system

Abstract

Provide a system. [Solution] means for capturing audio input; means for transmitting the captured audio data to a server; a means for the server to convert the voice data into text; A means for analyzing text data using natural language processing to identify the user's intent; A means for determining a task to be performed based on the specified intention and generating a command for the task; means for transmitting the generated command to a terminal and for the terminal to execute the command; A system including a means for notifying a user of the results of task execution.

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] To provide an environment where elderly people and users unfamiliar with PC operation can easily operate a computer. In particular, there is a need for a system that allows instructions to be given through voice input, accurately recognizes those instructions, and executes specific tasks. Existing systems often have difficulty accurately understanding the user's intentions, resulting in insufficient operation. This raises the barrier to using a PC and reduces user convenience. [Means for solving the problem]

[0005] The present invention solves the above-mentioned problems by providing a system including: means for capturing voice input; means for transmitting the captured voice data to a server; means for the server to convert the voice data into text; means for analyzing the text data using natural language processing to identify the user's intention; means for determining a task to be executed based on the identified intention and generating a command for that task; means for transmitting the generated command to a terminal and for the terminal to execute the command; and means for notifying the user of the result of task execution. This system allows users to easily operate a PC through voice, providing great convenience, especially for elderly people and users who are unfamiliar with PC operation.

[0006] "Voice input" refers to verbal instructions or commands given by a user and is voice data captured using a microphone or similar acoustic device.

[0007] "Capture" refers to recording or capturing data, such as voice input, and includes the process of temporarily storing that data.

[0008] "Audio Data" means a digital representation of captured audio input.

[0009] A "server" is a computer system for processing voice and text data, and performs the calculations and data management required to perform a specific task.

[0010] "Speech recognition" refers to the technology of converting captured voice data into text data, which is the preprocessing required for natural language processing.

[0011] "Natural language processing" is a technology that allows computers to understand human language, and involves analyzing text data to identify user intent.

[0012] "Intention" refers to the operation instructions or requests that the user is trying to convey through voice input.

[0013] A "task" refers to a specific computer operation that is executed based on the user's intention, and includes commands and processes for achieving a specific purpose.

[0014] A "command" is a statement that instructs a computer to perform a particular task.

[0015] A "terminal" is a device that is directly operated by a user, and that captures voice input and receives and executes commands from a server.

[0016] "Task execution result" refers to the outcome or state obtained after a terminal executes a specific command.

[0017] "Notification" is the act of informing the user of the results of the execution of a task, and includes audio or visual feedback. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0019] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0020] First, the terms used in the following description will be explained.

[0021] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0022] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0023] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0024] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0026] [First embodiment]

[0027] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0028] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0031] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0034] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0038] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0039] The present invention relates to a system that allows elderly people and users who are unfamiliar with PC operation to easily operate a computer. The system captures the user's voice input, analyzes their intention, and automatically executes the corresponding task.

[0040] System Overview

[0041] In this system, the user verbally commands an operation, and the device captures the speech and sends it to the server. The server performs speech recognition and natural language processing to identify the task to be performed and generate the command. The generated command is then sent to the device and executed. Finally, the user is notified of the result of the task execution.

[0042] Program processing overview

[0043] 1. Capture voice input

[0044] The device captures voice input from the user through a microphone, for example, by giving verbal commands such as "open a browser."

[0045] 2. Sending audio data

[0046] The device transmits the captured audio data to the server, which uses a secure communication method.

[0047] 3. Voice Recognition

[0048] The server converts the received voice data into text data, using a speech recognition engine on the server.

[0049] 4. Natural Language Processing

[0050] The server analyzes the text data and identifies the user's intent. For example, it uses a natural language processing engine to recognize the intent of "open a browser."

[0051] 5. Generate task execution commands

[0052] Based on the intent recognized, the server determines the task to be performed and generates the corresponding command, for example, "start chrome" as the command to "open a browser."

[0053] 6. Sending Commands

[0054] The server generates commands and sends them to the terminal.

[0055] 7. Executing commands

[0056] The device executes the received command and performs the specified task, in this case launching a browser.

[0057] 8. Notification of execution results

[0058] The device notifies the user of the result of the task execution, for example, by voice or on-screen notification that "the browser has been opened."

[0059] Specific examples

[0060] Example 1: Open a browser

[0061] 1. User: Says "Open a browser."

[0062] 2. Device: Captures the user's voice and sends it to the server.

[0063] 3. Server: Converts the voice data into text and recognizes it as "opening a browser."

[0064] 4. Server: Analyze the intent to "open a browser" using natural language processing.

[0065] 5. Server: Generate the command "start chrome".

[0066] 6. Server: Sends the generated commands to the terminal.

[0067] 7. Terminal: Execute the command and launch the browser.

[0068] 8. Terminal: Notify the user that the browser has been opened.

[0069] Example 2: Sending an email

[0070] 1. User: Says "Send me an email."

[0071] 2. Device: Captures the user's voice and sends it to the server.

[0072] 3. Server: Converts the voice data into text and recognizes it as "send an email."

[0073] 4. Server: Analyze the intent to "send an email" using natural language processing.

[0074] 5. Server: Obtains additional information from the user (recipient, subject, and body) needed to send the email.

[0075] 6. Server: Generates the command to "launch the mail client and send the email."

[0076] 7. Server: Sends the generated commands to the terminal.

[0077] 8. Terminal: Execute commands and send emails.

[0078] 9. Terminal: Notify the user that the email has been sent.

[0079] This system allows users to operate a computer using only voice input, providing great convenience, especially for elderly people and those who are unfamiliar with PC operation.

[0080] The processing flow will be explained below.

[0081] Step 1:

[0082] The user says, "Open a browser." The user verbally instructs the computer to operate, and this instruction is the starting point for the entire system.

[0083] Step 2:

[0084] The device captures the user's voice input through a microphone, where the user's speech is temporarily stored in memory as audio data.

[0085] Step 3:

[0086] The device sends the captured audio data to the server using an HTTP request, using a secure communication method.

[0087] Step 4:

[0088] The server passes the received voice data to a voice recognition engine and converts it into text data. For example, the voice saying "Open the browser" is analyzed and converted into text.

[0089] Step 5:

[0090] The server passes the text data to a natural language processing engine, which analyzes the user's intent. Here, the combination of the word "browser" and the verb "open" identifies the intent as "open a browser."

[0091] Step 6:

[0092] The server determines the task based on the specified intent and generates a command to perform that task, for example "start chrome".

[0093] Step 7:

[0094] The server generates commands and sends them to the device, which then receives them using HTTP requests.

[0095] Step 8:

[0096] The terminal executes the received command. In this case, a browser (for example, GOOGLE CHROME (registered trademark)) is launched by the received "start chrome" command.

[0097] Step 9:

[0098] The terminal notifies the user of the result of the task execution. If the browser is successfully launched, the user is given a voice or visual message saying "Browser opened."

[0099] Example 1

[0100] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0101] There was a need to provide an environment where elderly people and users unfamiliar with PC operation could use computers easily without complex operations. However, conventional systems had low accuracy in recognizing voice input and instructions, and intended operations were often not executed. This reduced user convenience and impaired the practicality of voice operation.

[0102] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0103] In this invention, the server includes means for converting voice data into text data, means for analyzing the text data using natural language processing technology to identify the user's intention, and means for determining the process to be performed based on the identified intention and generating instructions for that process. This makes it possible to recognize the user's intention with high accuracy and generate and execute appropriate instructions.

[0104] "Means for capturing voice input" refers to a device or method that captures voice spoken by a user and converts it into a digital signal.

[0105] "Means for transmitting captured audio data to a computer using a communication network" refers to the technology or method for transferring acquired audio data to a computer located in a remote location, such as a server, via a network such as the Internet.

[0106] "Means for converting voice data into text data by an electronic computer" refers to the process by which a server or computer uses voice recognition technology to convert voice into text information.

[0107] "Means of analyzing text data using natural language processing technology to identify the user's intent" refers to the technology and methods for analyzing acquired text data and understanding its meaning and requests.

[0108] "Means for determining the processing to be performed based on the identified intent and generating instructions for that" refers to the process of determining what operations or processing to perform based on the analysis results and generating the corresponding commands or instructions.

[0109] "Means for sending the generated instructions to the terminal and for the terminal to execute the instructions" refers to a technology or method for sending operation instructions generated by a server to a user's device and for the device to execute the received instructions.

[0110] "Means for notifying the user of the results of processing" refers to techniques or methods for visually or audibly conveying the results of the processing to the user.

[0111] "Electronic computer" means a computing device, server, or other computing equipment capable of receiving, processing, and analyzing data.

[0112] "Audio collection device" means a device for collecting a user's voice, including a microphone or other audio input device.

[0113] This invention provides a system that allows elderly people and users who are unfamiliar with PC operation to easily operate a computer. Specifically, it is a system that performs various tasks using voice input. This system uses the following main hardware and software:

[0114] Hardware and software used

[0115] 1. Terminal (user device)

[0116] Microphone: An audio collection device used to capture a user's voice input. This includes a computer's built-in microphone and an external USB microphone.

[0117] Network interface: A communication device for sending audio data to a server. An internet connection is required.

[0118] 2. Server

[0119] Speech recognition engine: Software for converting voice data into text data. For example, Google® Cloud Speech-to-Text API is used.

[0120] Natural language processing engine: Software used to analyze text data and identify user intent. For example, the Google Cloud Natural Language API is used.

[0121] Task execution command generation engine: Software that determines the process to be performed based on the user's intentions and generates the instructions.

[0122] System examples and operation procedures

[0123] As a concrete example for implementing this system, the tasks "open a browser" and "send an email" are taken.

[0124] Example 1: Open a browser

[0125] 1. User: Says "Open a browser."

[0126] 2. Terminal: Captures the user's voice with a microphone and sends the voice data to the server, which is securely transmitted using TLS.

[0127] 3. Server: Receives the audio data and converts it to text using the Google Cloud Speech-to-Text API.

[0128] 4. Server: Analyze the text data using the Google Cloud Natural Language API to identify the intent to "open a browser."

[0129] 5. Server: Generates a task execution command called "start chrome" based on the instruction "open browser".

[0130] 6. Server: Sends the generated commands to the terminal.

[0131] 7. Terminal: Executes the received command and launches the browser.

[0132] 8. On the device: The user is notified by voice or on screen that the browser has been opened.

[0133] Example 2: Sending an email

[0134] 1. User: Says "Send me an email."

[0135] 2. Terminal: Captures the user's voice with a microphone and sends the voice data to the server, which is securely transmitted using TLS.

[0136] 3. Server: Receives the audio data and converts it to text using the Google Cloud Speech-to-Text API.

[0137] 4. Server: Analyze the text data using the Google Cloud Natural Language API and identify the intent to "send an email."

[0138] 5. Server: Obtain additional information from the user (recipient, subject, and message) needed to send the email. For example, confirm "recipient is example@example.com, subject is test, message is hello."

[0139] 6. Server: Generates a command to "launch the mail client and send the email" and includes the necessary information.

[0140] 7. Server: Sends the generated commands to the terminal.

[0141] 8. Terminal: Executes the received command and sends the email.

[0142] 9. Terminal: Notify the user by voice or on-screen message that "Email has been sent."

[0143] This system allows users to easily operate a computer by voice input, providing great convenience especially for the elderly and people who are unfamiliar with using computers.

[0144] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0145] Step 1:

[0146] Input: User's voice input.

[0147] How it works: The device uses the microphone to capture the user's voice input, as the user speaks into the microphone, "open browser."

[0148] Output: Audio data in digital format.

[0149] Step 2:

[0150] Input: Audio data in digital format.

[0151] How it works: The device captures audio data and sends it to the server using a secure communication protocol (e.g., HTTPS). The audio data is encrypted with TLS and transmitted securely over the network.

[0152] Output: The audio data sent to the server.

[0153] Step 3:

[0154] Input: The audio data sent to the server.

[0155] How it works: The server converts the received voice data into text data using the Google Cloud Speech-to-Text API. The server's speech recognition engine analyzes the voice data and generates a string of text.

[0156] Output: Data in text format (e.g. "Open browser").

[0157] Step 4:

[0158] Input: Data in text format.

[0159] How it works: The server parses the text data using the Google Cloud Natural Language API. A natural language processing engine on the server tokenizes the text data and identifies the intent "open a browser." This process involves morphological analysis and contextual understanding.

[0160] Output: The user intent (e.g., open a browser).

[0161] Step 5:

[0162] Input: User intent.

[0163] How it works: The server generates an execution command such as "start chrome" based on the user's intent. The command generation engine on the server creates a script or shell command corresponding to the specified task and validates it as necessary.

[0164] Output: The generated command to run (e.g. start chrome).

[0165] Step 6:

[0166] Input: The generated execution command.

[0167] How it works: The server sends the generated command to the device. A secure communication protocol is used to ensure the command reaches the device. The command is encrypted and then decrypted by the device.

[0168] Output: Execution instructions sent to the terminal.

[0169] Step 7:

[0170] Input: Execution instructions sent to the terminal.

[0171] Operation: The device executes the received command and performs the specified task. In this case, the "start chrome" command is executed and the browser is launched. The device monitors the command execution result and determines whether it was successful or failed.

[0172] Output: The result of the task execution (e.g., a browser was opened).

[0173] Step 8:

[0174] Input: The result of the task execution.

[0175] Action: The device notifies the user of the execution result. For example, it uses a speech synthesis engine (e.g., Google Text-to-Speech) to announce "Browser opened" by voice, or displays the message "Browser opened" on the screen.

[0176] Output: Notification to the user (e.g. audio notification, screen display).

[0177] (Application example 1)

[0178] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0179] Security management systems require easy voice control of important security functions, such as camera operation, alarm setting, locking and unlocking, without requiring complex operations, especially for elderly users and those unfamiliar with technology. However, existing technologies lack a consistent system for achieving such voice control. Other issues include the accuracy of voice recognition, appropriate generation of execution commands, and clear notification of execution results to users.

[0180] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0181] In this invention, the server includes means for capturing voice input, means for transmitting the captured voice data to the server, means for the server to convert the voice data into text, means for analyzing the text data using natural language processing and identifying the user's intention, means for determining a task to be executed based on the identified intention and generating a command for that task, means for transmitting the generated command to the terminal and having the terminal execute the command, means for controlling security functions such as operating a camera, setting an alarm, locking or unlocking using voice input, and means for notifying the user of the result of task execution. This enables even elderly people and users who are unfamiliar with technology to easily control security functions using only voice input.

[0182] A "means for capturing voice input" is a technical device for detecting voice from a user and collecting that data.

[0183] The "means for transmitting captured voice data to a server" is a technical device that transmits collected voice data to a server using a communication means such as the Internet.

[0184] The "means for the server to convert voice data into text" refers to a technical device that converts voice data into text information using a voice recognition engine.

[0185] "Means for analyzing text data using natural language processing and identifying user intent" refers to a technical device that uses natural language processing technology to analyze text data and understand what the user wants.

[0186] The "means for determining a task to be executed based on the identified intention and generating a command for that task" is a technical device that determines an operation according to the user's intention and creates a corresponding command.

[0187] The "means for sending the generated command to the terminal and for the terminal to execute the command" is a technical device that sends the generated command to the terminal and the terminal executes the task according to the command.

[0188] "Means for controlling security functions such as operating cameras, setting alarms, locking and unlocking doors, etc. using voice input" refers to a technical device that uses voice input to operate security functions such as changing the direction of cameras, setting alarms, and locking and unlocking doors.

[0189] The "means for notifying the user of the results of the task execution" is a technical device that notifies the user of the results of the executed task by voice, display, or other means.

[0190] The present invention is a system for controlling a security control system using voice input, which takes voice input from a user and transmits the voice data to a server, which analyzes the voice data, identifies a corresponding task, and generates a command to perform the task.

[0191] The system includes the following components:

[0192] 1. Capture voice input

[0193] The user issues a command by voice, such as "point the camera at the living room." The microphone on the device captures this voice. The hardware used is a standard voice capture device.

[0194] 2. Sending audio data to the server

[0195] The device transmits the captured audio data to a server via a communication method such as the Internet. The communication uses a secure protocol (e.g., HTTPS).

[0196] 3. Voice Recognition

[0197] The server analyzes the received voice data and converts it into text information. The software used for this is a speech recognition engine (for example, Google Speech-to-Text API). The converted text data is used in the next step.

[0198] 4. Natural Language Processing

[0199] The server analyzes the text data obtained by the speech recognition engine using a natural language processing engine (e.g., spaCy or NLTK) to identify the user's intent. For example, the server recognizes the instruction "point the camera toward the living room" as "adjust the camera toward the living room."

[0200] 5. Generate task execution commands

[0201] The server determines the task to be performed based on the user's intent and generates a command for that task. For example, for the intent "point the camera at the living room," the server generates a command to "change the camera's orientation to the living room."

[0202] 6. Sending commands to the terminal and executing them

[0203] The generated command is then sent back to the device, which then executes it. For example, a mechanical control system for adjusting the orientation of a camera on the device is activated.

[0204] 7. Notifying the user of the task execution results

[0205] The device notifies the user of the execution result of the instructed task. For example, the device may notify the user by voice or by displaying a message on the screen saying, "The camera has been redirected to the living room."

[0206] Specific examples

[0207] Example 1: Camera operation

[0208] 1. The user says, "Point the camera at the living room."

[0209] 2. The device captures this audio and sends it to the server.

[0210] 3. The server converts the voice data into text and recognizes it as "point the camera towards the living room."

[0211] 4. The server parses this intent and generates a command to point the camera towards the living room.

[0212] 5. The generated command is sent to the device, and the camera turns as instructed.

[0213] 6. The device will announce, "The camera has been pointed at the living room."

[0214] Prompt Sentence Examples

[0215] When the user says "point the camera towards the living room," how does the system adjust the camera's orientation to face the living room? Please explain step by step.

[0216] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0217] Step 1:

[0218] Capturing voice input

[0219] The user issues a command, for example, "point the camera at the living room," and the microphone on the device captures the user's voice. The input is voice data, which is used for the next step.

[0220] Step 2:

[0221] Sending audio data to the server

[0222] The device sends the captured audio data to the server over the Internet using a secure protocol (e.g., HTTPS). The input is the captured audio data, and the output is the data sent to the server.

[0223] Step 3:

[0224] Voice Recognition

[0225] The server analyzes the received voice data and converts it into text information. The software used is a speech recognition engine (e.g., Google Speech-to-Text API). The input is voice data and the output is text data. Specifically, the speech recognition engine extracts acoustic features and matches them with known character patterns.

[0226] Step 4:

[0227] Natural Language Processing

[0228] The server analyzes the text data obtained through speech recognition to determine the user's intent. The software used is a natural language processing engine (e.g., spaCy or NLTK). The input is text data, and the output is information indicating the user's intent. Specifically, grammar analysis and context understanding are performed.

[0229] Step 5:

[0230] Generate task execution commands

[0231] The server determines the task to be executed based on the identified user intent and generates the command for that task. The input is information indicating the user intent, and the output is the specific command to be executed. For example, a command such as "change the camera's orientation to the living room" may be generated.

[0232] Step 6:

[0233] Sending commands to the terminal and executing them

[0234] The server sends the generated command to the terminal, which then executes it. The input is the generated command, and the output is the result of the command execution. Specifically, the terminal controls the camera motor and changes the camera's orientation as instructed.

[0235] Step 7:

[0236] Notifying users of task execution results

[0237] The device notifies the user of the results of the task. The input is the result of the command execution, and the output is a notification to the user. Specifically, the device tells the user, "The camera has been pointed at the living room," through a voice output device or a screen display.

[0238] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0239] This invention relates to a system that allows elderly people and users who are unfamiliar with PC operation to operate a computer easily. This system takes in the user's voice input, analyzes their intention, and automatically executes the corresponding task. It also incorporates an emotion engine that recognizes the user's emotions and enables responses that take those emotions into consideration.

[0240] System Overview

[0241] In this system, the user verbally commands an operation, and the device captures the speech and sends it to the server. The server performs speech recognition and natural language processing to identify the task to be performed and generate the command. The generated command is then sent to the device and executed. The system also recognizes the user's emotions and takes those emotions into consideration when carrying out the task and providing feedback. Finally, the user is notified of the results of the task.

[0242] Program processing overview

[0243] 1. Capture voice input

[0244] The device captures voice input from the user through a microphone. The user verbally issues commands such as "open a browser." The voice input may contain emotions.

[0245] 2. Sending audio data

[0246] The device sends the captured audio data to the server using an HTTP request, using secure communication methods.

[0247] 3. Voice Recognition

[0248] The server passes the received voice data to a voice recognition engine and converts it into text data. For example, the voice saying "Open the browser" is analyzed and converted into text.

[0249] 4. Natural Language Processing

[0250] The server analyzes the text data and identifies the user's intent. For example, it uses a natural language processing engine to recognize the intent of "open a browser."

[0251] 5. Emotion recognition

[0252] The server analyzes the user's emotions from the voice data. Using an emotion engine, it analyzes the tone and speed of the voice to identify the user's emotional state. For example, it can recognize whether the user is angry or happy.

[0253] 6. Generate task execution commands

[0254] The server determines the task to be performed based on the recognized intent and emotion, and generates a command for it. For example, the command "start chrome" is generated.

[0255] 7. Sending Commands

[0256] The server generates commands and sends them to the terminal. The commands are sent using HTTP requests.

[0257] 8. Executing Commands

[0258] The terminal executes the received command. In this case, the received command "start chrome" launches a browser (for example, Google Chrome).

[0259] 9. Notification of execution results

[0260] The device notifies the user of the results of the task. Based on emotion recognition, if the user is angry, for example, a message such as "I've opened the browser as you requested. Is there anything else I can help you with?" is provided to the user via voice or visual means.

[0261] Specific examples

[0262] Example 1: Open a browser

[0263] 1. User: "Open a browser." At this point, the user is in a hurry, so their voice is high-pitched and fast.

[0264] 2. Device: Captures audio and sends it to the server.

[0265] 3. Server: Converts the voice data into text and recognizes it as "opening a browser."

[0266] 4. Server: Analyze the intent to "open a browser" using natural language processing.

[0267] 5. Server: The emotion engine detects urgent situations and prioritizes quick responses.

[0268] 6. Server: Generate the command "start chrome".

[0269] 7. Server: Sends the generated commands to the terminal.

[0270] 8. Terminal: Run the command and launch the browser.

[0271] 9. On the device: The user is notified, "I've opened your browser. I see you're short on time. Is there anything else I can help you with?"

[0272] Example 2: Sending an email

[0273] 1. User: "Send me an email." The user is calm and their voice is level.

[0274] 2. Device: Captures audio and sends it to the server.

[0275] 3. Server: Converts the voice data into text and recognizes it as "send an email."

[0276] 4. Server: Analyze the intent to "send an email" using natural language processing.

[0277] 5. Server: The emotion engine detects the calm situation and proceeds with the normal procedure.

[0278] 6. Server: Obtains additional information from the user (recipient, subject, and body) needed to send the email.

[0279] 7. Server: Generates the command to "launch the mail client and send the email."

[0280] 8. Server: Sends the generated commands to the terminal.

[0281] 9. Terminal: Execute commands and send emails.

[0282] 10. Terminal: Notify the user that "Email has been sent. Please check."

[0283] In this way, the system recognizes the user's voice commands and emotions, converts them into specific PC operations, and automatically executes them, providing great convenience to elderly people and users who are unfamiliar with PC operation.

[0284] The processing flow will be explained below.

[0285] Step 1:

[0286] The user says, "Open a browser." This verbal command is the starting point of the entire system. The user's voice contains their current emotional state.

[0287] Step 2:

[0288] The device captures the user's voice input through a microphone, where the user's speech is temporarily stored in memory as audio data.

[0289] Step 3:

[0290] The device sends the captured audio data to the server using an HTTP request, using secure communication methods.

[0291] Step 4:

[0292] The server passes the received voice data to a voice recognition engine and converts it into text data. For example, the voice saying "Open the browser" is analyzed and converted into text.

[0293] Step 5:

[0294] The server passes the text data to a natural language processing engine to identify the user's intent. Here, the combination of the word "browser" and the verb "open" identifies the intent as "open a browser."

[0295] Step 6:

[0296] The server analyzes the user's emotions from the voice data, and the emotion engine analyzes voice characteristics such as tone, speed, and emphasis to identify the user's emotional state, such as whether they are in a hurry or relaxed.

[0297] Step 7:

[0298] Based on the identified intent and emotion, the server determines the task to be performed and generates the command. For example, if the user is in a hurry, the command "start chrome" is generated to "open the browser immediately," eliminating the additional confirmation step.

[0299] Step 8:

[0300] The server generates commands and sends them to the terminal. The commands are sent using HTTP requests.

[0301] Step 9:

[0302] The terminal executes the received command. In this case, the received command "start chrome" launches a browser (for example, Google Chrome).

[0303] Step 10:

[0304] The device notifies the user of the results of the task execution. Based on the emotion recognition results, if the user is in a hurry, a message such as "I've opened the browser. Is there anything else I can help you with?" is provided to the user via voice or visual means.

[0305] Specific examples

[0306] Example 1: Open a browser

[0307] 1. The user says, "Open a browser." The voice is quick and high-pitched because they're in a hurry.

[0308] 2. The device captures the audio and sends it to the server.

[0309] 3. The server converts the voice data into text and recognizes it as "opening a browser."

[0310] 4. The server uses natural language processing to analyze the intent to "open a browser."

[0311] 5. The server detects urgent situations using an emotion engine.

[0312] 6. The server generates the command "start chrome".

[0313] 7. The server sends the generated command to the terminal.

[0314] 8. The terminal executes the command and launches the browser.

[0315] 9. The device notifies the user, "Your browser is open. Is there anything else I can help you with?"

[0316] Example 2: Sending an email

[0317] 1. The user says, "Send me an email." The user is calm and the voice is even.

[0318] 2. The device captures the audio and sends it to the server.

[0319] 3. The server converts the voice data into text and recognizes it as "send an email."

[0320] 4. The server uses natural language processing to analyze the intent to "send an email."

[0321] 5. The server detects calm situations using the emotion engine.

[0322] 6. The server obtains additional information from the user (recipient, subject, and body) needed to send the email.

[0323] 7. The server generates a command to launch the mail client and send the email.

[0324] 8. The server sends the generated command to the terminal.

[0325] 9. The terminal executes the command and sends the email.

[0326] 10. The device notifies the user, "Email has been sent. Please check."

[0327] In this way, the system recognizes the user's voice commands and emotions, converts them into specific PC operations, and automatically executes them, providing great convenience to elderly people and users who are unfamiliar with PC operation.

[0328] Example 2

[0329] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0330] Today's elderly and users unfamiliar with PC operation face many difficulties when operating computers. In particular, conventional user interfaces require visual and tactile operation, which poses technical hurdles. For this reason, more intuitive and easier operation methods are needed. Furthermore, there is a lack of support that takes into account the user's emotional state. This increases stress and burden on users, resulting in a decrease in frequency of use.

[0331] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for converting voice data into text, a means for analyzing the text data by natural language processing to identify the user's intention, and a means for analyzing the user's emotions from the voice data. This allows the user to intuitively operate the computer using only voice input, and further enables the computer to respond according to their emotional state.

[0332] "Voice input" refers to instructions or information verbally given by a user.

[0333] "Capture" refers to the process of recording and importing data such as voice input.

[0334] "Audio Data" means captured audio converted into digital form.

[0335] A "server" is a computer system that processes data over a network and provides the results to other devices.

[0336] "Text conversion" refers to the process of converting audio data into corresponding text data.

[0337] "Natural language processing" is a technology that uses computers to analyze and understand human language.

[0338] "Intent" refers to the goal or request that the user is trying to achieve through voice input.

[0339] "Emotion recognition" is a technology that analyzes and identifies a user's emotional state from voice data.

[0340] A "task" is a specific operation or process that a computer or terminal must perform.

[0341] A "command" is a specific instruction or direction that a computer system is to carry out.

[0342] A "terminal" is a device with input and output capabilities that is directly operated by a user.

[0343] "Execution results" refers to the results or output obtained after a computer or terminal executes a command.

[0344] "Notification" refers to the act of informing the user of execution results or information.

[0345] This invention relates to a system that allows elderly people and users who are unfamiliar with PC operation to easily operate a computer. This system captures the user's voice input, analyzes their intention, and automatically executes the corresponding task. It also incorporates an emotion engine that recognizes the user's emotions and enables responses that take those emotions into consideration.

[0346] System Overview

[0347] The system consists of the following main components:

[0348] 1. A means to capture audio input (microphone)

[0349] 2. A means of sending the captured audio data to the server

[0350] 3. A means of converting voice data into text on the server side (voice recognition engine)

[0351] 4. A means of analyzing text data using natural language processing to identify the user's intent (natural language processing engine)

[0352] 5. A means of analyzing user emotions from voice data (emotion engine)

[0353] 6. A means to determine the task to be performed based on the identified intention and emotion and generate the command.

[0354] 7. A means for sending the generated command to the terminal and for the terminal to execute the command

[0355] 8. Means of notifying the user of the results of the task execution

[0356] Specific operation explanation

[0357] 1. Capture voice input

[0358] The device captures voice input from the user. The device has a built-in microphone and records the user's commands in real time. For example, the user might say, "Open a browser."

[0359] 2. Sending audio data

[0360] The device sends the captured audio data to the server, where it is encrypted and transmitted over a secure channel.

[0361] 3. Voice Recognition

[0362] The server passes the received voice data to a speech recognition engine, which converts the voice into text data. The speech recognition engine uses the Google Cloud Speech-to-Text API or similar. For example, the voice saying "Open your browser" is converted into text saying "Open your browser."

[0363] 4. Natural Language Processing

[0364] The server passes the text data to a natural language processing engine, which analyzes the user's intent. The natural language processing engine uses Google Cloud Natural Language API or similar. From the analysis results, the specific intent of "open a browser" is identified.

[0365] 5. Emotion recognition

[0366] The server passes the voice data to an emotion recognition engine, which analyzes the tone and speed of the voice to recognize the user's emotions. The emotion recognition engine uses Microsoft® Azure® Emotion API, etc. For example, if the user is in a hurry, the server will understand their emotional state.

[0367] 6. Generate task execution commands

[0368] The server determines the task to be performed based on the user's intention and emotion, and generates the command for it. Specifically, the command "start chrome" is generated.

[0369] 7. Sending Commands

[0370] The server sends the generated command to the terminal using a secure communication method.

[0371] 8. Executing Commands

[0372] The device executes the received command, for example launching Google Chrome.

[0373] 9. Notification of execution results

[0374] The device notifies the user of the results of the task. Based on the emotion recognition results, feedback appropriate to the user's situation is provided. For example, a message such as "I've opened your browser. It seems you're short on time. Is there anything else I can help you with?" is displayed or spoken.

[0375] Specific examples

[0376] Example 1: Open a browser

[0377] 1. The user says, "Open a browser."

[0378] 2. The device captures the audio and sends it to the server.

[0379] 3. The server converts the voice data into text and recognizes it as "opening a browser."

[0380] 4. The server analyzes the intent.

[0381] 5. The server recognizes the user's sense of urgency from the voice.

[0382] 6. The server generates the command "start chrome".

[0383] 7. The server sends the command to the device.

[0384] 8. The terminal executes the command and launches the browser.

[0385] 9. Your device will say, "I've opened your browser. I see you're short on time. Is there anything else I can help you with?"

[0386] Example 2: Sending an email

[0387] 1. The user says, "Send me an email."

[0388] 2. The device captures the audio and sends it to the server.

[0389] 3. The server converts the voice data into text and recognizes it as "send an email."

[0390] 4. The server analyzes the intent.

[0391] 5. The server recognizes the user's neutral emotion from the voice.

[0392] 6. The server generates a prompt asking for additional information.

[0393] 7. The server sends a prompt to the terminal to get additional information from the user.

[0394] 8. The server generates a command to launch the email client and send the email.

[0395] 9. The server sends the command to the device.

[0396] 10. The terminal executes the command and sends the email.

[0397] 11. The device will notify you that "Email has been sent. Please check."

[0398] In this way, the system analyzes the user's voice commands in detail, converts them into specific PC operations, and automatically executes them, providing convenience that allows even elderly people and users who are unfamiliar with PC operation to easily operate a computer.

[0399] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0400] Step 1:

[0401] The device captures voice input from the user through the microphone. The input is the user's speech, for example, "Open the browser," in the form of voice data. The device receives this voice as an analog signal, converts it to a digital signal, and temporarily stores it in a buffer. Specifically, the device's built-in microphone converts the voice signal into digital data (for example, WAV format).

[0402] Step 2:

[0403] The device sends the captured audio data to the server. The input is the digital audio data obtained in step 1. This data is sent to the server using the HTTPS protocol. Specifically, the device's communication module creates an HTTP POST request and sends the audio data as a payload to the server.

[0404] Step 3:

[0405] The server passes the received voice data to a speech recognition engine, which converts the voice into text data. The input is the digital voice data sent in step 2. The output is the converted text data. Specifically, the server uses the Google Cloud Speech-to-Text API to convert the voice "Open your browser" into the text "Open your browser."

[0406] Step 4:

[0407] The server passes the text data to a natural language processing engine to analyze the user's intent. The input is the text data obtained in step 3. The output is data that identifies the user's intent. Specifically, the server uses the Google Cloud Natural Language API to analyze the text "open a browser" and identify the intent as "launch a browser."

[0408] Step 5:

[0409] The server analyzes the user's emotions from the voice data. The input is the voice data acquired in step 2. The output is the user's emotional state. Specifically, the server uses the Microsoft Azure Emotion API to analyze the tone and speed of the voice and determine whether the user is in a hurry.

[0410] Step 6:

[0411] The server determines the task to be performed based on the user's intention and emotion, and generates the command. The input is the user's intention identified in step 4 and the emotional state analyzed in step 5. The output is a specific command. Specifically, the server generates the command "start chrome".

[0412] Step 7:

[0413] The server sends the generated command to the terminal. The input is the command generated in step 6. The output is the command sent to the terminal. Specifically, the server's communication module creates an HTTP POST request and sends the command as a payload to the terminal.

[0414] Step 8:

[0415] The terminal executes the received command. The input is the command received in step 7. The output is the result of executing the command. Specifically, the terminal interprets the command "start chrome" and launches Google Chrome.

[0416] Step 9:

[0417] The terminal notifies the user of the results of the task execution. The input is the result of the command execution in step 8. The output is feedback to the user. Specifically, the terminal displays a message audibly or visually saying, "I've opened your browser. It looks like you're short on time. Is there anything else I can help you with?"

[0418] (Application example 2)

[0419] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0420] Conventional systems lack the means to intuitively operate computers and digital devices for elderly users and users unfamiliar with computer operation. This problem is particularly pronounced when complex operations are required. Furthermore, the lack of a system that flexibly responds to the user's emotions often results in a poor user experience. This creates a challenge for elderly users and users unfamiliar with technology to continue using digital environments.

[0421] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0422] In this invention, the server includes means for capturing voice input, means for transmitting the captured voice data to the server, means for the server to convert the voice data into text, means for analyzing the text data using natural language processing to identify the user's intention, means for analyzing the user's emotional state and providing flexible task execution and feedback based on the emotion, means for determining a task to be executed based on the identified intention and emotion and generating a command for the task, means for transmitting the generated command to a terminal and having the terminal execute the command, means for notifying the user of the task execution result, and navigation means for guiding the user using smart glasses. This enables even elderly people and users unfamiliar with computer operation to intuitively operate the device through voice input and smoothly use a digital environment while receiving flexible responses according to their emotional state.

[0423] A "means for capturing voice input" is a device or function for capturing voice uttered by a user.

[0424] The "means for transmitting captured audio data to a server" refers to a communication means or protocol for transmitting captured audio data to a server.

[0425] The "means for the server to convert voice data into text" is a voice recognition engine for converting received voice data into text format.

[0426] "Means for analyzing text data using natural language processing and identifying the user's intent" refers to a natural language processing engine that analyzes the converted text data to understand the user's instructions and intent.

[0427] "Means for analyzing the user's emotional state and providing flexible task execution and feedback based on those emotions" refers to an emotion recognition engine that analyzes the user's emotions from voice and other inputs and provides task execution and feedback appropriate to those emotions.

[0428] The "means for determining a task to be executed based on the identified intention and emotion and generating a command for that task" is a means for determining the optimal task according to the user's intention and emotional state and generating a command for that task.

[0429] The "means for transmitting the generated command to the terminal and for the terminal to execute the command" refers to a communication means for transmitting the generated command to the device and for causing the device to execute the command.

[0430] The "means for notifying the user of the execution result of the task" is a means for notifying the user of the execution result of the task.

[0431] A "navigation means for guiding a user using smart glasses" is a navigation system for providing visual and audio guidance and instructions to a user through smart glasses.

[0432] This invention is a system that allows elderly people and users who are unfamiliar with operating personal computers to easily operate a computer. A specific example of this system is shown below.

[0433] The main elements of the system include means for capturing a user's voice input, means for transmitting the captured voice data to a server, means for converting the voice data to text on the server side, means for analyzing the text data using natural language processing to identify the user's intention, means for analyzing the user's emotional state, means for determining a task to be performed based on the intention and emotion and generating a command for that task, means for transmitting the generated command to a terminal and for the terminal to execute the command, means for notifying the user of the result of the task execution, and navigation means for guiding the user using smart glasses.

[0434] Specifically, the system works as follows: The user wears the smart glasses and gives voice instructions (e.g., "Where is the tea shelf?"). The microphone in the smart glasses captures this voice and sends the voice data to the server using HTTPS communication. The server converts the voice data into text using Google Cloud Speech-to-Text.

[0435] The system then uses the Google Cloud Natural Language engine to analyze the converted text data and identify the user's intent. At the same time, it uses the Azure Emotion API to analyze the user's emotional state from the audio data. For example, if the system determines that the user is confused, that emotional state is also taken into account.

[0436] Based on the identified intent and emotion, the server determines the optimal task and generates a command to execute it. For example, commands such as "Start navigation" or "Guide to the tea shelf." This generated command is then sent to the smart glasses via HTTPS. The smart glasses then execute the received command and navigate the store using the built-in Google Maps API.

[0437] During navigation, the smart glasses provide real-time visual and audio guidance to the user, such as a message saying, "This way is the tea corner. Let us show you around." Once navigation is complete, feedback is also provided, such as, "You've arrived at the tea corner. Can we help you?"

[0438] Specific examples

[0439] Example 1: Guidance to the tea corner

[0440] 1. User: "Where is the tea cupboard?"

[0441] 2. Smart glasses: Capture the user's voice and send it to the server.

[0442] 3. Server: Converts speech to text and analyzes intent. Recognizes "Search for tea shelves."

[0443] 4. Server: The emotion engine recognizes "confusion."

[0444] 5. Server: Generate navigation command. "Guide me to the tea corner."

[0445] 6. Smart glasses: Start navigation and visually and audibly display "The tea corner is here. We will guide you."

[0446] 7. Smart glasses: After the guide is completed, the glasses will notify you, "You have arrived at the tea corner. Can I help you?"

[0447] Prompt Sentence Examples

[0448] The user puts on the smart glasses and asks a question by voice, for example, "Where is the tea cupboard?"

[0449] The smart glasses capture this audio and send it to the server. The server converts the audio into text using Google Cloud Speech-to-Text and identifies the intent using Google Cloud Natural Language. It then analyzes the user's emotions using the Azure Emotion API. Based on the acquired information, the server generates appropriate navigation commands and sends them to the smart glasses. The smart glasses then use the Google Maps API to perform real-time navigation and provide audio and visual guidance to the user.

[0450] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0451] Step 1:

[0452] Input: User's spoken command (e.g., "Where is the tea cupboard?")

[0453] How it works: The microphone on the device (smart glasses) captures the user's voice.

[0454] Output: Captured audio data

[0455] Step 2:

[0456] Input: Captured audio data

[0457] How it works: The device sends the captured audio data to the server using HTTPS.

[0458] Output: Audio data sent to the server

[0459] Step 3:

[0460] Input: Audio data received by the server

[0461] How it works: The server uses a speech recognition engine (Google Cloud Speech-to-Text) to convert the voice data into text.

[0462] Output: Text data (e.g., "Where is the tea shelf?")

[0463] Step 4:

[0464] Input: Converted text data

[0465] How it works: The server uses a natural language processing engine (Google Cloud Natural Language) to analyze the text data and determine the user's intent.

[0466] Output: Identified intent (e.g., "Find a tea cupboard")

[0467] Step 5:

[0468] Input: Audio data received by the server

[0469] How it works: The server uses an emotion recognition engine (Azure Emotion API) to analyze the user's emotional state from the voice data.

[0470] Output: User's emotional state (e.g., "confused")

[0471] Step 6:

[0472] Input: Identified intent and user emotional state

[0473] Action: The server determines the task to be performed based on the recognized intent and emotion, and generates the command.

[0474] Output: Generated command (e.g. "Take me to the tea corner")

[0475] Step 7:

[0476] Input: Generated command

[0477] Operation: The server sends the generated command to the device using HTTPS communication.

[0478] Output: Command sent to the terminal

[0479] Step 8:

[0480] Input: The command received by the terminal

[0481] How it works: The device (smart glasses) starts real-time navigation using the Google Maps API.

[0482] Output: Visual and audio navigation instructions

[0483] Step 9:

[0484] Input: Navigation result

[0485] Behavior: The device notifies the user of the results of the task, for example by displaying a message like "You've arrived at the tea corner. How can I help you?"

[0486] Output: Feedback notification to the user

[0487] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0488] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0489] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0490] [Second embodiment]

[0491] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0492] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0493] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0494] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0495] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0496] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0497] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0498] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0499] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0500] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0501] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0502] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0503] The present invention relates to a system that allows elderly people and users who are unfamiliar with PC operation to easily operate a computer. The system captures the user's voice input, analyzes their intention, and automatically executes the corresponding task.

[0504] System Overview

[0505] In this system, the user verbally commands an operation, and the device captures the speech and sends it to the server. The server performs speech recognition and natural language processing to identify the task to be performed and generate the command. The generated command is then sent to the device and executed. Finally, the user is notified of the result of the task execution.

[0506] Program processing overview

[0507] 1. Capture voice input

[0508] The device captures voice input from the user through a microphone, for example, by giving verbal commands such as "open a browser."

[0509] 2. Sending audio data

[0510] The device transmits the captured audio data to the server, which uses a secure communication method.

[0511] 3. Voice Recognition

[0512] The server converts the received voice data into text data, using a speech recognition engine on the server.

[0513] 4. Natural Language Processing

[0514] The server analyzes the text data and identifies the user's intent. For example, it uses a natural language processing engine to recognize the intent of "open a browser."

[0515] 5. Generate task execution commands

[0516] Based on the intent recognized, the server determines the task to be performed and generates the corresponding command, for example, "start chrome" as the command to "open a browser."

[0517] 6. Sending Commands

[0518] The server generates commands and sends them to the terminal.

[0519] 7. Executing commands

[0520] The device executes the received command and performs the specified task, in this case launching a browser.

[0521] 8. Notification of execution results

[0522] The device notifies the user of the result of the task execution, for example, by voice or on-screen notification that "the browser has been opened."

[0523] Specific examples

[0524] Example 1: Open a browser

[0525] 1. User: Says "Open a browser."

[0526] 2. Device: Captures the user's voice and sends it to the server.

[0527] 3. Server: Converts the voice data into text and recognizes it as "opening a browser."

[0528] 4. Server: Analyze the intent to "open a browser" using natural language processing.

[0529] 5. Server: Generate the command "start chrome".

[0530] 6. Server: Sends the generated commands to the terminal.

[0531] 7. Terminal: Execute the command and launch the browser.

[0532] 8. Terminal: Notify the user that the browser has been opened.

[0533] Example 2: Sending an email

[0534] 1. User: Says "Send me an email."

[0535] 2. Device: Captures the user's voice and sends it to the server.

[0536] 3. Server: Converts the voice data into text and recognizes it as "send an email."

[0537] 4. Server: Analyze the intent to "send an email" using natural language processing.

[0538] 5. Server: Obtains additional information from the user (recipient, subject, and body) needed to send the email.

[0539] 6. Server: Generates the command to "launch the mail client and send the email."

[0540] 7. Server: Sends the generated commands to the terminal.

[0541] 8. Terminal: Execute commands and send emails.

[0542] 9. Terminal: Notify the user that the email has been sent.

[0543] This system allows users to operate a computer using only voice input, providing great convenience, especially for elderly people and those who are unfamiliar with PC operation.

[0544] The processing flow will be explained below.

[0545] Step 1:

[0546] The user says, "Open a browser." The user verbally instructs the computer to operate, and this instruction is the starting point for the entire system.

[0547] Step 2:

[0548] The device captures the user's voice input through a microphone, where the user's speech is temporarily stored in memory as audio data.

[0549] Step 3:

[0550] The device sends the captured audio data to the server using an HTTP request, using a secure communication method.

[0551] Step 4:

[0552] The server passes the received voice data to a voice recognition engine and converts it into text data. For example, the voice saying "Open the browser" is analyzed and converted into text.

[0553] Step 5:

[0554] The server passes the text data to a natural language processing engine, which analyzes the user's intent. Here, the combination of the word "browser" and the verb "open" identifies the intent as "open a browser."

[0555] Step 6:

[0556] The server determines the task based on the specified intent and generates a command to perform that task, for example "start chrome".

[0557] Step 7:

[0558] The server generates commands and sends them to the device, which then receives them using HTTP requests.

[0559] Step 8:

[0560] The terminal executes the received command. In this case, the received command "start chrome" launches a browser (for example, Google Chrome).

[0561] Step 9:

[0562] The terminal notifies the user of the result of the task execution. If the browser is successfully launched, the user is given a voice or visual message saying "Browser opened."

[0563] Example 1

[0564] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0565] There was a need to provide an environment where elderly people and users unfamiliar with PC operation could use computers easily without complex operations. However, conventional systems had low accuracy in recognizing voice input and instructions, and intended operations were often not executed. This reduced user convenience and impaired the practicality of voice operation.

[0566] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0567] In this invention, the server includes means for converting voice data into text data, means for analyzing the text data using natural language processing technology to identify the user's intention, and means for determining the process to be performed based on the identified intention and generating instructions for that process. This makes it possible to recognize the user's intention with high accuracy and generate and execute appropriate instructions.

[0568] "Means for capturing voice input" refers to a device or method that captures voice spoken by a user and converts it into a digital signal.

[0569] "Means for transmitting captured audio data to a computer using a communication network" refers to the technology or method for transferring acquired audio data to a computer located in a remote location, such as a server, via a network such as the Internet.

[0570] "Means for converting voice data into text data by an electronic computer" refers to the process by which a server or computer uses voice recognition technology to convert voice into text information.

[0571] "Means of analyzing text data using natural language processing technology to identify the user's intent" refers to the technology and methods for analyzing acquired text data and understanding its meaning and requests.

[0572] "Means for determining the processing to be performed based on the identified intent and generating instructions for that" refers to the process of determining what operations or processing to perform based on the analysis results and generating the corresponding commands or instructions.

[0573] "Means for sending the generated instructions to the terminal and for the terminal to execute the instructions" refers to a technology or method for sending operation instructions generated by a server to a user's device and for the device to execute the received instructions.

[0574] "Means for notifying the user of the results of processing" refers to techniques or methods for visually or audibly conveying the results of the processing to the user.

[0575] "Electronic computer" means a computing device, server, or other computing equipment capable of receiving, processing, and analyzing data.

[0576] "Audio collection device" means a device for collecting a user's voice, including a microphone or other audio input device.

[0577] This invention provides a system that allows elderly people and users who are unfamiliar with PC operation to easily operate a computer. Specifically, it is a system that performs various tasks using voice input. This system uses the following main hardware and software:

[0578] Hardware and software used

[0579] 1. Terminal (user device)

[0580] Microphone: An audio collection device used to capture a user's voice input. This includes a computer's built-in microphone and an external USB microphone.

[0581] Network interface: A communication device for sending audio data to a server. An internet connection is required.

[0582] 2. Server

[0583] Speech recognition engine: Software for converting voice data into text data. For example, the Google Cloud Speech-to-Text API is used.

[0584] Natural language processing engine: Software used to analyze text data and identify user intent. For example, the Google Cloud Natural Language API is used.

[0585] Task execution command generation engine: Software that determines the process to be performed based on the user's intentions and generates the instructions.

[0586] System examples and operation procedures

[0587] As a concrete example for implementing this system, the tasks "open a browser" and "send an email" are taken.

[0588] Example 1: Open a browser

[0589] 1. User: Says "Open a browser."

[0590] 2. Terminal: Captures the user's voice with a microphone and sends the voice data to the server, which is securely transmitted using TLS.

[0591] 3. Server: Receives the audio data and converts it to text using the Google Cloud Speech-to-Text API.

[0592] 4. Server: Analyze the text data using the Google Cloud Natural Language API to identify the intent to "open a browser."

[0593] 5. Server: Generates a task execution command called "start chrome" based on the instruction "open browser".

[0594] 6. Server: Sends the generated commands to the terminal.

[0595] 7. Terminal: Executes the received command and launches the browser.

[0596] 8. On the device: The user is notified by voice or on screen that the browser has been opened.

[0597] Example 2: Sending an email

[0598] 1. User: Says "Send me an email."

[0599] 2. Terminal: Captures the user's voice with a microphone and sends the voice data to the server, which is securely transmitted using TLS.

[0600] 3. Server: Receives the audio data and converts it to text using the Google Cloud Speech-to-Text API.

[0601] 4. Server: Analyze the text data using the Google Cloud Natural Language API and identify the intent to "send an email."

[0602] 5. Server: Obtain additional information from the user (recipient, subject, and message) needed to send the email. For example, confirm "recipient is example@example.com, subject is test, message is hello."

[0603] 6. Server: Generates a command to "launch the mail client and send the email" and includes the necessary information.

[0604] 7. Server: Sends the generated commands to the terminal.

[0605] 8. Terminal: Executes the received command and sends the email.

[0606] 9. Terminal: Notify the user by voice or on-screen message that "Email has been sent."

[0607] This system allows users to easily operate a computer by voice input, providing great convenience especially for the elderly and people who are unfamiliar with using computers.

[0608] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0609] Step 1:

[0610] Input: User's voice input.

[0611] How it works: The device uses the microphone to capture the user's voice input, as the user speaks into the microphone, "open browser."

[0612] Output: Audio data in digital format.

[0613] Step 2:

[0614] Input: Audio data in digital format.

[0615] How it works: The device captures audio data and sends it to the server using a secure communication protocol (e.g., HTTPS). The audio data is encrypted with TLS and transmitted securely over the network.

[0616] Output: The audio data sent to the server.

[0617] Step 3:

[0618] Input: The audio data sent to the server.

[0619] How it works: The server converts the received voice data into text data using the Google Cloud Speech-to-Text API. The server's speech recognition engine analyzes the voice data and generates a string of text.

[0620] Output: Data in text format (e.g. "Open browser").

[0621] Step 4:

[0622] Input: Data in text format.

[0623] How it works: The server parses the text data using the Google Cloud Natural Language API. A natural language processing engine on the server tokenizes the text data and identifies the intent "open a browser." This process involves morphological analysis and contextual understanding.

[0624] Output: The user intent (e.g., open a browser).

[0625] Step 5:

[0626] Input: User intent.

[0627] How it works: The server generates an execution command such as "start chrome" based on the user's intent. The command generation engine on the server creates a script or shell command corresponding to the specified task and validates it as necessary.

[0628] Output: The generated command to run (e.g. start chrome).

[0629] Step 6:

[0630] Input: The generated execution command.

[0631] How it works: The server sends the generated command to the device. A secure communication protocol is used to ensure the command reaches the device. The command is encrypted and then decrypted by the device.

[0632] Output: Execution instructions sent to the terminal.

[0633] Step 7:

[0634] Input: Execution instructions sent to the terminal.

[0635] Operation: The device executes the received command and performs the specified task. In this case, the "start chrome" command is executed and the browser is launched. The device monitors the command execution result and determines whether it was successful or failed.

[0636] Output: The result of the task execution (e.g., a browser was opened).

[0637] Step 8:

[0638] Input: The result of the task execution.

[0639] Action: The device notifies the user of the execution result. For example, it uses a speech synthesis engine (e.g., Google Text-to-Speech) to announce "Browser opened" by voice, or displays the message "Browser opened" on the screen.

[0640] Output: Notification to the user (e.g. audio notification, screen display).

[0641] (Application example 1)

[0642] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0643] Security management systems require easy voice control of important security functions, such as camera operation, alarm setting, locking and unlocking, without requiring complex operations, especially for elderly users and those unfamiliar with technology. However, existing technologies lack a consistent system for achieving such voice control. Other issues include the accuracy of voice recognition, appropriate generation of execution commands, and clear notification of execution results to users.

[0644] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0645] In this invention, the server includes means for capturing voice input, means for transmitting the captured voice data to the server, means for the server to convert the voice data into text, means for analyzing the text data using natural language processing and identifying the user's intention, means for determining a task to be executed based on the identified intention and generating a command for that task, means for transmitting the generated command to the terminal and having the terminal execute the command, means for controlling security functions such as operating a camera, setting an alarm, locking or unlocking using voice input, and means for notifying the user of the result of task execution. This enables even elderly people and users who are unfamiliar with technology to easily control security functions using only voice input.

[0646] A "means for capturing voice input" is a technical device for detecting voice from a user and collecting that data.

[0647] The "means for transmitting captured voice data to a server" is a technical device that transmits collected voice data to a server using a communication means such as the Internet.

[0648] The "means for the server to convert voice data into text" refers to a technical device that converts voice data into text information using a voice recognition engine.

[0649] "Means for analyzing text data using natural language processing and identifying user intent" refers to a technical device that uses natural language processing technology to analyze text data and understand what the user wants.

[0650] The "means for determining a task to be executed based on the identified intention and generating a command for that task" is a technical device that determines an operation according to the user's intention and creates a corresponding command.

[0651] The "means for sending the generated command to the terminal and for the terminal to execute the command" is a technical device that sends the generated command to the terminal and the terminal executes the task according to the command.

[0652] "Means for controlling security functions such as operating cameras, setting alarms, locking and unlocking doors, etc. using voice input" refers to a technical device that uses voice input to operate security functions such as changing the direction of cameras, setting alarms, and locking and unlocking doors.

[0653] The "means for notifying the user of the results of the task execution" is a technical device that notifies the user of the results of the executed task by voice, display, or other means.

[0654] The present invention is a system for controlling a security control system using voice input, which takes voice input from a user and transmits the voice data to a server, which analyzes the voice data, identifies a corresponding task, and generates a command to perform the task.

[0655] The system includes the following components:

[0656] 1. Capture voice input

[0657] The user issues a command by voice, such as "point the camera at the living room." The microphone on the device captures this voice. The hardware used is a standard voice capture device.

[0658] 2. Sending audio data to the server

[0659] The device transmits the captured audio data to a server via a communication method such as the Internet. The communication uses a secure protocol (e.g., HTTPS).

[0660] 3. Voice Recognition

[0661] The server analyzes the received voice data and converts it into text information. The software used for this is a speech recognition engine (for example, Google Speech-to-Text API). The converted text data is used in the next step.

[0662] 4. Natural Language Processing

[0663] The server analyzes the text data obtained by the speech recognition engine using a natural language processing engine (e.g., spaCy or NLTK) to identify the user's intent. For example, the server recognizes the instruction "point the camera toward the living room" as "adjust the camera toward the living room."

[0664] 5. Generate task execution commands

[0665] The server determines the task to be performed based on the user's intent and generates a command for that task. For example, for the intent "point the camera at the living room," the server generates a command to "change the camera's orientation to the living room."

[0666] 6. Sending commands to the terminal and executing them

[0667] The generated command is then sent back to the device, which then executes it. For example, a mechanical control system for adjusting the orientation of a camera on the device is activated.

[0668] 7. Notifying the user of the task execution results

[0669] The device notifies the user of the execution result of the instructed task. For example, the device may notify the user by voice or by displaying a message on the screen saying, "The camera has been redirected to the living room."

[0670] Specific examples

[0671] Example 1: Camera operation

[0672] 1. The user says, "Point the camera at the living room."

[0673] 2. The device captures this audio and sends it to the server.

[0674] 3. The server converts the voice data into text and recognizes it as "point the camera towards the living room."

[0675] 4. The server parses this intent and generates a command to point the camera towards the living room.

[0676] 5. The generated command is sent to the device, and the camera turns as instructed.

[0677] 6. The device will announce, "The camera has been pointed at the living room."

[0678] Prompt Sentence Examples

[0679] When the user says "point the camera towards the living room," how does the system adjust the camera's orientation to face the living room? Please explain step by step.

[0680] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0681] Step 1:

[0682] Capturing voice input

[0683] The user issues a command, for example, "point the camera at the living room," and the microphone on the device captures the user's voice. The input is voice data, which is used for the next step.

[0684] Step 2:

[0685] Sending audio data to the server

[0686] The device sends the captured audio data to the server over the Internet using a secure protocol (e.g., HTTPS). The input is the captured audio data, and the output is the data sent to the server.

[0687] Step 3:

[0688] Voice Recognition

[0689] The server analyzes the received voice data and converts it into text information. The software used is a speech recognition engine (e.g., Google Speech-to-Text API). The input is voice data and the output is text data. Specifically, the speech recognition engine extracts acoustic features and matches them with known character patterns.

[0690] Step 4:

[0691] Natural Language Processing

[0692] The server analyzes the text data obtained through speech recognition to determine the user's intent. The software used is a natural language processing engine (e.g., spaCy or NLTK). The input is text data, and the output is information indicating the user's intent. Specifically, grammar analysis and context understanding are performed.

[0693] Step 5:

[0694] Generate task execution commands

[0695] The server determines the task to be executed based on the identified user intent and generates the command for that task. The input is information indicating the user intent, and the output is the specific command to be executed. For example, a command such as "change the camera's orientation to the living room" may be generated.

[0696] Step 6:

[0697] Sending commands to the terminal and executing them

[0698] The server sends the generated command to the terminal, which then executes it. The input is the generated command, and the output is the result of the command execution. Specifically, the terminal controls the camera motor and changes the camera's orientation as instructed.

[0699] Step 7:

[0700] Notifying users of task execution results

[0701] The device notifies the user of the results of the task. The input is the result of the command execution, and the output is a notification to the user. Specifically, the device tells the user, "The camera has been pointed at the living room," through a voice output device or a screen display.

[0702] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0703] This invention relates to a system that allows elderly people and users who are unfamiliar with PC operation to operate a computer easily. This system takes in the user's voice input, analyzes their intention, and automatically executes the corresponding task. It also incorporates an emotion engine that recognizes the user's emotions and enables responses that take those emotions into consideration.

[0704] System Overview

[0705] In this system, the user verbally commands an operation, and the device captures the speech and sends it to the server. The server performs speech recognition and natural language processing to identify the task to be performed and generate the command. The generated command is then sent to the device and executed. The system also recognizes the user's emotions and takes those emotions into consideration when carrying out the task and providing feedback. Finally, the user is notified of the results of the task.

[0706] Program processing overview

[0707] 1. Capture voice input

[0708] The device captures voice input from the user through a microphone. The user verbally issues commands such as "open a browser." The voice input may contain emotions.

[0709] 2. Sending audio data

[0710] The device sends the captured audio data to the server using an HTTP request, using secure communication methods.

[0711] 3. Voice Recognition

[0712] The server passes the received voice data to a voice recognition engine and converts it into text data. For example, the voice saying "Open the browser" is analyzed and converted into text.

[0713] 4. Natural Language Processing

[0714] The server analyzes the text data and identifies the user's intent. For example, it uses a natural language processing engine to recognize the intent of "open a browser."

[0715] 5. Emotion recognition

[0716] The server analyzes the user's emotions from the voice data. Using an emotion engine, it analyzes the tone and speed of the voice to identify the user's emotional state. For example, it can recognize whether the user is angry or happy.

[0717] 6. Generate task execution commands

[0718] The server determines the task to be performed based on the recognized intent and emotion, and generates a command for it. For example, the command "start chrome" is generated.

[0719] 7. Sending Commands

[0720] The server generates commands and sends them to the terminal. The commands are sent using HTTP requests.

[0721] 8. Executing Commands

[0722] The terminal executes the received command. In this case, the received command "start chrome" launches a browser (for example, Google Chrome).

[0723] 9. Notification of execution results

[0724] The device notifies the user of the results of the task. Based on emotion recognition, if the user is angry, for example, a message such as "I've opened the browser as you requested. Is there anything else I can help you with?" is provided to the user via voice or visual means.

[0725] Specific examples

[0726] Example 1: Open a browser

[0727] 1. User: "Open a browser." At this point, the user is in a hurry, so their voice is high-pitched and fast.

[0728] 2. Device: Captures audio and sends it to the server.

[0729] 3. Server: Converts the voice data into text and recognizes it as "opening a browser."

[0730] 4. Server: Analyze the intent to "open a browser" using natural language processing.

[0731] 5. Server: The emotion engine detects urgent situations and prioritizes quick responses.

[0732] 6. Server: Generate the command "start chrome".

[0733] 7. Server: Sends the generated commands to the terminal.

[0734] 8. Terminal: Run the command and launch the browser.

[0735] 9. On the device: The user is notified, "I've opened your browser. I see you're short on time. Is there anything else I can help you with?"

[0736] Example 2: Sending an email

[0737] 1. User: "Send me an email." The user is calm and their voice is level.

[0738] 2. Device: Captures audio and sends it to the server.

[0739] 3. Server: Converts the voice data into text and recognizes it as "send an email."

[0740] 4. Server: Analyze the intent to "send an email" using natural language processing.

[0741] 5. Server: The emotion engine detects the calm situation and proceeds with the normal procedure.

[0742] 6. Server: Obtains additional information from the user (recipient, subject, and body) needed to send the email.

[0743] 7. Server: Generates the command to "launch the mail client and send the email."

[0744] 8. Server: Sends the generated commands to the terminal.

[0745] 9. Terminal: Execute commands and send emails.

[0746] 10. Terminal: Notify the user that "Email has been sent. Please check."

[0747] In this way, the system recognizes the user's voice commands and emotions, converts them into specific PC operations, and automatically executes them, providing great convenience to elderly people and users who are unfamiliar with PC operation.

[0748] The processing flow will be explained below.

[0749] Step 1:

[0750] The user says, "Open a browser." This verbal command is the starting point of the entire system. The user's voice contains their current emotional state.

[0751] Step 2:

[0752] The device captures the user's voice input through a microphone, where the user's speech is temporarily stored in memory as audio data.

[0753] Step 3:

[0754] The device sends the captured audio data to the server using an HTTP request, using secure communication methods.

[0755] Step 4:

[0756] The server passes the received voice data to a voice recognition engine and converts it into text data. For example, the voice saying "Open the browser" is analyzed and converted into text.

[0757] Step 5:

[0758] The server passes the text data to a natural language processing engine to identify the user's intent. Here, the combination of the word "browser" and the verb "open" identifies the intent as "open a browser."

[0759] Step 6:

[0760] The server analyzes the user's emotions from the voice data, and the emotion engine analyzes voice characteristics such as tone, speed, and emphasis to identify the user's emotional state, such as whether they are in a hurry or relaxed.

[0761] Step 7:

[0762] Based on the identified intent and emotion, the server determines the task to be performed and generates the command. For example, if the user is in a hurry, the command "start chrome" is generated to "open the browser immediately," eliminating the additional confirmation step.

[0763] Step 8:

[0764] The server generates commands and sends them to the terminal. The commands are sent using HTTP requests.

[0765] Step 9:

[0766] The terminal executes the received command. In this case, the received command "start chrome" launches a browser (for example, Google Chrome).

[0767] Step 10:

[0768] The device notifies the user of the results of the task execution. Based on the emotion recognition results, if the user is in a hurry, a message such as "I've opened the browser. Is there anything else I can help you with?" is provided to the user via voice or visual means.

[0769] Specific examples

[0770] Example 1: Open a browser

[0771] 1. The user says, "Open a browser." The voice is quick and high-pitched because they're in a hurry.

[0772] 2. The device captures the audio and sends it to the server.

[0773] 3. The server converts the voice data into text and recognizes it as "opening a browser."

[0774] 4. The server uses natural language processing to analyze the intent to "open a browser."

[0775] 5. The server detects urgent situations using an emotion engine.

[0776] 6. The server generates the command "start chrome".

[0777] 7. The server sends the generated command to the terminal.

[0778] 8. The terminal executes the command and launches the browser.

[0779] 9. The device notifies the user, "Your browser is open. Is there anything else I can help you with?"

[0780] Example 2: Sending an email

[0781] 1. The user says, "Send me an email." The user is calm and the voice is even.

[0782] 2. The device captures the audio and sends it to the server.

[0783] 3. The server converts the voice data into text and recognizes it as "send an email."

[0784] 4. The server uses natural language processing to analyze the intent to "send an email."

[0785] 5. The server detects calm situations using the emotion engine.

[0786] 6. The server obtains additional information from the user (recipient, subject, and body) needed to send the email.

[0787] 7. The server generates a command to launch the mail client and send the email.

[0788] 8. The server sends the generated command to the terminal.

[0789] 9. The terminal executes the command and sends the email.

[0790] 10. The device notifies the user, "Email has been sent. Please check."

[0791] In this way, the system recognizes the user's voice commands and emotions, converts them into specific PC operations, and automatically executes them, providing great convenience to elderly people and users who are unfamiliar with PC operation.

[0792] Example 2

[0793] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0794] Today's elderly and users unfamiliar with PC operation face many difficulties when operating computers. In particular, conventional user interfaces require visual and tactile operation, which poses technical hurdles. For this reason, more intuitive and easier operation methods are needed. Furthermore, there is a lack of support that takes into account the user's emotional state. This increases stress and burden on users, resulting in a decrease in frequency of use.

[0795] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for converting voice data into text, a means for analyzing the text data by natural language processing to identify the user's intention, and a means for analyzing the user's emotions from the voice data. This allows the user to intuitively operate the computer using only voice input, and further enables the computer to respond according to their emotional state.

[0796] "Voice input" refers to instructions or information verbally given by a user.

[0797] "Capture" refers to the process of recording and importing data such as voice input.

[0798] "Audio Data" means captured audio converted into digital form.

[0799] A "server" is a computer system that processes data over a network and provides the results to other devices.

[0800] "Text conversion" refers to the process of converting audio data into corresponding text data.

[0801] "Natural language processing" is a technology that uses computers to analyze and understand human language.

[0802] "Intent" refers to the goal or request that the user is trying to achieve through voice input.

[0803] "Emotion recognition" is a technology that analyzes and identifies a user's emotional state from voice data.

[0804] A "task" is a specific operation or process that a computer or terminal must perform.

[0805] A "command" is a specific instruction or direction that a computer system is to carry out.

[0806] A "terminal" is a device with input and output capabilities that is directly operated by a user.

[0807] "Execution results" refers to the results or output obtained after a computer or terminal executes a command.

[0808] "Notification" refers to the act of informing the user of execution results or information.

[0809] This invention relates to a system that allows elderly people and users who are unfamiliar with PC operation to easily operate a computer. This system captures the user's voice input, analyzes their intention, and automatically executes the corresponding task. It also incorporates an emotion engine that recognizes the user's emotions and enables responses that take those emotions into consideration.

[0810] System Overview

[0811] The system consists of the following main components:

[0812] 1. A means to capture audio input (microphone)

[0813] 2. A means of sending the captured audio data to the server

[0814] 3. A means of converting voice data into text on the server side (voice recognition engine)

[0815] 4. A means of analyzing text data using natural language processing to identify the user's intent (natural language processing engine)

[0816] 5. A means of analyzing user emotions from voice data (emotion engine)

[0817] 6. A means to determine the task to be performed based on the identified intention and emotion and generate the command.

[0818] 7. A means for sending the generated command to the terminal and for the terminal to execute the command

[0819] 8. Means of notifying the user of the results of the task execution

[0820] Specific operation explanation

[0821] 1. Capture voice input

[0822] The device captures voice input from the user. The device has a built-in microphone and records the user's commands in real time. For example, the user might say, "Open a browser."

[0823] 2. Sending audio data

[0824] The device sends the captured audio data to the server, where it is encrypted and transmitted over a secure channel.

[0825] 3. Voice Recognition

[0826] The server passes the received voice data to a speech recognition engine, which converts the voice into text data. The speech recognition engine uses the Google Cloud Speech-to-Text API or similar. For example, the voice saying "Open your browser" is converted into text saying "Open your browser."

[0827] 4. Natural Language Processing

[0828] The server passes the text data to a natural language processing engine, which analyzes the user's intent. The natural language processing engine uses Google Cloud Natural Language API or similar. From the analysis results, the specific intent of "open a browser" is identified.

[0829] 5. Emotion recognition

[0830] The server passes the voice data to an emotion recognition engine, which analyzes the tone and speed of the voice to recognize the user's emotions. The emotion recognition engine uses the Microsoft Azure Emotion API, etc. For example, if the user is in a hurry, the server will understand their emotional state.

[0831] 6. Generate task execution commands

[0832] The server determines the task to be performed based on the user's intention and emotion, and generates the command for it. Specifically, the command "start chrome" is generated.

[0833] 7. Sending Commands

[0834] The server sends the generated command to the terminal using a secure communication method.

[0835] 8. Executing Commands

[0836] The device executes the received command, for example launching Google Chrome.

[0837] 9. Notification of execution results

[0838] The device notifies the user of the results of the task. Based on the emotion recognition results, feedback appropriate to the user's situation is provided. For example, a message such as "I've opened your browser. It seems you're short on time. Is there anything else I can help you with?" is displayed or spoken.

[0839] Specific examples

[0840] Example 1: Open a browser

[0841] 1. The user says, "Open a browser."

[0842] 2. The device captures the audio and sends it to the server.

[0843] 3. The server converts the voice data into text and recognizes it as "opening a browser."

[0844] 4. The server analyzes the intent.

[0845] 5. The server recognizes the user's sense of urgency from the voice.

[0846] 6. The server generates the command "start chrome".

[0847] 7. The server sends the command to the device.

[0848] 8. The terminal executes the command and launches the browser.

[0849] 9. Your device will say, "I've opened your browser. I see you're short on time. Is there anything else I can help you with?"

[0850] Example 2: Sending an email

[0851] 1. The user says, "Send me an email."

[0852] 2. The device captures the audio and sends it to the server.

[0853] 3. The server converts the voice data into text and recognizes it as "send an email."

[0854] 4. The server analyzes the intent.

[0855] 5. The server recognizes the user's neutral emotion from the voice.

[0856] 6. The server generates a prompt asking for additional information.

[0857] 7. The server sends a prompt to the terminal to get additional information from the user.

[0858] 8. The server generates a command to launch the email client and send the email.

[0859] 9. The server sends the command to the device.

[0860] 10. The terminal executes the command and sends the email.

[0861] 11. The device will notify you that "Email has been sent. Please check."

[0862] In this way, the system analyzes the user's voice commands in detail, converts them into specific PC operations, and automatically executes them, providing convenience that allows even elderly people and users who are unfamiliar with PC operation to easily operate a computer.

[0863] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0864] Step 1:

[0865] The device captures voice input from the user through the microphone. The input is the user's speech, for example, "Open the browser," in the form of voice data. The device receives this voice as an analog signal, converts it to a digital signal, and temporarily stores it in a buffer. Specifically, the device's built-in microphone converts the voice signal into digital data (for example, WAV format).

[0866] Step 2:

[0867] The device sends the captured audio data to the server. The input is the digital audio data obtained in step 1. This data is sent to the server using the HTTPS protocol. Specifically, the device's communication module creates an HTTP POST request and sends the audio data as a payload to the server.

[0868] Step 3:

[0869] The server passes the received voice data to a speech recognition engine, which converts the voice into text data. The input is the digital voice data sent in step 2. The output is the converted text data. Specifically, the server uses the Google Cloud Speech-to-Text API to convert the voice "Open your browser" into the text "Open your browser."

[0870] Step 4:

[0871] The server passes the text data to a natural language processing engine to analyze the user's intent. The input is the text data obtained in step 3. The output is data that identifies the user's intent. Specifically, the server uses the Google Cloud Natural Language API to analyze the text "open a browser" and identify the intent as "launch a browser."

[0872] Step 5:

[0873] The server analyzes the user's emotions from the voice data. The input is the voice data acquired in step 2. The output is the user's emotional state. Specifically, the server uses the Microsoft Azure Emotion API to analyze the tone and speed of the voice and determine whether the user is in a hurry.

[0874] Step 6:

[0875] The server determines the task to be performed based on the user's intention and emotion, and generates the command. The input is the user's intention identified in step 4 and the emotional state analyzed in step 5. The output is a specific command. Specifically, the server generates the command "start chrome".

[0876] Step 7:

[0877] The server sends the generated command to the terminal. The input is the command generated in step 6. The output is the command sent to the terminal. Specifically, the server's communication module creates an HTTP POST request and sends the command as a payload to the terminal.

[0878] Step 8:

[0879] The terminal executes the received command. The input is the command received in step 7. The output is the result of executing the command. Specifically, the terminal interprets the command "start chrome" and launches Google Chrome.

[0880] Step 9:

[0881] The terminal notifies the user of the results of the task execution. The input is the result of the command execution in step 8. The output is feedback to the user. Specifically, the terminal displays a message audibly or visually saying, "I've opened your browser. It looks like you're short on time. Is there anything else I can help you with?"

[0882] (Application example 2)

[0883] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0884] Conventional systems lack the means to intuitively operate computers and digital devices for elderly users and users unfamiliar with computer operation. This problem is particularly pronounced when complex operations are required. Furthermore, the lack of a system that flexibly responds to the user's emotions often results in a poor user experience. This creates a challenge for elderly users and users unfamiliar with technology to continue using digital environments.

[0885] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0886] In this invention, the server includes means for capturing voice input, means for transmitting the captured voice data to the server, means for the server to convert the voice data into text, means for analyzing the text data using natural language processing to identify the user's intention, means for analyzing the user's emotional state and providing flexible task execution and feedback based on the emotion, means for determining a task to be executed based on the identified intention and emotion and generating a command for the task, means for transmitting the generated command to a terminal and having the terminal execute the command, means for notifying the user of the task execution result, and navigation means for guiding the user using smart glasses. This enables even elderly people and users unfamiliar with computer operation to intuitively operate the device through voice input and smoothly use a digital environment while receiving flexible responses according to their emotional state.

[0887] A "means for capturing voice input" is a device or function for capturing voice uttered by a user.

[0888] The "means for transmitting captured audio data to a server" refers to a communication means or protocol for transmitting captured audio data to a server.

[0889] The "means for the server to convert voice data into text" is a voice recognition engine for converting received voice data into text format.

[0890] "Means for analyzing text data using natural language processing and identifying the user's intent" refers to a natural language processing engine that analyzes the converted text data to understand the user's instructions and intent.

[0891] "Means for analyzing the user's emotional state and providing flexible task execution and feedback based on those emotions" refers to an emotion recognition engine that analyzes the user's emotions from voice and other inputs and provides task execution and feedback appropriate to those emotions.

[0892] The "means for determining a task to be executed based on the identified intention and emotion and generating a command for that task" is a means for determining the optimal task according to the user's intention and emotional state and generating a command for that task.

[0893] The "means for transmitting the generated command to the terminal and for the terminal to execute the command" refers to a communication means for transmitting the generated command to the device and for causing the device to execute the command.

[0894] The "means for notifying the user of the execution result of the task" is a means for notifying the user of the execution result of the task.

[0895] A "navigation means for guiding a user using smart glasses" is a navigation system for providing visual and audio guidance and instructions to a user through smart glasses.

[0896] This invention is a system that allows elderly people and users who are unfamiliar with operating personal computers to easily operate a computer. A specific example of this system is shown below.

[0897] The main elements of the system include means for capturing a user's voice input, means for transmitting the captured voice data to a server, means for converting the voice data to text on the server side, means for analyzing the text data using natural language processing to identify the user's intention, means for analyzing the user's emotional state, means for determining a task to be performed based on the intention and emotion and generating a command for that task, means for transmitting the generated command to a terminal and for the terminal to execute the command, means for notifying the user of the result of the task execution, and navigation means for guiding the user using smart glasses.

[0898] Specifically, the system works as follows: The user wears the smart glasses and gives voice instructions (e.g., "Where is the tea shelf?"). The microphone in the smart glasses captures this voice and sends the voice data to the server using HTTPS communication. The server converts the voice data into text using Google Cloud Speech-to-Text.

[0899] The system then uses the Google Cloud Natural Language engine to analyze the converted text data and identify the user's intent. At the same time, it uses the Azure Emotion API to analyze the user's emotional state from the audio data. For example, if the system determines that the user is confused, that emotional state is also taken into account.

[0900] Based on the identified intent and emotion, the server determines the optimal task and generates a command to execute it. For example, commands such as "Start navigation" or "Guide to the tea shelf." This generated command is then sent to the smart glasses via HTTPS. The smart glasses then execute the received command and navigate the store using the built-in Google Maps API.

[0901] During navigation, the smart glasses provide real-time visual and audio guidance to the user, such as a message saying, "This way is the tea corner. Let us show you around." Once navigation is complete, feedback is also provided, such as, "You've arrived at the tea corner. Can we help you?"

[0902] Specific examples

[0903] Example 1: Guidance to the tea corner

[0904] 1. User: "Where is the tea cupboard?"

[0905] 2. Smart glasses: Capture the user's voice and send it to the server.

[0906] 3. Server: Converts speech to text and analyzes intent. Recognizes "Search for tea shelves."

[0907] 4. Server: The emotion engine recognizes "confusion."

[0908] 5. Server: Generate navigation command. "Guide me to the tea corner."

[0909] 6. Smart glasses: Start navigation and visually and audibly display "The tea corner is here. We will guide you."

[0910] 7. Smart glasses: After the guide is completed, the glasses will notify you, "You have arrived at the tea corner. Can I help you?"

[0911] Prompt Sentence Examples

[0912] The user puts on the smart glasses and asks a question by voice, for example, "Where is the tea cupboard?"

[0913] The smart glasses capture this audio and send it to the server. The server converts the audio into text using Google Cloud Speech-to-Text and identifies the intent using Google Cloud Natural Language. It then analyzes the user's emotions using the Azure Emotion API. Based on the acquired information, the server generates appropriate navigation commands and sends them to the smart glasses. The smart glasses then use the Google Maps API to perform real-time navigation and provide audio and visual guidance to the user.

[0914] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0915] Step 1:

[0916] Input: User's spoken command (e.g., "Where is the tea cupboard?")

[0917] How it works: The microphone on the device (smart glasses) captures the user's voice.

[0918] Output: Captured audio data

[0919] Step 2:

[0920] Input: Captured audio data

[0921] How it works: The device sends the captured audio data to the server using HTTPS.

[0922] Output: Audio data sent to the server

[0923] Step 3:

[0924] Input: Audio data received by the server

[0925] How it works: The server uses a speech recognition engine (Google Cloud Speech-to-Text) to convert the voice data into text.

[0926] Output: Text data (e.g., "Where is the tea shelf?")

[0927] Step 4:

[0928] Input: Converted text data

[0929] How it works: The server uses a natural language processing engine (Google Cloud Natural Language) to analyze the text data and determine the user's intent.

[0930] Output: Identified intent (e.g., "Find a tea cupboard")

[0931] Step 5:

[0932] Input: Audio data received by the server

[0933] How it works: The server uses an emotion recognition engine (Azure Emotion API) to analyze the user's emotional state from the voice data.

[0934] Output: User's emotional state (e.g., "confused")

[0935] Step 6:

[0936] Input: Identified intent and user emotional state

[0937] Action: The server determines the task to be performed based on the recognized intent and emotion, and generates the command.

[0938] Output: Generated command (e.g. "Take me to the tea corner")

[0939] Step 7:

[0940] Input: Generated command

[0941] Operation: The server sends the generated command to the device using HTTPS communication.

[0942] Output: Command sent to the terminal

[0943] Step 8:

[0944] Input: The command received by the terminal

[0945] How it works: The device (smart glasses) starts real-time navigation using the Google Maps API.

[0946] Output: Visual and audio navigation instructions

[0947] Step 9:

[0948] Input: Navigation result

[0949] Behavior: The device notifies the user of the results of the task, for example by displaying a message like "You've arrived at the tea corner. How can I help you?"

[0950] Output: Feedback notification to the user

[0951] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0952] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0953] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0954] [Third embodiment]

[0955] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0956] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0957] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0958] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0959] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0960] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0961] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0962] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0963] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0964] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0965] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0966] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0967] The present invention relates to a system that allows elderly people and users who are unfamiliar with PC operation to easily operate a computer. The system captures the user's voice input, analyzes their intention, and automatically executes the corresponding task.

[0968] System Overview

[0969] In this system, the user verbally commands an operation, and the device captures the speech and sends it to the server. The server performs speech recognition and natural language processing to identify the task to be performed and generate the command. The generated command is then sent to the device and executed. Finally, the user is notified of the result of the task execution.

[0970] Program processing overview

[0971] 1. Capture voice input

[0972] The device captures voice input from the user through a microphone, for example, by giving verbal commands such as "open a browser."

[0973] 2. Sending audio data

[0974] The device transmits the captured audio data to the server, which uses a secure communication method.

[0975] 3. Voice Recognition

[0976] The server converts the received voice data into text data, using a speech recognition engine on the server.

[0977] 4. Natural Language Processing

[0978] The server analyzes the text data and identifies the user's intent. For example, it uses a natural language processing engine to recognize the intent of "open a browser."

[0979] 5. Generate task execution commands

[0980] Based on the intent recognized, the server determines the task to be performed and generates the corresponding command, for example, "start chrome" as the command to "open a browser."

[0981] 6. Sending Commands

[0982] The server generates commands and sends them to the terminal.

[0983] 7. Executing commands

[0984] The device executes the received command and performs the specified task, in this case launching a browser.

[0985] 8. Notification of execution results

[0986] The device notifies the user of the result of the task execution, for example, by voice or on-screen notification that "the browser has been opened."

[0987] Specific examples

[0988] Example 1: Open a browser

[0989] 1. User: Says "Open a browser."

[0990] 2. Device: Captures the user's voice and sends it to the server.

[0991] 3. Server: Converts the voice data into text and recognizes it as "opening a browser."

[0992] 4. Server: Analyze the intent to "open a browser" using natural language processing.

[0993] 5. Server: Generate the command "start chrome".

[0994] 6. Server: Sends the generated commands to the terminal.

[0995] 7. Terminal: Execute the command and launch the browser.

[0996] 8. Terminal: Notify the user that the browser has been opened.

[0997] Example 2: Sending an email

[0998] 1. User: Says "Send me an email."

[0999] 2. Device: Captures the user's voice and sends it to the server.

[1000] 3. Server: Converts the voice data into text and recognizes it as "send an email."

[1001] 4. Server: Analyze the intent to "send an email" using natural language processing.

[1002] 5. Server: Obtains additional information from the user (recipient, subject, and body) needed to send the email.

[1003] 6. Server: Generates the command to "launch the mail client and send the email."

[1004] 7. Server: Sends the generated commands to the terminal.

[1005] 8. Terminal: Execute commands and send emails.

[1006] 9. Terminal: Notify the user that the email has been sent.

[1007] This system allows users to operate a computer using only voice input, providing great convenience, especially for elderly people and those who are unfamiliar with PC operation.

[1008] The processing flow will be explained below.

[1009] Step 1:

[1010] The user says, "Open a browser." The user verbally instructs the computer to operate, and this instruction is the starting point for the entire system.

[1011] Step 2:

[1012] The device captures the user's voice input through a microphone, where the user's speech is temporarily stored in memory as audio data.

[1013] Step 3:

[1014] The device sends the captured audio data to the server using an HTTP request, using a secure communication method.

[1015] Step 4:

[1016] The server passes the received voice data to a voice recognition engine and converts it into text data. For example, the voice saying "Open the browser" is analyzed and converted into text.

[1017] Step 5:

[1018] The server passes the text data to a natural language processing engine, which analyzes the user's intent. Here, the combination of the word "browser" and the verb "open" identifies the intent as "open a browser."

[1019] Step 6:

[1020] The server determines the task based on the specified intent and generates a command to perform that task, for example "start chrome".

[1021] Step 7:

[1022] The server generates commands and sends them to the device, which then receives them using HTTP requests.

[1023] Step 8:

[1024] The terminal executes the received command. In this case, the received command "start chrome" launches a browser (for example, Google Chrome).

[1025] Step 9:

[1026] The terminal notifies the user of the result of the task execution. If the browser is successfully launched, the user is given a voice or visual message saying "Browser opened."

[1027] Example 1

[1028] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1029] There was a need to provide an environment where elderly people and users unfamiliar with PC operation could use computers easily without complex operations. However, conventional systems had low accuracy in recognizing voice input and instructions, and intended operations were often not executed. This reduced user convenience and impaired the practicality of voice operation.

[1030] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1031] In this invention, the server includes means for converting voice data into text data, means for analyzing the text data using natural language processing technology to identify the user's intention, and means for determining the process to be performed based on the identified intention and generating instructions for that process. This makes it possible to recognize the user's intention with high accuracy and generate and execute appropriate instructions.

[1032] "Means for capturing voice input" refers to a device or method that captures voice spoken by a user and converts it into a digital signal.

[1033] "Means for transmitting captured audio data to a computer using a communication network" refers to the technology or method for transferring acquired audio data to a computer located in a remote location, such as a server, via a network such as the Internet.

[1034] "Means for converting voice data into text data by an electronic computer" refers to the process by which a server or computer uses voice recognition technology to convert voice into text information.

[1035] "Means of analyzing text data using natural language processing technology to identify the user's intent" refers to the technology and methods for analyzing acquired text data and understanding its meaning and requests.

[1036] "Means for determining the processing to be performed based on the identified intent and generating instructions for that" refers to the process of determining what operations or processing to perform based on the analysis results and generating the corresponding commands or instructions.

[1037] "Means for sending the generated instructions to the terminal and for the terminal to execute the instructions" refers to a technology or method for sending operation instructions generated by a server to a user's device and for the device to execute the received instructions.

[1038] "Means for notifying the user of the results of processing" refers to techniques or methods for visually or audibly conveying the results of the processing to the user.

[1039] "Electronic computer" means a computing device, server, or other computing equipment capable of receiving, processing, and analyzing data.

[1040] "Audio collection device" means a device for collecting a user's voice, including a microphone or other audio input device.

[1041] This invention provides a system that allows elderly people and users who are unfamiliar with PC operation to easily operate a computer. Specifically, it is a system that performs various tasks using voice input. This system uses the following main hardware and software:

[1042] Hardware and software used

[1043] 1. Terminal (user device)

[1044] Microphone: An audio collection device used to capture a user's voice input. This includes a computer's built-in microphone and an external USB microphone.

[1045] Network interface: A communication device for sending audio data to a server. An internet connection is required.

[1046] 2. Server

[1047] Speech recognition engine: Software for converting voice data into text data. For example, the Google Cloud Speech-to-Text API is used.

[1048] Natural language processing engine: Software used to analyze text data and identify user intent. For example, the Google Cloud Natural Language API is used.

[1049] Task execution command generation engine: Software that determines the process to be performed based on the user's intentions and generates the instructions.

[1050] System examples and operation procedures

[1051] As a concrete example for implementing this system, the tasks "open a browser" and "send an email" are taken.

[1052] Example 1: Open a browser

[1053] 1. User: Says "Open a browser."

[1054] 2. Terminal: Captures the user's voice with a microphone and sends the voice data to the server, which is securely transmitted using TLS.

[1055] 3. Server: Receives the audio data and converts it to text using the Google Cloud Speech-to-Text API.

[1056] 4. Server: Analyze the text data using the Google Cloud Natural Language API to identify the intent to "open a browser."

[1057] 5. Server: Generates a task execution command called "start chrome" based on the instruction "open browser".

[1058] 6. Server: Sends the generated commands to the terminal.

[1059] 7. Terminal: Executes the received command and launches the browser.

[1060] 8. On the device: The user is notified by voice or on screen that the browser has been opened.

[1061] Example 2: Sending an email

[1062] 1. User: Says "Send me an email."

[1063] 2. Terminal: Captures the user's voice with a microphone and sends the voice data to the server, which is securely transmitted using TLS.

[1064] 3. Server: Receives the audio data and converts it to text using the Google Cloud Speech-to-Text API.

[1065] 4. Server: Analyze the text data using the Google Cloud Natural Language API and identify the intent to "send an email."

[1066] 5. Server: Obtain additional information from the user (recipient, subject, and message) needed to send the email. For example, confirm "recipient is example@example.com, subject is test, message is hello."

[1067] 6. Server: Generates a command to "launch the mail client and send the email" and includes the necessary information.

[1068] 7. Server: Sends the generated commands to the terminal.

[1069] 8. Terminal: Executes the received command and sends the email.

[1070] 9. Terminal: Notify the user by voice or on-screen message that "Email has been sent."

[1071] This system allows users to easily operate a computer by voice input, providing great convenience especially for the elderly and people who are unfamiliar with using computers.

[1072] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1073] Step 1:

[1074] Input: User's voice input.

[1075] How it works: The device uses the microphone to capture the user's voice input, as the user speaks into the microphone, "open browser."

[1076] Output: Audio data in digital format.

[1077] Step 2:

[1078] Input: Audio data in digital format.

[1079] How it works: The device captures audio data and sends it to the server using a secure communication protocol (e.g., HTTPS). The audio data is encrypted with TLS and transmitted securely over the network.

[1080] Output: The audio data sent to the server.

[1081] Step 3:

[1082] Input: The audio data sent to the server.

[1083] How it works: The server converts the received voice data into text data using the Google Cloud Speech-to-Text API. The server's speech recognition engine analyzes the voice data and generates a string of text.

[1084] Output: Data in text format (e.g. "Open browser").

[1085] Step 4:

[1086] Input: Data in text format.

[1087] How it works: The server parses the text data using the Google Cloud Natural Language API. A natural language processing engine on the server tokenizes the text data and identifies the intent "open a browser." This process involves morphological analysis and contextual understanding.

[1088] Output: The user intent (e.g., open a browser).

[1089] Step 5:

[1090] Input: User intent.

[1091] How it works: The server generates an execution command such as "start chrome" based on the user's intent. The command generation engine on the server creates a script or shell command corresponding to the specified task and validates it as necessary.

[1092] Output: The generated command to run (e.g. start chrome).

[1093] Step 6:

[1094] Input: The generated execution command.

[1095] How it works: The server sends the generated command to the device. A secure communication protocol is used to ensure the command reaches the device. The command is encrypted and then decrypted by the device.

[1096] Output: Execution instructions sent to the terminal.

[1097] Step 7:

[1098] Input: Execution instructions sent to the terminal.

[1099] Operation: The device executes the received command and performs the specified task. In this case, the "start chrome" command is executed and the browser is launched. The device monitors the command execution result and determines whether it was successful or failed.

[1100] Output: The result of the task execution (e.g., a browser was opened).

[1101] Step 8:

[1102] Input: The result of the task execution.

[1103] Action: The device notifies the user of the execution result. For example, it uses a speech synthesis engine (e.g., Google Text-to-Speech) to announce "Browser opened" by voice, or displays the message "Browser opened" on the screen.

[1104] Output: Notification to the user (e.g. audio notification, screen display).

[1105] (Application example 1)

[1106] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1107] Security management systems require easy voice control of important security functions, such as camera operation, alarm setting, locking and unlocking, without requiring complex operations, especially for elderly users and those unfamiliar with technology. However, existing technologies lack a consistent system for achieving such voice control. Other issues include the accuracy of voice recognition, appropriate generation of execution commands, and clear notification of execution results to users.

[1108] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1109] In this invention, the server includes means for capturing voice input, means for transmitting the captured voice data to the server, means for the server to convert the voice data into text, means for analyzing the text data using natural language processing and identifying the user's intention, means for determining a task to be executed based on the identified intention and generating a command for that task, means for transmitting the generated command to the terminal and having the terminal execute the command, means for controlling security functions such as operating a camera, setting an alarm, locking or unlocking using voice input, and means for notifying the user of the result of task execution. This enables even elderly people and users who are unfamiliar with technology to easily control security functions using only voice input.

[1110] A "means for capturing voice input" is a technical device for detecting voice from a user and collecting that data.

[1111] The "means for transmitting captured voice data to a server" is a technical device that transmits collected voice data to a server using a communication means such as the Internet.

[1112] The "means for the server to convert voice data into text" refers to a technical device that converts voice data into text information using a voice recognition engine.

[1113] "Means for analyzing text data using natural language processing and identifying user intent" refers to a technical device that uses natural language processing technology to analyze text data and understand what the user wants.

[1114] The "means for determining a task to be executed based on the identified intention and generating a command for that task" is a technical device that determines an operation according to the user's intention and creates a corresponding command.

[1115] The "means for sending the generated command to the terminal and for the terminal to execute the command" is a technical device that sends the generated command to the terminal and the terminal executes the task according to the command.

[1116] "Means for controlling security functions such as operating cameras, setting alarms, locking and unlocking doors, etc. using voice input" refers to a technical device that uses voice input to operate security functions such as changing the direction of cameras, setting alarms, and locking and unlocking doors.

[1117] The "means for notifying the user of the results of the task execution" is a technical device that notifies the user of the results of the executed task by voice, display, or other means.

[1118] The present invention is a system for controlling a security control system using voice input, which takes voice input from a user and transmits the voice data to a server, which analyzes the voice data, identifies a corresponding task, and generates a command to perform the task.

[1119] The system includes the following components:

[1120] 1. Capture voice input

[1121] The user issues a command by voice, such as "point the camera at the living room." The microphone on the device captures this voice. The hardware used is a standard voice capture device.

[1122] 2. Sending audio data to the server

[1123] The device transmits the captured audio data to a server via a communication method such as the Internet. The communication uses a secure protocol (e.g., HTTPS).

[1124] 3. Voice Recognition

[1125] The server analyzes the received voice data and converts it into text information. The software used for this is a speech recognition engine (for example, Google Speech-to-Text API). The converted text data is used in the next step.

[1126] 4. Natural Language Processing

[1127] The server analyzes the text data obtained by the speech recognition engine using a natural language processing engine (e.g., spaCy or NLTK) to identify the user's intent. For example, the server recognizes the instruction "point the camera toward the living room" as "adjust the camera toward the living room."

[1128] 5. Generate task execution commands

[1129] The server determines the task to be performed based on the user's intent and generates a command for that task. For example, for the intent "point the camera at the living room," the server generates a command to "change the camera's orientation to the living room."

[1130] 6. Sending commands to the terminal and executing them

[1131] The generated command is then sent back to the device, which then executes it. For example, a mechanical control system for adjusting the orientation of a camera on the device is activated.

[1132] 7. Notifying the user of the task execution results

[1133] The device notifies the user of the execution result of the instructed task. For example, the device may notify the user by voice or by displaying a message on the screen saying, "The camera has been redirected to the living room."

[1134] Specific examples

[1135] Example 1: Camera operation

[1136] 1. The user says, "Point the camera at the living room."

[1137] 2. The device captures this audio and sends it to the server.

[1138] 3. The server converts the voice data into text and recognizes it as "point the camera towards the living room."

[1139] 4. The server parses this intent and generates a command to point the camera towards the living room.

[1140] 5. The generated command is sent to the device, and the camera turns as instructed.

[1141] 6. The device will announce, "The camera has been pointed at the living room."

[1142] Prompt Sentence Examples

[1143] When the user says "point the camera towards the living room," how does the system adjust the camera's orientation to face the living room? Please explain step by step.

[1144] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1145] Step 1:

[1146] Capturing voice input

[1147] The user issues a command, for example, "point the camera at the living room," and the microphone on the device captures the user's voice. The input is voice data, which is used for the next step.

[1148] Step 2:

[1149] Sending audio data to the server

[1150] The device sends the captured audio data to the server over the Internet using a secure protocol (e.g., HTTPS). The input is the captured audio data, and the output is the data sent to the server.

[1151] Step 3:

[1152] Voice Recognition

[1153] The server analyzes the received voice data and converts it into text information. The software used is a speech recognition engine (e.g., Google Speech-to-Text API). The input is voice data and the output is text data. Specifically, the speech recognition engine extracts acoustic features and matches them with known character patterns.

[1154] Step 4:

[1155] Natural Language Processing

[1156] The server analyzes the text data obtained through speech recognition to determine the user's intent. The software used is a natural language processing engine (e.g., spaCy or NLTK). The input is text data, and the output is information indicating the user's intent. Specifically, grammar analysis and context understanding are performed.

[1157] Step 5:

[1158] Generate task execution commands

[1159] The server determines the task to be executed based on the identified user intent and generates the command for that task. The input is information indicating the user intent, and the output is the specific command to be executed. For example, a command such as "change the camera's orientation to the living room" may be generated.

[1160] Step 6:

[1161] Sending commands to the terminal and executing them

[1162] The server sends the generated command to the terminal, which then executes it. The input is the generated command, and the output is the result of the command execution. Specifically, the terminal controls the camera motor and changes the camera's orientation as instructed.

[1163] Step 7:

[1164] Notifying users of task execution results

[1165] The device notifies the user of the results of the task. The input is the result of the command execution, and the output is a notification to the user. Specifically, the device tells the user, "The camera has been pointed at the living room," through a voice output device or a screen display.

[1166] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1167] This invention relates to a system that allows elderly people and users who are unfamiliar with PC operation to operate a computer easily. This system takes in the user's voice input, analyzes their intention, and automatically executes the corresponding task. It also incorporates an emotion engine that recognizes the user's emotions and enables responses that take those emotions into consideration.

[1168] System Overview

[1169] In this system, the user verbally commands an operation, and the device captures the speech and sends it to the server. The server performs speech recognition and natural language processing to identify the task to be performed and generate the command. The generated command is then sent to the device and executed. The system also recognizes the user's emotions and takes those emotions into consideration when carrying out the task and providing feedback. Finally, the user is notified of the results of the task.

[1170] Program processing overview

[1171] 1. Capture voice input

[1172] The device captures voice input from the user through a microphone. The user verbally issues commands such as "open a browser." The voice input may contain emotions.

[1173] 2. Sending audio data

[1174] The device sends the captured audio data to the server using an HTTP request, using secure communication methods.

[1175] 3. Voice Recognition

[1176] The server passes the received voice data to a voice recognition engine and converts it into text data. For example, the voice saying "Open the browser" is analyzed and converted into text.

[1177] 4. Natural Language Processing

[1178] The server analyzes the text data and identifies the user's intent. For example, it uses a natural language processing engine to recognize the intent of "open a browser."

[1179] 5. Emotion recognition

[1180] The server analyzes the user's emotions from the voice data. Using an emotion engine, it analyzes the tone and speed of the voice to identify the user's emotional state. For example, it can recognize whether the user is angry or happy.

[1181] 6. Generate task execution commands

[1182] The server determines the task to be performed based on the recognized intent and emotion, and generates a command for it. For example, the command "start chrome" is generated.

[1183] 7. Sending Commands

[1184] The server generates commands and sends them to the terminal. The commands are sent using HTTP requests.

[1185] 8. Executing Commands

[1186] The terminal executes the received command. In this case, the received command "start chrome" launches a browser (for example, Google Chrome).

[1187] 9. Notification of execution results

[1188] The device notifies the user of the results of the task. Based on emotion recognition, if the user is angry, for example, a message such as "I've opened the browser as you requested. Is there anything else I can help you with?" is provided to the user via voice or visual means.

[1189] Specific examples

[1190] Example 1: Open a browser

[1191] 1. User: "Open a browser." At this point, the user is in a hurry, so their voice is high-pitched and fast.

[1192] 2. Device: Captures audio and sends it to the server.

[1193] 3. Server: Converts the voice data into text and recognizes it as "opening a browser."

[1194] 4. Server: Analyze the intent to "open a browser" using natural language processing.

[1195] 5. Server: The emotion engine detects urgent situations and prioritizes quick responses.

[1196] 6. Server: Generate the command "start chrome".

[1197] 7. Server: Sends the generated commands to the terminal.

[1198] 8. Terminal: Run the command and launch the browser.

[1199] 9. On the device: The user is notified, "I've opened your browser. I see you're short on time. Is there anything else I can help you with?"

[1200] Example 2: Sending an email

[1201] 1. User: "Send me an email." The user is calm and their voice is level.

[1202] 2. Device: Captures audio and sends it to the server.

[1203] 3. Server: Converts the voice data into text and recognizes it as "send an email."

[1204] 4. Server: Analyze the intent to "send an email" using natural language processing.

[1205] 5. Server: The emotion engine detects the calm situation and proceeds with the normal procedure.

[1206] 6. Server: Obtains additional information from the user (recipient, subject, and body) needed to send the email.

[1207] 7. Server: Generates the command to "launch the mail client and send the email."

[1208] 8. Server: Sends the generated commands to the terminal.

[1209] 9. Terminal: Execute commands and send emails.

[1210] 10. Terminal: Notify the user that "Email has been sent. Please check."

[1211] In this way, the system recognizes the user's voice commands and emotions, converts them into specific PC operations, and automatically executes them, providing great convenience to elderly people and users who are unfamiliar with PC operation.

[1212] The processing flow will be explained below.

[1213] Step 1:

[1214] The user says, "Open a browser." This verbal command is the starting point of the entire system. The user's voice contains their current emotional state.

[1215] Step 2:

[1216] The device captures the user's voice input through a microphone, where the user's speech is temporarily stored in memory as audio data.

[1217] Step 3:

[1218] The device sends the captured audio data to the server using an HTTP request, using secure communication methods.

[1219] Step 4:

[1220] The server passes the received voice data to a voice recognition engine and converts it into text data. For example, the voice saying "Open the browser" is analyzed and converted into text.

[1221] Step 5:

[1222] The server passes the text data to a natural language processing engine to identify the user's intent. Here, the combination of the word "browser" and the verb "open" identifies the intent as "open a browser."

[1223] Step 6:

[1224] The server analyzes the user's emotions from the voice data, and the emotion engine analyzes voice characteristics such as tone, speed, and emphasis to identify the user's emotional state, such as whether they are in a hurry or relaxed.

[1225] Step 7:

[1226] Based on the identified intent and emotion, the server determines the task to be performed and generates the command. For example, if the user is in a hurry, the command "start chrome" is generated to "open the browser immediately," eliminating the additional confirmation step.

[1227] Step 8:

[1228] The server generates commands and sends them to the terminal. The commands are sent using HTTP requests.

[1229] Step 9:

[1230] The terminal executes the received command. In this case, the received command "start chrome" launches a browser (for example, Google Chrome).

[1231] Step 10:

[1232] The device notifies the user of the results of the task execution. Based on the emotion recognition results, if the user is in a hurry, a message such as "I've opened the browser. Is there anything else I can help you with?" is provided to the user via voice or visual means.

[1233] Specific examples

[1234] Example 1: Open a browser

[1235] 1. The user says, "Open a browser." The voice is quick and high-pitched because they're in a hurry.

[1236] 2. The device captures the audio and sends it to the server.

[1237] 3. The server converts the voice data into text and recognizes it as "opening a browser."

[1238] 4. The server uses natural language processing to analyze the intent to "open a browser."

[1239] 5. The server detects urgent situations using an emotion engine.

[1240] 6. The server generates the command "start chrome".

[1241] 7. The server sends the generated command to the terminal.

[1242] 8. The terminal executes the command and launches the browser.

[1243] 9. The device notifies the user, "Your browser is open. Is there anything else I can help you with?"

[1244] Example 2: Sending an email

[1245] 1. The user says, "Send me an email." The user is calm and the voice is even.

[1246] 2. The device captures the audio and sends it to the server.

[1247] 3. The server converts the voice data into text and recognizes it as "send an email."

[1248] 4. The server uses natural language processing to analyze the intent to "send an email."

[1249] 5. The server detects calm situations using the emotion engine.

[1250] 6. The server obtains additional information from the user (recipient, subject, and body) needed to send the email.

[1251] 7. The server generates a command to launch the mail client and send the email.

[1252] 8. The server sends the generated command to the terminal.

[1253] 9. The terminal executes the command and sends the email.

[1254] 10. The device notifies the user, "Email has been sent. Please check."

[1255] In this way, the system recognizes the user's voice commands and emotions, converts them into specific PC operations, and automatically executes them, providing great convenience to elderly people and users who are unfamiliar with PC operation.

[1256] Example 2

[1257] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1258] Today's elderly and users unfamiliar with PC operation face many difficulties when operating computers. In particular, conventional user interfaces require visual and tactile operation, which poses technical hurdles. For this reason, more intuitive and easier operation methods are needed. Furthermore, there is a lack of support that takes into account the user's emotional state. This increases stress and burden on users, resulting in a decrease in frequency of use.

[1259] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for converting voice data into text, a means for analyzing the text data by natural language processing to identify the user's intention, and a means for analyzing the user's emotions from the voice data. This allows the user to intuitively operate the computer using only voice input, and further enables the computer to respond according to their emotional state.

[1260] "Voice input" refers to instructions or information verbally given by a user.

[1261] "Capture" refers to the process of recording and importing data such as voice input.

[1262] "Audio Data" means captured audio converted into digital form.

[1263] A "server" is a computer system that processes data over a network and provides the results to other devices.

[1264] "Text conversion" refers to the process of converting audio data into corresponding text data.

[1265] "Natural language processing" is a technology that uses computers to analyze and understand human language.

[1266] "Intent" refers to the goal or request that the user is trying to achieve through voice input.

[1267] "Emotion recognition" is a technology that analyzes and identifies a user's emotional state from voice data.

[1268] A "task" is a specific operation or process that a computer or terminal must perform.

[1269] A "command" is a specific instruction or direction that a computer system is to carry out.

[1270] A "terminal" is a device with input and output capabilities that is directly operated by a user.

[1271] "Execution results" refers to the results or output obtained after a computer or terminal executes a command.

[1272] "Notification" refers to the act of informing the user of execution results or information.

[1273] This invention relates to a system that allows elderly people and users who are unfamiliar with PC operation to easily operate a computer. This system captures the user's voice input, analyzes their intention, and automatically executes the corresponding task. It also incorporates an emotion engine that recognizes the user's emotions and enables responses that take those emotions into consideration.

[1274] System Overview

[1275] The system consists of the following main components:

[1276] 1. A means to capture audio input (microphone)

[1277] 2. A means of sending the captured audio data to the server

[1278] 3. A means of converting voice data into text on the server side (voice recognition engine)

[1279] 4. A means of analyzing text data using natural language processing to identify the user's intent (natural language processing engine)

[1280] 5. A means of analyzing user emotions from voice data (emotion engine)

[1281] 6. A means to determine the task to be performed based on the identified intention and emotion and generate the command.

[1282] 7. A means for sending the generated command to the terminal and for the terminal to execute the command

[1283] 8. Means of notifying the user of the results of the task execution

[1284] Specific operation explanation

[1285] 1. Capture voice input

[1286] The device captures voice input from the user. The device has a built-in microphone and records the user's commands in real time. For example, the user might say, "Open a browser."

[1287] 2. Sending audio data

[1288] The device sends the captured audio data to the server, where it is encrypted and transmitted over a secure channel.

[1289] 3. Voice Recognition

[1290] The server passes the received voice data to a speech recognition engine, which converts the voice into text data. The speech recognition engine uses the Google Cloud Speech-to-Text API or similar. For example, the voice saying "Open your browser" is converted into text saying "Open your browser."

[1291] 4. Natural Language Processing

[1292] The server passes the text data to a natural language processing engine, which analyzes the user's intent. The natural language processing engine uses Google Cloud Natural Language API or similar. From the analysis results, the specific intent of "open a browser" is identified.

[1293] 5. Emotion recognition

[1294] The server passes the voice data to an emotion recognition engine, which analyzes the tone and speed of the voice to recognize the user's emotions. The emotion recognition engine uses the Microsoft Azure Emotion API, etc. For example, if the user is in a hurry, the server will understand their emotional state.

[1295] 6. Generate task execution commands

[1296] The server determines the task to be performed based on the user's intention and emotion, and generates the command for it. Specifically, the command "start chrome" is generated.

[1297] 7. Sending Commands

[1298] The server sends the generated command to the terminal using a secure communication method.

[1299] 8. Executing Commands

[1300] The device executes the received command, for example launching Google Chrome.

[1301] 9. Notification of execution results

[1302] The device notifies the user of the results of the task. Based on the emotion recognition results, feedback appropriate to the user's situation is provided. For example, a message such as "I've opened your browser. It seems you're short on time. Is there anything else I can help you with?" is displayed or spoken.

[1303] Specific examples

[1304] Example 1: Open a browser

[1305] 1. The user says, "Open a browser."

[1306] 2. The device captures the audio and sends it to the server.

[1307] 3. The server converts the voice data into text and recognizes it as "opening a browser."

[1308] 4. The server analyzes the intent.

[1309] 5. The server recognizes the user's sense of urgency from the voice.

[1310] 6. The server generates the command "start chrome".

[1311] 7. The server sends the command to the device.

[1312] 8. The terminal executes the command and launches the browser.

[1313] 9. Your device will say, "I've opened your browser. I see you're short on time. Is there anything else I can help you with?"

[1314] Example 2: Sending an email

[1315] 1. The user says, "Send me an email."

[1316] 2. The device captures the audio and sends it to the server.

[1317] 3. The server converts the voice data into text and recognizes it as "send an email."

[1318] 4. The server analyzes the intent.

[1319] 5. The server recognizes the user's neutral emotion from the voice.

[1320] 6. The server generates a prompt asking for additional information.

[1321] 7. The server sends a prompt to the terminal to get additional information from the user.

[1322] 8. The server generates a command to launch the email client and send the email.

[1323] 9. The server sends the command to the device.

[1324] 10. The terminal executes the command and sends the email.

[1325] 11. The device will notify you that "Email has been sent. Please check."

[1326] In this way, the system analyzes the user's voice commands in detail, converts them into specific PC operations, and automatically executes them, providing convenience that allows even elderly people and users who are unfamiliar with PC operation to easily operate a computer.

[1327] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1328] Step 1:

[1329] The device captures voice input from the user through the microphone. The input is the user's speech, for example, "Open the browser," in the form of voice data. The device receives this voice as an analog signal, converts it to a digital signal, and temporarily stores it in a buffer. Specifically, the device's built-in microphone converts the voice signal into digital data (for example, WAV format).

[1330] Step 2:

[1331] The device sends the captured audio data to the server. The input is the digital audio data obtained in step 1. This data is sent to the server using the HTTPS protocol. Specifically, the device's communication module creates an HTTP POST request and sends the audio data as a payload to the server.

[1332] Step 3:

[1333] The server passes the received voice data to a speech recognition engine, which converts the voice into text data. The input is the digital voice data sent in step 2. The output is the converted text data. Specifically, the server uses the Google Cloud Speech-to-Text API to convert the voice "Open your browser" into the text "Open your browser."

[1334] Step 4:

[1335] The server passes the text data to a natural language processing engine to analyze the user's intent. The input is the text data obtained in step 3. The output is data that identifies the user's intent. Specifically, the server uses the Google Cloud Natural Language API to analyze the text "open a browser" and identify the intent as "launch a browser."

[1336] Step 5:

[1337] The server analyzes the user's emotions from the voice data. The input is the voice data acquired in step 2. The output is the user's emotional state. Specifically, the server uses the Microsoft Azure Emotion API to analyze the tone and speed of the voice and determine whether the user is in a hurry.

[1338] Step 6:

[1339] The server determines the task to be performed based on the user's intention and emotion, and generates the command. The input is the user's intention identified in step 4 and the emotional state analyzed in step 5. The output is a specific command. Specifically, the server generates the command "start chrome".

[1340] Step 7:

[1341] The server sends the generated command to the terminal. The input is the command generated in step 6. The output is the command sent to the terminal. Specifically, the server's communication module creates an HTTP POST request and sends the command as a payload to the terminal.

[1342] Step 8:

[1343] The terminal executes the received command. The input is the command received in step 7. The output is the result of executing the command. Specifically, the terminal interprets the command "start chrome" and launches Google Chrome.

[1344] Step 9:

[1345] The terminal notifies the user of the results of the task execution. The input is the result of the command execution in step 8. The output is feedback to the user. Specifically, the terminal displays a message audibly or visually saying, "I've opened your browser. It looks like you're short on time. Is there anything else I can help you with?"

[1346] (Application example 2)

[1347] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1348] Conventional systems lack the means to intuitively operate computers and digital devices for elderly users and users unfamiliar with computer operation. This problem is particularly pronounced when complex operations are required. Furthermore, the lack of a system that flexibly responds to the user's emotions often results in a poor user experience. This creates a challenge for elderly users and users unfamiliar with technology to continue using digital environments.

[1349] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1350] In this invention, the server includes means for capturing voice input, means for transmitting the captured voice data to the server, means for the server to convert the voice data into text, means for analyzing the text data using natural language processing to identify the user's intention, means for analyzing the user's emotional state and providing flexible task execution and feedback based on the emotion, means for determining a task to be executed based on the identified intention and emotion and generating a command for the task, means for transmitting the generated command to a terminal and having the terminal execute the command, means for notifying the user of the task execution result, and navigation means for guiding the user using smart glasses. This enables even elderly people and users unfamiliar with computer operation to intuitively operate the device through voice input and smoothly use a digital environment while receiving flexible responses according to their emotional state.

[1351] A "means for capturing voice input" is a device or function for capturing voice uttered by a user.

[1352] The "means for transmitting captured audio data to a server" refers to a communication means or protocol for transmitting captured audio data to a server.

[1353] The "means for the server to convert voice data into text" is a voice recognition engine for converting received voice data into text format.

[1354] "Means for analyzing text data using natural language processing and identifying the user's intent" refers to a natural language processing engine that analyzes the converted text data to understand the user's instructions and intent.

[1355] "Means for analyzing the user's emotional state and providing flexible task execution and feedback based on those emotions" refers to an emotion recognition engine that analyzes the user's emotions from voice and other inputs and provides task execution and feedback appropriate to those emotions.

[1356] The "means for determining a task to be executed based on the identified intention and emotion and generating a command for that task" is a means for determining the optimal task according to the user's intention and emotional state and generating a command for that task.

[1357] The "means for transmitting the generated command to the terminal and for the terminal to execute the command" refers to a communication means for transmitting the generated command to the device and for causing the device to execute the command.

[1358] The "means for notifying the user of the execution result of the task" is a means for notifying the user of the execution result of the task.

[1359] A "navigation means for guiding a user using smart glasses" is a navigation system for providing visual and audio guidance and instructions to a user through smart glasses.

[1360] This invention is a system that allows elderly people and users who are unfamiliar with operating personal computers to easily operate a computer. A specific example of this system is shown below.

[1361] The main elements of the system include means for capturing a user's voice input, means for transmitting the captured voice data to a server, means for converting the voice data to text on the server side, means for analyzing the text data using natural language processing to identify the user's intention, means for analyzing the user's emotional state, means for determining a task to be performed based on the intention and emotion and generating a command for that task, means for transmitting the generated command to a terminal and for the terminal to execute the command, means for notifying the user of the result of the task execution, and navigation means for guiding the user using smart glasses.

[1362] Specifically, the system works as follows: The user wears the smart glasses and gives voice instructions (e.g., "Where is the tea shelf?"). The microphone in the smart glasses captures this voice and sends the voice data to the server using HTTPS communication. The server converts the voice data into text using Google Cloud Speech-to-Text.

[1363] The system then uses the Google Cloud Natural Language engine to analyze the converted text data and identify the user's intent. At the same time, it uses the Azure Emotion API to analyze the user's emotional state from the audio data. For example, if the system determines that the user is confused, that emotional state is also taken into account.

[1364] Based on the identified intent and emotion, the server determines the optimal task and generates a command to execute it. For example, commands such as "Start navigation" or "Guide to the tea shelf." This generated command is then sent to the smart glasses via HTTPS. The smart glasses then execute the received command and navigate the store using the built-in Google Maps API.

[1365] During navigation, the smart glasses provide real-time visual and audio guidance to the user, such as a message saying, "This way is the tea corner. Let us show you around." Once navigation is complete, feedback is also provided, such as, "You've arrived at the tea corner. Can we help you?"

[1366] Specific examples

[1367] Example 1: Guidance to the tea corner

[1368] 1. User: "Where is the tea cupboard?"

[1369] 2. Smart glasses: Capture the user's voice and send it to the server.

[1370] 3. Server: Converts speech to text and analyzes intent. Recognizes "Search for tea shelves."

[1371] 4. Server: The emotion engine recognizes "confusion."

[1372] 5. Server: Generate navigation command. "Guide me to the tea corner."

[1373] 6. Smart glasses: Start navigation and visually and audibly display "The tea corner is here. We will guide you."

[1374] 7. Smart glasses: After the guide is completed, the glasses will notify you, "You have arrived at the tea corner. Can I help you?"

[1375] Prompt Sentence Examples

[1376] The user puts on the smart glasses and asks a question by voice, for example, "Where is the tea cupboard?"

[1377] The smart glasses capture this audio and send it to the server. The server converts the audio into text using Google Cloud Speech-to-Text and identifies the intent using Google Cloud Natural Language. It then analyzes the user's emotions using the Azure Emotion API. Based on the acquired information, the server generates appropriate navigation commands and sends them to the smart glasses. The smart glasses then use the Google Maps API to perform real-time navigation and provide audio and visual guidance to the user.

[1378] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1379] Step 1:

[1380] Input: User's spoken command (e.g., "Where is the tea cupboard?")

[1381] How it works: The microphone on the device (smart glasses) captures the user's voice.

[1382] Output: Captured audio data

[1383] Step 2:

[1384] Input: Captured audio data

[1385] How it works: The device sends the captured audio data to the server using HTTPS.

[1386] Output: Audio data sent to the server

[1387] Step 3:

[1388] Input: Audio data received by the server

[1389] How it works: The server uses a speech recognition engine (Google Cloud Speech-to-Text) to convert the voice data into text.

[1390] Output: Text data (e.g., "Where is the tea shelf?")

[1391] Step 4:

[1392] Input: Converted text data

[1393] How it works: The server uses a natural language processing engine (Google Cloud Natural Language) to analyze the text data and determine the user's intent.

[1394] Output: Identified intent (e.g., "Find a tea cupboard")

[1395] Step 5:

[1396] Input: Audio data received by the server

[1397] How it works: The server uses an emotion recognition engine (Azure Emotion API) to analyze the user's emotional state from the voice data.

[1398] Output: User's emotional state (e.g., "confused")

[1399] Step 6:

[1400] Input: Identified intent and user emotional state

[1401] Action: The server determines the task to be performed based on the recognized intent and emotion, and generates the command.

[1402] Output: Generated command (e.g. "Take me to the tea corner")

[1403] Step 7:

[1404] Input: Generated command

[1405] Operation: The server sends the generated command to the device using HTTPS communication.

[1406] Output: Command sent to the terminal

[1407] Step 8:

[1408] Input: The command received by the terminal

[1409] How it works: The device (smart glasses) starts real-time navigation using the Google Maps API.

[1410] Output: Visual and audio navigation instructions

[1411] Step 9:

[1412] Input: Navigation result

[1413] Behavior: The device notifies the user of the results of the task, for example by displaying a message like "You've arrived at the tea corner. How can I help you?"

[1414] Output: Feedback notification to the user

[1415] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1416] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1417] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1418] [Fourth embodiment]

[1419] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1420] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1421] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1422] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1423] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1424] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1425] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1426] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1427] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1428] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1429] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1430] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1431] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1432] The present invention relates to a system that allows elderly people and users who are unfamiliar with PC operation to easily operate a computer. The system captures the user's voice input, analyzes their intention, and automatically executes the corresponding task.

[1433] System Overview

[1434] In this system, the user verbally commands an operation, and the device captures the speech and sends it to the server. The server performs speech recognition and natural language processing to identify the task to be performed and generate the command. The generated command is then sent to the device and executed. Finally, the user is notified of the result of the task execution.

[1435] Program processing overview

[1436] 1. Capture voice input

[1437] The device captures voice input from the user through a microphone, for example, by giving verbal commands such as "open a browser."

[1438] 2. Sending audio data

[1439] The device transmits the captured audio data to the server, which uses a secure communication method.

[1440] 3. Voice Recognition

[1441] The server converts the received voice data into text data, using a speech recognition engine on the server.

[1442] 4. Natural Language Processing

[1443] The server analyzes the text data and identifies the user's intent. For example, it uses a natural language processing engine to recognize the intent of "open a browser."

[1444] 5. Generate task execution commands

[1445] Based on the intent recognized, the server determines the task to be performed and generates the corresponding command, for example, "start chrome" as the command to "open a browser."

[1446] 6. Sending Commands

[1447] The server generates commands and sends them to the terminal.

[1448] 7. Executing commands

[1449] The device executes the received command and performs the specified task, in this case launching a browser.

[1450] 8. Notification of execution results

[1451] The device notifies the user of the result of the task execution, for example, by voice or on-screen notification that "the browser has been opened."

[1452] Specific examples

[1453] Example 1: Open a browser

[1454] 1. User: Says "Open a browser."

[1455] 2. Device: Captures the user's voice and sends it to the server.

[1456] 3. Server: Converts the voice data into text and recognizes it as "opening a browser."

[1457] 4. Server: Analyze the intent to "open a browser" using natural language processing.

[1458] 5. Server: Generate the command "start chrome".

[1459] 6. Server: Sends the generated commands to the terminal.

[1460] 7. Terminal: Execute the command and launch the browser.

[1461] 8. Terminal: Notify the user that the browser has been opened.

[1462] Example 2: Sending an email

[1463] 1. User: Says "Send me an email."

[1464] 2. Device: Captures the user's voice and sends it to the server.

[1465] 3. Server: Converts the voice data into text and recognizes it as "send an email."

[1466] 4. Server: Analyze the intent to "send an email" using natural language processing.

[1467] 5. Server: Obtains additional information from the user (recipient, subject, and body) needed to send the email.

[1468] 6. Server: Generates the command to "launch the mail client and send the email."

[1469] 7. Server: Sends the generated commands to the terminal.

[1470] 8. Terminal: Execute commands and send emails.

[1471] 9. Terminal: Notify the user that the email has been sent.

[1472] This system allows users to operate a computer using only voice input, providing great convenience, especially for elderly people and those who are unfamiliar with PC operation.

[1473] The processing flow will be explained below.

[1474] Step 1:

[1475] The user says, "Open a browser." The user verbally instructs the computer to operate, and this instruction is the starting point for the entire system.

[1476] Step 2:

[1477] The device captures the user's voice input through a microphone, where the user's speech is temporarily stored in memory as audio data.

[1478] Step 3:

[1479] The device sends the captured audio data to the server using an HTTP request, using a secure communication method.

[1480] Step 4:

[1481] The server passes the received voice data to a voice recognition engine and converts it into text data. For example, the voice saying "Open the browser" is analyzed and converted into text.

[1482] Step 5:

[1483] The server passes the text data to a natural language processing engine, which analyzes the user's intent. Here, the combination of the word "browser" and the verb "open" identifies the intent as "open a browser."

[1484] Step 6:

[1485] The server determines the task based on the specified intent and generates a command to perform that task, for example "start chrome".

[1486] Step 7:

[1487] The server generates commands and sends them to the device, which then receives them using HTTP requests.

[1488] Step 8:

[1489] The terminal executes the received command. In this case, the received command "start chrome" launches a browser (for example, Google Chrome).

[1490] Step 9:

[1491] The terminal notifies the user of the result of the task execution. If the browser is successfully launched, the user is given a voice or visual message saying "Browser opened."

[1492] Example 1

[1493] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1494] There was a need to provide an environment where elderly people and users unfamiliar with PC operation could use computers easily without complex operations. However, conventional systems had low accuracy in recognizing voice input and instructions, and intended operations were often not executed. This reduced user convenience and impaired the practicality of voice operation.

[1495] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1496] In this invention, the server includes means for converting voice data into text data, means for analyzing the text data using natural language processing technology to identify the user's intention, and means for determining the process to be performed based on the identified intention and generating instructions for that process. This makes it possible to recognize the user's intention with high accuracy and generate and execute appropriate instructions.

[1497] "Means for capturing voice input" refers to a device or method that captures voice spoken by a user and converts it into a digital signal.

[1498] "Means for transmitting captured audio data to a computer using a communication network" refers to the technology or method for transferring acquired audio data to a computer located in a remote location, such as a server, via a network such as the Internet.

[1499] "Means for converting voice data into text data by an electronic computer" refers to the process by which a server or computer uses voice recognition technology to convert voice into text information.

[1500] "Means of analyzing text data using natural language processing technology to identify the user's intent" refers to the technology and methods for analyzing acquired text data and understanding its meaning and requests.

[1501] "Means for determining the processing to be performed based on the identified intent and generating instructions for that" refers to the process of determining what operations or processing to perform based on the analysis results and generating the corresponding commands or instructions.

[1502] "Means for sending the generated instructions to the terminal and for the terminal to execute the instructions" refers to a technology or method for sending operation instructions generated by a server to a user's device and for the device to execute the received instructions.

[1503] "Means for notifying the user of the results of processing" refers to techniques or methods for visually or audibly conveying the results of the processing to the user.

[1504] "Electronic computer" means a computing device, server, or other computing equipment capable of receiving, processing, and analyzing data.

[1505] "Audio collection device" means a device for collecting a user's voice, including a microphone or other audio input device.

[1506] This invention provides a system that allows elderly people and users who are unfamiliar with PC operation to easily operate a computer. Specifically, it is a system that performs various tasks using voice input. This system uses the following main hardware and software:

[1507] Hardware and software used

[1508] 1. Terminal (user device)

[1509] Microphone: An audio collection device used to capture a user's voice input. This includes a computer's built-in microphone and an external USB microphone.

[1510] Network interface: A communication device for sending audio data to a server. An internet connection is required.

[1511] 2. Server

[1512] Speech recognition engine: Software for converting voice data into text data. For example, the Google Cloud Speech-to-Text API is used.

[1513] Natural language processing engine: Software used to analyze text data and identify user intent. For example, the Google Cloud Natural Language API is used.

[1514] Task execution command generation engine: Software that determines the process to be performed based on the user's intentions and generates the instructions.

[1515] System examples and operation procedures

[1516] As a concrete example for implementing this system, the tasks "open a browser" and "send an email" are taken.

[1517] Example 1: Open a browser

[1518] 1. User: Says "Open a browser."

[1519] 2. Terminal: Captures the user's voice with a microphone and sends the voice data to the server, which is securely transmitted using TLS.

[1520] 3. Server: Receives the audio data and converts it to text using the Google Cloud Speech-to-Text API.

[1521] 4. Server: Analyze the text data using the Google Cloud Natural Language API to identify the intent to "open a browser."

[1522] 5. Server: Generates a task execution command called "start chrome" based on the instruction "open browser".

[1523] 6. Server: Sends the generated commands to the terminal.

[1524] 7. Terminal: Executes the received command and launches the browser.

[1525] 8. On the device: The user is notified by voice or on screen that the browser has been opened.

[1526] Example 2: Sending an email

[1527] 1. User: Says "Send me an email."

[1528] 2. Terminal: Captures the user's voice with a microphone and sends the voice data to the server, which is securely transmitted using TLS.

[1529] 3. Server: Receives the audio data and converts it to text using the Google Cloud Speech-to-Text API.

[1530] 4. Server: Analyze the text data using the Google Cloud Natural Language API and identify the intent to "send an email."

[1531] 5. Server: Obtain additional information from the user (recipient, subject, and message) needed to send the email. For example, confirm "recipient is example@example.com, subject is test, message is hello."

[1532] 6. Server: Generates a command to "launch the mail client and send the email" and includes the necessary information.

[1533] 7. Server: Sends the generated commands to the terminal.

[1534] 8. Terminal: Executes the received command and sends the email.

[1535] 9. Terminal: Notify the user by voice or on-screen message that "Email has been sent."

[1536] This system allows users to easily operate a computer by voice input, providing great convenience especially for the elderly and people who are unfamiliar with using computers.

[1537] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1538] Step 1:

[1539] Input: User's voice input.

[1540] How it works: The device uses the microphone to capture the user's voice input, as the user speaks into the microphone, "open browser."

[1541] Output: Audio data in digital format.

[1542] Step 2:

[1543] Input: Audio data in digital format.

[1544] How it works: The device captures audio data and sends it to the server using a secure communication protocol (e.g., HTTPS). The audio data is encrypted with TLS and transmitted securely over the network.

[1545] Output: The audio data sent to the server.

[1546] Step 3:

[1547] Input: The audio data sent to the server.

[1548] How it works: The server converts the received voice data into text data using the Google Cloud Speech-to-Text API. The server's speech recognition engine analyzes the voice data and generates a string of text.

[1549] Output: Data in text format (e.g. "Open browser").

[1550] Step 4:

[1551] Input: Data in text format.

[1552] How it works: The server parses the text data using the Google Cloud Natural Language API. A natural language processing engine on the server tokenizes the text data and identifies the intent "open a browser." This process involves morphological analysis and contextual understanding.

[1553] Output: The user intent (e.g., open a browser).

[1554] Step 5:

[1555] Input: User intent.

[1556] How it works: The server generates an execution command such as "start chrome" based on the user's intent. The command generation engine on the server creates a script or shell command corresponding to the specified task and validates it as necessary.

[1557] Output: The generated command to run (e.g. start chrome).

[1558] Step 6:

[1559] Input: The generated execution command.

[1560] How it works: The server sends the generated command to the device. A secure communication protocol is used to ensure the command reaches the device. The command is encrypted and then decrypted by the device.

[1561] Output: Execution instructions sent to the terminal.

[1562] Step 7:

[1563] Input: Execution instructions sent to the terminal.

[1564] Operation: The device executes the received command and performs the specified task. In this case, the "start chrome" command is executed and the browser is launched. The device monitors the command execution result and determines whether it was successful or failed.

[1565] Output: The result of the task execution (e.g., a browser was opened).

[1566] Step 8:

[1567] Input: The result of the task execution.

[1568] Action: The device notifies the user of the execution result. For example, it uses a speech synthesis engine (e.g., Google Text-to-Speech) to announce "Browser opened" by voice, or displays the message "Browser opened" on the screen.

[1569] Output: Notification to the user (e.g. audio notification, screen display).

[1570] (Application example 1)

[1571] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1572] Security management systems require easy voice control of important security functions, such as camera operation, alarm setting, locking and unlocking, without requiring complex operations, especially for elderly users and those unfamiliar with technology. However, existing technologies lack a consistent system for achieving such voice control. Other issues include the accuracy of voice recognition, appropriate generation of execution commands, and clear notification of execution results to users.

[1573] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1574] In this invention, the server includes means for capturing voice input, means for transmitting the captured voice data to the server, means for the server to convert the voice data into text, means for analyzing the text data using natural language processing and identifying the user's intention, means for determining a task to be executed based on the identified intention and generating a command for that task, means for transmitting the generated command to the terminal and having the terminal execute the command, means for controlling security functions such as operating a camera, setting an alarm, locking or unlocking using voice input, and means for notifying the user of the result of task execution. This enables even elderly people and users who are unfamiliar with technology to easily control security functions using only voice input.

[1575] A "means for capturing voice input" is a technical device for detecting voice from a user and collecting that data.

[1576] The "means for transmitting captured voice data to a server" is a technical device that transmits collected voice data to a server using a communication means such as the Internet.

[1577] The "means for the server to convert voice data into text" refers to a technical device that converts voice data into text information using a voice recognition engine.

[1578] "Means for analyzing text data using natural language processing and identifying user intent" refers to a technical device that uses natural language processing technology to analyze text data and understand what the user wants.

[1579] The "means for determining a task to be executed based on the identified intention and generating a command for that task" is a technical device that determines an operation according to the user's intention and creates a corresponding command.

[1580] The "means for sending the generated command to the terminal and for the terminal to execute the command" is a technical device that sends the generated command to the terminal and the terminal executes the task according to the command.

[1581] "Means for controlling security functions such as operating cameras, setting alarms, locking and unlocking doors, etc. using voice input" refers to a technical device that uses voice input to operate security functions such as changing the direction of cameras, setting alarms, and locking and unlocking doors.

[1582] The "means for notifying the user of the results of the task execution" is a technical device that notifies the user of the results of the executed task by voice, display, or other means.

[1583] The present invention is a system for controlling a security control system using voice input, which takes voice input from a user and transmits the voice data to a server, which analyzes the voice data, identifies a corresponding task, and generates a command to perform the task.

[1584] The system includes the following components:

[1585] 1. Capture voice input

[1586] The user issues a command by voice, such as "point the camera at the living room." The microphone on the device captures this voice. The hardware used is a standard voice capture device.

[1587] 2. Sending audio data to the server

[1588] The device transmits the captured audio data to a server via a communication method such as the Internet. The communication uses a secure protocol (e.g., HTTPS).

[1589] 3. Voice Recognition

[1590] The server analyzes the received voice data and converts it into text information. The software used for this is a speech recognition engine (for example, Google Speech-to-Text API). The converted text data is used in the next step.

[1591] 4. Natural Language Processing

[1592] The server analyzes the text data obtained by the speech recognition engine using a natural language processing engine (e.g., spaCy or NLTK) to identify the user's intent. For example, the server recognizes the instruction "point the camera toward the living room" as "adjust the camera toward the living room."

[1593] 5. Generate task execution commands

[1594] The server determines the task to be performed based on the user's intent and generates a command for that task. For example, for the intent "point the camera at the living room," the server generates a command to "change the camera's orientation to the living room."

[1595] 6. Sending commands to the terminal and executing them

[1596] The generated command is then sent back to the device, which then executes it. For example, a mechanical control system for adjusting the orientation of a camera on the device is activated.

[1597] 7. Notifying the user of the task execution results

[1598] The device notifies the user of the execution result of the instructed task. For example, the device may notify the user by voice or by displaying a message on the screen saying, "The camera has been redirected to the living room."

[1599] Specific examples

[1600] Example 1: Camera operation

[1601] 1. The user says, "Point the camera at the living room."

[1602] 2. The device captures this audio and sends it to the server.

[1603] 3. The server converts the voice data into text and recognizes it as "point the camera towards the living room."

[1604] 4. The server parses this intent and generates a command to point the camera towards the living room.

[1605] 5. The generated command is sent to the device, and the camera turns as instructed.

[1606] 6. The device will announce, "The camera has been pointed at the living room."

[1607] Prompt Sentence Examples

[1608] When the user says "point the camera towards the living room," how does the system adjust the camera's orientation to face the living room? Please explain step by step.

[1609] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1610] Step 1:

[1611] Capturing voice input

[1612] The user issues a command, for example, "point the camera at the living room," and the microphone on the device captures the user's voice. The input is voice data, which is used for the next step.

[1613] Step 2:

[1614] Sending audio data to the server

[1615] The device sends the captured audio data to the server over the Internet using a secure protocol (e.g., HTTPS). The input is the captured audio data, and the output is the data sent to the server.

[1616] Step 3:

[1617] Voice Recognition

[1618] The server analyzes the received voice data and converts it into text information. The software used is a speech recognition engine (e.g., Google Speech-to-Text API). The input is voice data and the output is text data. Specifically, the speech recognition engine extracts acoustic features and matches them with known character patterns.

[1619] Step 4:

[1620] Natural Language Processing

[1621] The server analyzes the text data obtained through speech recognition to determine the user's intent. The software used is a natural language processing engine (e.g., spaCy or NLTK). The input is text data, and the output is information indicating the user's intent. Specifically, grammar analysis and context understanding are performed.

[1622] Step 5:

[1623] Generate task execution commands

[1624] The server determines the task to be executed based on the identified user intent and generates the command for that task. The input is information indicating the user intent, and the output is the specific command to be executed. For example, a command such as "change the camera's orientation to the living room" may be generated.

[1625] Step 6:

[1626] Sending commands to the terminal and executing them

[1627] The server sends the generated command to the terminal, which then executes it. The input is the generated command, and the output is the result of the command execution. Specifically, the terminal controls the camera motor and changes the camera's orientation as instructed.

[1628] Step 7:

[1629] Notifying users of task execution results

[1630] The device notifies the user of the results of the task. The input is the result of the command execution, and the output is a notification to the user. Specifically, the device tells the user, "The camera has been pointed at the living room," through a voice output device or a screen display.

[1631] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1632] This invention relates to a system that allows elderly people and users who are unfamiliar with PC operation to operate a computer easily. This system takes in the user's voice input, analyzes their intention, and automatically executes the corresponding task. It also incorporates an emotion engine that recognizes the user's emotions and enables responses that take those emotions into consideration.

[1633] System Overview

[1634] In this system, the user verbally commands an operation, and the device captures the speech and sends it to the server. The server performs speech recognition and natural language processing to identify the task to be performed and generate the command. The generated command is then sent to the device and executed. The system also recognizes the user's emotions and takes those emotions into consideration when carrying out the task and providing feedback. Finally, the user is notified of the results of the task.

[1635] Program processing overview

[1636] 1. Capture voice input

[1637] The device captures voice input from the user through a microphone. The user verbally issues commands such as "open a browser." The voice input may contain emotions.

[1638] 2. Sending audio data

[1639] The device sends the captured audio data to the server using an HTTP request, using secure communication methods.

[1640] 3. Voice Recognition

[1641] The server passes the received voice data to a voice recognition engine and converts it into text data. For example, the voice saying "Open the browser" is analyzed and converted into text.

[1642] 4. Natural Language Processing

[1643] The server analyzes the text data and identifies the user's intent. For example, it uses a natural language processing engine to recognize the intent of "open a browser."

[1644] 5. Emotion recognition

[1645] The server analyzes the user's emotions from the voice data. Using an emotion engine, it analyzes the tone and speed of the voice to identify the user's emotional state. For example, it can recognize whether the user is angry or happy.

[1646] 6. Generate task execution commands

[1647] The server determines the task to be performed based on the recognized intent and emotion, and generates a command for it. For example, the command "start chrome" is generated.

[1648] 7. Sending Commands

[1649] The server generates commands and sends them to the terminal. The commands are sent using HTTP requests.

[1650] 8. Executing Commands

[1651] The terminal executes the received command. In this case, the received command "start chrome" launches a browser (for example, Google Chrome).

[1652] 9. Notification of execution results

[1653] The device notifies the user of the results of the task. Based on emotion recognition, if the user is angry, for example, a message such as "I've opened the browser as you requested. Is there anything else I can help you with?" is provided to the user via voice or visual means.

[1654] Specific examples

[1655] Example 1: Open a browser

[1656] 1. User: "Open a browser." At this point, the user is in a hurry, so their voice is high-pitched and fast.

[1657] 2. Device: Captures audio and sends it to the server.

[1658] 3. Server: Converts the voice data into text and recognizes it as "opening a browser."

[1659] 4. Server: Analyze the intent to "open a browser" using natural language processing.

[1660] 5. Server: The emotion engine detects urgent situations and prioritizes quick responses.

[1661] 6. Server: Generate the command "start chrome".

[1662] 7. Server: Sends the generated commands to the terminal.

[1663] 8. Terminal: Run the command and launch the browser.

[1664] 9. On the device: The user is notified, "I've opened your browser. I see you're short on time. Is there anything else I can help you with?"

[1665] Example 2: Sending an email

[1666] 1. User: "Send me an email." The user is calm and their voice is level.

[1667] 2. Device: Captures audio and sends it to the server.

[1668] 3. Server: Converts the voice data into text and recognizes it as "send an email."

[1669] 4. Server: Analyze the intent to "send an email" using natural language processing.

[1670] 5. Server: The emotion engine detects the calm situation and proceeds with the normal procedure.

[1671] 6. Server: Obtains additional information from the user (recipient, subject, and body) needed to send the email.

[1672] 7. Server: Generates the command to "launch the mail client and send the email."

[1673] 8. Server: Sends the generated commands to the terminal.

[1674] 9. Terminal: Execute commands and send emails.

[1675] 10. Terminal: Notify the user that "Email has been sent. Please check."

[1676] In this way, the system recognizes the user's voice commands and emotions, converts them into specific PC operations, and automatically executes them, providing great convenience to elderly people and users who are unfamiliar with PC operation.

[1677] The processing flow will be explained below.

[1678] Step 1:

[1679] The user says, "Open a browser." This verbal command is the starting point of the entire system. The user's voice contains their current emotional state.

[1680] Step 2:

[1681] The device captures the user's voice input through a microphone, where the user's speech is temporarily stored in memory as audio data.

[1682] Step 3:

[1683] The device sends the captured audio data to the server using an HTTP request, using secure communication methods.

[1684] Step 4:

[1685] The server passes the received voice data to a voice recognition engine and converts it into text data. For example, the voice saying "Open the browser" is analyzed and converted into text.

[1686] Step 5:

[1687] The server passes the text data to a natural language processing engine to identify the user's intent. Here, the combination of the word "browser" and the verb "open" identifies the intent as "open a browser."

[1688] Step 6:

[1689] The server analyzes the user's emotions from the voice data, and the emotion engine analyzes voice characteristics such as tone, speed, and emphasis to identify the user's emotional state, such as whether they are in a hurry or relaxed.

[1690] Step 7:

[1691] Based on the identified intent and emotion, the server determines the task to be performed and generates the command. For example, if the user is in a hurry, the command "start chrome" is generated to "open the browser immediately," eliminating the additional confirmation step.

[1692] Step 8:

[1693] The server generates commands and sends them to the terminal. The commands are sent using HTTP requests.

[1694] Step 9:

[1695] The terminal executes the received command. In this case, the received command "start chrome" launches a browser (for example, Google Chrome).

[1696] Step 10:

[1697] The device notifies the user of the results of the task execution. Based on the emotion recognition results, if the user is in a hurry, a message such as "I've opened the browser. Is there anything else I can help you with?" is provided to the user via voice or visual means.

[1698] Specific examples

[1699] Example 1: Open a browser

[1700] 1. The user says, "Open a browser." The voice is quick and high-pitched because they're in a hurry.

[1701] 2. The device captures the audio and sends it to the server.

[1702] 3. The server converts the voice data into text and recognizes it as "opening a browser."

[1703] 4. The server uses natural language processing to analyze the intent to "open a browser."

[1704] 5. The server detects urgent situations using an emotion engine.

[1705] 6. The server generates the command "start chrome".

[1706] 7. The server sends the generated command to the terminal.

[1707] 8. The terminal executes the command and launches the browser.

[1708] 9. The device notifies the user, "Your browser is open. Is there anything else I can help you with?"

[1709] Example 2: Sending an email

[1710] 1. The user says, "Send me an email." The user is calm and the voice is even.

[1711] 2. The device captures the audio and sends it to the server.

[1712] 3. The server converts the voice data into text and recognizes it as "send an email."

[1713] 4. The server uses natural language processing to analyze the intent to "send an email."

[1714] 5. The server detects calm situations using the emotion engine.

[1715] 6. The server obtains additional information from the user (recipient, subject, and body) needed to send the email.

[1716] 7. The server generates a command to launch the mail client and send the email.

[1717] 8. The server sends the generated command to the terminal.

[1718] 9. The terminal executes the command and sends the email.

[1719] 10. The device notifies the user, "Email has been sent. Please check."

[1720] In this way, the system recognizes the user's voice commands and emotions, converts them into specific PC operations, and automatically executes them, providing great convenience to elderly people and users who are unfamiliar with PC operation.

[1721] Example 2

[1722] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1723] Today's elderly and users unfamiliar with PC operation face many difficulties when operating computers. In particular, conventional user interfaces require visual and tactile operation, which poses technical hurdles. For this reason, more intuitive and easier operation methods are needed. Furthermore, there is a lack of support that takes into account the user's emotional state. This increases stress and burden on users, resulting in a decrease in frequency of use.

[1724] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for converting voice data into text, a means for analyzing the text data by natural language processing to identify the user's intention, and a means for analyzing the user's emotions from the voice data. This allows the user to intuitively operate the computer using only voice input, and further enables the computer to respond according to their emotional state.

[1725] "Voice input" refers to instructions or information verbally given by a user.

[1726] "Capture" refers to the process of recording and importing data such as voice input.

[1727] "Audio Data" means captured audio converted into digital form.

[1728] A "server" is a computer system that processes data over a network and provides the results to other devices.

[1729] "Text conversion" refers to the process of converting audio data into corresponding text data.

[1730] "Natural language processing" is a technology that uses computers to analyze and understand human language.

[1731] "Intent" refers to the goal or request that the user is trying to achieve through voice input.

[1732] "Emotion recognition" is a technology that analyzes and identifies a user's emotional state from voice data.

[1733] A "task" is a specific operation or process that a computer or terminal must perform.

[1734] A "command" is a specific instruction or direction that a computer system is to carry out.

[1735] A "terminal" is a device with input and output capabilities that is directly operated by a user.

[1736] "Execution results" refers to the results or output obtained after a computer or terminal executes a command.

[1737] "Notification" refers to the act of informing the user of execution results or information.

[1738] This invention relates to a system that allows elderly people and users who are unfamiliar with PC operation to easily operate a computer. This system captures the user's voice input, analyzes their intention, and automatically executes the corresponding task. It also incorporates an emotion engine that recognizes the user's emotions and enables responses that take those emotions into consideration.

[1739] System Overview

[1740] The system consists of the following main components:

[1741] 1. A means to capture audio input (microphone)

[1742] 2. A means of sending the captured audio data to the server

[1743] 3. A means of converting voice data into text on the server side (voice recognition engine)

[1744] 4. A means of analyzing text data using natural language processing to identify the user's intent (natural language processing engine)

[1745] 5. A means of analyzing user emotions from voice data (emotion engine)

[1746] 6. A means to determine the task to be performed based on the identified intention and emotion and generate the command.

[1747] 7. A means for sending the generated command to the terminal and for the terminal to execute the command

[1748] 8. Means of notifying the user of the results of the task execution

[1749] Specific operation explanation

[1750] 1. Capture voice input

[1751] The device captures voice input from the user. The device has a built-in microphone and records the user's commands in real time. For example, the user might say, "Open a browser."

[1752] 2. Sending audio data

[1753] The device sends the captured audio data to the server, where it is encrypted and transmitted over a secure channel.

[1754] 3. Voice Recognition

[1755] The server passes the received voice data to a speech recognition engine, which converts the voice into text data. The speech recognition engine uses the Google Cloud Speech-to-Text API or similar. For example, the voice saying "Open your browser" is converted into text saying "Open your browser."

[1756] 4. Natural Language Processing

[1757] The server passes the text data to a natural language processing engine, which analyzes the user's intent. The natural language processing engine uses Google Cloud Natural Language API or similar. From the analysis results, the specific intent of "open a browser" is identified.

[1758] 5. Emotion recognition

[1759] The server passes the voice data to an emotion recognition engine, which analyzes the tone and speed of the voice to recognize the user's emotions. The emotion recognition engine uses the Microsoft Azure Emotion API, etc. For example, if the user is in a hurry, the server will understand their emotional state.

[1760] 6. Generate task execution commands

[1761] The server determines the task to be performed based on the user's intention and emotion, and generates the command for it. Specifically, the command "start chrome" is generated.

[1762] 7. Sending Commands

[1763] The server sends the generated command to the terminal using a secure communication method.

[1764] 8. Executing Commands

[1765] The device executes the received command, for example launching Google Chrome.

[1766] 9. Notification of execution results

[1767] The device notifies the user of the results of the task. Based on the emotion recognition results, feedback appropriate to the user's situation is provided. For example, a message such as "I've opened your browser. It seems you're short on time. Is there anything else I can help you with?" is displayed or spoken.

[1768] Specific examples

[1769] Example 1: Open a browser

[1770] 1. The user says, "Open a browser."

[1771] 2. The device captures the audio and sends it to the server.

[1772] 3. The server converts the voice data into text and recognizes it as "opening a browser."

[1773] 4. The server analyzes the intent.

[1774] 5. The server recognizes the user's sense of urgency from the voice.

[1775] 6. The server generates the command "start chrome".

[1776] 7. The server sends the command to the device.

[1777] 8. The terminal executes the command and launches the browser.

[1778] 9. Your device will say, "I've opened your browser. I see you're short on time. Is there anything else I can help you with?"

[1779] Example 2: Sending an email

[1780] 1. The user says, "Send me an email."

[1781] 2. The device captures the audio and sends it to the server.

[1782] 3. The server converts the voice data into text and recognizes it as "send an email."

[1783] 4. The server analyzes the intent.

[1784] 5. The server recognizes the user's neutral emotion from the voice.

[1785] 6. The server generates a prompt asking for additional information.

[1786] 7. The server sends a prompt to the terminal to get additional information from the user.

[1787] 8. The server generates a command to launch the email client and send the email.

[1788] 9. The server sends the command to the device.

[1789] 10. The terminal executes the command and sends the email.

[1790] 11. The device will notify you that "Email has been sent. Please check."

[1791] In this way, the system analyzes the user's voice commands in detail, converts them into specific PC operations, and automatically executes them, providing convenience that allows even elderly people and users who are unfamiliar with PC operation to easily operate a computer.

[1792] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1793] Step 1:

[1794] The device captures voice input from the user through the microphone. The input is the user's speech, for example, "Open the browser," in the form of voice data. The device receives this voice as an analog signal, converts it to a digital signal, and temporarily stores it in a buffer. Specifically, the device's built-in microphone converts the voice signal into digital data (for example, WAV format).

[1795] Step 2:

[1796] The device sends the captured audio data to the server. The input is the digital audio data obtained in step 1. This data is sent to the server using the HTTPS protocol. Specifically, the device's communication module creates an HTTP POST request and sends the audio data as a payload to the server.

[1797] Step 3:

[1798] The server passes the received voice data to a speech recognition engine, which converts the voice into text data. The input is the digital voice data sent in step 2. The output is the converted text data. Specifically, the server uses the Google Cloud Speech-to-Text API to convert the voice "Open your browser" into the text "Open your browser."

[1799] Step 4:

[1800] The server passes the text data to a natural language processing engine to analyze the user's intent. The input is the text data obtained in step 3. The output is data that identifies the user's intent. Specifically, the server uses the Google Cloud Natural Language API to analyze the text "open a browser" and identify the intent as "launch a browser."

[1801] Step 5:

[1802] The server analyzes the user's emotions from the voice data. The input is the voice data acquired in step 2. The output is the user's emotional state. Specifically, the server uses the Microsoft Azure Emotion API to analyze the tone and speed of the voice and determine whether the user is in a hurry.

[1803] Step 6:

[1804] The server determines the task to be performed based on the user's intention and emotion, and generates the command. The input is the user's intention identified in step 4 and the emotional state analyzed in step 5. The output is a specific command. Specifically, the server generates the command "start chrome".

[1805] Step 7:

[1806] The server sends the generated command to the terminal. The input is the command generated in step 6. The output is the command sent to the terminal. Specifically, the server's communication module creates an HTTP POST request and sends the command as a payload to the terminal.

[1807] Step 8:

[1808] The terminal executes the received command. The input is the command received in step 7. The output is the result of executing the command. Specifically, the terminal interprets the command "start chrome" and launches Google Chrome.

[1809] Step 9:

[1810] The terminal notifies the user of the results of the task execution. The input is the result of the command execution in step 8. The output is feedback to the user. Specifically, the terminal displays a message audibly or visually saying, "I've opened your browser. It looks like you're short on time. Is there anything else I can help you with?"

[1811] (Application example 2)

[1812] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1813] Conventional systems lack the means to intuitively operate computers and digital devices for elderly users and users unfamiliar with computer operation. This problem is particularly pronounced when complex operations are required. Furthermore, the lack of a system that flexibly responds to the user's emotions often results in a poor user experience. This creates a challenge for elderly users and users unfamiliar with technology to continue using digital environments.

[1814] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1815] In this invention, the server includes means for capturing voice input, means for transmitting the captured voice data to the server, means for the server to convert the voice data into text, means for analyzing the text data using natural language processing to identify the user's intention, means for analyzing the user's emotional state and providing flexible task execution and feedback based on the emotion, means for determining a task to be executed based on the identified intention and emotion and generating a command for the task, means for transmitting the generated command to a terminal and having the terminal execute the command, means for notifying the user of the task execution result, and navigation means for guiding the user using smart glasses. This enables even elderly people and users unfamiliar with computer operation to intuitively operate the device through voice input and smoothly use a digital environment while receiving flexible responses according to their emotional state.

[1816] A "means for capturing voice input" is a device or function for capturing voice uttered by a user.

[1817] The "means for transmitting captured audio data to a server" refers to a communication means or protocol for transmitting captured audio data to a server.

[1818] The "means for the server to convert voice data into text" is a voice recognition engine for converting received voice data into text format.

[1819] "Means for analyzing text data using natural language processing and identifying the user's intent" refers to a natural language processing engine that analyzes the converted text data to understand the user's instructions and intent.

[1820] "Means for analyzing the user's emotional state and providing flexible task execution and feedback based on those emotions" refers to an emotion recognition engine that analyzes the user's emotions from voice and other inputs and provides task execution and feedback appropriate to those emotions.

[1821] The "means for determining a task to be executed based on the identified intention and emotion and generating a command for that task" is a means for determining the optimal task according to the user's intention and emotional state and generating a command for that task.

[1822] The "means for transmitting the generated command to the terminal and for the terminal to execute the command" refers to a communication means for transmitting the generated command to the device and for causing the device to execute the command.

[1823] The "means for notifying the user of the execution result of the task" is a means for notifying the user of the execution result of the task.

[1824] A "navigation means for guiding a user using smart glasses" is a navigation system for providing visual and audio guidance and instructions to a user through smart glasses.

[1825] This invention is a system that allows elderly people and users who are unfamiliar with operating personal computers to easily operate a computer. A specific example of this system is shown below.

[1826] The main elements of the system include means for capturing a user's voice input, means for transmitting the captured voice data to a server, means for converting the voice data to text on the server side, means for analyzing the text data using natural language processing to identify the user's intention, means for analyzing the user's emotional state, means for determining a task to be performed based on the intention and emotion and generating a command for that task, means for transmitting the generated command to a terminal and for the terminal to execute the command, means for notifying the user of the result of the task execution, and navigation means for guiding the user using smart glasses.

[1827] Specifically, the system works as follows: The user wears the smart glasses and gives voice instructions (e.g., "Where is the tea shelf?"). The microphone in the smart glasses captures this voice and sends the voice data to the server using HTTPS communication. The server converts the voice data into text using Google Cloud Speech-to-Text.

[1828] The system then uses the Google Cloud Natural Language engine to analyze the converted text data and identify the user's intent. At the same time, it uses the Azure Emotion API to analyze the user's emotional state from the audio data. For example, if the system determines that the user is confused, that emotional state is also taken into account.

[1829] Based on the identified intent and emotion, the server determines the optimal task and generates a command to execute it. For example, commands such as "Start navigation" or "Guide to the tea shelf." This generated command is then sent to the smart glasses via HTTPS. The smart glasses then execute the received command and navigate the store using the built-in Google Maps API.

[1830] During navigation, the smart glasses provide real-time visual and audio guidance to the user, such as a message saying, "This way is the tea corner. Let us show you around." Once navigation is complete, feedback is also provided, such as, "You've arrived at the tea corner. Can we help you?"

[1831] Specific examples

[1832] Example 1: Guidance to the tea corner

[1833] 1. User: "Where is the tea cupboard?"

[1834] 2. Smart glasses: Capture the user's voice and send it to the server.

[1835] 3. Server: Converts speech to text and analyzes intent. Recognizes "Search for tea shelves."

[1836] 4. Server: The emotion engine recognizes "confusion."

[1837] 5. Server: Generate navigation command. "Guide me to the tea corner."

[1838] 6. Smart glasses: Start navigation and visually and audibly display "The tea corner is here. We will guide you."

[1839] 7. Smart glasses: After the guide is completed, the glasses will notify you, "You have arrived at the tea corner. Can I help you?"

[1840] Prompt Sentence Examples

[1841] The user puts on the smart glasses and asks a question by voice, for example, "Where is the tea cupboard?"

[1842] The smart glasses capture this audio and send it to the server. The server converts the audio into text using Google Cloud Speech-to-Text and identifies the intent using Google Cloud Natural Language. It then analyzes the user's emotions using the Azure Emotion API. Based on the acquired information, the server generates appropriate navigation commands and sends them to the smart glasses. The smart glasses then use the Google Maps API to perform real-time navigation and provide audio and visual guidance to the user.

[1843] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1844] Step 1:

[1845] Input: User's spoken command (e.g., "Where is the tea cupboard?")

[1846] How it works: The microphone on the device (smart glasses) captures the user's voice.

[1847] Output: Captured audio data

[1848] Step 2:

[1849] Input: Captured audio data

[1850] How it works: The device sends the captured audio data to the server using HTTPS.

[1851] Output: Audio data sent to the server

[1852] Step 3:

[1853] Input: Audio data received by the server

[1854] How it works: The server uses a speech recognition engine (Google Cloud Speech-to-Text) to convert the voice data into text.

[1855] Output: Text data (e.g., "Where is the tea shelf?")

[1856] Step 4:

[1857] Input: Converted text data

[1858] How it works: The server uses a natural language processing engine (Google Cloud Natural Language) to analyze the text data and determine the user's intent.

[1859] Output: Identified intent (e.g., "Find a tea cupboard")

[1860] Step 5:

[1861] Input: Audio data received by the server

[1862] How it works: The server uses an emotion recognition engine (Azure Emotion API) to analyze the user's emotional state from the voice data.

[1863] Output: User's emotional state (e.g., "confused")

[1864] Step 6:

[1865] Input: Identified intent and user emotional state

[1866] Action: The server determines the task to be performed based on the recognized intent and emotion, and generates the command.

[1867] Output: Generated command (e.g. "Take me to the tea corner")

[1868] Step 7:

[1869] Input: Generated command

[1870] Operation: The server sends the generated command to the device using HTTPS communication.

[1871] Output: Command sent to the terminal

[1872] Step 8:

[1873] Input: The command received by the terminal

[1874] How it works: The device (smart glasses) starts real-time navigation using the Google Maps API.

[1875] Output: Visual and audio navigation instructions

[1876] Step 9:

[1877] Input: Navigation result

[1878] Behavior: The device notifies the user of the results of the task, for example by displaying a message like "You've arrived at the tea corner. How can I help you?"

[1879] Output: Feedback notification to the user

[1880] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1881] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1882] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1883] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1884] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1885] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1886] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1887] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1888] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1889] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1890] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1891] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1892] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1893] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1894] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1895] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1896] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1897] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1898] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1899] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1900] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1901] The following is further disclosed regarding the above embodiment.

[1902] (Claim 1)

[1903] means for capturing audio input;

[1904] means for transmitting the captured audio data to a server;

[1905] a means for the server to convert the voice data into text;

[1906] A means for analyzing text data using natural language processing to identify the user's intent;

[1907] A means for determining a task to be performed based on the specified intention and generating a command for the task;

[1908] means for transmitting the generated command to a terminal and for the terminal to execute the command;

[1909] A system including a means for notifying a user of the results of task execution.

[1910] (Claim 2)

[1911] 10. The system of claim 1, wherein the means for capturing audio input uses a microphone.

[1912] (Claim 3)

[1913] 10. The system of claim 1, wherein the means for converting the voice data to text uses a voice recognition engine.

[1914] (Claim 4)

[1915] 2. The system of claim 1, wherein the means for analyzing using natural language processing uses a natural language processing engine.

[1916] (Claim 5)

[1917] The system of claim 1 , wherein the means for notifying the user of the result of the execution of the task includes means for providing audio or visual feedback.

[1918] "Example 1"

[1919] (Claim 1)

[1920] means for capturing audio input;

[1921] means for transmitting the captured voice data to a computer via a communication network;

[1922] A means for converting voice data into text data by an electronic computer;

[1923] A means for analyzing text data using natural language processing technology and identifying the user's intent;

[1924] A means for determining a process to be performed based on the specified intention and generating instructions for the process;

[1925] means for transmitting the generated instructions to a terminal and for the terminal to execute the instructions;

[1926] A system that includes a means for notifying the user of the results of processing.

[1927] (Claim 2)

[1928] 10. The system of claim 1, wherein the means for capturing audio input uses an audio collection device.

[1929] (Claim 3)

[1930] 10. The system of claim 1, wherein the means for converting the voice data into text data uses voice recognition technology.

[1931] "Application Example 1"

[1932] (Claim 1)

[1933] means for capturing audio input;

[1934] means for transmitting the captured audio data to a server;

[1935] a means for the server to convert the voice data into text;

[1936] A means for analyzing text data using natural language processing to identify the user's intent;

[1937] A means for determining a task to be performed based on the specified intention and generating a command for the task;

[1938] means for transmitting the generated command to a terminal and for the terminal to execute the command;

[1939] A means for controlling security functions such as camera operation, alarm setting, locking and unlocking using voice input;

[1940] A system including a means for notifying a user of the results of task execution.

[1941] (Claim 2)

[1942] 10. The system of claim 1, wherein the means for capturing audio input uses a microphone.

[1943] (Claim 3)

[1944] 10. The system of claim 1, wherein the means for converting the voice data to text uses a voice recognition engine.

[1945] "Example 2: Combining Emotion Engines"

[1946] (Claim 1)

[1947] means for capturing audio input;

[1948] means for transmitting the captured audio data to a server;

[1949] a means for the server to convert the voice data into text;

[1950] A means for analyzing text data using natural language processing to identify the user's intent;

[1951] A means for analyzing user emotions from voice data;

[1952] A means for determining a task to be performed based on the identified intention and emotion and generating a command for the task;

[1953] means for transmitting the generated command to a terminal and for the terminal to execute the command;

[1954] a means for notifying a user of the execution result of the task;

[1955] A system including:

[1956] (Claim 2)

[1957] 10. The system of claim 1, wherein the means for capturing audio input uses a microphone.

[1958] (Claim 3)

[1959] 10. The system of claim 1, wherein the means for converting the voice data to text uses a voice recognition engine.

[1960] "Application example 2 when combining emotion engines"

[1961] (Claim 1)

[1962] means for capturing audio input;

[1963] means for transmitting the captured audio data to a server;

[1964] a means for the server to convert the voice data into text;

[1965] A means for analyzing text data using natural language processing to identify the user's intent;

[1966] A means for analyzing the user's emotional state and providing flexible task execution and feedback based on the emotional state;

[1967] A means for determining a task to be performed based on the identified intention and emotion and generating a command for the task;

[1968] means for transmitting the generated command to a terminal and for the terminal to execute the command;

[1969] a means for notifying a user of the execution result of the task;

[1970] A system including a navigation means for guiding a user using smart glasses.

[1971] (Claim 2)

[1972] 10. The system of claim 1, wherein the means for capturing audio input uses a microphone.

[1973] (Claim 3)

[1974] 10. The system of claim 1, wherein the means for converting the voice data to text uses a voice recognition engine. [Explanation of symbols]

[1975] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a means for capturing audio input; means for transmitting the captured audio data to a server; a means for the server to convert the voice data into text; A means for analyzing text data using natural language processing to identify the user's intent; A means for determining a task to be performed based on the specified intention and generating a command for the task; means for transmitting the generated command to a terminal and for the terminal to execute the command; A system including a means for notifying a user of the results of task execution.

2. The system of claim 1 , wherein the means for capturing audio input uses a microphone.

3. 10. The system of claim 1, wherein the means for converting voice data to text uses a voice recognition engine.

4. The system of claim 1 , wherein the means for analyzing by natural language processing uses a natural language processing engine.

5. The system of claim 1 , wherein the means for notifying the user of the result of the execution of the task includes means for providing audio or visual feedback.

Images