system

Abstract

We provide the system. [Solution] A means of obtaining user voice commands, A means of analyzing acquired voice commands, A means of acquiring screen information based on analyzed voice commands, Means including a generative artificial intelligence model for summarizing or editing acquired screen information, A means of providing summarized or edited information to the user in audio format, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] When visually impaired people use smartphones or personal computers, it is difficult to efficiently understand the information on the screen because conventional reading readers read all the information. Also, when the characters are scattered or there are various notations, it becomes even more difficult to accurately understand the information on the screen. As a result, visually impaired people not only cannot quickly obtain the necessary information, but also have problems of requiring a great deal of time and labor for information acquisition and operation.

Means for Solving the Problems

[0005] The present invention provides a system that enables visually impaired individuals to efficiently operate a device and obtain information using voice commands. Specifically, the system comprises means for acquiring user voice commands, means for analyzing acquired voice commands, means for acquiring screen information based on the analyzed voice commands, means including a generative artificial intelligence model for summarizing or editing the acquired screen information, and means for providing the summarized or edited information to the user by voice. Furthermore, the system further includes means for transcribing or editing text according to user instructions based on acquired voice commands, and means for creating, checking, and sending emails, thereby enabling visually impaired individuals to use devices more efficiently.

[0006] "User" refers to a visually impaired person who issues voice commands and operates the device.

[0007] A "voice command" is a voice instruction that a user issues to a device to direct it to perform a specific operation or retrieve information.

[0008] "Means of acquisition" refers to methods and devices for collecting user voice commands using the device's microphone or voice recognition system.

[0009] "Means of analysis" refers to software and algorithms that convert acquired voice commands into text data and then understand and process its content.

[0010] "Screen information" refers to text and image information displayed on the device's screen.

[0011] "Summarizing" refers to the process of concisely summarizing screen information and extracting the main points.

[0012] "Editing" refers to operations such as adding, changing, or deleting information on the screen.

[0013] A "generative artificial intelligence model" refers to a machine learning model that uses AI technology to analyze, summarize, and generate text data.

[0014] "Means of providing information via audio" refers to speech synthesis technology and device speakers that convert text data into speech and allow users to hear it.

[0015] "Transcribing" refers to the process of copying and pasting text data to a specific location.

[0016] "Creating an email" refers to the process of generating a new email according to the user's instructions.

[0017] "Confirmation" refers to the process of having the user reconfirm and approve the generated or edited content.

[0018] "Sending" refers to the action of sending a created or edited email to the specified recipient. [Brief explanation of the drawing]

[0019] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8]It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when the emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when the emotion engine is combined.

Mode for Carrying Out the Invention

[0020] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0021] First, the terms used in the following description will be explained.

[0022] In the following embodiments, a processor with a reference number (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of a plurality of arithmetic units. Further, the processor may be a single type of arithmetic unit or a combination of a plurality of types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0023] In the following embodiments, signed RAM (Random Access Memory) is a memory that temporarily stores information and is used as work memory by the processor.

[0024] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0025] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0026] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0027] [First Embodiment]

[0028] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0029] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0030] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0031] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0032] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0033] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0034] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0035] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0036] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0037] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0038] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0039] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0040] This invention provides a system that allows visually impaired individuals to operate a device using voice commands and efficiently acquire information. The program processing of this system is described below.

[0041] First, the user issues a voice command. For example, the user might say, "Please summarize this page." This voice command is picked up by the device's microphone and processed as voice data by the device's internal voice recognition software. The voice data is then sent to a server for analysis. The server uses its voice recognition system to convert the voice data into text data and understands the command "Please summarize."

[0042] Next, the server sends instructions to the terminal based on the analyzed command. In this case, it specifies "summarize screen information." The terminal retrieves information from the currently displayed screen. For example, it takes a screen capture and extracts text information using optical character recognition (OCR) technology. The extracted text data is then sent back to the server for summarization.

[0043] The server analyzes the received text data using a generation AI model, extracts key information, and creates a summary. The generated summary data is sent to the terminal, which then reads the summarized information aloud using speech synthesis. This allows the user to easily grasp the main information on the screen.

[0044] Furthermore, if the user gives specific instructions, such as "Copy this sentence and paste it into an email," they can issue a voice command. The terminal retrieves the voice data again and sends it to the server. The server analyzes the voice data and prepares text editing operations based on the instructions. The server sends the edited content to the terminal, and the terminal copies and edits the text in the appropriate place. The editing results are provided to the user via voice, allowing the user to confirm them.

[0045] When a user composes an email, they might be instructed, for example, "Create a new email. The subject should be 'Work Report,' and I submitted my work report today..." The device receives the voice data and sends it to the server. The server analyzes the voice data and generates the subject "Work Report" and the body "I submitted my work report today...". The generated email content is sent to the device, which reads the email content aloud and asks the user for confirmation. When the user gives a confirmation voice such as "Send," the server executes the email sending operation.

[0046] Thus, a highly functional assistance system that allows for a smooth series of operations, from acquiring voice commands to summarizing information, editing text, and creating and sending emails, enables visually impaired individuals to make better use of their devices, greatly improving their quality of life and work efficiency.

[0047] The following describes the processing flow.

[0048] Step 1:

[0049] The user issues a voice command, such as dictating, "Please summarize this page."

[0050] Step 2:

[0051] The device acquires voice commands via the microphone. The voice is recorded and captured as audio data.

[0052] Step 3:

[0053] The device temporarily stores the audio data and processes it using speech recognition software. The audio data is then converted into text data.

[0054] Step 4:

[0055] The device converts the audio data into text data and then sends it to the server. The data is sent to the server via the network.

[0056] Step 5:

[0057] The server uses a speech recognition system to analyze the voice data and generate text data. It analyzes the voice command and understands the meaning of "Please summarize."

[0058] Step 6:

[0059] Based on the analysis performed by the server, it instructs the terminal to retrieve screen information. The instructions are then packaged and sent to the terminal.

[0060] Step 7:

[0061] The device retrieves information from the currently displayed screen. It takes a screen capture and retrieves the text information from the screen.

[0062] Step 8:

[0063] The device converts screen information into text data using optical character recognition (OCR) technology. It extracts characters from image data.

[0064] Step 9:

[0065] The terminal extracts text data and sends it to the server. The converted text data is packaged and sent to the server.

[0066] Step 10:

[0067] The server analyzes the received text data using a generation AI model. It extracts key information from the text data to summarize its main points.

[0068] Step 11:

[0069] The server generates summary data and sends it to the terminal. The summary results are packaged and sent to the terminal.

[0070] Step 12:

[0071] The device receives summary data, converts it into speech using speech synthesis technology, and provides it to the user. It also converts text data into speech format and plays it through the speaker.

[0072] Specific example 1:

[0073] User: "Could you please summarize this page?"

[0074] Terminal: Receives voice commands and sends them to the server for analysis.

[0075] Server: Analyzes the audio and sends a summary of the screen information back to the terminal.

[0076] Terminal: Captures screen information, converts it to text using OCR, and sends it to the server.

[0077] Server: Summarizes the received text and sends it to the terminal.

[0078] Terminal: Provides a summary to the user in audio format.

[0079] Specific example 2:

[0080] User: "Please create a new email. The subject line should be 'Work Report,' and I submitted my work report today..."

[0081] Terminal: Receives voice commands and sends them to the server for analysis.

[0082] Server: Generates subject and body text and sends it to the terminal.

[0083] Terminal: Reads the email content aloud and asks the user for confirmation.

[0084] User: "Send"

[0085] Terminal: Retrieves confirmation audio and sends it to the server.

[0086] Server: Sending email.

[0087] In this way, a system is realized that allows visually impaired people to operate devices and acquire information more efficiently.

[0088] (Example 1)

[0089] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0090] Visually impaired individuals face challenges in operating devices and efficiently obtaining information. Furthermore, complex operations such as retrieving, summarizing, editing, and composing emails are difficult to perform smoothly using voice commands. To address these issues, a system is needed that analyzes voice commands and responds quickly and accurately.

[0091] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0092] In this invention, the server includes means for acquiring user voice commands, means for analyzing the acquired voice commands, and means for acquiring screen information based on the analyzed voice commands. This enables visually impaired individuals to operate a device using voice commands, efficiently acquire screen information, and compose and send emails.

[0093] "User" refers to the person who operates the system in question.

[0094] "Voice commands" refer to instructions given by the user using their voice.

[0095] A "terminal" refers to an electronic device that has functions such as acquiring voice commands, converting voice data, and acquiring screen information.

[0096] A "server" refers to a central processing unit that performs tasks such as analyzing voice data, understanding commands, using generative AI models, and sending and receiving data.

[0097] "Voice data" refers to the digital data of a user's voice commands.

[0098] "Voice recognition software" refers to a program that converts acquired voice commands into text data.

[0099] "Text data" refers to character information converted by speech recognition software.

[0100] "Screen information" refers to all information displayed on the device, such as text and images.

[0101] "Optical character recognition technology" refers to the technology that recognizes characters from images and extracts them as text data.

[0102] A "generative AI model" refers to an artificial intelligence model that analyzes received data and performs tasks such as summarizing and editing.

[0103] A "summary" is information that extracts the important parts from screen information and presents them concisely.

[0104] "Editing" refers to the process of modifying or rewriting acquired text data based on user instructions.

[0105] "Speech synthesis" refers to a technology that converts text data into speech and provides it to the user.

[0106] This invention is a system for visually impaired individuals to operate a device using voice commands and efficiently acquire information. The following describes specific embodiments of this system.

[0107] Hardware and software usage

[0108] terminal

[0109] The device includes a microphone, speech recognition software, and optical character recognition technology. Speech recognition software such as Google® Speech-to-Text API is used. Optical character recognition technology such as Tesseract OCR is used.

[0110] server

[0111] The server includes functions for analyzing voice data, understanding commands, using generative AI models, and sending and receiving data. DeepSpeech is used for the speech recognition system, and GPT-3 (registered trademark) is used for the generative AI model.

[0112] Operation Description

[0113] Acquisition and recognition of voice commands

[0114] 1. The user issues a voice command. For example, they might say, "Please summarize this page."

[0115] 2. The device's microphone acquires voice commands, which are then converted into voice data by speech recognition software.

[0116] 3. The converted audio data is sent to the server.

[0117] Audio data analysis and command understanding

[0118] 1. The server receives the audio data and converts it into text data using a speech recognition system such as DeepSpeech.

[0119] 2. The server understands the command from the text data and interprets the instruction, "Please summarize this page."

[0120] Acquiring screen information and extracting text

[0121] 1. The server sends an instruction to the terminal called "Summarize screen information".

[0122] 2. Take a screenshot of the screen currently displayed on the device.

[0123] 3. Extract text data from the screen capture using Tesseract OCR or similar software, and send it back to the server.

[0124] Text data summary generation

[0125] 1. The server analyzes the received text data using a generative AI model such as GPT-3, extracts important information, and generates a summary.

[0126] 2. The generated summary data is sent to the terminal.

[0127] Summary audio output

[0128] 1. The device generates a summary of the received information using speech synthesis (such as the Google Text-to-Speech API) and provides it to the user in audio format.

[0129] Specific examples and prompt statements

[0130] Specific example: Summary of page information

[0131] When a user issues a voice command such as "Please summarize this page," the device's microphone captures the voice, which is then converted into digital data by speech recognition software. The converted voice data is sent to a server, where DeepSpeech converts it back into text, and GPT-3 extracts key information to generate a summary. The generated summary data is then sent to the device, and speech synthesis software reads the summary aloud to the user.

[0132] Specific example: Creating and sending an email

[0133] When a user instructs, "Create a new email. The subject line should be 'Work Report,' and I submitted a work report today," the device's microphone captures the voice, which is then converted into digital data by speech recognition software. The converted voice data is sent to a server, which analyzes the voice data and generates a subject line and body. The generated email content is sent to the device, and the user is asked to confirm it as a voice message. Once the user confirms "Send," the server executes the email sending operation.

[0134] Thus, this system allows for a smooth series of operations, from acquiring voice commands to summarizing information, editing text, and creating and sending emails. This system enables visually impaired individuals to efficiently utilize devices and improve their quality of life and work efficiency.

[0135] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0136] Step 1:

[0137] Voice command generation and acquisition:

[0138] The user speaks a voice command into the device's microphone. For example, they might say, "Please summarize this page."

[0139] Input: User's voice command

[0140] Output: Audio data acquired as an audio signal by the device's microphone.

[0141] Specific actions:

[0142] When a user speaks into the microphone, their voice is converted into a digital signal by the microphone.

[0143] Step 2:

[0144] Speech recognition and data conversion:

[0145] The device uses built-in speech recognition software to convert speech data into text data.

[0146] Input: Audio data (audio signal)

[0147] Output: Text data (content of voice command)

[0148] Specific actions:

[0149] The device's Google Speech-to-Text API analyzes the audio signal and converts it into text data that says, "Please summarize this page."

[0150] Step 3:

[0151] Sending audio data:

[0152] The terminal sends the converted text data to the server.

[0153] Input: Text data (content of voice command)

[0154] Output: Text data sent to the server

[0155] Specific actions:

[0156] The terminal's communication module sends text data based on voice commands to the server.

[0157] Step 4:

[0158] Text data analysis and command understanding:

[0159] The server analyzes the received text data to understand the content of the voice command.

[0160] Input: Received text data

[0161] Output: Analysis results (understanding of command content)

[0162] Specific actions:

[0163] The server's speech recognition system, such as DeepSpeech, analyzes the text data and understands the command "Please summarize."

[0164] Step 5:

[0165] Sending instructions based on commands:

[0166] In response to the command, the server sends an instruction to the terminal requesting a "summary of screen information."

[0167] Input: Analysis results (command: "Please provide a summary")

[0168] Output: Signal containing the instruction content (summary of screen information)

[0169] Specific actions:

[0170] The server sends a signal to the terminal instructing it to retrieve and summarize the screen information.

[0171] Step 6:

[0172] Retrieving screen information:

[0173] Capture a screenshot of the screen currently displayed on the device.

[0174] Input: Instructions from the server (summary of screen information)

[0175] Output: Screen capture image

[0176] Specific actions:

[0177] The device uses its screen capture function to obtain the current screen content as an image.

[0178] Step 7:

[0179] Optical character recognition (OCR):

[0180] The device extracts text data from the acquired screen capture image using optical character recognition technology.

[0181] Input: Screen capture image

[0182] Output: Extracted text data

[0183] Specific actions:

[0184] The device's Tesseract OCR analyzes the captured image and extracts the text information displayed on the screen.

[0185] Step 8:

[0186] Sending text data:

[0187] The terminal then sends the extracted text data back to the server.

[0188] Input: Extracted text data

[0189] Output: Text data sent to the server

[0190] Specific actions:

[0191] The extracted text data is sent to the server via the terminal's communication module.

[0192] Step 9:

[0193] Text data summary generation:

[0194] The server analyzes the received text data using a generating AI model (e.g., GPT-3), extracts important information, and generates a summary.

[0195] Input: Received text data

[0196] Output: Generated summary text

[0197] Specific actions:

[0198] A GPT-3 generation AI model on the server analyzes the text data and generates a summary.

[0199] Step 10:

[0200] Send summary data:

[0201] The server sends the generated summary data to the terminal.

[0202] Input: Generated summary text

[0203] Output: Summary text sent to the terminal

[0204] Specific actions:

[0205] The server sends a signal containing summary data to the terminal.

[0206] Step 11:

[0207] Summary audio output:

[0208] The terminal provides the user with the received summary data in voice using speech synthesis technology.

[0209] Input: Received summary text

[0210] Output: Summary provided in audio

[0211] Specific actions:

[0212] The device's Google Text-to-Speech API converts the text into speech and reads it aloud to the user.

[0213] (Application Example 1)

[0214] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0215] Visually impaired individuals face difficulties in conducting electronic transactions and making payments due to the need to understand screen information and edit text, hindering their ability to easily perform daily activities and work. This problem is particularly pronounced in transaction confirmation procedures and payment processes, causing significant stress for visually impaired individuals. Therefore, there is a need for systems that support visually impaired individuals in conducting electronic transactions quickly and accurately using voice commands.

[0216] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0217] In this invention, the server includes means for analyzing the user's voice commands, means for acquiring screen information or electronic transaction information based on the analyzed voice commands, and means including a generative artificial intelligence model for summarizing or editing the acquired information. This makes it possible for visually impaired individuals to confirm the contents of electronic transactions using voice commands and to perform all necessary editing and operations by voice.

[0218] A "voice command" is a voice input that includes instructions or requests that a user makes to a device.

[0219] "Means of analysis" refers to a device or software that converts acquired voice commands into text data and performs the process of understanding and interpreting its content.

[0220] "Screen information" refers to visual information such as text, images, and icons displayed on the device screen.

[0221] "Electronic transaction information" refers to information regarding the purchase, sale, and payment of goods and services conducted in digital format.

[0222] A "generative artificial intelligence model" is an artificial intelligence system that learns from large amounts of data and can automatically perform tasks such as summarizing, editing, and generating text.

[0223] "Means of providing information to the user in audio" refers to a device or software that performs the process of audibly communicating generated or acquired information to the user using speech synthesis technology.

[0224] "Transaction details" refers to detailed information about a specific electronic transaction, such as the goods, services, price, quantity, and trading partner.

[0225] "Transcribing or editing means" refers to any device or software that performs the process of copying, editing, or modifying text or transaction information to another location based on user instructions.

[0226] "Means for creating, reviewing, and sending emails" refers to a device or software that performs the process of creating email content according to the user's voice instructions, having the user review the content, and sending the email after approval.

[0227] "Means for executing a transaction" refers to a device or software that executes a process to complete electronic transaction procedures based on the user's voice instructions.

[0228] To implement this invention, a system will be constructed that allows visually impaired individuals to easily conduct electronic transactions using voice commands. The details of this system are described below.

[0229] First, the user issues a voice command using their smartphone's microphone. This voice command might be something like "Start payment." This voice command is processed as audio data by speech recognition software (e.g., Google Cloud Speech-to-Text) inside the device. The audio data is sent to a server for analysis. The server uses its speech recognition system to convert the audio data into text data and understands the content of the command (in this case, "Start payment").

[0230] Next, the server instructs the terminal to retrieve the current screen information based on the voice command. The terminal uses OCR technology (e.g., Tesseract OCR) to analyze the screen information and extract the data necessary for the transaction (product name, price, etc.). The extracted information is sent back to the server, which uses a generative artificial intelligence model (e.g., OpenAI® GPT-4®) to analyze the information and generate the data necessary for the payment process.

[0231] The generated payment information is provided to the user audibly using speech synthesis software (e.g., Google Text-to-Speech). For example, it might ask for confirmation by voice, such as, "The payment amount for product name 'ABC' is 3000 yen. Is that correct?"

[0232] When the user issues the voice command "confirm," the terminal retrieves the voice data again and sends it to the server. The server analyzes the voice data and begins the payment process. After the transaction is complete, the processing result is sent to the terminal and the user is notified by voice, such as "Payment completed."

[0233] This system enables visually impaired individuals to conduct electronic transactions quickly and accurately using only voice commands, significantly improving their convenience in daily life. A concrete example of this system involves a user giving a voice command such as "Start payment," followed by a series of processes where the server analyzes the data, retrieves information, and synthesizes the speech. The following is an example of a prompt.

[0234] As a concrete example, the following prompt is input to the generating AI model:

[0235] "Start payment"

[0236] The user says "Start payment." The app uses voice recognition to convert the command into text and sends it to the server.

[0237] The server analyzes the text, understands the "start payment" command, and uses OCR to retrieve information from the screen.

[0238] Analyze the acquired information,

[0239] Prompt example: "Product name: "XYZ", Price: 5000 yen, Please confirm payment method."

[0240] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0241] Step 1:

[0242] The user issues a voice command such as "Start payment." This voice command is captured by the device's microphone.

[0243] Input: User's voice command

[0244] Output: Audio data

[0245] Action: The user says "Start payment" to their smartphone.

[0246] Step 2:

[0247] The speech recognition software inside the device (e.g., Google Cloud Speech-to-Text) converts the speech data into text data.

[0248] Input: Audio data

[0249] Output: Text data

[0250] Operation: Speech recognition software analyzes the speech and generates the text data "Start payment".

[0251] Step 3:

[0252] Text data is sent to the server. The server analyzes the received text data using a speech recognition system and understands the command.

[0253] Input: Text data

[0254] Output: Command content

[0255] Operation: The server recognizes the "start payment" command and determines the next step.

[0256] Step 4:

[0257] The server instructs the terminal to acquire screen information and electronic transaction information. The terminal uses OCR technology (e.g., Tesseract OCR) to analyze the screen information and extract the transaction information.

[0258] Input: Instructions from the server

[0259] Output: Transaction information (text format)

[0260] Operation: The device takes a screenshot of the current screen and uses OCR technology to extract product names and prices as text data.

[0261] Step 5:

[0262] The extracted transaction information is sent to the server. The server uses a generative artificial intelligence model (e.g., OpenAI GPT-4) to analyze the information and generate the data necessary for the payment process.

[0263] Input: Transaction information (text format)

[0264] Output: Payment procedure data

[0265] Operation: The server inputs transaction information into an AI model, which then generates summaries and confirmation messages regarding the payment process.

[0266] Step 6:

[0267] The generated payment processing data is provided to the user via voice using speech synthesis software (e.g., Google Text-to-Speech).

[0268] Input: Payment procedure data

[0269] Output: Voice message

[0270] Operation: The device's speaker will voice-over, "The payment amount for product name 'ABC' is 3000 yen. Is that correct?" to request confirmation.

[0271] Step 7:

[0272] The user issues the voice command "Confirm." The terminal retrieves the voice data again and sends it to the server. The server analyzes the voice data and executes the payment process.

[0273] Input: User's voice command

[0274] Output: Instructions to execute payment procedure

[0275] Operation: Speech recognition software generates the text data "confirm," which the server then analyzes to initiate the payment process.

[0276] Step 8:

[0277] Once the payment process is complete, the processing result is sent from the server to the terminal. The terminal then notifies the user of the processing result via voice.

[0278] Input: Payment procedure result data

[0279] Output: Audio notification of processing results

[0280] Action: The device's speaker will notify the user via voice message, "Payment complete."

[0281] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0282] This invention combines a system for visually impaired individuals to operate a device using voice commands and efficiently acquire information with an emotion engine for recognizing the user's emotions. The program processing of this system is described below.

[0283] First, the user issues a voice command. For example, an instruction such as "Please summarize this page" is given. This voice command is acquired by the microphone of the terminal and processed as voice data by the voice recognition software inside the terminal. The terminal temporarily stores this voice data and transmits it to the server for analysis.

[0284] The server uses a voice recognition system to convert the voice data into text data, and further uses an emotion engine to recognize the user's emotion from the voice data. At this time, the server analyzes not only the voice command but also features such as the tone, speed, and intonation of the user's voice, and estimates the user's emotional state. For example, when the user is feeling anxious, the emotion engine recognizes it.

[0285] Next, based on the analyzed content, the server instructs the terminal to acquire screen information. The terminal acquires the information of the currently displayed screen and converts it into text data using optical character recognition (OCR) technology. This text data is also transmitted to the server again.

[0286] The server analyzes the received text data using a generated artificial intelligence model, extracts important information, and creates a summary. Based on the user's emotional state recognized by the emotion engine, the information providing method is adjusted. For example, when the user is feeling anxious, the summary is provided in a concise and quickly understandable form.

[0287] The generated summary data is transmitted to the terminal, and the terminal converts the summarized information into voice using voice synthesis technology and provides it to the user. Through this process, the user can quickly and accurately grasp the necessary information.

[0288] When a user gives instructions for a specific action, they might issue a voice command such as, "Copy this sentence and paste it into an email." The terminal receives the voice data and sends it to the server. The server analyzes the voice data and, taking into account the user's emotional state, prepares the text editing operation in the most optimal way. The server sends the edited content to the terminal, which then copies and edits the text in the appropriate location. The editing result is provided to the user via voice for confirmation.

[0289] When composing an email, the user might instruct, "Create a new email. The subject should be 'Work Report,' and today I submitted a work report..." In this case, the device also acquires voice data and sends it to the server. The server analyzes the voice data and generates the subject and body of the email. The generated email content is sent to the device, which reads the content aloud and asks the user for confirmation. When the user gives a confirmation voice such as "Send," the server executes the email sending operation.

[0290] In this way, by analyzing both voice commands and user emotions, we provide a highly functional support system that enables visually impaired individuals to efficiently operate devices and acquire information. For example, when a user requests a summary, if the emotion engine recognizes the user's impatience, the system adjusts to provide a quick and concise summary. Also, when composing an email, the system can prevent accidental sending by repeatedly confirming the content according to the user's emotions. This significantly improves the quality of life and work efficiency of visually impaired individuals.

[0291] The following describes the processing flow.

[0292] Step 1:

[0293] The user issues a voice command, such as dictating, "Please summarize this page."

[0294] Step 2:

[0295] The device acquires voice commands via the microphone. The voice is recorded and captured as audio data.

[0296] Step 3:

[0297] The terminal temporarily stores the voice data and processes it with voice recognition software. The voice data is converted into text data.

[0298] Step 4:

[0299] The terminal transmits the converted text data to the server. The data is sent to the server via the network.

[0300] Step 5:

[0301] The server analyzes the text data using a voice recognition system. It understands the voice command "Please summarize".

[0302] Step 6:

[0303] The emotion engine incorporated in the server recognizes the user's emotion from the voice data. It analyzes the tone, speed, intonation, etc. of the voice and estimates the user's emotional state (e.g., anxiety, joy, tension, etc.).

[0304] Step 7:

[0305] Based on the analyzed content and the result of emotion recognition by the server, the server instructs the terminal to acquire screen information. The instruction is packaged and sent to the terminal.

[0306] Step 8:

[0307] The terminal acquires the information of the currently displayed screen. It takes a screen capture to acquire the character information of the screen.

[0308] Step 9:

[0309] The terminal converts the acquired screen information into text data using optical character recognition (OCR) technology. Characters are extracted from the image data. [[ID=

[0310] Step 10:

[0311] The terminal sends the converted text data to the server.

[0312] Step 11:

[0313] The server receives text data, which is then analyzed by an artificial intelligence model. Important information from the text data is extracted, and a summary is created.

[0314] Step 12:

[0315] The emotion engine adjusts how summaries are delivered based on the user's emotional state. For example, if the user is feeling anxious, it will provide a concise and easily understandable summary.

[0316] Step 13:

[0317] The server generates summary data and sends it to the terminal. The summary results are packaged and sent to the terminal.

[0318] Step 14:

[0319] The device converts the received summary data into speech using speech synthesis technology. It also converts text data into audio format and plays it through the speaker.

[0320] Step 15:

[0321] The user gives voice commands for specific actions. For example, they might say, "Copy this text and paste it into an email."

[0322] Step 16:

[0323] The terminal acquires audio data and sends it to the server. A package for converting the audio recording to text is generated.

[0324] Step 17:

[0325] The server analyzes the audio data and understands the instructions given. It converts the audio to text and then analyzes the content.

[0326] Step 18:

[0327] The emotion engine optimizes text editing operations based on the user's emotional state. For example, if the user is emotionally agitated, the operation steps will be simplified.

[0328] Step 19:

[0329] The server sends the edited content to the terminal. The edited content is packaged and transferred.

[0330] Step 20:

[0331] The device provides the user with the edited results via voice. It reads aloud, "The text has been pasted into the email. Do you want to confirm?"

[0332] Step 21:

[0333] The user requests a voice message to create an email. They instruct, "Create a new email. The subject line should be 'Work Report,' and I submitted a work report today..."

[0334] Step 22:

[0335] The terminal acquires audio data and sends it to the server. A package for analysis is generated from the audio recording.

[0336] Step 23:

[0337] The server analyzes the audio data and generates a subject and body. It generates text such as "Subject: Work Report" and "Body: Today I gave a work report..."

[0338] Step 24:

[0339] The generated email content is sent to the terminal. The subject and body data are packaged and forwarded.

[0340] Step 25:

[0341] The device reads the email content aloud and asks the user for confirmation. It instructs the user, "Do you want to send it? If yes, please say 'Send'."

[0342] Step 26:

[0343] The user gives a confirmation voice command, saying "Send."

[0344] Step 27:

[0345] The device acquires the confirmation audio and sends it to the server. The audio data for confirmation is sent to the server.

[0346] Step 28:

[0347] The server performs a final check and then sends the email. The email is sent to the specified recipient.

[0348] (Example 2)

[0349] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0350] Traditional systems primarily relied on voice commands for operation and information provision, enabling visually impaired individuals to efficiently operate devices and quickly and appropriately obtain information. However, these systems were insufficient in responding to the user's emotional state. As a result, users may experience frustration or be unable to obtain appropriate information. Therefore, there is a need for a system that can adjust device operation and information provision methods according to the user's emotional state, making it more efficient and reassuring to use.

[0351] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0352] In this invention, the server includes means for recognizing the user's emotions from voice data, means for acquiring screen information based on the emotion recognition results, and means including a generative artificial intelligence model for summarizing or editing the acquired screen information. This makes it possible to adjust the appropriate information provision method according to the user's emotional state, thereby improving convenience and peace of mind for visually impaired users when using the device.

[0353] A "voice command" refers to a voice command that a user makes to operate or instruct a device.

[0354] "Voice data" refers to data that represents acquired voice commands in digital format.

[0355] "Speech recognition" refers to the technology that analyzes speech data and converts it into text data.

[0356] "Emotion recognition" refers to a technology that analyzes and estimates a user's emotional state based on voice data.

[0357] "Screen information" refers to the visual information displayed on a device's screen.

[0358] Optical Character Recognition (OCR) refers to a technology that analyzes characters in an image and converts them into text data.

[0359] A "generative artificial intelligence model" refers to AI technology that analyzes input data and generates output tailored to a specific task.

[0360] A "summary" refers to a short, concise compilation of key information extracted from acquired text data.

[0361] "Editing" refers to the operation of changing or modifying the content of text data based on user instructions.

[0362] "Speech synthesis" refers to the technology that analyzes text data and converts it into human speech.

[0363] "Email" refers to a method of communication for creating and sending messages electronically.

[0364] Modes for carrying out the invention

[0365] This invention is a system for visually impaired individuals to operate a device using voice commands and efficiently acquire information, and it incorporates an emotion engine to recognize the user's emotions. Specific embodiments are described below.

[0366] Hardware and software to use

[0367] This system uses the following hardware and software:

[0368] Speech recognition software: For example, Google's speech recognition API

[0369] Emotion recognition software: For example, IBM Watson® Tone Analyzer

[0370] Optical Character Recognition (OCR) technology: e.g., Tesseract OCR

[0371] Generative artificial intelligence models: e.g., OpenAI's GPT-3

[0372] Speech synthesis technology: For example, IBM Watson Text to Speech

[0373] System configuration and operation

[0374] 1. Obtaining voice commands

[0375] The user issues a voice command to the device. For example, they might give a command such as, "Please summarize this page."

[0376] The device's microphone picks up this voice command, and it is processed as voice data by speech recognition software.

[0377] 2. Sending audio data

[0378] The device temporarily stores the voice data and sends it to the server via a secure protocol (e.g., HTTPS).

[0379] 3. Analysis of audio data

[0380] The server receives the audio data and converts it into text data using a speech recognition system.

[0381] Furthermore, the server recognizes the user's emotions from the voice data. It uses an emotion engine to analyze tone, speed, intonation, and other factors.

[0382] 4. Obtaining screen information

[0383] Based on the analyzed voice commands and emotion recognition results, the server instructs the terminal to retrieve screen information.

[0384] The device captures the currently displayed screen and converts it into text data using OCR technology.

[0385] 5. Generating a summary

[0386] The server receives text data sent from the terminal and analyzes it using a generative AI model. It then extracts important information and creates a summary.

[0387] The way information is delivered is adjusted based on the user's emotional state recognized by the emotion engine.

[0388] 6. Transmission of summary data and speech synthesis

[0389] The generated summary data is sent from the server to the terminal.

[0390] The device converts the summarized information into speech using speech synthesis technology and provides it to the user.

[0391] Specific example

[0392] For example, if a user issues a voice command such as "Please summarize this page," the device's microphone captures the voice, and a speech recognition API processes it into audio data. The server receives this audio data and uses an emotion engine to analyze the user's emotions. If the user is feeling anxious, the server generates a quick and concise summary, converts it into speech using speech synthesis technology, and provides it to the user through the device.

[0393] Example of a prompt

[0394] "Please describe a scenario in which a visually impaired person issues a voice command saying, 'Please summarize this page,' and the system, recognizing the user's impatience, quickly and concisely provides a summary."

[0395] In this way, this system provides flexible and rapid information tailored to the user's emotional state, creating an environment where visually impaired individuals can effectively use devices.

[0396] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0397] Step 1:

[0398] The user issues a voice command, specifically a request such as, "Please summarize this page." The device's microphone picks up the voice command and processes it as audio data using speech recognition software (e.g., Google's Speech Recognition API). The input is the user's voice command, and the output is audio data.

[0399] Step 2:

[0400] The terminal temporarily stores the audio data and sends it to the server via a secure protocol (e.g., HTTPS). Specifically, the binary data of the audio data is transmitted using a secure connection. The input is the audio data, and the output is the audio data sent to the server.

[0401] Step 3:

[0402] The server receives audio data and converts it into text data using a speech recognition system (e.g., Google Cloud Speech-to-Text). This process involves dividing the audio signal and mapping it to language units. The input is audio data, and the output is text data.

[0403] Step 4:

[0404] The server uses an emotion engine (e.g., IBM Watson Tone Analyzer) to recognize the user's emotions from voice data. It analyzes features such as tone, speed, and intonation of the voice to estimate emotional states such as anxiety or relaxation. The input is voice data and text data, and the output is emotional state data.

[0405] Step 5:

[0406] Based on the voice commands and emotion recognition results analyzed by the server, it instructs the terminal to retrieve screen information. Specifically, it sends instructions to the terminal via an HTTP request. The input is the voice command text and emotion state data, and the output is an instruction to retrieve screen information.

[0407] Step 6:

[0408] The device captures the information displayed on the screen and converts it into text data using optical character recognition (OCR) technology (e.g., Tesseract OCR). Alternatively, it can take a screenshot of the screen and extract text information from the image. The input is a screen image, and the output is text data.

[0409] Step 7:

[0410] The terminal sends the converted text data back to the server. Specifically, the text data is sent using a secure connection. The input is text data, and the output is the text data sent to the server.

[0411] Step 8:

[0412] The server analyzes the received text using a generative AI model (e.g., OpenAI's GPT-3), extracts important information, and creates a summary. The information delivery method is adjusted based on the user's emotional state recognized by the emotion engine. The input is text data and emotional state data, and the output is summary data.

[0413] Step 9:

[0414] The generated summary data is sent from the server to the terminal. A secure protocol is used to transmit the data. The input is the summary data, and the output is the summary data sent to the terminal.

[0415] Step 10:

[0416] The terminal converts summarized information into speech using speech synthesis technology (e.g., IBM Watson Text to Speech) and provides it to the user. The generated audio data is played back through the terminal's speaker. The input is summarized data, and the output is audio data.

[0417] Step 11:

[0418] The user gives a specific command (e.g., "Copy this text and paste it into an email"). The terminal acquires the voice data and sends it to the server. The input is the user's voice command, and the output is the voice data sent to the server.

[0419] Step 12:

[0420] The server analyzes the audio data and prepares the optimal text editing method, taking into account the user's emotional state. The prepared editing content is then sent to the terminal. Input consists of audio data and emotional state data, while output is editing instruction data.

[0421] Step 13:

[0422] The terminal transcribes or edits text to the appropriate location based on editing instruction data received from the server. The editing result is provided to the user via audio for user confirmation. The input is the editing instruction data, and the output is the edited text data and its audio data.

[0423] Step 14:

[0424] When a user instructs, "Create a new email. The subject line should be 'Work Report,' and I submitted my work report today...", the terminal acquires voice data and sends it to the server. The input is the user's voice command, and the output is the voice data sent to the server.

[0425] Step 15:

[0426] The server analyzes the audio data and generates a subject and body for the email. The generated email content is then sent to the terminal. The input consists of audio data and emotional state data, while the output consists of the email subject and body data.

[0427] Step 16:

[0428] The terminal reads the content aloud and prompts the user for confirmation. Once the user confirms "Send," the server executes the email sending operation. The input consists of the email subject, body data, and the user's send instruction, and the output is the sent email.

[0429] (Application Example 2)

[0430] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0431] For visually impaired individuals to find products appropriately and shop efficiently in physical stores, a voice-command navigation system is necessary. However, conventional voice command systems do not take user emotions into consideration, and at the same time, they do not adequately provide specific directions within stores or location information for products. As a result, shopping becomes difficult, especially for users who experience anxiety or stress.

[0432] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring a user's voice command, means for analyzing the acquired voice command, means for acquiring screen information based on the analyzed voice command, means including a generative artificial intelligence model for summarizing or editing the acquired screen information, means for providing the summarized or edited information to the user by voice, means including an emotion engine for analyzing emotions using the user's voice command and voice characteristics, and means for adjusting the information provision method based on the user's emotions. This makes it possible to provide navigation and product information quickly and concisely in a physical store while taking into account the user's emotional state.

[0433] "Means for acquiring user voice commands" refers to a microphone or voice input device that recognizes voice commands uttered by the user and inputs them into the device.

[0434] "Means for analyzing voice commands" refers to a voice recognition system that converts acquired voice commands into digital data, analyzes its content, and converts it into appropriate instructions.

[0435] "Means of acquiring screen information" refers to cameras and screen capture functions used to capture and analyze information displayed on a device.

[0436] A "generative artificial intelligence model" refers to an AI algorithm or system that extracts important information from large amounts of data and performs summarization or editing.

[0437] "Means of providing information to users in audio format" refers to speech synthesis technology and speakers used to convey analysis results and summaries to users as audio.

[0438] An "emotion engine" refers to software or a system that analyzes a user's voice characteristics and estimates their emotional state.

[0439] "Means for adjusting the method of information provision" refers to a function that dynamically changes the content and format of the information provided based on the user's emotional state.

[0440] "Means for transcribing or editing text" refers to software functions that copy digital text to the appropriate location and edit it based on the user's voice commands.

[0441] "Means for creating, reviewing, and sending emails" refers to systems and software functions that automatically generate emails based on user voice instructions, review their content, and then send them.

[0442] This invention is a system designed to assist visually impaired individuals in shopping at physical stores. The system includes a set of functions that allow the user, wearing smart glasses and using voice commands, to obtain information on product locations and store navigation. Furthermore, it reads the user's emotions from their voice and adjusts the assistance accordingly.

[0443] First, the user puts on smart glasses and makes a voice command such as "Tell me where the product is." The device's microphone captures this voice, and the Google Cloud Speech-to-Text API converts the voice into text data.

[0444] Next, along with the text data, characteristics such as tone, speed, and intonation of the voice are sent to the Azure® Emotion API for emotion analysis. As a result of the analysis, a specific emotional state is recognized, such as when the user is anxious.

[0445] Screen information is acquired, for example, using the camera of smart glasses. The camera takes pictures of the store's map and product shelves, and converts them into text using Tesseract OCR technology. Based on this text data, the system works with the Google Maps API to determine the location within the physical store.

[0446] A specific prompt would be sent to the Generative AI model (GPT-4) with a request such as, "The Emotion API has detected that the user is feeling anxious. The user is currently looking for the location of a sticky note in the store. Please generate concise and easy-to-understand navigation instructions."

[0447] A generative AI model generates optimal feedback (for example, "Turn right and go straight down the next corridor. The sticky note is on the shelf to your right."), and this feedback is converted into speech by the Google Text-to-Speech API. The information is then delivered to the user through the smart glasses' speaker.

[0448] This system allows users to receive voice-based navigation and product location information tailored to their emotional state, enabling them to shop efficiently and comfortably in physical stores.

[0449] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0450] Step 1:

[0451] The user wears smart glasses and issues a voice command. Voice input is acquired through the smart glasses' microphone. The input voice data is the output of this step.

[0452] Step 2:

[0453] The device sends the acquired audio data to the Google Cloud Speech-to-Text API, which converts the audio into text data. The input is audio data, and the output is the converted text data.

[0454] Step 3:

[0455] The device sends text data along with characteristic data such as tone, speed, and intonation of speech to the Azure Emotion API. The input consists of text data and speech characteristic data, and the output is the result of the user's emotion analysis.

[0456] Step 4:

[0457] The server processes data based on the sentiment analysis results to determine a navigation method that corresponds to the user's emotions. The input here is the sentiment analysis results, and the output is navigation data corresponding to those emotions.

[0458] Step 5:

[0459] The device uses the camera on smart glasses to take pictures of the store interior and product shelves, and converts them into text data using Tesseract OCR. The input is the image taken by the camera, and the output is the converted text data.

[0460] Step 6:

[0461] The server uses OCR data in conjunction with the Google Maps API to identify location information within the store. The input is text data converted by OCR, and the output is the identified location information.

[0462] Step 7:

[0463] The device uses a generative AI model (GPT-4) to generate navigation instructions for the user based on sentiment analysis results and location information. The input is sentiment analysis results and location information, and the output is navigation instruction text.

[0464] Step 8:

[0465] The device sends the generated navigation instruction text to the Google Text-to-Speech API, where it is converted into speech. The input is navigation instruction text, and the output is audio data.

[0466] Step 9:

[0467] The device provides voice data to the user through the smart glasses' speaker. This allows the user to receive accurate directions and product information within the store via voice. The input is voice data, and the output is voice guidance.

[0468] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0469] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0470] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0471] [Second Embodiment]

[0472] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0473] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0474] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0475] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0476] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0477] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0478] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0479] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0480] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0481] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0482] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0483] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0484] This invention provides a system that allows visually impaired individuals to operate a device using voice commands and efficiently acquire information. The program processing of this system is described below.

[0485] First, the user issues a voice command. For example, the user might say, "Please summarize this page." This voice command is picked up by the device's microphone and processed as voice data by the device's internal voice recognition software. The voice data is then sent to a server for analysis. The server uses its voice recognition system to convert the voice data into text data and understands the command "Please summarize."

[0486] Next, the server sends instructions to the terminal based on the analyzed command. In this case, it specifies "summarize screen information." The terminal retrieves information from the currently displayed screen. For example, it takes a screen capture and extracts text information using optical character recognition (OCR) technology. The extracted text data is then sent back to the server for summarization.

[0487] The server analyzes the received text data using a generation AI model, extracts key information, and creates a summary. The generated summary data is sent to the terminal, which then reads the summarized information aloud using speech synthesis. This allows the user to easily grasp the main information on the screen.

[0488] Furthermore, if the user gives specific instructions, such as "Copy this sentence and paste it into an email," they can issue a voice command. The terminal retrieves the voice data again and sends it to the server. The server analyzes the voice data and prepares text editing operations based on the instructions. The server sends the edited content to the terminal, and the terminal copies and edits the text in the appropriate place. The editing results are provided to the user via voice, allowing the user to confirm them.

[0489] When a user composes an email, they might be instructed, for example, "Create a new email. The subject should be 'Work Report,' and I submitted my work report today..." The device receives the voice data and sends it to the server. The server analyzes the voice data and generates the subject "Work Report" and the body "I submitted my work report today...". The generated email content is sent to the device, which reads the email content aloud and asks the user for confirmation. When the user gives a confirmation voice such as "Send," the server executes the email sending operation.

[0490] Thus, a highly functional assistance system that allows for a smooth series of operations, from acquiring voice commands to summarizing information, editing text, and creating and sending emails, enables visually impaired individuals to make better use of their devices, greatly improving their quality of life and work efficiency.

[0491] The following describes the processing flow.

[0492] Step 1:

[0493] The user issues a voice command, such as dictating, "Please summarize this page."

[0494] Step 2:

[0495] The device acquires voice commands via the microphone. The voice is recorded and captured as audio data.

[0496] Step 3:

[0497] The device temporarily stores the audio data and processes it using speech recognition software. The audio data is then converted into text data.

[0498] Step 4:

[0499] The device converts the audio data into text data and then sends it to the server. The data is sent to the server via the network.

[0500] Step 5:

[0501] The server uses a speech recognition system to analyze the voice data and generate text data. It analyzes the voice command and understands the meaning of "Please summarize."

[0502] Step 6:

[0503] Based on the analysis performed by the server, it instructs the terminal to retrieve screen information. The instructions are then packaged and sent to the terminal.

[0504] Step 7:

[0505] The device retrieves information from the currently displayed screen. It takes a screen capture and retrieves the text information from the screen.

[0506] Step 8:

[0507] The device converts screen information into text data using optical character recognition (OCR) technology. It extracts characters from image data.

[0508] Step 9:

[0509] The terminal extracts text data and sends it to the server. The converted text data is packaged and sent to the server.

[0510] Step 10:

[0511] The server analyzes the received text data using a generation AI model. It extracts key information from the text data to summarize its main points.

[0512] Step 11:

[0513] The server generates summary data and sends it to the terminal. The summary results are packaged and sent to the terminal.

[0514] Step 12:

[0515] The device receives summary data, converts it into speech using speech synthesis technology, and provides it to the user. It also converts text data into speech format and plays it through the speaker.

[0516] Specific example 1:

[0517] User: "Could you please summarize this page?"

[0518] Terminal: Receives voice commands and sends them to the server for analysis.

[0519] Server: Analyzes the audio and sends a summary of the screen information back to the terminal.

[0520] Terminal: Captures screen information, converts it to text using OCR, and sends it to the server.

[0521] Server: Summarizes the received text and sends it to the terminal.

[0522] Terminal: Provides a summary to the user in audio format.

[0523] Specific example 2:

[0524] User: "Please create a new email. The subject line should be 'Work Report,' and I submitted my work report today..."

[0525] Terminal: Receives voice commands and sends them to the server for analysis.

[0526] Server: Generates subject and body text and sends it to the terminal.

[0527] Terminal: Reads the email content aloud and asks the user for confirmation.

[0528] User: "Send"

[0529] Terminal: Retrieves confirmation audio and sends it to the server.

[0530] Server: Sending email.

[0531] In this way, a system is realized that allows visually impaired people to operate devices and acquire information more efficiently.

[0532] (Example 1)

[0533] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0534] Visually impaired individuals face challenges in operating devices and efficiently obtaining information. Furthermore, complex operations such as retrieving, summarizing, editing, and composing emails are difficult to perform smoothly using voice commands. To address these issues, a system is needed that analyzes voice commands and responds quickly and accurately.

[0535] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0536] In this invention, the server includes means for acquiring user voice commands, means for analyzing the acquired voice commands, and means for acquiring screen information based on the analyzed voice commands. This enables visually impaired individuals to operate a device using voice commands, efficiently acquire screen information, and compose and send emails.

[0537] "User" refers to the person who operates the system in question.

[0538] "Voice commands" refer to instructions given by the user using their voice.

[0539] A "terminal" refers to an electronic device that has functions such as acquiring voice commands, converting voice data, and acquiring screen information.

[0540] A "server" refers to a central processing unit that performs tasks such as analyzing voice data, understanding commands, using generative AI models, and sending and receiving data.

[0541] "Voice data" refers to the digital data of a user's voice commands.

[0542] "Voice recognition software" refers to a program that converts acquired voice commands into text data.

[0543] "Text data" refers to character information converted by speech recognition software.

[0544] "Screen information" refers to all information displayed on the device, such as text and images.

[0545] "Optical character recognition technology" refers to the technology that recognizes characters from images and extracts them as text data.

[0546] A "generative AI model" refers to an artificial intelligence model that analyzes received data and performs tasks such as summarizing and editing.

[0547] A "summary" is information that extracts the important parts from screen information and presents them concisely.

[0548] "Editing" refers to the process of modifying or rewriting acquired text data based on user instructions.

[0549] "Speech synthesis" refers to a technology that converts text data into speech and provides it to the user.

[0550] This invention is a system for visually impaired individuals to operate a device using voice commands and efficiently acquire information. The following describes specific embodiments of this system.

[0551] Hardware and software usage

[0552] terminal

[0553] The device includes a microphone, speech recognition software, and optical character recognition (OCR) technology. Speech recognition software such as Google Speech-to-Text API is used. Optical character recognition technology such as Tesseract OCR is used.

[0554] server

[0555] The server includes functions for analyzing voice data, understanding commands, using generative AI models, and sending and receiving data. DeepSpeech is used for the speech recognition system, and GPT-3 is used for the generative AI model.

[0556] Operation Description

[0557] Acquisition and recognition of voice commands

[0558] 1. The user issues a voice command. For example, they might say, "Please summarize this page."

[0559] 2. The device's microphone acquires voice commands, which are then converted into voice data by speech recognition software.

[0560] 3. The converted audio data is sent to the server.

[0561] Audio data analysis and command understanding

[0562] 1. The server receives the audio data and converts it into text data using a speech recognition system such as DeepSpeech.

[0563] 2. The server understands the command from the text data and interprets the instruction, "Please summarize this page."

[0564] Acquiring screen information and extracting text

[0565] 1. The server sends an instruction to the terminal called "Summarize screen information".

[0566] 2. Take a screenshot of the screen currently displayed on the device.

[0567] 3. Extract text data from the screen capture using Tesseract OCR or similar software, and send it back to the server.

[0568] Text data summary generation

[0569] 1. The server analyzes the received text data using a generative AI model such as GPT-3, extracts important information, and generates a summary.

[0570] 2. The generated summary data is sent to the terminal.

[0571] Summary audio output

[0572] 1. The device generates a summary of the received information using speech synthesis (such as the Google Text-to-Speech API) and provides it to the user in audio format.

[0573] Specific examples and prompt statements

[0574] Specific example: Summary of page information

[0575] When a user issues a voice command such as "Please summarize this page," the device's microphone captures the voice, which is then converted into digital data by speech recognition software. The converted voice data is sent to a server, where DeepSpeech converts it back into text, and GPT-3 extracts key information to generate a summary. The generated summary data is then sent to the device, and speech synthesis software reads the summary aloud to the user.

[0576] Specific example: Creating and sending an email

[0577] When a user instructs, "Create a new email. The subject line should be 'Work Report,' and I submitted a work report today," the device's microphone captures the voice, which is then converted into digital data by speech recognition software. The converted voice data is sent to a server, which analyzes the voice data and generates a subject line and body. The generated email content is sent to the device, and the user is asked to confirm it as a voice message. Once the user confirms "Send," the server executes the email sending operation.

[0578] Thus, this system allows for a smooth series of operations, from acquiring voice commands to summarizing information, editing text, and creating and sending emails. This system enables visually impaired individuals to efficiently utilize devices and improve their quality of life and work efficiency.

[0579] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0580] Step 1:

[0581] Voice command generation and acquisition:

[0582] The user speaks a voice command into the device's microphone. For example, they might say, "Please summarize this page."

[0583] Input: User's voice command

[0584] Output: Audio data acquired as an audio signal by the device's microphone.

[0585] Specific actions:

[0586] When a user speaks into the microphone, their voice is converted into a digital signal by the microphone.

[0587] Step 2:

[0588] Speech recognition and data conversion:

[0589] The device uses built-in speech recognition software to convert speech data into text data.

[0590] Input: Audio data (audio signal)

[0591] Output: Text data (content of voice command)

[0592] Specific actions:

[0593] The device's Google Speech-to-Text API analyzes the audio signal and converts it into text data that says, "Please summarize this page."

[0594] Step 3:

[0595] Sending audio data:

[0596] The terminal sends the converted text data to the server.

[0597] Input: Text data (content of voice command)

[0598] Output: Text data sent to the server

[0599] Specific actions:

[0600] The terminal's communication module sends text data based on voice commands to the server.

[0601] Step 4:

[0602] Text data analysis and command understanding:

[0603] The server analyzes the received text data to understand the content of the voice command.

[0604] Input: Received text data

[0605] Output: Analysis results (understanding of command content)

[0606] Specific actions:

[0607] The server's speech recognition system, such as DeepSpeech, analyzes the text data and understands the command "Please summarize."

[0608] Step 5:

[0609] Sending instructions based on commands:

[0610] In response to the command, the server sends an instruction to the terminal requesting a "summary of screen information."

[0611] Input: Analysis results (command: "Please provide a summary")

[0612] Output: Signal containing the instruction content (summary of screen information)

[0613] Specific actions:

[0614] The server sends a signal to the terminal instructing it to retrieve and summarize the screen information.

[0615] Step 6:

[0616] Retrieving screen information:

[0617] Capture a screenshot of the screen currently displayed on the device.

[0618] Input: Instructions from the server (summary of screen information)

[0619] Output: Screen capture image

[0620] Specific actions:

[0621] The device uses its screen capture function to obtain the current screen content as an image.

[0622] Step 7:

[0623] Optical character recognition (OCR):

[0624] The device extracts text data from the acquired screen capture image using optical character recognition technology.

[0625] Input: Screen capture image

[0626] Output: Extracted text data

[0627] Specific actions:

[0628] The device's Tesseract OCR analyzes the captured image and extracts the text information displayed on the screen.

[0629] Step 8:

[0630] Sending text data:

[0631] The terminal then sends the extracted text data back to the server.

[0632] Input: Extracted text data

[0633] Output: Text data sent to the server

[0634] Specific actions:

[0635] The extracted text data is sent to the server via the terminal's communication module.

[0636] Step 9:

[0637] Text data summary generation:

[0638] The server analyzes the received text data using a generating AI model (e.g., GPT-3), extracts important information, and generates a summary.

[0639] Input: Received text data

[0640] Output: Generated summary text

[0641] Specific actions:

[0642] A GPT-3 generation AI model on the server analyzes the text data and generates a summary.

[0643] Step 10:

[0644] Send summary data:

[0645] The server sends the generated summary data to the terminal.

[0646] Input: Generated summary text

[0647] Output: Summary text sent to the terminal

[0648] Specific actions:

[0649] The server sends a signal containing summary data to the terminal.

[0650] Step 11:

[0651] Summary audio output:

[0652] The terminal provides the user with the received summary data in voice using speech synthesis technology.

[0653] Input: Received summary text

[0654] Output: Summary provided in audio

[0655] Specific actions:

[0656] The device's Google Text-to-Speech API converts the text into speech and reads it aloud to the user.

[0657] (Application Example 1)

[0658] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0659] Visually impaired individuals face difficulties in conducting electronic transactions and making payments due to the need to understand screen information and edit text, hindering their ability to easily perform daily activities and work. This problem is particularly pronounced in transaction confirmation procedures and payment processes, causing significant stress for visually impaired individuals. Therefore, there is a need for systems that support visually impaired individuals in conducting electronic transactions quickly and accurately using voice commands.

[0660] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0661] In this invention, the server includes means for analyzing the user's voice commands, means for acquiring screen information or electronic transaction information based on the analyzed voice commands, and means including a generative artificial intelligence model for summarizing or editing the acquired information. This makes it possible for visually impaired individuals to confirm the contents of electronic transactions using voice commands and to perform all necessary editing and operations by voice.

[0662] A "voice command" is a voice input that includes instructions or requests that a user makes to a device.

[0663] "Means of analysis" refers to a device or software that converts acquired voice commands into text data and performs the process of understanding and interpreting its content.

[0664] "Screen information" refers to visual information such as text, images, and icons displayed on the device screen.

[0665] "Electronic transaction information" refers to information regarding the purchase, sale, and payment of goods and services conducted in digital format.

[0666] A "generative artificial intelligence model" is an artificial intelligence system that learns from large amounts of data and can automatically perform tasks such as summarizing, editing, and generating text.

[0667] "Means of providing information to the user in audio" refers to a device or software that performs the process of audibly communicating generated or acquired information to the user using speech synthesis technology.

[0668] "Transaction details" refers to detailed information about a specific electronic transaction, such as the goods, services, price, quantity, and trading partner.

[0669] "Transcribing or editing means" refers to any device or software that performs the process of copying, editing, or modifying text or transaction information to another location based on user instructions.

[0670] "Means for creating, reviewing, and sending emails" refers to a device or software that performs the process of creating email content according to the user's voice instructions, having the user review the content, and sending the email after approval.

[0671] "Means for executing a transaction" refers to a device or software that executes a process to complete electronic transaction procedures based on the user's voice instructions.

[0672] To implement this invention, a system will be constructed that allows visually impaired individuals to easily conduct electronic transactions using voice commands. The details of this system are described below.

[0673] First, the user issues a voice command using their smartphone's microphone. This voice command might be something like "Start payment." This voice command is processed as audio data by speech recognition software (e.g., Google Cloud Speech-to-Text) inside the device. The audio data is sent to a server for analysis. The server uses its speech recognition system to convert the audio data into text data and understands the content of the command (in this case, "Start payment").

[0674] Next, the server instructs the terminal to retrieve the current screen information based on the voice command. The terminal uses OCR technology (e.g., Tesseract OCR) to analyze the screen information and extract the data necessary for the transaction (product name, price, etc.). The extracted information is sent back to the server, which uses a generative artificial intelligence model (e.g., OpenAI GPT-4) to analyze the information and generate the data necessary for the payment process.

[0675] The generated payment information is provided to the user audibly using speech synthesis software (e.g., Google Text-to-Speech). For example, it might ask for confirmation by voice, such as, "The payment amount for product name 'ABC' is 3000 yen. Is that correct?"

[0676] When the user issues the voice command "confirm," the terminal retrieves the voice data again and sends it to the server. The server analyzes the voice data and begins the payment process. After the transaction is complete, the processing result is sent to the terminal and the user is notified by voice, such as "Payment completed."

[0677] This system enables visually impaired individuals to conduct electronic transactions quickly and accurately using only voice commands, significantly improving their convenience in daily life. A concrete example of this system involves a user giving a voice command such as "Start payment," followed by a series of processes where the server analyzes the data, retrieves information, and synthesizes the speech. The following is an example of a prompt.

[0678] As a concrete example, the following prompt is input to the generating AI model:

[0679] "Start payment"

[0680] The user says "Start payment." The app uses voice recognition to convert the command into text and sends it to the server.

[0681] The server analyzes the text, understands the "start payment" command, and uses OCR to retrieve information from the screen.

[0682] Analyze the acquired information,

[0683] Prompt example: "Product name: "XYZ", Price: 5000 yen, Please confirm payment method."

[0684] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0685] Step 1:

[0686] The user issues a voice command such as "Start payment." This voice command is captured by the device's microphone.

[0687] Input: User's voice command

[0688] Output: Audio data

[0689] Action: The user says "Start payment" to their smartphone.

[0690] Step 2:

[0691] The speech recognition software inside the device (e.g., Google Cloud Speech-to-Text) converts the speech data into text data.

[0692] Input: Audio data

[0693] Output: Text data

[0694] Operation: Speech recognition software analyzes the speech and generates the text data "Start payment".

[0695] Step 3:

[0696] Text data is sent to the server. The server analyzes the received text data using a speech recognition system and understands the command.

[0697] Input: Text data

[0698] Output: Command content

[0699] Operation: The server recognizes the "start payment" command and determines the next step.

[0700] Step 4:

[0701] The server instructs the terminal to acquire screen information and electronic transaction information. The terminal uses OCR technology (e.g., Tesseract OCR) to analyze the screen information and extract the transaction information.

[0702] Input: Instructions from the server

[0703] Output: Transaction information (text format)

[0704] Operation: The device takes a screenshot of the current screen and uses OCR technology to extract product names and prices as text data.

[0705] Step 5:

[0706] The extracted transaction information is sent to the server. The server uses a generative artificial intelligence model (e.g., OpenAI GPT-4) to analyze the information and generate the data necessary for the payment process.

[0707] Input: Transaction information (text format)

[0708] Output: Payment procedure data

[0709] Operation: The server inputs transaction information into an AI model, which then generates summaries and confirmation messages regarding the payment process.

[0710] Step 6:

[0711] The generated payment processing data is provided to the user via voice using speech synthesis software (e.g., Google Text-to-Speech).

[0712] Input: Payment procedure data

[0713] Output: Voice message

[0714] Operation: The device's speaker will voice-over, "The payment amount for product name 'ABC' is 3000 yen. Is that correct?" to request confirmation.

[0715] Step 7:

[0716] The user issues the voice command "Confirm." The terminal retrieves the voice data again and sends it to the server. The server analyzes the voice data and executes the payment process.

[0717] Input: User's voice command

[0718] Output: Instructions to execute payment procedure

[0719] Operation: Speech recognition software generates the text data "confirm," which the server then analyzes to initiate the payment process.

[0720] Step 8:

[0721] Once the payment process is complete, the processing result is sent from the server to the terminal. The terminal then notifies the user of the processing result via voice.

[0722] Input: Payment procedure result data

[0723] Output: Audio notification of processing results

[0724] Action: The device's speaker will notify the user via voice message, "Payment complete."

[0725] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0726] This invention combines a system for visually impaired individuals to operate a device using voice commands and efficiently acquire information with an emotion engine for recognizing the user's emotions. The program processing of this system is described below.

[0727] First, the user issues a voice command, such as "Please summarize this page." This voice command is captured by the device's microphone and processed as audio data by the device's internal speech recognition software. The device temporarily stores this audio data and sends it to a server for analysis.

[0728] The server uses a speech recognition system to convert voice data into text data, and then uses an emotion engine to recognize the user's emotions from the voice data. At this time, the server analyzes not only voice commands but also features such as the tone, speed, and intonation of the user's voice to estimate the user's emotional state. For example, if the user is feeling anxious, the emotion engine will recognize this.

[0729] Next, the server instructs the terminal to retrieve screen information based on the analysis. The terminal retrieves the information from the currently displayed screen and converts it into text data using optical character recognition (OCR) technology. This text data is also sent back to the server.

[0730] The server analyzes the received text data using an artificial intelligence model, extracts important information, and creates a summary. Based on the user's emotional state recognized by the emotion engine, the way information is presented is adjusted. For example, if the user is feeling anxious, the summary is provided in a concise and easily understandable format.

[0731] The generated summary data is sent to the device, which then uses speech synthesis technology to convert the summarized information into speech and provides it to the user. This process allows the user to quickly and accurately grasp the information they need.

[0732] When a user gives instructions for a specific action, they might issue a voice command such as, "Copy this sentence and paste it into an email." The terminal receives the voice data and sends it to the server. The server analyzes the voice data and, taking into account the user's emotional state, prepares the text editing operation in the most optimal way. The server sends the edited content to the terminal, which then copies and edits the text in the appropriate location. The editing result is provided to the user via voice for confirmation.

[0733] When composing an email, the user might instruct, "Create a new email. The subject should be 'Work Report,' and today I submitted a work report..." In this case, the device also acquires voice data and sends it to the server. The server analyzes the voice data and generates the subject and body of the email. The generated email content is sent to the device, which reads the content aloud and asks the user for confirmation. When the user gives a confirmation voice such as "Send," the server executes the email sending operation.

[0734] In this way, by analyzing both voice commands and user emotions, we provide a highly functional support system that enables visually impaired individuals to efficiently operate devices and acquire information. For example, when a user requests a summary, if the emotion engine recognizes the user's impatience, the system adjusts to provide a quick and concise summary. Also, when composing an email, the system can prevent accidental sending by repeatedly confirming the content according to the user's emotions. This significantly improves the quality of life and work efficiency of visually impaired individuals.

[0735] The following describes the processing flow.

[0736] Step 1:

[0737] The user issues a voice command, such as dictating, "Please summarize this page."

[0738] Step 2:

[0739] The device acquires voice commands via the microphone. The voice is recorded and captured as audio data.

[0740] Step 3:

[0741] The device temporarily stores the audio data and processes it using speech recognition software. The audio data is then converted into text data.

[0742] Step 4:

[0743] The terminal sends the converted text data to the server. The data is sent to the server via the network.

[0744] Step 5:

[0745] The server uses a speech recognition system to analyze text data. It understands the voice command "Please summarize."

[0746] Step 6:

[0747] An emotion engine built into the server recognizes the user's emotions from the voice data. It analyzes the tone, speed, and intonation of the voice to estimate the user's emotional state (e.g., anxiety, joy, tension, etc.).

[0748] Step 7:

[0749] Based on the server's analysis and emotion recognition results, it instructs the terminal to acquire screen information. The instructions are then packaged and sent to the terminal.

[0750] Step 8:

[0751] The device retrieves information from the currently displayed screen. A screen capture is performed to obtain text information from the screen.

[0752] Step 9:

[0753] The device converts screen information it acquires into text data using optical character recognition (OCR) technology. It extracts characters from image data.

[0754] Step 10:

[0755] The terminal sends the converted text data to the server.

[0756] Step 11:

[0757] The server receives text data, which is then analyzed by an artificial intelligence model. Important information from the text data is extracted, and a summary is created.

[0758] Step 12:

[0759] The emotion engine adjusts how summaries are delivered based on the user's emotional state. For example, if the user is feeling anxious, it will provide a concise and easily understandable summary.

[0760] Step 13:

[0761] The server generates summary data and sends it to the terminal. The summary results are packaged and sent to the terminal.

[0762] Step 14:

[0763] The device converts the received summary data into speech using speech synthesis technology. It also converts text data into audio format and plays it through the speaker.

[0764] Step 15:

[0765] The user gives voice commands for specific actions. For example, they might say, "Copy this text and paste it into an email."

[0766] Step 16:

[0767] The terminal acquires audio data and sends it to the server. A package for converting the audio recording to text is generated.

[0768] Step 17:

[0769] The server analyzes the audio data and understands the instructions given. It converts the audio to text and then analyzes the content.

[0770] Step 18:

[0771] The emotion engine optimizes text editing operations based on the user's emotional state. For example, if the user is emotionally agitated, the operation steps will be simplified.

[0772] Step 19:

[0773] The server sends the edited content to the terminal. The edited content is packaged and transferred.

[0774] Step 20:

[0775] The device provides the user with the edited results via voice. It reads aloud, "The text has been pasted into the email. Do you want to confirm?"

[0776] Step 21:

[0777] The user requests a voice message to create an email. They instruct, "Create a new email. The subject line should be 'Work Report,' and I submitted a work report today..."

[0778] Step 22:

[0779] The terminal acquires audio data and sends it to the server. A package for analysis is generated from the audio recording.

[0780] Step 23:

[0781] The server analyzes the audio data and generates a subject and body. It generates text such as "Subject: Work Report" and "Body: Today I gave a work report..."

[0782] Step 24:

[0783] The generated email content is sent to the terminal. The subject and body data are packaged and forwarded.

[0784] Step 25:

[0785] The device reads the email content aloud and asks the user for confirmation. It instructs the user, "Do you want to send it? If yes, please say 'Send'."

[0786] Step 26:

[0787] The user gives a confirmation voice command, saying "Send."

[0788] Step 27:

[0789] The device acquires the confirmation audio and sends it to the server. The audio data for confirmation is sent to the server.

[0790] Step 28:

[0791] The server performs a final check and then sends the email. The email is sent to the specified recipient.

[0792] (Example 2)

[0793] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0794] Traditional systems primarily relied on voice commands for operation and information provision, enabling visually impaired individuals to efficiently operate devices and quickly and appropriately obtain information. However, these systems were insufficient in responding to the user's emotional state. As a result, users may experience frustration or be unable to obtain appropriate information. Therefore, there is a need for a system that can adjust device operation and information provision methods according to the user's emotional state, making it more efficient and reassuring to use.

[0795] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0796] In this invention, the server includes means for recognizing the user's emotions from voice data, means for acquiring screen information based on the emotion recognition results, and means including a generative artificial intelligence model for summarizing or editing the acquired screen information. This makes it possible to adjust the appropriate information provision method according to the user's emotional state, thereby improving convenience and peace of mind for visually impaired users when using the device.

[0797] A "voice command" refers to a voice command that a user makes to operate or instruct a device.

[0798] "Voice data" refers to data that represents acquired voice commands in digital format.

[0799] "Speech recognition" refers to the technology that analyzes speech data and converts it into text data.

[0800] "Emotion recognition" refers to a technology that analyzes and estimates a user's emotional state based on voice data.

[0801] "Screen information" refers to the visual information displayed on a device's screen.

[0802] Optical Character Recognition (OCR) refers to a technology that analyzes characters in an image and converts them into text data.

[0803] A "generative artificial intelligence model" refers to AI technology that analyzes input data and generates output tailored to a specific task.

[0804] A "summary" refers to a short, concise compilation of key information extracted from acquired text data.

[0805] "Editing" refers to the operation of changing or modifying the content of text data based on user instructions.

[0806] "Speech synthesis" refers to the technology that analyzes text data and converts it into human speech.

[0807] "Email" refers to a method of communication for creating and sending messages electronically.

[0808] Modes for carrying out the invention

[0809] This invention is a system for visually impaired individuals to operate a device using voice commands and efficiently acquire information, and it incorporates an emotion engine to recognize the user's emotions. Specific embodiments are described below.

[0810] Hardware and software to use

[0811] This system uses the following hardware and software:

[0812] Speech recognition software: For example, Google's speech recognition API

[0813] Emotion recognition software: e.g., IBM Watson Tone Analyzer

[0814] Optical Character Recognition (OCR) technology: e.g., Tesseract OCR

[0815] Generative artificial intelligence models: e.g., OpenAI's GPT-3

[0816] Speech synthesis technology: For example, IBM Watson Text to Speech

[0817] System configuration and operation

[0818] 1. Obtaining voice commands

[0819] The user issues a voice command to the device. For example, they might give a command such as, "Please summarize this page."

[0820] The device's microphone picks up this voice command, and it is processed as voice data by speech recognition software.

[0821] 2. Sending audio data

[0822] The device temporarily stores the voice data and sends it to the server via a secure protocol (e.g., HTTPS).

[0823] 3. Analysis of audio data

[0824] The server receives the audio data and converts it into text data using a speech recognition system.

[0825] Furthermore, the server recognizes the user's emotions from the voice data. It uses an emotion engine to analyze tone, speed, intonation, and other factors.

[0826] 4. Obtaining screen information

[0827] Based on the analyzed voice commands and emotion recognition results, the server instructs the terminal to retrieve screen information.

[0828] The device captures the currently displayed screen and converts it into text data using OCR technology.

[0829] 5. Generating a summary

[0830] The server receives text data sent from the terminal and analyzes it using a generative AI model. It then extracts important information and creates a summary.

[0831] The way information is delivered is adjusted based on the user's emotional state recognized by the emotion engine.

[0832] 6. Transmission of summary data and speech synthesis

[0833] The generated summary data is sent from the server to the terminal.

[0834] The device converts the summarized information into speech using speech synthesis technology and provides it to the user.

[0835] Specific example

[0836] For example, if a user issues a voice command such as "Please summarize this page," the device's microphone captures the voice, and a speech recognition API processes it into audio data. The server receives this audio data and uses an emotion engine to analyze the user's emotions. If the user is feeling anxious, the server generates a quick and concise summary, converts it into speech using speech synthesis technology, and provides it to the user through the device.

[0837] Example of a prompt

[0838] "Please describe a scenario in which a visually impaired person issues a voice command saying, 'Please summarize this page,' and the system, recognizing the user's impatience, quickly and concisely provides a summary."

[0839] In this way, this system provides flexible and rapid information tailored to the user's emotional state, creating an environment where visually impaired individuals can effectively use devices.

[0840] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0841] Step 1:

[0842] The user issues a voice command, specifically a request such as, "Please summarize this page." The device's microphone picks up the voice command and processes it as audio data using speech recognition software (e.g., Google's Speech Recognition API). The input is the user's voice command, and the output is audio data.

[0843] Step 2:

[0844] The terminal temporarily stores the audio data and sends it to the server via a secure protocol (e.g., HTTPS). Specifically, the binary data of the audio data is transmitted using a secure connection. The input is the audio data, and the output is the audio data sent to the server.

[0845] Step 3:

[0846] The server receives audio data and converts it into text data using a speech recognition system (e.g., Google Cloud Speech-to-Text). This process involves dividing the audio signal and mapping it to language units. The input is audio data, and the output is text data.

[0847] Step 4:

[0848] The server uses an emotion engine (e.g., IBM Watson Tone Analyzer) to recognize the user's emotions from voice data. It analyzes features such as tone, speed, and intonation of the voice to estimate emotional states such as anxiety or relaxation. The input is voice data and text data, and the output is emotional state data.

[0849] Step 5:

[0850] Based on the voice commands and emotion recognition results analyzed by the server, it instructs the terminal to retrieve screen information. Specifically, it sends instructions to the terminal via an HTTP request. The input is the voice command text and emotion state data, and the output is an instruction to retrieve screen information.

[0851] Step 6:

[0852] The device captures the information displayed on the screen and converts it into text data using optical character recognition (OCR) technology (e.g., Tesseract OCR). Alternatively, it can take a screenshot of the screen and extract text information from the image. The input is a screen image, and the output is text data.

[0853] Step 7:

[0854] The terminal sends the converted text data back to the server. Specifically, the text data is sent using a secure connection. The input is text data, and the output is the text data sent to the server.

[0855] Step 8:

[0856] The server analyzes the received text using a generative AI model (e.g., OpenAI's GPT-3), extracts important information, and creates a summary. The information delivery method is adjusted based on the user's emotional state recognized by the emotion engine. The input is text data and emotional state data, and the output is summary data.

[0857] Step 9:

[0858] The generated summary data is sent from the server to the terminal. A secure protocol is used to transmit the data. The input is the summary data, and the output is the summary data sent to the terminal.

[0859] Step 10:

[0860] The terminal converts summarized information into speech using speech synthesis technology (e.g., IBM Watson Text to Speech) and provides it to the user. The generated audio data is played back through the terminal's speaker. The input is summarized data, and the output is audio data.

[0861] Step 11:

[0862] The user gives a specific command (e.g., "Copy this text and paste it into an email"). The terminal acquires the voice data and sends it to the server. The input is the user's voice command, and the output is the voice data sent to the server.

[0863] Step 12:

[0864] The server analyzes the audio data and prepares the optimal text editing method, taking into account the user's emotional state. The prepared editing content is then sent to the terminal. Input consists of audio data and emotional state data, while output is editing instruction data.

[0865] Step 13:

[0866] The terminal transcribes or edits text to the appropriate location based on editing instruction data received from the server. The editing result is provided to the user via audio for user confirmation. The input is the editing instruction data, and the output is the edited text data and its audio data.

[0867] Step 14:

[0868] When a user instructs, "Create a new email. The subject line should be 'Work Report,' and I submitted my work report today...", the terminal acquires voice data and sends it to the server. The input is the user's voice command, and the output is the voice data sent to the server.

[0869] Step 15:

[0870] The server analyzes the audio data and generates a subject and body for the email. The generated email content is then sent to the terminal. The input consists of audio data and emotional state data, while the output consists of the email subject and body data.

[0871] Step 16:

[0872] The terminal reads the content aloud and prompts the user for confirmation. Once the user confirms "Send," the server executes the email sending operation. The input consists of the email subject, body data, and the user's send instruction, and the output is the sent email.

[0873] (Application Example 2)

[0874] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0875] For visually impaired individuals to find products appropriately and shop efficiently in physical stores, a voice-command navigation system is necessary. However, conventional voice command systems do not take user emotions into consideration, and at the same time, they do not adequately provide specific directions within stores or location information for products. As a result, shopping becomes difficult, especially for users who experience anxiety or stress.

[0876] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring a user's voice command, means for analyzing the acquired voice command, means for acquiring screen information based on the analyzed voice command, means including a generative artificial intelligence model for summarizing or editing the acquired screen information, means for providing the summarized or edited information to the user by voice, means including an emotion engine for analyzing emotions using the user's voice command and voice characteristics, and means for adjusting the information provision method based on the user's emotions. This makes it possible to provide navigation and product information quickly and concisely in a physical store while taking into account the user's emotional state.

[0877] "Means for acquiring user voice commands" refers to a microphone or voice input device that recognizes voice commands uttered by the user and inputs them into the device.

[0878] "Means for analyzing voice commands" refers to a voice recognition system that converts acquired voice commands into digital data, analyzes its content, and converts it into appropriate instructions.

[0879] "Means of acquiring screen information" refers to cameras and screen capture functions used to capture and analyze information displayed on a device.

[0880] A "generative artificial intelligence model" refers to an AI algorithm or system that extracts important information from large amounts of data and performs summarization or editing.

[0881] "Means of providing information to users in audio format" refers to speech synthesis technology and speakers used to convey analysis results and summaries to users as audio.

[0882] An "emotion engine" refers to software or a system that analyzes a user's voice characteristics and estimates their emotional state.

[0883] "Means for adjusting the method of information provision" refers to a function that dynamically changes the content and format of the information provided based on the user's emotional state.

[0884] "Means for transcribing or editing text" refers to software functions that copy digital text to the appropriate location and edit it based on the user's voice commands.

[0885] "Means for creating, reviewing, and sending emails" refers to systems and software functions that automatically generate emails based on user voice instructions, review their content, and then send them.

[0886] This invention is a system designed to assist visually impaired individuals in shopping at physical stores. The system includes a set of functions that allow the user, wearing smart glasses and using voice commands, to obtain information on product locations and store navigation. Furthermore, it reads the user's emotions from their voice and adjusts the assistance accordingly.

[0887] First, the user puts on smart glasses and makes a voice command such as "Tell me where the product is." The device's microphone captures this voice, and the Google Cloud Speech-to-Text API converts the voice into text data.

[0888] Next, along with the text data, characteristics such as tone, speed, and intonation of the voice are sent to the Azure Emotion API for emotion analysis. As a result of the analysis, a specific emotional state is recognized, such as when the user is anxious.

[0889] Screen information is acquired, for example, using the camera of smart glasses. The camera takes pictures of the store's map and product shelves, and converts them into text using Tesseract OCR technology. Based on this text data, the system works with the Google Maps API to determine the location within the physical store.

[0890] A specific prompt would be sent to the Generative AI model (GPT-4) with a request such as, "The Emotion API has detected that the user is feeling anxious. The user is currently looking for the location of a sticky note in the store. Please generate concise and easy-to-understand navigation instructions."

[0891] A generative AI model generates optimal feedback (for example, "Turn right and go straight down the next corridor. The sticky note is on the shelf to your right."), and this feedback is converted into speech by the Google Text-to-Speech API. The information is then delivered to the user through the smart glasses' speaker.

[0892] This system allows users to receive voice-based navigation and product location information tailored to their emotional state, enabling them to shop efficiently and comfortably in physical stores.

[0893] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0894] Step 1:

[0895] The user wears smart glasses and issues a voice command. Voice input is acquired through the smart glasses' microphone. The input voice data is the output of this step.

[0896] Step 2:

[0897] The device sends the acquired audio data to the Google Cloud Speech-to-Text API, which converts the audio into text data. The input is audio data, and the output is the converted text data.

[0898] Step 3:

[0899] The device sends text data along with characteristic data such as tone, speed, and intonation of speech to the Azure Emotion API. The input consists of text data and speech characteristic data, and the output is the result of the user's emotion analysis.

[0900] Step 4:

[0901] The server processes data based on the sentiment analysis results to determine a navigation method that corresponds to the user's emotions. The input here is the sentiment analysis results, and the output is navigation data corresponding to those emotions.

[0902] Step 5:

[0903] The device uses the camera on smart glasses to take pictures of the store interior and product shelves, and converts them into text data using Tesseract OCR. The input is the image taken by the camera, and the output is the converted text data.

[0904] Step 6:

[0905] The server uses OCR data in conjunction with the Google Maps API to identify location information within the store. The input is text data converted by OCR, and the output is the identified location information.

[0906] Step 7:

[0907] The device uses a generative AI model (GPT-4) to generate navigation instructions for the user based on sentiment analysis results and location information. The input is sentiment analysis results and location information, and the output is navigation instruction text.

[0908] Step 8:

[0909] The device sends the generated navigation instruction text to the Google Text-to-Speech API, where it is converted into speech. The input is navigation instruction text, and the output is audio data.

[0910] Step 9:

[0911] The device provides voice data to the user through the smart glasses' speaker. This allows the user to receive accurate directions and product information within the store via voice. The input is voice data, and the output is voice guidance.

[0912] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0913] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0914] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0915] [Third Embodiment]

[0916] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0917] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0918] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0919] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0920] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0921] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0922] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0923] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0924] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0925] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0926] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0927] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0928] This invention provides a system that allows visually impaired individuals to operate a device using voice commands and efficiently acquire information. The program processing of this system is described below.

[0929] First, the user issues a voice command. For example, the user might say, "Please summarize this page." This voice command is picked up by the device's microphone and processed as voice data by the device's internal voice recognition software. The voice data is then sent to a server for analysis. The server uses its voice recognition system to convert the voice data into text data and understands the command "Please summarize."

[0930] Next, the server sends instructions to the terminal based on the analyzed command. In this case, it specifies "summarize screen information." The terminal retrieves information from the currently displayed screen. For example, it takes a screen capture and extracts text information using optical character recognition (OCR) technology. The extracted text data is then sent back to the server for summarization.

[0931] The server analyzes the received text data using a generation AI model, extracts key information, and creates a summary. The generated summary data is sent to the terminal, which then reads the summarized information aloud using speech synthesis. This allows the user to easily grasp the main information on the screen.

[0932] Furthermore, if the user gives specific instructions, such as "Copy this sentence and paste it into an email," they can issue a voice command. The terminal retrieves the voice data again and sends it to the server. The server analyzes the voice data and prepares text editing operations based on the instructions. The server sends the edited content to the terminal, and the terminal copies and edits the text in the appropriate place. The editing results are provided to the user via voice, allowing the user to confirm them.

[0933] When a user composes an email, they might be instructed, for example, "Create a new email. The subject should be 'Work Report,' and I submitted my work report today..." The device receives the voice data and sends it to the server. The server analyzes the voice data and generates the subject "Work Report" and the body "I submitted my work report today...". The generated email content is sent to the device, which reads the email content aloud and asks the user for confirmation. When the user gives a confirmation voice such as "Send," the server executes the email sending operation.

[0934] Thus, a highly functional assistance system that allows for a smooth series of operations, from acquiring voice commands to summarizing information, editing text, and creating and sending emails, enables visually impaired individuals to make better use of their devices, greatly improving their quality of life and work efficiency.

[0935] The following describes the processing flow.

[0936] Step 1:

[0937] The user issues a voice command, such as dictating, "Please summarize this page."

[0938] Step 2:

[0939] The device acquires voice commands via the microphone. The voice is recorded and captured as audio data.

[0940] Step 3:

[0941] The device temporarily stores the audio data and processes it using speech recognition software. The audio data is then converted into text data.

[0942] Step 4:

[0943] The device converts the audio data into text data and then sends it to the server. The data is sent to the server via the network.

[0944] Step 5:

[0945] The server uses a speech recognition system to analyze the voice data and generate text data. It analyzes the voice command and understands the meaning of "Please summarize."

[0946] Step 6:

[0947] Based on the analysis performed by the server, it instructs the terminal to retrieve screen information. The instructions are then packaged and sent to the terminal.

[0948] Step 7:

[0949] The device retrieves information from the currently displayed screen. It takes a screen capture and retrieves the text information from the screen.

[0950] Step 8:

[0951] The device converts screen information into text data using optical character recognition (OCR) technology. It extracts characters from image data.

[0952] Step 9:

[0953] The terminal extracts text data and sends it to the server. The converted text data is packaged and sent to the server.

[0954] Step 10:

[0955] The server analyzes the received text data using a generation AI model. It extracts key information from the text data to summarize its main points.

[0956] Step 11:

[0957] The server generates summary data and sends it to the terminal. The summary results are packaged and sent to the terminal.

[0958] Step 12:

[0959] The device receives summary data, converts it into speech using speech synthesis technology, and provides it to the user. It also converts text data into speech format and plays it through the speaker.

[0960] Specific example 1:

[0961] User: "Could you please summarize this page?"

[0962] Terminal: Receives voice commands and sends them to the server for analysis.

[0963] Server: Analyzes the audio and sends a summary of the screen information back to the terminal.

[0964] Terminal: Captures screen information, converts it to text using OCR, and sends it to the server.

[0965] Server: Summarizes the received text and sends it to the terminal.

[0966] Terminal: Provides a summary to the user in audio format.

[0967] Specific example 2:

[0968] User: "Please create a new email. The subject line should be 'Work Report,' and I submitted my work report today..."

[0969] Terminal: Receives voice commands and sends them to the server for analysis.

[0970] Server: Generates subject and body text and sends it to the terminal.

[0971] Terminal: Reads the email content aloud and asks the user for confirmation.

[0972] User: "Send"

[0973] Terminal: Retrieves confirmation audio and sends it to the server.

[0974] Server: Sending email.

[0975] In this way, a system is realized that allows visually impaired people to operate devices and acquire information more efficiently.

[0976] (Example 1)

[0977] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0978] Visually impaired individuals face challenges in operating devices and efficiently obtaining information. Furthermore, complex operations such as retrieving, summarizing, editing, and composing emails are difficult to perform smoothly using voice commands. To address these issues, a system is needed that analyzes voice commands and responds quickly and accurately.

[0979] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0980] In this invention, the server includes means for acquiring user voice commands, means for analyzing the acquired voice commands, and means for acquiring screen information based on the analyzed voice commands. This enables visually impaired individuals to operate a device using voice commands, efficiently acquire screen information, and compose and send emails.

[0981] "User" refers to the person who operates the system in question.

[0982] "Voice commands" refer to instructions given by the user using their voice.

[0983] A "terminal" refers to an electronic device that has functions such as acquiring voice commands, converting voice data, and acquiring screen information.

[0984] A "server" refers to a central processing unit that performs tasks such as analyzing voice data, understanding commands, using generative AI models, and sending and receiving data.

[0985] "Voice data" refers to the digital data of a user's voice commands.

[0986] "Voice recognition software" refers to a program that converts acquired voice commands into text data.

[0987] "Text data" refers to character information converted by speech recognition software.

[0988] "Screen information" refers to all information displayed on the device, such as text and images.

[0989] "Optical character recognition technology" refers to the technology that recognizes characters from images and extracts them as text data.

[0990] A "generative AI model" refers to an artificial intelligence model that analyzes received data and performs tasks such as summarizing and editing.

[0991] A "summary" is information that extracts the important parts from screen information and presents them concisely.

[0992] "Editing" refers to the process of modifying or rewriting acquired text data based on user instructions.

[0993] "Speech synthesis" refers to a technology that converts text data into speech and provides it to the user.

[0994] This invention is a system for visually impaired individuals to operate a device using voice commands and efficiently acquire information. The following describes specific embodiments of this system.

[0995] Hardware and software usage

[0996] terminal

[0997] The device includes a microphone, speech recognition software, and optical character recognition (OCR) technology. Speech recognition software such as Google Speech-to-Text API is used. Optical character recognition technology such as Tesseract OCR is used.

[0998] server

[0999] The server includes functions for analyzing voice data, understanding commands, using generative AI models, and sending and receiving data. DeepSpeech is used for the speech recognition system, and GPT-3 is used for the generative AI model.

[1000] Operation Description

[1001] Acquisition and recognition of voice commands

[1002] 1. The user issues a voice command. For example, they might say, "Please summarize this page."

[1003] 2. The device's microphone acquires voice commands, which are then converted into voice data by speech recognition software.

[1004] 3. The converted audio data is sent to the server.

[1005] Audio data analysis and command understanding

[1006] 1. The server receives the audio data and converts it into text data using a speech recognition system such as DeepSpeech.

[1007] 2. The server understands the command from the text data and interprets the instruction, "Please summarize this page."

[1008] Acquiring screen information and extracting text

[1009] 1. The server sends an instruction to the terminal called "Summarize screen information".

[1010] 2. Take a screenshot of the screen currently displayed on the device.

[1011] 3. Extract text data from the screen capture using Tesseract OCR or similar software, and send it back to the server.

[1012] Text data summary generation

[1013] 1. The server analyzes the received text data using a generative AI model such as GPT-3, extracts important information, and generates a summary.

[1014] 2. The generated summary data is sent to the terminal.

[1015] Summary audio output

[1016] 1. The device generates a summary of the received information using speech synthesis (such as the Google Text-to-Speech API) and provides it to the user in audio format.

[1017] Specific examples and prompt statements

[1018] Specific example: Summary of page information

[1019] When a user issues a voice command such as "Please summarize this page," the device's microphone captures the voice, which is then converted into digital data by speech recognition software. The converted voice data is sent to a server, where DeepSpeech converts it back into text, and GPT-3 extracts key information to generate a summary. The generated summary data is then sent to the device, and speech synthesis software reads the summary aloud to the user.

[1020] Specific example: Creating and sending an email

[1021] When a user instructs, "Create a new email. The subject line should be 'Work Report,' and I submitted a work report today," the device's microphone captures the voice, which is then converted into digital data by speech recognition software. The converted voice data is sent to a server, which analyzes the voice data and generates a subject line and body. The generated email content is sent to the device, and the user is asked to confirm it as a voice message. Once the user confirms "Send," the server executes the email sending operation.

[1022] Thus, this system allows for a smooth series of operations, from acquiring voice commands to summarizing information, editing text, and creating and sending emails. This system enables visually impaired individuals to efficiently utilize devices and improve their quality of life and work efficiency.

[1023] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1024] Step 1:

[1025] Voice command generation and acquisition:

[1026] The user speaks a voice command into the device's microphone. For example, they might say, "Please summarize this page."

[1027] Input: User's voice command

[1028] Output: Audio data acquired as an audio signal by the device's microphone.

[1029] Specific actions:

[1030] When a user speaks into the microphone, their voice is converted into a digital signal by the microphone.

[1031] Step 2:

[1032] Speech recognition and data conversion:

[1033] The device uses built-in speech recognition software to convert speech data into text data.

[1034] Input: Audio data (audio signal)

[1035] Output: Text data (content of voice command)

[1036] Specific actions:

[1037] The device's Google Speech-to-Text API analyzes the audio signal and converts it into text data that says, "Please summarize this page."

[1038] Step 3:

[1039] Sending audio data:

[1040] The terminal sends the converted text data to the server.

[1041] Input: Text data (content of voice command)

[1042] Output: Text data sent to the server

[1043] Specific actions:

[1044] The terminal's communication module sends text data based on voice commands to the server.

[1045] Step 4:

[1046] Text data analysis and command understanding:

[1047] The server analyzes the received text data to understand the content of the voice command.

[1048] Input: Received text data

[1049] Output: Analysis results (understanding of command content)

[1050] Specific actions:

[1051] The server's speech recognition system, such as DeepSpeech, analyzes the text data and understands the command "Please summarize."

[1052] Step 5:

[1053] Sending instructions based on commands:

[1054] In response to the command, the server sends an instruction to the terminal requesting a "summary of screen information."

[1055] Input: Analysis results (command: "Please provide a summary")

[1056] Output: Signal containing the instruction content (summary of screen information)

[1057] Specific actions:

[1058] The server sends a signal to the terminal instructing it to retrieve and summarize the screen information.

[1059] Step 6:

[1060] Retrieving screen information:

[1061] Capture a screenshot of the screen currently displayed on the device.

[1062] Input: Instructions from the server (summary of screen information)

[1063] Output: Screen capture image

[1064] Specific actions:

[1065] The device uses its screen capture function to obtain the current screen content as an image.

[1066] Step 7:

[1067] Optical character recognition (OCR):

[1068] The device extracts text data from the acquired screen capture image using optical character recognition technology.

[1069] Input: Screen capture image

[1070] Output: Extracted text data

[1071] Specific actions:

[1072] The device's Tesseract OCR analyzes the captured image and extracts the text information displayed on the screen.

[1073] Step 8:

[1074] Sending text data:

[1075] The terminal then sends the extracted text data back to the server.

[1076] Input: Extracted text data

[1077] Output: Text data sent to the server

[1078] Specific actions:

[1079] The extracted text data is sent to the server via the terminal's communication module.

[1080] Step 9:

[1081] Text data summary generation:

[1082] The server analyzes the received text data using a generating AI model (e.g., GPT-3), extracts important information, and generates a summary.

[1083] Input: Received text data

[1084] Output: Generated summary text

[1085] Specific actions:

[1086] A GPT-3 generation AI model on the server analyzes the text data and generates a summary.

[1087] Step 10:

[1088] Send summary data:

[1089] The server sends the generated summary data to the terminal.

[1090] Input: Generated summary text

[1091] Output: Summary text sent to the terminal

[1092] Specific actions:

[1093] The server sends a signal containing summary data to the terminal.

[1094] Step 11:

[1095] Summary audio output:

[1096] The terminal provides the user with the received summary data in voice using speech synthesis technology.

[1097] Input: Received summary text

[1098] Output: Summary provided in audio

[1099] Specific actions:

[1100] The device's Google Text-to-Speech API converts the text into speech and reads it aloud to the user.

[1101] (Application Example 1)

[1102] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1103] Visually impaired individuals face difficulties in conducting electronic transactions and making payments due to the need to understand screen information and edit text, hindering their ability to easily perform daily activities and work. This problem is particularly pronounced in transaction confirmation procedures and payment processes, causing significant stress for visually impaired individuals. Therefore, there is a need for systems that support visually impaired individuals in conducting electronic transactions quickly and accurately using voice commands.

[1104] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1105] In this invention, the server includes means for analyzing the user's voice commands, means for acquiring screen information or electronic transaction information based on the analyzed voice commands, and means including a generative artificial intelligence model for summarizing or editing the acquired information. This makes it possible for visually impaired individuals to confirm the contents of electronic transactions using voice commands and to perform all necessary editing and operations by voice.

[1106] A "voice command" is a voice input that includes instructions or requests that a user makes to a device.

[1107] "Means of analysis" refers to a device or software that converts acquired voice commands into text data and performs the process of understanding and interpreting its content.

[1108] "Screen information" refers to visual information such as text, images, and icons displayed on the device screen.

[1109] "Electronic transaction information" refers to information regarding the purchase, sale, and payment of goods and services conducted in digital format.

[1110] A "generative artificial intelligence model" is an artificial intelligence system that learns from large amounts of data and can automatically perform tasks such as summarizing, editing, and generating text.

[1111] "Means of providing information to the user in audio" refers to a device or software that performs the process of audibly communicating generated or acquired information to the user using speech synthesis technology.

[1112] "Transaction details" refers to detailed information about a specific electronic transaction, such as the goods, services, price, quantity, and trading partner.

[1113] "Transcribing or editing means" refers to any device or software that performs the process of copying, editing, or modifying text or transaction information to another location based on user instructions.

[1114] "Means for creating, reviewing, and sending emails" refers to a device or software that performs the process of creating email content according to the user's voice instructions, having the user review the content, and sending the email after approval.

[1115] "Means for executing a transaction" refers to a device or software that executes a process to complete electronic transaction procedures based on the user's voice instructions.

[1116] To implement this invention, a system will be constructed that allows visually impaired individuals to easily conduct electronic transactions using voice commands. The details of this system are described below.

[1117] First, the user issues a voice command using their smartphone's microphone. This voice command might be something like "Start payment." This voice command is processed as audio data by speech recognition software (e.g., Google Cloud Speech-to-Text) inside the device. The audio data is sent to a server for analysis. The server uses its speech recognition system to convert the audio data into text data and understands the content of the command (in this case, "Start payment").

[1118] Next, the server instructs the terminal to retrieve the current screen information based on the voice command. The terminal uses OCR technology (e.g., Tesseract OCR) to analyze the screen information and extract the data necessary for the transaction (product name, price, etc.). The extracted information is sent back to the server, which uses a generative artificial intelligence model (e.g., OpenAI GPT-4) to analyze the information and generate the data necessary for the payment process.

[1119] The generated payment information is provided to the user audibly using speech synthesis software (e.g., Google Text-to-Speech). For example, it might ask for confirmation by voice, such as, "The payment amount for product name 'ABC' is 3000 yen. Is that correct?"

[1120] When the user issues the voice command "confirm," the terminal retrieves the voice data again and sends it to the server. The server analyzes the voice data and begins the payment process. After the transaction is complete, the processing result is sent to the terminal and the user is notified by voice, such as "Payment completed."

[1121] This system enables visually impaired individuals to conduct electronic transactions quickly and accurately using only voice commands, significantly improving their convenience in daily life. A concrete example of this system involves a user giving a voice command such as "Start payment," followed by a series of processes where the server analyzes the data, retrieves information, and synthesizes the speech. The following is an example of a prompt.

[1122] As a concrete example, the following prompt is input to the generating AI model:

[1123] "Start payment"

[1124] The user says "Start payment." The app uses voice recognition to convert the command into text and sends it to the server.

[1125] The server analyzes the text, understands the "start payment" command, and uses OCR to retrieve information from the screen.

[1126] Analyze the acquired information,

[1127] Prompt example: "Product name: "XYZ", Price: 5000 yen, Please confirm payment method."

[1128] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1129] Step 1:

[1130] The user issues a voice command such as "Start payment." This voice command is captured by the device's microphone.

[1131] Input: User's voice command

[1132] Output: Audio data

[1133] Action: The user says "Start payment" to their smartphone.

[1134] Step 2:

[1135] The speech recognition software inside the device (e.g., Google Cloud Speech-to-Text) converts the speech data into text data.

[1136] Input: Audio data

[1137] Output: Text data

[1138] Operation: Speech recognition software analyzes the speech and generates the text data "Start payment".

[1139] Step 3:

[1140] Text data is sent to the server. The server analyzes the received text data using a speech recognition system and understands the command.

[1141] Input: Text data

[1142] Output: Command content

[1143] Operation: The server recognizes the "start payment" command and determines the next step.

[1144] Step 4:

[1145] The server instructs the terminal to acquire screen information and electronic transaction information. The terminal uses OCR technology (e.g., Tesseract OCR) to analyze the screen information and extract the transaction information.

[1146] Input: Instructions from the server

[1147] Output: Transaction information (text format)

[1148] Operation: The device takes a screenshot of the current screen and uses OCR technology to extract product names and prices as text data.

[1149] Step 5:

[1150] The extracted transaction information is sent to the server. The server uses a generative artificial intelligence model (e.g., OpenAI GPT-4) to analyze the information and generate the data necessary for the payment process.

[1151] Input: Transaction information (text format)

[1152] Output: Payment procedure data

[1153] Operation: The server inputs transaction information into an AI model, which then generates summaries and confirmation messages regarding the payment process.

[1154] Step 6:

[1155] The generated payment processing data is provided to the user via voice using speech synthesis software (e.g., Google Text-to-Speech).

[1156] Input: Payment procedure data

[1157] Output: Voice message

[1158] Operation: The device's speaker will voice-over, "The payment amount for product name 'ABC' is 3000 yen. Is that correct?" to request confirmation.

[1159] Step 7:

[1160] The user issues the voice command "Confirm." The terminal retrieves the voice data again and sends it to the server. The server analyzes the voice data and executes the payment process.

[1161] Input: User's voice command

[1162] Output: Instructions to execute payment procedure

[1163] Operation: Speech recognition software generates the text data "confirm," which the server then analyzes to initiate the payment process.

[1164] Step 8:

[1165] Once the payment process is complete, the processing result is sent from the server to the terminal. The terminal then notifies the user of the processing result via voice.

[1166] Input: Payment procedure result data

[1167] Output: Audio notification of processing results

[1168] Action: The device's speaker will notify the user via voice message, "Payment complete."

[1169] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1170] This invention combines a system for visually impaired individuals to operate a device using voice commands and efficiently acquire information with an emotion engine for recognizing the user's emotions. The program processing of this system is described below.

[1171] First, the user issues a voice command, such as "Please summarize this page." This voice command is captured by the device's microphone and processed as audio data by the device's internal speech recognition software. The device temporarily stores this audio data and sends it to a server for analysis.

[1172] The server uses a speech recognition system to convert voice data into text data, and then uses an emotion engine to recognize the user's emotions from the voice data. At this time, the server analyzes not only voice commands but also features such as the tone, speed, and intonation of the user's voice to estimate the user's emotional state. For example, if the user is feeling anxious, the emotion engine will recognize this.

[1173] Next, the server instructs the terminal to retrieve screen information based on the analysis. The terminal retrieves the information from the currently displayed screen and converts it into text data using optical character recognition (OCR) technology. This text data is also sent back to the server.

[1174] The server analyzes the received text data using an artificial intelligence model, extracts important information, and creates a summary. Based on the user's emotional state recognized by the emotion engine, the way information is presented is adjusted. For example, if the user is feeling anxious, the summary is provided in a concise and easily understandable format.

[1175] The generated summary data is sent to the device, which then uses speech synthesis technology to convert the summarized information into speech and provides it to the user. This process allows the user to quickly and accurately grasp the information they need.

[1176] When a user gives instructions for a specific action, they might issue a voice command such as, "Copy this sentence and paste it into an email." The terminal receives the voice data and sends it to the server. The server analyzes the voice data and, taking into account the user's emotional state, prepares the text editing operation in the most optimal way. The server sends the edited content to the terminal, which then copies and edits the text in the appropriate location. The editing result is provided to the user via voice for confirmation.

[1177] When composing an email, the user might instruct, "Create a new email. The subject should be 'Work Report,' and today I submitted a work report..." In this case, the device also acquires voice data and sends it to the server. The server analyzes the voice data and generates the subject and body of the email. The generated email content is sent to the device, which reads the content aloud and asks the user for confirmation. When the user gives a confirmation voice such as "Send," the server executes the email sending operation.

[1178] In this way, by analyzing both voice commands and user emotions, we provide a highly functional support system that enables visually impaired individuals to efficiently operate devices and acquire information. For example, when a user requests a summary, if the emotion engine recognizes the user's impatience, the system adjusts to provide a quick and concise summary. Also, when composing an email, the system can prevent accidental sending by repeatedly confirming the content according to the user's emotions. This significantly improves the quality of life and work efficiency of visually impaired individuals.

[1179] The following describes the processing flow.

[1180] Step 1:

[1181] The user issues a voice command, such as dictating, "Please summarize this page."

[1182] Step 2:

[1183] The device acquires voice commands via the microphone. The voice is recorded and captured as audio data.

[1184] Step 3:

[1185] The device temporarily stores the audio data and processes it using speech recognition software. The audio data is then converted into text data.

[1186] Step 4:

[1187] The terminal sends the converted text data to the server. The data is sent to the server via the network.

[1188] Step 5:

[1189] The server uses a speech recognition system to analyze text data. It understands the voice command "Please summarize."

[1190] Step 6:

[1191] An emotion engine built into the server recognizes the user's emotions from the voice data. It analyzes the tone, speed, and intonation of the voice to estimate the user's emotional state (e.g., anxiety, joy, tension, etc.).

[1192] Step 7:

[1193] Based on the server's analysis and emotion recognition results, it instructs the terminal to acquire screen information. The instructions are then packaged and sent to the terminal.

[1194] Step 8:

[1195] The device retrieves information from the currently displayed screen. A screen capture is performed to obtain text information from the screen.

[1196] Step 9:

[1197] The device converts screen information it acquires into text data using optical character recognition (OCR) technology. It extracts characters from image data.

[1198] Step 10:

[1199] The terminal sends the converted text data to the server.

[1200] Step 11:

[1201] The server receives text data, which is then analyzed by an artificial intelligence model. Important information from the text data is extracted, and a summary is created.

[1202] Step 12:

[1203] The emotion engine adjusts how summaries are delivered based on the user's emotional state. For example, if the user is feeling anxious, it will provide a concise and easily understandable summary.

[1204] Step 13:

[1205] The server generates summary data and sends it to the terminal. The summary results are packaged and sent to the terminal.

[1206] Step 14:

[1207] The device converts the received summary data into speech using speech synthesis technology. It also converts text data into audio format and plays it through the speaker.

[1208] Step 15:

[1209] The user gives voice commands for specific actions. For example, they might say, "Copy this text and paste it into an email."

[1210] Step 16:

[1211] The terminal acquires audio data and sends it to the server. A package for converting the audio recording to text is generated.

[1212] Step 17:

[1213] The server analyzes the audio data and understands the instructions given. It converts the audio to text and then analyzes the content.

[1214] Step 18:

[1215] The emotion engine optimizes text editing operations based on the user's emotional state. For example, if the user is emotionally agitated, the operation steps will be simplified.

[1216] Step 19:

[1217] The server sends the edited content to the terminal. The edited content is packaged and transferred.

[1218] Step 20:

[1219] The device provides the user with the edited results via voice. It reads aloud, "The text has been pasted into the email. Do you want to confirm?"

[1220] Step 21:

[1221] The user requests a voice message to create an email. They instruct, "Create a new email. The subject line should be 'Work Report,' and I submitted a work report today..."

[1222] Step 22:

[1223] The terminal acquires audio data and sends it to the server. A package for analysis is generated from the audio recording.

[1224] Step 23:

[1225] The server analyzes the audio data and generates a subject and body. It generates text such as "Subject: Work Report" and "Body: Today I gave a work report..."

[1226] Step 24:

[1227] The generated email content is sent to the terminal. The subject and body data are packaged and forwarded.

[1228] Step 25:

[1229] The device reads the email content aloud and asks the user for confirmation. It instructs the user, "Do you want to send it? If yes, please say 'Send'."

[1230] Step 26:

[1231] The user gives a confirmation voice command, saying "Send."

[1232] Step 27:

[1233] The device acquires the confirmation audio and sends it to the server. The audio data for confirmation is sent to the server.

[1234] Step 28:

[1235] The server performs a final check and then sends the email. The email is sent to the specified recipient.

[1236] (Example 2)

[1237] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1238] Traditional systems primarily relied on voice commands for operation and information provision, enabling visually impaired individuals to efficiently operate devices and quickly and appropriately obtain information. However, these systems were insufficient in responding to the user's emotional state. As a result, users may experience frustration or be unable to obtain appropriate information. Therefore, there is a need for a system that can adjust device operation and information provision methods according to the user's emotional state, making it more efficient and reassuring to use.

[1239] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1240] In this invention, the server includes means for recognizing the user's emotions from voice data, means for acquiring screen information based on the emotion recognition results, and means including a generative artificial intelligence model for summarizing or editing the acquired screen information. This makes it possible to adjust the appropriate information provision method according to the user's emotional state, thereby improving convenience and peace of mind for visually impaired users when using the device.

[1241] A "voice command" refers to a voice command that a user makes to operate or instruct a device.

[1242] "Voice data" refers to data that represents acquired voice commands in digital format.

[1243] "Speech recognition" refers to the technology that analyzes speech data and converts it into text data.

[1244] "Emotion recognition" refers to a technology that analyzes and estimates a user's emotional state based on voice data.

[1245] "Screen information" refers to the visual information displayed on a device's screen.

[1246] Optical Character Recognition (OCR) refers to a technology that analyzes characters in an image and converts them into text data.

[1247] A "generative artificial intelligence model" refers to AI technology that analyzes input data and generates output tailored to a specific task.

[1248] A "summary" refers to a short, concise compilation of key information extracted from acquired text data.

[1249] "Editing" refers to the operation of changing or modifying the content of text data based on user instructions.

[1250] "Speech synthesis" refers to the technology that analyzes text data and converts it into human speech.

[1251] "Email" refers to a method of communication for creating and sending messages electronically.

[1252] Modes for carrying out the invention

[1253] This invention is a system for visually impaired individuals to operate a device using voice commands and efficiently acquire information, and it incorporates an emotion engine to recognize the user's emotions. Specific embodiments are described below.

[1254] Hardware and software to use

[1255] This system uses the following hardware and software:

[1256] Speech recognition software: For example, Google's speech recognition API

[1257] Emotion recognition software: e.g., IBM Watson Tone Analyzer

[1258] Optical Character Recognition (OCR) technology: e.g., Tesseract OCR

[1259] Generative artificial intelligence models: e.g., OpenAI's GPT-3

[1260] Speech synthesis technology: For example, IBM Watson Text to Speech

[1261] System configuration and operation

[1262] 1. Obtaining voice commands

[1263] The user issues a voice command to the device. For example, they might give a command such as, "Please summarize this page."

[1264] The device's microphone picks up this voice command, and it is processed as voice data by speech recognition software.

[1265] 2. Sending audio data

[1266] The device temporarily stores the voice data and sends it to the server via a secure protocol (e.g., HTTPS).

[1267] 3. Analysis of audio data

[1268] The server receives the audio data and converts it into text data using a speech recognition system.

[1269] Furthermore, the server recognizes the user's emotions from the voice data. It uses an emotion engine to analyze tone, speed, intonation, and other factors.

[1270] 4. Obtaining screen information

[1271] Based on the analyzed voice commands and emotion recognition results, the server instructs the terminal to retrieve screen information.

[1272] The device captures the currently displayed screen and converts it into text data using OCR technology.

[1273] 5. Generating a summary

[1274] The server receives text data sent from the terminal and analyzes it using a generative AI model. It then extracts important information and creates a summary.

[1275] The way information is delivered is adjusted based on the user's emotional state recognized by the emotion engine.

[1276] 6. Transmission of summary data and speech synthesis

[1277] The generated summary data is sent from the server to the terminal.

[1278] The device converts the summarized information into speech using speech synthesis technology and provides it to the user.

[1279] Specific example

[1280] For example, if a user issues a voice command such as "Please summarize this page," the device's microphone captures the voice, and a speech recognition API processes it into audio data. The server receives this audio data and uses an emotion engine to analyze the user's emotions. If the user is feeling anxious, the server generates a quick and concise summary, converts it into speech using speech synthesis technology, and provides it to the user through the device.

[1281] Example of a prompt

[1282] "Please describe a scenario in which a visually impaired person issues a voice command saying, 'Please summarize this page,' and the system, recognizing the user's impatience, quickly and concisely provides a summary."

[1283] In this way, this system provides flexible and rapid information tailored to the user's emotional state, creating an environment where visually impaired individuals can effectively use devices.

[1284] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1285] Step 1:

[1286] The user issues a voice command, specifically a request such as, "Please summarize this page." The device's microphone picks up the voice command and processes it as audio data using speech recognition software (e.g., Google's Speech Recognition API). The input is the user's voice command, and the output is audio data.

[1287] Step 2:

[1288] The terminal temporarily stores the audio data and sends it to the server via a secure protocol (e.g., HTTPS). Specifically, the binary data of the audio data is transmitted using a secure connection. The input is the audio data, and the output is the audio data sent to the server.

[1289] Step 3:

[1290] The server receives audio data and converts it into text data using a speech recognition system (e.g., Google Cloud Speech-to-Text). This process involves dividing the audio signal and mapping it to language units. The input is audio data, and the output is text data.

[1291] Step 4:

[1292] The server uses an emotion engine (e.g., IBM Watson Tone Analyzer) to recognize the user's emotions from voice data. It analyzes features such as tone, speed, and intonation of the voice to estimate emotional states such as anxiety or relaxation. The input is voice data and text data, and the output is emotional state data.

[1293] Step 5:

[1294] Based on the voice commands and emotion recognition results analyzed by the server, it instructs the terminal to retrieve screen information. Specifically, it sends instructions to the terminal via an HTTP request. The input is the voice command text and emotion state data, and the output is an instruction to retrieve screen information.

[1295] Step 6:

[1296] The device captures the information displayed on the screen and converts it into text data using optical character recognition (OCR) technology (e.g., Tesseract OCR). Alternatively, it can take a screenshot of the screen and extract text information from the image. The input is a screen image, and the output is text data.

[1297] Step 7:

[1298] The terminal sends the converted text data back to the server. Specifically, the text data is sent using a secure connection. The input is text data, and the output is the text data sent to the server.

[1299] Step 8:

[1300] The server analyzes the received text using a generative AI model (e.g., OpenAI's GPT-3), extracts important information, and creates a summary. The information delivery method is adjusted based on the user's emotional state recognized by the emotion engine. The input is text data and emotional state data, and the output is summary data.

[1301] Step 9:

[1302] The generated summary data is sent from the server to the terminal. A secure protocol is used to transmit the data. The input is the summary data, and the output is the summary data sent to the terminal.

[1303] Step 10:

[1304] The terminal converts summarized information into speech using speech synthesis technology (e.g., IBM Watson Text to Speech) and provides it to the user. The generated audio data is played back through the terminal's speaker. The input is summarized data, and the output is audio data.

[1305] Step 11:

[1306] The user gives a specific command (e.g., "Copy this text and paste it into an email"). The terminal acquires the voice data and sends it to the server. The input is the user's voice command, and the output is the voice data sent to the server.

[1307] Step 12:

[1308] The server analyzes the audio data and prepares the optimal text editing method, taking into account the user's emotional state. The prepared editing content is then sent to the terminal. Input consists of audio data and emotional state data, while output is editing instruction data.

[1309] Step 13:

[1310] The terminal transcribes or edits text to the appropriate location based on editing instruction data received from the server. The editing result is provided to the user via audio for user confirmation. The input is the editing instruction data, and the output is the edited text data and its audio data.

[1311] Step 14:

[1312] When a user instructs, "Create a new email. The subject line should be 'Work Report,' and I submitted my work report today...", the terminal acquires voice data and sends it to the server. The input is the user's voice command, and the output is the voice data sent to the server.

[1313] Step 15:

[1314] The server analyzes the audio data and generates a subject and body for the email. The generated email content is then sent to the terminal. The input consists of audio data and emotional state data, while the output consists of the email subject and body data.

[1315] Step 16:

[1316] The terminal reads the content aloud and prompts the user for confirmation. Once the user confirms "Send," the server executes the email sending operation. The input consists of the email subject, body data, and the user's send instruction, and the output is the sent email.

[1317] (Application Example 2)

[1318] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1319] For visually impaired individuals to find products appropriately and shop efficiently in physical stores, a voice-command navigation system is necessary. However, conventional voice command systems do not take user emotions into consideration, and at the same time, they do not adequately provide specific directions within stores or location information for products. As a result, shopping becomes difficult, especially for users who experience anxiety or stress.

[1320] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring a user's voice command, means for analyzing the acquired voice command, means for acquiring screen information based on the analyzed voice command, means including a generative artificial intelligence model for summarizing or editing the acquired screen information, means for providing the summarized or edited information to the user by voice, means including an emotion engine for analyzing emotions using the user's voice command and voice characteristics, and means for adjusting the information provision method based on the user's emotions. This makes it possible to provide navigation and product information quickly and concisely in a physical store while taking into account the user's emotional state.

[1321] "Means for acquiring user voice commands" refers to a microphone or voice input device that recognizes voice commands uttered by the user and inputs them into the device.

[1322] "Means for analyzing voice commands" refers to a voice recognition system that converts acquired voice commands into digital data, analyzes its content, and converts it into appropriate instructions.

[1323] "Means of acquiring screen information" refers to cameras and screen capture functions used to capture and analyze information displayed on a device.

[1324] A "generative artificial intelligence model" refers to an AI algorithm or system that extracts important information from large amounts of data and performs summarization or editing.

[1325] "Means of providing information to users in audio format" refers to speech synthesis technology and speakers used to convey analysis results and summaries to users as audio.

[1326] An "emotion engine" refers to software or a system that analyzes a user's voice characteristics and estimates their emotional state.

[1327] "Means for adjusting the method of information provision" refers to a function that dynamically changes the content and format of the information provided based on the user's emotional state.

[1328] "Means for transcribing or editing text" refers to software functions that copy digital text to the appropriate location and edit it based on the user's voice commands.

[1329] "Means for creating, reviewing, and sending emails" refers to systems and software functions that automatically generate emails based on user voice instructions, review their content, and then send them.

[1330] This invention is a system designed to assist visually impaired individuals in shopping at physical stores. The system includes a set of functions that allow the user, wearing smart glasses and using voice commands, to obtain information on product locations and store navigation. Furthermore, it reads the user's emotions from their voice and adjusts the assistance accordingly.

[1331] First, the user puts on smart glasses and makes a voice command such as "Tell me where the product is." The device's microphone captures this voice, and the Google Cloud Speech-to-Text API converts the voice into text data.

[1332] Next, along with the text data, characteristics such as tone, speed, and intonation of the voice are sent to the Azure Emotion API for emotion analysis. As a result of the analysis, a specific emotional state is recognized, such as when the user is anxious.

[1333] Screen information is acquired, for example, using the camera of smart glasses. The camera takes pictures of the store's map and product shelves, and converts them into text using Tesseract OCR technology. Based on this text data, the system works with the Google Maps API to determine the location within the physical store.

[1334] A specific prompt would be sent to the Generative AI model (GPT-4) with a request such as, "The Emotion API has detected that the user is feeling anxious. The user is currently looking for the location of a sticky note in the store. Please generate concise and easy-to-understand navigation instructions."

[1335] A generative AI model generates optimal feedback (for example, "Turn right and go straight down the next corridor. The sticky note is on the shelf to your right."), and this feedback is converted into speech by the Google Text-to-Speech API. The information is then delivered to the user through the smart glasses' speaker.

[1336] This system allows users to receive voice-based navigation and product location information tailored to their emotional state, enabling them to shop efficiently and comfortably in physical stores.

[1337] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1338] Step 1:

[1339] The user wears smart glasses and issues a voice command. Voice input is acquired through the smart glasses' microphone. The input voice data is the output of this step.

[1340] Step 2:

[1341] The device sends the acquired audio data to the Google Cloud Speech-to-Text API, which converts the audio into text data. The input is audio data, and the output is the converted text data.

[1342] Step 3:

[1343] The device sends text data along with characteristic data such as tone, speed, and intonation of speech to the Azure Emotion API. The input consists of text data and speech characteristic data, and the output is the result of the user's emotion analysis.

[1344] Step 4:

[1345] The server processes data based on the sentiment analysis results to determine a navigation method that corresponds to the user's emotions. The input here is the sentiment analysis results, and the output is navigation data corresponding to those emotions.

[1346] Step 5:

[1347] The device uses the camera on smart glasses to take pictures of the store interior and product shelves, and converts them into text data using Tesseract OCR. The input is the image taken by the camera, and the output is the converted text data.

[1348] Step 6:

[1349] The server uses OCR data in conjunction with the Google Maps API to identify location information within the store. The input is text data converted by OCR, and the output is the identified location information.

[1350] Step 7:

[1351] The device uses a generative AI model (GPT-4) to generate navigation instructions for the user based on sentiment analysis results and location information. The input is sentiment analysis results and location information, and the output is navigation instruction text.

[1352] Step 8:

[1353] The device sends the generated navigation instruction text to the Google Text-to-Speech API, where it is converted into speech. The input is navigation instruction text, and the output is audio data.

[1354] Step 9:

[1355] The device provides voice data to the user through the smart glasses' speaker. This allows the user to receive accurate directions and product information within the store via voice. The input is voice data, and the output is voice guidance.

[1356] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1357] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1358] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[1359] [Fourth Embodiment]

[1360] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[1361] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1362] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1363] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[1364] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1365] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1366] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1367] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[1368] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1369] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1370] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1371] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1372] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1373] This invention provides a system that allows visually impaired individuals to operate a device using voice commands and efficiently acquire information. The program processing of this system is described below.

[1374] First, the user issues a voice command. For example, the user might say, "Please summarize this page." This voice command is picked up by the device's microphone and processed as voice data by the device's internal voice recognition software. The voice data is then sent to a server for analysis. The server uses its voice recognition system to convert the voice data into text data and understands the command "Please summarize."

[1375] Next, the server sends instructions to the terminal based on the analyzed command. In this case, it specifies "summarize screen information." The terminal retrieves information from the currently displayed screen. For example, it takes a screen capture and extracts text information using optical character recognition (OCR) technology. The extracted text data is then sent back to the server for summarization.

[1376] The server analyzes the received text data using a generation AI model, extracts key information, and creates a summary. The generated summary data is sent to the terminal, which then reads the summarized information aloud using speech synthesis. This allows the user to easily grasp the main information on the screen.

[1377] Furthermore, if the user gives specific instructions, such as "Copy this sentence and paste it into an email," they can issue a voice command. The terminal retrieves the voice data again and sends it to the server. The server analyzes the voice data and prepares text editing operations based on the instructions. The server sends the edited content to the terminal, and the terminal copies and edits the text in the appropriate place. The editing results are provided to the user via voice, allowing the user to confirm them.

[1378] When a user composes an email, they might be instructed, for example, "Create a new email. The subject should be 'Work Report,' and I submitted my work report today..." The device receives the voice data and sends it to the server. The server analyzes the voice data and generates the subject "Work Report" and the body "I submitted my work report today...". The generated email content is sent to the device, which reads the email content aloud and asks the user for confirmation. When the user gives a confirmation voice such as "Send," the server executes the email sending operation.

[1379] Thus, a highly functional assistance system that allows for a smooth series of operations, from acquiring voice commands to summarizing information, editing text, and creating and sending emails, enables visually impaired individuals to make better use of their devices, greatly improving their quality of life and work efficiency.

[1380] The following describes the processing flow.

[1381] Step 1:

[1382] The user issues a voice command, such as dictating, "Please summarize this page."

[1383] Step 2:

[1384] The device acquires voice commands via the microphone. The voice is recorded and captured as audio data.

[1385] Step 3:

[1386] The device temporarily stores the audio data and processes it using speech recognition software. The audio data is then converted into text data.

[1387] Step 4:

[1388] The device converts the audio data into text data and then sends it to the server. The data is sent to the server via the network.

[1389] Step 5:

[1390] The server uses a speech recognition system to analyze the voice data and generate text data. It analyzes the voice command and understands the meaning of "Please summarize."

[1391] Step 6:

[1392] Based on the analysis performed by the server, it instructs the terminal to retrieve screen information. The instructions are then packaged and sent to the terminal.

[1393] Step 7:

[1394] The device retrieves information from the currently displayed screen. It takes a screen capture and retrieves the text information from the screen.

[1395] Step 8:

[1396] The device converts screen information into text data using optical character recognition (OCR) technology. It extracts characters from image data.

[1397] Step 9:

[1398] The terminal extracts text data and sends it to the server. The converted text data is packaged and sent to the server.

[1399] Step 10:

[1400] The server analyzes the received text data using a generation AI model. It extracts key information from the text data to summarize its main points.

[1401] Step 11:

[1402] The server generates summary data and sends it to the terminal. The summary results are packaged and sent to the terminal.

[1403] Step 12:

[1404] The device receives summary data, converts it into speech using speech synthesis technology, and provides it to the user. It also converts text data into speech format and plays it through the speaker.

[1405] Specific example 1:

[1406] User: "Could you please summarize this page?"

[1407] Terminal: Receives voice commands and sends them to the server for analysis.

[1408] Server: Analyzes the audio and sends a summary of the screen information back to the terminal.

[1409] Terminal: Captures screen information, converts it to text using OCR, and sends it to the server.

[1410] Server: Summarizes the received text and sends it to the terminal.

[1411] Terminal: Provides a summary to the user in audio format.

[1412] Specific example 2:

[1413] User: "Please create a new email. The subject line should be 'Work Report,' and I submitted my work report today..."

[1414] Terminal: Receives voice commands and sends them to the server for analysis.

[1415] Server: Generates subject and body text and sends it to the terminal.

[1416] Terminal: Reads the email content aloud and asks the user for confirmation.

[1417] User: "Send"

[1418] Terminal: Retrieves confirmation audio and sends it to the server.

[1419] Server: Sending email.

[1420] In this way, a system is realized that allows visually impaired people to operate devices and acquire information more efficiently.

[1421] (Example 1)

[1422] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1423] Visually impaired individuals face challenges in operating devices and efficiently obtaining information. Furthermore, complex operations such as retrieving, summarizing, editing, and composing emails are difficult to perform smoothly using voice commands. To address these issues, a system is needed that analyzes voice commands and responds quickly and accurately.

[1424] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1425] In this invention, the server includes means for acquiring user voice commands, means for analyzing the acquired voice commands, and means for acquiring screen information based on the analyzed voice commands. This enables visually impaired individuals to operate a device using voice commands, efficiently acquire screen information, and compose and send emails.

[1426] "User" refers to the person who operates the system in question.

[1427] "Voice commands" refer to instructions given by the user using their voice.

[1428] A "terminal" refers to an electronic device that has functions such as acquiring voice commands, converting voice data, and acquiring screen information.

[1429] A "server" refers to a central processing unit that performs tasks such as analyzing voice data, understanding commands, using generative AI models, and sending and receiving data.

[1430] "Voice data" refers to the digital data of a user's voice commands.

[1431] "Voice recognition software" refers to a program that converts acquired voice commands into text data.

[1432] "Text data" refers to character information converted by speech recognition software.

[1433] "Screen information" refers to all information displayed on the device, such as text and images.

[1434] "Optical character recognition technology" refers to the technology that recognizes characters from images and extracts them as text data.

[1435] A "generative AI model" refers to an artificial intelligence model that analyzes received data and performs tasks such as summarizing and editing.

[1436] A "summary" is information that extracts the important parts from screen information and presents them concisely.

[1437] "Editing" refers to the process of modifying or rewriting acquired text data based on user instructions.

[1438] "Speech synthesis" refers to a technology that converts text data into speech and provides it to the user.

[1439] This invention is a system for visually impaired individuals to operate a device using voice commands and efficiently acquire information. The following describes specific embodiments of this system.

[1440] Hardware and software usage

[1441] terminal

[1442] The device includes a microphone, speech recognition software, and optical character recognition (OCR) technology. Speech recognition software such as Google Speech-to-Text API is used. Optical character recognition technology such as Tesseract OCR is used.

[1443] server

[1444] The server includes functions for analyzing voice data, understanding commands, using generative AI models, and sending and receiving data. DeepSpeech is used for the speech recognition system, and GPT-3 is used for the generative AI model.

[1445] Operation Description

[1446] Acquisition and recognition of voice commands

[1447] 1. The user issues a voice command. For example, they might say, "Please summarize this page."

[1448] 2. The device's microphone acquires voice commands, which are then converted into voice data by speech recognition software.

[1449] 3. The converted audio data is sent to the server.

[1450] Audio data analysis and command understanding

[1451] 1. The server receives the audio data and converts it into text data using a speech recognition system such as DeepSpeech.

[1452] 2. The server understands the command from the text data and interprets the instruction, "Please summarize this page."

[1453] Acquiring screen information and extracting text

[1454] 1. The server sends an instruction to the terminal called "Summarize screen information".

[1455] 2. Take a screenshot of the screen currently displayed on the device.

[1456] 3. Extract text data from the screen capture using Tesseract OCR or similar software, and send it back to the server.

[1457] Text data summary generation

[1458] 1. The server analyzes the received text data using a generative AI model such as GPT-3, extracts important information, and generates a summary.

[1459] 2. The generated summary data is sent to the terminal.

[1460] Summary audio output

[1461] 1. The device generates a summary of the received information using speech synthesis (such as the Google Text-to-Speech API) and provides it to the user in audio format.

[1462] Specific examples and prompt statements

[1463] Specific example: Summary of page information

[1464] When a user issues a voice command such as "Please summarize this page," the device's microphone captures the voice, which is then converted into digital data by speech recognition software. The converted voice data is sent to a server, where DeepSpeech converts it back into text, and GPT-3 extracts key information to generate a summary. The generated summary data is then sent to the device, and speech synthesis software reads the summary aloud to the user.

[1465] Specific example: Creating and sending an email

[1466] When a user instructs, "Create a new email. The subject line should be 'Work Report,' and I submitted a work report today," the device's microphone captures the voice, which is then converted into digital data by speech recognition software. The converted voice data is sent to a server, which analyzes the voice data and generates a subject line and body. The generated email content is sent to the device, and the user is asked to confirm it as a voice message. Once the user confirms "Send," the server executes the email sending operation.

[1467] Thus, this system allows for a smooth series of operations, from acquiring voice commands to summarizing information, editing text, and creating and sending emails. This system enables visually impaired individuals to efficiently utilize devices and improve their quality of life and work efficiency.

[1468] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1469] Step 1:

[1470] Voice command generation and acquisition:

[1471] The user speaks a voice command into the device's microphone. For example, they might say, "Please summarize this page."

[1472] Input: User's voice command

[1473] Output: Audio data acquired as an audio signal by the device's microphone.

[1474] Specific actions:

[1475] When a user speaks into the microphone, their voice is converted into a digital signal by the microphone.

[1476] Step 2:

[1477] Speech recognition and data conversion:

[1478] The device uses built-in speech recognition software to convert speech data into text data.

[1479] Input: Audio data (audio signal)

[1480] Output: Text data (content of voice command)

[1481] Specific actions:

[1482] The device's Google Speech-to-Text API analyzes the audio signal and converts it into text data that says, "Please summarize this page."

[1483] Step 3:

[1484] Sending audio data:

[1485] The terminal sends the converted text data to the server.

[1486] Input: Text data (content of voice command)

[1487] Output: Text data sent to the server

[1488] Specific actions:

[1489] The terminal's communication module sends text data based on voice commands to the server.

[1490] Step 4:

[1491] Text data analysis and command understanding:

[1492] The server analyzes the received text data to understand the content of the voice command.

[1493] Input: Received text data

[1494] Output: Analysis results (understanding of command content)

[1495] Specific actions:

[1496] The server's speech recognition system, such as DeepSpeech, analyzes the text data and understands the command "Please summarize."

[1497] Step 5:

[1498] Sending instructions based on commands:

[1499] In response to the command, the server sends an instruction to the terminal requesting a "summary of screen information."

[1500] Input: Analysis results (command: "Please provide a summary")

[1501] Output: Signal containing the instruction content (summary of screen information)

[1502] Specific actions:

[1503] The server sends a signal to the terminal instructing it to retrieve and summarize the screen information.

[1504] Step 6:

[1505] Retrieving screen information:

[1506] Capture a screenshot of the screen currently displayed on the device.

[1507] Input: Instructions from the server (summary of screen information)

[1508] Output: Screen capture image

[1509] Specific actions:

[1510] The device uses its screen capture function to obtain the current screen content as an image.

[1511] Step 7:

[1512] Optical character recognition (OCR):

[1513] The device extracts text data from the acquired screen capture image using optical character recognition technology.

[1514] Input: Screen capture image

[1515] Output: Extracted text data

[1516] Specific actions:

[1517] The device's Tesseract OCR analyzes the captured image and extracts the text information displayed on the screen.

[1518] Step 8:

[1519] Sending text data:

[1520] The terminal then sends the extracted text data back to the server.

[1521] Input: Extracted text data

[1522] Output: Text data sent to the server

[1523] Specific actions:

[1524] The extracted text data is sent to the server via the terminal's communication module.

[1525] Step 9:

[1526] Text data summary generation:

[1527] The server analyzes the received text data using a generating AI model (e.g., GPT-3), extracts important information, and generates a summary.

[1528] Input: Received text data

[1529] Output: Generated summary text

[1530] Specific actions:

[1531] A GPT-3 generation AI model on the server analyzes the text data and generates a summary.

[1532] Step 10:

[1533] Send summary data:

[1534] The server sends the generated summary data to the terminal.

[1535] Input: Generated summary text

[1536] Output: Summary text sent to the terminal

[1537] Specific actions:

[1538] The server sends a signal containing summary data to the terminal.

[1539] Step 11:

[1540] Summary audio output:

[1541] The terminal provides the user with the received summary data in voice using speech synthesis technology.

[1542] Input: Received summary text

[1543] Output: Summary provided in audio

[1544] Specific actions:

[1545] The device's Google Text-to-Speech API converts the text into speech and reads it aloud to the user.

[1546] (Application Example 1)

[1547] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1548] Visually impaired individuals face difficulties in conducting electronic transactions and making payments due to the need to understand screen information and edit text, hindering their ability to easily perform daily activities and work. This problem is particularly pronounced in transaction confirmation procedures and payment processes, causing significant stress for visually impaired individuals. Therefore, there is a need for systems that support visually impaired individuals in conducting electronic transactions quickly and accurately using voice commands.

[1549] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1550] In this invention, the server includes means for analyzing the user's voice commands, means for acquiring screen information or electronic transaction information based on the analyzed voice commands, and means including a generative artificial intelligence model for summarizing or editing the acquired information. This makes it possible for visually impaired individuals to confirm the contents of electronic transactions using voice commands and to perform all necessary editing and operations by voice.

[1551] A "voice command" is a voice input that includes instructions or requests that a user makes to a device.

[1552] "Means of analysis" refers to a device or software that converts acquired voice commands into text data and performs the process of understanding and interpreting its content.

[1553] "Screen information" refers to visual information such as text, images, and icons displayed on the device screen.

[1554] "Electronic transaction information" refers to information regarding the purchase, sale, and payment of goods and services conducted in digital format.

[1555] A "generative artificial intelligence model" is an artificial intelligence system that learns from large amounts of data and can automatically perform tasks such as summarizing, editing, and generating text.

[1556] "Means of providing information to the user in audio" refers to a device or software that performs the process of audibly communicating generated or acquired information to the user using speech synthesis technology.

[1557] "Transaction details" refers to detailed information about a specific electronic transaction, such as the goods, services, price, quantity, and trading partner.

[1558] "Transcribing or editing means" refers to any device or software that performs the process of copying, editing, or modifying text or transaction information to another location based on user instructions.

[1559] "Means for creating, reviewing, and sending emails" refers to a device or software that performs the process of creating email content according to the user's voice instructions, having the user review the content, and sending the email after approval.

[1560] "Means for executing a transaction" refers to a device or software that executes a process to complete electronic transaction procedures based on the user's voice instructions.

[1561] To implement this invention, a system will be constructed that allows visually impaired individuals to easily conduct electronic transactions using voice commands. The details of this system are described below.

[1562] First, the user issues a voice command using their smartphone's microphone. This voice command might be something like "Start payment." This voice command is processed as audio data by speech recognition software (e.g., Google Cloud Speech-to-Text) inside the device. The audio data is sent to a server for analysis. The server uses its speech recognition system to convert the audio data into text data and understands the content of the command (in this case, "Start payment").

[1563] Next, the server instructs the terminal to retrieve the current screen information based on the voice command. The terminal uses OCR technology (e.g., Tesseract OCR) to analyze the screen information and extract the data necessary for the transaction (product name, price, etc.). The extracted information is sent back to the server, which uses a generative artificial intelligence model (e.g., OpenAI GPT-4) to analyze the information and generate the data necessary for the payment process.

[1564] The generated payment information is provided to the user audibly using speech synthesis software (e.g., Google Text-to-Speech). For example, it might ask for confirmation by voice, such as, "The payment amount for product name 'ABC' is 3000 yen. Is that correct?"

[1565] When the user issues the voice command "confirm," the terminal retrieves the voice data again and sends it to the server. The server analyzes the voice data and begins the payment process. After the transaction is complete, the processing result is sent to the terminal and the user is notified by voice, such as "Payment completed."

[1566] This system enables visually impaired individuals to conduct electronic transactions quickly and accurately using only voice commands, significantly improving their convenience in daily life. A concrete example of this system involves a user giving a voice command such as "Start payment," followed by a series of processes where the server analyzes the data, retrieves information, and synthesizes the speech. The following is an example of a prompt.

[1567] As a concrete example, the following prompt is input to the generating AI model:

[1568] "Start payment"

[1569] The user says "Start payment." The app uses voice recognition to convert the command into text and sends it to the server.

[1570] The server analyzes the text, understands the "start payment" command, and uses OCR to retrieve information from the screen.

[1571] Analyze the acquired information,

[1572] Prompt example: "Product name: "XYZ", Price: 5000 yen, Please confirm payment method."

[1573] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1574] Step 1:

[1575] The user issues a voice command such as "Start payment." This voice command is captured by the device's microphone.

[1576] Input: User's voice command

[1577] Output: Audio data

[1578] Action: The user says "Start payment" to their smartphone.

[1579] Step 2:

[1580] The speech recognition software inside the device (e.g., Google Cloud Speech-to-Text) converts the speech data into text data.

[1581] Input: Audio data

[1582] Output: Text data

[1583] Operation: Speech recognition software analyzes the speech and generates the text data "Start payment".

[1584] Step 3:

[1585] Text data is sent to the server. The server analyzes the received text data using a speech recognition system and understands the command.

[1586] Input: Text data

[1587] Output: Command content

[1588] Operation: The server recognizes the "start payment" command and determines the next step.

[1589] Step 4:

[1590] The server instructs the terminal to acquire screen information and electronic transaction information. The terminal uses OCR technology (e.g., Tesseract OCR) to analyze the screen information and extract the transaction information.

[1591] Input: Instructions from the server

[1592] Output: Transaction information (text format)

[1593] Operation: The device takes a screenshot of the current screen and uses OCR technology to extract product names and prices as text data.

[1594] Step 5:

[1595] The extracted transaction information is sent to the server. The server uses a generative artificial intelligence model (e.g., OpenAI GPT-4) to analyze the information and generate the data necessary for the payment process.

[1596] Input: Transaction information (text format)

[1597] Output: Payment procedure data

[1598] Operation: The server inputs transaction information into an AI model, which then generates summaries and confirmation messages regarding the payment process.

[1599] Step 6:

[1600] The generated payment processing data is provided to the user via voice using speech synthesis software (e.g., Google Text-to-Speech).

[1601] Input: Payment procedure data

[1602] Output: Voice message

[1603] Operation: The device's speaker will voice-over, "The payment amount for product name 'ABC' is 3000 yen. Is that correct?" to request confirmation.

[1604] Step 7:

[1605] The user issues the voice command "Confirm." The terminal retrieves the voice data again and sends it to the server. The server analyzes the voice data and executes the payment process.

[1606] Input: User's voice command

[1607] Output: Instructions to execute payment procedure

[1608] Operation: Speech recognition software generates the text data "confirm," which the server then analyzes to initiate the payment process.

[1609] Step 8:

[1610] Once the payment process is complete, the processing result is sent from the server to the terminal. The terminal then notifies the user of the processing result via voice.

[1611] Input: Payment procedure result data

[1612] Output: Audio notification of processing results

[1613] Action: The device's speaker will notify the user via voice message, "Payment complete."

[1614] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1615] This invention combines a system for visually impaired individuals to operate a device using voice commands and efficiently acquire information with an emotion engine for recognizing the user's emotions. The program processing of this system is described below.

[1616] First, the user issues a voice command, such as "Please summarize this page." This voice command is captured by the device's microphone and processed as audio data by the device's internal speech recognition software. The device temporarily stores this audio data and sends it to a server for analysis.

[1617] The server uses a speech recognition system to convert voice data into text data, and then uses an emotion engine to recognize the user's emotions from the voice data. At this time, the server analyzes not only voice commands but also features such as the tone, speed, and intonation of the user's voice to estimate the user's emotional state. For example, if the user is feeling anxious, the emotion engine will recognize this.

[1618] Next, the server instructs the terminal to retrieve screen information based on the analysis. The terminal retrieves the information from the currently displayed screen and converts it into text data using optical character recognition (OCR) technology. This text data is also sent back to the server.

[1619] The server analyzes the received text data using an artificial intelligence model, extracts important information, and creates a summary. Based on the user's emotional state recognized by the emotion engine, the way information is presented is adjusted. For example, if the user is feeling anxious, the summary is provided in a concise and easily understandable format.

[1620] The generated summary data is sent to the device, which then uses speech synthesis technology to convert the summarized information into speech and provides it to the user. This process allows the user to quickly and accurately grasp the information they need.

[1621] When a user gives instructions for a specific action, they might issue a voice command such as, "Copy this sentence and paste it into an email." The terminal receives the voice data and sends it to the server. The server analyzes the voice data and, taking into account the user's emotional state, prepares the text editing operation in the most optimal way. The server sends the edited content to the terminal, which then copies and edits the text in the appropriate location. The editing result is provided to the user via voice for confirmation.

[1622] When composing an email, the user might instruct, "Create a new email. The subject should be 'Work Report,' and today I submitted a work report..." In this case, the device also acquires voice data and sends it to the server. The server analyzes the voice data and generates the subject and body of the email. The generated email content is sent to the device, which reads the content aloud and asks the user for confirmation. When the user gives a confirmation voice such as "Send," the server executes the email sending operation.

[1623] In this way, by analyzing both voice commands and user emotions, we provide a highly functional support system that enables visually impaired individuals to efficiently operate devices and acquire information. For example, when a user requests a summary, if the emotion engine recognizes the user's impatience, the system adjusts to provide a quick and concise summary. Also, when composing an email, the system can prevent accidental sending by repeatedly confirming the content according to the user's emotions. This significantly improves the quality of life and work efficiency of visually impaired individuals.

[1624] The following describes the processing flow.

[1625] Step 1:

[1626] The user issues a voice command, such as dictating, "Please summarize this page."

[1627] Step 2:

[1628] The device acquires voice commands via the microphone. The voice is recorded and captured as audio data.

[1629] Step 3:

[1630] The device temporarily stores the audio data and processes it using speech recognition software. The audio data is then converted into text data.

[1631] Step 4:

[1632] The terminal sends the converted text data to the server. The data is sent to the server via the network.

[1633] Step 5:

[1634] The server uses a speech recognition system to analyze text data. It understands the voice command "Please summarize."

[1635] Step 6:

[1636] An emotion engine built into the server recognizes the user's emotions from the voice data. It analyzes the tone, speed, and intonation of the voice to estimate the user's emotional state (e.g., anxiety, joy, tension, etc.).

[1637] Step 7:

[1638] Based on the server's analysis and emotion recognition results, it instructs the terminal to acquire screen information. The instructions are then packaged and sent to the terminal.

[1639] Step 8:

[1640] The device retrieves information from the currently displayed screen. A screen capture is performed to obtain text information from the screen.

[1641] Step 9:

[1642] The device converts screen information it acquires into text data using optical character recognition (OCR) technology. It extracts characters from image data.

[1643] Step 10:

[1644] The terminal sends the converted text data to the server.

[1645] Step 11:

[1646] The server receives text data, which is then analyzed by an artificial intelligence model. Important information from the text data is extracted, and a summary is created.

[1647] Step 12:

[1648] The emotion engine adjusts how summaries are delivered based on the user's emotional state. For example, if the user is feeling anxious, it will provide a concise and easily understandable summary.

[1649] Step 13:

[1650] The server generates summary data and sends it to the terminal. The summary results are packaged and sent to the terminal.

[1651] Step 14:

[1652] The device converts the received summary data into speech using speech synthesis technology. It also converts text data into audio format and plays it through the speaker.

[1653] Step 15:

[1654] The user gives voice commands for specific actions. For example, they might say, "Copy this text and paste it into an email."

[1655] Step 16:

[1656] The terminal acquires audio data and sends it to the server. A package for converting the audio recording to text is generated.

[1657] Step 17:

[1658] The server analyzes the audio data and understands the instructions given. It converts the audio to text and then analyzes the content.

[1659] Step 18:

[1660] The emotion engine optimizes text editing operations based on the user's emotional state. For example, if the user is emotionally agitated, the operation steps will be simplified.

[1661] Step 19:

[1662] The server sends the edited content to the terminal. The edited content is packaged and transferred.

[1663] Step 20:

[1664] The device provides the user with the edited results via voice. It reads aloud, "The text has been pasted into the email. Do you want to confirm?"

[1665] Step 21:

[1666] The user requests a voice message to create an email. They instruct, "Create a new email. The subject line should be 'Work Report,' and I submitted a work report today..."

[1667] Step 22:

[1668] The terminal acquires audio data and sends it to the server. A package for analysis is generated from the audio recording.

[1669] Step 23:

[1670] The server analyzes the audio data and generates a subject and body. It generates text such as "Subject: Work Report" and "Body: Today I gave a work report..."

[1671] Step 24:

[1672] The generated email content is sent to the terminal. The subject and body data are packaged and forwarded.

[1673] Step 25:

[1674] The device reads the email content aloud and asks the user for confirmation. It instructs the user, "Do you want to send it? If yes, please say 'Send'."

[1675] Step 26:

[1676] The user gives a confirmation voice command, saying "Send."

[1677] Step 27:

[1678] The device acquires the confirmation audio and sends it to the server. The audio data for confirmation is sent to the server.

[1679] Step 28:

[1680] The server performs a final check and then sends the email. The email is sent to the specified recipient.

[1681] (Example 2)

[1682] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1683] Traditional systems primarily relied on voice commands for operation and information provision, enabling visually impaired individuals to efficiently operate devices and quickly and appropriately obtain information. However, these systems were insufficient in responding to the user's emotional state. As a result, users may experience frustration or be unable to obtain appropriate information. Therefore, there is a need for a system that can adjust device operation and information provision methods according to the user's emotional state, making it more efficient and reassuring to use.

[1684] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1685] In this invention, the server includes means for recognizing the user's emotions from voice data, means for acquiring screen information based on the emotion recognition results, and means including a generative artificial intelligence model for summarizing or editing the acquired screen information. This makes it possible to adjust the appropriate information provision method according to the user's emotional state, thereby improving convenience and peace of mind for visually impaired users when using the device.

[1686] A "voice command" refers to a voice command that a user makes to operate or instruct a device.

[1687] "Voice data" refers to data that represents acquired voice commands in digital format.

[1688] "Speech recognition" refers to the technology that analyzes speech data and converts it into text data.

[1689] "Emotion recognition" refers to a technology that analyzes and estimates a user's emotional state based on voice data.

[1690] "Screen information" refers to the visual information displayed on a device's screen.

[1691] Optical Character Recognition (OCR) refers to a technology that analyzes characters in an image and converts them into text data.

[1692] A "generative artificial intelligence model" refers to AI technology that analyzes input data and generates output tailored to a specific task.

[1693] A "summary" refers to a short, concise compilation of key information extracted from acquired text data.

[1694] "Editing" refers to the operation of changing or modifying the content of text data based on user instructions.

[1695] "Speech synthesis" refers to the technology that analyzes text data and converts it into human speech.

[1696] "Email" refers to a method of communication for creating and sending messages electronically.

[1697] Modes for carrying out the invention

[1698] This invention is a system for visually impaired individuals to operate a device using voice commands and efficiently acquire information, and it incorporates an emotion engine to recognize the user's emotions. Specific embodiments are described below.

[1699] Hardware and software to use

[1700] This system uses the following hardware and software:

[1701] Speech recognition software: For example, Google's speech recognition API

[1702] Emotion recognition software: e.g., IBM Watson Tone Analyzer

[1703] Optical Character Recognition (OCR) technology: e.g., Tesseract OCR

[1704] Generative artificial intelligence models: e.g., OpenAI's GPT-3

[1705] Speech synthesis technology: For example, IBM Watson Text to Speech

[1706] System configuration and operation

[1707] 1. Obtaining voice commands

[1708] The user issues a voice command to the device. For example, they might give a command such as, "Please summarize this page."

[1709] The device's microphone picks up this voice command, and it is processed as voice data by speech recognition software.

[1710] 2. Sending audio data

[1711] The device temporarily stores the voice data and sends it to the server via a secure protocol (e.g., HTTPS).

[1712] 3. Analysis of audio data

[1713] The server receives the audio data and converts it into text data using a speech recognition system.

[1714] Furthermore, the server recognizes the user's emotions from the voice data. It uses an emotion engine to analyze tone, speed, intonation, and other factors.

[1715] 4. Obtaining screen information

[1716] Based on the analyzed voice commands and emotion recognition results, the server instructs the terminal to retrieve screen information.

[1717] The device captures the currently displayed screen and converts it into text data using OCR technology.

[1718] 5. Generating a summary

[1719] The server receives text data sent from the terminal and analyzes it using a generative AI model. It then extracts important information and creates a summary.

[1720] The way information is delivered is adjusted based on the user's emotional state recognized by the emotion engine.

[1721] 6. Transmission of summary data and speech synthesis

[1722] The generated summary data is sent from the server to the terminal.

[1723] The device converts the summarized information into speech using speech synthesis technology and provides it to the user.

[1724] Specific example

[1725] For example, if a user issues a voice command such as "Please summarize this page," the device's microphone captures the voice, and a speech recognition API processes it into audio data. The server receives this audio data and uses an emotion engine to analyze the user's emotions. If the user is feeling anxious, the server generates a quick and concise summary, converts it into speech using speech synthesis technology, and provides it to the user through the device.

[1726] Example of a prompt

[1727] "Please describe a scenario in which a visually impaired person issues a voice command saying, 'Please summarize this page,' and the system, recognizing the user's impatience, quickly and concisely provides a summary."

[1728] In this way, this system provides flexible and rapid information tailored to the user's emotional state, creating an environment where visually impaired individuals can effectively use devices.

[1729] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1730] Step 1:

[1731] The user issues a voice command, specifically a request such as, "Please summarize this page." The device's microphone picks up the voice command and processes it as audio data using speech recognition software (e.g., Google's Speech Recognition API). The input is the user's voice command, and the output is audio data.

[1732] Step 2:

[1733] The terminal temporarily stores the audio data and sends it to the server via a secure protocol (e.g., HTTPS). Specifically, the binary data of the audio data is transmitted using a secure connection. The input is the audio data, and the output is the audio data sent to the server.

[1734] Step 3:

[1735] The server receives audio data and converts it into text data using a speech recognition system (e.g., Google Cloud Speech-to-Text). This process involves dividing the audio signal and mapping it to language units. The input is audio data, and the output is text data.

[1736] Step 4:

[1737] The server uses an emotion engine (e.g., IBM Watson Tone Analyzer) to recognize the user's emotions from voice data. It analyzes features such as tone, speed, and intonation of the voice to estimate emotional states such as anxiety or relaxation. The input is voice data and text data, and the output is emotional state data.

[1738] Step 5:

[1739] Based on the voice commands and emotion recognition results analyzed by the server, it instructs the terminal to retrieve screen information. Specifically, it sends instructions to the terminal via an HTTP request. The input is the voice command text and emotion state data, and the output is an instruction to retrieve screen information.

[1740] Step 6:

[1741] The device captures the information displayed on the screen and converts it into text data using optical character recognition (OCR) technology (e.g., Tesseract OCR). Alternatively, it can take a screenshot of the screen and extract text information from the image. The input is a screen image, and the output is text data.

[1742] Step 7:

[1743] The terminal sends the converted text data back to the server. Specifically, the text data is sent using a secure connection. The input is text data, and the output is the text data sent to the server.

[1744] Step 8:

[1745] The server analyzes the received text using a generative AI model (e.g., OpenAI's GPT-3), extracts important information, and creates a summary. The information delivery method is adjusted based on the user's emotional state recognized by the emotion engine. The input is text data and emotional state data, and the output is summary data.

[1746] Step 9:

[1747] The generated summary data is sent from the server to the terminal. A secure protocol is used to transmit the data. The input is the summary data, and the output is the summary data sent to the terminal.

[1748] Step 10:

[1749] The terminal converts summarized information into speech using speech synthesis technology (e.g., IBM Watson Text to Speech) and provides it to the user. The generated audio data is played back through the terminal's speaker. The input is summarized data, and the output is audio data.

[1750] Step 11:

[1751] The user gives a specific command (e.g., "Copy this text and paste it into an email"). The terminal acquires the voice data and sends it to the server. The input is the user's voice command, and the output is the voice data sent to the server.

[1752] Step 12:

[1753] The server analyzes the audio data and prepares the optimal text editing method, taking into account the user's emotional state. The prepared editing content is then sent to the terminal. Input consists of audio data and emotional state data, while output is editing instruction data.

[1754] Step 13:

[1755] The terminal transcribes or edits text to the appropriate location based on editing instruction data received from the server. The editing result is provided to the user via audio for user confirmation. The input is the editing instruction data, and the output is the edited text data and its audio data.

[1756] Step 14:

[1757] When a user instructs, "Create a new email. The subject line should be 'Work Report,' and I submitted my work report today...", the terminal acquires voice data and sends it to the server. The input is the user's voice command, and the output is the voice data sent to the server.

[1758] Step 15:

[1759] The server analyzes the audio data and generates a subject and body for the email. The generated email content is then sent to the terminal. The input consists of audio data and emotional state data, while the output consists of the email subject and body data.

[1760] Step 16:

[1761] The terminal reads the content aloud and prompts the user for confirmation. Once the user confirms "Send," the server executes the email sending operation. The input consists of the email subject, body data, and the user's send instruction, and the output is the sent email.

[1762] (Application Example 2)

[1763] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1764] For visually impaired individuals to find products appropriately and shop efficiently in physical stores, a voice-command navigation system is necessary. However, conventional voice command systems do not take user emotions into consideration, and at the same time, they do not adequately provide specific directions within stores or location information for products. As a result, shopping becomes difficult, especially for users who experience anxiety or stress.

[1765] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring a user's voice command, means for analyzing the acquired voice command, means for acquiring screen information based on the analyzed voice command, means including a generative artificial intelligence model for summarizing or editing the acquired screen information, means for providing the summarized or edited information to the user by voice, means including an emotion engine for analyzing emotions using the user's voice command and voice characteristics, and means for adjusting the information provision method based on the user's emotions. This makes it possible to provide navigation and product information quickly and concisely in a physical store while taking into account the user's emotional state.

[1766] "Means for acquiring user voice commands" refers to a microphone or voice input device that recognizes voice commands uttered by the user and inputs them into the device.

[1767] "Means for analyzing voice commands" refers to a voice recognition system that converts acquired voice commands into digital data, analyzes its content, and converts it into appropriate instructions.

[1768] "Means of acquiring screen information" refers to cameras and screen capture functions used to capture and analyze information displayed on a device.

[1769] A "generative artificial intelligence model" refers to an AI algorithm or system that extracts important information from large amounts of data and performs summarization or editing.

[1770] "Means of providing information to users in audio format" refers to speech synthesis technology and speakers used to convey analysis results and summaries to users as audio.

[1771] An "emotion engine" refers to software or a system that analyzes a user's voice characteristics and estimates their emotional state.

[1772] "Means for adjusting the method of information provision" refers to a function that dynamically changes the content and format of the information provided based on the user's emotional state.

[1773] "Means for transcribing or editing text" refers to software functions that copy digital text to the appropriate location and edit it based on the user's voice commands.

[1774] "Means for creating, reviewing, and sending emails" refers to systems and software functions that automatically generate emails based on user voice instructions, review their content, and then send them.

[1775] This invention is a system designed to assist visually impaired individuals in shopping at physical stores. The system includes a set of functions that allow the user, wearing smart glasses and using voice commands, to obtain information on product locations and store navigation. Furthermore, it reads the user's emotions from their voice and adjusts the assistance accordingly.

[1776] First, the user puts on smart glasses and makes a voice command such as "Tell me where the product is." The device's microphone captures this voice, and the Google Cloud Speech-to-Text API converts the voice into text data.

[1777] Next, along with the text data, characteristics such as tone, speed, and intonation of the voice are sent to the Azure Emotion API for emotion analysis. As a result of the analysis, a specific emotional state is recognized, such as when the user is anxious.

[1778] Screen information is acquired, for example, using the camera of smart glasses. The camera takes pictures of the store's map and product shelves, and converts them into text using Tesseract OCR technology. Based on this text data, the system works with the Google Maps API to determine the location within the physical store.

[1779] A specific prompt would be sent to the Generative AI model (GPT-4) with a request such as, "The Emotion API has detected that the user is feeling anxious. The user is currently looking for the location of a sticky note in the store. Please generate concise and easy-to-understand navigation instructions."

[1780] A generative AI model generates optimal feedback (for example, "Turn right and go straight down the next corridor. The sticky note is on the shelf to your right."), and this feedback is converted into speech by the Google Text-to-Speech API. The information is then delivered to the user through the smart glasses' speaker.

[1781] This system allows users to receive voice-based navigation and product location information tailored to their emotional state, enabling them to shop efficiently and comfortably in physical stores.

[1782] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1783] Step 1:

[1784] The user wears smart glasses and issues a voice command. Voice input is acquired through the smart glasses' microphone. The input voice data is the output of this step.

[1785] Step 2:

[1786] The device sends the acquired audio data to the Google Cloud Speech-to-Text API, which converts the audio into text data. The input is audio data, and the output is the converted text data.

[1787] Step 3:

[1788] The device sends text data along with characteristic data such as tone, speed, and intonation of speech to the Azure Emotion API. The input consists of text data and speech characteristic data, and the output is the result of the user's emotion analysis.

[1789] Step 4:

[1790] The server processes data based on the sentiment analysis results to determine a navigation method that corresponds to the user's emotions. The input here is the sentiment analysis results, and the output is navigation data corresponding to those emotions.

[1791] Step 5:

[1792] The device uses the camera on smart glasses to take pictures of the store interior and product shelves, and converts them into text data using Tesseract OCR. The input is the image taken by the camera, and the output is the converted text data.

[1793] Step 6:

[1794] The server uses OCR data in conjunction with the Google Maps API to identify location information within the store. The input is text data converted by OCR, and the output is the identified location information.

[1795] Step 7:

[1796] The device uses a generative AI model (GPT-4) to generate navigation instructions for the user based on sentiment analysis results and location information. The input is sentiment analysis results and location information, and the output is navigation instruction text.

[1797] Step 8:

[1798] The device sends the generated navigation instruction text to the Google Text-to-Speech API, where it is converted into speech. The input is navigation instruction text, and the output is audio data.

[1799] Step 9:

[1800] The device provides voice data to the user through the smart glasses' speaker. This allows the user to receive accurate directions and product information within the store via voice. The input is voice data, and the output is voice guidance.

[1801] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1802] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1803] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[1804] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1805] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[1806] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[1807] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[1808] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[1809] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[1810] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[1811] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[1812] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[1813] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[1814] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1815] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[1816] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[1817] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[1818] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[1819] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[1820] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[1821] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[1822] The following is further disclosed regarding the embodiments described above.

[1823] (Claim 1)

[1824] A means of obtaining user voice commands,

[1825] A means of analyzing acquired voice commands,

[1826] A means of acquiring screen information based on analyzed voice commands,

[1827] Means including a generative artificial intelligence model for summarizing or editing acquired screen information,

[1828] A means of providing summarized or edited information to the user in audio format,

[1829] A system that includes this.

[1830] (Claim 2)

[1831] The system according to claim 1, further comprising means for transcribing or editing text according to user instructions based on acquired voice commands.

[1832] (Claim 3)

[1833] The system according to claim 1, further comprising means for creating, confirming, and sending emails based on acquired voice commands.

[1834]

[1835] "Example 1"

[1836] (Claim 1)

[1837] A means of obtaining user voice commands,

[1838] A means of analyzing acquired voice commands,

[1839] A means of acquiring screen information based on analyzed voice commands,

[1840] Means including an artificial intelligence model for summarizing or editing acquired screen information,

[1841] A means of providing summarized or edited information to the user in audio format,

[1842] A means for extracting text data from acquired screen information using optical character recognition technology,

[1843] A means including an artificial intelligence model that analyzes extracted text data to extract important information,

[1844] A means of sending voice data to a server based on voice commands uttered by the user,

[1845] A means of obtaining screen information based on instructions from the server,

[1846] A system that includes this.

[1847] (Claim 2)

[1848] The system according to claim 1, further comprising means for transcribing or editing text according to user instructions based on acquired voice commands.

[1849] (Claim 3)

[1850] The system according to claim 1, further comprising means for creating, confirming, and sending emails based on acquired voice commands.

[1851] "Application Example 1"

[1852] (Claim 1)

[1853] A means of obtaining user voice commands,

[1854] A means of analyzing acquired voice commands,

[1855] A means for acquiring screen information or electronic transaction information based on analyzed voice commands,

[1856] Means including a generative artificial intelligence model for summarizing or editing acquired information,

[1857] A means of providing summarized or edited information to the user in audio format,

[1858] A system that includes this.

[1859] (Claim 2)

[1860] The system according to claim 1, further comprising means for transcribing or editing text or transaction details in accordance with user instructions based on acquired voice commands.

[1861] (Claim 3)

[1862] The system according to claim 1, further comprising means for creating, confirming, sending, and executing emails based on acquired voice commands.

[1863] "Example 2 of combining an emotion engine"

[1864] (Claim 1)

[1865] A means of obtaining user voice commands,

[1866] A means of analyzing acquired voice commands,

[1867] A means of recognizing a user's emotions from voice data,

[1868] A means for acquiring screen information based on analyzed voice commands and emotion recognition results,

[1869] A means for converting acquired screen information into text data using optical character recognition technology,

[1870] Means including a generative artificial intelligence model for summarizing or editing converted text data,

[1871] A means of providing summarized or edited information in audio format, adjusted according to the user's emotional state,

[1872] A system that includes this.

[1873] (Claim 2)

[1874] The system according to claim 1, further comprising means for transcribing or editing text in an optimal manner, taking into account the user's emotional state, based on acquired voice commands.

[1875] (Claim 3)

[1876] The system according to claim 1, further comprising means for creating, confirming, and sending an email, taking into account the user's emotional state based on acquired voice commands.

[1877] "Application example 2 when combining with an emotional engine"

[1878] (Claim 1)

[1879] A means of obtaining user voice commands,

[1880] A means of analyzing acquired voice commands,

[1881] A means of acquiring screen information based on analyzed voice commands,

[1882] Means including a generative artificial intelligence model for summarizing or editing acquired screen information,

[1883] A means of providing summarized or edited information to the user in audio format,

[1884] A means including an emotion engine that analyzes emotions using the user's voice commands and voice characteristics,

[1885] A means of adjusting the way information is delivered based on user emotions,

[1886] A system that includes this.

[1887] (Claim 2)

[1888] The system according to claim 1, further comprising means for transcribing or editing text according to user instructions based on acquired voice commands.

[1889] (Claim 3)

[1890] The system according to claim 1, further comprising means for creating, confirming, and sending emails based on acquired voice commands. [Explanation of Symbols]

[1891] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of obtaining user voice commands, A means of analyzing acquired voice commands, A means of acquiring screen information based on analyzed voice commands, Means including a generative artificial intelligence model for summarizing or editing acquired screen information, A means of providing summarized or edited information to the user in audio format, A system that includes this.

2. The system according to claim 1, further comprising means for transcribing or editing text according to user instructions based on acquired voice commands.

3. The system according to claim 1, further comprising means for creating, confirming, and sending emails based on acquired voice commands.

Images