System

Abstract

A system is provided.SOLUTION: A system comprising: a camera for capturing visual information; means for analyzing the captured visual information to extract textual information; and means for converting the extracted textual information to speech and outputting the speech.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] People with visual or hearing impairments have difficulty obtaining information in their daily lives. Therefore, they need assistance to obtain visually provided information as audio, and to obtain audio-based information visually. In particular, there is a need to convert and provide this information in real time, but existing technologies do not adequately meet the needs of such users. Therefore, there is a need for a system that can convert visual information into audio and read it aloud, and convert audio information into text and display it, thereby enabling people with visual and hearing impairments to lead more independent lives. [Means for solving the problem]

[0005] This invention relates to a system including a device with a camera and means for capturing visual information, means for analyzing the captured visual information and extracting text information, and means for converting the extracted text information into audio and outputting it. It also includes a microphone for capturing audio information, means for converting the captured audio information into text information, and means for displaying the converted text information. This system supports the visually impaired by reading out visual information aloud and the hearing impaired by displaying audio information as text. Furthermore, by including both a camera and a microphone, the system can meet the needs of users by providing both visual and audio information to them in real time.

[0006] "Visual information" generally refers to information that can be seen with the eyes, and specifically includes images and video captured by a camera.

[0007] "Audio information" generally refers to information that can be heard by the ear, and specifically includes sounds and voices captured by a microphone.

[0008] A "camera" is a device that converts light into an electrical signal and is used to capture objects or scenes as still or moving images.

[0009] "Text information" refers to information made up of characters and symbols in general, including information that is displayed in a visually recognizable form.

[0010] "Convert to speech" refers to the process of outputting text or other non-speech information as speech, including using text-to-speech or speech synthesis technologies.

[0011] A "microphone" is a device that converts sound into an electrical signal, specifically a device that captures ambient voices and sounds.

[0012] "Speech recognition" refers to the technology of analyzing speech signals and converting them into corresponding text information, and specifically includes the process of generating text input from speech input.

[0013] "Device" refers to a machine or system that includes a camera and microphone as described above and is designed to capture and analyze visual and audio information.

[0014] "Means for outputting" refers to means for presenting the analyzed information to the user, and particularly includes displaying the information on a screen and reading it aloud.

[0015] "Display" refers to the visual presentation of textual or other digital information to a user, and specifically includes the process of rendering information on a display or screen. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9]1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] The present invention provides an apparatus and means for assisting people with visual and hearing impairments. The system includes a camera and a microphone, and a set of means for capturing visual and audio information and providing each of them to the user in an appropriate format.

[0038] System configuration and operation

[0039] visual information processing

[0040] server

[0041] The server initializes a camera to capture visual information and an OCR module to analyze the captured image.

[0042] The image obtained from the camera is passed to the OCR module to extract text information.

[0043] Terminal

[0044] The terminal has a voice output device that converts text information sent from the server into voice and reads it out to the user.

[0045] User

[0046] A visually impaired user can understand the captured visual information by listening to the audio information provided by the system.

[0047] Auditory information processing

[0048] server

[0049] The server initializes a microphone to capture audio information and a speech recognition module to convert speech to text.

[0050] The voice data acquired by the microphone is passed to a voice recognition module and converted into text information.

[0051] Terminal

[0052] The terminal has a display that displays the text information sent from the server.

[0053] User

[0054] A hearing-impaired user can understand the captured audio information by reading the text information provided by the system.

[0055] Specific examples

[0056] Example 1: Use in a bookstore

[0057] User

[0058] When a visually impaired user is in a bookstore and wants to know the contents of a book, he or she uses a camera to take a picture of the book's cover.

[0059] Terminal

[0060] The image captured by the camera is sent to the server, where text information is extracted using an OCR module.

[0061] The extracted text information is converted into audio and read aloud to the user.

[0062] server

[0063] The server analyzes the visual information and generates audio data such as the book title and author name.

[0064] Example 2: Understanding everyday conversation

[0065] User

[0066] If a hearing impaired user wants to know the content of everyday conversations, he or she can use a microphone to record the conversations.

[0067] Terminal

[0068] The voice data recorded by the microphone is sent to the server and converted into text information by a voice recognition module.

[0069] The converted text information is displayed on the terminal display so that the user can check it.

[0070] server

[0071] The server processes the captured audio information, converts it into text information in real time, and transmits it.

[0072] Program processing description

[0073] server

[0074] When the server starts up, it loads the necessary libraries and modules and initializes the camera and microphone.

[0075] The server periodically captures visual information from the camera, analyzes it using an OCR module, and converts it into text information.

[0076] Similarly, audio information is captured from the microphone, analyzed by a voice recognition module, and converted into text information.

[0077] Terminal

[0078] The terminal provides the text information received from the server to the user through voice output or display.

[0079] The system of the present invention can provide necessary information in real time to users with visual and hearing impairments, thereby increasing their independence in daily life and overcoming obstacles to information acquisition.

[0080] The processing flow will be explained below.

[0081] Specific processing explanation of the program

[0082] visual information processing

[0083] Step 1:

[0084] The server initializes the system, loading the necessary libraries and modules (OpenCV, OCR module, etc.) and initializing the camera.

[0085] Step 2:

[0086] It uses the device's camera to capture visual information: when the user points an object at the camera, the camera takes a picture of it.

[0087] Step 3:

[0088] The server sends the captured image data to the OCR module, which analyzes the image data and extracts text information.

[0089] Step 4:

[0090] The server sends the extracted text information to a text-to-speech (TTS) engine, which converts the text information into audio data.

[0091] Step 5:

[0092] The device plays the audio data to the user, allowing the user to obtain and understand the visual information as audio.

[0093] Auditory information processing

[0094] Step 1:

[0095] The server initializes the system, loading the necessary libraries and modules (SpeechRecognition, voice recognition module, etc.) and initializing the microphone.

[0096] Step 2:

[0097] Use the device's microphone to capture audio information, either by the user speaking into the microphone or by recording ambient sounds.

[0098] Step 3:

[0099] The server sends the captured voice data to a voice recognition module, which analyzes the voice data and generates text information.

[0100] Step 4:

[0101] The server transmits the generated text information to the terminal.

[0102] Step 5:

[0103] The device displays the text information on the display, allowing the user to receive and understand the audio information as text.

[0104] Specific examples

[0105] Example 1: Use in a bookstore

[0106] Step 1:

[0107] The server initializes the camera and OCR module.

[0108] Step 2:

[0109] The user captures the book cover through the device's camera.

[0110] Step 3:

[0111] The server analyzes the image data using an OCR module and extracts text information.

[0112] Step 4:

[0113] The server converts the extracted text information into speech using a TTS engine.

[0114] Step 5:

[0115] The terminal plays the audio data to the user, and the user understands the book title and author name by listening to the audio.

[0116] Example 2: Understanding everyday conversation

[0117] Step 1:

[0118] The server initializes the microphone and voice recognition module.

[0119] Step 2:

[0120] The user captures the audio of the conversation through the device's microphone.

[0121] Step 3:

[0122] The server analyzes the voice data using a voice recognition module and generates text information.

[0123] Step 4:

[0124] The server transmits the generated text information to the terminal.

[0125] Step 5:

[0126] The terminal displays text information on the display, and the user understands the content of the conversation through text.

[0127] In this way, visual and auditory information is processed sequentially along each step, making it possible to provide the user with the information they need in an appropriate format.

[0128] Example 1

[0129] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0130] In modern society, people with visual or hearing impairments face significant challenges in obtaining and understanding information. In particular, visually impaired people lack the means to receive visual information as audio, while hearing impaired people have limited means to receive audio information as text. Effective systems are needed to overcome these obstacles and improve users' independence and quality of life.

[0131] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0132] In this invention, the server includes means having an imaging device for capturing visual information, means for analyzing the captured visual information and extracting text information, means for converting the extracted text information into speech and outputting it, means for generating speech using a language conversion device for generating speech, means having an acoustic collection device for capturing speech information, means for converting the captured speech information into text information, means for displaying the converted text information, means using a speech recognition device for analyzing the speech information in real time, and means including a communication device for providing information to users in real time, thereby enabling users with visual and hearing impairments to understand the captured visual and speech information in real time and overcome obstacles to information acquisition.

[0133] "Visual information" refers to image data captured by an imaging device such as a camera.

[0134] "Imaging device" refers generically to cameras and other devices for capturing visual data.

[0135] "Capture" refers to the process of obtaining visual or audio information.

[0136] "OCR (Optical Character Recognition)" refers to a technology for extracting text information from image data.

[0137] "Audio information" refers to audio data captured by an audio collecting device such as a microphone.

[0138] "Sound collection device" refers to a general term for microphones and other devices used to capture audio data.

[0139] "Speech recognition device" refers to technology or devices for converting voice information into text information.

[0140] "User" refers to a person with a visual or hearing impairment who wishes to obtain information using this system.

[0141] "Means for converting into voice and outputting" refers to a general term for devices and programs that convert extracted text information into voice information and provide it to the user.

[0142] "Language conversion device" refers to technology or devices for converting text data into voice data.

[0143] "Text information" refers to character data extracted or converted from visual or audio information.

[0144] "Means for displaying" refers to a display or other device for visually presenting textual information to a user.

[0145] "Communication means" refers to the protocols and technologies used to send and receive data between the server and the terminal.

[0146] "Providing in real time" refers to rapidly processing captured information and providing it to the user simultaneously.

[0147] This invention is a system for assisting people with visual or hearing impairments, and aims to overcome barriers to information acquisition by capturing visual and audio information and providing each to the user in an appropriate format.

[0148] First, we will explain how visual information is processed. The server initializes an imaging device to capture visual information and loads an OCR (Optical Character Recognition) module to analyze the captured image data. The image data obtained from the camera is converted into text information using the OCR module. Specifically, the camera is initialized using Python's cv2 library, and OCR analysis is performed using the pytesseract library. This text information is sent to the device via an HTTP request or WebSocket.

[0149] The device converts the text information received from the server into audio using a Text-to-Speech engine (e.g., gTTS) and provides the audio data to the user through a built-in speaker. A visually impaired user can understand the captured visual information through this audio information.

[0150] Next, we will explain how to process auditory information. The server initializes an audio collection device to capture audio information and loads a speech recognition module to convert the audio into text. The audio data captured by the microphone is converted into text information using the speech recognition module. Specifically, the microphone is initialized using the pyaudio library, and speech recognition is performed using Google Cloud Speech-to-Text. This text information is sent to the device via an HTTP request or WebSocket.

[0151] The terminal displays the text information received from the server on a display. The display uses GUI libraries such as tkinter and PyQt to display the text information in a format that is easy for users to read. Hearing-impaired users can understand the captured audio information by reading this text information.

[0152] As a concrete example, consider a bookstore. A visually impaired user takes a picture of a book cover using a camera. The image data is sent to a server, where an OCR module extracts text information (e.g., the book title and author's name). The extracted text information is converted into audio and read aloud to the user. The server analyzes the visual information and provides the content to the user via audio.

[0153] Let's take a concrete example of understanding everyday conversation. If a hearing-impaired user wants to know the content of an everyday conversation, they can record the conversation using a microphone. The audio data is sent to a server, where it is converted into text information by a speech recognition module. The converted text information is then displayed on the device display for the user to confirm. The server then converts the captured audio information into text in real time and provides the content to the user.

[0154] An example of a prompt is:

[0155] "Please explain how a system that reads the contents of books aloud in bookstores for the visually impaired works. Please provide detailed steps from camera initialization to audio output."

[0156] "Please explain how a system that displays everyday conversations as text for the hearing impaired works. Please provide detailed steps from microphone initialization to text display."

[0157] Examples include:

[0158] In this way, the system of the present invention can provide visually and hearing impaired users with the information they need in real time, increasing their independence.

[0159] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0160] visual information processing

[0161] Step 1:

[0162] Server: Initialize the imaging device (camera). Using the Python cv2 library, initialize the camera with the cv2.VideoCapture(0) command. This initialization process allows the camera to function properly and enable image capture. The input is the camera device, and the output is the initialized camera object.

[0163] Step 2:

[0164] Server: Captures image data. Periodically, it uses the cap.read() command to obtain image data from the camera. The obtained image data is passed to the subsequent OCR module. The input is the initialized camera object, and the output is the captured image data.

[0165] Step 3:

[0166] Server: Pass the captured image data to the OCR module and extract text information. Specifically, use the pytesseract library to extract text from the image data with the pytesseract.image_to_string(image) command. The input is the captured image data, and the output is the extracted text data.

[0167] Step 4:

[0168] Server: Send the extracted text data to the terminal via an HTTP request or WebSocket. Use the requests.post(url, data={'text': text}) command. The input is the extracted text data, and the output is the text data sent to the terminal.

[0169] Step 5:

[0170] Terminal: The received text data is passed to the Text-to-Speech engine to generate audio data. The gTTS library is used to generate audio data using the gTTS(text) command. The generated audio file is played through the built-in speaker. The input is the text data received from the server, and the output is the generated audio data.

[0171] Step 6:

[0172] User: Understands the captured visual information by listening to the audio information provided by the device. The input is the audio data played from the device, and the output is the information comprehension obtained by listening to the audio data.

[0173] Auditory information processing

[0174] Step 1:

[0175] Server: Initialize the sound collection device (microphone). Use Python's pyaudio library to initialize the microphone with the audio = pyaudio.PyAudio() command. The input is the microphone device, and the output is the initialized microphone object.

[0176] Step 2:

[0177] Server: Captures audio data. Periodically obtains audio data from the microphone using the audio.open() and stream.read() commands. The obtained audio data is passed to the subsequent speech recognition module. The input is the initialized microphone object, and the output is the captured audio data.

[0178] Step 3:

[0179] Server: Pass the captured voice data to the speech recognition module and convert it into text information. Use Google Cloud Speech-to-Text to convert the voice data into text using the speech_to_text(audio_data) command. The input is the captured voice data, and the output is the converted text data.

[0180] Step 4:

[0181] Server: The converted text data is sent to the terminal via an HTTP request or WebSocket. Use the requests.post(url, data={'text': text}) command. The input is the converted text data, and the output is the text data sent to the terminal.

[0182] Step 5:

[0183] Terminal: The received text data is displayed on the screen. Using a GUI library such as tkinter or PyQt, the text data is displayed with the label.config(text=text) command. The input is the text data received from the server, and the output is the text data displayed on the screen.

[0184] Step 6:

[0185] User: Understands the captured audio information by reading the text information displayed on the display. The input is the text data displayed on the display, and the output is the understanding of the information obtained by reading the text data.

[0186] (Application example 1)

[0187] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0188] People with visual or hearing impairments have difficulty understanding product information and conversations in virtual stores. To address this issue, a system is needed that can convert visual information into speech and speech into text in real time.

[0189] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0190] In this invention, the server is a device equipped with a camera for capturing visual information, including means for analyzing the captured visual information and extracting text information, means for converting the extracted text information into audio and outputting it, means for saving the extracted text information as an audio file, and means for playing the saved audio file; a device equipped with an acoustic sensor for capturing audio information, including means for converting the captured audio information into text information, means for displaying the converted text information, and means for displaying the text information on a display of a user's terminal; a device equipped with both a camera and an acoustic sensor, including means for converting the extracted text information from text to audio and saving it as an audio file, means for playing the saved audio file, and means for using a generative AI model to generate prompt sentences. This enables users with visual or hearing impairments to intuitively and easily understand product information and conversations in a virtual store.

[0191] "Visual information" refers to image or video information captured using a camera or other photographic device.

[0192] "Image capture device" refers to a camera or other device for capturing visual information.

[0193] "Text information" refers to text data extracted from visual information.

[0194] "Audio information" refers to audio data captured using a microphone or acoustic sensor.

[0195] "Acoustic sensor" refers to a microphone or other device for capturing audio information.

[0196] "Transformation" refers to the act or process of transforming information into a different form.

[0197] "Users" refers to visually and hearing impaired people who use the system.

[0198] "Terminal" refers to a device used by a user, such as a smartphone or tablet.

[0199] "Generative AI model" refers to an artificial intelligence model for generating prompts and other text.

[0200] A "prompt" refers to the text of instructions or questions that are input to a generative AI model.

[0201] "Audio File" means a digital file that stores audio information.

[0202] This invention is a system for supporting people with visual or hearing impairments, making it easier for them to understand product information and conversations in a virtual store.

[0203] visual information processing

[0204] The user takes a picture of a product image or screen using a camera to capture visual information. The server analyzes the captured visual information using OCR (optical character recognition) technology and extracts text information. This text information is then used to generate a prompt using a generative AI model, which is then converted into audio and saved as an audio file. The device then plays the saved audio file, allowing the user to listen to the product description.

[0205] Auditory information processing

[0206] Using an acoustic sensor (microphone) to capture auditory information, the user captures audio information in the virtual store (for example, conversations with sales staff). The server analyzes the captured audio information using voice recognition technology and converts it into text information. This text information is displayed on the user's device display, allowing the user to confirm the information provided by voice as text.

[0207] System Configuration

[0208] 1. Server

[0209] The server analyzes the captured visual information using an OCR module (e.g., pytesseract) and converts it into text information.

[0210] The server analyzes the captured voice information using a voice recognition module (e.g., speech_recognition) and converts it into text information.

[0211] The text information is processed by a generative AI model to generate a prompt sentence.

[0212] To convert the generated text information into audio, an audio file is created and saved using Google Text-to-Speech (gTTS).

[0213] 2. Terminal

[0214] The user's terminal plays the audio file sent from the server and provides the user with information by voice.

[0215] The user's terminal displays the text information sent from the server on a display, allowing the user to confirm the text information.

[0216] Specific examples

[0217] Example 1: Retrieving product information

[0218] While a user is viewing a product page in a virtual store, they take a picture of the screen with their smartphone camera. The image is sent to a server, where the product description is converted into text using OCR. A generative AI model generates a prompt, which is then converted into speech and played back to the user.

[0219] Example 2: Understanding conversation content

[0220] The user records their conversation with the sales staff in the virtual store, and the audio data is sent to the server, where it is converted into text and displayed on the device's display. The user can understand the content of the conversation by reading the text.

[0221] Prompt Sentence Examples

[0222] "Extract Japanese characters from the input image file and convert the content into an audio file (MP3 format). Also, play the audio file and display the extracted text."

[0223] This system enables visually or hearing impaired users to understand product information and conversations in a virtual store in real time, allowing them to enjoy a comfortable shopping experience.

[0224] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0225] Step 1:

[0226] The user uses the smartphone camera to take pictures of the virtual store screen and product images. The input is image data from the camera. The output is the captured image file.

[0227] Step 2:

[0228] The terminal sends the captured image file to the server. The input is the image file. The output is the image data sent to the server.

[0229] Step 3:

[0230] The server parses the received image file into text using an OCR module (e.g., pytesseract). The input is the image data received by the server. The output is the extracted text information.

[0231] Step 4:

[0232] The server uses a generative AI model to generate a prompt from the extracted text, and then converts it into an audio file using Google Text-to-Speech (gTTS). The input is the extracted text, and the output is the generated prompt and audio file.

[0233] Step 5:

[0234] The server sends the generated audio file to the terminal. The input is the audio file. The output is the audio data sent to the terminal.

[0235] Step 6:

[0236] The terminal plays the received audio file and provides audio information to the user. The input is the audio data sent from the server. The output is the audio played back from the audio data.

[0237] Step 7:

[0238] A user uses the microphone on their smartphone to record conversations in a virtual store. The input is the audio data of the conversation. The output is a recorded audio file.

[0239] Step 8:

[0240] The terminal sends the recorded audio file to the server. The input is the recorded audio file. The output is the audio data sent to the server.

[0241] Step 9:

[0242] The server converts the received audio file into text using a speech recognition module (e.g., speech_recognition). The input is the received audio data. The output is the converted text.

[0243] Step 10:

[0244] The server sends the converted character information to the terminal. The input is the converted character information. The output is the character information sent to the terminal.

[0245] Step 11:

[0246] The terminal displays the received text information on the display and provides the text information to the user. The input is the text information sent from the server. The output is the text information displayed on the display.

[0247] This allows users with visual or hearing impairments to understand product information and conversations in a virtual store in real time.

[0248] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0249] This invention provides a system that combines an emotion engine that recognizes the user's emotions with a device for assisting people with visual or hearing impairments. The system is equipped with a camera and a microphone, and includes a series of means for acquiring visual and audio information in real time and providing it to the user in an appropriate format. Furthermore, the emotion engine recognizes the user's emotions and adjusts the information output based on the results.

[0250] System configuration and operation

[0251] visual information processing

[0252] server

[0253] The server initializes a camera to capture visual information and an OCR module to analyze the captured image.

[0254] The image obtained from the camera is passed to the OCR module to extract text information.

[0255] The extracted text information is sent to an emotion engine, which selects an appropriate voice output format depending on the user's emotional state.

[0256] Terminal

[0257] The terminal is equipped with a voice output device that converts text information sent from the server into voice and reads it out to the user. The emotion engine adjusts the tone and speed of the voice according to the user's emotions.

[0258] User

[0259] A visually impaired user can understand the captured visual information by listening to audio information provided by the system, for example, a slower voice when the user is tired, or a clearer voice when the user is energetic.

[0260] Auditory information processing

[0261] server

[0262] The server initializes a microphone to capture audio information and a speech recognition module to convert speech to text.

[0263] The voice data acquired by the microphone is passed to a voice recognition module and converted into text information.

[0264] The generated text information is sent to an emotion engine, which adjusts the format and size of the text display according to the user's emotional state.

[0265] Terminal

[0266] The terminal has a display that displays text information sent from the server, and the emotion engine adjusts the font size and color of the text according to the user's emotions.

[0267] User

[0268] A hearing-impaired user can understand the captured audio information by reading the text information provided by the system, for example, in normal font size when the user is calm and in larger font size when the user is anxious.

[0269] Specific examples

[0270] Example 1: Use in a bookstore

[0271] User

[0272] When a visually impaired user is in a bookstore and wants to know the contents of a book, he or she uses a camera to take a picture of the book's cover.

[0273] Terminal

[0274] The image captured by the camera is sent to the server, where text information is extracted using an OCR module.

[0275] The extracted text information is sent to the TTS engine via the emotion engine, and is read aloud to the user in a voice format that corresponds to the user's emotion.

[0276] server

[0277] The server analyzes visual information, generates audio data such as the book title and author's name, and uses an emotion engine to select the optimal audio output that takes the user's emotions into consideration.

[0278] Example 2: Understanding everyday conversation

[0279] User

[0280] If a hearing impaired user wants to know the content of everyday conversations, he or she can use a microphone to record the conversations.

[0281] Terminal

[0282] The voice data recorded by the microphone is sent to the server and converted into text information by a voice recognition module.

[0283] The converted text information is sent to the terminal via the emotion engine and displayed on the display in a format that corresponds to the user's emotion.

[0284] server

[0285] The server processes the captured voice information, converts it into text information in real time, and selects the optimal text display format taking into account the user's emotions using an emotion engine.

[0286] In this way, by adding an emotion engine to the processing of visual and auditory information, it becomes possible to provide information flexibly according to the user's emotional state, further improving the user experience.

[0287] The processing flow will be explained below.

[0288] visual information processing

[0289] Step 1:

[0290] The server initializes the system, loading the necessary libraries and modules (OpenCV, OCR module, etc.) and initializing the camera.

[0291] Step 2:

[0292] The user shows the book cover to the device's camera, and the device uses the camera to capture the book cover.

[0293] Step 3:

[0294] The server sends the captured image data to the OCR module to extract text information, which is then analyzed and generated.

[0295] Step 4:

[0296] The server sends the extracted text information to the emotion engine, which assesses the user's emotional state and adjusts the tone and speed of the speech accordingly.

[0297] Step 5:

[0298] The server sends the text information adjusted by the emotion engine to a TTS (text-to-speech) engine, which converts the text information into voice data.

[0299] Step 6:

[0300] The device plays the audio data back to the user. For example, if the device determines that the user is tired, it will read the data in a slower tone, and if the user is energetic, it will read the data in a clearer tone.

[0301] Auditory information processing

[0302] Step 1:

[0303] The server initializes the system, loading the necessary libraries and modules (SpeechRecognition, voice recognition module, etc.) and initializing the microphone.

[0304] Step 2:

[0305] The user speaks into the device's microphone or records the surrounding audio. The device captures the audio information.

[0306] Step 3:

[0307] The server sends the captured voice data to a voice recognition module to generate text information, which then analyzes the voice data and converts it into text information.

[0308] Step 4:

[0309] The server sends the generated text information to the emotion engine, which evaluates the user's emotional state and adjusts the format and size of the text display accordingly.

[0310] Step 5:

[0311] The server sends text information adjusted by the emotion engine to the terminal.

[0312] Step 6:

[0313] The device will display text information on the display, for example in a larger font size if it determines that the user is anxious, or in a normal font size if the user is calm.

[0314] Specific examples

[0315] Example 1: Use in a bookstore

[0316] Step 1:

[0317] The server initializes the camera and OCR module.

[0318] Step 2:

[0319] The user points the device camera at the book cover.

[0320] Step 3:

[0321] The device captures the book cover with its camera and sends the image data to the server.

[0322] Step 4:

[0323] The server analyzes the image data using an OCR module and extracts text information.

[0324] Step 5:

[0325] The server sends the text information to the emotion engine, which determines how to read it out based on the user's emotion.

[0326] Step 6:

[0327] The server converts the adjusted text information into voice data using a TTS engine.

[0328] Step 7:

[0329] The device plays the audio data to the user, for example, reading it in a slower tone if it determines that the user is tired.

[0330] Example 2: Understanding everyday conversation

[0331] Step 1:

[0332] The server initializes the microphone and voice recognition module.

[0333] Step 2:

[0334] The user speaks into the device microphone or captures the conversation audio.

[0335] Step 3:

[0336] The terminal transmits the voice information to the server.

[0337] Step 4:

[0338] The server analyzes the voice data using a voice recognition module and converts it into text information.

[0339] Step 5:

[0340] The server sends the generated text information to the emotion engine, which determines the format and size according to the user's emotion.

[0341] Step 6:

[0342] The server sends the adjusted text information to the terminal.

[0343] Step 7:

[0344] The device displays text information on the display, for example, in a larger font size if it determines that the user is in a hurry.

[0345] In this way, by combining the processing of visual and auditory information with an emotion engine, it becomes possible to provide information flexibly according to the user's emotional state, further improving the user experience.

[0346] Example 2

[0347] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0348] People with visual or hearing impairments have difficulty obtaining and understanding information in their daily lives. Furthermore, conventional information support systems do not take the user's emotional state into consideration, which results in a lack of improvement in the user experience. The objective of this invention is to provide flexible information provision according to the user's emotional state in such information support systems for the visually and hearing impaired, thereby improving the user experience.

[0349] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0350] In this invention, the server includes a camera for capturing visual information, means for analyzing the captured visual information and extracting text information, means for sending the extracted text information to an emotion engine for analyzing the user's emotion, and means for converting the text information into audio based on the analysis results of the emotion engine and adjusting the audio format for output. This makes it possible to provide information in an appropriate format that takes into account the user's emotional state when converting visual information into audio. The server also includes a microphone for capturing audio information, means for converting the captured audio information into text information, means for sending the converted text information to the emotion engine for analyzing the user's emotion, and means for adjusting and displaying the text format based on the analysis results of the emotion engine. This makes it possible to provide information in an appropriate format that takes into account the user's emotional state when converting audio information into text.

[0351] "Visual information" refers to image or video data captured using a device such as a camera.

[0352] A "camera" is a photographing device for acquiring visual information.

[0353] An "OCR (Optical Character Recognition) module" is a software module for extracting character information from captured image data.

[0354] "Audio information" refers to sound or voice data captured using a device such as a microphone.

[0355] A "microphone" is a sound pickup device for acquiring audio information.

[0356] An "emotion engine" is a software component for analyzing a user's emotional state.

[0357] A "voice recognition module" is a software module for converting captured voice data into text information.

[0358] "Text information" refers to data that can be represented and stored as characters.

[0359] A "TTS (Text-to-Speech) engine" is a software engine for converting text information into speech.

[0360] A "terminal" is a device used by a user to receive information, including an audio output device and a display.

[0361] A "server" is a centralized computing device for managing and executing system-wide processing.

[0362] A "display" is a screen device for visually presenting text information to a user.

[0363] "Format" refers to the form or style in which text or audio information is presented to the user appropriately.

[0364] MODE FOR CARRYING OUT THE INVENTION

[0365] This invention relates to a system for assisting people with visual or hearing impairments, and provides a set of means for acquiring visual and audio information in real time and providing the information in a flexible format according to the user's emotional state. The system is equipped with a camera and a microphone, acquires visual and audio information in real time, and provides the information to the user in an appropriate format. It also uses an emotion engine to analyze the user's emotional state and adjusts the information output format based on the results.

[0366] visual information processing

[0367] server

[0368] The server is equipped with a camera for capturing visual information. After the camera is initialized, it captures visual information (e.g., text or images). The captured image data is passed to an OCR (Optical Character Recognition) module for analysis. This analysis extracts text information from the image.

[0369] The server then sends the extracted text information to the emotion engine, which analyzes the user's emotional state and selects a voice format (tone, speed, etc.) based on the analysis results. For example, if the user is tired, a slower voice format is selected, and if the user is active, a clearer, crisper voice format is selected.

[0370] Terminal

[0371] The terminal is equipped with a voice output device and receives text information sent from the server. This text information is converted into voice using a TTS (Text-to-Speech) engine and read aloud to the user through the voice output device.

[0372] User

[0373] A visually impaired user can understand the content of a book by receiving visual information captured by a camera as audio. For example, a user can take a picture of a book cover in a bookstore and receive text information extracted from the captured image as audio.

[0374] Auditory information processing

[0375] server

[0376] The server is equipped with a microphone for capturing voice information. After the microphone is initialized, it captures voice information (e.g., conversation content). The captured voice data is passed to a voice recognition module and converted into text information. The converted text information is also sent to the emotion engine.

[0377] The emotion engine analyzes the user's emotional state and adjusts the text formatting (font size, color, etc.) based on the analysis. For example, if the user is calm, text will be displayed in normal font size, and if the user is anxious, text will be displayed in large font size.

[0378] Terminal

[0379] The device is equipped with a display and receives text information sent from the server. The received text information is displayed on the display in an appropriate format based on the analysis results of the emotion engine.

[0380] User

[0381] Hearing-impaired users can understand the content of audio information captured by a microphone by receiving it as text. For example, a user can record their everyday conversation, and text information generated from the recorded audio data is displayed on a screen.

[0382] Specific examples

[0383] Example 1: Use in a bookstore

[0384] User

[0385] When a visually impaired user is in a bookstore and wants to know the contents of a book, he or she uses a camera to take a picture of the book's cover.

[0386] Terminal

[0387] The image captured by the camera is sent to the server, where the OCR module extracts text information, which is then sent to the TTS engine via the emotion engine, where it is read aloud to the user in a voice format that corresponds to the user's emotion.

[0388] server

[0389] The server analyzes the visual information, generates text data such as the book title and author's name, and uses an emotion engine to select the optimal voice output based on the user's emotions.

[0390] Example 2: Understanding everyday conversation

[0391] User

[0392] If a hearing impaired user wants to know the content of everyday conversations, he or she can use a microphone to record the conversations.

[0393] Terminal

[0394] The voice data recorded by the microphone is sent to the server and converted into text information by a voice recognition module. The converted text information is sent to the device via an emotion engine and displayed on the screen in a format that corresponds to the user's emotion.

[0395] server

[0396] The server processes the captured audio information, converts it into text information in real time, and uses an emotion engine to select the optimal text display format according to the user's emotions.

[0397] In a specific embodiment of this invention, users with visual and hearing impairments can receive visual information as audio and audio information as text, and enjoy information in a flexible format that adapts to their emotional state. This system improves the user experience and makes it easier to obtain information in daily life.

[0398] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0399] visual information processing

[0400] Step 1:

[0401] server

[0402] The server initializes the camera to capture visual information, and the initialized camera is ready to capture visual information in real time.

[0403] Specific operation: The server sends an initialization command to the camera and confirms that the camera is ready for operation.

[0404] Step 2:

[0405] User

[0406] If the user is visually impaired, they place the source of information they want to know about (e.g., a book cover) in front of the camera and use the camera to photograph the object.

[0407] Specific Action: The user presses the camera button to capture an image.

[0408] Step 3:

[0409] server

[0410] The server receives the captured image data and passes it to the OCR module, which analyzes the image data and extracts text information. The input is the image data and the output is the extracted text information.

[0411] Specific operation: The server sends an API request to the OCR module to perform image analysis.

[0412] Step 4:

[0413] server

[0414] The server sends the extracted text information to the emotion engine, which analyzes the user's emotional state. The input is the text information, and the output is the emotion analysis result.

[0415] Specific operation: The server sends text data to the emotion engine and obtains the analysis results.

[0416] Step 5:

[0417] server

[0418] The server selects an audio format for converting text information into audio based on the emotion analysis results. The input is the emotion analysis results, and the output is the audio format.

[0419] What happens: The server selects the appropriate voice format (tone, speed, etc.) and prepares the data to send to the TTS engine.

[0420] Step 6:

[0421] Terminal

[0422] The terminal receives text information sent from the server and converts it into speech using a TTS engine. The input is text information and speech format, and the output is speech data.

[0423] Specific operation: The device sends an API request to the TTS engine and obtains the generated voice data.

[0424] Step 7:

[0425] Terminal

[0426] The terminal uses an audio output device to read the generated audio data to the user.

[0427] Specific operation: The terminal transmits audio data to the audio output device, and the user listens to the audio.

[0428] Auditory information processing

[0429] Step 1:

[0430] server

[0431] The server initializes the microphone to capture audio information. Once initialized, the microphone is ready to capture audio information in real time.

[0432] Specific operation: The server sends a microphone initialization command and verifies that the microphone is ready for operation.

[0433] Step 2:

[0434] User

[0435] If the user is hearing impaired, the desired conversation content is recorded using a microphone.

[0436] Specific action: The user presses the microphone button to capture audio.

[0437] Step 3:

[0438] server

[0439] The server receives the captured voice data and passes it to the voice recognition module, which analyzes the voice data and converts it into text information. The input is voice data and the output is text information.

[0440] Specific operation: The server sends an API request to the speech recognition module to perform speech analysis.

[0441] Step 4:

[0442] server

[0443] The server sends the generated text information to the emotion engine, which analyzes the user's emotional state. The input is the text information, and the output is the emotion analysis result.

[0444] Specific operation: The server sends text data to the emotion engine and obtains the analysis results.

[0445] Step 5:

[0446] server

[0447] The server selects the text format (font size, color, etc.) based on the sentiment analysis results. The input is the sentiment analysis results, and the output is the formatted text information.

[0448] Specific behavior: The server selects the appropriate text format and prepares the data to be displayed on the display.

[0449] Step 6:

[0450] Terminal

[0451] The terminal receives the formatted text information sent from the server and displays it on the display. The input is the formatted text information, and the output is the text displayed on the display.

[0452] Specific operation: The terminal sends formatted text information to the display and checks the display status.

[0453] Step 7:

[0454] User

[0455] The user understands the content of the conversation by reading the text information displayed on the display.

[0456] Specific operation: The user looks at the display and visually recognizes the text information.

[0457] The above are the specific processing steps of the system for processing visual and auditory information, which enables visually and auditory impaired users to obtain and understand information in real time.

[0458] (Application example 2)

[0459] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0460] The challenge is to solve the difficulties that visually and hearing impaired users have in safely and effectively performing complex operations such as electronic payments in their daily lives. Another problem is that the information provided is not adapted to the user's current emotional state, resulting in a poor user experience.

[0461] The specification processing by the specification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means having a camera for capturing visual information, means for analyzing the captured visual information and extracting text information, means for converting the extracted text information into voice and outputting it, an emotion engine for adjusting the extracted text information and the converted text information in accordance with the user's emotional state, means for outputting the information adjusted by the emotion engine to the user in an appropriate tone and speed, and means for providing the user with information captured in an electronic payment situation. This enables visually or hearing impaired users to receive appropriate information according to their emotional state when making electronic payments.

[0462] "Visual information" generally refers to video data captured using an image capture device such as a camera.

[0463] "Camera" refers to a photographic device used to capture visual information.

[0464] "Analysis" refers to the process of breaking down captured visual and audio information and extracting useful data.

[0465] "Text information" refers to character data extracted from visual or audio information.

[0466] "Extraction" refers to the process of extracting necessary information from analyzed data.

[0467] "Convert to speech" refers to the process of converting extracted text information into speech data.

[0468] "Output" refers to the act of conveying converted speech or displayed text to the user.

[0469] "Audio information" generally refers to sound data captured using an audio capture device such as a microphone.

[0470] "Microphone" refers to a sound pickup device used to capture audio information.

[0471] "Display" refers to visually conveying text information to a user using a display device or the like.

[0472] An "emotion engine" refers to a system that analyzes the user's emotional state and adjusts the format of information output based on the results.

[0473] "Adjustment" refers to the act of changing the output format or content depending on the user's emotional state.

[0474] "Tone" refers to the sound quality of voice, such as pitch, volume, and speed.

[0475] "Speed" refers to the speed at which voice or information is output.

[0476] "Electronic payment" refers to a payment transaction made using digital technology.

[0477] "Providing" refers to the act of giving information to a user.

[0478] The present invention realizes a system that enables users with visual or hearing impairments to make electronic payments safely and effectively. Specific embodiments are described below.

[0479] Hardware Configuration

[0480] The server is equipped with a camera to capture visual information and a microphone to capture audio information. The camera captures visual information and the microphone captures audio information. The terminal used by the user is equipped with a display and speaker.

[0481] Software Configuration

[0482] The server is installed with an OCR (Optical Character Recognition) module to analyze visual information and extract text information, and a speech recognition module to convert audio information into text. The speech synthesis engine pyttsx3 is used for reading out the audio, and EmotionRecognizer is used to recognize the user's emotional state.

[0483] Data processing and calculation

[0484] 1. Processing of visual information:

[0485] The server uses a camera to capture visual information. Images are captured in real time using the OpenCV library.

[0486] The acquired image data is subjected to OCR processing using pytesseract, and text information is extracted.

[0487] 2. Audio processing:

[0488] It uses a microphone to capture audio information. The speech_recognition library is used to obtain the audio data.

[0489] The captured voice data is converted into text information via the Google Speech API.

[0490] 3. Emotion recognition and output adjustment:

[0491] Using EmotionRecognizer, the user's emotional state is analyzed, which allows for assessment of the user's level of fatigue, energy, etc.

[0492] Based on the analysis results, the acquired text information is outputted using the speech synthesis engine pyttsx3 in an appropriate tone and speed, and the font size and color are also adjusted for text display.

[0493] Specific examples

[0494] As a concrete example, consider a visually impaired user making an electronic payment at a cafe. The user scans a QR code with their smartphone camera, and the server analyzes the image using OCR to extract text information. The emotion engine analyzes the user's emotional state, and if it determines that the user is tired, for example, it lowers the tone and speed of the voice to announce, "This code is for payment at the cafe."

[0495] Example of input prompt for generative AI model

[0496] "When a user scans a QR code with a camera, design a system that extracts the code's contents using an OCR module, determines the user's emotion using an emotion recognition engine, and provides information in an appropriate format based on that emotion. For example, if the user is tired, slow down the reading speed, and if the user is energetic, speed up the reading speed."

[0497] In this way, a support system can be provided that allows visually or hearing impaired users to make electronic payments with peace of mind.

[0498] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0499] Step 1:

[0500] A user uses a camera to capture visual information, and obtains an image containing a QR code and text information. The input is the image data captured by the camera, and this data is passed to the next step.

[0501] Step 2:

[0502] The server passes the captured image data as input to the OCR module to extract text information. Specifically, it reads the image using the OpenCV library and performs text extraction using pytesseract. The output of this process is the extracted text information.

[0503] Step 3:

[0504] The extracted text information is passed as input to the server's speech synthesis engine (pyttsx3) to prepare for conversion into speech. At this time, the server receives additional information from the camera or microphone as input and performs user emotion analysis using EmotionRecognizer to recognize the user's emotional state. The output of this process is emotional state data.

[0505] Step 4:

[0506] The server adjusts the tone and speed of the speech synthesis based on the emotional state data. Specifically, it sets pyttsx3 properties, for example, slowing down the speech speed if the user is tired and speeding up if the user is energetic. The output of this process is speech data adapted to the user's emotions.

[0507] Step 5:

[0508] The server outputs the adjusted voice data to the device's speaker and provides the information to the user. The input is the adjusted voice data, and this data is transmitted to the user as voice output. The processing is completed when the device's speaker plays the voice and the user receives the information.

[0509] Step 6:

[0510] If the target is audio information, the user uses a microphone to capture audio data, which is input to the server and passed to the next step.

[0511] Step 7:

[0512] The server passes the captured voice data as input to a speech recognition module, which converts the voice into text. Specifically, it uses the speech_recognition library and the Google Speech API to analyze the voice data and extract text information. The output of this process is the converted text information.

[0513] Step 8:

[0514] The extracted text information is passed as input to the emotion engine on the server, which adjusts the display format based on the user's emotional state. Specifically, the font size and color are changed based on the emotional state analyzed by EmotionRecognizer. The output of this process is a text format adapted to the emotion.

[0515] Step 9:

[0516] The server outputs the adjusted text format to the terminal display and provides the information to the user. The input is in the adjusted text format, and by displaying this data on the display, the user receives the information visually. This step completes the process.

[0517] As a result, a system is provided that allows visually or hearing impaired users to safely and effectively make electronic payments.

[0518] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0519] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0520] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0521] [Second embodiment]

[0522] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0523] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0524] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0525] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0526] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0527] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0528] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0529] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0530] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0531] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0532] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0533] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0534] The present invention provides an apparatus and means for assisting people with visual and hearing impairments. The system includes a camera and a microphone, and a set of means for capturing visual and audio information and providing each of them to the user in an appropriate format.

[0535] System configuration and operation

[0536] visual information processing

[0537] server

[0538] The server initializes a camera to capture visual information and an OCR module to analyze the captured image.

[0539] The image obtained from the camera is passed to the OCR module to extract text information.

[0540] Terminal

[0541] The terminal has a voice output device that converts text information sent from the server into voice and reads it out to the user.

[0542] User

[0543] A visually impaired user can understand the captured visual information by listening to the audio information provided by the system.

[0544] Auditory information processing

[0545] server

[0546] The server initializes a microphone to capture audio information and a speech recognition module to convert speech to text.

[0547] The voice data acquired by the microphone is passed to a voice recognition module and converted into text information.

[0548] Terminal

[0549] The terminal has a display that displays the text information sent from the server.

[0550] User

[0551] A hearing-impaired user can understand the captured audio information by reading the text information provided by the system.

[0552] Specific examples

[0553] Example 1: Use in a bookstore

[0554] User

[0555] When a visually impaired user is in a bookstore and wants to know the contents of a book, he or she uses a camera to take a picture of the book's cover.

[0556] Terminal

[0557] The image captured by the camera is sent to the server, where text information is extracted using an OCR module.

[0558] The extracted text information is converted into audio and read aloud to the user.

[0559] server

[0560] The server analyzes the visual information and generates audio data such as the book title and author name.

[0561] Example 2: Understanding everyday conversation

[0562] User

[0563] If a hearing impaired user wants to know the content of everyday conversations, he or she can use a microphone to record the conversations.

[0564] Terminal

[0565] The voice data recorded by the microphone is sent to the server and converted into text information by a voice recognition module.

[0566] The converted text information is displayed on the terminal display so that the user can check it.

[0567] server

[0568] The server processes the captured audio information, converts it into text information in real time, and transmits it.

[0569] Program processing description

[0570] server

[0571] When the server starts up, it loads the necessary libraries and modules and initializes the camera and microphone.

[0572] The server periodically captures visual information from the camera, analyzes it using an OCR module, and converts it into text information.

[0573] Similarly, audio information is captured from the microphone, analyzed by a voice recognition module, and converted into text information.

[0574] Terminal

[0575] The terminal provides the text information received from the server to the user through voice output or display.

[0576] The system of the present invention can provide necessary information in real time to users with visual and hearing impairments, thereby increasing their independence in daily life and overcoming obstacles to information acquisition.

[0577] The processing flow will be explained below.

[0578] Specific processing explanation of the program

[0579] visual information processing

[0580] Step 1:

[0581] The server initializes the system, loading the necessary libraries and modules (OpenCV, OCR module, etc.) and initializing the camera.

[0582] Step 2:

[0583] It uses the device's camera to capture visual information: when the user points an object at the camera, the camera takes a picture of it.

[0584] Step 3:

[0585] The server sends the captured image data to the OCR module, which analyzes the image data and extracts text information.

[0586] Step 4:

[0587] The server sends the extracted text information to a text-to-speech (TTS) engine, which converts the text information into audio data.

[0588] Step 5:

[0589] The device plays the audio data to the user, allowing the user to obtain and understand the visual information as audio.

[0590] Auditory information processing

[0591] Step 1:

[0592] The server initializes the system, loading the necessary libraries and modules (SpeechRecognition, voice recognition module, etc.) and initializing the microphone.

[0593] Step 2:

[0594] Use the device's microphone to capture audio information, either by the user speaking into the microphone or by recording ambient sounds.

[0595] Step 3:

[0596] The server sends the captured voice data to a voice recognition module, which analyzes the voice data and generates text information.

[0597] Step 4:

[0598] The server transmits the generated text information to the terminal.

[0599] Step 5:

[0600] The device displays the text information on the display, allowing the user to receive and understand the audio information as text.

[0601] Specific examples

[0602] Example 1: Use in a bookstore

[0603] Step 1:

[0604] The server initializes the camera and OCR module.

[0605] Step 2:

[0606] The user captures the book cover through the device's camera.

[0607] Step 3:

[0608] The server analyzes the image data using an OCR module and extracts text information.

[0609] Step 4:

[0610] The server converts the extracted text information into speech using a TTS engine.

[0611] Step 5:

[0612] The terminal plays the audio data to the user, and the user understands the book title and author name by listening to the audio.

[0613] Example 2: Understanding everyday conversation

[0614] Step 1:

[0615] The server initializes the microphone and voice recognition module.

[0616] Step 2:

[0617] The user captures the audio of the conversation through the device's microphone.

[0618] Step 3:

[0619] The server analyzes the voice data using a voice recognition module and generates text information.

[0620] Step 4:

[0621] The server transmits the generated text information to the terminal.

[0622] Step 5:

[0623] The terminal displays text information on the display, and the user understands the content of the conversation through text.

[0624] In this way, visual and auditory information is processed sequentially along each step, making it possible to provide the user with the information they need in an appropriate format.

[0625] Example 1

[0626] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0627] In modern society, people with visual or hearing impairments face significant challenges in obtaining and understanding information. In particular, visually impaired people lack the means to receive visual information as audio, while hearing impaired people have limited means to receive audio information as text. Effective systems are needed to overcome these obstacles and improve users' independence and quality of life.

[0628] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0629] In this invention, the server includes means having an imaging device for capturing visual information, means for analyzing the captured visual information and extracting text information, means for converting the extracted text information into speech and outputting it, means for generating speech using a language conversion device for generating speech, means having an acoustic collection device for capturing speech information, means for converting the captured speech information into text information, means for displaying the converted text information, means using a speech recognition device for analyzing the speech information in real time, and means including a communication device for providing information to users in real time, thereby enabling users with visual and hearing impairments to understand the captured visual and speech information in real time and overcome obstacles to information acquisition.

[0630] "Visual information" refers to image data captured by an imaging device such as a camera.

[0631] "Imaging device" refers generically to cameras and other devices for capturing visual data.

[0632] "Capture" refers to the process of obtaining visual or audio information.

[0633] "OCR (Optical Character Recognition)" refers to a technology for extracting text information from image data.

[0634] "Audio information" refers to audio data captured by an audio collecting device such as a microphone.

[0635] "Sound collection device" refers to a general term for microphones and other devices used to capture audio data.

[0636] "Speech recognition device" refers to technology or devices for converting voice information into text information.

[0637] "User" refers to a person with a visual or hearing impairment who wishes to obtain information using this system.

[0638] "Means for converting into voice and outputting" refers to a general term for devices and programs that convert extracted text information into voice information and provide it to the user.

[0639] "Language conversion device" refers to technology or devices for converting text data into voice data.

[0640] "Text information" refers to character data extracted or converted from visual or audio information.

[0641] "Means for displaying" refers to a display or other device for visually presenting textual information to a user.

[0642] "Communication means" refers to the protocols and technologies used to send and receive data between the server and the terminal.

[0643] "Providing in real time" refers to rapidly processing captured information and providing it to the user simultaneously.

[0644] This invention is a system for assisting people with visual or hearing impairments, and aims to overcome barriers to information acquisition by capturing visual and audio information and providing each to the user in an appropriate format.

[0645] First, we will explain how visual information is processed. The server initializes an imaging device to capture visual information and loads an OCR (Optical Character Recognition) module to analyze the captured image data. The image data obtained from the camera is converted into text information using the OCR module. Specifically, the camera is initialized using Python's cv2 library, and OCR analysis is performed using the pytesseract library. This text information is sent to the device via an HTTP request or WebSocket.

[0646] The device converts the text information received from the server into audio using a Text-to-Speech engine (e.g., gTTS) and provides the audio data to the user through a built-in speaker. A visually impaired user can understand the captured visual information through this audio information.

[0647] Next, we will explain how to process auditory information. The server initializes an audio collection device to capture audio information and loads a speech recognition module to convert the audio into text. The audio data captured by the microphone is converted into text information using the speech recognition module. Specifically, the microphone is initialized using the pyaudio library, and speech recognition is performed using Google Cloud Speech-to-Text. This text information is sent to the device via an HTTP request or WebSocket.

[0648] The terminal displays the text information received from the server on a display. The display uses GUI libraries such as tkinter and PyQt to display the text information in a format that is easy for users to read. Hearing-impaired users can understand the captured audio information by reading this text information.

[0649] As a concrete example, consider a bookstore. A visually impaired user takes a picture of a book cover using a camera. The image data is sent to a server, where an OCR module extracts text information (e.g., the book title and author's name). The extracted text information is converted into audio and read aloud to the user. The server analyzes the visual information and provides the content to the user via audio.

[0650] Let's take a concrete example of understanding everyday conversation. If a hearing-impaired user wants to know the content of an everyday conversation, they can record the conversation using a microphone. The audio data is sent to a server, where it is converted into text information by a speech recognition module. The converted text information is then displayed on the device display for the user to confirm. The server then converts the captured audio information into text in real time and provides the content to the user.

[0651] An example of a prompt is:

[0652] "Please explain how a system that reads the contents of books aloud in bookstores for the visually impaired works. Please provide detailed steps from camera initialization to audio output."

[0653] "Please explain how a system that displays everyday conversations as text for the hearing impaired works. Please provide detailed steps from microphone initialization to text display."

[0654] Examples include:

[0655] In this way, the system of the present invention can provide visually and hearing impaired users with the information they need in real time, increasing their independence.

[0656] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0657] visual information processing

[0658] Step 1:

[0659] Server: Initialize the imaging device (camera). Using the Python cv2 library, initialize the camera with the cv2.VideoCapture(0) command. This initialization process allows the camera to function properly and enable image capture. The input is the camera device, and the output is the initialized camera object.

[0660] Step 2:

[0661] Server: Captures image data. Periodically, it uses the cap.read() command to obtain image data from the camera. The obtained image data is passed to the subsequent OCR module. The input is the initialized camera object, and the output is the captured image data.

[0662] Step 3:

[0663] Server: Pass the captured image data to the OCR module and extract text information. Specifically, use the pytesseract library to extract text from the image data with the pytesseract.image_to_string(image) command. The input is the captured image data, and the output is the extracted text data.

[0664] Step 4:

[0665] Server: Send the extracted text data to the terminal via an HTTP request or WebSocket. Use the requests.post(url, data={'text': text}) command. The input is the extracted text data, and the output is the text data sent to the terminal.

[0666] Step 5:

[0667] Terminal: The received text data is passed to the Text-to-Speech engine to generate audio data. The gTTS library is used to generate audio data using the gTTS(text) command. The generated audio file is played through the built-in speaker. The input is the text data received from the server, and the output is the generated audio data.

[0668] Step 6:

[0669] User: Understands the captured visual information by listening to the audio information provided by the device. The input is the audio data played from the device, and the output is the information comprehension obtained by listening to the audio data.

[0670] Auditory information processing

[0671] Step 1:

[0672] Server: Initialize the sound collection device (microphone). Use Python's pyaudio library to initialize the microphone with the audio = pyaudio.PyAudio() command. The input is the microphone device, and the output is the initialized microphone object.

[0673] Step 2:

[0674] Server: Captures audio data. Periodically obtains audio data from the microphone using the audio.open() and stream.read() commands. The obtained audio data is passed to the subsequent speech recognition module. The input is the initialized microphone object, and the output is the captured audio data.

[0675] Step 3:

[0676] Server: Pass the captured voice data to the speech recognition module and convert it into text information. Use Google Cloud Speech-to-Text to convert the voice data into text using the speech_to_text(audio_data) command. The input is the captured voice data, and the output is the converted text data.

[0677] Step 4:

[0678] Server: The converted text data is sent to the terminal via an HTTP request or WebSocket. Use the requests.post(url, data={'text': text}) command. The input is the converted text data, and the output is the text data sent to the terminal.

[0679] Step 5:

[0680] Terminal: The received text data is displayed on the screen. Using a GUI library such as tkinter or PyQt, the text data is displayed with the label.config(text=text) command. The input is the text data received from the server, and the output is the text data displayed on the screen.

[0681] Step 6:

[0682] User: Understands the captured audio information by reading the text information displayed on the display. The input is the text data displayed on the display, and the output is the understanding of the information obtained by reading the text data.

[0683] (Application example 1)

[0684] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0685] People with visual or hearing impairments have difficulty understanding product information and conversations in virtual stores. To address this issue, a system is needed that can convert visual information into speech and speech into text in real time.

[0686] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0687] In this invention, the server is a device equipped with a camera for capturing visual information, including means for analyzing the captured visual information and extracting text information, means for converting the extracted text information into audio and outputting it, means for saving the extracted text information as an audio file, and means for playing the saved audio file; a device equipped with an acoustic sensor for capturing audio information, including means for converting the captured audio information into text information, means for displaying the converted text information, and means for displaying the text information on a display of a user's terminal; a device equipped with both a camera and an acoustic sensor, including means for converting the extracted text information from text to audio and saving it as an audio file, means for playing the saved audio file, and means for using a generative AI model to generate prompt sentences. This enables users with visual or hearing impairments to intuitively and easily understand product information and conversations in a virtual store.

[0688] "Visual information" refers to image or video information captured using a camera or other photographic device.

[0689] "Image capture device" refers to a camera or other device for capturing visual information.

[0690] "Text information" refers to text data extracted from visual information.

[0691] "Audio information" refers to audio data captured using a microphone or acoustic sensor.

[0692] "Acoustic sensor" refers to a microphone or other device for capturing audio information.

[0693] "Transformation" refers to the act or process of transforming information into a different form.

[0694] "Users" refers to visually and hearing impaired people who use the system.

[0695] "Terminal" refers to a device used by a user, such as a smartphone or tablet.

[0696] "Generative AI model" refers to an artificial intelligence model for generating prompts and other text.

[0697] A "prompt" refers to the text of instructions or questions that are input to a generative AI model.

[0698] "Audio File" means a digital file that stores audio information.

[0699] This invention is a system for supporting people with visual or hearing impairments, making it easier for them to understand product information and conversations in a virtual store.

[0700] visual information processing

[0701] The user takes a picture of a product image or screen using a camera to capture visual information. The server analyzes the captured visual information using OCR (optical character recognition) technology and extracts text information. This text information is then used to generate a prompt using a generative AI model, which is then converted into audio and saved as an audio file. The device then plays the saved audio file, allowing the user to listen to the product description.

[0702] Auditory information processing

[0703] Using an acoustic sensor (microphone) to capture auditory information, the user captures audio information in the virtual store (for example, conversations with sales staff). The server analyzes the captured audio information using voice recognition technology and converts it into text information. This text information is displayed on the user's device display, allowing the user to confirm the information provided by voice as text.

[0704] System Configuration

[0705] 1. Server

[0706] The server analyzes the captured visual information using an OCR module (e.g., pytesseract) and converts it into text information.

[0707] The server analyzes the captured voice information using a voice recognition module (e.g., speech_recognition) and converts it into text information.

[0708] The text information is processed by a generative AI model to generate a prompt sentence.

[0709] To convert the generated text information into audio, an audio file is created and saved using Google Text-to-Speech (gTTS).

[0710] 2. Terminal

[0711] The user's terminal plays the audio file sent from the server and provides the user with information by voice.

[0712] The user's terminal displays the text information sent from the server on a display, allowing the user to confirm the text information.

[0713] Specific examples

[0714] Example 1: Retrieving product information

[0715] While a user is viewing a product page in a virtual store, they take a picture of the screen with their smartphone camera. The image is sent to a server, where the product description is converted into text using OCR. A generative AI model generates a prompt, which is then converted into speech and played back to the user.

[0716] Example 2: Understanding conversation content

[0717] The user records their conversation with the sales staff in the virtual store, and the audio data is sent to the server, where it is converted into text and displayed on the device's display. The user can understand the content of the conversation by reading the text.

[0718] Prompt Sentence Examples

[0719] "Extract Japanese characters from the input image file and convert the content into an audio file (MP3 format). Also, play the audio file and display the extracted text."

[0720] This system enables visually or hearing impaired users to understand product information and conversations in a virtual store in real time, allowing them to enjoy a comfortable shopping experience.

[0721] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0722] Step 1:

[0723] The user uses the smartphone camera to take pictures of the virtual store screen and product images. The input is image data from the camera. The output is the captured image file.

[0724] Step 2:

[0725] The terminal sends the captured image file to the server. The input is the image file. The output is the image data sent to the server.

[0726] Step 3:

[0727] The server parses the received image file into text using an OCR module (e.g., pytesseract). The input is the image data received by the server. The output is the extracted text information.

[0728] Step 4:

[0729] The server uses a generative AI model to generate a prompt from the extracted text, and then converts it into an audio file using Google Text-to-Speech (gTTS). The input is the extracted text, and the output is the generated prompt and audio file.

[0730] Step 5:

[0731] The server sends the generated audio file to the terminal. The input is the audio file. The output is the audio data sent to the terminal.

[0732] Step 6:

[0733] The terminal plays the received audio file and provides audio information to the user. The input is the audio data sent from the server. The output is the audio played back from the audio data.

[0734] Step 7:

[0735] A user uses the microphone on their smartphone to record conversations in a virtual store. The input is the audio data of the conversation. The output is a recorded audio file.

[0736] Step 8:

[0737] The terminal sends the recorded audio file to the server. The input is the recorded audio file. The output is the audio data sent to the server.

[0738] Step 9:

[0739] The server converts the received audio file into text using a speech recognition module (e.g., speech_recognition). The input is the received audio data. The output is the converted text.

[0740] Step 10:

[0741] The server sends the converted character information to the terminal. The input is the converted character information. The output is the character information sent to the terminal.

[0742] Step 11:

[0743] The terminal displays the received text information on the display and provides the text information to the user. The input is the text information sent from the server. The output is the text information displayed on the display.

[0744] This allows users with visual or hearing impairments to understand product information and conversations in a virtual store in real time.

[0745] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0746] This invention provides a system that combines an emotion engine that recognizes the user's emotions with a device for assisting people with visual or hearing impairments. The system is equipped with a camera and a microphone, and includes a series of means for acquiring visual and audio information in real time and providing it to the user in an appropriate format. Furthermore, the emotion engine recognizes the user's emotions and adjusts the information output based on the results.

[0747] System configuration and operation

[0748] visual information processing

[0749] server

[0750] The server initializes a camera to capture visual information and an OCR module to analyze the captured image.

[0751] The image obtained from the camera is passed to the OCR module to extract text information.

[0752] The extracted text information is sent to an emotion engine, which selects an appropriate voice output format depending on the user's emotional state.

[0753] Terminal

[0754] The terminal is equipped with a voice output device that converts text information sent from the server into voice and reads it out to the user. The emotion engine adjusts the tone and speed of the voice according to the user's emotions.

[0755] User

[0756] A visually impaired user can understand the captured visual information by listening to audio information provided by the system, for example, a slower voice when the user is tired, or a clearer voice when the user is energetic.

[0757] Auditory information processing

[0758] server

[0759] The server initializes a microphone to capture audio information and a speech recognition module to convert speech to text.

[0760] The voice data acquired by the microphone is passed to a voice recognition module and converted into text information.

[0761] The generated text information is sent to an emotion engine, which adjusts the format and size of the text display according to the user's emotional state.

[0762] Terminal

[0763] The terminal has a display that displays text information sent from the server, and the emotion engine adjusts the font size and color of the text according to the user's emotions.

[0764] User

[0765] A hearing-impaired user can understand the captured audio information by reading the text information provided by the system, for example, in normal font size when the user is calm and in larger font size when the user is anxious.

[0766] Specific examples

[0767] Example 1: Use in a bookstore

[0768] User

[0769] When a visually impaired user is in a bookstore and wants to know the contents of a book, he or she uses a camera to take a picture of the book's cover.

[0770] Terminal

[0771] The image captured by the camera is sent to the server, where text information is extracted using an OCR module.

[0772] The extracted text information is sent to the TTS engine via the emotion engine, and is read aloud to the user in a voice format that corresponds to the user's emotion.

[0773] server

[0774] The server analyzes visual information, generates audio data such as the book title and author's name, and uses an emotion engine to select the optimal audio output that takes the user's emotions into consideration.

[0775] Example 2: Understanding everyday conversation

[0776] User

[0777] If a hearing impaired user wants to know the content of everyday conversations, he or she can use a microphone to record the conversations.

[0778] Terminal

[0779] The voice data recorded by the microphone is sent to the server and converted into text information by a voice recognition module.

[0780] The converted text information is sent to the terminal via the emotion engine and displayed on the display in a format that corresponds to the user's emotion.

[0781] server

[0782] The server processes the captured voice information, converts it into text information in real time, and selects the optimal text display format taking into account the user's emotions using an emotion engine.

[0783] In this way, by adding an emotion engine to the processing of visual and auditory information, it becomes possible to provide information flexibly according to the user's emotional state, further improving the user experience.

[0784] The processing flow will be explained below.

[0785] visual information processing

[0786] Step 1:

[0787] The server initializes the system, loading the necessary libraries and modules (OpenCV, OCR module, etc.) and initializing the camera.

[0788] Step 2:

[0789] The user shows the book cover to the device's camera, and the device uses the camera to capture the book cover.

[0790] Step 3:

[0791] The server sends the captured image data to the OCR module to extract text information, which is then analyzed and generated.

[0792] Step 4:

[0793] The server sends the extracted text information to the emotion engine, which assesses the user's emotional state and adjusts the tone and speed of the speech accordingly.

[0794] Step 5:

[0795] The server sends the text information adjusted by the emotion engine to a TTS (text-to-speech) engine, which converts the text information into voice data.

[0796] Step 6:

[0797] The device plays the audio data back to the user. For example, if the device determines that the user is tired, it will read the data in a slower tone, and if the user is energetic, it will read the data in a clearer tone.

[0798] Auditory information processing

[0799] Step 1:

[0800] The server initializes the system, loading the necessary libraries and modules (SpeechRecognition, voice recognition module, etc.) and initializing the microphone.

[0801] Step 2:

[0802] The user speaks into the device's microphone or records the surrounding audio. The device captures the audio information.

[0803] Step 3:

[0804] The server sends the captured voice data to a voice recognition module to generate text information, which then analyzes the voice data and converts it into text information.

[0805] Step 4:

[0806] The server sends the generated text information to the emotion engine, which evaluates the user's emotional state and adjusts the format and size of the text display accordingly.

[0807] Step 5:

[0808] The server sends text information adjusted by the emotion engine to the terminal.

[0809] Step 6:

[0810] The device will display text information on the display, for example in a larger font size if it determines that the user is anxious, or in a normal font size if the user is calm.

[0811] Specific examples

[0812] Example 1: Use in a bookstore

[0813] Step 1:

[0814] The server initializes the camera and OCR module.

[0815] Step 2:

[0816] The user points the device camera at the book cover.

[0817] Step 3:

[0818] The device captures the book cover with its camera and sends the image data to the server.

[0819] Step 4:

[0820] The server analyzes the image data using an OCR module and extracts text information.

[0821] Step 5:

[0822] The server sends the text information to the emotion engine, which determines how to read it out based on the user's emotion.

[0823] Step 6:

[0824] The server converts the adjusted text information into voice data using a TTS engine.

[0825] Step 7:

[0826] The device plays the audio data to the user, for example, reading it in a slower tone if it determines that the user is tired.

[0827] Example 2: Understanding everyday conversation

[0828] Step 1:

[0829] The server initializes the microphone and voice recognition module.

[0830] Step 2:

[0831] The user speaks into the device microphone or captures the conversation audio.

[0832] Step 3:

[0833] The terminal transmits the voice information to the server.

[0834] Step 4:

[0835] The server analyzes the voice data using a voice recognition module and converts it into text information.

[0836] Step 5:

[0837] The server sends the generated text information to the emotion engine, which determines the format and size according to the user's emotion.

[0838] Step 6:

[0839] The server sends the adjusted text information to the terminal.

[0840] Step 7:

[0841] The device displays text information on the display, for example, in a larger font size if it determines that the user is in a hurry.

[0842] In this way, by combining the processing of visual and auditory information with an emotion engine, it becomes possible to provide information flexibly according to the user's emotional state, further improving the user experience.

[0843] Example 2

[0844] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0845] People with visual or hearing impairments have difficulty obtaining and understanding information in their daily lives. Furthermore, conventional information support systems do not take the user's emotional state into consideration, which results in a lack of improvement in the user experience. The objective of this invention is to provide flexible information provision according to the user's emotional state in such information support systems for the visually and hearing impaired, thereby improving the user experience.

[0846] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0847] In this invention, the server includes a camera for capturing visual information, means for analyzing the captured visual information and extracting text information, means for sending the extracted text information to an emotion engine for analyzing the user's emotion, and means for converting the text information into audio based on the analysis results of the emotion engine and adjusting the audio format for output. This makes it possible to provide information in an appropriate format that takes into account the user's emotional state when converting visual information into audio. The server also includes a microphone for capturing audio information, means for converting the captured audio information into text information, means for sending the converted text information to the emotion engine for analyzing the user's emotion, and means for adjusting and displaying the text format based on the analysis results of the emotion engine. This makes it possible to provide information in an appropriate format that takes into account the user's emotional state when converting audio information into text.

[0848] "Visual information" refers to image or video data captured using a device such as a camera.

[0849] A "camera" is a photographing device for acquiring visual information.

[0850] An "OCR (Optical Character Recognition) module" is a software module for extracting character information from captured image data.

[0851] "Audio information" refers to sound or voice data captured using a device such as a microphone.

[0852] A "microphone" is a sound pickup device for acquiring audio information.

[0853] An "emotion engine" is a software component for analyzing a user's emotional state.

[0854] A "voice recognition module" is a software module for converting captured voice data into text information.

[0855] "Text information" refers to data that can be represented and stored as characters.

[0856] A "TTS (Text-to-Speech) engine" is a software engine for converting text information into speech.

[0857] A "terminal" is a device used by a user to receive information, including an audio output device and a display.

[0858] A "server" is a centralized computing device for managing and executing system-wide processing.

[0859] A "display" is a screen device for visually presenting text information to a user.

[0860] "Format" refers to the form or style in which text or audio information is presented to the user appropriately.

[0861] MODE FOR CARRYING OUT THE INVENTION

[0862] This invention relates to a system for assisting people with visual or hearing impairments, and provides a set of means for acquiring visual and audio information in real time and providing the information in a flexible format according to the user's emotional state. The system is equipped with a camera and a microphone, acquires visual and audio information in real time, and provides the information to the user in an appropriate format. It also uses an emotion engine to analyze the user's emotional state and adjusts the information output format based on the results.

[0863] visual information processing

[0864] server

[0865] The server is equipped with a camera for capturing visual information. After the camera is initialized, it captures visual information (e.g., text or images). The captured image data is passed to an OCR (Optical Character Recognition) module for analysis. This analysis extracts text information from the image.

[0866] The server then sends the extracted text information to the emotion engine, which analyzes the user's emotional state and selects a voice format (tone, speed, etc.) based on the analysis results. For example, if the user is tired, a slower voice format is selected, and if the user is active, a clearer, crisper voice format is selected.

[0867] Terminal

[0868] The terminal is equipped with a voice output device and receives text information sent from the server. This text information is converted into voice using a TTS (Text-to-Speech) engine and read aloud to the user through the voice output device.

[0869] User

[0870] A visually impaired user can understand the content of a book by receiving visual information captured by a camera as audio. For example, a user can take a picture of a book cover in a bookstore and receive text information extracted from the captured image as audio.

[0871] Auditory information processing

[0872] server

[0873] The server is equipped with a microphone for capturing voice information. After the microphone is initialized, it captures voice information (e.g., conversation content). The captured voice data is passed to a voice recognition module and converted into text information. The converted text information is also sent to the emotion engine.

[0874] The emotion engine analyzes the user's emotional state and adjusts the text formatting (font size, color, etc.) based on the analysis. For example, if the user is calm, text will be displayed in normal font size, and if the user is anxious, text will be displayed in large font size.

[0875] Terminal

[0876] The device is equipped with a display and receives text information sent from the server. The received text information is displayed on the display in an appropriate format based on the analysis results of the emotion engine.

[0877] User

[0878] Hearing-impaired users can understand the content of audio information captured by a microphone by receiving it as text. For example, a user can record their everyday conversation, and text information generated from the recorded audio data is displayed on a screen.

[0879] Specific examples

[0880] Example 1: Use in a bookstore

[0881] User

[0882] When a visually impaired user is in a bookstore and wants to know the contents of a book, he or she uses a camera to take a picture of the book's cover.

[0883] Terminal

[0884] The image captured by the camera is sent to the server, where the OCR module extracts text information, which is then sent to the TTS engine via the emotion engine, where it is read aloud to the user in a voice format that corresponds to the user's emotion.

[0885] server

[0886] The server analyzes the visual information, generates text data such as the book title and author's name, and uses an emotion engine to select the optimal voice output based on the user's emotions.

[0887] Example 2: Understanding everyday conversation

[0888] User

[0889] If a hearing impaired user wants to know the content of everyday conversations, he or she can use a microphone to record the conversations.

[0890] Terminal

[0891] The voice data recorded by the microphone is sent to the server and converted into text information by a voice recognition module. The converted text information is sent to the device via an emotion engine and displayed on the screen in a format that corresponds to the user's emotion.

[0892] server

[0893] The server processes the captured audio information, converts it into text information in real time, and uses an emotion engine to select the optimal text display format according to the user's emotions.

[0894] In a specific embodiment of this invention, users with visual and hearing impairments can receive visual information as audio and audio information as text, and enjoy information in a flexible format that adapts to their emotional state. This system improves the user experience and makes it easier to obtain information in daily life.

[0895] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0896] visual information processing

[0897] Step 1:

[0898] server

[0899] The server initializes the camera to capture visual information, and the initialized camera is ready to capture visual information in real time.

[0900] Specific operation: The server sends an initialization command to the camera and confirms that the camera is ready for operation.

[0901] Step 2:

[0902] User

[0903] If the user is visually impaired, they place the source of information they want to know about (e.g., a book cover) in front of the camera and use the camera to photograph the object.

[0904] Specific Action: The user presses the camera button to capture an image.

[0905] Step 3:

[0906] server

[0907] The server receives the captured image data and passes it to the OCR module, which analyzes the image data and extracts text information. The input is the image data and the output is the extracted text information.

[0908] Specific operation: The server sends an API request to the OCR module to perform image analysis.

[0909] Step 4:

[0910] server

[0911] The server sends the extracted text information to the emotion engine, which analyzes the user's emotional state. The input is the text information, and the output is the emotion analysis result.

[0912] Specific operation: The server sends text data to the emotion engine and obtains the analysis results.

[0913] Step 5:

[0914] server

[0915] The server selects an audio format for converting text information into audio based on the emotion analysis results. The input is the emotion analysis results, and the output is the audio format.

[0916] What happens: The server selects the appropriate voice format (tone, speed, etc.) and prepares the data to send to the TTS engine.

[0917] Step 6:

[0918] Terminal

[0919] The terminal receives text information sent from the server and converts it into speech using a TTS engine. The input is text information and speech format, and the output is speech data.

[0920] Specific operation: The device sends an API request to the TTS engine and obtains the generated voice data.

[0921] Step 7:

[0922] Terminal

[0923] The terminal uses an audio output device to read the generated audio data to the user.

[0924] Specific operation: The terminal transmits audio data to the audio output device, and the user listens to the audio.

[0925] Auditory information processing

[0926] Step 1:

[0927] server

[0928] The server initializes the microphone to capture audio information. Once initialized, the microphone is ready to capture audio information in real time.

[0929] Specific operation: The server sends a microphone initialization command and verifies that the microphone is ready for operation.

[0930] Step 2:

[0931] User

[0932] If the user is hearing impaired, the desired conversation content is recorded using a microphone.

[0933] Specific action: The user presses the microphone button to capture audio.

[0934] Step 3:

[0935] server

[0936] The server receives the captured voice data and passes it to the voice recognition module, which analyzes the voice data and converts it into text information. The input is voice data and the output is text information.

[0937] Specific operation: The server sends an API request to the speech recognition module to perform speech analysis.

[0938] Step 4:

[0939] server

[0940] The server sends the generated text information to the emotion engine, which analyzes the user's emotional state. The input is the text information, and the output is the emotion analysis result.

[0941] Specific operation: The server sends text data to the emotion engine and obtains the analysis results.

[0942] Step 5:

[0943] server

[0944] The server selects the text format (font size, color, etc.) based on the sentiment analysis results. The input is the sentiment analysis results, and the output is the formatted text information.

[0945] Specific behavior: The server selects the appropriate text format and prepares the data to be displayed on the display.

[0946] Step 6:

[0947] Terminal

[0948] The terminal receives the formatted text information sent from the server and displays it on the display. The input is the formatted text information, and the output is the text displayed on the display.

[0949] Specific operation: The terminal sends formatted text information to the display and checks the display status.

[0950] Step 7:

[0951] User

[0952] The user understands the content of the conversation by reading the text information displayed on the display.

[0953] Specific operation: The user looks at the display and visually recognizes the text information.

[0954] The above are the specific processing steps of the system for processing visual and auditory information, which enables visually and auditory impaired users to obtain and understand information in real time.

[0955] (Application example 2)

[0956] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0957] The challenge is to solve the difficulties that visually and hearing impaired users have in safely and effectively performing complex operations such as electronic payments in their daily lives. Another problem is that the information provided is not adapted to the user's current emotional state, resulting in a poor user experience.

[0958] The specification processing by the specification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means having a camera for capturing visual information, means for analyzing the captured visual information and extracting text information, means for converting the extracted text information into voice and outputting it, an emotion engine for adjusting the extracted text information and the converted text information in accordance with the user's emotional state, means for outputting the information adjusted by the emotion engine to the user in an appropriate tone and speed, and means for providing the user with information captured in an electronic payment situation. This enables visually or hearing impaired users to receive appropriate information according to their emotional state when making electronic payments.

[0959] "Visual information" generally refers to video data captured using an image capture device such as a camera.

[0960] "Camera" refers to a photographic device used to capture visual information.

[0961] "Analysis" refers to the process of breaking down captured visual and audio information and extracting useful data.

[0962] "Text information" refers to character data extracted from visual or audio information.

[0963] "Extraction" refers to the process of extracting necessary information from analyzed data.

[0964] "Convert to speech" refers to the process of converting extracted text information into speech data.

[0965] "Output" refers to the act of conveying converted speech or displayed text to the user.

[0966] "Audio information" generally refers to sound data captured using an audio capture device such as a microphone.

[0967] "Microphone" refers to a sound pickup device used to capture audio information.

[0968] "Display" refers to visually conveying text information to a user using a display device or the like.

[0969] An "emotion engine" refers to a system that analyzes the user's emotional state and adjusts the format of information output based on the results.

[0970] "Adjustment" refers to the act of changing the output format or content depending on the user's emotional state.

[0971] "Tone" refers to the sound quality of voice, such as pitch, volume, and speed.

[0972] "Speed" refers to the speed at which voice or information is output.

[0973] "Electronic payment" refers to a payment transaction made using digital technology.

[0974] "Providing" refers to the act of giving information to a user.

[0975] The present invention realizes a system that enables users with visual or hearing impairments to make electronic payments safely and effectively. Specific embodiments are described below.

[0976] Hardware Configuration

[0977] The server is equipped with a camera to capture visual information and a microphone to capture audio information. The camera captures visual information and the microphone captures audio information. The terminal used by the user is equipped with a display and speaker.

[0978] Software Configuration

[0979] The server is installed with an OCR (Optical Character Recognition) module to analyze visual information and extract text information, and a speech recognition module to convert audio information into text. The speech synthesis engine pyttsx3 is used for reading out the audio, and EmotionRecognizer is used to recognize the user's emotional state.

[0980] Data processing and calculation

[0981] 1. Processing of visual information:

[0982] The server uses a camera to capture visual information. Images are captured in real time using the OpenCV library.

[0983] The acquired image data is subjected to OCR processing using pytesseract, and text information is extracted.

[0984] 2. Audio processing:

[0985] It uses a microphone to capture audio information. The speech_recognition library is used to obtain the audio data.

[0986] The captured voice data is converted into text information via the Google Speech API.

[0987] 3. Emotion recognition and output adjustment:

[0988] Using EmotionRecognizer, the user's emotional state is analyzed, which allows for assessment of the user's level of fatigue, energy, etc.

[0989] Based on the analysis results, the acquired text information is outputted using the speech synthesis engine pyttsx3 in an appropriate tone and speed, and the font size and color are also adjusted for text display.

[0990] Specific examples

[0991] As a concrete example, consider a visually impaired user making an electronic payment at a cafe. The user scans a QR code with their smartphone camera, and the server analyzes the image using OCR to extract text information. The emotion engine analyzes the user's emotional state, and if it determines that the user is tired, for example, it lowers the tone and speed of the voice to announce, "This code is for payment at the cafe."

[0992] Example of input prompt for generative AI model

[0993] "When a user scans a QR code with a camera, design a system that extracts the code's contents using an OCR module, determines the user's emotion using an emotion recognition engine, and provides information in an appropriate format based on that emotion. For example, if the user is tired, slow down the reading speed, and if the user is energetic, speed up the reading speed."

[0994] In this way, a support system can be provided that allows visually or hearing impaired users to make electronic payments with peace of mind.

[0995] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0996] Step 1:

[0997] A user uses a camera to capture visual information, and obtains an image containing a QR code and text information. The input is the image data captured by the camera, and this data is passed to the next step.

[0998] Step 2:

[0999] The server passes the captured image data as input to the OCR module to extract text information. Specifically, it reads the image using the OpenCV library and performs text extraction using pytesseract. The output of this process is the extracted text information.

[1000] Step 3:

[1001] The extracted text information is passed as input to the server's speech synthesis engine (pyttsx3) to prepare for conversion into speech. At this time, the server receives additional information from the camera or microphone as input and performs user emotion analysis using EmotionRecognizer to recognize the user's emotional state. The output of this process is emotional state data.

[1002] Step 4:

[1003] The server adjusts the tone and speed of the speech synthesis based on the emotional state data. Specifically, it sets pyttsx3 properties, for example, slowing down the speech speed if the user is tired and speeding up if the user is energetic. The output of this process is speech data adapted to the user's emotions.

[1004] Step 5:

[1005] The server outputs the adjusted voice data to the device's speaker and provides the information to the user. The input is the adjusted voice data, and this data is transmitted to the user as voice output. The processing is completed when the device's speaker plays the voice and the user receives the information.

[1006] Step 6:

[1007] If the target is audio information, the user uses a microphone to capture audio data, which is input to the server and passed to the next step.

[1008] Step 7:

[1009] The server passes the captured voice data as input to a speech recognition module, which converts the voice into text. Specifically, it uses the speech_recognition library and the Google Speech API to analyze the voice data and extract text information. The output of this process is the converted text information.

[1010] Step 8:

[1011] The extracted text information is passed as input to the emotion engine on the server, which adjusts the display format based on the user's emotional state. Specifically, the font size and color are changed based on the emotional state analyzed by EmotionRecognizer. The output of this process is a text format adapted to the emotion.

[1012] Step 9:

[1013] The server outputs the adjusted text format to the terminal display and provides the information to the user. The input is in the adjusted text format, and by displaying this data on the display, the user receives the information visually. This step completes the process.

[1014] As a result, a system is provided that allows visually or hearing impaired users to safely and effectively make electronic payments.

[1015] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1016] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1017] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1018] [Third embodiment]

[1019] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1020] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1021] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1022] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1023] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1024] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1025] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1026] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1027] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1028] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1029] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1030] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1031] The present invention provides an apparatus and means for assisting people with visual and hearing impairments. The system includes a camera and a microphone, and a set of means for capturing visual and audio information and providing each of them to the user in an appropriate format.

[1032] System configuration and operation

[1033] visual information processing

[1034] server

[1035] The server initializes a camera to capture visual information and an OCR module to analyze the captured image.

[1036] The image obtained from the camera is passed to the OCR module to extract text information.

[1037] Terminal

[1038] The terminal has a voice output device that converts text information sent from the server into voice and reads it out to the user.

[1039] User

[1040] A visually impaired user can understand the captured visual information by listening to the audio information provided by the system.

[1041] Auditory information processing

[1042] server

[1043] The server initializes a microphone to capture audio information and a speech recognition module to convert speech to text.

[1044] The voice data acquired by the microphone is passed to a voice recognition module and converted into text information.

[1045] Terminal

[1046] The terminal has a display that displays the text information sent from the server.

[1047] User

[1048] A hearing-impaired user can understand the captured audio information by reading the text information provided by the system.

[1049] Specific examples

[1050] Example 1: Use in a bookstore

[1051] User

[1052] When a visually impaired user is in a bookstore and wants to know the contents of a book, he or she uses a camera to take a picture of the book's cover.

[1053] Terminal

[1054] The image captured by the camera is sent to the server, where text information is extracted using an OCR module.

[1055] The extracted text information is converted into audio and read aloud to the user.

[1056] server

[1057] The server analyzes the visual information and generates audio data such as the book title and author name.

[1058] Example 2: Understanding everyday conversation

[1059] User

[1060] If a hearing impaired user wants to know the content of everyday conversations, he or she can use a microphone to record the conversations.

[1061] Terminal

[1062] The voice data recorded by the microphone is sent to the server and converted into text information by a voice recognition module.

[1063] The converted text information is displayed on the terminal display so that the user can check it.

[1064] server

[1065] The server processes the captured audio information, converts it into text information in real time, and transmits it.

[1066] Program processing description

[1067] server

[1068] When the server starts up, it loads the necessary libraries and modules and initializes the camera and microphone.

[1069] The server periodically captures visual information from the camera, analyzes it using an OCR module, and converts it into text information.

[1070] Similarly, audio information is captured from the microphone, analyzed by a voice recognition module, and converted into text information.

[1071] Terminal

[1072] The terminal provides the text information received from the server to the user through voice output or display.

[1073] The system of the present invention can provide necessary information in real time to users with visual and hearing impairments, thereby increasing their independence in daily life and overcoming obstacles to information acquisition.

[1074] The processing flow will be explained below.

[1075] Specific processing explanation of the program

[1076] visual information processing

[1077] Step 1:

[1078] The server initializes the system, loading the necessary libraries and modules (OpenCV, OCR module, etc.) and initializing the camera.

[1079] Step 2:

[1080] It uses the device's camera to capture visual information: when the user points an object at the camera, the camera takes a picture of it.

[1081] Step 3:

[1082] The server sends the captured image data to the OCR module, which analyzes the image data and extracts text information.

[1083] Step 4:

[1084] The server sends the extracted text information to a text-to-speech (TTS) engine, which converts the text information into audio data.

[1085] Step 5:

[1086] The device plays the audio data to the user, allowing the user to obtain and understand the visual information as audio.

[1087] Auditory information processing

[1088] Step 1:

[1089] The server initializes the system, loading the necessary libraries and modules (SpeechRecognition, voice recognition module, etc.) and initializing the microphone.

[1090] Step 2:

[1091] Use the device's microphone to capture audio information, either by the user speaking into the microphone or by recording ambient sounds.

[1092] Step 3:

[1093] The server sends the captured voice data to a voice recognition module, which analyzes the voice data and generates text information.

[1094] Step 4:

[1095] The server transmits the generated text information to the terminal.

[1096] Step 5:

[1097] The device displays the text information on the display, allowing the user to receive and understand the audio information as text.

[1098] Specific examples

[1099] Example 1: Use in a bookstore

[1100] Step 1:

[1101] The server initializes the camera and OCR module.

[1102] Step 2:

[1103] The user captures the book cover through the device's camera.

[1104] Step 3:

[1105] The server analyzes the image data using an OCR module and extracts text information.

[1106] Step 4:

[1107] The server converts the extracted text information into speech using a TTS engine.

[1108] Step 5:

[1109] The terminal plays the audio data to the user, and the user understands the book title and author name by listening to the audio.

[1110] Example 2: Understanding everyday conversation

[1111] Step 1:

[1112] The server initializes the microphone and voice recognition module.

[1113] Step 2:

[1114] The user captures the audio of the conversation through the device's microphone.

[1115] Step 3:

[1116] The server analyzes the voice data using a voice recognition module and generates text information.

[1117] Step 4:

[1118] The server transmits the generated text information to the terminal.

[1119] Step 5:

[1120] The terminal displays text information on the display, and the user understands the content of the conversation through text.

[1121] In this way, visual and auditory information is processed sequentially along each step, making it possible to provide the user with the information they need in an appropriate format.

[1122] Example 1

[1123] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1124] In modern society, people with visual or hearing impairments face significant challenges in obtaining and understanding information. In particular, visually impaired people lack the means to receive visual information as audio, while hearing impaired people have limited means to receive audio information as text. Effective systems are needed to overcome these obstacles and improve users' independence and quality of life.

[1125] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1126] In this invention, the server includes means having an imaging device for capturing visual information, means for analyzing the captured visual information and extracting text information, means for converting the extracted text information into speech and outputting it, means for generating speech using a language conversion device for generating speech, means having an acoustic collection device for capturing speech information, means for converting the captured speech information into text information, means for displaying the converted text information, means using a speech recognition device for analyzing the speech information in real time, and means including a communication device for providing information to users in real time, thereby enabling users with visual and hearing impairments to understand the captured visual and speech information in real time and overcome obstacles to information acquisition.

[1127] "Visual information" refers to image data captured by an imaging device such as a camera.

[1128] "Imaging device" refers generically to cameras and other devices for capturing visual data.

[1129] "Capture" refers to the process of obtaining visual or audio information.

[1130] "OCR (Optical Character Recognition)" refers to a technology for extracting text information from image data.

[1131] "Audio information" refers to audio data captured by an audio collecting device such as a microphone.

[1132] "Sound collection device" refers to a general term for microphones and other devices used to capture audio data.

[1133] "Speech recognition device" refers to technology or devices for converting voice information into text information.

[1134] "User" refers to a person with a visual or hearing impairment who wishes to obtain information using this system.

[1135] "Means for converting into voice and outputting" refers to a general term for devices and programs that convert extracted text information into voice information and provide it to the user.

[1136] "Language conversion device" refers to technology or devices for converting text data into voice data.

[1137] "Text information" refers to character data extracted or converted from visual or audio information.

[1138] "Means for displaying" refers to a display or other device for visually presenting textual information to a user.

[1139] "Communication means" refers to the protocols and technologies used to send and receive data between the server and the terminal.

[1140] "Providing in real time" refers to rapidly processing captured information and providing it to the user simultaneously.

[1141] This invention is a system for assisting people with visual or hearing impairments, and aims to overcome barriers to information acquisition by capturing visual and audio information and providing each to the user in an appropriate format.

[1142] First, we will explain how visual information is processed. The server initializes an imaging device to capture visual information and loads an OCR (Optical Character Recognition) module to analyze the captured image data. The image data obtained from the camera is converted into text information using the OCR module. Specifically, the camera is initialized using Python's cv2 library, and OCR analysis is performed using the pytesseract library. This text information is sent to the device via an HTTP request or WebSocket.

[1143] The device converts the text information received from the server into audio using a Text-to-Speech engine (e.g., gTTS) and provides the audio data to the user through a built-in speaker. A visually impaired user can understand the captured visual information through this audio information.

[1144] Next, we will explain how to process auditory information. The server initializes an audio collection device to capture audio information and loads a speech recognition module to convert the audio into text. The audio data captured by the microphone is converted into text information using the speech recognition module. Specifically, the microphone is initialized using the pyaudio library, and speech recognition is performed using Google Cloud Speech-to-Text. This text information is sent to the device via an HTTP request or WebSocket.

[1145] The terminal displays the text information received from the server on a display. The display uses GUI libraries such as tkinter and PyQt to display the text information in a format that is easy for users to read. Hearing-impaired users can understand the captured audio information by reading this text information.

[1146] As a concrete example, consider a bookstore. A visually impaired user takes a picture of a book cover using a camera. The image data is sent to a server, where an OCR module extracts text information (e.g., the book title and author's name). The extracted text information is converted into audio and read aloud to the user. The server analyzes the visual information and provides the content to the user via audio.

[1147] Let's take a concrete example of understanding everyday conversation. If a hearing-impaired user wants to know the content of an everyday conversation, they can record the conversation using a microphone. The audio data is sent to a server, where it is converted into text information by a speech recognition module. The converted text information is then displayed on the device display for the user to confirm. The server then converts the captured audio information into text in real time and provides the content to the user.

[1148] An example of a prompt is:

[1149] "Please explain how a system that reads the contents of books aloud in bookstores for the visually impaired works. Please provide detailed steps from camera initialization to audio output."

[1150] "Please explain how a system that displays everyday conversations as text for the hearing impaired works. Please provide detailed steps from microphone initialization to text display."

[1151] Examples include:

[1152] In this way, the system of the present invention can provide visually and hearing impaired users with the information they need in real time, increasing their independence.

[1153] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1154] visual information processing

[1155] Step 1:

[1156] Server: Initialize the imaging device (camera). Using the Python cv2 library, initialize the camera with the cv2.VideoCapture(0) command. This initialization process allows the camera to function properly and enable image capture. The input is the camera device, and the output is the initialized camera object.

[1157] Step 2:

[1158] Server: Captures image data. Periodically, it uses the cap.read() command to obtain image data from the camera. The obtained image data is passed to the subsequent OCR module. The input is the initialized camera object, and the output is the captured image data.

[1159] Step 3:

[1160] Server: Pass the captured image data to the OCR module and extract text information. Specifically, use the pytesseract library to extract text from the image data with the pytesseract.image_to_string(image) command. The input is the captured image data, and the output is the extracted text data.

[1161] Step 4:

[1162] Server: Send the extracted text data to the terminal via an HTTP request or WebSocket. Use the requests.post(url, data={'text': text}) command. The input is the extracted text data, and the output is the text data sent to the terminal.

[1163] Step 5:

[1164] Terminal: The received text data is passed to the Text-to-Speech engine to generate audio data. The gTTS library is used to generate audio data using the gTTS(text) command. The generated audio file is played through the built-in speaker. The input is the text data received from the server, and the output is the generated audio data.

[1165] Step 6:

[1166] User: Understands the captured visual information by listening to the audio information provided by the device. The input is the audio data played from the device, and the output is the information comprehension obtained by listening to the audio data.

[1167] Auditory information processing

[1168] Step 1:

[1169] Server: Initialize the sound collection device (microphone). Use Python's pyaudio library to initialize the microphone with the audio = pyaudio.PyAudio() command. The input is the microphone device, and the output is the initialized microphone object.

[1170] Step 2:

[1171] Server: Captures audio data. Periodically obtains audio data from the microphone using the audio.open() and stream.read() commands. The obtained audio data is passed to the subsequent speech recognition module. The input is the initialized microphone object, and the output is the captured audio data.

[1172] Step 3:

[1173] Server: Pass the captured voice data to the speech recognition module and convert it into text information. Use Google Cloud Speech-to-Text to convert the voice data into text using the speech_to_text(audio_data) command. The input is the captured voice data, and the output is the converted text data.

[1174] Step 4:

[1175] Server: The converted text data is sent to the terminal via an HTTP request or WebSocket. Use the requests.post(url, data={'text': text}) command. The input is the converted text data, and the output is the text data sent to the terminal.

[1176] Step 5:

[1177] Terminal: The received text data is displayed on the screen. Using a GUI library such as tkinter or PyQt, the text data is displayed with the label.config(text=text) command. The input is the text data received from the server, and the output is the text data displayed on the screen.

[1178] Step 6:

[1179] User: Understands the captured audio information by reading the text information displayed on the display. The input is the text data displayed on the display, and the output is the understanding of the information obtained by reading the text data.

[1180] (Application example 1)

[1181] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1182] People with visual or hearing impairments have difficulty understanding product information and conversations in virtual stores. To address this issue, a system is needed that can convert visual information into speech and speech into text in real time.

[1183] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1184] In this invention, the server is a device equipped with a camera for capturing visual information, including means for analyzing the captured visual information and extracting text information, means for converting the extracted text information into audio and outputting it, means for saving the extracted text information as an audio file, and means for playing the saved audio file; a device equipped with an acoustic sensor for capturing audio information, including means for converting the captured audio information into text information, means for displaying the converted text information, and means for displaying the text information on a display of a user's terminal; a device equipped with both a camera and an acoustic sensor, including means for converting the extracted text information from text to audio and saving it as an audio file, means for playing the saved audio file, and means for using a generative AI model to generate prompt sentences. This enables users with visual or hearing impairments to intuitively and easily understand product information and conversations in a virtual store.

[1185] "Visual information" refers to image or video information captured using a camera or other photographic device.

[1186] "Image capture device" refers to a camera or other device for capturing visual information.

[1187] "Text information" refers to text data extracted from visual information.

[1188] "Audio information" refers to audio data captured using a microphone or acoustic sensor.

[1189] "Acoustic sensor" refers to a microphone or other device for capturing audio information.

[1190] "Transformation" refers to the act or process of transforming information into a different form.

[1191] "Users" refers to visually and hearing impaired people who use the system.

[1192] "Terminal" refers to a device used by a user, such as a smartphone or tablet.

[1193] "Generative AI model" refers to an artificial intelligence model for generating prompts and other text.

[1194] A "prompt" refers to the text of instructions or questions that are input to a generative AI model.

[1195] "Audio File" means a digital file that stores audio information.

[1196] This invention is a system for supporting people with visual or hearing impairments, making it easier for them to understand product information and conversations in a virtual store.

[1197] visual information processing

[1198] The user takes a picture of a product image or screen using a camera to capture visual information. The server analyzes the captured visual information using OCR (optical character recognition) technology and extracts text information. This text information is then used to generate a prompt using a generative AI model, which is then converted into audio and saved as an audio file. The device then plays the saved audio file, allowing the user to listen to the product description.

[1199] Auditory information processing

[1200] Using an acoustic sensor (microphone) to capture auditory information, the user captures audio information in the virtual store (for example, conversations with sales staff). The server analyzes the captured audio information using voice recognition technology and converts it into text information. This text information is displayed on the user's device display, allowing the user to confirm the information provided by voice as text.

[1201] System Configuration

[1202] 1. Server

[1203] The server analyzes the captured visual information using an OCR module (e.g., pytesseract) and converts it into text information.

[1204] The server analyzes the captured voice information using a voice recognition module (e.g., speech_recognition) and converts it into text information.

[1205] The text information is processed by a generative AI model to generate a prompt sentence.

[1206] To convert the generated text information into audio, an audio file is created and saved using Google Text-to-Speech (gTTS).

[1207] 2. Terminal

[1208] The user's terminal plays the audio file sent from the server and provides the user with information by voice.

[1209] The user's terminal displays the text information sent from the server on a display, allowing the user to confirm the text information.

[1210] Specific examples

[1211] Example 1: Retrieving product information

[1212] While a user is viewing a product page in a virtual store, they take a picture of the screen with their smartphone camera. The image is sent to a server, where the product description is converted into text using OCR. A generative AI model generates a prompt, which is then converted into speech and played back to the user.

[1213] Example 2: Understanding conversation content

[1214] The user records their conversation with the sales staff in the virtual store, and the audio data is sent to the server, where it is converted into text and displayed on the device's display. The user can understand the content of the conversation by reading the text.

[1215] Prompt Sentence Examples

[1216] "Extract Japanese characters from the input image file and convert the content into an audio file (MP3 format). Also, play the audio file and display the extracted text."

[1217] This system enables visually or hearing impaired users to understand product information and conversations in a virtual store in real time, allowing them to enjoy a comfortable shopping experience.

[1218] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1219] Step 1:

[1220] The user uses the smartphone camera to take pictures of the virtual store screen and product images. The input is image data from the camera. The output is the captured image file.

[1221] Step 2:

[1222] The terminal sends the captured image file to the server. The input is the image file. The output is the image data sent to the server.

[1223] Step 3:

[1224] The server parses the received image file into text using an OCR module (e.g., pytesseract). The input is the image data received by the server. The output is the extracted text information.

[1225] Step 4:

[1226] The server uses a generative AI model to generate a prompt from the extracted text, and then converts it into an audio file using Google Text-to-Speech (gTTS). The input is the extracted text, and the output is the generated prompt and audio file.

[1227] Step 5:

[1228] The server sends the generated audio file to the terminal. The input is the audio file. The output is the audio data sent to the terminal.

[1229] Step 6:

[1230] The terminal plays the received audio file and provides audio information to the user. The input is the audio data sent from the server. The output is the audio played back from the audio data.

[1231] Step 7:

[1232] A user uses the microphone on their smartphone to record conversations in a virtual store. The input is the audio data of the conversation. The output is a recorded audio file.

[1233] Step 8:

[1234] The terminal sends the recorded audio file to the server. The input is the recorded audio file. The output is the audio data sent to the server.

[1235] Step 9:

[1236] The server converts the received audio file into text using a speech recognition module (e.g., speech_recognition). The input is the received audio data. The output is the converted text.

[1237] Step 10:

[1238] The server sends the converted character information to the terminal. The input is the converted character information. The output is the character information sent to the terminal.

[1239] Step 11:

[1240] The terminal displays the received text information on the display and provides the text information to the user. The input is the text information sent from the server. The output is the text information displayed on the display.

[1241] This allows users with visual or hearing impairments to understand product information and conversations in a virtual store in real time.

[1242] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1243] This invention provides a system that combines an emotion engine that recognizes the user's emotions with a device for assisting people with visual or hearing impairments. The system is equipped with a camera and a microphone, and includes a series of means for acquiring visual and audio information in real time and providing it to the user in an appropriate format. Furthermore, the emotion engine recognizes the user's emotions and adjusts the information output based on the results.

[1244] System configuration and operation

[1245] visual information processing

[1246] server

[1247] The server initializes a camera to capture visual information and an OCR module to analyze the captured image.

[1248] The image obtained from the camera is passed to the OCR module to extract text information.

[1249] The extracted text information is sent to an emotion engine, which selects an appropriate voice output format depending on the user's emotional state.

[1250] Terminal

[1251] The terminal is equipped with a voice output device that converts text information sent from the server into voice and reads it out to the user. The emotion engine adjusts the tone and speed of the voice according to the user's emotions.

[1252] User

[1253] A visually impaired user can understand the captured visual information by listening to audio information provided by the system, for example, a slower voice when the user is tired, or a clearer voice when the user is energetic.

[1254] Auditory information processing

[1255] server

[1256] The server initializes a microphone to capture audio information and a speech recognition module to convert speech to text.

[1257] The voice data acquired by the microphone is passed to a voice recognition module and converted into text information.

[1258] The generated text information is sent to an emotion engine, which adjusts the format and size of the text display according to the user's emotional state.

[1259] Terminal

[1260] The terminal has a display that displays text information sent from the server, and the emotion engine adjusts the font size and color of the text according to the user's emotions.

[1261] User

[1262] A hearing-impaired user can understand the captured audio information by reading the text information provided by the system, for example, in normal font size when the user is calm and in larger font size when the user is anxious.

[1263] Specific examples

[1264] Example 1: Use in a bookstore

[1265] User

[1266] When a visually impaired user is in a bookstore and wants to know the contents of a book, he or she uses a camera to take a picture of the book's cover.

[1267] Terminal

[1268] The image captured by the camera is sent to the server, where text information is extracted using an OCR module.

[1269] The extracted text information is sent to the TTS engine via the emotion engine, and is read aloud to the user in a voice format that corresponds to the user's emotion.

[1270] server

[1271] The server analyzes visual information, generates audio data such as the book title and author's name, and uses an emotion engine to select the optimal audio output that takes the user's emotions into consideration.

[1272] Example 2: Understanding everyday conversation

[1273] User

[1274] If a hearing impaired user wants to know the content of everyday conversations, he or she can use a microphone to record the conversations.

[1275] Terminal

[1276] The voice data recorded by the microphone is sent to the server and converted into text information by a voice recognition module.

[1277] The converted text information is sent to the terminal via the emotion engine and displayed on the display in a format that corresponds to the user's emotion.

[1278] server

[1279] The server processes the captured voice information, converts it into text information in real time, and selects the optimal text display format taking into account the user's emotions using an emotion engine.

[1280] In this way, by adding an emotion engine to the processing of visual and auditory information, it becomes possible to provide information flexibly according to the user's emotional state, further improving the user experience.

[1281] The processing flow will be explained below.

[1282] visual information processing

[1283] Step 1:

[1284] The server initializes the system, loading the necessary libraries and modules (OpenCV, OCR module, etc.) and initializing the camera.

[1285] Step 2:

[1286] The user shows the book cover to the device's camera, and the device uses the camera to capture the book cover.

[1287] Step 3:

[1288] The server sends the captured image data to the OCR module to extract text information, which is then analyzed and generated.

[1289] Step 4:

[1290] The server sends the extracted text information to the emotion engine, which assesses the user's emotional state and adjusts the tone and speed of the speech accordingly.

[1291] Step 5:

[1292] The server sends the text information adjusted by the emotion engine to a TTS (text-to-speech) engine, which converts the text information into voice data.

[1293] Step 6:

[1294] The device plays the audio data back to the user. For example, if the device determines that the user is tired, it will read the data in a slower tone, and if the user is energetic, it will read the data in a clearer tone.

[1295] Auditory information processing

[1296] Step 1:

[1297] The server initializes the system, loading the necessary libraries and modules (SpeechRecognition, voice recognition module, etc.) and initializing the microphone.

[1298] Step 2:

[1299] The user speaks into the device's microphone or records the surrounding audio. The device captures the audio information.

[1300] Step 3:

[1301] The server sends the captured voice data to a voice recognition module to generate text information, which then analyzes the voice data and converts it into text information.

[1302] Step 4:

[1303] The server sends the generated text information to the emotion engine, which evaluates the user's emotional state and adjusts the format and size of the text display accordingly.

[1304] Step 5:

[1305] The server sends text information adjusted by the emotion engine to the terminal.

[1306] Step 6:

[1307] The device will display text information on the display, for example in a larger font size if it determines that the user is anxious, or in a normal font size if the user is calm.

[1308] Specific examples

[1309] Example 1: Use in a bookstore

[1310] Step 1:

[1311] The server initializes the camera and OCR module.

[1312] Step 2:

[1313] The user points the device camera at the book cover.

[1314] Step 3:

[1315] The device captures the book cover with its camera and sends the image data to the server.

[1316] Step 4:

[1317] The server analyzes the image data using an OCR module and extracts text information.

[1318] Step 5:

[1319] The server sends the text information to the emotion engine, which determines how to read it out based on the user's emotion.

[1320] Step 6:

[1321] The server converts the adjusted text information into voice data using a TTS engine.

[1322] Step 7:

[1323] The device plays the audio data to the user, for example, reading it in a slower tone if it determines that the user is tired.

[1324] Example 2: Understanding everyday conversation

[1325] Step 1:

[1326] The server initializes the microphone and voice recognition module.

[1327] Step 2:

[1328] The user speaks into the device microphone or captures the conversation audio.

[1329] Step 3:

[1330] The terminal transmits the voice information to the server.

[1331] Step 4:

[1332] The server analyzes the voice data using a voice recognition module and converts it into text information.

[1333] Step 5:

[1334] The server sends the generated text information to the emotion engine, which determines the format and size according to the user's emotion.

[1335] Step 6:

[1336] The server sends the adjusted text information to the terminal.

[1337] Step 7:

[1338] The device displays text information on the display, for example, in a larger font size if it determines that the user is in a hurry.

[1339] In this way, by combining the processing of visual and auditory information with an emotion engine, it becomes possible to provide information flexibly according to the user's emotional state, further improving the user experience.

[1340] Example 2

[1341] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1342] People with visual or hearing impairments have difficulty obtaining and understanding information in their daily lives. Furthermore, conventional information support systems do not take the user's emotional state into consideration, which results in a lack of improvement in the user experience. The objective of this invention is to provide flexible information provision according to the user's emotional state in such information support systems for the visually and hearing impaired, thereby improving the user experience.

[1343] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1344] In this invention, the server includes a camera for capturing visual information, means for analyzing the captured visual information and extracting text information, means for sending the extracted text information to an emotion engine for analyzing the user's emotion, and means for converting the text information into audio based on the analysis results of the emotion engine and adjusting the audio format for output. This makes it possible to provide information in an appropriate format that takes into account the user's emotional state when converting visual information into audio. The server also includes a microphone for capturing audio information, means for converting the captured audio information into text information, means for sending the converted text information to the emotion engine for analyzing the user's emotion, and means for adjusting and displaying the text format based on the analysis results of the emotion engine. This makes it possible to provide information in an appropriate format that takes into account the user's emotional state when converting audio information into text.

[1345] "Visual information" refers to image or video data captured using a device such as a camera.

[1346] A "camera" is a photographing device for acquiring visual information.

[1347] An "OCR (Optical Character Recognition) module" is a software module for extracting character information from captured image data.

[1348] "Audio information" refers to sound or voice data captured using a device such as a microphone.

[1349] A "microphone" is a sound pickup device for acquiring audio information.

[1350] An "emotion engine" is a software component for analyzing a user's emotional state.

[1351] A "voice recognition module" is a software module for converting captured voice data into text information.

[1352] "Text information" refers to data that can be represented and stored as characters.

[1353] A "TTS (Text-to-Speech) engine" is a software engine for converting text information into speech.

[1354] A "terminal" is a device used by a user to receive information, including an audio output device and a display.

[1355] A "server" is a centralized computing device for managing and executing system-wide processing.

[1356] A "display" is a screen device for visually presenting text information to a user.

[1357] "Format" refers to the form or style in which text or audio information is presented to the user appropriately.

[1358] MODE FOR CARRYING OUT THE INVENTION

[1359] This invention relates to a system for assisting people with visual or hearing impairments, and provides a set of means for acquiring visual and audio information in real time and providing the information in a flexible format according to the user's emotional state. The system is equipped with a camera and a microphone, acquires visual and audio information in real time, and provides the information to the user in an appropriate format. It also uses an emotion engine to analyze the user's emotional state and adjusts the information output format based on the results.

[1360] visual information processing

[1361] server

[1362] The server is equipped with a camera for capturing visual information. After the camera is initialized, it captures visual information (e.g., text or images). The captured image data is passed to an OCR (Optical Character Recognition) module for analysis. This analysis extracts text information from the image.

[1363] The server then sends the extracted text information to the emotion engine, which analyzes the user's emotional state and selects a voice format (tone, speed, etc.) based on the analysis results. For example, if the user is tired, a slower voice format is selected, and if the user is active, a clearer, crisper voice format is selected.

[1364] Terminal

[1365] The terminal is equipped with a voice output device and receives text information sent from the server. This text information is converted into voice using a TTS (Text-to-Speech) engine and read aloud to the user through the voice output device.

[1366] User

[1367] A visually impaired user can understand the content of a book by receiving visual information captured by a camera as audio. For example, a user can take a picture of a book cover in a bookstore and receive text information extracted from the captured image as audio.

[1368] Auditory information processing

[1369] server

[1370] The server is equipped with a microphone for capturing voice information. After the microphone is initialized, it captures voice information (e.g., conversation content). The captured voice data is passed to a voice recognition module and converted into text information. The converted text information is also sent to the emotion engine.

[1371] The emotion engine analyzes the user's emotional state and adjusts the text formatting (font size, color, etc.) based on the analysis. For example, if the user is calm, text will be displayed in normal font size, and if the user is anxious, text will be displayed in large font size.

[1372] Terminal

[1373] The device is equipped with a display and receives text information sent from the server. The received text information is displayed on the display in an appropriate format based on the analysis results of the emotion engine.

[1374] User

[1375] Hearing-impaired users can understand the content of audio information captured by a microphone by receiving it as text. For example, a user can record their everyday conversation, and text information generated from the recorded audio data is displayed on a screen.

[1376] Specific examples

[1377] Example 1: Use in a bookstore

[1378] User

[1379] When a visually impaired user is in a bookstore and wants to know the contents of a book, he or she uses a camera to take a picture of the book's cover.

[1380] Terminal

[1381] The image captured by the camera is sent to the server, where the OCR module extracts text information, which is then sent to the TTS engine via the emotion engine, where it is read aloud to the user in a voice format that corresponds to the user's emotion.

[1382] server

[1383] The server analyzes the visual information, generates text data such as the book title and author's name, and uses an emotion engine to select the optimal voice output based on the user's emotions.

[1384] Example 2: Understanding everyday conversation

[1385] User

[1386] If a hearing impaired user wants to know the content of everyday conversations, he or she can use a microphone to record the conversations.

[1387] Terminal

[1388] The voice data recorded by the microphone is sent to the server and converted into text information by a voice recognition module. The converted text information is sent to the device via an emotion engine and displayed on the screen in a format that corresponds to the user's emotion.

[1389] server

[1390] The server processes the captured audio information, converts it into text information in real time, and uses an emotion engine to select the optimal text display format according to the user's emotions.

[1391] In a specific embodiment of this invention, users with visual and hearing impairments can receive visual information as audio and audio information as text, and enjoy information in a flexible format that adapts to their emotional state. This system improves the user experience and makes it easier to obtain information in daily life.

[1392] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1393] visual information processing

[1394] Step 1:

[1395] server

[1396] The server initializes the camera to capture visual information, and the initialized camera is ready to capture visual information in real time.

[1397] Specific operation: The server sends an initialization command to the camera and confirms that the camera is ready for operation.

[1398] Step 2:

[1399] User

[1400] If the user is visually impaired, they place the source of information they want to know about (e.g., a book cover) in front of the camera and use the camera to photograph the object.

[1401] Specific Action: The user presses the camera button to capture an image.

[1402] Step 3:

[1403] server

[1404] The server receives the captured image data and passes it to the OCR module, which analyzes the image data and extracts text information. The input is the image data and the output is the extracted text information.

[1405] Specific operation: The server sends an API request to the OCR module to perform image analysis.

[1406] Step 4:

[1407] server

[1408] The server sends the extracted text information to the emotion engine, which analyzes the user's emotional state. The input is the text information, and the output is the emotion analysis result.

[1409] Specific operation: The server sends text data to the emotion engine and obtains the analysis results.

[1410] Step 5:

[1411] server

[1412] The server selects an audio format for converting text information into audio based on the emotion analysis results. The input is the emotion analysis results, and the output is the audio format.

[1413] What happens: The server selects the appropriate voice format (tone, speed, etc.) and prepares the data to send to the TTS engine.

[1414] Step 6:

[1415] Terminal

[1416] The terminal receives text information sent from the server and converts it into speech using a TTS engine. The input is text information and speech format, and the output is speech data.

[1417] Specific operation: The device sends an API request to the TTS engine and obtains the generated voice data.

[1418] Step 7:

[1419] Terminal

[1420] The terminal uses an audio output device to read the generated audio data to the user.

[1421] Specific operation: The terminal transmits audio data to the audio output device, and the user listens to the audio.

[1422] Auditory information processing

[1423] Step 1:

[1424] server

[1425] The server initializes the microphone to capture audio information. Once initialized, the microphone is ready to capture audio information in real time.

[1426] Specific operation: The server sends a microphone initialization command and verifies that the microphone is ready for operation.

[1427] Step 2:

[1428] User

[1429] If the user is hearing impaired, the desired conversation content is recorded using a microphone.

[1430] Specific action: The user presses the microphone button to capture audio.

[1431] Step 3:

[1432] server

[1433] The server receives the captured voice data and passes it to the voice recognition module, which analyzes the voice data and converts it into text information. The input is voice data and the output is text information.

[1434] Specific operation: The server sends an API request to the speech recognition module to perform speech analysis.

[1435] Step 4:

[1436] server

[1437] The server sends the generated text information to the emotion engine, which analyzes the user's emotional state. The input is the text information, and the output is the emotion analysis result.

[1438] Specific operation: The server sends text data to the emotion engine and obtains the analysis results.

[1439] Step 5:

[1440] server

[1441] The server selects the text format (font size, color, etc.) based on the sentiment analysis results. The input is the sentiment analysis results, and the output is the formatted text information.

[1442] Specific behavior: The server selects the appropriate text format and prepares the data to be displayed on the display.

[1443] Step 6:

[1444] Terminal

[1445] The terminal receives the formatted text information sent from the server and displays it on the display. The input is the formatted text information, and the output is the text displayed on the display.

[1446] Specific operation: The terminal sends formatted text information to the display and checks the display status.

[1447] Step 7:

[1448] User

[1449] The user understands the content of the conversation by reading the text information displayed on the display.

[1450] Specific operation: The user looks at the display and visually recognizes the text information.

[1451] The above are the specific processing steps of the system for processing visual and auditory information, which enables visually and auditory impaired users to obtain and understand information in real time.

[1452] (Application example 2)

[1453] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1454] The challenge is to solve the difficulties that visually and hearing impaired users have in safely and effectively performing complex operations such as electronic payments in their daily lives. Another problem is that the information provided is not adapted to the user's current emotional state, resulting in a poor user experience.

[1455] The specification processing by the specification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means having a camera for capturing visual information, means for analyzing the captured visual information and extracting text information, means for converting the extracted text information into voice and outputting it, an emotion engine for adjusting the extracted text information and the converted text information in accordance with the user's emotional state, means for outputting the information adjusted by the emotion engine to the user in an appropriate tone and speed, and means for providing the user with information captured in an electronic payment situation. This enables visually or hearing impaired users to receive appropriate information according to their emotional state when making electronic payments.

[1456] "Visual information" generally refers to video data captured using an image capture device such as a camera.

[1457] "Camera" refers to a photographic device used to capture visual information.

[1458] "Analysis" refers to the process of breaking down captured visual and audio information and extracting useful data.

[1459] "Text information" refers to character data extracted from visual or audio information.

[1460] "Extraction" refers to the process of extracting necessary information from analyzed data.

[1461] "Convert to speech" refers to the process of converting extracted text information into speech data.

[1462] "Output" refers to the act of conveying converted speech or displayed text to the user.

[1463] "Audio information" generally refers to sound data captured using an audio capture device such as a microphone.

[1464] "Microphone" refers to a sound pickup device used to capture audio information.

[1465] "Display" refers to visually conveying text information to a user using a display device or the like.

[1466] An "emotion engine" refers to a system that analyzes the user's emotional state and adjusts the format of information output based on the results.

[1467] "Adjustment" refers to the act of changing the output format or content depending on the user's emotional state.

[1468] "Tone" refers to the sound quality of voice, such as pitch, volume, and speed.

[1469] "Speed" refers to the speed at which voice or information is output.

[1470] "Electronic payment" refers to a payment transaction made using digital technology.

[1471] "Providing" refers to the act of giving information to a user.

[1472] The present invention realizes a system that enables users with visual or hearing impairments to make electronic payments safely and effectively. Specific embodiments are described below.

[1473] Hardware Configuration

[1474] The server is equipped with a camera to capture visual information and a microphone to capture audio information. The camera captures visual information and the microphone captures audio information. The terminal used by the user is equipped with a display and speaker.

[1475] Software Configuration

[1476] The server is installed with an OCR (Optical Character Recognition) module to analyze visual information and extract text information, and a speech recognition module to convert audio information into text. The speech synthesis engine pyttsx3 is used for reading out the audio, and EmotionRecognizer is used to recognize the user's emotional state.

[1477] Data processing and calculation

[1478] 1. Processing of visual information:

[1479] The server uses a camera to capture visual information. Images are captured in real time using the OpenCV library.

[1480] The acquired image data is subjected to OCR processing using pytesseract, and text information is extracted.

[1481] 2. Audio processing:

[1482] It uses a microphone to capture audio information. The speech_recognition library is used to obtain the audio data.

[1483] The captured voice data is converted into text information via the Google Speech API.

[1484] 3. Emotion recognition and output adjustment:

[1485] Using EmotionRecognizer, the user's emotional state is analyzed, which allows for assessment of the user's level of fatigue, energy, etc.

[1486] Based on the analysis results, the acquired text information is outputted using the speech synthesis engine pyttsx3 in an appropriate tone and speed, and the font size and color are also adjusted for text display.

[1487] Specific examples

[1488] As a concrete example, consider a visually impaired user making an electronic payment at a cafe. The user scans a QR code with their smartphone camera, and the server analyzes the image using OCR to extract text information. The emotion engine analyzes the user's emotional state, and if it determines that the user is tired, for example, it lowers the tone and speed of the voice to announce, "This code is for payment at the cafe."

[1489] Example of input prompt for generative AI model

[1490] "When a user scans a QR code with a camera, design a system that extracts the code's contents using an OCR module, determines the user's emotion using an emotion recognition engine, and provides information in an appropriate format based on that emotion. For example, if the user is tired, slow down the reading speed, and if the user is energetic, speed up the reading speed."

[1491] In this way, a support system can be provided that allows visually or hearing impaired users to make electronic payments with peace of mind.

[1492] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1493] Step 1:

[1494] A user uses a camera to capture visual information, and obtains an image containing a QR code and text information. The input is the image data captured by the camera, and this data is passed to the next step.

[1495] Step 2:

[1496] The server passes the captured image data as input to the OCR module to extract text information. Specifically, it reads the image using the OpenCV library and performs text extraction using pytesseract. The output of this process is the extracted text information.

[1497] Step 3:

[1498] The extracted text information is passed as input to the server's speech synthesis engine (pyttsx3) to prepare for conversion into speech. At this time, the server receives additional information from the camera or microphone as input and performs user emotion analysis using EmotionRecognizer to recognize the user's emotional state. The output of this process is emotional state data.

[1499] Step 4:

[1500] The server adjusts the tone and speed of the speech synthesis based on the emotional state data. Specifically, it sets pyttsx3 properties, for example, slowing down the speech speed if the user is tired and speeding up if the user is energetic. The output of this process is speech data adapted to the user's emotions.

[1501] Step 5:

[1502] The server outputs the adjusted voice data to the device's speaker and provides the information to the user. The input is the adjusted voice data, and this data is transmitted to the user as voice output. The processing is completed when the device's speaker plays the voice and the user receives the information.

[1503] Step 6:

[1504] If the target is audio information, the user uses a microphone to capture audio data, which is input to the server and passed to the next step.

[1505] Step 7:

[1506] The server passes the captured voice data as input to a speech recognition module, which converts the voice into text. Specifically, it uses the speech_recognition library and the Google Speech API to analyze the voice data and extract text information. The output of this process is the converted text information.

[1507] Step 8:

[1508] The extracted text information is passed as input to the emotion engine on the server, which adjusts the display format based on the user's emotional state. Specifically, the font size and color are changed based on the emotional state analyzed by EmotionRecognizer. The output of this process is a text format adapted to the emotion.

[1509] Step 9:

[1510] The server outputs the adjusted text format to the terminal display and provides the information to the user. The input is in the adjusted text format, and by displaying this data on the display, the user receives the information visually. This step completes the process.

[1511] As a result, a system is provided that allows visually or hearing impaired users to safely and effectively make electronic payments.

[1512] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1513] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1514] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1515] [Fourth embodiment]

[1516] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1517] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1518] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1519] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1520] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1521] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1522] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1523] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1524] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1525] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1526] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1527] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1528] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1529] The present invention provides an apparatus and means for assisting people with visual and hearing impairments. The system includes a camera and a microphone, and a set of means for capturing visual and audio information and providing each of them to the user in an appropriate format.

[1530] System configuration and operation

[1531] visual information processing

[1532] server

[1533] The server initializes a camera to capture visual information and an OCR module to analyze the captured image.

[1534] The image obtained from the camera is passed to the OCR module to extract text information.

[1535] Terminal

[1536] The terminal has a voice output device that converts text information sent from the server into voice and reads it out to the user.

[1537] User

[1538] A visually impaired user can understand the captured visual information by listening to the audio information provided by the system.

[1539] Auditory information processing

[1540] server

[1541] The server initializes a microphone to capture audio information and a speech recognition module to convert speech to text.

[1542] The voice data acquired by the microphone is passed to a voice recognition module and converted into text information.

[1543] Terminal

[1544] The terminal has a display that displays the text information sent from the server.

[1545] User

[1546] A hearing-impaired user can understand the captured audio information by reading the text information provided by the system.

[1547] Specific examples

[1548] Example 1: Use in a bookstore

[1549] User

[1550] When a visually impaired user is in a bookstore and wants to know the contents of a book, he or she uses a camera to take a picture of the book's cover.

[1551] Terminal

[1552] The image captured by the camera is sent to the server, where text information is extracted using an OCR module.

[1553] The extracted text information is converted into audio and read aloud to the user.

[1554] server

[1555] The server analyzes the visual information and generates audio data such as the book title and author name.

[1556] Example 2: Understanding everyday conversation

[1557] User

[1558] If a hearing impaired user wants to know the content of everyday conversations, he or she can use a microphone to record the conversations.

[1559] Terminal

[1560] The voice data recorded by the microphone is sent to the server and converted into text information by a voice recognition module.

[1561] The converted text information is displayed on the terminal display so that the user can check it.

[1562] server

[1563] The server processes the captured audio information, converts it into text information in real time, and transmits it.

[1564] Program processing description

[1565] server

[1566] When the server starts up, it loads the necessary libraries and modules and initializes the camera and microphone.

[1567] The server periodically captures visual information from the camera, analyzes it using an OCR module, and converts it into text information.

[1568] Similarly, audio information is captured from the microphone, analyzed by a voice recognition module, and converted into text information.

[1569] Terminal

[1570] The terminal provides the text information received from the server to the user through voice output or display.

[1571] The system of the present invention can provide necessary information in real time to users with visual and hearing impairments, thereby increasing their independence in daily life and overcoming obstacles to information acquisition.

[1572] The processing flow will be explained below.

[1573] Specific processing explanation of the program

[1574] visual information processing

[1575] Step 1:

[1576] The server initializes the system, loading the necessary libraries and modules (OpenCV, OCR module, etc.) and initializing the camera.

[1577] Step 2:

[1578] It uses the device's camera to capture visual information: when the user points an object at the camera, the camera takes a picture of it.

[1579] Step 3:

[1580] The server sends the captured image data to the OCR module, which analyzes the image data and extracts text information.

[1581] Step 4:

[1582] The server sends the extracted text information to a text-to-speech (TTS) engine, which converts the text information into audio data.

[1583] Step 5:

[1584] The device plays the audio data to the user, allowing the user to obtain and understand the visual information as audio.

[1585] Auditory information processing

[1586] Step 1:

[1587] The server initializes the system, loading the necessary libraries and modules (SpeechRecognition, voice recognition module, etc.) and initializing the microphone.

[1588] Step 2:

[1589] Use the device's microphone to capture audio information, either by the user speaking into the microphone or by recording ambient sounds.

[1590] Step 3:

[1591] The server sends the captured voice data to a voice recognition module, which analyzes the voice data and generates text information.

[1592] Step 4:

[1593] The server transmits the generated text information to the terminal.

[1594] Step 5:

[1595] The device displays the text information on the display, allowing the user to receive and understand the audio information as text.

[1596] Specific examples

[1597] Example 1: Use in a bookstore

[1598] Step 1:

[1599] The server initializes the camera and OCR module.

[1600] Step 2:

[1601] The user captures the book cover through the device's camera.

[1602] Step 3:

[1603] The server analyzes the image data using an OCR module and extracts text information.

[1604] Step 4:

[1605] The server converts the extracted text information into speech using a TTS engine.

[1606] Step 5:

[1607] The terminal plays the audio data to the user, and the user understands the book title and author name by listening to the audio.

[1608] Example 2: Understanding everyday conversation

[1609] Step 1:

[1610] The server initializes the microphone and voice recognition module.

[1611] Step 2:

[1612] The user captures the audio of the conversation through the device's microphone.

[1613] Step 3:

[1614] The server analyzes the voice data using a voice recognition module and generates text information.

[1615] Step 4:

[1616] The server transmits the generated text information to the terminal.

[1617] Step 5:

[1618] The terminal displays text information on the display, and the user understands the content of the conversation through text.

[1619] In this way, visual and auditory information is processed sequentially along each step, making it possible to provide the user with the information they need in an appropriate format.

[1620] Example 1

[1621] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1622] In modern society, people with visual or hearing impairments face significant challenges in obtaining and understanding information. In particular, visually impaired people lack the means to receive visual information as audio, while hearing impaired people have limited means to receive audio information as text. Effective systems are needed to overcome these obstacles and improve users' independence and quality of life.

[1623] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1624] In this invention, the server includes means having an imaging device for capturing visual information, means for analyzing the captured visual information and extracting text information, means for converting the extracted text information into speech and outputting it, means for generating speech using a language conversion device for generating speech, means having an acoustic collection device for capturing speech information, means for converting the captured speech information into text information, means for displaying the converted text information, means using a speech recognition device for analyzing the speech information in real time, and means including a communication device for providing information to users in real time, thereby enabling users with visual and hearing impairments to understand the captured visual and speech information in real time and overcome obstacles to information acquisition.

[1625] "Visual information" refers to image data captured by an imaging device such as a camera.

[1626] "Imaging device" refers generically to cameras and other devices for capturing visual data.

[1627] "Capture" refers to the process of obtaining visual or audio information.

[1628] "OCR (Optical Character Recognition)" refers to a technology for extracting text information from image data.

[1629] "Audio information" refers to audio data captured by an audio collecting device such as a microphone.

[1630] "Sound collection device" refers to a general term for microphones and other devices used to capture audio data.

[1631] "Speech recognition device" refers to technology or devices for converting voice information into text information.

[1632] "User" refers to a person with a visual or hearing impairment who wishes to obtain information using this system.

[1633] "Means for converting into voice and outputting" refers to a general term for devices and programs that convert extracted text information into voice information and provide it to the user.

[1634] "Language conversion device" refers to technology or devices for converting text data into voice data.

[1635] "Text information" refers to character data extracted or converted from visual or audio information.

[1636] "Means for displaying" refers to a display or other device for visually presenting textual information to a user.

[1637] "Communication means" refers to the protocols and technologies used to send and receive data between the server and the terminal.

[1638] "Providing in real time" refers to rapidly processing captured information and providing it to the user simultaneously.

[1639] This invention is a system for assisting people with visual or hearing impairments, and aims to overcome barriers to information acquisition by capturing visual and audio information and providing each to the user in an appropriate format.

[1640] First, we will explain how visual information is processed. The server initializes an imaging device to capture visual information and loads an OCR (Optical Character Recognition) module to analyze the captured image data. The image data obtained from the camera is converted into text information using the OCR module. Specifically, the camera is initialized using Python's cv2 library, and OCR analysis is performed using the pytesseract library. This text information is sent to the device via an HTTP request or WebSocket.

[1641] The device converts the text information received from the server into audio using a Text-to-Speech engine (e.g., gTTS) and provides the audio data to the user through a built-in speaker. A visually impaired user can understand the captured visual information through this audio information.

[1642] Next, we will explain how to process auditory information. The server initializes an audio collection device to capture audio information and loads a speech recognition module to convert the audio into text. The audio data captured by the microphone is converted into text information using the speech recognition module. Specifically, the microphone is initialized using the pyaudio library, and speech recognition is performed using Google Cloud Speech-to-Text. This text information is sent to the device via an HTTP request or WebSocket.

[1643] The terminal displays the text information received from the server on a display. The display uses GUI libraries such as tkinter and PyQt to display the text information in a format that is easy for users to read. Hearing-impaired users can understand the captured audio information by reading this text information.

[1644] As a concrete example, consider a bookstore. A visually impaired user takes a picture of a book cover using a camera. The image data is sent to a server, where an OCR module extracts text information (e.g., the book title and author's name). The extracted text information is converted into audio and read aloud to the user. The server analyzes the visual information and provides the content to the user via audio.

[1645] Let's take a concrete example of understanding everyday conversation. If a hearing-impaired user wants to know the content of an everyday conversation, they can record the conversation using a microphone. The audio data is sent to a server, where it is converted into text information by a speech recognition module. The converted text information is then displayed on the device display for the user to confirm. The server then converts the captured audio information into text in real time and provides the content to the user.

[1646] An example of a prompt is:

[1647] "Please explain how a system that reads the contents of books aloud in bookstores for the visually impaired works. Please provide detailed steps from camera initialization to audio output."

[1648] "Please explain how a system that displays everyday conversations as text for the hearing impaired works. Please provide detailed steps from microphone initialization to text display."

[1649] Examples include:

[1650] In this way, the system of the present invention can provide visually and hearing impaired users with the information they need in real time, increasing their independence.

[1651] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1652] visual information processing

[1653] Step 1:

[1654] Server: Initialize the imaging device (camera). Using the Python cv2 library, initialize the camera with the cv2.VideoCapture(0) command. This initialization process allows the camera to function properly and enable image capture. The input is the camera device, and the output is the initialized camera object.

[1655] Step 2:

[1656] Server: Captures image data. Periodically, it uses the cap.read() command to obtain image data from the camera. The obtained image data is passed to the subsequent OCR module. The input is the initialized camera object, and the output is the captured image data.

[1657] Step 3:

[1658] Server: Pass the captured image data to the OCR module and extract text information. Specifically, use the pytesseract library to extract text from the image data with the pytesseract.image_to_string(image) command. The input is the captured image data, and the output is the extracted text data.

[1659] Step 4:

[1660] Server: Send the extracted text data to the terminal via an HTTP request or WebSocket. Use the requests.post(url, data={'text': text}) command. The input is the extracted text data, and the output is the text data sent to the terminal.

[1661] Step 5:

[1662] Terminal: The received text data is passed to the Text-to-Speech engine to generate audio data. The gTTS library is used to generate audio data using the gTTS(text) command. The generated audio file is played through the built-in speaker. The input is the text data received from the server, and the output is the generated audio data.

[1663] Step 6:

[1664] User: Understands the captured visual information by listening to the audio information provided by the device. The input is the audio data played from the device, and the output is the information comprehension obtained by listening to the audio data.

[1665] Auditory information processing

[1666] Step 1:

[1667] Server: Initialize the sound collection device (microphone). Use Python's pyaudio library to initialize the microphone with the audio = pyaudio.PyAudio() command. The input is the microphone device, and the output is the initialized microphone object.

[1668] Step 2:

[1669] Server: Captures audio data. Periodically obtains audio data from the microphone using the audio.open() and stream.read() commands. The obtained audio data is passed to the subsequent speech recognition module. The input is the initialized microphone object, and the output is the captured audio data.

[1670] Step 3:

[1671] Server: Pass the captured voice data to the speech recognition module and convert it into text information. Use Google Cloud Speech-to-Text to convert the voice data into text using the speech_to_text(audio_data) command. The input is the captured voice data, and the output is the converted text data.

[1672] Step 4:

[1673] Server: The converted text data is sent to the terminal via an HTTP request or WebSocket. Use the requests.post(url, data={'text': text}) command. The input is the converted text data, and the output is the text data sent to the terminal.

[1674] Step 5:

[1675] Terminal: The received text data is displayed on the screen. Using a GUI library such as tkinter or PyQt, the text data is displayed with the label.config(text=text) command. The input is the text data received from the server, and the output is the text data displayed on the screen.

[1676] Step 6:

[1677] User: Understands the captured audio information by reading the text information displayed on the display. The input is the text data displayed on the display, and the output is the understanding of the information obtained by reading the text data.

[1678] (Application example 1)

[1679] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1680] People with visual or hearing impairments have difficulty understanding product information and conversations in virtual stores. To address this issue, a system is needed that can convert visual information into speech and speech into text in real time.

[1681] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1682] In this invention, the server is a device equipped with a camera for capturing visual information, including means for analyzing the captured visual information and extracting text information, means for converting the extracted text information into audio and outputting it, means for saving the extracted text information as an audio file, and means for playing the saved audio file; a device equipped with an acoustic sensor for capturing audio information, including means for converting the captured audio information into text information, means for displaying the converted text information, and means for displaying the text information on a display of a user's terminal; a device equipped with both a camera and an acoustic sensor, including means for converting the extracted text information from text to audio and saving it as an audio file, means for playing the saved audio file, and means for using a generative AI model to generate prompt sentences. This enables users with visual or hearing impairments to intuitively and easily understand product information and conversations in a virtual store.

[1683] "Visual information" refers to image or video information captured using a camera or other photographic device.

[1684] "Image capture device" refers to a camera or other device for capturing visual information.

[1685] "Text information" refers to text data extracted from visual information.

[1686] "Audio information" refers to audio data captured using a microphone or acoustic sensor.

[1687] "Acoustic sensor" refers to a microphone or other device for capturing audio information.

[1688] "Transformation" refers to the act or process of transforming information into a different form.

[1689] "Users" refers to visually and hearing impaired people who use the system.

[1690] "Terminal" refers to a device used by a user, such as a smartphone or tablet.

[1691] "Generative AI model" refers to an artificial intelligence model for generating prompts and other text.

[1692] A "prompt" refers to the text of instructions or questions that are input to a generative AI model.

[1693] "Audio File" means a digital file that stores audio information.

[1694] This invention is a system for supporting people with visual or hearing impairments, making it easier for them to understand product information and conversations in a virtual store.

[1695] visual information processing

[1696] The user takes a picture of a product image or screen using a camera to capture visual information. The server analyzes the captured visual information using OCR (optical character recognition) technology and extracts text information. This text information is then used to generate a prompt using a generative AI model, which is then converted into audio and saved as an audio file. The device then plays the saved audio file, allowing the user to listen to the product description.

[1697] Auditory information processing

[1698] Using an acoustic sensor (microphone) to capture auditory information, the user captures audio information in the virtual store (for example, conversations with sales staff). The server analyzes the captured audio information using voice recognition technology and converts it into text information. This text information is displayed on the user's device display, allowing the user to confirm the information provided by voice as text.

[1699] System Configuration

[1700] 1. Server

[1701] The server analyzes the captured visual information using an OCR module (e.g., pytesseract) and converts it into text information.

[1702] The server analyzes the captured voice information using a voice recognition module (e.g., speech_recognition) and converts it into text information.

[1703] The text information is processed by a generative AI model to generate a prompt sentence.

[1704] To convert the generated text information into audio, an audio file is created and saved using Google Text-to-Speech (gTTS).

[1705] 2. Terminal

[1706] The user's terminal plays the audio file sent from the server and provides the user with information by voice.

[1707] The user's terminal displays the text information sent from the server on a display, allowing the user to confirm the text information.

[1708] Specific examples

[1709] Example 1: Retrieving product information

[1710] While a user is viewing a product page in a virtual store, they take a picture of the screen with their smartphone camera. The image is sent to a server, where the product description is converted into text using OCR. A generative AI model generates a prompt, which is then converted into speech and played back to the user.

[1711] Example 2: Understanding conversation content

[1712] The user records their conversation with the sales staff in the virtual store, and the audio data is sent to the server, where it is converted into text and displayed on the device's display. The user can understand the content of the conversation by reading the text.

[1713] Prompt Sentence Examples

[1714] "Extract Japanese characters from the input image file and convert the content into an audio file (MP3 format). Also, play the audio file and display the extracted text."

[1715] This system enables visually or hearing impaired users to understand product information and conversations in a virtual store in real time, allowing them to enjoy a comfortable shopping experience.

[1716] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1717] Step 1:

[1718] The user uses the smartphone camera to take pictures of the virtual store screen and product images. The input is image data from the camera. The output is the captured image file.

[1719] Step 2:

[1720] The terminal sends the captured image file to the server. The input is the image file. The output is the image data sent to the server.

[1721] Step 3:

[1722] The server parses the received image file into text using an OCR module (e.g., pytesseract). The input is the image data received by the server. The output is the extracted text information.

[1723] Step 4:

[1724] The server uses a generative AI model to generate a prompt from the extracted text, and then converts it into an audio file using Google Text-to-Speech (gTTS). The input is the extracted text, and the output is the generated prompt and audio file.

[1725] Step 5:

[1726] The server sends the generated audio file to the terminal. The input is the audio file. The output is the audio data sent to the terminal.

[1727] Step 6:

[1728] The terminal plays the received audio file and provides audio information to the user. The input is the audio data sent from the server. The output is the audio played back from the audio data.

[1729] Step 7:

[1730] A user uses the microphone on their smartphone to record conversations in a virtual store. The input is the audio data of the conversation. The output is a recorded audio file.

[1731] Step 8:

[1732] The terminal sends the recorded audio file to the server. The input is the recorded audio file. The output is the audio data sent to the server.

[1733] Step 9:

[1734] The server converts the received audio file into text using a speech recognition module (e.g., speech_recognition). The input is the received audio data. The output is the converted text.

[1735] Step 10:

[1736] The server sends the converted character information to the terminal. The input is the converted character information. The output is the character information sent to the terminal.

[1737] Step 11:

[1738] The terminal displays the received text information on the display and provides the text information to the user. The input is the text information sent from the server. The output is the text information displayed on the display.

[1739] This allows users with visual or hearing impairments to understand product information and conversations in a virtual store in real time.

[1740] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1741] This invention provides a system that combines an emotion engine that recognizes the user's emotions with a device for assisting people with visual or hearing impairments. The system is equipped with a camera and a microphone, and includes a series of means for acquiring visual and audio information in real time and providing it to the user in an appropriate format. Furthermore, the emotion engine recognizes the user's emotions and adjusts the information output based on the results.

[1742] System configuration and operation

[1743] visual information processing

[1744] server

[1745] The server initializes a camera to capture visual information and an OCR module to analyze the captured image.

[1746] The image obtained from the camera is passed to the OCR module to extract text information.

[1747] The extracted text information is sent to an emotion engine, which selects an appropriate voice output format depending on the user's emotional state.

[1748] Terminal

[1749] The terminal is equipped with a voice output device that converts text information sent from the server into voice and reads it out to the user. The emotion engine adjusts the tone and speed of the voice according to the user's emotions.

[1750] User

[1751] A visually impaired user can understand the captured visual information by listening to audio information provided by the system, for example, a slower voice when the user is tired, or a clearer voice when the user is energetic.

[1752] Auditory information processing

[1753] server

[1754] The server initializes a microphone to capture audio information and a speech recognition module to convert speech to text.

[1755] The voice data acquired by the microphone is passed to a voice recognition module and converted into text information.

[1756] The generated text information is sent to an emotion engine, which adjusts the format and size of the text display according to the user's emotional state.

[1757] Terminal

[1758] The terminal has a display that displays text information sent from the server, and the emotion engine adjusts the font size and color of the text according to the user's emotions.

[1759] User

[1760] A hearing-impaired user can understand the captured audio information by reading the text information provided by the system, for example, in normal font size when the user is calm and in larger font size when the user is anxious.

[1761] Specific examples

[1762] Example 1: Use in a bookstore

[1763] User

[1764] When a visually impaired user is in a bookstore and wants to know the contents of a book, he or she uses a camera to take a picture of the book's cover.

[1765] Terminal

[1766] The image captured by the camera is sent to the server, where text information is extracted using an OCR module.

[1767] The extracted text information is sent to the TTS engine via the emotion engine, and is read aloud to the user in a voice format that corresponds to the user's emotion.

[1768] server

[1769] The server analyzes visual information, generates audio data such as the book title and author's name, and uses an emotion engine to select the optimal audio output that takes the user's emotions into consideration.

[1770] Example 2: Understanding everyday conversation

[1771] User

[1772] If a hearing impaired user wants to know the content of everyday conversations, he or she can use a microphone to record the conversations.

[1773] Terminal

[1774] The voice data recorded by the microphone is sent to the server and converted into text information by a voice recognition module.

[1775] The converted text information is sent to the terminal via the emotion engine and displayed on the display in a format that corresponds to the user's emotion.

[1776] server

[1777] The server processes the captured voice information, converts it into text information in real time, and selects the optimal text display format taking into account the user's emotions using an emotion engine.

[1778] In this way, by adding an emotion engine to the processing of visual and auditory information, it becomes possible to provide information flexibly according to the user's emotional state, further improving the user experience.

[1779] The processing flow will be explained below.

[1780] visual information processing

[1781] Step 1:

[1782] The server initializes the system, loading the necessary libraries and modules (OpenCV, OCR module, etc.) and initializing the camera.

[1783] Step 2:

[1784] The user shows the book cover to the device's camera, and the device uses the camera to capture the book cover.

[1785] Step 3:

[1786] The server sends the captured image data to the OCR module to extract text information, which is then analyzed and generated.

[1787] Step 4:

[1788] The server sends the extracted text information to the emotion engine, which assesses the user's emotional state and adjusts the tone and speed of the speech accordingly.

[1789] Step 5:

[1790] The server sends the text information adjusted by the emotion engine to a TTS (text-to-speech) engine, which converts the text information into voice data.

[1791] Step 6:

[1792] The device plays the audio data back to the user. For example, if the device determines that the user is tired, it will read the data in a slower tone, and if the user is energetic, it will read the data in a clearer tone.

[1793] Auditory information processing

[1794] Step 1:

[1795] The server initializes the system, loading the necessary libraries and modules (SpeechRecognition, voice recognition module, etc.) and initializing the microphone.

[1796] Step 2:

[1797] The user speaks into the device's microphone or records the surrounding audio. The device captures the audio information.

[1798] Step 3:

[1799] The server sends the captured voice data to a voice recognition module to generate text information, which then analyzes the voice data and converts it into text information.

[1800] Step 4:

[1801] The server sends the generated text information to the emotion engine, which evaluates the user's emotional state and adjusts the format and size of the text display accordingly.

[1802] Step 5:

[1803] The server sends text information adjusted by the emotion engine to the terminal.

[1804] Step 6:

[1805] The device will display text information on the display, for example in a larger font size if it determines that the user is anxious, or in a normal font size if the user is calm.

[1806] Specific examples

[1807] Example 1: Use in a bookstore

[1808] Step 1:

[1809] The server initializes the camera and OCR module.

[1810] Step 2:

[1811] The user points the device camera at the book cover.

[1812] Step 3:

[1813] The device captures the book cover with its camera and sends the image data to the server.

[1814] Step 4:

[1815] The server analyzes the image data using an OCR module and extracts text information.

[1816] Step 5:

[1817] The server sends the text information to the emotion engine, which determines how to read it out based on the user's emotion.

[1818] Step 6:

[1819] The server converts the adjusted text information into voice data using a TTS engine.

[1820] Step 7:

[1821] The device plays the audio data to the user, for example, reading it in a slower tone if it determines that the user is tired.

[1822] Example 2: Understanding everyday conversation

[1823] Step 1:

[1824] The server initializes the microphone and voice recognition module.

[1825] Step 2:

[1826] The user speaks into the device microphone or captures the conversation audio.

[1827] Step 3:

[1828] The terminal transmits the voice information to the server.

[1829] Step 4:

[1830] The server analyzes the voice data using a voice recognition module and converts it into text information.

[1831] Step 5:

[1832] The server sends the generated text information to the emotion engine, which determines the format and size according to the user's emotion.

[1833] Step 6:

[1834] The server sends the adjusted text information to the terminal.

[1835] Step 7:

[1836] The device displays text information on the display, for example, in a larger font size if it determines that the user is in a hurry.

[1837] In this way, by combining the processing of visual and auditory information with an emotion engine, it becomes possible to provide information flexibly according to the user's emotional state, further improving the user experience.

[1838] Example 2

[1839] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1840] People with visual or hearing impairments have difficulty obtaining and understanding information in their daily lives. Furthermore, conventional information support systems do not take the user's emotional state into consideration, which results in a lack of improvement in the user experience. The objective of this invention is to provide flexible information provision according to the user's emotional state in such information support systems for the visually and hearing impaired, thereby improving the user experience.

[1841] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1842] In this invention, the server includes a camera for capturing visual information, means for analyzing the captured visual information and extracting text information, means for sending the extracted text information to an emotion engine for analyzing the user's emotion, and means for converting the text information into audio based on the analysis results of the emotion engine and adjusting the audio format for output. This makes it possible to provide information in an appropriate format that takes into account the user's emotional state when converting visual information into audio. The server also includes a microphone for capturing audio information, means for converting the captured audio information into text information, means for sending the converted text information to the emotion engine for analyzing the user's emotion, and means for adjusting and displaying the text format based on the analysis results of the emotion engine. This makes it possible to provide information in an appropriate format that takes into account the user's emotional state when converting audio information into text.

[1843] "Visual information" refers to image or video data captured using a device such as a camera.

[1844] A "camera" is a photographing device for acquiring visual information.

[1845] An "OCR (Optical Character Recognition) module" is a software module for extracting character information from captured image data.

[1846] "Audio information" refers to sound or voice data captured using a device such as a microphone.

[1847] A "microphone" is a sound pickup device for acquiring audio information.

[1848] An "emotion engine" is a software component for analyzing a user's emotional state.

[1849] A "voice recognition module" is a software module for converting captured voice data into text information.

[1850] "Text information" refers to data that can be represented and stored as characters.

[1851] A "TTS (Text-to-Speech) engine" is a software engine for converting text information into speech.

[1852] A "terminal" is a device used by a user to receive information, including an audio output device and a display.

[1853] A "server" is a centralized computing device for managing and executing system-wide processing.

[1854] A "display" is a screen device for visually presenting text information to a user.

[1855] "Format" refers to the form or style in which text or audio information is presented to the user appropriately.

[1856] MODE FOR CARRYING OUT THE INVENTION

[1857] This invention relates to a system for assisting people with visual or hearing impairments, and provides a set of means for acquiring visual and audio information in real time and providing the information in a flexible format according to the user's emotional state. The system is equipped with a camera and a microphone, acquires visual and audio information in real time, and provides the information to the user in an appropriate format. It also uses an emotion engine to analyze the user's emotional state and adjusts the information output format based on the results.

[1858] visual information processing

[1859] server

[1860] The server is equipped with a camera for capturing visual information. After the camera is initialized, it captures visual information (e.g., text or images). The captured image data is passed to an OCR (Optical Character Recognition) module for analysis. This analysis extracts text information from the image.

[1861] The server then sends the extracted text information to the emotion engine, which analyzes the user's emotional state and selects a voice format (tone, speed, etc.) based on the analysis results. For example, if the user is tired, a slower voice format is selected, and if the user is active, a clearer, crisper voice format is selected.

[1862] Terminal

[1863] The terminal is equipped with a voice output device and receives text information sent from the server. This text information is converted into voice using a TTS (Text-to-Speech) engine and read aloud to the user through the voice output device.

[1864] User

[1865] A visually impaired user can understand the content of a book by receiving visual information captured by a camera as audio. For example, a user can take a picture of a book cover in a bookstore and receive text information extracted from the captured image as audio.

[1866] Auditory information processing

[1867] server

[1868] The server is equipped with a microphone for capturing voice information. After the microphone is initialized, it captures voice information (e.g., conversation content). The captured voice data is passed to a voice recognition module and converted into text information. The converted text information is also sent to the emotion engine.

[1869] The emotion engine analyzes the user's emotional state and adjusts the text formatting (font size, color, etc.) based on the analysis. For example, if the user is calm, text will be displayed in normal font size, and if the user is anxious, text will be displayed in large font size.

[1870] Terminal

[1871] The device is equipped with a display and receives text information sent from the server. The received text information is displayed on the display in an appropriate format based on the analysis results of the emotion engine.

[1872] User

[1873] Hearing-impaired users can understand the content of audio information captured by a microphone by receiving it as text. For example, a user can record their everyday conversation, and text information generated from the recorded audio data is displayed on a screen.

[1874] Specific examples

[1875] Example 1: Use in a bookstore

[1876] User

[1877] When a visually impaired user is in a bookstore and wants to know the contents of a book, he or she uses a camera to take a picture of the book's cover.

[1878] Terminal

[1879] The image captured by the camera is sent to the server, where the OCR module extracts text information, which is then sent to the TTS engine via the emotion engine, where it is read aloud to the user in a voice format that corresponds to the user's emotion.

[1880] server

[1881] The server analyzes the visual information, generates text data such as the book title and author's name, and uses an emotion engine to select the optimal voice output based on the user's emotions.

[1882] Example 2: Understanding everyday conversation

[1883] User

[1884] If a hearing impaired user wants to know the content of everyday conversations, he or she can use a microphone to record the conversations.

[1885] Terminal

[1886] The voice data recorded by the microphone is sent to the server and converted into text information by a voice recognition module. The converted text information is sent to the device via an emotion engine and displayed on the screen in a format that corresponds to the user's emotion.

[1887] server

[1888] The server processes the captured audio information, converts it into text information in real time, and uses an emotion engine to select the optimal text display format according to the user's emotions.

[1889] In a specific embodiment of this invention, users with visual and hearing impairments can receive visual information as audio and audio information as text, and enjoy information in a flexible format that adapts to their emotional state. This system improves the user experience and makes it easier to obtain information in daily life.

[1890] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1891] visual information processing

[1892] Step 1:

[1893] server

[1894] The server initializes the camera to capture visual information, and the initialized camera is ready to capture visual information in real time.

[1895] Specific operation: The server sends an initialization command to the camera and confirms that the camera is ready for operation.

[1896] Step 2:

[1897] User

[1898] If the user is visually impaired, they place the source of information they want to know about (e.g., a book cover) in front of the camera and use the camera to photograph the object.

[1899] Specific Action: The user presses the camera button to capture an image.

[1900] Step 3:

[1901] server

[1902] The server receives the captured image data and passes it to the OCR module, which analyzes the image data and extracts text information. The input is the image data and the output is the extracted text information.

[1903] Specific operation: The server sends an API request to the OCR module to perform image analysis.

[1904] Step 4:

[1905] server

[1906] The server sends the extracted text information to the emotion engine, which analyzes the user's emotional state. The input is the text information, and the output is the emotion analysis result.

[1907] Specific operation: The server sends text data to the emotion engine and obtains the analysis results.

[1908] Step 5:

[1909] server

[1910] The server selects an audio format for converting text information into audio based on the emotion analysis results. The input is the emotion analysis results, and the output is the audio format.

[1911] What happens: The server selects the appropriate voice format (tone, speed, etc.) and prepares the data to send to the TTS engine.

[1912] Step 6:

[1913] Terminal

[1914] The terminal receives text information sent from the server and converts it into speech using a TTS engine. The input is text information and speech format, and the output is speech data.

[1915] Specific operation: The device sends an API request to the TTS engine and obtains the generated voice data.

[1916] Step 7:

[1917] Terminal

[1918] The terminal uses an audio output device to read the generated audio data to the user.

[1919] Specific operation: The terminal transmits audio data to the audio output device, and the user listens to the audio.

[1920] Auditory information processing

[1921] Step 1:

[1922] server

[1923] The server initializes the microphone to capture audio information. Once initialized, the microphone is ready to capture audio information in real time.

[1924] Specific operation: The server sends a microphone initialization command and verifies that the microphone is ready for operation.

[1925] Step 2:

[1926] User

[1927] If the user is hearing impaired, the desired conversation content is recorded using a microphone.

[1928] Specific action: The user presses the microphone button to capture audio.

[1929] Step 3:

[1930] server

[1931] The server receives the captured voice data and passes it to the voice recognition module, which analyzes the voice data and converts it into text information. The input is voice data and the output is text information.

[1932] Specific operation: The server sends an API request to the speech recognition module to perform speech analysis.

[1933] Step 4:

[1934] server

[1935] The server sends the generated text information to the emotion engine, which analyzes the user's emotional state. The input is the text information, and the output is the emotion analysis result.

[1936] Specific operation: The server sends text data to the emotion engine and obtains the analysis results.

[1937] Step 5:

[1938] server

[1939] The server selects the text format (font size, color, etc.) based on the sentiment analysis results. The input is the sentiment analysis results, and the output is the formatted text information.

[1940] Specific behavior: The server selects the appropriate text format and prepares the data to be displayed on the display.

[1941] Step 6:

[1942] Terminal

[1943] The terminal receives the formatted text information sent from the server and displays it on the display. The input is the formatted text information, and the output is the text displayed on the display.

[1944] Specific operation: The terminal sends formatted text information to the display and checks the display status.

[1945] Step 7:

[1946] User

[1947] The user understands the content of the conversation by reading the text information displayed on the display.

[1948] Specific operation: The user looks at the display and visually recognizes the text information.

[1949] The above are the specific processing steps of the system for processing visual and auditory information, which enables visually and auditory impaired users to obtain and understand information in real time.

[1950] (Application example 2)

[1951] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1952] The challenge is to solve the difficulties that visually and hearing impaired users have in safely and effectively performing complex operations such as electronic payments in their daily lives. Another problem is that the information provided is not adapted to the user's current emotional state, resulting in a poor user experience.

[1953] The specification processing by the specification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means having a camera for capturing visual information, means for analyzing the captured visual information and extracting text information, means for converting the extracted text information into voice and outputting it, an emotion engine for adjusting the extracted text information and the converted text information in accordance with the user's emotional state, means for outputting the information adjusted by the emotion engine to the user in an appropriate tone and speed, and means for providing the user with information captured in an electronic payment situation. This enables visually or hearing impaired users to receive appropriate information according to their emotional state when making electronic payments.

[1954] "Visual information" generally refers to video data captured using an image capture device such as a camera.

[1955] "Camera" refers to a photographic device used to capture visual information.

[1956] "Analysis" refers to the process of breaking down captured visual and audio information and extracting useful data.

[1957] "Text information" refers to character data extracted from visual or audio information.

[1958] "Extraction" refers to the process of extracting necessary information from analyzed data.

[1959] "Convert to speech" refers to the process of converting extracted text information into speech data.

[1960] "Output" refers to the act of conveying converted speech or displayed text to the user.

[1961] "Audio information" generally refers to sound data captured using an audio capture device such as a microphone.

[1962] "Microphone" refers to a sound pickup device used to capture audio information.

[1963] "Display" refers to visually conveying text information to a user using a display device or the like.

[1964] An "emotion engine" refers to a system that analyzes the user's emotional state and adjusts the format of information output based on the results.

[1965] "Adjustment" refers to the act of changing the output format or content depending on the user's emotional state.

[1966] "Tone" refers to the sound quality of voice, such as pitch, volume, and speed.

[1967] "Speed" refers to the speed at which voice or information is output.

[1968] "Electronic payment" refers to a payment transaction made using digital technology.

[1969] "Providing" refers to the act of giving information to a user.

[1970] The present invention realizes a system that enables users with visual or hearing impairments to make electronic payments safely and effectively. Specific embodiments are described below.

[1971] Hardware Configuration

[1972] The server is equipped with a camera to capture visual information and a microphone to capture audio information. The camera captures visual information and the microphone captures audio information. The terminal used by the user is equipped with a display and speaker.

[1973] Software Configuration

[1974] The server is installed with an OCR (Optical Character Recognition) module to analyze visual information and extract text information, and a speech recognition module to convert audio information into text. The speech synthesis engine pyttsx3 is used for reading out the audio, and EmotionRecognizer is used to recognize the user's emotional state.

[1975] Data processing and calculation

[1976] 1. Processing of visual information:

[1977] The server uses a camera to capture visual information. Images are captured in real time using the OpenCV library.

[1978] The acquired image data is subjected to OCR processing using pytesseract, and text information is extracted.

[1979] 2. Audio processing:

[1980] It uses a microphone to capture audio information. The speech_recognition library is used to obtain the audio data.

[1981] The captured voice data is converted into text information via the Google Speech API.

[1982] 3. Emotion recognition and output adjustment:

[1983] Using EmotionRecognizer, the user's emotional state is analyzed, which allows for assessment of the user's level of fatigue, energy, etc.

[1984] Based on the analysis results, the acquired text information is outputted using the speech synthesis engine pyttsx3 in an appropriate tone and speed, and the font size and color are also adjusted for text display.

[1985] Specific examples

[1986] As a concrete example, consider a visually impaired user making an electronic payment at a cafe. The user scans a QR code with their smartphone camera, and the server analyzes the image using OCR to extract text information. The emotion engine analyzes the user's emotional state, and if it determines that the user is tired, for example, it lowers the tone and speed of the voice to announce, "This code is for payment at the cafe."

[1987] Example of input prompt for generative AI model

[1988] "When a user scans a QR code with a camera, design a system that extracts the code's contents using an OCR module, determines the user's emotion using an emotion recognition engine, and provides information in an appropriate format based on that emotion. For example, if the user is tired, slow down the reading speed, and if the user is energetic, speed up the reading speed."

[1989] In this way, a support system can be provided that allows visually or hearing impaired users to make electronic payments with peace of mind.

[1990] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1991] Step 1:

[1992] A user uses a camera to capture visual information, and obtains an image containing a QR code and text information. The input is the image data captured by the camera, and this data is passed to the next step.

[1993] Step 2:

[1994] The server passes the captured image data as input to the OCR module to extract text information. Specifically, it reads the image using the OpenCV library and performs text extraction using pytesseract. The output of this process is the extracted text information.

[1995] Step 3:

[1996] The extracted text information is passed as input to the server's speech synthesis engine (pyttsx3) to prepare for conversion into speech. At this time, the server receives additional information from the camera or microphone as input and performs user emotion analysis using EmotionRecognizer to recognize the user's emotional state. The output of this process is emotional state data.

[1997] Step 4:

[1998] The server adjusts the tone and speed of the speech synthesis based on the emotional state data. Specifically, it sets pyttsx3 properties, for example, slowing down the speech speed if the user is tired and speeding up if the user is energetic. The output of this process is speech data adapted to the user's emotions.

[1999] Step 5:

[2000] The server outputs the adjusted voice data to the device's speaker and provides the information to the user. The input is the adjusted voice data, and this data is transmitted to the user as voice output. The processing is completed when the device's speaker plays the voice and the user receives the information.

[2001] Step 6:

[2002] If the target is audio information, the user uses a microphone to capture audio data, which is input to the server and passed to the next step.

[2003] Step 7:

[2004] The server passes the captured voice data as input to a speech recognition module, which converts the voice into text. Specifically, it uses the speech_recognition library and the Google Speech API to analyze the voice data and extract text information. The output of this process is the converted text information.

[2005] Step 8:

[2006] The extracted text information is passed as input to the emotion engine on the server, which adjusts the display format based on the user's emotional state. Specifically, the font size and color are changed based on the emotional state analyzed by EmotionRecognizer. The output of this process is a text format adapted to the emotion.

[2007] Step 9:

[2008] The server outputs the adjusted text format to the terminal display and provides the information to the user. The input is in the adjusted text format, and by displaying this data on the display, the user receives the information visually. This step completes the process.

[2009] As a result, a system is provided that allows visually or hearing impaired users to safely and effectively make electronic payments.

[2010] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2011] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2012] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2013] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2014] FIG. 9 illustrates an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and behaviors arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2015] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2016] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2017] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2018] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2019] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2020] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2021] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2022] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[2023] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2024] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[2025] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[2026] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[2027] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[2028] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[2029] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[2030] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[2031] The following is further disclosed regarding the above embodiment.

[2032] (Claim 1)

[2033] 1. A device having a camera for capturing visual information,

[2034] means for analyzing the captured visual information to extract textual information;

[2035] means for converting the extracted text information into speech and outputting the speech;

[2036] A system including:

[2037] (Claim 2)

[2038] 1. A device having a microphone for capturing audio information,

[2039] means for converting the captured audio information into text information;

[2040] means for displaying the converted text information;

[2041] 10. The system of claim 1.

[2042] (Claim 3)

[2043] A device equipped with both a camera and a microphone,

[2044] means for analyzing the visual information captured by the camera to extract textual information;

[2045] means for converting audio information captured by the microphone into text information;

[2046] means for outputting the extracted text information and the converted text information to a user;

[2047] 10. The system of claim 1.

[2048]

[2049] "Example 1"

[2050] (Claim 1)

[2051] 1. An apparatus comprising an imaging device for capturing visual information,

[2052] means for analyzing the captured visual information to extract textual information;

[2053] means for converting the extracted text information into speech and outputting the speech;

[2054] means for generating speech using a language conversion device for generating speech;

[2055] A system including:

[2056] (Claim 2)

[2057] 1. An apparatus comprising an acoustic collection device for capturing audio information,

[2058] means for converting the captured audio information into text information;

[2059] means for displaying the converted text information;

[2060] means for using a speech recognizer to analyze speech information in real time;

[2061] 10. The system of claim 1.

[2062] (Claim 3)

[2063] A device having both an imaging device and an acoustic collection device,

[2064] means for analyzing the visual information captured by the imaging device to extract text information;

[2065] means for converting the audio information captured by the audio collecting device into text information;

[2066] means for outputting the extracted text information and the converted text information to a user;

[2067] means including a communication means for providing information to a user in real time;

[2068] 10. The system of claim 1.

[2069] "Application Example 1"

[2070] (Claim 1)

[2071] 1. A device comprising an imaging device for capturing visual information,

[2072] means for analyzing the captured visual information to extract textual information;

[2073] means for converting the extracted character information into sound and outputting the sound;

[2074] A means for saving the extracted text information as an audio file;

[2075] means for playing the stored audio file;

[2076] A system including:

[2077] (Claim 2)

[2078] 1. A device comprising an acoustic sensor for capturing audio information,

[2079] means for converting the captured audio information into textual information;

[2080] a means for displaying the converted character information;

[2081] means for displaying text information on a display of a user's terminal;

[2082] 10. The system of claim 1.

[2083] (Claim 3)

[2084] A device equipped with both an imaging device and an acoustic sensor,

[2085] means for analyzing the visual information captured by the imaging device and extracting text information;

[2086] means for converting audio information captured by the acoustic sensor into text information;

[2087] means for outputting the extracted character information and the converted character information to a user;

[2088] a means for using a generative AI model to generate prompt sentences;

[2089] A means for converting the extracted character information from text to audio and saving it as an audio file;

[2090] means for playing the stored audio file;

[2091] 10. The system of claim 1.

[2092] "Example 2: Combining Emotion Engines"

[2093] (Claim 1)

[2094] 1. A device having a camera for capturing visual information,

[2095] means for analyzing the captured visual information to extract textual information;

[2096] means for transmitting the extracted text information to an emotion engine to analyze the user's emotion;

[2097] means for converting text information into speech based on the analysis result of the emotion engine, adjusting the speech format, and outputting the speech;

[2098] A system including:

[2099] (Claim 2)

[2100] 1. A device having a microphone for capturing audio information,

[2101] means for converting the captured audio information into text information;

[2102] means for transmitting the converted text information to an emotion engine to analyze the user's emotion;

[2103] a means for adjusting and displaying a text format based on the analysis result of the emotion engine;

[2104] 10. The system of claim 1.

[2105] (Claim 3)

[2106] A device equipped with both a camera and a microphone,

[2107] means for analyzing the visual information captured by the camera to extract textual information;

[2108] means for converting audio information captured by the microphone into text information;

[2109] means for transmitting the extracted text information and the converted text information to an emotion engine to analyze the user's emotion;

[2110] means for outputting audio and text information to the user in a tailored format based on the analysis results of the emotion engine;

[2111] 10. The system of claim 1.

[2112] "Application example 2 when combining emotion engines"

[2113] (Claim 1)

[2114] 1. A device having a camera for capturing visual information,

[2115] means for analyzing the captured visual information to extract textual information;

[2116] means for converting the extracted text information into speech and outputting the speech;

[2117] an emotion engine that adjusts the extracted text information and the converted text information according to the user's emotional state;

[2118] A means for outputting information adjusted by the emotion engine to a user in an appropriate tone and speed;

[2119] means for providing captured information to a user in the context of an electronic payment;

[2120] A system including:

[2121] (Claim 2)

[2122] 1. A device having a microphone for capturing audio information,

[2123] means for converting the captured audio information into text information;

[2124] means for displaying the converted text information;

[2125] a means for adjusting the converted text information in accordance with the emotional state of the user and displaying the adjusted text information on a display;

[2126] 10. The system of claim 1.

[2127] (Claim 3)

[2128] A device equipped with both a camera and a microphone,

[2129] means for analyzing the visual information captured by the camera to extract textual information;

[2130] means for converting audio information captured by the microphone into text information;

[2131] an emotion engine that adjusts the extracted text information and the converted text information according to the user's emotional state;

[2132] means for outputting the adjusted information to a user;

[2133] means for providing information adjusted by the emotion engine to a user in an electronic payment context;

[2134] 10. The system of claim 1. [Explanation of symbols]

[2135] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. 1. A device having a camera for capturing visual information, means for analyzing the captured visual information to extract textual information; means for converting the extracted text information into speech and outputting the speech; A system including:

2. 1. A device having a microphone for capturing audio information, means for converting the captured audio information into text information; means for displaying the converted text information; The system of claim 1 .

3. A device equipped with both a camera and a microphone, means for analyzing the visual information captured by the camera to extract textual information; means for converting audio information captured by the microphone into text information; means for outputting the extracted text information and the converted text information to a user; The system of claim 1 .

Images