System

Abstract

A system is provided.SOLUTION: The system includes an image acquisition means, a means for transmitting the acquired image to an external server by a communication means, a character recognition means for extracting text information from the image received by the external server, a voice generation means for converting the extracted text information into voice data, and a voice output means for receiving and reproducing the generated voice data via the communication means.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Visually impaired people and people with reading and writing difficulties face the challenge of quickly and accurately obtaining written information in their daily lives. In particular, they often need the help of others to understand written information on electronic bulletin boards at stations, signs around town, books, etc. This situation is a factor that hinders the independent living of visually impaired people. [Means for solving the problem]

[0005] To solve the above problems, the present invention provides the following means. Specifically, a visually impaired person or a person who has difficulty reading and writing wears smart glasses and acquires an image containing text information from a camera built into the glasses. The acquired image is transmitted to an external server via a communication means, where it is analyzed using character recognition means that extracts text information from the image. The extracted text information is then converted into audio data, and the generated audio data is sent back to the smart glasses via the communication means. An audio output means built into the smart glasses plays back the audio data, providing the user with the text information by voice. In this way, visually impaired people can acquire information independently.

[0006] The "image acquisition means" is a means for capturing visual information as an image using a device such as a camera built into the smart glasses.

[0007] "Communication means" refers to a means for sending and receiving information between smart glasses and an external server, and includes wireless communication technologies such as Wi-Fi and Bluetooth.

[0008] An "external server" is a remote computer system used to receive data sent from smart glasses or devices and perform advanced processing.

[0009] "Character recognition means" refers to a means of extracting text information from an image using OCR (optical character recognition) technology or the like.

[0010] The "voice generation means" is a means for converting text information into voice data using a TTS (Text-to-Speech) engine or the like.

[0011] The "audio output means" refers to a device such as a speaker or earphone for reproducing the generated audio data to the user.

[0012] The term "wearable device" refers to an electronic device that can be worn by a user, and in the present invention, this particularly refers to smart glasses.

[0013] "Cloud storage" refers to online storage services for storing and accessing data over the Internet.

[0014] An "HTTP request" is a protocol-based request that allows a client (smart glasses or a terminal) to send data to a server.

[0015] A "REST API" is an API based on an architectural style for integrating with and exchanging data with web services.

[0016] "User" refers to a visually impaired or illiterate person using the smart glasses of the present invention. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8]FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] The present invention provides a smart glasses system that enables visually impaired people and people with reading and writing difficulties to acquire text information by voice in their daily lives. The system includes an image acquisition unit, a communication unit, an external server, a character recognition unit, a voice generation unit, and a voice output unit.

[0039] Program processing overview

[0040] Image capture and transmission

[0041] User:

[0042] The user puts on the smart glasses and directs their gaze toward the text information they want to read (for example, an electronic bulletin board at a station or a sign in town). By pressing a dedicated button on the side of the smart glasses, the image capture means is activated.

[0043] Device:

[0044] The smart glasses' camera is activated and captures an image containing text information. This image is temporarily stored in the smart glasses and then sent to a connected smartphone or computer (device) via Wi-Fi or Bluetooth.

[0045] Image upload and analysis

[0046] Device:

[0047] The device uploads the received image data to an external server using an HTTP request. Specifically, the image data is sent via a REST API endpoint using a POST request.

[0048] server:

[0049] The server temporarily stores the received image data in cloud storage, then passes the stored image data to an OCR engine (e.g., a general character recognition algorithm or a cloud-based OCR service) to extract text information from the image.

[0050] Generate and return audio data

[0051] server:

[0052] The extracted text information is then passed to a TTS engine (e.g., a cloud-based speech generation service) and converted into audio data, which is temporarily stored on the server and then sent back to the device using an HTTP response.

[0053] Device:

[0054] The device analyzes the received audio data, stores it in temporary storage, and then transmits it to the smart glasses via Bluetooth.

[0055] Playing audio

[0056] User:

[0057] Smart glasses can play audio data using built-in audio output means (e.g., bone conduction speakers or earphones), allowing users to hear text information displayed in front of them aloud.

[0058] Specific examples

[0059] In the case of station electronic bulletin boards

[0060] The user presses a button on the smart glasses while looking at the station's electronic bulletin board. This action causes the smart glasses' camera to capture an image of the bulletin board and send it to the device. The device then uploads the image to the server, which uses an OCR engine to extract text information such as "To XX, departure time XX:XX." The TTS engine then converts this text into voice data and sends it back to the smart glasses via the device. Finally, the smart glasses play back this voice data, allowing the user to hear "To XX, departure time XX:XX."

[0061] For signs in the city

[0062] The same procedure applies when a user finds a sign in the city. When the user presses a button on the smart glasses, an image of the sign is captured and uploaded to the server via the device. The server's OCR engine extracts the sign's text information (e.g., "Restaurant XX, business hours XX to XX"), and the TTS engine generates voice data. Finally, the voice data is sent back to the smart glasses and played back, allowing the user to obtain this information by voice.

[0063] This will realize a system that allows visually impaired people to independently and efficiently obtain text information about their daily lives through audio.

[0064] The processing flow will be explained below.

[0065] Step 1:

[0066] User: The user puts on the smart glasses and brings the text information they want to read (e.g., magazines, signs, train station electronic bulletin boards, etc.) into their field of view. Then, they press the button on the side of the smart glasses.

[0067] Step 2:

[0068] Device: The camera built into the smart glasses is activated and captures images containing text information within the field of view. The captured images are temporarily stored in a buffer in the smart glasses in JPEG or PNG format.

[0069] Step 3:

[0070] Terminal: Image data captured from smart glasses is sent to a connected smartphone or computer (terminal) via Wi-Fi or Bluetooth communication.

[0071] Step 4:

[0072] Terminal: The smartphone or PC uploads the received image data to an external server using an HTTP request. Specifically, it sends the image data as a POST request via a REST API endpoint.

[0073] Step 5:

[0074] Server: The server processes the received HTTP request and temporarily stores the image data in cloud storage. After storage is complete, the server prepares to call an OCR engine (e.g., optical character recognition technology).

[0075] Step 6:

[0076] Server: The OCR engine analyzes the images stored in cloud storage and extracts the text information from the images. The extracted text information is temporarily stored in the server's memory as text data.

[0077] Step 7:

[0078] Server: The server then passes the extracted text data to a Text-to-Speech (TTS) engine, which converts the text into audio data, and the generated audio data is saved in a specified directory on the server.

[0079] Step 8:

[0080] Server: The generated audio data is returned to the device as an HTTP response. The successful transmission of the audio data is recorded in the server log.

[0081] Step 9:

[0082] Device: The device analyzes the voice data received from the server and stores it in temporary storage. It then transmits the voice data to the smart glasses via Bluetooth.

[0083] Step 10:

[0084] Terminal: The smart glasses receive the audio data and play it back using the built-in audio output means (e.g., bone conduction speaker or earphones).

[0085] Step 11:

[0086] User: The user can listen to the audio being played through the smart glasses. Use the volume control buttons to adjust the volume as needed.

[0087] The above steps realize a system that enables visually impaired people to quickly and accurately obtain text information in daily life through speech.

[0088] Example 1

[0089] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0090] It is important in daily life for visually impaired people and those who have difficulty reading and writing to quickly and accurately acquire text information. However, conventional methods for acquiring text information by voice have problems such as cumbersome and time-consuming operations, difficulty in synchronizing images and voice, and lack of portability. The purpose of this invention is to solve these problems and provide a system that allows users to easily and quickly acquire text information by voice.

[0091] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0092] In this invention, the server includes image acquisition means for acquiring an image in the direction in which the user is looking, means for transmitting the acquired image to an external server via communication means, character recognition means for extracting text information from the image received by the external server, voice generation means for converting the extracted text information into voice data, means for receiving the generated voice data via the communication means and transmitting it to voice output means via Bluetooth, and voice output means for playing back the generated voice data. This enables visually impaired people and people who have difficulty reading and writing to efficiently acquire text information necessary in daily life by voice.

[0093] The "image acquisition means" is a device or component that has the function of capturing an image in the direction in which the user is looking.

[0094] "Communication means" refers to the technical means for sending and receiving image data and audio data between external servers and terminals, including Wi-Fi and Bluetooth.

[0095] An "external server" is a computer system that communicates with a terminal via a network and has the functions of receiving, analyzing, and saving image data, and generating and returning audio data.

[0096] "Character recognition means" means a technology or device that has the function of extracting text information from an image, and includes character recognition algorithms and services such as OCR.

[0097] "Speech generation means" refers to the technology and devices that convert extracted text information into speech data, and includes text-to-speech engines (TTS engines).

[0098] The "audio output means" is a device that has the function of reproducing the generated audio data, and includes bone conduction speakers, earphones, etc.

[0099] "Cloud storage" is a storage service for storing and managing data on a network, and functions as part of data storage within a server.

[0100] A "wearable device" is a portable electronic device that is worn by a user and includes a camera and audio output means.

[0101] This invention relates to a smart glasses system that enables visually impaired people and people with reading and writing difficulties to obtain text information by voice in their daily lives. The system includes an image acquisition unit, a communication unit, an external server, a character recognition unit, a voice generation unit, and a voice output unit.

[0102] The user wears the smart glasses and directs their gaze toward text information (e.g., a train station bulletin board or sign). The user presses a dedicated button on the side of the smart glasses to activate the image capture means.

[0103] At the device, the camera built into the smart glasses captures an image in the direction of gaze, which is temporarily stored in the smart glasses and then sent to a connected smartphone or computer (device) via Wi-Fi or Bluetooth.

[0104] The device uploads the received image data to an external server using an HTTP request. Specifically, the image data is sent via a REST API endpoint using the POST method.

[0105] The server temporarily stores the received image data in cloud storage, then processes the image using the Python library OpenCV and extracts text information from the image using Tesseract OCR or a cloud-based OCR engine.

[0106] The extracted text information is then passed to a TTS (Text-to-Speech) engine, for example, using a cloud-based speech generation service (e.g., Amazon Polly or Google® Cloud Text-to-Speech), which converts the text into audio data. This audio data is then stored in cloud storage and sent back to the device using an HTTP response.

[0107] The device analyzes the received voice data and temporarily stores it in local storage, then transmits the data to the smart glasses via Bluetooth.

[0108] Finally, users can play audio data using the smart glasses' built-in audio output means (e.g., bone conduction speakers or earphones), allowing them to obtain the text information they are viewing by audio.

[0109] Specific examples

[0110] For example, a user looks at a train station's electronic bulletin board and presses a button on the smart glasses. This operation causes the camera on the smart glasses to capture an image of the bulletin board and send it to the device. The device uploads the image to the server, and the server uses an OCR engine to extract text information such as "To XX, departure time XX:XX." The TTS engine then converts this text into voice data and sends it back to the smart glasses via the device. The user can hear "To XX, departure time XX:XX" aloud through the voice output means of the smart glasses.

[0111] An example of a prompt sentence could be, "Please create a voice guide for when you see the electronic bulletin board at a station." This system will enable visually impaired people and people who have difficulty reading and writing to efficiently obtain text information in their daily lives by voice.

[0112] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0113] Step 1:

[0114] The user puts on the smart glasses and directs their gaze towards text information (e.g., an electronic bulletin board at a station or a sign in town). At this time, the user presses a dedicated button on the smart glasses. This activates the image capture means, and the camera captures an image in the direction of the gaze. The input is the user's gaze and button operation, and the output is the captured image data.

[0115] Step 2:

[0116] The device receives image data temporarily stored in the smart glasses and communicates via Wi-Fi or Bluetooth. The input is image data from the smart glasses, and the output is image data sent to the device. Specifically, the device receives the image and temporarily stores it in memory.

[0117] Step 3:

[0118] The device uploads the received image data to an external server using an HTTP request. Specifically, the device sends the image data to a REST API endpoint using the POST method. The input is the image data stored on the device, and the output is the HTTP request sent to the server. During this process, the device performs communication processing to send the data over the network.

[0119] Step 4:

[0120] The server temporarily stores the received image data in cloud storage. Using the stored image data, the server passes the image to an OCR engine (e.g., Tesseract OCR or Google Cloud Vision API) to extract text information. The input is the image data stored in cloud storage, and the output is the extracted text information. Specifically, the server analyzes the image data using an image processing library.

[0121] Step 5:

[0122] The server passes the extracted text information to a TTS engine (e.g., Amazon Polly or Google Cloud Text-to-Speech) and converts it into speech data. The input is the extracted text information, and the output is the generated speech data. The server executes the text-to-speech conversion process and stores the results in cloud storage.

[0123] Step 6:

[0124] The server returns the generated audio data to the device using an HTTP response. The input is the audio data stored in cloud storage, and the output is an HTTP response to the device. Specifically, the server performs communication processing to send the audio data to the device.

[0125] Step 7:

[0126] The device analyzes the received voice data and temporarily stores it in local storage. The device then transmits the voice data to the smart glasses using Bluetooth. The input is the voice data received from the server, and the output is the voice data transmitted to the smart glasses. Specifically, the device transfers the voice data via Bluetooth communication.

[0127] Step 8:

[0128] The user plays audio data using the smart glasses' built-in audio output means (bone conduction speaker or earphones). The input is audio data received via Bluetooth, and the output is the played audio. Specifically, the smart glasses process the received audio data and provide audio information to the user through the audio output means.

[0129] The above steps realize a system that enables visually impaired people and users who have difficulty reading and writing to efficiently obtain text information by voice.

[0130] (Application example 1)

[0131] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0132] In factories, workers need to quickly and accurately obtain information from machine control panels and signs, but visual confirmation alone takes time and carries the risk of misreading. This results in problems of reduced work efficiency. Furthermore, visually impaired workers in particular have difficulty operating equipment and checking signs, making it difficult to ensure work safety and efficiency.

[0133] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0134] In this invention, the server includes image acquisition means, means for transmitting images to an external server via communication means, character recognition means for extracting text information from the images received by the external server, voice generation means for converting the extracted text information into voice data, voice output means for receiving and playing the generated voice data via the communication means, and means for recognizing machine operation panels and signs in the factory and conveying that information to workers by voice. This allows workers to obtain necessary information by voice without having to confirm the information visually, improving work efficiency and ensuring safety.

[0135] An "image capture means" is a device for capturing visual information as digital data.

[0136] A "communication means" is a device or technology for sending and receiving digital data to other devices or servers.

[0137] An "external server" is a remote computer system for receiving, processing, storing, or transmitting digital data.

[0138] "Character recognition means" refers to software or hardware that analyzes characters and symbols in image data and extracts them as text data.

[0139] "Speech generation means" refers to technology or devices for converting text data into speech data.

[0140] The "audio output means" is a device for reproducing audio data so that it can be recognized by the human ear.

[0141] "Means of recognizing machine control panels and signs in a factory and communicating that information to workers by voice" refers to technology or equipment that uses a camera to capture important information from machine control panels and signs in a factory, converts that information into text data, and then converts it into audio data to communicate it to workers by voice.

[0142] "Cloud storage" is a remote server where data can be stored and accessed over the internet.

[0143] A "wearable device" is an electronic device that can be worn by a user and is often equipped with a camera, communication means, etc.

[0144] A "mobile device" is a portable electronic device such as a mobile phone or tablet.

[0145] A "factory robot" is an automated machine designed to perform work within a factory, and may have the role of monitoring work and conveying instructions.

[0146] The present invention provides a system for communicating text information displayed on machine control panels and signs to workers in a factory by voice. The system includes an image acquisition unit, a communication unit, an external server, a character recognition unit, a voice generation unit, and a voice output unit.

[0147] System Program Overview

[0148] 1. Image capture and transmission

[0149] The robot uses a camera to capture images of control panels and signs in the factory, and the images are temporarily stored in the robot's internal computer.

[0150] 2. Image upload and analysis

[0151] The robot's internal computer uploads image data to an external server using an HTTP request, which then stores the data in cloud storage and uses an OCR engine to extract text information from the image.

[0152] 3. Generating and returning audio data

[0153] The server passes the extracted text information to the TTS engine, which generates voice data, which is then sent back to the robot's computer using an HTTP response.

[0154] 4. Audio playback

[0155] The computer inside the robot analyzes the received audio data and plays the audio using the built-in speaker or an audio output device connected via Bluetooth.

[0156] Hardware and software used

[0157] Hardware: A high-resolution camera mounted on the robot is used for image acquisition, a high-performance computer within the robot is used for image processing and communication, and built-in speakers or a Bluetooth headset are used for audio output.

[0158] Software: Use a cloud-based OCR engine (e.g., Google Cloud Vision API) for image analysis and a cloud-based TTS engine (e.g., Amazon Polly) for voice generation.

[0159] Specific examples of processing

[0160] Example 1:

[0161] The robot uses a camera to capture an image of the factory's control panel and recognizes the text, "Machine status: Operating normally." This text information is converted into voice data, which is then played back from the robot's speaker: "Machine status is operating normally."

[0162] Example 2:

[0163] The robot uses its camera to capture factory signs, recognizes the text "Warehouse A: Material Delivery Area," and relays this information to workers via voice, allowing them to obtain the necessary information without having to visually confirm it.

[0164] Prompt Sentence Examples

[0165] "When the robot is in front of an operation panel in the factory, please generate a program that will capture the text on the operation panel using a camera and convey this text information to the worker by voice."

[0166] This configuration allows for efficient and safe work within the factory.

[0167] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0168] Step 1:

[0169] Image capture and saving

[0170] The robot uses a camera to capture images of control panels and signs in the factory, and the image data captured by the camera is temporarily stored in the robot's internal computer.

[0171] Input: Actual images of control panels and signs.

[0172] Data processing: The camera converts visual information into digital image data.

[0173] Output: Image data stored in the robot's internal computer.

[0174] Step 2:

[0175] Uploading an image

[0176] The computer inside the robot uploads the captured image data to an external server using HTTP requests, which uses a REST API.

[0177] Input: Stored image data.

[0178] Data processing: Convert image data into HTTP request format.

[0179] Output: Image data sent to an external server.

[0180] Step 3:

[0181] Extracting text information

[0182] The server stores the received image data in cloud storage and then uses an OCR engine to extract text information from the image.

[0183] Input: Image data stored on an external server.

[0184] Data processing: The OCR engine analyzes characters from image data and extracts text data.

[0185] Output: The extracted text information.

[0186] Step 4:

[0187] Generate audio data

[0188] The server passes the extracted text information to a TTS engine to generate audio data, which then converts the text data into an audio file.

[0189] Input: The extracted text information.

[0190] Data processing: The TTS engine converts the text into an audio file.

[0191] Output: The generated audio data.

[0192] Step 5:

[0193] Returning audio data

[0194] The server sends the generated audio data back to the computer in the robot using an HTTP response.

[0195] Input: The generated audio data.

[0196] Data processing: Converts audio data into HTTP response format.

[0197] Output: Audio data sent to the computer inside the robot.

[0198] Step 6:

[0199] Playing audio

[0200] The computer inside the robot analyzes the received audio data and plays the audio using the built-in speaker or an audio output device connected via Bluetooth.

[0201] Input: Audio data stored in the robot's internal computer.

[0202] Data processing: Converting audio data into a playable format.

[0203] Output: Audio played through speakers or headset.

[0204] The above are the specific processing steps of the program for the system that realizes the application example.

[0205] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0206] The present invention provides a smart glasses system that allows visually impaired people and people with reading and writing difficulties to acquire textual information by voice in their daily lives, recognizes the user's emotional state during the information acquisition process, and responds appropriately accordingly. The system includes an image acquisition unit, a communication unit, an external server, a character recognition unit, a voice generation unit, a voice output unit, and an emotion recognition unit.

[0207] Program processing overview

[0208] Image capture and transmission

[0209] User:

[0210] The user puts on the smart glasses and directs their gaze toward the text information they want to read. By pressing a button on the side of the smart glasses, the image capture means is activated.

[0211] Device:

[0212] The camera built into the smart glasses is activated to capture images, including text information, of the field of view. These images are temporarily stored in a buffer in the smart glasses in JPEG or PNG format and then sent to a connected smartphone or computer (device) via Wi-Fi or Bluetooth.

[0213] Image upload and analysis

[0214] Device:

[0215] The device uploads the received image data to an external server using an HTTP request. Specifically, the device sends the image data as a POST request via a REST API endpoint.

[0216] server:

[0217] The server temporarily stores the received image data in cloud storage.The image stored in cloud storage is then analyzed using an OCR engine to extract text information from the image.The extracted text data is temporarily stored in the server's memory.

[0218] Generate and return audio data

[0219] server:

[0220] The extracted text information is then passed to the TTS engine and converted into voice data, which is then saved in a specified directory on the server and sent back to the device as an HTTP response.

[0221] Device:

[0222] The device analyzes the received voice data, stores it in temporary storage, and then transmits the data to the smart glasses via Bluetooth.

[0223] Playing audio

[0224] Device:

[0225] The smart glasses receive the audio data and play it back using the built-in audio output means.

[0226] User:

[0227] The user can listen to the audio being played through the smart glasses.

[0228] Emotion recognition and response

[0229] Operation of emotion recognition means

[0230] Device:

[0231] The emotion recognition means is designed to analyze the user's facial expressions and voice data in real time, for example, by capturing the user's facial expressions and tone of voice through a camera and microphone installed in the smart glasses and analyzing the data.

[0232] server:

[0233] The analyzed data is sent to a server, where an emotion recognition engine determines the user's emotional state. If the emotional state is negative (sadness, anger, discomfort, etc.), the server generates a message of comfort or encouragement. On the other hand, if the emotional state is positive (joy, relief, excitement, etc.), it generates a recommended message to maintain that state.

[0234] Responding according to emotions

[0235] server:

[0236] The generated message is converted into voice data and sent from the server to the device, which then transmits the received voice data to the smart glasses.

[0237] Device:

[0238] An appropriate message is played to the user through the audio output means of the smart glasses.

[0239] Specific examples

[0240] In the case of station electronic bulletin boards

[0241] The user looks at the station's electronic display board and presses a button on the smart glasses. This action causes the smart glasses' camera to capture an image of the display board and send it to the device. The device then uploads the image to the server, which uses an OCR engine to extract "Travel to XX, departure time XX:XX." The TTS engine generates voice data and sends it back to the smart glasses via the device. The user listens to this voice. Meanwhile, the emotion recognition means analyzes the user's emotions and generates an appropriate response. For example, if a negative emotion is detected, a comforting message such as "Don't worry, the next train will arrive at XX:XX" is played.

[0242] For signs in the city

[0243] A similar procedure is applied when a user finds a sign in the city and presses a button on the smart glasses. An image of the sign is captured and sent to the server via the device. The server's OCR engine extracts the text information and generates audio data such as "Restaurant XX, business hours XX to XX." The audio data is sent back to the smart glasses, which the user listens to. During this time, if the emotion recognition means detects a positive emotion, a recommendation message such as "This restaurant has a very good reputation" is played.

[0244] In this way, a system is realized that enables visually impaired people to remain independent while efficiently obtaining textual information in daily life through speech, and in the process provides appropriate responses according to their emotions.

[0245] The processing flow will be explained below.

[0246] Step 1:

[0247] User: The user puts on the smart glasses and directs their gaze towards the text information they want to read (e.g., a magazine, a sign, a train station electronic bulletin board, etc.), then presses the button on the side of the smart glasses.

[0248] Step 2:

[0249] Device: The camera built into the smart glasses is activated and captures images containing text information within the field of view. The captured images are temporarily stored in a buffer in the smart glasses in JPEG or PNG format.

[0250] Step 3:

[0251] Terminal: Image data captured from smart glasses is sent to a connected smartphone or computer (terminal) via Wi-Fi or Bluetooth communication.

[0252] Step 4:

[0253] Terminal: The smartphone or PC uploads the received image data to an external server using an HTTP request. Specifically, it sends the image data as a POST request via a REST API endpoint.

[0254] Step 5:

[0255] Server: The server processes the received HTTP request and temporarily stores the image data in cloud storage. After storage is complete, the server prepares to call an OCR engine (e.g., optical character recognition technology).

[0256] Step 6:

[0257] Server: The OCR engine analyzes the images stored in cloud storage and extracts the text information from the images. The extracted text information is temporarily stored in the server's memory as text data.

[0258] Step 7:

[0259] Server: The server then passes the extracted text data to a Text-to-Speech (TTS) engine, which converts the text into audio data, and the generated audio data is saved in a specified directory on the server.

[0260] Step 8:

[0261] Server: The generated audio data is returned to the device as an HTTP response. The successful transmission of the audio data is recorded in the server log.

[0262] Step 9:

[0263] Device: The device analyzes the voice data received from the server and stores it in temporary storage. It then transmits the voice data to the smart glasses via Bluetooth.

[0264] Step 10:

[0265] Terminal: The smart glasses receive the audio data and play it back using the built-in audio output means (e.g., bone conduction speaker or earphones).

[0266] Step 11:

[0267] User: The user can listen to the audio being played through the smart glasses. Use the volume control buttons to adjust the volume as needed.

[0268] Step 12:

[0269] Device: The emotion recognition system captures the user's facial expressions and tone of voice through the smart glasses' camera and microphone. The captured data is then analyzed in real time by the device.

[0270] Step 13:

[0271] Terminal: The emotion recognition means determines the user's emotional state based on the analysis data, and determines whether the user is feeling negative (e.g., sadness, anger, discomfort) or positive (e.g., joy, relief, excitement).

[0272] Step 14:

[0273] Device: If a negative emotional state is detected, the device generates a comforting or encouraging message and prepares to send it to the server as audio data. On the other hand, if a positive emotional state is detected, the device generates a recommendation message for maintaining that state and prepares to send it to the server as audio data.

[0274] Step 15:

[0275] Server: Receives message data generated by the device and converts it into voice data using a TTS engine. The generated voice data is then sent back to the device.

[0276] Step 16:

[0277] Terminal: Transmits the voice data received from the server to the smart glasses.

[0278] Step 17:

[0279] Terminal: An appropriate emotional message is played to the user through the audio output means of the smart glasses.

[0280] Specific examples

[0281] In the case of station electronic bulletin boards

[0282] The user looks at the station's electronic display board and presses a button on the smart glasses. This action causes the smart glasses' camera to capture an image of the display board and send it to the device. The device then uploads the image to the server, which uses an OCR engine to extract "Travel to XX, departure time XX:XX." The TTS engine generates voice data and sends it back to the smart glasses via the device. The user listens to this voice. Meanwhile, the emotion recognition means analyzes the user's emotions and generates an appropriate response. For example, if a negative emotion is detected, a comforting message such as "Don't worry, the next train will arrive at XX:XX" is played.

[0283] For signs in the city

[0284] A similar procedure is applied when a user finds a sign in the city and presses a button on the smart glasses. An image of the sign is captured and sent to the server via the device. The server's OCR engine extracts the text information and generates audio data such as "Restaurant XX, business hours XX to XX." The audio data is sent back to the smart glasses, which the user listens to. During this time, if the emotion recognition means detects a positive emotion, a recommendation message such as "This restaurant has a very good reputation" is played.

[0285] This will enable visually impaired people to remain independent and efficiently obtain textual information in their daily lives through audio, and a system will be realized that provides appropriate responses based on emotions in the process.

[0286] Example 2

[0287] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0288] Visually impaired people and those with reading and writing difficulties face the challenge of efficiently acquiring textual information in their daily lives. Even when textual information is acquired, the user's emotional state is often not properly understood, and necessary support and encouragement are often not provided. To address this issue, a system that not only converts textual information into speech, but also recognizes the user's emotional state in real time and responds accordingly is needed.

[0289] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0290] In this invention, the server includes image acquisition means, means for transmitting the acquired image to an external server via communication means, character recognition means for extracting text information from the image received by the external server, voice generation means for converting the extracted text information into voice data, voice output means for receiving the generated voice data via the communication means and playing it back, emotion recognition means for analyzing the emotional state of the user, and means for generating an appropriate message according to the user's emotional state and converting it into voice data. This allows the user to efficiently acquire text information by voice, recognize the emotional state in real time during the process, and provide an appropriate response.

[0291] "Image capture means" refers to a device or method that electronically images information visually captured by a user using a device's camera.

[0292] "Communication means" refers to a method or device for transmitting acquired data to other devices or systems, including, for example, Wi-Fi and Bluetooth.

[0293] An "external server" is an information processing device that is connected via a network, separate from the device used by the user, and has the role of analyzing and storing data.

[0294] "Character recognition means" refers to technology or devices that detect character information contained in image data and extract it as text data. For example, this includes OCR (optical character recognition) technology.

[0295] "Speech generation means" refers to technology or equipment that converts text data into speech data, for example, using TTS (Text to Speech) technology.

[0296] "Audio output means" refers to a device or method for outputting generated audio data in a form that can be heard by the user, and includes speakers and earphones.

[0297] "Emotion recognition means" refers to technology or devices that analyze a user's facial expression or tone of voice to determine their emotional state. Examples include facial expression recognition and voice emotion analysis.

[0298] The "means for generating an appropriate message and converting it into voice data" refers to a technology or device that creates a message corresponding to the user's emotional state based on the user's emotional state obtained by the emotion recognition means, and converts that message into voice data.

[0299] This invention is a smart glasses system that allows visually impaired people and people with reading and writing difficulties to acquire text information by voice in their daily lives, recognize the user's emotional state during the information acquisition process, and respond appropriately accordingly. This system includes image acquisition means, communication means, an external server, character recognition means, voice generation means, voice output means, and emotion recognition means.

[0300] Image Acquisition Method

[0301] The user puts on the smart glasses, directs their gaze at the text information they want to read, and presses a button on the smart glasses. This action activates the camera built into the smart glasses, which captures an image containing the text information in their field of view. The image is temporarily saved in a buffer in the smart glasses in JPEG or PNG format.

[0302] communication means

[0303] The acquired image data is sent to a connected smartphone or PC (terminal) via Wi-Fi or Bluetooth. The terminal then sends the received image data to an external server as an HTTP request. Specifically, the image data is sent as a POST request via a REST API endpoint.

[0304] External Server

[0305] The server receives the POST request and temporarily stores the image data in cloud storage.The image stored in cloud storage is then analyzed using an OCR engine (for example, Google Cloud Vision API) to extract text information from the image.The extracted text data is temporarily stored in the server's memory.

[0306] Character recognition means

[0307] The character recognition means is a technology that uses an OCR engine to extract characters from image data. The extracted text data is passed to the voice generation means.

[0308] Voice generation means

[0309] The voice generation means converts the extracted text information into voice data using a TTS engine (for example, Google Text-to-Speech API). The generated voice data is saved in a specified directory on the server and then returned to the terminal as an HTTP response.

[0310] Audio output means

[0311] The device analyzes the received audio data and stores it in temporary storage. It then transmits the audio data to the smart glasses via Bluetooth. The smart glasses receive the audio data and play it back through their built-in speakers or earphones, allowing the user to listen to the audio.

[0312] emotion recognition means

[0313] The emotion recognition means uses a camera and microphone installed in the smart glasses to capture and analyze the user's facial expressions and tone of voice in real time. This determines the user's emotional state in relation to the recognized text information. Analysis is performed by an emotion recognition engine (e.g., Amazon Rekognition or Microsoft® Azure® Emotion API). If a negative emotion (sadness, anger, displeasure, etc.) is detected, a comforting or encouraging message is generated. If a positive emotion (joy, relief, excitement, etc.) is detected, a recommended message to maintain that state is generated.

[0314] For example, when a user looks at a train station's electronic display board and presses a button on the smart glasses, the camera in the smart glasses captures an image of the display board. The device uploads the image to a server, and the server's OCR engine extracts "To XX, departure time XX:XX." The TTS engine generates voice data and sends it back to the smart glasses via the device. The user listens to this voice. Meanwhile, the emotion recognition means analyzes the user's emotions, and if a negative emotion is detected, a comforting message such as "Don't worry, the next train will arrive at XX:XX" is played.

[0315] The same process is also performed when the user gazes at a sign in the city. Text information is extracted from the image of the sign, and audio data such as "Restaurant XX, business hours XX to XX" is generated and sent back to the smart glasses. If the emotion recognition means detects a positive emotion while the user is listening to the audio, a message such as "This restaurant has a very good reputation" is played.

[0316] This system enables visually impaired people to independently access textual information in their daily lives efficiently through speech, and provides appropriate responses according to their emotions in the process.

[0317] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0318] Step 1:

[0319] The user puts on the smart glasses, directs their gaze at the text information they want to read, and presses a button on the smart glasses. This action activates the camera built into the smart glasses, which captures an image containing the text information in their field of view. The captured image is temporarily saved in a buffer within the smart glasses in JPEG or PNG format. The input is the user's gaze and button press, and the output is the image data saved in the smart glasses buffer.

[0320] Step 2:

[0321] The device receives temporarily stored image data from the smart glasses via Wi-Fi or Bluetooth. The received image data is temporarily stored in the device's internal storage. The device then uploads this image data to an external server using an HTTP request. Specifically, the device sends the image data as a POST request via a REST API endpoint. The input is the image data from the smart glasses, and the output is an HTTP request to the external server.

[0322] Step 3:

[0323] The server processes the received POST request and temporarily stores the image data in cloud storage. Next, the image data stored in cloud storage is passed to an OCR engine (for example, Google Cloud Vision API) for analysis. The OCR engine detects character information in the image and extracts it as text data. The extracted text data is temporarily stored in the server's memory. The input is the image data in cloud storage, and the output is the extracted text data.

[0324] Step 4:

[0325] The server passes the extracted text data to a TTS engine (for example, Google Text-to-Speech API) and converts it into audio data. This audio data is saved in a specified directory on the server and then sent back to the device as an HTTP response. The input is the text data obtained by the OCR engine, and the output is the generated audio data.

[0326] Step 5:

[0327] The device analyzes the audio data in the received HTTP response and stores it in temporary storage. It then transmits the audio data to the smart glasses via Bluetooth. The input is the audio data returned from the server, and the output is the audio data sent to the smart glasses.

[0328] Step 6:

[0329] Smart glasses play back audio data received via Bluetooth. The audio is transmitted to the user through built-in speakers or earphones. The input is the audio data sent from the device, and the output is the played audio. The user can listen to this audio.

[0330] Step 7:

[0331] Using the smart glasses' built-in camera and microphone, the device captures the user's facial expressions and tone of voice in real time and passes them to the emotion recognition means. The emotion recognition means analyzes the data and determines the user's emotional state. The input is the facial expressions and tone of voice captured in real time, and the output is the analyzed emotional data.

[0332] Step 8:

[0333] The server generates a message of comfort or encouragement based on the user's emotional state obtained from the emotion recognition means if a negative emotion is detected, and generates a message of recommendation to maintain that state if a positive emotion is detected. The generated message is converted back into voice data and sent to the device. The input is the analyzed emotion data, and the output is the voice data of the generated message.

[0334] Step 9:

[0335] The device sends audio data based on the received emotional response to the smart glasses, which then plays it through the built-in speaker or earphones. This allows the user to receive an appropriate response message according to their emotional state. The input is the emotional response audio data from the server, and the output is the audio played by the smart glasses.

[0336] (Application example 2)

[0337] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0338] Visually impaired people and people with reading and writing difficulties face difficulties in efficiently obtaining product information in physical stores. Furthermore, there is a need for a system that can accurately recognize the user's emotional state during the information acquisition process and provide appropriate responses and support according to that emotion. Furthermore, it is necessary to realize a system in which a wearable device worn by the user can perform these tasks in an integrated manner.

[0339] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes image acquisition means, means for transmitting the acquired image to an external server via communication means, character recognition means for extracting text information from the image received by the external server, voice generation means for converting the extracted text information into voice data, voice output means for receiving and playing the generated voice data via the communication means, emotion recognition means for recognizing an emotional state, and means for generating an appropriate message according to the recognized emotional state and converting it into voice data. This enables visually impaired people and people who have difficulty reading and writing to not only efficiently obtain product information in physical stores, but also receive appropriate responses and support according to their emotions during the process.

[0340] "Image acquisition means" refers to a camera or imaging device for capturing visual information as digital data.

[0341] "Communication means" refers to a means for bidirectionally communicating information such as image data and audio data with external devices or servers, and utilizes wireless communication technologies such as Wi-Fi and Bluetooth.

[0342] An "external server" is a remote server that performs processes such as analyzing image data, extracting text information, and generating audio data.

[0343] "Character recognition means" refers to optical character recognition (OCR) technology for extracting text information from captured image data.

[0344] The "voice generation means" is a voice synthesis technology for converting the extracted text information into voice data.

[0345] The "audio output means" is a device that plays back audio to notify the user of the generated audio data, and includes a speaker, a bone conduction speaker, and the like.

[0346] "Emotion recognition means" is a technology for analyzing the user's emotional state, and uses a camera and microphone to analyze facial expressions and vocal tones.

[0347] The "means for generating an appropriate message" is a technology for generating an appropriate message to be provided to the user in accordance with the recognized emotional state and converting it into voice data.

[0348] A "wearable device" is a portable electronic device that can be worn by a user and is used to acquire biometric data and present information.

[0349] This invention provides a system that enables visually impaired people and people with reading and writing difficulties to efficiently obtain product information in physical stores and receive appropriate support according to their emotions during the process. The system includes smart glasses, image acquisition means, communication means, an external server, character recognition means, voice generation means, voice output means, emotion recognition means, means for generating appropriate messages, and a wearable device.

[0350] System Program

[0351] Image capture and transmission

[0352] When a user wears the smart glasses, points their eyes at a product, and presses a button, the camera built into the smart glasses captures the product and sends the image to a device. The device then uploads the received image data to an external server via Wi-Fi or Bluetooth. This process uses software tools such as OpenCV and requests.

[0353] Image upload and analysis

[0354] The device sends the received image data to an external server using an HTTP request. The server temporarily stores the image in cloud storage and extracts text information from the image using an OCR engine. The extracted text information is temporarily stored in the server's memory.

[0355] Generate and return audio data

[0356] The server passes the extracted text information to the TTS engine to generate voice data. The generated voice data is saved in a specified directory on the server and sent back to the device as an HTTP response. The device then sends the received voice data to the smart glasses and plays the voice.

[0357] Emotion recognition and appropriate responses

[0358] The emotion recognition means captures the user's facial expressions and voice using the smart glasses' camera and microphone and analyzes the data. The analyzed data is sent to the server, where the emotion recognition engine determines the user's emotional state. If the emotional state is negative, the server generates a comforting or encouraging message. If the emotional state is positive, it generates a recommendation message to maintain that state. The generated message is converted into audio data and sent to the smart glasses via the terminal, where an appropriate message is played for the user.

[0359] Hardware and software used

[0360] Hardware: Smart glasses (including built-in camera and audio output means), smartphone or laptop

[0361] Software: Python, OpenCV (image capture and processing), requests (HTTP requests), playsound (audio playback), OCR engine, TTS engine

[0362] Specific examples

[0363] Prompt Sentence Examples

[0364] Image Recognition: "Please read out the product information I'm looking at"

[0365] Emotion recognition: "The user looks sad, so generate a comforting message."

[0366] This system enables visually impaired and people with reading and writing difficulties to efficiently obtain product information in physical stores, while also receiving appropriate emotional support in the process, thereby supporting users' independence and improving the quality of their daily lives.

[0367] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0368] Step 1:

[0369] The user puts on the smart glasses, looks at the product, and presses a button. The camera built into the smart glasses captures the product and sends the image to the device. The input is the product image, and the output is the image data sent to the device.

[0370] Step 2:

[0371] The device uploads the received image data to an external server via Wi-Fi or Bluetooth. During this process, the device sends the image using an HTTP POST request. The input is the image data received from the smart glasses, and the output is the image data sent to the server.

[0372] Step 3:

[0373] The server temporarily stores the received image data in cloud storage. Then, it uses an OCR engine to extract text information from the image. The input is the image data stored in cloud storage, and the output is the extracted text information.

[0374] Step 4:

[0375] The server passes the extracted text information to the TTS engine to generate voice data. The generated voice data is saved in a specified directory on the server and sent back to the device as an HTTP response. The input is the extracted text information, and the output is the generated voice data.

[0376] Step 5:

[0377] The device analyzes the received voice data and stores it in temporary storage.Then, it transmits the voice data to the smart glasses via Bluetooth.The input is the voice data received from the server, and the output is the voice data transmitted to the smart glasses.

[0378] Step 6:

[0379] The audio output means of the smart glasses receives the audio data and plays the audio through the built-in speaker. The input is the audio data received from the terminal, and the output is the audio heard by the user.

[0380] Step 7:

[0381] The emotion recognition means captures the user's facial expressions and voice through the camera and microphone of the smart glasses and analyzes the data. The input is the user's facial expressions and voice data, and the output is the analyzed emotion data.

[0382] Step 8:

[0383] The server passes the analyzed emotional data to an emotion recognition engine to determine the user's emotional state. If the emotional state is negative, it generates a comforting or encouraging message, and if it is positive, it generates a recommendation message. The input is the analyzed emotional data, and the output is an appropriate message.

[0384] Step 9:

[0385] The server passes the generated message to the TTS engine to convert it into voice data, which is then sent to the smart glasses via the device. The input is the generated message, and the output is voice data.

[0386] Step 10:

[0387] The smart glasses receive the audio data and play it back on the built-in speaker. The input is the audio data sent from the server, and the output is the audio the user hears.

[0388] The above series of processes realizes a system that allows users to obtain product information through voice and receive appropriate responses according to their emotions during the process.

[0389] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0390] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0391] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0392] [Second embodiment]

[0393] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0394] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0395] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0396] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0397] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0398] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0399] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0400] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0401] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0402] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0403] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0404] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0405] The present invention provides a smart glasses system that enables visually impaired people and people with reading and writing difficulties to acquire text information by voice in their daily lives. The system includes an image acquisition unit, a communication unit, an external server, a character recognition unit, a voice generation unit, and a voice output unit.

[0406] Program processing overview

[0407] Image capture and transmission

[0408] User:

[0409] The user puts on the smart glasses and directs their gaze toward the text information they want to read (for example, an electronic bulletin board at a station or a sign in town). By pressing a dedicated button on the side of the smart glasses, the image capture means is activated.

[0410] Device:

[0411] The smart glasses' camera is activated and captures an image containing text information. This image is temporarily stored in the smart glasses and then sent to a connected smartphone or computer (device) via Wi-Fi or Bluetooth.

[0412] Image upload and analysis

[0413] Device:

[0414] The device uploads the received image data to an external server using an HTTP request. Specifically, the image data is sent via a REST API endpoint using a POST request.

[0415] server:

[0416] The server temporarily stores the received image data in cloud storage, then passes the stored image data to an OCR engine (e.g., a general character recognition algorithm or a cloud-based OCR service) to extract text information from the image.

[0417] Generate and return audio data

[0418] server:

[0419] The extracted text information is then passed to a TTS engine (e.g., a cloud-based speech generation service) and converted into audio data, which is temporarily stored on the server and then sent back to the device using an HTTP response.

[0420] Device:

[0421] The device analyzes the received audio data, stores it in temporary storage, and then transmits it to the smart glasses via Bluetooth.

[0422] Playing audio

[0423] User:

[0424] Smart glasses can play audio data using built-in audio output means (e.g., bone conduction speakers or earphones), allowing users to hear text information displayed in front of them aloud.

[0425] Specific examples

[0426] In the case of station electronic bulletin boards

[0427] The user presses a button on the smart glasses while looking at the station's electronic bulletin board. This action causes the smart glasses' camera to capture an image of the bulletin board and send it to the device. The device then uploads the image to the server, which uses an OCR engine to extract text information such as "To XX, departure time XX:XX." The TTS engine then converts this text into voice data and sends it back to the smart glasses via the device. Finally, the smart glasses play back this voice data, allowing the user to hear "To XX, departure time XX:XX."

[0428] For signs in the city

[0429] The same procedure applies when a user finds a sign in the city. When the user presses a button on the smart glasses, an image of the sign is captured and uploaded to the server via the device. The server's OCR engine extracts the sign's text information (e.g., "Restaurant XX, business hours XX to XX"), and the TTS engine generates voice data. Finally, the voice data is sent back to the smart glasses and played back, allowing the user to obtain this information by voice.

[0430] This will realize a system that allows visually impaired people to independently and efficiently obtain text information about their daily lives through audio.

[0431] The processing flow will be explained below.

[0432] Step 1:

[0433] User: The user puts on the smart glasses and brings the text information they want to read (e.g., magazines, signs, train station electronic bulletin boards, etc.) into their field of view. Then, they press the button on the side of the smart glasses.

[0434] Step 2:

[0435] Device: The camera built into the smart glasses is activated and captures images containing text information within the field of view. The captured images are temporarily stored in a buffer in the smart glasses in JPEG or PNG format.

[0436] Step 3:

[0437] Terminal: Image data captured from smart glasses is sent to a connected smartphone or computer (terminal) via Wi-Fi or Bluetooth communication.

[0438] Step 4:

[0439] Terminal: The smartphone or PC uploads the received image data to an external server using an HTTP request. Specifically, it sends the image data as a POST request via a REST API endpoint.

[0440] Step 5:

[0441] Server: The server processes the received HTTP request and temporarily stores the image data in cloud storage. After storage is complete, the server prepares to call an OCR engine (e.g., optical character recognition technology).

[0442] Step 6:

[0443] Server: The OCR engine analyzes the images stored in cloud storage and extracts the text information from the images. The extracted text information is temporarily stored in the server's memory as text data.

[0444] Step 7:

[0445] Server: The server then passes the extracted text data to a Text-to-Speech (TTS) engine, which converts the text into audio data, and the generated audio data is saved in a specified directory on the server.

[0446] Step 8:

[0447] Server: The generated audio data is returned to the device as an HTTP response. The successful transmission of the audio data is recorded in the server log.

[0448] Step 9:

[0449] Device: The device analyzes the voice data received from the server and stores it in temporary storage. It then transmits the voice data to the smart glasses via Bluetooth.

[0450] Step 10:

[0451] Terminal: The smart glasses receive the audio data and play it back using the built-in audio output means (e.g., bone conduction speaker or earphones).

[0452] Step 11:

[0453] User: The user can listen to the audio being played through the smart glasses. Use the volume control buttons to adjust the volume as needed.

[0454] The above steps realize a system that enables visually impaired people to quickly and accurately obtain text information in daily life through speech.

[0455] Example 1

[0456] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0457] It is important in daily life for visually impaired people and those who have difficulty reading and writing to quickly and accurately acquire text information. However, conventional methods for acquiring text information by voice have problems such as cumbersome and time-consuming operations, difficulty in synchronizing images and voice, and lack of portability. The purpose of this invention is to solve these problems and provide a system that allows users to easily and quickly acquire text information by voice.

[0458] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0459] In this invention, the server includes image acquisition means for acquiring an image in the direction in which the user is looking, means for transmitting the acquired image to an external server via communication means, character recognition means for extracting text information from the image received by the external server, voice generation means for converting the extracted text information into voice data, means for receiving the generated voice data via the communication means and transmitting it to voice output means via Bluetooth, and voice output means for playing back the generated voice data. This enables visually impaired people and people who have difficulty reading and writing to efficiently acquire text information necessary in daily life by voice.

[0460] The "image acquisition means" is a device or component that has the function of capturing an image in the direction in which the user is looking.

[0461] "Communication means" refers to the technical means for sending and receiving image data and audio data between external servers and terminals, including Wi-Fi and Bluetooth.

[0462] An "external server" is a computer system that communicates with a terminal via a network and has the functions of receiving, analyzing, and saving image data, and generating and returning audio data.

[0463] "Character recognition means" means a technology or device that has the function of extracting text information from an image, and includes character recognition algorithms and services such as OCR.

[0464] "Speech generation means" refers to the technology and devices that convert extracted text information into speech data, and includes text-to-speech engines (TTS engines).

[0465] The "audio output means" is a device that has the function of reproducing the generated audio data, and includes bone conduction speakers, earphones, etc.

[0466] "Cloud storage" is a storage service for storing and managing data on a network, and functions as part of data storage within a server.

[0467] A "wearable device" is a portable electronic device that is worn by a user and includes a camera and audio output means.

[0468] This invention relates to a smart glasses system that enables visually impaired people and people with reading and writing difficulties to obtain text information by voice in their daily lives. The system includes an image acquisition unit, a communication unit, an external server, a character recognition unit, a voice generation unit, and a voice output unit.

[0469] The user wears the smart glasses and directs their gaze toward text information (e.g., a train station bulletin board or sign). The user presses a dedicated button on the side of the smart glasses to activate the image capture means.

[0470] At the device, the camera built into the smart glasses captures an image in the direction of gaze, which is temporarily stored in the smart glasses and then sent to a connected smartphone or computer (device) via Wi-Fi or Bluetooth.

[0471] The device uploads the received image data to an external server using an HTTP request. Specifically, the image data is sent via a REST API endpoint using the POST method.

[0472] The server temporarily stores the received image data in cloud storage, then processes the image using the Python library OpenCV and extracts text information from the image using Tesseract OCR or a cloud-based OCR engine.

[0473] The extracted text information is then passed to a TTS (Text-to-Speech) engine, for example, using a cloud-based speech generation service (e.g., Amazon Polly or Google Cloud Text-to-Speech), which converts the text into audio data that is again stored in cloud storage and then sent back to the device using an HTTP response.

[0474] The device analyzes the received voice data and temporarily stores it in local storage, then transmits the data to the smart glasses via Bluetooth.

[0475] Finally, users can play audio data using the smart glasses' built-in audio output means (e.g., bone conduction speakers or earphones), allowing them to obtain the text information they are viewing by audio.

[0476] Specific examples

[0477] For example, a user looks at a train station's electronic bulletin board and presses a button on the smart glasses. This operation causes the camera on the smart glasses to capture an image of the bulletin board and send it to the device. The device uploads the image to the server, and the server uses an OCR engine to extract text information such as "To XX, departure time XX:XX." The TTS engine then converts this text into voice data and sends it back to the smart glasses via the device. The user can hear "To XX, departure time XX:XX" aloud through the voice output means of the smart glasses.

[0478] An example of a prompt sentence could be, "Please create a voice guide for when you see the electronic bulletin board at a station." This system will enable visually impaired people and people who have difficulty reading and writing to efficiently obtain text information in their daily lives by voice.

[0479] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0480] Step 1:

[0481] The user puts on the smart glasses and directs their gaze towards text information (e.g., an electronic bulletin board at a station or a sign in town). At this time, the user presses a dedicated button on the smart glasses. This activates the image capture means, and the camera captures an image in the direction of the gaze. The input is the user's gaze and button operation, and the output is the captured image data.

[0482] Step 2:

[0483] The device receives image data temporarily stored in the smart glasses and communicates via Wi-Fi or Bluetooth. The input is image data from the smart glasses, and the output is image data sent to the device. Specifically, the device receives the image and temporarily stores it in memory.

[0484] Step 3:

[0485] The device uploads the received image data to an external server using an HTTP request. Specifically, the device sends the image data to a REST API endpoint using the POST method. The input is the image data stored on the device, and the output is the HTTP request sent to the server. During this process, the device performs communication processing to send the data over the network.

[0486] Step 4:

[0487] The server temporarily stores the received image data in cloud storage. Using the stored image data, the server passes the image to an OCR engine (e.g., Tesseract OCR or Google Cloud Vision API) to extract text information. The input is the image data stored in cloud storage, and the output is the extracted text information. Specifically, the server analyzes the image data using an image processing library.

[0488] Step 5:

[0489] The server passes the extracted text information to a TTS engine (e.g., Amazon Polly or Google Cloud Text-to-Speech) and converts it into speech data. The input is the extracted text information, and the output is the generated speech data. The server executes the text-to-speech conversion process and stores the results in cloud storage.

[0490] Step 6:

[0491] The server returns the generated audio data to the device using an HTTP response. The input is the audio data stored in cloud storage, and the output is an HTTP response to the device. Specifically, the server performs communication processing to send the audio data to the device.

[0492] Step 7:

[0493] The device analyzes the received voice data and temporarily stores it in local storage. The device then transmits the voice data to the smart glasses using Bluetooth. The input is the voice data received from the server, and the output is the voice data transmitted to the smart glasses. Specifically, the device transfers the voice data via Bluetooth communication.

[0494] Step 8:

[0495] The user plays audio data using the smart glasses' built-in audio output means (bone conduction speaker or earphones). The input is audio data received via Bluetooth, and the output is the played audio. Specifically, the smart glasses process the received audio data and provide audio information to the user through the audio output means.

[0496] The above steps realize a system that enables visually impaired people and users who have difficulty reading and writing to efficiently obtain text information by voice.

[0497] (Application example 1)

[0498] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0499] In factories, workers need to quickly and accurately obtain information from machine control panels and signs, but visual confirmation alone takes time and carries the risk of misreading. This results in problems of reduced work efficiency. Furthermore, visually impaired workers in particular have difficulty operating equipment and checking signs, making it difficult to ensure work safety and efficiency.

[0500] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0501] In this invention, the server includes image acquisition means, means for transmitting images to an external server via communication means, character recognition means for extracting text information from the images received by the external server, voice generation means for converting the extracted text information into voice data, voice output means for receiving and playing the generated voice data via the communication means, and means for recognizing machine operation panels and signs in the factory and conveying that information to workers by voice. This allows workers to obtain necessary information by voice without having to confirm the information visually, improving work efficiency and ensuring safety.

[0502] An "image capture means" is a device for capturing visual information as digital data.

[0503] A "communication means" is a device or technology for sending and receiving digital data to other devices or servers.

[0504] An "external server" is a remote computer system for receiving, processing, storing, or transmitting digital data.

[0505] "Character recognition means" refers to software or hardware that analyzes characters and symbols in image data and extracts them as text data.

[0506] "Speech generation means" refers to technology or devices for converting text data into speech data.

[0507] The "audio output means" is a device for reproducing audio data so that it can be recognized by the human ear.

[0508] "Means of recognizing machine control panels and signs in a factory and communicating that information to workers by voice" refers to technology or equipment that uses a camera to capture important information from machine control panels and signs in a factory, converts that information into text data, and then converts it into audio data to communicate it to workers by voice.

[0509] "Cloud storage" is a remote server where data can be stored and accessed over the internet.

[0510] A "wearable device" is an electronic device that can be worn by a user and is often equipped with a camera, communication means, etc.

[0511] A "mobile device" is a portable electronic device such as a mobile phone or tablet.

[0512] A "factory robot" is an automated machine designed to perform work within a factory, and may have the role of monitoring work and conveying instructions.

[0513] The present invention provides a system for communicating text information displayed on machine control panels and signs to workers in a factory by voice. The system includes an image acquisition unit, a communication unit, an external server, a character recognition unit, a voice generation unit, and a voice output unit.

[0514] System Program Overview

[0515] 1. Image capture and transmission

[0516] The robot uses a camera to capture images of control panels and signs in the factory, and the images are temporarily stored in the robot's internal computer.

[0517] 2. Image upload and analysis

[0518] The robot's internal computer uploads image data to an external server using an HTTP request, which then stores the data in cloud storage and uses an OCR engine to extract text information from the image.

[0519] 3. Generating and returning audio data

[0520] The server passes the extracted text information to the TTS engine, which generates voice data, which is then sent back to the robot's computer using an HTTP response.

[0521] 4. Audio playback

[0522] The computer inside the robot analyzes the received audio data and plays the audio using the built-in speaker or an audio output device connected via Bluetooth.

[0523] Hardware and software used

[0524] Hardware: A high-resolution camera mounted on the robot is used for image acquisition, a high-performance computer within the robot is used for image processing and communication, and built-in speakers or a Bluetooth headset are used for audio output.

[0525] Software: Use a cloud-based OCR engine (e.g., Google Cloud Vision API) for image analysis and a cloud-based TTS engine (e.g., Amazon Polly) for voice generation.

[0526] Specific examples of processing

[0527] Example 1:

[0528] The robot uses a camera to capture an image of the factory's control panel and recognizes the text, "Machine status: Operating normally." This text information is converted into voice data, which is then played back from the robot's speaker: "Machine status is operating normally."

[0529] Example 2:

[0530] The robot uses its camera to capture factory signs, recognizes the text "Warehouse A: Material Delivery Area," and relays this information to workers via voice, allowing them to obtain the necessary information without having to visually confirm it.

[0531] Prompt Sentence Examples

[0532] "When the robot is in front of an operation panel in the factory, please generate a program that will capture the text on the operation panel using a camera and convey this text information to the worker by voice."

[0533] This configuration allows for efficient and safe work within the factory.

[0534] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0535] Step 1:

[0536] Image capture and saving

[0537] The robot uses a camera to capture images of control panels and signs in the factory, and the image data captured by the camera is temporarily stored in the robot's internal computer.

[0538] Input: Actual images of control panels and signs.

[0539] Data processing: The camera converts visual information into digital image data.

[0540] Output: Image data stored in the robot's internal computer.

[0541] Step 2:

[0542] Uploading an image

[0543] The computer inside the robot uploads the captured image data to an external server using HTTP requests, which uses a REST API.

[0544] Input: Stored image data.

[0545] Data processing: Convert image data into HTTP request format.

[0546] Output: Image data sent to an external server.

[0547] Step 3:

[0548] Extracting text information

[0549] The server stores the received image data in cloud storage and then uses an OCR engine to extract text information from the image.

[0550] Input: Image data stored on an external server.

[0551] Data processing: The OCR engine analyzes characters from image data and extracts text data.

[0552] Output: The extracted text information.

[0553] Step 4:

[0554] Generate audio data

[0555] The server passes the extracted text information to a TTS engine to generate audio data, which then converts the text data into an audio file.

[0556] Input: The extracted text information.

[0557] Data processing: The TTS engine converts the text into an audio file.

[0558] Output: The generated audio data.

[0559] Step 5:

[0560] Returning audio data

[0561] The server sends the generated audio data back to the computer in the robot using an HTTP response.

[0562] Input: The generated audio data.

[0563] Data processing: Converts audio data into HTTP response format.

[0564] Output: Audio data sent to the computer inside the robot.

[0565] Step 6:

[0566] Playing audio

[0567] The computer inside the robot analyzes the received audio data and plays the audio using the built-in speaker or an audio output device connected via Bluetooth.

[0568] Input: Audio data stored in the robot's internal computer.

[0569] Data processing: Converting audio data into a playable format.

[0570] Output: Audio played through speakers or headset.

[0571] The above are the specific processing steps of the program for the system that realizes the application example.

[0572] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0573] The present invention provides a smart glasses system that allows visually impaired people and people with reading and writing difficulties to acquire textual information by voice in their daily lives, recognizes the user's emotional state during the information acquisition process, and responds appropriately accordingly. The system includes an image acquisition unit, a communication unit, an external server, a character recognition unit, a voice generation unit, a voice output unit, and an emotion recognition unit.

[0574] Program processing overview

[0575] Image capture and transmission

[0576] User:

[0577] The user puts on the smart glasses and directs their gaze toward the text information they want to read. By pressing a button on the side of the smart glasses, the image capture means is activated.

[0578] Device:

[0579] The camera built into the smart glasses is activated to capture images, including text information, of the field of view. These images are temporarily stored in a buffer in the smart glasses in JPEG or PNG format and then sent to a connected smartphone or computer (device) via Wi-Fi or Bluetooth.

[0580] Image upload and analysis

[0581] Device:

[0582] The device uploads the received image data to an external server using an HTTP request. Specifically, the device sends the image data as a POST request via a REST API endpoint.

[0583] server:

[0584] The server temporarily stores the received image data in cloud storage.The image stored in cloud storage is then analyzed using an OCR engine to extract text information from the image.The extracted text data is temporarily stored in the server's memory.

[0585] Generate and return audio data

[0586] server:

[0587] The extracted text information is then passed to the TTS engine and converted into voice data, which is then saved in a specified directory on the server and sent back to the device as an HTTP response.

[0588] Device:

[0589] The device analyzes the received voice data, stores it in temporary storage, and then transmits the data to the smart glasses via Bluetooth.

[0590] Playing audio

[0591] Device:

[0592] The smart glasses receive the audio data and play it back using the built-in audio output means.

[0593] User:

[0594] The user can listen to the audio being played through the smart glasses.

[0595] Emotion recognition and response

[0596] Operation of emotion recognition means

[0597] Device:

[0598] The emotion recognition means is designed to analyze the user's facial expressions and voice data in real time, for example, by capturing the user's facial expressions and tone of voice through a camera and microphone installed in the smart glasses and analyzing the data.

[0599] server:

[0600] The analyzed data is sent to a server, where an emotion recognition engine determines the user's emotional state. If the emotional state is negative (sadness, anger, discomfort, etc.), the server generates a message of comfort or encouragement. On the other hand, if the emotional state is positive (joy, relief, excitement, etc.), it generates a recommended message to maintain that state.

[0601] Responding according to emotions

[0602] server:

[0603] The generated message is converted into voice data and sent from the server to the device, which then transmits the received voice data to the smart glasses.

[0604] Device:

[0605] An appropriate message is played to the user through the audio output means of the smart glasses.

[0606] Specific examples

[0607] In the case of station electronic bulletin boards

[0608] The user looks at the station's electronic display board and presses a button on the smart glasses. This action causes the smart glasses' camera to capture an image of the display board and send it to the device. The device then uploads the image to the server, which uses an OCR engine to extract "Travel to XX, departure time XX:XX." The TTS engine generates voice data and sends it back to the smart glasses via the device. The user listens to this voice. Meanwhile, the emotion recognition means analyzes the user's emotions and generates an appropriate response. For example, if a negative emotion is detected, a comforting message such as "Don't worry, the next train will arrive at XX:XX" is played.

[0609] For signs in the city

[0610] A similar procedure is applied when a user finds a sign in the city and presses a button on the smart glasses. An image of the sign is captured and sent to the server via the device. The server's OCR engine extracts the text information and generates audio data such as "Restaurant XX, business hours XX to XX." The audio data is sent back to the smart glasses, which the user listens to. During this time, if the emotion recognition means detects a positive emotion, a recommendation message such as "This restaurant has a very good reputation" is played.

[0611] In this way, a system is realized that enables visually impaired people to remain independent while efficiently obtaining textual information in daily life through speech, and in the process provides appropriate responses according to their emotions.

[0612] The processing flow will be explained below.

[0613] Step 1:

[0614] User: The user puts on the smart glasses and directs their gaze towards the text information they want to read (e.g., a magazine, a sign, a train station electronic bulletin board, etc.), then presses the button on the side of the smart glasses.

[0615] Step 2:

[0616] Device: The camera built into the smart glasses is activated and captures images containing text information within the field of view. The captured images are temporarily stored in a buffer in the smart glasses in JPEG or PNG format.

[0617] Step 3:

[0618] Terminal: Image data captured from smart glasses is sent to a connected smartphone or computer (terminal) via Wi-Fi or Bluetooth communication.

[0619] Step 4:

[0620] Terminal: The smartphone or PC uploads the received image data to an external server using an HTTP request. Specifically, it sends the image data as a POST request via a REST API endpoint.

[0621] Step 5:

[0622] Server: The server processes the received HTTP request and temporarily stores the image data in cloud storage. After storage is complete, the server prepares to call an OCR engine (e.g., optical character recognition technology).

[0623] Step 6:

[0624] Server: The OCR engine analyzes the images stored in cloud storage and extracts the text information from the images. The extracted text information is temporarily stored in the server's memory as text data.

[0625] Step 7:

[0626] Server: The server then passes the extracted text data to a Text-to-Speech (TTS) engine, which converts the text into audio data, and the generated audio data is saved in a specified directory on the server.

[0627] Step 8:

[0628] Server: The generated audio data is returned to the device as an HTTP response. The successful transmission of the audio data is recorded in the server log.

[0629] Step 9:

[0630] Device: The device analyzes the voice data received from the server and stores it in temporary storage. It then transmits the voice data to the smart glasses via Bluetooth.

[0631] Step 10:

[0632] Terminal: The smart glasses receive the audio data and play it back using the built-in audio output means (e.g., bone conduction speaker or earphones).

[0633] Step 11:

[0634] User: The user can listen to the audio being played through the smart glasses. Use the volume control buttons to adjust the volume as needed.

[0635] Step 12:

[0636] Device: The emotion recognition system captures the user's facial expressions and tone of voice through the smart glasses' camera and microphone. The captured data is then analyzed in real time by the device.

[0637] Step 13:

[0638] Terminal: The emotion recognition means determines the user's emotional state based on the analysis data, and determines whether the user is feeling negative (e.g., sadness, anger, discomfort) or positive (e.g., joy, relief, excitement).

[0639] Step 14:

[0640] Device: If a negative emotional state is detected, the device generates a comforting or encouraging message and prepares to send it to the server as audio data. On the other hand, if a positive emotional state is detected, the device generates a recommendation message for maintaining that state and prepares to send it to the server as audio data.

[0641] Step 15:

[0642] Server: Receives message data generated by the device and converts it into voice data using a TTS engine. The generated voice data is then sent back to the device.

[0643] Step 16:

[0644] Terminal: Transmits the voice data received from the server to the smart glasses.

[0645] Step 17:

[0646] Terminal: An appropriate emotional message is played to the user through the audio output means of the smart glasses.

[0647] Specific examples

[0648] In the case of station electronic bulletin boards

[0649] The user looks at the station's electronic display board and presses a button on the smart glasses. This action causes the smart glasses' camera to capture an image of the display board and send it to the device. The device then uploads the image to the server, which uses an OCR engine to extract "Travel to XX, departure time XX:XX." The TTS engine generates voice data and sends it back to the smart glasses via the device. The user listens to this voice. Meanwhile, the emotion recognition means analyzes the user's emotions and generates an appropriate response. For example, if a negative emotion is detected, a comforting message such as "Don't worry, the next train will arrive at XX:XX" is played.

[0650] For signs in the city

[0651] A similar procedure is applied when a user finds a sign in the city and presses a button on the smart glasses. An image of the sign is captured and sent to the server via the device. The server's OCR engine extracts the text information and generates audio data such as "Restaurant XX, business hours XX to XX." The audio data is sent back to the smart glasses, which the user listens to. During this time, if the emotion recognition means detects a positive emotion, a recommendation message such as "This restaurant has a very good reputation" is played.

[0652] This will enable visually impaired people to remain independent and efficiently obtain textual information in their daily lives through audio, and a system will be realized that provides appropriate responses based on emotions in the process.

[0653] Example 2

[0654] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0655] Visually impaired people and those with reading and writing difficulties face the challenge of efficiently acquiring textual information in their daily lives. Even when textual information is acquired, the user's emotional state is often not properly understood, and necessary support and encouragement are often not provided. To address this issue, a system that not only converts textual information into speech, but also recognizes the user's emotional state in real time and responds accordingly is needed.

[0656] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0657] In this invention, the server includes image acquisition means, means for transmitting the acquired image to an external server via communication means, character recognition means for extracting text information from the image received by the external server, voice generation means for converting the extracted text information into voice data, voice output means for receiving the generated voice data via the communication means and playing it back, emotion recognition means for analyzing the emotional state of the user, and means for generating an appropriate message according to the user's emotional state and converting it into voice data. This allows the user to efficiently acquire text information by voice, recognize the emotional state in real time during the process, and provide an appropriate response.

[0658] "Image capture means" refers to a device or method that electronically images information visually captured by a user using a device's camera.

[0659] "Communication means" refers to a method or device for transmitting acquired data to other devices or systems, including, for example, Wi-Fi and Bluetooth.

[0660] An "external server" is an information processing device that is connected via a network, separate from the device used by the user, and has the role of analyzing and storing data.

[0661] "Character recognition means" refers to technology or devices that detect character information contained in image data and extract it as text data. For example, this includes OCR (optical character recognition) technology.

[0662] "Speech generation means" refers to technology or equipment that converts text data into speech data, for example, using TTS (Text to Speech) technology.

[0663] "Audio output means" refers to a device or method for outputting generated audio data in a form that can be heard by the user, and includes speakers and earphones.

[0664] "Emotion recognition means" refers to technology or devices that analyze a user's facial expression or tone of voice to determine their emotional state. Examples include facial expression recognition and voice emotion analysis.

[0665] The "means for generating an appropriate message and converting it into voice data" refers to a technology or device that creates a message corresponding to the user's emotional state based on the user's emotional state obtained by the emotion recognition means, and converts that message into voice data.

[0666] This invention is a smart glasses system that allows visually impaired people and people with reading and writing difficulties to acquire text information by voice in their daily lives, recognize the user's emotional state during the information acquisition process, and respond appropriately accordingly. This system includes image acquisition means, communication means, an external server, character recognition means, voice generation means, voice output means, and emotion recognition means.

[0667] Image Acquisition Method

[0668] The user puts on the smart glasses, directs their gaze at the text information they want to read, and presses a button on the smart glasses. This action activates the camera built into the smart glasses, which captures an image containing the text information in their field of view. The image is temporarily saved in a buffer in the smart glasses in JPEG or PNG format.

[0669] communication means

[0670] The acquired image data is sent to a connected smartphone or PC (terminal) via Wi-Fi or Bluetooth. The terminal then sends the received image data to an external server as an HTTP request. Specifically, the image data is sent as a POST request via a REST API endpoint.

[0671] External Server

[0672] The server receives the POST request and temporarily stores the image data in cloud storage.The image stored in cloud storage is then analyzed using an OCR engine (for example, Google Cloud Vision API) to extract text information from the image.The extracted text data is temporarily stored in the server's memory.

[0673] Character recognition means

[0674] The character recognition means is a technology that uses an OCR engine to extract characters from image data. The extracted text data is passed to the voice generation means.

[0675] Voice generation means

[0676] The voice generation means converts the extracted text information into voice data using a TTS engine (for example, Google Text-to-Speech API). The generated voice data is saved in a specified directory on the server and then returned to the terminal as an HTTP response.

[0677] Audio output means

[0678] The device analyzes the received audio data and stores it in temporary storage. It then transmits the audio data to the smart glasses via Bluetooth. The smart glasses receive the audio data and play it back through their built-in speakers or earphones, allowing the user to listen to the audio.

[0679] emotion recognition means

[0680] The emotion recognition means uses a camera and microphone installed in the smart glasses to capture and analyze the user's facial expressions and tone of voice in real time. This determines the user's emotional state in relation to the recognized text information. Analysis is performed by an emotion recognition engine (e.g., Amazon Rekognition or Microsoft Azure Emotion API). If a negative emotion (sadness, anger, displeasure, etc.) is detected, a comforting or encouraging message is generated. If a positive emotion (joy, relief, excitement, etc.) is detected, a recommended message that maintains that state is generated.

[0681] For example, when a user looks at a train station's electronic display board and presses a button on the smart glasses, the camera in the smart glasses captures an image of the display board. The device uploads the image to a server, and the server's OCR engine extracts "To XX, departure time XX:XX." The TTS engine generates voice data and sends it back to the smart glasses via the device. The user listens to this voice. Meanwhile, the emotion recognition means analyzes the user's emotions, and if a negative emotion is detected, a comforting message such as "Don't worry, the next train will arrive at XX:XX" is played.

[0682] The same process is also performed when the user gazes at a sign in the city. Text information is extracted from the image of the sign, and audio data such as "Restaurant XX, business hours XX to XX" is generated and sent back to the smart glasses. If the emotion recognition means detects a positive emotion while the user is listening to the audio, a message such as "This restaurant has a very good reputation" is played.

[0683] This system enables visually impaired people to independently access textual information in their daily lives efficiently through speech, and provides appropriate responses according to their emotions in the process.

[0684] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0685] Step 1:

[0686] The user puts on the smart glasses, directs their gaze at the text information they want to read, and presses a button on the smart glasses. This action activates the camera built into the smart glasses, which captures an image containing the text information in their field of view. The captured image is temporarily saved in a buffer within the smart glasses in JPEG or PNG format. The input is the user's gaze and button press, and the output is the image data saved in the smart glasses buffer.

[0687] Step 2:

[0688] The device receives temporarily stored image data from the smart glasses via Wi-Fi or Bluetooth. The received image data is temporarily stored in the device's internal storage. The device then uploads this image data to an external server using an HTTP request. Specifically, the device sends the image data as a POST request via a REST API endpoint. The input is the image data from the smart glasses, and the output is an HTTP request to the external server.

[0689] Step 3:

[0690] The server processes the received POST request and temporarily stores the image data in cloud storage. Next, the image data stored in cloud storage is passed to an OCR engine (for example, Google Cloud Vision API) for analysis. The OCR engine detects character information in the image and extracts it as text data. The extracted text data is temporarily stored in the server's memory. The input is the image data in cloud storage, and the output is the extracted text data.

[0691] Step 4:

[0692] The server passes the extracted text data to a TTS engine (for example, Google Text-to-Speech API) and converts it into audio data. This audio data is saved in a specified directory on the server and then sent back to the device as an HTTP response. The input is the text data obtained by the OCR engine, and the output is the generated audio data.

[0693] Step 5:

[0694] The device analyzes the audio data in the received HTTP response and stores it in temporary storage. It then transmits the audio data to the smart glasses via Bluetooth. The input is the audio data returned from the server, and the output is the audio data sent to the smart glasses.

[0695] Step 6:

[0696] Smart glasses play back audio data received via Bluetooth. The audio is transmitted to the user through built-in speakers or earphones. The input is the audio data sent from the device, and the output is the played audio. The user can listen to this audio.

[0697] Step 7:

[0698] Using the smart glasses' built-in camera and microphone, the device captures the user's facial expressions and tone of voice in real time and passes them to the emotion recognition means. The emotion recognition means analyzes the data and determines the user's emotional state. The input is the facial expressions and tone of voice captured in real time, and the output is the analyzed emotional data.

[0699] Step 8:

[0700] The server generates a message of comfort or encouragement based on the user's emotional state obtained from the emotion recognition means if a negative emotion is detected, and generates a message of recommendation to maintain that state if a positive emotion is detected. The generated message is converted back into voice data and sent to the device. The input is the analyzed emotion data, and the output is the voice data of the generated message.

[0701] Step 9:

[0702] The device sends audio data based on the received emotional response to the smart glasses, which then plays it through the built-in speaker or earphones. This allows the user to receive an appropriate response message according to their emotional state. The input is the emotional response audio data from the server, and the output is the audio played by the smart glasses.

[0703] (Application example 2)

[0704] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0705] Visually impaired people and people with reading and writing difficulties face difficulties in efficiently obtaining product information in physical stores. Furthermore, there is a need for a system that can accurately recognize the user's emotional state during the information acquisition process and provide appropriate responses and support according to that emotion. Furthermore, it is necessary to realize a system in which a wearable device worn by the user can perform these tasks in an integrated manner.

[0706] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes image acquisition means, means for transmitting the acquired image to an external server via communication means, character recognition means for extracting text information from the image received by the external server, voice generation means for converting the extracted text information into voice data, voice output means for receiving and playing the generated voice data via the communication means, emotion recognition means for recognizing an emotional state, and means for generating an appropriate message according to the recognized emotional state and converting it into voice data. This enables visually impaired people and people who have difficulty reading and writing to not only efficiently obtain product information in physical stores, but also receive appropriate responses and support according to their emotions during the process.

[0707] "Image acquisition means" refers to a camera or imaging device for capturing visual information as digital data.

[0708] "Communication means" refers to a means for bidirectionally communicating information such as image data and audio data with external devices or servers, and utilizes wireless communication technologies such as Wi-Fi and Bluetooth.

[0709] An "external server" is a remote server that performs processes such as analyzing image data, extracting text information, and generating audio data.

[0710] "Character recognition means" refers to optical character recognition (OCR) technology for extracting text information from captured image data.

[0711] The "voice generation means" is a voice synthesis technology for converting the extracted text information into voice data.

[0712] The "audio output means" is a device that plays back audio to notify the user of the generated audio data, and includes a speaker, a bone conduction speaker, and the like.

[0713] "Emotion recognition means" is a technology for analyzing the user's emotional state, and uses a camera and microphone to analyze facial expressions and vocal tones.

[0714] The "means for generating an appropriate message" is a technology for generating an appropriate message to be provided to the user in accordance with the recognized emotional state and converting it into voice data.

[0715] A "wearable device" is a portable electronic device that can be worn by a user and is used to acquire biometric data and present information.

[0716] This invention provides a system that enables visually impaired people and people with reading and writing difficulties to efficiently obtain product information in physical stores and receive appropriate support according to their emotions during the process. The system includes smart glasses, image acquisition means, communication means, an external server, character recognition means, voice generation means, voice output means, emotion recognition means, means for generating appropriate messages, and a wearable device.

[0717] System Program

[0718] Image capture and transmission

[0719] When a user wears the smart glasses, points their eyes at a product, and presses a button, the camera built into the smart glasses captures the product and sends the image to a device. The device then uploads the received image data to an external server via Wi-Fi or Bluetooth. This process uses software tools such as OpenCV and requests.

[0720] Image upload and analysis

[0721] The device sends the received image data to an external server using an HTTP request. The server temporarily stores the image in cloud storage and extracts text information from the image using an OCR engine. The extracted text information is temporarily stored in the server's memory.

[0722] Generate and return audio data

[0723] The server passes the extracted text information to the TTS engine to generate voice data. The generated voice data is saved in a specified directory on the server and sent back to the device as an HTTP response. The device then sends the received voice data to the smart glasses and plays the voice.

[0724] Emotion recognition and appropriate responses

[0725] The emotion recognition means captures the user's facial expressions and voice using the smart glasses' camera and microphone and analyzes the data. The analyzed data is sent to the server, where the emotion recognition engine determines the user's emotional state. If the emotional state is negative, the server generates a comforting or encouraging message. If the emotional state is positive, it generates a recommendation message to maintain that state. The generated message is converted into audio data and sent to the smart glasses via the terminal, where an appropriate message is played for the user.

[0726] Hardware and software used

[0727] Hardware: Smart glasses (including built-in camera and audio output means), smartphone or laptop

[0728] Software: Python, OpenCV (image capture and processing), requests (HTTP requests), playsound (audio playback), OCR engine, TTS engine

[0729] Specific examples

[0730] Prompt Sentence Examples

[0731] Image Recognition: "Please read out the product information I'm looking at"

[0732] Emotion recognition: "The user looks sad, so generate a comforting message."

[0733] This system enables visually impaired and people with reading and writing difficulties to efficiently obtain product information in physical stores, while also receiving appropriate emotional support in the process, thereby supporting users' independence and improving the quality of their daily lives.

[0734] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0735] Step 1:

[0736] The user puts on the smart glasses, looks at the product, and presses a button. The camera built into the smart glasses captures the product and sends the image to the device. The input is the product image, and the output is the image data sent to the device.

[0737] Step 2:

[0738] The device uploads the received image data to an external server via Wi-Fi or Bluetooth. During this process, the device sends the image using an HTTP POST request. The input is the image data received from the smart glasses, and the output is the image data sent to the server.

[0739] Step 3:

[0740] The server temporarily stores the received image data in cloud storage. Then, it uses an OCR engine to extract text information from the image. The input is the image data stored in cloud storage, and the output is the extracted text information.

[0741] Step 4:

[0742] The server passes the extracted text information to the TTS engine to generate voice data. The generated voice data is saved in a specified directory on the server and sent back to the device as an HTTP response. The input is the extracted text information, and the output is the generated voice data.

[0743] Step 5:

[0744] The device analyzes the received voice data and stores it in temporary storage.Then, it transmits the voice data to the smart glasses via Bluetooth.The input is the voice data received from the server, and the output is the voice data transmitted to the smart glasses.

[0745] Step 6:

[0746] The audio output means of the smart glasses receives the audio data and plays the audio through the built-in speaker. The input is the audio data received from the terminal, and the output is the audio heard by the user.

[0747] Step 7:

[0748] The emotion recognition means captures the user's facial expressions and voice through the camera and microphone of the smart glasses and analyzes the data. The input is the user's facial expressions and voice data, and the output is the analyzed emotion data.

[0749] Step 8:

[0750] The server passes the analyzed emotional data to an emotion recognition engine to determine the user's emotional state. If the emotional state is negative, it generates a comforting or encouraging message, and if it is positive, it generates a recommendation message. The input is the analyzed emotional data, and the output is an appropriate message.

[0751] Step 9:

[0752] The server passes the generated message to the TTS engine to convert it into voice data, which is then sent to the smart glasses via the device. The input is the generated message, and the output is voice data.

[0753] Step 10:

[0754] The smart glasses receive the audio data and play it back on the built-in speaker. The input is the audio data sent from the server, and the output is the audio the user hears.

[0755] The above series of processes realizes a system that allows users to obtain product information through voice and receive appropriate responses according to their emotions during the process.

[0756] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0757] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0758] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0759] [Third embodiment]

[0760] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0761] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0762] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0763] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0764] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0765] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0766] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0767] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0768] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0769] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0770] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0771] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0772] The present invention provides a smart glasses system that enables visually impaired people and people with reading and writing difficulties to acquire text information by voice in their daily lives. The system includes an image acquisition unit, a communication unit, an external server, a character recognition unit, a voice generation unit, and a voice output unit.

[0773] Program processing overview

[0774] Image capture and transmission

[0775] User:

[0776] The user puts on the smart glasses and directs their gaze toward the text information they want to read (for example, an electronic bulletin board at a station or a sign in town). By pressing a dedicated button on the side of the smart glasses, the image capture means is activated.

[0777] Device:

[0778] The smart glasses' camera is activated and captures an image containing text information. This image is temporarily stored in the smart glasses and then sent to a connected smartphone or computer (device) via Wi-Fi or Bluetooth.

[0779] Image upload and analysis

[0780] Device:

[0781] The device uploads the received image data to an external server using an HTTP request. Specifically, the image data is sent via a REST API endpoint using a POST request.

[0782] server:

[0783] The server temporarily stores the received image data in cloud storage, then passes the stored image data to an OCR engine (e.g., a general character recognition algorithm or a cloud-based OCR service) to extract text information from the image.

[0784] Generate and return audio data

[0785] server:

[0786] The extracted text information is then passed to a TTS engine (e.g., a cloud-based speech generation service) and converted into audio data, which is temporarily stored on the server and then sent back to the device using an HTTP response.

[0787] Device:

[0788] The device analyzes the received audio data, stores it in temporary storage, and then transmits it to the smart glasses via Bluetooth.

[0789] Playing audio

[0790] User:

[0791] Smart glasses can play audio data using built-in audio output means (e.g., bone conduction speakers or earphones), allowing users to hear text information displayed in front of them aloud.

[0792] Specific examples

[0793] In the case of station electronic bulletin boards

[0794] The user presses a button on the smart glasses while looking at the station's electronic bulletin board. This action causes the smart glasses' camera to capture an image of the bulletin board and send it to the device. The device then uploads the image to the server, which uses an OCR engine to extract text information such as "To XX, departure time XX:XX." The TTS engine then converts this text into voice data and sends it back to the smart glasses via the device. Finally, the smart glasses play back this voice data, allowing the user to hear "To XX, departure time XX:XX."

[0795] For signs in the city

[0796] The same procedure applies when a user finds a sign in the city. When the user presses a button on the smart glasses, an image of the sign is captured and uploaded to the server via the device. The server's OCR engine extracts the sign's text information (e.g., "Restaurant XX, business hours XX to XX"), and the TTS engine generates voice data. Finally, the voice data is sent back to the smart glasses and played back, allowing the user to obtain this information by voice.

[0797] This will realize a system that allows visually impaired people to independently and efficiently obtain text information about their daily lives through audio.

[0798] The processing flow will be explained below.

[0799] Step 1:

[0800] User: The user puts on the smart glasses and brings the text information they want to read (e.g., magazines, signs, train station electronic bulletin boards, etc.) into their field of view. Then, they press the button on the side of the smart glasses.

[0801] Step 2:

[0802] Device: The camera built into the smart glasses is activated and captures images containing text information within the field of view. The captured images are temporarily stored in a buffer in the smart glasses in JPEG or PNG format.

[0803] Step 3:

[0804] Terminal: Image data captured from smart glasses is sent to a connected smartphone or computer (terminal) via Wi-Fi or Bluetooth communication.

[0805] Step 4:

[0806] Terminal: The smartphone or PC uploads the received image data to an external server using an HTTP request. Specifically, it sends the image data as a POST request via a REST API endpoint.

[0807] Step 5:

[0808] Server: The server processes the received HTTP request and temporarily stores the image data in cloud storage. After storage is complete, the server prepares to call an OCR engine (e.g., optical character recognition technology).

[0809] Step 6:

[0810] Server: The OCR engine analyzes the images stored in cloud storage and extracts the text information from the images. The extracted text information is temporarily stored in the server's memory as text data.

[0811] Step 7:

[0812] Server: The server then passes the extracted text data to a Text-to-Speech (TTS) engine, which converts the text into audio data, and the generated audio data is saved in a specified directory on the server.

[0813] Step 8:

[0814] Server: The generated audio data is returned to the device as an HTTP response. The successful transmission of the audio data is recorded in the server log.

[0815] Step 9:

[0816] Device: The device analyzes the voice data received from the server and stores it in temporary storage. It then transmits the voice data to the smart glasses via Bluetooth.

[0817] Step 10:

[0818] Terminal: The smart glasses receive the audio data and play it back using the built-in audio output means (e.g., bone conduction speaker or earphones).

[0819] Step 11:

[0820] User: The user can listen to the audio being played through the smart glasses. Use the volume control buttons to adjust the volume as needed.

[0821] The above steps realize a system that enables visually impaired people to quickly and accurately obtain text information in daily life through speech.

[0822] Example 1

[0823] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0824] It is important in daily life for visually impaired people and those who have difficulty reading and writing to quickly and accurately acquire text information. However, conventional methods for acquiring text information by voice have problems such as cumbersome and time-consuming operations, difficulty in synchronizing images and voice, and lack of portability. The purpose of this invention is to solve these problems and provide a system that allows users to easily and quickly acquire text information by voice.

[0825] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0826] In this invention, the server includes image acquisition means for acquiring an image in the direction in which the user is looking, means for transmitting the acquired image to an external server via communication means, character recognition means for extracting text information from the image received by the external server, voice generation means for converting the extracted text information into voice data, means for receiving the generated voice data via the communication means and transmitting it to voice output means via Bluetooth, and voice output means for playing back the generated voice data. This enables visually impaired people and people who have difficulty reading and writing to efficiently acquire text information necessary in daily life by voice.

[0827] The "image acquisition means" is a device or component that has the function of capturing an image in the direction in which the user is looking.

[0828] "Communication means" refers to the technical means for sending and receiving image data and audio data between external servers and terminals, including Wi-Fi and Bluetooth.

[0829] An "external server" is a computer system that communicates with a terminal via a network and has the functions of receiving, analyzing, and saving image data, and generating and returning audio data.

[0830] "Character recognition means" means a technology or device that has the function of extracting text information from an image, and includes character recognition algorithms and services such as OCR.

[0831] "Speech generation means" refers to the technology and devices that convert extracted text information into speech data, and includes text-to-speech engines (TTS engines).

[0832] The "audio output means" is a device that has the function of reproducing the generated audio data, and includes bone conduction speakers, earphones, etc.

[0833] "Cloud storage" is a storage service for storing and managing data on a network, and functions as part of data storage within a server.

[0834] A "wearable device" is a portable electronic device that is worn by a user and includes a camera and audio output means.

[0835] This invention relates to a smart glasses system that enables visually impaired people and people with reading and writing difficulties to obtain text information by voice in their daily lives. The system includes an image acquisition unit, a communication unit, an external server, a character recognition unit, a voice generation unit, and a voice output unit.

[0836] The user wears the smart glasses and directs their gaze toward text information (e.g., a train station bulletin board or sign). The user presses a dedicated button on the side of the smart glasses to activate the image capture means.

[0837] At the device, the camera built into the smart glasses captures an image in the direction of gaze, which is temporarily stored in the smart glasses and then sent to a connected smartphone or computer (device) via Wi-Fi or Bluetooth.

[0838] The device uploads the received image data to an external server using an HTTP request. Specifically, the image data is sent via a REST API endpoint using the POST method.

[0839] The server temporarily stores the received image data in cloud storage, then processes the image using the Python library OpenCV and extracts text information from the image using Tesseract OCR or a cloud-based OCR engine.

[0840] The extracted text information is then passed to a TTS (Text-to-Speech) engine, for example, using a cloud-based speech generation service (e.g., Amazon Polly or Google Cloud Text-to-Speech), which converts the text into audio data that is again stored in cloud storage and then sent back to the device using an HTTP response.

[0841] The device analyzes the received voice data and temporarily stores it in local storage, then transmits the data to the smart glasses via Bluetooth.

[0842] Finally, users can play audio data using the smart glasses' built-in audio output means (e.g., bone conduction speakers or earphones), allowing them to obtain the text information they are viewing by audio.

[0843] Specific examples

[0844] For example, a user looks at a train station's electronic bulletin board and presses a button on the smart glasses. This operation causes the camera on the smart glasses to capture an image of the bulletin board and send it to the device. The device uploads the image to the server, and the server uses an OCR engine to extract text information such as "To XX, departure time XX:XX." The TTS engine then converts this text into voice data and sends it back to the smart glasses via the device. The user can hear "To XX, departure time XX:XX" aloud through the voice output means of the smart glasses.

[0845] An example of a prompt sentence could be, "Please create a voice guide for when you see the electronic bulletin board at a station." This system will enable visually impaired people and people who have difficulty reading and writing to efficiently obtain text information in their daily lives by voice.

[0846] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0847] Step 1:

[0848] The user puts on the smart glasses and directs their gaze towards text information (e.g., an electronic bulletin board at a station or a sign in town). At this time, the user presses a dedicated button on the smart glasses. This activates the image capture means, and the camera captures an image in the direction of the gaze. The input is the user's gaze and button operation, and the output is the captured image data.

[0849] Step 2:

[0850] The device receives image data temporarily stored in the smart glasses and communicates via Wi-Fi or Bluetooth. The input is image data from the smart glasses, and the output is image data sent to the device. Specifically, the device receives the image and temporarily stores it in memory.

[0851] Step 3:

[0852] The device uploads the received image data to an external server using an HTTP request. Specifically, the device sends the image data to a REST API endpoint using the POST method. The input is the image data stored on the device, and the output is the HTTP request sent to the server. During this process, the device performs communication processing to send the data over the network.

[0853] Step 4:

[0854] The server temporarily stores the received image data in cloud storage. Using the stored image data, the server passes the image to an OCR engine (e.g., Tesseract OCR or Google Cloud Vision API) to extract text information. The input is the image data stored in cloud storage, and the output is the extracted text information. Specifically, the server analyzes the image data using an image processing library.

[0855] Step 5:

[0856] The server passes the extracted text information to a TTS engine (e.g., Amazon Polly or Google Cloud Text-to-Speech) and converts it into speech data. The input is the extracted text information, and the output is the generated speech data. The server executes the text-to-speech conversion process and stores the results in cloud storage.

[0857] Step 6:

[0858] The server returns the generated audio data to the device using an HTTP response. The input is the audio data stored in cloud storage, and the output is an HTTP response to the device. Specifically, the server performs communication processing to send the audio data to the device.

[0859] Step 7:

[0860] The device analyzes the received voice data and temporarily stores it in local storage. The device then transmits the voice data to the smart glasses using Bluetooth. The input is the voice data received from the server, and the output is the voice data transmitted to the smart glasses. Specifically, the device transfers the voice data via Bluetooth communication.

[0861] Step 8:

[0862] The user plays audio data using the smart glasses' built-in audio output means (bone conduction speaker or earphones). The input is audio data received via Bluetooth, and the output is the played audio. Specifically, the smart glasses process the received audio data and provide audio information to the user through the audio output means.

[0863] The above steps realize a system that enables visually impaired people and users who have difficulty reading and writing to efficiently obtain text information by voice.

[0864] (Application example 1)

[0865] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0866] In factories, workers need to quickly and accurately obtain information from machine control panels and signs, but visual confirmation alone takes time and carries the risk of misreading. This results in problems of reduced work efficiency. Furthermore, visually impaired workers in particular have difficulty operating equipment and checking signs, making it difficult to ensure work safety and efficiency.

[0867] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0868] In this invention, the server includes image acquisition means, means for transmitting images to an external server via communication means, character recognition means for extracting text information from the images received by the external server, voice generation means for converting the extracted text information into voice data, voice output means for receiving and playing the generated voice data via the communication means, and means for recognizing machine operation panels and signs in the factory and conveying that information to workers by voice. This allows workers to obtain necessary information by voice without having to confirm the information visually, improving work efficiency and ensuring safety.

[0869] An "image capture means" is a device for capturing visual information as digital data.

[0870] A "communication means" is a device or technology for sending and receiving digital data to other devices or servers.

[0871] An "external server" is a remote computer system for receiving, processing, storing, or transmitting digital data.

[0872] "Character recognition means" refers to software or hardware that analyzes characters and symbols in image data and extracts them as text data.

[0873] "Speech generation means" refers to technology or devices for converting text data into speech data.

[0874] The "audio output means" is a device for reproducing audio data so that it can be recognized by the human ear.

[0875] "Means of recognizing machine control panels and signs in a factory and communicating that information to workers by voice" refers to technology or equipment that uses a camera to capture important information from machine control panels and signs in a factory, converts that information into text data, and then converts it into audio data to communicate it to workers by voice.

[0876] "Cloud storage" is a remote server where data can be stored and accessed over the internet.

[0877] A "wearable device" is an electronic device that can be worn by a user and is often equipped with a camera, communication means, etc.

[0878] A "mobile device" is a portable electronic device such as a mobile phone or tablet.

[0879] A "factory robot" is an automated machine designed to perform work within a factory, and may have the role of monitoring work and conveying instructions.

[0880] The present invention provides a system for communicating text information displayed on machine control panels and signs to workers in a factory by voice. The system includes an image acquisition unit, a communication unit, an external server, a character recognition unit, a voice generation unit, and a voice output unit.

[0881] System Program Overview

[0882] 1. Image capture and transmission

[0883] The robot uses a camera to capture images of control panels and signs in the factory, and the images are temporarily stored in the robot's internal computer.

[0884] 2. Image upload and analysis

[0885] The robot's internal computer uploads image data to an external server using an HTTP request, which then stores the data in cloud storage and uses an OCR engine to extract text information from the image.

[0886] 3. Generating and returning audio data

[0887] The server passes the extracted text information to the TTS engine, which generates voice data, which is then sent back to the robot's computer using an HTTP response.

[0888] 4. Audio playback

[0889] The computer inside the robot analyzes the received audio data and plays the audio using the built-in speaker or an audio output device connected via Bluetooth.

[0890] Hardware and software used

[0891] Hardware: A high-resolution camera mounted on the robot is used for image acquisition, a high-performance computer within the robot is used for image processing and communication, and built-in speakers or a Bluetooth headset are used for audio output.

[0892] Software: Use a cloud-based OCR engine (e.g., Google Cloud Vision API) for image analysis and a cloud-based TTS engine (e.g., Amazon Polly) for voice generation.

[0893] Specific examples of processing

[0894] Example 1:

[0895] The robot uses a camera to capture an image of the factory's control panel and recognizes the text, "Machine status: Operating normally." This text information is converted into voice data, which is then played back from the robot's speaker: "Machine status is operating normally."

[0896] Example 2:

[0897] The robot uses its camera to capture factory signs, recognizes the text "Warehouse A: Material Delivery Area," and relays this information to workers via voice, allowing them to obtain the necessary information without having to visually confirm it.

[0898] Prompt Sentence Examples

[0899] "When the robot is in front of an operation panel in the factory, please generate a program that will capture the text on the operation panel using a camera and convey this text information to the worker by voice."

[0900] This configuration allows for efficient and safe work within the factory.

[0901] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0902] Step 1:

[0903] Image capture and saving

[0904] The robot uses a camera to capture images of control panels and signs in the factory, and the image data captured by the camera is temporarily stored in the robot's internal computer.

[0905] Input: Actual images of control panels and signs.

[0906] Data processing: The camera converts visual information into digital image data.

[0907] Output: Image data stored in the robot's internal computer.

[0908] Step 2:

[0909] Uploading an image

[0910] The computer inside the robot uploads the captured image data to an external server using HTTP requests, which uses a REST API.

[0911] Input: Stored image data.

[0912] Data processing: Convert image data into HTTP request format.

[0913] Output: Image data sent to an external server.

[0914] Step 3:

[0915] Extracting text information

[0916] The server stores the received image data in cloud storage and then uses an OCR engine to extract text information from the image.

[0917] Input: Image data stored on an external server.

[0918] Data processing: The OCR engine analyzes characters from image data and extracts text data.

[0919] Output: The extracted text information.

[0920] Step 4:

[0921] Generate audio data

[0922] The server passes the extracted text information to a TTS engine to generate audio data, which then converts the text data into an audio file.

[0923] Input: The extracted text information.

[0924] Data processing: The TTS engine converts the text into an audio file.

[0925] Output: The generated audio data.

[0926] Step 5:

[0927] Returning audio data

[0928] The server sends the generated audio data back to the computer in the robot using an HTTP response.

[0929] Input: The generated audio data.

[0930] Data processing: Converts audio data into HTTP response format.

[0931] Output: Audio data sent to the computer inside the robot.

[0932] Step 6:

[0933] Playing audio

[0934] The computer inside the robot analyzes the received audio data and plays the audio using the built-in speaker or an audio output device connected via Bluetooth.

[0935] Input: Audio data stored in the robot's internal computer.

[0936] Data processing: Converting audio data into a playable format.

[0937] Output: Audio played through speakers or headset.

[0938] The above are the specific processing steps of the program for the system that realizes the application example.

[0939] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0940] The present invention provides a smart glasses system that allows visually impaired people and people with reading and writing difficulties to acquire textual information by voice in their daily lives, recognizes the user's emotional state during the information acquisition process, and responds appropriately accordingly. The system includes an image acquisition unit, a communication unit, an external server, a character recognition unit, a voice generation unit, a voice output unit, and an emotion recognition unit.

[0941] Program processing overview

[0942] Image capture and transmission

[0943] User:

[0944] The user puts on the smart glasses and directs their gaze toward the text information they want to read. By pressing a button on the side of the smart glasses, the image capture means is activated.

[0945] Device:

[0946] The camera built into the smart glasses is activated to capture images, including text information, of the field of view. These images are temporarily stored in a buffer in the smart glasses in JPEG or PNG format and then sent to a connected smartphone or computer (device) via Wi-Fi or Bluetooth.

[0947] Image upload and analysis

[0948] Device:

[0949] The device uploads the received image data to an external server using an HTTP request. Specifically, the device sends the image data as a POST request via a REST API endpoint.

[0950] server:

[0951] The server temporarily stores the received image data in cloud storage.The image stored in cloud storage is then analyzed using an OCR engine to extract text information from the image.The extracted text data is temporarily stored in the server's memory.

[0952] Generate and return audio data

[0953] server:

[0954] The extracted text information is then passed to the TTS engine and converted into voice data, which is then saved in a specified directory on the server and sent back to the device as an HTTP response.

[0955] Device:

[0956] The device analyzes the received voice data, stores it in temporary storage, and then transmits the data to the smart glasses via Bluetooth.

[0957] Playing audio

[0958] Device:

[0959] The smart glasses receive the audio data and play it back using the built-in audio output means.

[0960] User:

[0961] The user can listen to the audio being played through the smart glasses.

[0962] Emotion recognition and response

[0963] Operation of emotion recognition means

[0964] Device:

[0965] The emotion recognition means is designed to analyze the user's facial expressions and voice data in real time, for example, by capturing the user's facial expressions and tone of voice through a camera and microphone installed in the smart glasses and analyzing the data.

[0966] server:

[0967] The analyzed data is sent to a server, where an emotion recognition engine determines the user's emotional state. If the emotional state is negative (sadness, anger, discomfort, etc.), the server generates a message of comfort or encouragement. On the other hand, if the emotional state is positive (joy, relief, excitement, etc.), it generates a recommended message to maintain that state.

[0968] Responding according to emotions

[0969] server:

[0970] The generated message is converted into voice data and sent from the server to the device, which then transmits the received voice data to the smart glasses.

[0971] Device:

[0972] An appropriate message is played to the user through the audio output means of the smart glasses.

[0973] Specific examples

[0974] In the case of station electronic bulletin boards

[0975] The user looks at the station's electronic display board and presses a button on the smart glasses. This action causes the smart glasses' camera to capture an image of the display board and send it to the device. The device then uploads the image to the server, which uses an OCR engine to extract "Travel to XX, departure time XX:XX." The TTS engine generates voice data and sends it back to the smart glasses via the device. The user listens to this voice. Meanwhile, the emotion recognition means analyzes the user's emotions and generates an appropriate response. For example, if a negative emotion is detected, a comforting message such as "Don't worry, the next train will arrive at XX:XX" is played.

[0976] For signs in the city

[0977] A similar procedure is applied when a user finds a sign in the city and presses a button on the smart glasses. An image of the sign is captured and sent to the server via the device. The server's OCR engine extracts the text information and generates audio data such as "Restaurant XX, business hours XX to XX." The audio data is sent back to the smart glasses, which the user listens to. During this time, if the emotion recognition means detects a positive emotion, a recommendation message such as "This restaurant has a very good reputation" is played.

[0978] In this way, a system is realized that enables visually impaired people to remain independent while efficiently obtaining textual information in daily life through speech, and in the process provides appropriate responses according to their emotions.

[0979] The processing flow will be explained below.

[0980] Step 1:

[0981] User: The user puts on the smart glasses and directs their gaze towards the text information they want to read (e.g., a magazine, a sign, a train station electronic bulletin board, etc.), then presses the button on the side of the smart glasses.

[0982] Step 2:

[0983] Device: The camera built into the smart glasses is activated and captures images containing text information within the field of view. The captured images are temporarily stored in a buffer in the smart glasses in JPEG or PNG format.

[0984] Step 3:

[0985] Terminal: Image data captured from smart glasses is sent to a connected smartphone or computer (terminal) via Wi-Fi or Bluetooth communication.

[0986] Step 4:

[0987] Terminal: The smartphone or PC uploads the received image data to an external server using an HTTP request. Specifically, it sends the image data as a POST request via a REST API endpoint.

[0988] Step 5:

[0989] Server: The server processes the received HTTP request and temporarily stores the image data in cloud storage. After storage is complete, the server prepares to call an OCR engine (e.g., optical character recognition technology).

[0990] Step 6:

[0991] Server: The OCR engine analyzes the images stored in cloud storage and extracts the text information from the images. The extracted text information is temporarily stored in the server's memory as text data.

[0992] Step 7:

[0993] Server: The server then passes the extracted text data to a Text-to-Speech (TTS) engine, which converts the text into audio data, and the generated audio data is saved in a specified directory on the server.

[0994] Step 8:

[0995] Server: The generated audio data is returned to the device as an HTTP response. The successful transmission of the audio data is recorded in the server log.

[0996] Step 9:

[0997] Device: The device analyzes the voice data received from the server and stores it in temporary storage. It then transmits the voice data to the smart glasses via Bluetooth.

[0998] Step 10:

[0999] Terminal: The smart glasses receive the audio data and play it back using the built-in audio output means (e.g., bone conduction speaker or earphones).

[1000] Step 11:

[1001] User: The user can listen to the audio being played through the smart glasses. Use the volume control buttons to adjust the volume as needed.

[1002] Step 12:

[1003] Device: The emotion recognition system captures the user's facial expressions and tone of voice through the smart glasses' camera and microphone. The captured data is then analyzed in real time by the device.

[1004] Step 13:

[1005] Terminal: The emotion recognition means determines the user's emotional state based on the analysis data, and determines whether the user is feeling negative (e.g., sadness, anger, discomfort) or positive (e.g., joy, relief, excitement).

[1006] Step 14:

[1007] Device: If a negative emotional state is detected, the device generates a comforting or encouraging message and prepares to send it to the server as audio data. On the other hand, if a positive emotional state is detected, the device generates a recommendation message for maintaining that state and prepares to send it to the server as audio data.

[1008] Step 15:

[1009] Server: Receives message data generated by the device and converts it into voice data using a TTS engine. The generated voice data is then sent back to the device.

[1010] Step 16:

[1011] Terminal: Transmits the voice data received from the server to the smart glasses.

[1012] Step 17:

[1013] Terminal: An appropriate emotional message is played to the user through the audio output means of the smart glasses.

[1014] Specific examples

[1015] In the case of station electronic bulletin boards

[1016] The user looks at the station's electronic display board and presses a button on the smart glasses. This action causes the smart glasses' camera to capture an image of the display board and send it to the device. The device then uploads the image to the server, which uses an OCR engine to extract "Travel to XX, departure time XX:XX." The TTS engine generates voice data and sends it back to the smart glasses via the device. The user listens to this voice. Meanwhile, the emotion recognition means analyzes the user's emotions and generates an appropriate response. For example, if a negative emotion is detected, a comforting message such as "Don't worry, the next train will arrive at XX:XX" is played.

[1017] For signs in the city

[1018] A similar procedure is applied when a user finds a sign in the city and presses a button on the smart glasses. An image of the sign is captured and sent to the server via the device. The server's OCR engine extracts the text information and generates audio data such as "Restaurant XX, business hours XX to XX." The audio data is sent back to the smart glasses, which the user listens to. During this time, if the emotion recognition means detects a positive emotion, a recommendation message such as "This restaurant has a very good reputation" is played.

[1019] This will enable visually impaired people to remain independent and efficiently obtain textual information in their daily lives through audio, and a system will be realized that provides appropriate responses based on emotions in the process.

[1020] Example 2

[1021] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1022] Visually impaired people and those with reading and writing difficulties face the challenge of efficiently acquiring textual information in their daily lives. Even when textual information is acquired, the user's emotional state is often not properly understood, and necessary support and encouragement are often not provided. To address this issue, a system that not only converts textual information into speech, but also recognizes the user's emotional state in real time and responds accordingly is needed.

[1023] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1024] In this invention, the server includes image acquisition means, means for transmitting the acquired image to an external server via communication means, character recognition means for extracting text information from the image received by the external server, voice generation means for converting the extracted text information into voice data, voice output means for receiving the generated voice data via the communication means and playing it back, emotion recognition means for analyzing the emotional state of the user, and means for generating an appropriate message according to the user's emotional state and converting it into voice data. This allows the user to efficiently acquire text information by voice, recognize the emotional state in real time during the process, and provide an appropriate response.

[1025] "Image capture means" refers to a device or method that electronically images information visually captured by a user using a device's camera.

[1026] "Communication means" refers to a method or device for transmitting acquired data to other devices or systems, including, for example, Wi-Fi and Bluetooth.

[1027] An "external server" is an information processing device that is connected via a network, separate from the device used by the user, and has the role of analyzing and storing data.

[1028] "Character recognition means" refers to technology or devices that detect character information contained in image data and extract it as text data. For example, this includes OCR (optical character recognition) technology.

[1029] "Speech generation means" refers to technology or equipment that converts text data into speech data, for example, using TTS (Text to Speech) technology.

[1030] "Audio output means" refers to a device or method for outputting generated audio data in a form that can be heard by the user, and includes speakers and earphones.

[1031] "Emotion recognition means" refers to technology or devices that analyze a user's facial expression or tone of voice to determine their emotional state. Examples include facial expression recognition and voice emotion analysis.

[1032] The "means for generating an appropriate message and converting it into voice data" refers to a technology or device that creates a message corresponding to the user's emotional state based on the user's emotional state obtained by the emotion recognition means, and converts that message into voice data.

[1033] This invention is a smart glasses system that allows visually impaired people and people with reading and writing difficulties to acquire text information by voice in their daily lives, recognize the user's emotional state during the information acquisition process, and respond appropriately accordingly. This system includes image acquisition means, communication means, an external server, character recognition means, voice generation means, voice output means, and emotion recognition means.

[1034] Image Acquisition Method

[1035] The user puts on the smart glasses, directs their gaze at the text information they want to read, and presses a button on the smart glasses. This action activates the camera built into the smart glasses, which captures an image containing the text information in their field of view. The image is temporarily saved in a buffer in the smart glasses in JPEG or PNG format.

[1036] communication means

[1037] The acquired image data is sent to a connected smartphone or PC (terminal) via Wi-Fi or Bluetooth. The terminal then sends the received image data to an external server as an HTTP request. Specifically, the image data is sent as a POST request via a REST API endpoint.

[1038] External Server

[1039] The server receives the POST request and temporarily stores the image data in cloud storage.The image stored in cloud storage is then analyzed using an OCR engine (for example, Google Cloud Vision API) to extract text information from the image.The extracted text data is temporarily stored in the server's memory.

[1040] Character recognition means

[1041] The character recognition means is a technology that uses an OCR engine to extract characters from image data. The extracted text data is passed to the voice generation means.

[1042] Voice generation means

[1043] The voice generation means converts the extracted text information into voice data using a TTS engine (for example, Google Text-to-Speech API). The generated voice data is saved in a specified directory on the server and then returned to the terminal as an HTTP response.

[1044] Audio output means

[1045] The device analyzes the received audio data and stores it in temporary storage. It then transmits the audio data to the smart glasses via Bluetooth. The smart glasses receive the audio data and play it back through their built-in speakers or earphones, allowing the user to listen to the audio.

[1046] emotion recognition means

[1047] The emotion recognition means uses a camera and microphone installed in the smart glasses to capture and analyze the user's facial expressions and tone of voice in real time. This determines the user's emotional state in relation to the recognized text information. Analysis is performed by an emotion recognition engine (e.g., Amazon Rekognition or Microsoft Azure Emotion API). If a negative emotion (sadness, anger, displeasure, etc.) is detected, a comforting or encouraging message is generated. If a positive emotion (joy, relief, excitement, etc.) is detected, a recommended message that maintains that state is generated.

[1048] For example, when a user looks at a train station's electronic display board and presses a button on the smart glasses, the camera in the smart glasses captures an image of the display board. The device uploads the image to a server, and the server's OCR engine extracts "To XX, departure time XX:XX." The TTS engine generates voice data and sends it back to the smart glasses via the device. The user listens to this voice. Meanwhile, the emotion recognition means analyzes the user's emotions, and if a negative emotion is detected, a comforting message such as "Don't worry, the next train will arrive at XX:XX" is played.

[1049] The same process is also performed when the user gazes at a sign in the city. Text information is extracted from the image of the sign, and audio data such as "Restaurant XX, business hours XX to XX" is generated and sent back to the smart glasses. If the emotion recognition means detects a positive emotion while the user is listening to the audio, a message such as "This restaurant has a very good reputation" is played.

[1050] This system enables visually impaired people to independently access textual information in their daily lives efficiently through speech, and provides appropriate responses according to their emotions in the process.

[1051] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1052] Step 1:

[1053] The user puts on the smart glasses, directs their gaze at the text information they want to read, and presses a button on the smart glasses. This action activates the camera built into the smart glasses, which captures an image containing the text information in their field of view. The captured image is temporarily saved in a buffer within the smart glasses in JPEG or PNG format. The input is the user's gaze and button press, and the output is the image data saved in the smart glasses buffer.

[1054] Step 2:

[1055] The device receives temporarily stored image data from the smart glasses via Wi-Fi or Bluetooth. The received image data is temporarily stored in the device's internal storage. The device then uploads this image data to an external server using an HTTP request. Specifically, the device sends the image data as a POST request via a REST API endpoint. The input is the image data from the smart glasses, and the output is an HTTP request to the external server.

[1056] Step 3:

[1057] The server processes the received POST request and temporarily stores the image data in cloud storage. Next, the image data stored in cloud storage is passed to an OCR engine (for example, Google Cloud Vision API) for analysis. The OCR engine detects character information in the image and extracts it as text data. The extracted text data is temporarily stored in the server's memory. The input is the image data in cloud storage, and the output is the extracted text data.

[1058] Step 4:

[1059] The server passes the extracted text data to a TTS engine (for example, Google Text-to-Speech API) and converts it into audio data. This audio data is saved in a specified directory on the server and then sent back to the device as an HTTP response. The input is the text data obtained by the OCR engine, and the output is the generated audio data.

[1060] Step 5:

[1061] The device analyzes the audio data in the received HTTP response and stores it in temporary storage. It then transmits the audio data to the smart glasses via Bluetooth. The input is the audio data returned from the server, and the output is the audio data sent to the smart glasses.

[1062] Step 6:

[1063] Smart glasses play back audio data received via Bluetooth. The audio is transmitted to the user through built-in speakers or earphones. The input is the audio data sent from the device, and the output is the played audio. The user can listen to this audio.

[1064] Step 7:

[1065] Using the smart glasses' built-in camera and microphone, the device captures the user's facial expressions and tone of voice in real time and passes them to the emotion recognition means. The emotion recognition means analyzes the data and determines the user's emotional state. The input is the facial expressions and tone of voice captured in real time, and the output is the analyzed emotional data.

[1066] Step 8:

[1067] The server generates a message of comfort or encouragement based on the user's emotional state obtained from the emotion recognition means if a negative emotion is detected, and generates a message of recommendation to maintain that state if a positive emotion is detected. The generated message is converted back into voice data and sent to the device. The input is the analyzed emotion data, and the output is the voice data of the generated message.

[1068] Step 9:

[1069] The device sends audio data based on the received emotional response to the smart glasses, which then plays it through the built-in speaker or earphones. This allows the user to receive an appropriate response message according to their emotional state. The input is the emotional response audio data from the server, and the output is the audio played by the smart glasses.

[1070] (Application example 2)

[1071] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1072] Visually impaired people and people with reading and writing difficulties face difficulties in efficiently obtaining product information in physical stores. Furthermore, there is a need for a system that can accurately recognize the user's emotional state during the information acquisition process and provide appropriate responses and support according to that emotion. Furthermore, it is necessary to realize a system in which a wearable device worn by the user can perform these tasks in an integrated manner.

[1073] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes image acquisition means, means for transmitting the acquired image to an external server via communication means, character recognition means for extracting text information from the image received by the external server, voice generation means for converting the extracted text information into voice data, voice output means for receiving and playing the generated voice data via the communication means, emotion recognition means for recognizing an emotional state, and means for generating an appropriate message according to the recognized emotional state and converting it into voice data. This enables visually impaired people and people who have difficulty reading and writing to not only efficiently obtain product information in physical stores, but also receive appropriate responses and support according to their emotions during the process.

[1074] "Image acquisition means" refers to a camera or imaging device for capturing visual information as digital data.

[1075] "Communication means" refers to a means for bidirectionally communicating information such as image data and audio data with external devices or servers, and utilizes wireless communication technologies such as Wi-Fi and Bluetooth.

[1076] An "external server" is a remote server that performs processes such as analyzing image data, extracting text information, and generating audio data.

[1077] "Character recognition means" refers to optical character recognition (OCR) technology for extracting text information from captured image data.

[1078] The "voice generation means" is a voice synthesis technology for converting the extracted text information into voice data.

[1079] The "audio output means" is a device that plays back audio to notify the user of the generated audio data, and includes a speaker, a bone conduction speaker, and the like.

[1080] "Emotion recognition means" is a technology for analyzing the user's emotional state, and uses a camera and microphone to analyze facial expressions and vocal tones.

[1081] The "means for generating an appropriate message" is a technology for generating an appropriate message to be provided to the user in accordance with the recognized emotional state and converting it into voice data.

[1082] A "wearable device" is a portable electronic device that can be worn by a user and is used to acquire biometric data and present information.

[1083] This invention provides a system that enables visually impaired people and people with reading and writing difficulties to efficiently obtain product information in physical stores and receive appropriate support according to their emotions during the process. The system includes smart glasses, image acquisition means, communication means, an external server, character recognition means, voice generation means, voice output means, emotion recognition means, means for generating appropriate messages, and a wearable device.

[1084] System Program

[1085] Image capture and transmission

[1086] When a user wears the smart glasses, points their eyes at a product, and presses a button, the camera built into the smart glasses captures the product and sends the image to a device. The device then uploads the received image data to an external server via Wi-Fi or Bluetooth. This process uses software tools such as OpenCV and requests.

[1087] Image upload and analysis

[1088] The device sends the received image data to an external server using an HTTP request. The server temporarily stores the image in cloud storage and extracts text information from the image using an OCR engine. The extracted text information is temporarily stored in the server's memory.

[1089] Generate and return audio data

[1090] The server passes the extracted text information to the TTS engine to generate voice data. The generated voice data is saved in a specified directory on the server and sent back to the device as an HTTP response. The device then sends the received voice data to the smart glasses and plays the voice.

[1091] Emotion recognition and appropriate responses

[1092] The emotion recognition means captures the user's facial expressions and voice using the smart glasses' camera and microphone and analyzes the data. The analyzed data is sent to the server, where the emotion recognition engine determines the user's emotional state. If the emotional state is negative, the server generates a comforting or encouraging message. If the emotional state is positive, it generates a recommendation message to maintain that state. The generated message is converted into audio data and sent to the smart glasses via the terminal, where an appropriate message is played for the user.

[1093] Hardware and software used

[1094] Hardware: Smart glasses (including built-in camera and audio output means), smartphone or laptop

[1095] Software: Python, OpenCV (image capture and processing), requests (HTTP requests), playsound (audio playback), OCR engine, TTS engine

[1096] Specific examples

[1097] Prompt Sentence Examples

[1098] Image Recognition: "Please read out the product information I'm looking at"

[1099] Emotion recognition: "The user looks sad, so generate a comforting message."

[1100] This system enables visually impaired and people with reading and writing difficulties to efficiently obtain product information in physical stores, while also receiving appropriate emotional support in the process, thereby supporting users' independence and improving the quality of their daily lives.

[1101] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1102] Step 1:

[1103] The user puts on the smart glasses, looks at the product, and presses a button. The camera built into the smart glasses captures the product and sends the image to the device. The input is the product image, and the output is the image data sent to the device.

[1104] Step 2:

[1105] The device uploads the received image data to an external server via Wi-Fi or Bluetooth. During this process, the device sends the image using an HTTP POST request. The input is the image data received from the smart glasses, and the output is the image data sent to the server.

[1106] Step 3:

[1107] The server temporarily stores the received image data in cloud storage. Then, it uses an OCR engine to extract text information from the image. The input is the image data stored in cloud storage, and the output is the extracted text information.

[1108] Step 4:

[1109] The server passes the extracted text information to the TTS engine to generate voice data. The generated voice data is saved in a specified directory on the server and sent back to the device as an HTTP response. The input is the extracted text information, and the output is the generated voice data.

[1110] Step 5:

[1111] The device analyzes the received voice data and stores it in temporary storage.Then, it transmits the voice data to the smart glasses via Bluetooth.The input is the voice data received from the server, and the output is the voice data transmitted to the smart glasses.

[1112] Step 6:

[1113] The audio output means of the smart glasses receives the audio data and plays the audio through the built-in speaker. The input is the audio data received from the terminal, and the output is the audio heard by the user.

[1114] Step 7:

[1115] The emotion recognition means captures the user's facial expressions and voice through the camera and microphone of the smart glasses and analyzes the data. The input is the user's facial expressions and voice data, and the output is the analyzed emotion data.

[1116] Step 8:

[1117] The server passes the analyzed emotional data to an emotion recognition engine to determine the user's emotional state. If the emotional state is negative, it generates a comforting or encouraging message, and if it is positive, it generates a recommendation message. The input is the analyzed emotional data, and the output is an appropriate message.

[1118] Step 9:

[1119] The server passes the generated message to the TTS engine to convert it into voice data, which is then sent to the smart glasses via the device. The input is the generated message, and the output is voice data.

[1120] Step 10:

[1121] The smart glasses receive the audio data and play it back on the built-in speaker. The input is the audio data sent from the server, and the output is the audio the user hears.

[1122] The above series of processes realizes a system that allows users to obtain product information through voice and receive appropriate responses according to their emotions during the process.

[1123] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1124] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1125] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1126] [Fourth embodiment]

[1127] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1128] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1129] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1130] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1131] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1132] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1133] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1134] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1135] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1136] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1137] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1138] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1139] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1140] The present invention provides a smart glasses system that enables visually impaired people and people with reading and writing difficulties to acquire text information by voice in their daily lives. The system includes an image acquisition unit, a communication unit, an external server, a character recognition unit, a voice generation unit, and a voice output unit.

[1141] Program processing overview

[1142] Image capture and transmission

[1143] User:

[1144] The user puts on the smart glasses and directs their gaze toward the text information they want to read (for example, an electronic bulletin board at a station or a sign in town). By pressing a dedicated button on the side of the smart glasses, the image capture means is activated.

[1145] Device:

[1146] The smart glasses' camera is activated and captures an image containing text information. This image is temporarily stored in the smart glasses and then sent to a connected smartphone or computer (device) via Wi-Fi or Bluetooth.

[1147] Image upload and analysis

[1148] Device:

[1149] The device uploads the received image data to an external server using an HTTP request. Specifically, the image data is sent via a REST API endpoint using a POST request.

[1150] server:

[1151] The server temporarily stores the received image data in cloud storage, then passes the stored image data to an OCR engine (e.g., a general character recognition algorithm or a cloud-based OCR service) to extract text information from the image.

[1152] Generate and return audio data

[1153] server:

[1154] The extracted text information is then passed to a TTS engine (e.g., a cloud-based speech generation service) and converted into audio data, which is temporarily stored on the server and then sent back to the device using an HTTP response.

[1155] Device:

[1156] The device analyzes the received audio data, stores it in temporary storage, and then transmits it to the smart glasses via Bluetooth.

[1157] Playing audio

[1158] User:

[1159] Smart glasses can play audio data using built-in audio output means (e.g., bone conduction speakers or earphones), allowing users to hear text information displayed in front of them aloud.

[1160] Specific examples

[1161] In the case of station electronic bulletin boards

[1162] The user presses a button on the smart glasses while looking at the station's electronic bulletin board. This action causes the smart glasses' camera to capture an image of the bulletin board and send it to the device. The device then uploads the image to the server, which uses an OCR engine to extract text information such as "To XX, departure time XX:XX." The TTS engine then converts this text into voice data and sends it back to the smart glasses via the device. Finally, the smart glasses play back this voice data, allowing the user to hear "To XX, departure time XX:XX."

[1163] For signs in the city

[1164] The same procedure applies when a user finds a sign in the city. When the user presses a button on the smart glasses, an image of the sign is captured and uploaded to the server via the device. The server's OCR engine extracts the sign's text information (e.g., "Restaurant XX, business hours XX to XX"), and the TTS engine generates voice data. Finally, the voice data is sent back to the smart glasses and played back, allowing the user to obtain this information by voice.

[1165] This will realize a system that allows visually impaired people to independently and efficiently obtain text information about their daily lives through audio.

[1166] The processing flow will be explained below.

[1167] Step 1:

[1168] User: The user puts on the smart glasses and brings the text information they want to read (e.g., magazines, signs, train station electronic bulletin boards, etc.) into their field of view. Then, they press the button on the side of the smart glasses.

[1169] Step 2:

[1170] Device: The camera built into the smart glasses is activated and captures images containing text information within the field of view. The captured images are temporarily stored in a buffer in the smart glasses in JPEG or PNG format.

[1171] Step 3:

[1172] Terminal: Image data captured from smart glasses is sent to a connected smartphone or computer (terminal) via Wi-Fi or Bluetooth communication.

[1173] Step 4:

[1174] Terminal: The smartphone or PC uploads the received image data to an external server using an HTTP request. Specifically, it sends the image data as a POST request via a REST API endpoint.

[1175] Step 5:

[1176] Server: The server processes the received HTTP request and temporarily stores the image data in cloud storage. After storage is complete, the server prepares to call an OCR engine (e.g., optical character recognition technology).

[1177] Step 6:

[1178] Server: The OCR engine analyzes the images stored in cloud storage and extracts the text information from the images. The extracted text information is temporarily stored in the server's memory as text data.

[1179] Step 7:

[1180] Server: The server then passes the extracted text data to a Text-to-Speech (TTS) engine, which converts the text into audio data, and the generated audio data is saved in a specified directory on the server.

[1181] Step 8:

[1182] Server: The generated audio data is returned to the device as an HTTP response. The successful transmission of the audio data is recorded in the server log.

[1183] Step 9:

[1184] Device: The device analyzes the voice data received from the server and stores it in temporary storage. It then transmits the voice data to the smart glasses via Bluetooth.

[1185] Step 10:

[1186] Terminal: The smart glasses receive the audio data and play it back using the built-in audio output means (e.g., bone conduction speaker or earphones).

[1187] Step 11:

[1188] User: The user can listen to the audio being played through the smart glasses. Use the volume control buttons to adjust the volume as needed.

[1189] The above steps realize a system that enables visually impaired people to quickly and accurately obtain text information in daily life through speech.

[1190] Example 1

[1191] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1192] It is important in daily life for visually impaired people and those who have difficulty reading and writing to quickly and accurately acquire text information. However, conventional methods for acquiring text information by voice have problems such as cumbersome and time-consuming operations, difficulty in synchronizing images and voice, and lack of portability. The purpose of this invention is to solve these problems and provide a system that allows users to easily and quickly acquire text information by voice.

[1193] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1194] In this invention, the server includes image acquisition means for acquiring an image in the direction in which the user is looking, means for transmitting the acquired image to an external server via communication means, character recognition means for extracting text information from the image received by the external server, voice generation means for converting the extracted text information into voice data, means for receiving the generated voice data via the communication means and transmitting it to voice output means via Bluetooth, and voice output means for playing back the generated voice data. This enables visually impaired people and people who have difficulty reading and writing to efficiently acquire text information necessary in daily life by voice.

[1195] The "image acquisition means" is a device or component that has the function of capturing an image in the direction in which the user is looking.

[1196] "Communication means" refers to the technical means for sending and receiving image data and audio data between external servers and terminals, including Wi-Fi and Bluetooth.

[1197] An "external server" is a computer system that communicates with a terminal via a network and has the functions of receiving, analyzing, and saving image data, and generating and returning audio data.

[1198] "Character recognition means" means a technology or device that has the function of extracting text information from an image, and includes character recognition algorithms and services such as OCR.

[1199] "Speech generation means" refers to the technology and devices that convert extracted text information into speech data, and includes text-to-speech engines (TTS engines).

[1200] The "audio output means" is a device that has the function of reproducing the generated audio data, and includes bone conduction speakers, earphones, etc.

[1201] "Cloud storage" is a storage service for storing and managing data on a network, and functions as part of data storage within a server.

[1202] A "wearable device" is a portable electronic device that is worn by a user and includes a camera and audio output means.

[1203] This invention relates to a smart glasses system that enables visually impaired people and people with reading and writing difficulties to obtain text information by voice in their daily lives. The system includes an image acquisition unit, a communication unit, an external server, a character recognition unit, a voice generation unit, and a voice output unit.

[1204] The user wears the smart glasses and directs their gaze toward text information (e.g., a train station bulletin board or sign). The user presses a dedicated button on the side of the smart glasses to activate the image capture means.

[1205] At the device, the camera built into the smart glasses captures an image in the direction of gaze, which is temporarily stored in the smart glasses and then sent to a connected smartphone or computer (device) via Wi-Fi or Bluetooth.

[1206] The device uploads the received image data to an external server using an HTTP request. Specifically, the image data is sent via a REST API endpoint using the POST method.

[1207] The server temporarily stores the received image data in cloud storage, then processes the image using the Python library OpenCV and extracts text information from the image using Tesseract OCR or a cloud-based OCR engine.

[1208] The extracted text information is then passed to a TTS (Text-to-Speech) engine, for example, using a cloud-based speech generation service (e.g., Amazon Polly or Google Cloud Text-to-Speech), which converts the text into audio data that is again stored in cloud storage and then sent back to the device using an HTTP response.

[1209] The device analyzes the received voice data and temporarily stores it in local storage, then transmits the data to the smart glasses via Bluetooth.

[1210] Finally, users can play audio data using the smart glasses' built-in audio output means (e.g., bone conduction speakers or earphones), allowing them to obtain the text information they are viewing by audio.

[1211] Specific examples

[1212] For example, a user looks at a train station's electronic bulletin board and presses a button on the smart glasses. This operation causes the camera on the smart glasses to capture an image of the bulletin board and send it to the device. The device uploads the image to the server, and the server uses an OCR engine to extract text information such as "To XX, departure time XX:XX." The TTS engine then converts this text into voice data and sends it back to the smart glasses via the device. The user can hear "To XX, departure time XX:XX" aloud through the voice output means of the smart glasses.

[1213] An example of a prompt sentence could be, "Please create a voice guide for when you see the electronic bulletin board at a station." This system will enable visually impaired people and people who have difficulty reading and writing to efficiently obtain text information in their daily lives by voice.

[1214] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1215] Step 1:

[1216] The user puts on the smart glasses and directs their gaze towards text information (e.g., an electronic bulletin board at a station or a sign in town). At this time, the user presses a dedicated button on the smart glasses. This activates the image capture means, and the camera captures an image in the direction of the gaze. The input is the user's gaze and button operation, and the output is the captured image data.

[1217] Step 2:

[1218] The device receives image data temporarily stored in the smart glasses and communicates via Wi-Fi or Bluetooth. The input is image data from the smart glasses, and the output is image data sent to the device. Specifically, the device receives the image and temporarily stores it in memory.

[1219] Step 3:

[1220] The device uploads the received image data to an external server using an HTTP request. Specifically, the device sends the image data to a REST API endpoint using the POST method. The input is the image data stored on the device, and the output is the HTTP request sent to the server. During this process, the device performs communication processing to send the data over the network.

[1221] Step 4:

[1222] The server temporarily stores the received image data in cloud storage. Using the stored image data, the server passes the image to an OCR engine (e.g., Tesseract OCR or Google Cloud Vision API) to extract text information. The input is the image data stored in cloud storage, and the output is the extracted text information. Specifically, the server analyzes the image data using an image processing library.

[1223] Step 5:

[1224] The server passes the extracted text information to a TTS engine (e.g., Amazon Polly or Google Cloud Text-to-Speech) and converts it into speech data. The input is the extracted text information, and the output is the generated speech data. The server executes the text-to-speech conversion process and stores the results in cloud storage.

[1225] Step 6:

[1226] The server returns the generated audio data to the device using an HTTP response. The input is the audio data stored in cloud storage, and the output is an HTTP response to the device. Specifically, the server performs communication processing to send the audio data to the device.

[1227] Step 7:

[1228] The device analyzes the received voice data and temporarily stores it in local storage. The device then transmits the voice data to the smart glasses using Bluetooth. The input is the voice data received from the server, and the output is the voice data transmitted to the smart glasses. Specifically, the device transfers the voice data via Bluetooth communication.

[1229] Step 8:

[1230] The user plays audio data using the smart glasses' built-in audio output means (bone conduction speaker or earphones). The input is audio data received via Bluetooth, and the output is the played audio. Specifically, the smart glasses process the received audio data and provide audio information to the user through the audio output means.

[1231] The above steps realize a system that enables visually impaired people and users who have difficulty reading and writing to efficiently obtain text information by voice.

[1232] (Application example 1)

[1233] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1234] In factories, workers need to quickly and accurately obtain information from machine control panels and signs, but visual confirmation alone takes time and carries the risk of misreading. This results in problems of reduced work efficiency. Furthermore, visually impaired workers in particular have difficulty operating equipment and checking signs, making it difficult to ensure work safety and efficiency.

[1235] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1236] In this invention, the server includes image acquisition means, means for transmitting images to an external server via communication means, character recognition means for extracting text information from the images received by the external server, voice generation means for converting the extracted text information into voice data, voice output means for receiving and playing the generated voice data via the communication means, and means for recognizing machine operation panels and signs in the factory and conveying that information to workers by voice. This allows workers to obtain necessary information by voice without having to confirm the information visually, improving work efficiency and ensuring safety.

[1237] An "image capture means" is a device for capturing visual information as digital data.

[1238] A "communication means" is a device or technology for sending and receiving digital data to other devices or servers.

[1239] An "external server" is a remote computer system for receiving, processing, storing, or transmitting digital data.

[1240] "Character recognition means" refers to software or hardware that analyzes characters and symbols in image data and extracts them as text data.

[1241] "Speech generation means" refers to technology or devices for converting text data into speech data.

[1242] The "audio output means" is a device for reproducing audio data so that it can be recognized by the human ear.

[1243] "Means of recognizing machine control panels and signs in a factory and communicating that information to workers by voice" refers to technology or equipment that uses a camera to capture important information from machine control panels and signs in a factory, converts that information into text data, and then converts it into audio data to communicate it to workers by voice.

[1244] "Cloud storage" is a remote server where data can be stored and accessed over the internet.

[1245] A "wearable device" is an electronic device that can be worn by a user and is often equipped with a camera, communication means, etc.

[1246] A "mobile device" is a portable electronic device such as a mobile phone or tablet.

[1247] A "factory robot" is an automated machine designed to perform work within a factory, and may have the role of monitoring work and conveying instructions.

[1248] The present invention provides a system for communicating text information displayed on machine control panels and signs to workers in a factory by voice. The system includes an image acquisition unit, a communication unit, an external server, a character recognition unit, a voice generation unit, and a voice output unit.

[1249] System Program Overview

[1250] 1. Image capture and transmission

[1251] The robot uses a camera to capture images of control panels and signs in the factory, and the images are temporarily stored in the robot's internal computer.

[1252] 2. Image upload and analysis

[1253] The robot's internal computer uploads image data to an external server using an HTTP request, which then stores the data in cloud storage and uses an OCR engine to extract text information from the image.

[1254] 3. Generating and returning audio data

[1255] The server passes the extracted text information to the TTS engine, which generates voice data, which is then sent back to the robot's computer using an HTTP response.

[1256] 4. Audio playback

[1257] The computer inside the robot analyzes the received audio data and plays the audio using the built-in speaker or an audio output device connected via Bluetooth.

[1258] Hardware and software used

[1259] Hardware: A high-resolution camera mounted on the robot is used for image acquisition, a high-performance computer within the robot is used for image processing and communication, and built-in speakers or a Bluetooth headset are used for audio output.

[1260] Software: Use a cloud-based OCR engine (e.g., Google Cloud Vision API) for image analysis and a cloud-based TTS engine (e.g., Amazon Polly) for voice generation.

[1261] Specific examples of processing

[1262] Example 1:

[1263] The robot uses a camera to capture an image of the factory's control panel and recognizes the text, "Machine status: Operating normally." This text information is converted into voice data, which is then played back from the robot's speaker: "Machine status is operating normally."

[1264] Example 2:

[1265] The robot uses its camera to capture factory signs, recognizes the text "Warehouse A: Material Delivery Area," and relays this information to workers via voice, allowing them to obtain the necessary information without having to visually confirm it.

[1266] Prompt Sentence Examples

[1267] "When the robot is in front of an operation panel in the factory, please generate a program that will capture the text on the operation panel using a camera and convey this text information to the worker by voice."

[1268] This configuration allows for efficient and safe work within the factory.

[1269] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1270] Step 1:

[1271] Image capture and saving

[1272] The robot uses a camera to capture images of control panels and signs in the factory, and the image data captured by the camera is temporarily stored in the robot's internal computer.

[1273] Input: Actual images of control panels and signs.

[1274] Data processing: The camera converts visual information into digital image data.

[1275] Output: Image data stored in the robot's internal computer.

[1276] Step 2:

[1277] Uploading an image

[1278] The computer inside the robot uploads the captured image data to an external server using HTTP requests, which uses a REST API.

[1279] Input: Stored image data.

[1280] Data processing: Convert image data into HTTP request format.

[1281] Output: Image data sent to an external server.

[1282] Step 3:

[1283] Extracting text information

[1284] The server stores the received image data in cloud storage and then uses an OCR engine to extract text information from the image.

[1285] Input: Image data stored on an external server.

[1286] Data processing: The OCR engine analyzes characters from image data and extracts text data.

[1287] Output: The extracted text information.

[1288] Step 4:

[1289] Generate audio data

[1290] The server passes the extracted text information to a TTS engine to generate audio data, which then converts the text data into an audio file.

[1291] Input: The extracted text information.

[1292] Data processing: The TTS engine converts the text into an audio file.

[1293] Output: The generated audio data.

[1294] Step 5:

[1295] Returning audio data

[1296] The server sends the generated audio data back to the computer in the robot using an HTTP response.

[1297] Input: The generated audio data.

[1298] Data processing: Converts audio data into HTTP response format.

[1299] Output: Audio data sent to the computer inside the robot.

[1300] Step 6:

[1301] Playing audio

[1302] The computer inside the robot analyzes the received audio data and plays the audio using the built-in speaker or an audio output device connected via Bluetooth.

[1303] Input: Audio data stored in the robot's internal computer.

[1304] Data processing: Converting audio data into a playable format.

[1305] Output: Audio played through speakers or headset.

[1306] The above are the specific processing steps of the program for the system that realizes the application example.

[1307] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1308] The present invention provides a smart glasses system that allows visually impaired people and people with reading and writing difficulties to acquire textual information by voice in their daily lives, recognizes the user's emotional state during the information acquisition process, and responds appropriately accordingly. The system includes an image acquisition unit, a communication unit, an external server, a character recognition unit, a voice generation unit, a voice output unit, and an emotion recognition unit.

[1309] Program processing overview

[1310] Image capture and transmission

[1311] User:

[1312] The user puts on the smart glasses and directs their gaze toward the text information they want to read. By pressing a button on the side of the smart glasses, the image capture means is activated.

[1313] Device:

[1314] The camera built into the smart glasses is activated to capture images, including text information, of the field of view. These images are temporarily stored in a buffer in the smart glasses in JPEG or PNG format and then sent to a connected smartphone or computer (device) via Wi-Fi or Bluetooth.

[1315] Image upload and analysis

[1316] Device:

[1317] The device uploads the received image data to an external server using an HTTP request. Specifically, the device sends the image data as a POST request via a REST API endpoint.

[1318] server:

[1319] The server temporarily stores the received image data in cloud storage.The image stored in cloud storage is then analyzed using an OCR engine to extract text information from the image.The extracted text data is temporarily stored in the server's memory.

[1320] Generate and return audio data

[1321] server:

[1322] The extracted text information is then passed to the TTS engine and converted into voice data, which is then saved in a specified directory on the server and sent back to the device as an HTTP response.

[1323] Device:

[1324] The device analyzes the received voice data, stores it in temporary storage, and then transmits the data to the smart glasses via Bluetooth.

[1325] Playing audio

[1326] Device:

[1327] The smart glasses receive the audio data and play it back using the built-in audio output means.

[1328] User:

[1329] The user can listen to the audio being played through the smart glasses.

[1330] Emotion recognition and response

[1331] Operation of emotion recognition means

[1332] Device:

[1333] The emotion recognition means is designed to analyze the user's facial expressions and voice data in real time, for example, by capturing the user's facial expressions and tone of voice through a camera and microphone installed in the smart glasses and analyzing the data.

[1334] server:

[1335] The analyzed data is sent to a server, where an emotion recognition engine determines the user's emotional state. If the emotional state is negative (sadness, anger, discomfort, etc.), the server generates a message of comfort or encouragement. On the other hand, if the emotional state is positive (joy, relief, excitement, etc.), it generates a recommended message to maintain that state.

[1336] Responding according to emotions

[1337] server:

[1338] The generated message is converted into voice data and sent from the server to the device, which then transmits the received voice data to the smart glasses.

[1339] Device:

[1340] An appropriate message is played to the user through the audio output means of the smart glasses.

[1341] Specific examples

[1342] In the case of station electronic bulletin boards

[1343] The user looks at the station's electronic display board and presses a button on the smart glasses. This action causes the smart glasses' camera to capture an image of the display board and send it to the device. The device then uploads the image to the server, which uses an OCR engine to extract "Travel to XX, departure time XX:XX." The TTS engine generates voice data and sends it back to the smart glasses via the device. The user listens to this voice. Meanwhile, the emotion recognition means analyzes the user's emotions and generates an appropriate response. For example, if a negative emotion is detected, a comforting message such as "Don't worry, the next train will arrive at XX:XX" is played.

[1344] For signs in the city

[1345] A similar procedure is applied when a user finds a sign in the city and presses a button on the smart glasses. An image of the sign is captured and sent to the server via the device. The server's OCR engine extracts the text information and generates audio data such as "Restaurant XX, business hours XX to XX." The audio data is sent back to the smart glasses, which the user listens to. During this time, if the emotion recognition means detects a positive emotion, a recommendation message such as "This restaurant has a very good reputation" is played.

[1346] In this way, a system is realized that enables visually impaired people to remain independent while efficiently obtaining textual information in daily life through speech, and in the process provides appropriate responses according to their emotions.

[1347] The processing flow will be explained below.

[1348] Step 1:

[1349] User: The user puts on the smart glasses and directs their gaze towards the text information they want to read (e.g., a magazine, a sign, a train station electronic bulletin board, etc.), then presses the button on the side of the smart glasses.

[1350] Step 2:

[1351] Device: The camera built into the smart glasses is activated and captures images containing text information within the field of view. The captured images are temporarily stored in a buffer in the smart glasses in JPEG or PNG format.

[1352] Step 3:

[1353] Terminal: Image data captured from smart glasses is sent to a connected smartphone or computer (terminal) via Wi-Fi or Bluetooth communication.

[1354] Step 4:

[1355] Terminal: The smartphone or PC uploads the received image data to an external server using an HTTP request. Specifically, it sends the image data as a POST request via a REST API endpoint.

[1356] Step 5:

[1357] Server: The server processes the received HTTP request and temporarily stores the image data in cloud storage. After storage is complete, the server prepares to call an OCR engine (e.g., optical character recognition technology).

[1358] Step 6:

[1359] Server: The OCR engine analyzes the images stored in cloud storage and extracts the text information from the images. The extracted text information is temporarily stored in the server's memory as text data.

[1360] Step 7:

[1361] Server: The server then passes the extracted text data to a Text-to-Speech (TTS) engine, which converts the text into audio data, and the generated audio data is saved in a specified directory on the server.

[1362] Step 8:

[1363] Server: The generated audio data is returned to the device as an HTTP response. The successful transmission of the audio data is recorded in the server log.

[1364] Step 9:

[1365] Device: The device analyzes the voice data received from the server and stores it in temporary storage. It then transmits the voice data to the smart glasses via Bluetooth.

[1366] Step 10:

[1367] Terminal: The smart glasses receive the audio data and play it back using the built-in audio output means (e.g., bone conduction speaker or earphones).

[1368] Step 11:

[1369] User: The user can listen to the audio being played through the smart glasses. Use the volume control buttons to adjust the volume as needed.

[1370] Step 12:

[1371] Device: The emotion recognition system captures the user's facial expressions and tone of voice through the smart glasses' camera and microphone. The captured data is then analyzed in real time by the device.

[1372] Step 13:

[1373] Terminal: The emotion recognition means determines the user's emotional state based on the analysis data, and determines whether the user is feeling negative (e.g., sadness, anger, discomfort) or positive (e.g., joy, relief, excitement).

[1374] Step 14:

[1375] Device: If a negative emotional state is detected, the device generates a comforting or encouraging message and prepares to send it to the server as audio data. On the other hand, if a positive emotional state is detected, the device generates a recommendation message for maintaining that state and prepares to send it to the server as audio data.

[1376] Step 15:

[1377] Server: Receives message data generated by the device and converts it into voice data using a TTS engine. The generated voice data is then sent back to the device.

[1378] Step 16:

[1379] Terminal: Transmits the voice data received from the server to the smart glasses.

[1380] Step 17:

[1381] Terminal: An appropriate emotional message is played to the user through the audio output means of the smart glasses.

[1382] Specific examples

[1383] In the case of station electronic bulletin boards

[1384] The user looks at the station's electronic display board and presses a button on the smart glasses. This action causes the smart glasses' camera to capture an image of the display board and send it to the device. The device then uploads the image to the server, which uses an OCR engine to extract "Travel to XX, departure time XX:XX." The TTS engine generates voice data and sends it back to the smart glasses via the device. The user listens to this voice. Meanwhile, the emotion recognition means analyzes the user's emotions and generates an appropriate response. For example, if a negative emotion is detected, a comforting message such as "Don't worry, the next train will arrive at XX:XX" is played.

[1385] For signs in the city

[1386] A similar procedure is applied when a user finds a sign in the city and presses a button on the smart glasses. An image of the sign is captured and sent to the server via the device. The server's OCR engine extracts the text information and generates audio data such as "Restaurant XX, business hours XX to XX." The audio data is sent back to the smart glasses, which the user listens to. During this time, if the emotion recognition means detects a positive emotion, a recommendation message such as "This restaurant has a very good reputation" is played.

[1387] This will enable visually impaired people to remain independent and efficiently obtain textual information in their daily lives through audio, and a system will be realized that provides appropriate responses based on emotions in the process.

[1388] Example 2

[1389] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1390] Visually impaired people and those with reading and writing difficulties face the challenge of efficiently acquiring textual information in their daily lives. Even when textual information is acquired, the user's emotional state is often not properly understood, and necessary support and encouragement are often not provided. To address this issue, a system that not only converts textual information into speech, but also recognizes the user's emotional state in real time and responds accordingly is needed.

[1391] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1392] In this invention, the server includes image acquisition means, means for transmitting the acquired image to an external server via communication means, character recognition means for extracting text information from the image received by the external server, voice generation means for converting the extracted text information into voice data, voice output means for receiving the generated voice data via the communication means and playing it back, emotion recognition means for analyzing the emotional state of the user, and means for generating an appropriate message according to the user's emotional state and converting it into voice data. This allows the user to efficiently acquire text information by voice, recognize the emotional state in real time during the process, and provide an appropriate response.

[1393] "Image capture means" refers to a device or method that electronically images information visually captured by a user using a device's camera.

[1394] "Communication means" refers to a method or device for transmitting acquired data to other devices or systems, including, for example, Wi-Fi and Bluetooth.

[1395] An "external server" is an information processing device that is connected via a network, separate from the device used by the user, and has the role of analyzing and storing data.

[1396] "Character recognition means" refers to technology or devices that detect character information contained in image data and extract it as text data. For example, this includes OCR (optical character recognition) technology.

[1397] "Speech generation means" refers to technology or equipment that converts text data into speech data, for example, using TTS (Text to Speech) technology.

[1398] "Audio output means" refers to a device or method for outputting generated audio data in a form that can be heard by the user, and includes speakers and earphones.

[1399] "Emotion recognition means" refers to technology or devices that analyze a user's facial expression or tone of voice to determine their emotional state. Examples include facial expression recognition and voice emotion analysis.

[1400] The "means for generating an appropriate message and converting it into voice data" refers to a technology or device that creates a message corresponding to the user's emotional state based on the user's emotional state obtained by the emotion recognition means, and converts that message into voice data.

[1401] This invention is a smart glasses system that allows visually impaired people and people with reading and writing difficulties to acquire text information by voice in their daily lives, recognize the user's emotional state during the information acquisition process, and respond appropriately accordingly. This system includes image acquisition means, communication means, an external server, character recognition means, voice generation means, voice output means, and emotion recognition means.

[1402] Image Acquisition Method

[1403] The user puts on the smart glasses, directs their gaze at the text information they want to read, and presses a button on the smart glasses. This action activates the camera built into the smart glasses, which captures an image containing the text information in their field of view. The image is temporarily saved in a buffer in the smart glasses in JPEG or PNG format.

[1404] communication means

[1405] The acquired image data is sent to a connected smartphone or PC (terminal) via Wi-Fi or Bluetooth. The terminal then sends the received image data to an external server as an HTTP request. Specifically, the image data is sent as a POST request via a REST API endpoint.

[1406] External Server

[1407] The server receives the POST request and temporarily stores the image data in cloud storage.The image stored in cloud storage is then analyzed using an OCR engine (for example, Google Cloud Vision API) to extract text information from the image.The extracted text data is temporarily stored in the server's memory.

[1408] Character recognition means

[1409] The character recognition means is a technology that uses an OCR engine to extract characters from image data. The extracted text data is passed to the voice generation means.

[1410] Voice generation means

[1411] The voice generation means converts the extracted text information into voice data using a TTS engine (for example, Google Text-to-Speech API). The generated voice data is saved in a specified directory on the server and then returned to the terminal as an HTTP response.

[1412] Audio output means

[1413] The device analyzes the received audio data and stores it in temporary storage. It then transmits the audio data to the smart glasses via Bluetooth. The smart glasses receive the audio data and play it back through their built-in speakers or earphones, allowing the user to listen to the audio.

[1414] emotion recognition means

[1415] The emotion recognition means uses a camera and microphone installed in the smart glasses to capture and analyze the user's facial expressions and tone of voice in real time. This determines the user's emotional state in relation to the recognized text information. Analysis is performed by an emotion recognition engine (e.g., Amazon Rekognition or Microsoft Azure Emotion API). If a negative emotion (sadness, anger, displeasure, etc.) is detected, a comforting or encouraging message is generated. If a positive emotion (joy, relief, excitement, etc.) is detected, a recommended message that maintains that state is generated.

[1416] For example, when a user looks at a train station's electronic display board and presses a button on the smart glasses, the camera in the smart glasses captures an image of the display board. The device uploads the image to a server, and the server's OCR engine extracts "To XX, departure time XX:XX." The TTS engine generates voice data and sends it back to the smart glasses via the device. The user listens to this voice. Meanwhile, the emotion recognition means analyzes the user's emotions, and if a negative emotion is detected, a comforting message such as "Don't worry, the next train will arrive at XX:XX" is played.

[1417] The same process is also performed when the user gazes at a sign in the city. Text information is extracted from the image of the sign, and audio data such as "Restaurant XX, business hours XX to XX" is generated and sent back to the smart glasses. If the emotion recognition means detects a positive emotion while the user is listening to the audio, a message such as "This restaurant has a very good reputation" is played.

[1418] This system enables visually impaired people to independently access textual information in their daily lives efficiently through speech, and provides appropriate responses according to their emotions in the process.

[1419] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1420] Step 1:

[1421] The user puts on the smart glasses, directs their gaze at the text information they want to read, and presses a button on the smart glasses. This action activates the camera built into the smart glasses, which captures an image containing the text information in their field of view. The captured image is temporarily saved in a buffer within the smart glasses in JPEG or PNG format. The input is the user's gaze and button press, and the output is the image data saved in the smart glasses buffer.

[1422] Step 2:

[1423] The device receives temporarily stored image data from the smart glasses via Wi-Fi or Bluetooth. The received image data is temporarily stored in the device's internal storage. The device then uploads this image data to an external server using an HTTP request. Specifically, the device sends the image data as a POST request via a REST API endpoint. The input is the image data from the smart glasses, and the output is an HTTP request to the external server.

[1424] Step 3:

[1425] The server processes the received POST request and temporarily stores the image data in cloud storage. Next, the image data stored in cloud storage is passed to an OCR engine (for example, Google Cloud Vision API) for analysis. The OCR engine detects character information in the image and extracts it as text data. The extracted text data is temporarily stored in the server's memory. The input is the image data in cloud storage, and the output is the extracted text data.

[1426] Step 4:

[1427] The server passes the extracted text data to a TTS engine (for example, Google Text-to-Speech API) and converts it into audio data. This audio data is saved in a specified directory on the server and then sent back to the device as an HTTP response. The input is the text data obtained by the OCR engine, and the output is the generated audio data.

[1428] Step 5:

[1429] The device analyzes the audio data in the received HTTP response and stores it in temporary storage. It then transmits the audio data to the smart glasses via Bluetooth. The input is the audio data returned from the server, and the output is the audio data sent to the smart glasses.

[1430] Step 6:

[1431] Smart glasses play back audio data received via Bluetooth. The audio is transmitted to the user through built-in speakers or earphones. The input is the audio data sent from the device, and the output is the played audio. The user can listen to this audio.

[1432] Step 7:

[1433] Using the smart glasses' built-in camera and microphone, the device captures the user's facial expressions and tone of voice in real time and passes them to the emotion recognition means. The emotion recognition means analyzes the data and determines the user's emotional state. The input is the facial expressions and tone of voice captured in real time, and the output is the analyzed emotional data.

[1434] Step 8:

[1435] The server generates a message of comfort or encouragement based on the user's emotional state obtained from the emotion recognition means if a negative emotion is detected, and generates a message of recommendation to maintain that state if a positive emotion is detected. The generated message is converted back into voice data and sent to the device. The input is the analyzed emotion data, and the output is the voice data of the generated message.

[1436] Step 9:

[1437] The device sends audio data based on the received emotional response to the smart glasses, which then plays it through the built-in speaker or earphones. This allows the user to receive an appropriate response message according to their emotional state. The input is the emotional response audio data from the server, and the output is the audio played by the smart glasses.

[1438] (Application example 2)

[1439] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1440] Visually impaired people and people with reading and writing difficulties face difficulties in efficiently obtaining product information in physical stores. Furthermore, there is a need for a system that can accurately recognize the user's emotional state during the information acquisition process and provide appropriate responses and support according to that emotion. Furthermore, it is necessary to realize a system in which a wearable device worn by the user can perform these tasks in an integrated manner.

[1441] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes image acquisition means, means for transmitting the acquired image to an external server via communication means, character recognition means for extracting text information from the image received by the external server, voice generation means for converting the extracted text information into voice data, voice output means for receiving and playing the generated voice data via the communication means, emotion recognition means for recognizing an emotional state, and means for generating an appropriate message according to the recognized emotional state and converting it into voice data. This enables visually impaired people and people who have difficulty reading and writing to not only efficiently obtain product information in physical stores, but also receive appropriate responses and support according to their emotions during the process.

[1442] "Image acquisition means" refers to a camera or imaging device for capturing visual information as digital data.

[1443] "Communication means" refers to a means for bidirectionally communicating information such as image data and audio data with external devices or servers, and utilizes wireless communication technologies such as Wi-Fi and Bluetooth.

[1444] An "external server" is a remote server that performs processes such as analyzing image data, extracting text information, and generating audio data.

[1445] "Character recognition means" refers to optical character recognition (OCR) technology for extracting text information from captured image data.

[1446] The "voice generation means" is a voice synthesis technology for converting the extracted text information into voice data.

[1447] The "audio output means" is a device that plays back audio to notify the user of the generated audio data, and includes a speaker, a bone conduction speaker, and the like.

[1448] "Emotion recognition means" is a technology for analyzing the user's emotional state, and uses a camera and microphone to analyze facial expressions and vocal tones.

[1449] The "means for generating an appropriate message" is a technology for generating an appropriate message to be provided to the user in accordance with the recognized emotional state and converting it into voice data.

[1450] A "wearable device" is a portable electronic device that can be worn by a user and is used to acquire biometric data and present information.

[1451] This invention provides a system that enables visually impaired people and people with reading and writing difficulties to efficiently obtain product information in physical stores and receive appropriate support according to their emotions during the process. The system includes smart glasses, image acquisition means, communication means, an external server, character recognition means, voice generation means, voice output means, emotion recognition means, means for generating appropriate messages, and a wearable device.

[1452] System Program

[1453] Image capture and transmission

[1454] When a user wears the smart glasses, points their eyes at a product, and presses a button, the camera built into the smart glasses captures the product and sends the image to a device. The device then uploads the received image data to an external server via Wi-Fi or Bluetooth. This process uses software tools such as OpenCV and requests.

[1455] Image upload and analysis

[1456] The device sends the received image data to an external server using an HTTP request. The server temporarily stores the image in cloud storage and extracts text information from the image using an OCR engine. The extracted text information is temporarily stored in the server's memory.

[1457] Generate and return audio data

[1458] The server passes the extracted text information to the TTS engine to generate voice data. The generated voice data is saved in a specified directory on the server and sent back to the device as an HTTP response. The device then sends the received voice data to the smart glasses and plays the voice.

[1459] Emotion recognition and appropriate responses

[1460] The emotion recognition means captures the user's facial expressions and voice using the smart glasses' camera and microphone and analyzes the data. The analyzed data is sent to the server, where the emotion recognition engine determines the user's emotional state. If the emotional state is negative, the server generates a comforting or encouraging message. If the emotional state is positive, it generates a recommendation message to maintain that state. The generated message is converted into audio data and sent to the smart glasses via the terminal, where an appropriate message is played for the user.

[1461] Hardware and software used

[1462] Hardware: Smart glasses (including built-in camera and audio output means), smartphone or laptop

[1463] Software: Python, OpenCV (image capture and processing), requests (HTTP requests), playsound (audio playback), OCR engine, TTS engine

[1464] Specific examples

[1465] Prompt Sentence Examples

[1466] Image Recognition: "Please read out the product information I'm looking at"

[1467] Emotion recognition: "The user looks sad, so generate a comforting message."

[1468] This system enables visually impaired and people with reading and writing difficulties to efficiently obtain product information in physical stores, while also receiving appropriate emotional support in the process, thereby supporting users' independence and improving the quality of their daily lives.

[1469] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1470] Step 1:

[1471] The user puts on the smart glasses, looks at the product, and presses a button. The camera built into the smart glasses captures the product and sends the image to the device. The input is the product image, and the output is the image data sent to the device.

[1472] Step 2:

[1473] The device uploads the received image data to an external server via Wi-Fi or Bluetooth. During this process, the device sends the image using an HTTP POST request. The input is the image data received from the smart glasses, and the output is the image data sent to the server.

[1474] Step 3:

[1475] The server temporarily stores the received image data in cloud storage. Then, it uses an OCR engine to extract text information from the image. The input is the image data stored in cloud storage, and the output is the extracted text information.

[1476] Step 4:

[1477] The server passes the extracted text information to the TTS engine to generate voice data. The generated voice data is saved in a specified directory on the server and sent back to the device as an HTTP response. The input is the extracted text information, and the output is the generated voice data.

[1478] Step 5:

[1479] The device analyzes the received voice data and stores it in temporary storage.Then, it transmits the voice data to the smart glasses via Bluetooth.The input is the voice data received from the server, and the output is the voice data transmitted to the smart glasses.

[1480] Step 6:

[1481] The audio output means of the smart glasses receives the audio data and plays the audio through the built-in speaker. The input is the audio data received from the terminal, and the output is the audio heard by the user.

[1482] Step 7:

[1483] The emotion recognition means captures the user's facial expressions and voice through the camera and microphone of the smart glasses and analyzes the data. The input is the user's facial expressions and voice data, and the output is the analyzed emotion data.

[1484] Step 8:

[1485] The server passes the analyzed emotional data to an emotion recognition engine to determine the user's emotional state. If the emotional state is negative, it generates a comforting or encouraging message, and if it is positive, it generates a recommendation message. The input is the analyzed emotional data, and the output is an appropriate message.

[1486] Step 9:

[1487] The server passes the generated message to the TTS engine to convert it into voice data, which is then sent to the smart glasses via the device. The input is the generated message, and the output is voice data.

[1488] Step 10:

[1489] The smart glasses receive the audio data and play it back on the built-in speaker. The input is the audio data sent from the server, and the output is the audio the user hears.

[1490] The above series of processes realizes a system that allows users to obtain product information through voice and receive appropriate responses according to their emotions during the process.

[1491] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1492] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1493] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1494] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1495] FIG. 9 illustrates an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and behaviors arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1496] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1497] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1498] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1499] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1500] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1501] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1502] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1503] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1504] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1505] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1506] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1507] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1508] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1509] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1510] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1511] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1512] The following is further disclosed regarding the above embodiment.

[1513] (Claim 1)

[1514] Image acquisition means;

[1515] a means for transmitting the acquired image to an external server via a communication means;

[1516] character recognition means for extracting text information from the received image in the external server;

[1517] a voice generating means for converting the extracted text information into voice data;

[1518] The system includes an audio output means for receiving the generated audio data via the communication means and playing it back.

[1519] (Claim 2)

[1520] the external server includes a means for saving the image received from the image acquisition means in a cloud storage;

[1521] 2. The system according to claim 1, wherein the character recognition means and the voice generation means operate using images stored in cloud storage.

[1522] (Claim 3)

[1523] the audio output means is incorporated into a wearable device worn by a user,

[1524] The system of claim 1 , wherein the wearable device includes a camera and functions as the acquisition means.

[1525] "Example 1"

[1526] (Claim 1)

[1527] image acquisition means for acquiring an image in the direction in which the user is looking;

[1528] a means for transmitting the acquired image to an external server via a communication means;

[1529] character recognition means for extracting text information from the received image in the external server;

[1530] a voice generating means for converting the extracted text information into voice data;

[1531] a means for receiving the generated voice data via the communication means and transmitting the voice data to the voice output means via Bluetooth;

[1532] The system includes an audio output means for playing back the generated audio data.

[1533] (Claim 2)

[1534] the external server includes a means for saving the image received from the image acquisition means in a cloud storage;

[1535] 2. The system according to claim 1, wherein the character recognition means and the voice generation means operate using images stored in cloud storage.

[1536] (Claim 3)

[1537] the audio output means is incorporated into a wearable device worn by a user,

[1538] The system of claim 1 , wherein the wearable device comprises a camera and functions as the image acquisition means.

[1539] "Application Example 1"

[1540] result

[1541] (Claim 1)

[1542] Image acquisition means;

[1543] a means for transmitting the acquired image to an external server via a communication means;

[1544] character recognition means for extracting text information from the received image in the external server;

[1545] a voice generating means for converting the extracted text information into voice data;

[1546] an audio output means for receiving the generated audio data via the communication means and playing it back;

[1547] A system that includes a means of recognizing machine control panels and signs within a factory and conveying that information to workers via voice.

[1548] (Claim 2)

[1549] the external server includes a means for saving the image received from the image acquisition means in a cloud storage;

[1550] 2. The system according to claim 1, wherein the character recognition means and the voice generation means operate using images stored in cloud storage.

[1551] (Claim 3)

[1552] the voice output means is built into a mobile terminal used by a worker or a factory robot,

[1553] The system of claim 1 , wherein the camera functions as the acquisition means.

[1554] "Example 2: Combining Emotion Engines"

[1555] (Claim 1)

[1556] Image acquisition means;

[1557] a means for transmitting the acquired image to an external server via a communication means;

[1558] character recognition means for extracting text information from the received image in the external server;

[1559] a voice generating means for converting the extracted text information into voice data;

[1560] an audio output means for receiving the generated audio data via the communication means and playing it back;

[1561] emotion recognition means for analyzing the emotional state of a user;

[1562] means for generating an appropriate message according to the emotional state of the user and converting it into voice data;

[1563] A system including:

[1564] (Claim 2)

[1565] the external server includes a means for saving the image received from the image acquisition means in a cloud storage;

[1566] 2. The system according to claim 1, wherein the character recognition means and the voice generation means operate using images stored in cloud storage.

[1567] (Claim 3)

[1568] the audio output means is incorporated into a wearable device worn by a user,

[1569] the wearable device is equipped with a camera and functions as the acquisition means;

[1570] The system of claim 1, wherein the emotion recognition means further analyzes facial expressions and tone of voice using a wearable device.

[1571] "Application example 2 when combining emotion engines"

[1572] (Claim 1)

[1573] Image acquisition means;

[1574] a means for transmitting the acquired image to an external server via a communication means;

[1575] character recognition means for extracting text information from the received image in the external server;

[1576] a voice generating means for converting the extracted text information into voice data;

[1577] an audio output means for receiving the generated audio data via the communication means and playing it back;

[1578] an emotion recognition means for recognizing an emotional state;

[1579] The system includes a means for generating an appropriate message in response to the recognized emotional state and converting the message into audio data.

[1580] (Claim 2)

[1581] the external server includes a means for saving the image received from the image acquisition means in a cloud storage;

[1582] 2. The system according to claim 1, wherein the character recognition means and the voice generation means operate using images stored in cloud storage.

[1583] (Claim 3)

[1584] the audio output means is incorporated into a wearable device worn by a user,

[1585] the wearable device is equipped with a camera and functions as the acquisition means;

[1586] Furthermore, the emotion recognition means includes a camera or a microphone for analyzing the facial expression or voice of the user,

[1587] 2. The system according to claim 1, further comprising means for generating an appropriate message according to the user's emotion. [Explanation of symbols]

[1588] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. Image acquisition means; a means for transmitting the acquired image to an external server via a communication means; character recognition means for extracting text information from the received image in the external server; a voice generating means for converting the extracted text information into voice data; The system includes an audio output means for receiving the generated audio data via the communication means and playing it back.

2. the external server includes a means for saving the image received from the image acquisition means in a cloud storage; The system according to claim 1 , wherein the character recognition means and the voice generation means operate using images stored in cloud storage.

3. the audio output means is incorporated into a wearable device worn by a user, The system of claim 1 , wherein the wearable device includes a camera and functions as the acquisition means.

Images