System

Abstract

A system is provided.SOLUTION: A system for a visually impaired person to understand braille, comprising: means for activating an application using voice commands; means for acquiring an image of braille using a camera; means for transmitting the acquired image to a server; means including a generation AI for analyzing the image and identifying the braille on the server side; means for converting the braille identified by the generation AI to text data; and means for converting the text data received on the server side to voice and reading aloud the text data.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Only a small minority of visually impaired people can read Braille, and for many people who have become blind or are elderly, obtaining Braille information is a major barrier to their daily lives. The difficulty of obtaining Braille information limits the range of activities that visually impaired people can undertake. Furthermore, the limited opportunities to learn Braille make it difficult, and many visually impaired people go about their lives without ever being able to master it. [Means for solving the problem]

[0005] To solve this problem, we propose a system that allows visually impaired people to understand Braille through voice. This system includes the following means: First, it provides a means for launching an application using a voice command. Second, it includes a means for acquiring Braille image data using a camera and sending that image data to a server. On the server side, it uses a generation AI that analyzes the image data and identifies Braille, and it converts the identified Braille into text data. It then provides a means for sending the converted text data to a terminal, and includes a means for converting the received text data into voice and reading it aloud. In this way, even visually impaired people who have difficulty reading Braille can easily access Braille information through voice. This system reduces the barriers to information acquisition in everyday life and can expand the range of activities available to visually impaired people.

[0006] "Visually impaired persons" refers to people who are visually impaired and have reduced or no vision.

[0007] "Braille" is a pattern of raised dots used by visually impaired people to read letters and symbols using their sense of touch.

[0008] A "system" is a general term for a mechanism in which multiple elements and means work together to perform a specific function.

[0009] A "voice command" is an input method in which a user issues instructions to a system using voice.

[0010] An "application" is a software program designed to accomplish a particular purpose.

[0011] A "camera" is a device that captures light and converts it into image data.

[0012] "Image data" refers to still image information expressed in a digital format.

[0013] A "server" is a computer system that processes and manages data over a network.

[0014] "Generative AI" refers to algorithms and software that use artificial intelligence to analyze and generate data.

[0015] "Text data" refers to data that represents character information in digital form.

[0016] A "speech synthesis engine" is a system that artificially generates speech based on text data.

[0017] "Reading aloud" refers to the act of pronouncing letters or text information aloud.

[0018] "Noise removal" is a process of removing unnecessary information from acquired image data or audio data. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8]FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0020] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0021] First, the terms used in the following description will be explained.

[0022] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0023] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0024] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0025] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0026] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0027] [First embodiment]

[0028] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0029] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0030] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0031] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0032] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0033] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0034] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0035] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0036] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0037] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0038] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0039] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0040] The present invention is a system for visually impaired people to understand Braille, which uses a smartphone camera and generation AI to read Braille aloud. A specific embodiment of this system is described below.

[0041] System configuration

[0042] This system consists of a smartphone (terminal) operated by the user, a server that processes data, and an application that runs between them. The application runs on the smartphone and is activated by voice commands. A camera is used to capture an object with Braille printed on it, and the acquired image data is sent to the server. On the server, a generative AI analyzes the Braille from the image data and converts the recognized Braille into text data. Finally, the smartphone converts the text data into speech and reads it out loud to the user.

[0043] Program processing

[0044] The specific program processing executed by each device will be explained in natural language below.

[0045] Launching the app

[0046] User: Says the voice command to their smartphone: "Launch the Braille app."

[0047] Device: The voice recognition engine analyzes the voice command and identifies the corresponding application. The application launch command is executed, and the Braille reading application is launched.

[0048] Braille capture and transmission

[0049] User: Point the smartphone camera at an object with Braille on it.

[0050] Terminal: The camera acquires image data, stores it in memory, prepares it for transmission to the server, compresses it, and uploads it to the server.

[0051] Image data analysis

[0052] Server: Stores the received image data for analysis. Analysis begins and performs preprocessing such as noise removal. The generation AI inputs the preprocessed image data and analyzes the Braille patterns. The recognized Braille is converted into text data and sent to the device.

[0053] Converting text data into speech

[0054] Device: The device receives the text data and passes it to the speech synthesis engine. The speech synthesis engine converts the text data into voice data, which is then played back from the speaker.

[0055] User: Checks the content of Braille through audio data. This allows visually impaired people to obtain the content through audio without having to read Braille.

[0056] Specific examples

[0057] Scenario: Reading Braille on a station sign

[0058] 1. User: Say to the smartphone, "Launch the Braille app."

[0059] 2. Device: The app will launch and the main screen will appear.

[0060] 3. User: Point the camera at the Braille on the station sign.

[0061] 4. Terminal: Operate the camera to capture Braille and obtain image data.

[0062] 5. Terminal: Compresses the image data and sends it to the server.

[0063] 6. Server: Receives the image data and converts the Braille into text using the generation AI. The generated text is, "Platform 1 goes to Tokyo, and platform 2 goes to Shinjuku."

[0064] 7. Server: Sends text data to the device.

[0065] 8. Terminal: Converts the received text data into speech and reads out loud, "Platform 1 goes to Tokyo, and platform 2 goes to Shinjuku."

[0066] 9. User: Listen to the audio and understand the instructions.

[0067] Such an embodiment of the system will enable visually impaired people to easily obtain Braille information, which is expected to reduce barriers to obtaining information in daily life and expand the range of activities available to visually impaired people.

[0068] The processing flow will be explained below.

[0069] Step 1:

[0070] User: Says the voice command to their smartphone: "Launch the Braille app."

[0071] A voice recognition engine receives a user's voice command and analyzes the voice data.

[0072] Based on the recognized command, it determines whether to launch a Braille reading application.

[0073] Step 2:

[0074] Terminal: Execute the application launch command to launch the Braille reading app.

[0075] The app will start and the main screen will be displayed.

[0076] The camera control module is ready and waiting to capture braille.

[0077] Step 3:

[0078] User: Point the smartphone camera at an object with Braille on it.

[0079] Refer to the on-screen guidelines and audio guide to point the camera in the correct position.

[0080] Give a voice command for the camera button to prepare for capturing Braille.

[0081] Step 4:

[0082] Terminal: Braille image data is acquired using a camera.

[0083] The camera captures the image in the frame and stores the image data in memory.

[0084] The stored image data undergoes a compression process to prepare it for transmission to the server.

[0085] Step 5:

[0086] Terminal: Sends compressed image data to a server via the Internet.

[0087] Upload image data to the server's API endpoint.

[0088] Monitor the transmission status to ensure that the data was sent successfully.

[0089] Step 6:

[0090] Server: Initiates the process of analyzing the received image data.

[0091] The reception of the data is confirmed and the image data is stored in the database.

[0092] Generative AI performs pre-processing, noise removal, and image enhancement.

[0093] Step 7:

[0094] Server: Preprocessed image data is passed to the generation AI, which then identifies the Braille characters.

[0095] A generative AI model analyzes the image data and recognizes Braille patterns.

[0096] The recognized braille pattern is converted into text data.

[0097] Step 8:

[0098] Server: Sends the generated text data to the device.

[0099] Converts text data into an optimal format and sends it to the device's API endpoint.

[0100] Step 9:

[0101] Terminal: Converts the received text data into audio data and prepares it to be read to the user.

[0102] Check the integrity of the text data and pass it to the speech synthesis engine.

[0103] A speech synthesis engine converts the text data into speech data.

[0104] Step 10:

[0105] Device: The generated audio data is played through the device's speaker.

[0106] The audio data is played and read to the user.

[0107] Step 11:

[0108] User: Check the braille content through audio from the device.

[0109] Listen to the audio and get the information you need.

[0110] Example 1

[0111] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0112] It is extremely difficult for visually impaired people to understand Braille in their daily lives. In particular, there is a lack of means to quickly and accurately understand the Braille on information and guide signs provided in public places, which poses a major challenge for visually impaired people, as it takes a lot of time and effort for them to obtain the information they need.

[0113] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0114] In this invention, the server includes a means for processing image data acquired by an image acquisition device and identifying Braille, a means for converting Braille into character data using a generative AI model, and a means for transmitting the converted character data to a terminal, thereby enabling visually impaired people to efficiently and quickly understand Braille by voice using a terminal such as a smartphone.

[0115] "Visually impaired persons" refer to people who have impaired vision and are unable to recognize letters or figures using normal visual methods.

[0116] "Braille" refers to a writing system made up of raised and recessed patterns that is used by visually impaired people to understand written information using their sense of touch.

[0117] A "system" refers to a set of mechanisms in which multiple devices and software work together.

[0118] "Voice command" refers to a means by which a user gives instructions to electronic devices or software using voice.

[0119] "Application" means a software program designed to provide a particular function or service.

[0120] "Image capture device" refers to equipment that uses a device such as a camera to capture visual information as a digital image.

[0121] "Data processing device" refers to a computer or server system for storing and processing acquired data.

[0122] "Analysis" refers to the process of using algorithms and programs to decipher acquired data and extract meaningful information.

[0123] A "generative AI model" refers to an artificial intelligence model that is trained to perform a specific task using techniques such as machine learning and deep learning.

[0124] "Character data" refers to character information represented in digital form.

[0125] "Terminal" refers to an electronic device that can be directly operated by a user, such as a smartphone or tablet.

[0126] "Means for converting into audio" refers to the technology or software for converting text data into audio data and playing it back through an audio output device.

[0127] The present invention is a system for visually impaired people to understand Braille. An application is started using a voice command, Braille image data acquired by an image acquisition device is sent to a data processing device, and the Braille is converted into character data using a generative AI model, which is then read aloud. Specific embodiments of this system are described below.

[0128] System configuration

[0129] This system consists of a terminal operated by the user, a server that processes data, and an application that runs between them. The application runs on the terminal and is activated by voice commands. An image capture device (e.g., a smartphone camera) is used to capture an object with Braille printed on it, and the captured image data is sent to the server. On the server, a generative AI model analyzes the Braille from the image data and converts the recognized Braille into text data. Finally, the terminal converts the text data into audio and reads it to the user.

[0130] Application Behavior

[0131] Launching the app

[0132] The user issues a voice command to the device, saying, "Launch the Braille app." This voice command is analyzed by the device's voice recognition engine. The voice recognition engine uses technology such as Google Assistant or Siri to identify the corresponding application. The application launch command is executed, and the Braille reading app launches.

[0133] Braille capture and transmission

[0134] The user points the device's camera at an object with Braille. The camera captures image data, which is then stored in the device's memory. The stored image data is then compressed and uploaded to the server.

[0135] Image data analysis

[0136] The server stores the received image data. It then begins analyzing it and performs preprocessing such as noise removal using an image processing library such as OpenCV. The generative AI model uses the preprocessed image data as input and analyzes the Braille patterns. The recognized Braille characters are converted into character data and sent to the device.

[0137] Converting text data into speech

[0138] The device receives the text data and passes it to a speech synthesis engine such as Google Text-to-Speech or Apple's VoiceOver function. The speech synthesis engine converts the text data into audio data, which is then played back through the device's speaker. The user can then confirm the braille content through the audio data.

[0139] Specific examples

[0140] Scenario: Reading Braille on a station sign

[0141] 1. User: Say to the smartphone, "Launch the Braille app."

[0142] 2. Device: The app will launch and the main screen will appear.

[0143] 3. User: Point the camera at the Braille on the station sign.

[0144] 4. Terminal: Operate the camera to capture Braille and obtain image data.

[0145] 5. Terminal: Compresses the image data and sends it to the server.

[0146] 6. Server: Receives the image data and converts the Braille into text using a generative AI model. This generates text such as "Platform 1 goes to Tokyo, and platform 2 goes to Shinjuku."

[0147] 7. Server: Sends character data to the terminal.

[0148] 8. Terminal: Converts the received text data into speech and reads out loud, "Platform 1 goes to Tokyo, and platform 2 goes to Shinjuku."

[0149] 9. User: Listen to the audio and understand the instructions.

[0150] Examples of prompt statements

[0151] Using the voice command "Launch the Braille app" as input, the prompt sentence can be designed as follows:

[0152] "Just tell your smartphone to 'launch the Braille app' and point the camera at the Braille. The app will then capture the image and send it to the server. On the server side, a generative AI model will convert the image data from Braille to text data. Finally, the text data will be communicated to the user as an audio alert."

[0153] By inputting this prompt into the generative AI model, the entire process of reading Braille can be explained to visually impaired people, allowing them to understand the specific steps to take.

[0154] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0155] Step 1: Launch the app

[0156] User: Says the voice command to their smartphone: "Launch the Braille app."

[0157] Input: The user's voice command.

[0158] Output: Text data of the voice command.

[0159] Specific actions: The user holds the smartphone close to the microphone and speaks.

[0160] On the device: The voice recognition engine analyzes the voice command and converts it into text data. The converted text data is used to identify the corresponding application and execute the command to launch the application.

[0161] Input: Text data of the voice command.

[0162] Output: Launches a Braille reading app.

[0163] Specific operation: The device's internal voice recognition engine converts the voice into text and searches for and launches the target app.

[0164] Step 2: Capturing Braille

[0165] User: Point the smartphone camera at an object with Braille on it.

[0166] Input: An object with Braille inscribed on it.

[0167] Output: A focused image from the camera.

[0168] Specific actions: The user focuses the camera on the Braille and sees the Braille on the app screen.

[0169] Terminal: The camera captures image data and stores it in memory.

[0170] Input: An image captured by the camera.

[0171] Output: Image data stored in memory.

[0172] Specific operation: The device's camera app takes a photo and stores the data in memory.

[0173] Step 3: Sending image data

[0174] Terminal: Compresses the stored image data and prepares it for transmission to the data processing device (server). Compresses the data and uploads it to the server.

[0175] Input: Image data stored in memory.

[0176] Output: Send compressed image data to the server.

[0177] Specific operation: Data compression and uploading is performed using the communication module inside the terminal.

[0178] Step 4: Analyzing the image data

[0179] Server: Stores the received image data and begins analysis. Performs preprocessing such as noise reduction using an image processing library (e.g., OpenCV).

[0180] Input: Compressed image data.

[0181] Output: Preprocessed image data.

[0182] Specific operation: Image data is stored in the server storage and noise is removed using an image processing library.

[0183] Server: The preprocessed image data is input into the generative AI model, which analyzes the Braille patterns. The generative AI model converts the Braille into character data.

[0184] Input: Preprocessed image data.

[0185] Output: Character data.

[0186] Specific operation: Image data is input into the generative AI model, and Braille analysis and character conversion are performed.

[0187] Step 5: Sending text data

[0188] Server: Sends the generated character data to the terminal.

[0189] Input: Character data.

[0190] Output: Sending character data to the terminal.

[0191] Specific operation: Sends character data to the terminal via the network.

[0192] Step 6: Converting text data into speech

[0193] Terminal: Receives the received text data and passes it to the speech synthesis engine, which converts it into voice data.

[0194] Input: Character data.

[0195] Output: Audio data.

[0196] Specific operation: Converts text data into audio data using Google Text-to-Speech or Apple's VoiceOver function.

[0197] Device: Plays the generated audio data through the speaker.

[0198] Input: Audio data.

[0199] Output: The audio played through the speakers.

[0200] Specific operation: Plays audio data through the device's speaker.

[0201] User: Check the braille content through audio data.

[0202] Input: The audio coming from the device's speaker.

[0203] Output: Information comprehension.

[0204] What happens: The user is near the smartphone and hears a voice notification.

[0205] (Application example 1)

[0206] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0207] Currently, there are limited means for visually impaired people to safely and quickly check security-related information. Even if they can read Braille, they may not be able to understand or communicate it properly. Therefore, there is a need for a system that allows visually impaired people to obtain information accurately and quickly through Braille.

[0208] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0209] In this invention, the server includes a means for pre-processing image data acquired by the camera, such as noise removal, a means including a generation AI for analyzing the image data, identifying Braille, and converting it into text data, and a means for recognizing security-related information based on Braille and notifying it by voice, thereby enabling visually impaired people to accurately and quickly obtain security-related information by voice.

[0210] "Visually impaired persons" refers to people with visual impairments, including those who have difficulty obtaining visual information on their own.

[0211] Braille is a writing system that consists of special symbols that allow visually impaired people to read written information using their sense of touch.

[0212] A "system" refers to a complex of multiple components that interact with each other to achieve a specific function.

[0213] "Voice command" refers to a voice input means for recognizing a user's speech and operating the system based on that instruction.

[0214] "Application" refers to computer software designed to provide a specific function.

[0215] "Camera" refers to an optical device for acquiring image data, and in this context refers specifically to a camera built into a mobile device such as a smartphone.

[0216] "Image data" refers to data obtained by converting visual information acquired by an imaging device such as a camera into digital values.

[0217] A "server" refers to a computer system that provides services such as data processing, storage, and use over a network.

[0218] "Generative AI" refers to algorithms or models that use artificial intelligence techniques to analyze, generate, or transform data.

[0219] "Text data" refers to digital data consisting of characters.

[0220] Converting to "audio" and "reading aloud" refers to the act of converting text data into an audio signal using speech synthesis technology and playing it back through an output device.

[0221] "Security-related information" refers to confidential information such as personal information, authentication codes, and PIN codes, and includes information that must be verified in a secure manner.

[0222] This invention is a system that enables visually impaired people to understand Braille and safely check security-related information. This system uses a smartphone camera and generation AI to allow visually impaired people to understand Braille through audio. Specific embodiments of this system are described below.

[0223] This system consists of a terminal operated by a user, a server that processes data, and an application that runs between them.

[0224] System configuration

[0225] Device:

[0226] It is a smartphone operated by the user, and applications are launched using voice commands. By saying "Launch the Braille app," the voice recognition engine analyzes the voice command and identifies the corresponding application, which then launches the Braille reading app.

[0227] camera:

[0228] The system uses the smartphone's built-in camera: when the user points the camera at an object with Braille, the camera captures the image data and stores it in memory.

[0229] server:

[0230] The acquired image data is sent to the server, where it undergoes preprocessing such as noise removal. The preprocessed image data is then input into a generative AI model, which analyzes the Braille patterns. The recognized Braille characters are converted into text data, which is then sent back to the device.

[0231] Text to speech conversion:

[0232] The received text data is passed to a speech synthesis engine on the device. The speech synthesis engine (for example, gTTS: Google Text-to-Speech) converts the text data into voice data. The generated voice data is played back through the device's speaker.

[0233] Specific examples

[0234] Scenario: Checking the Braille PIN code on your cash card at a bank ATM

[0235] 1. User: Say to the smartphone, "Launch the Braille app."

[0236] 2. Device: The app will launch and display the main screen, with the camera activated and ready to take a photo of an object with Braille.

[0237] 3. User: Point the smartphone camera at the card with the PIN code written on it.

[0238] 4. Terminal: The camera takes a picture of the card.

[0239] 5. Terminal: The acquired image data is compressed and sent to the server.

[0240] 6. Server: Receives the image data and performs preprocessing such as noise removal. Then, the generative AI model analyzes the Braille and generates text data. The text "1234" is generated.

[0241] 7. Server: Sends text data to the device.

[0242] 8. Device: Converts the received text data into speech and reads out "1234."

[0243] 9. User: Listen to the audio and confirm the PIN code.

[0244] This system allows visually impaired people to easily access Braille information, and is particularly important for enabling them to check security-related information safely and quickly.

[0245] Prompt Sentence Examples

[0246] Say "Launch the Braille app." Then point the camera at the Braille. The app will launch and the camera will capture the image. The Braille will be analyzed and the results will be read aloud.

[0247] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0248] Step 1:

[0249] User: Says to the smartphone, "Launch the Braille app." Requests the launch of the application via voice command.

[0250] Input: User's voice command

[0251] Output: Text data of voice commands

[0252] What happens: The smartphone uses a speech recognition engine (e.g., Google Speech Recognition API) to convert the voice command into text and identify the phrase "Launch Braille app."

[0253] Step 2:

[0254] Device: Analyzes voice commands, identifies the corresponding application, and launches it.

[0255] Input: Text data of voice command

[0256] Output: Launching a Braille reading application

[0257] Specific operation: Based on the text data, the smartphone's operating system launches the Braille reading application, the application's main screen is displayed, and the camera becomes available for use.

[0258] Step 3:

[0259] User: Point the smartphone camera at an object with Braille on it.

[0260] Input: An object with braille on it

[0261] Output: Camera image data

[0262] What it does: The user positions the camera in the right position, and the smartphone camera captures an image of the object with Braille on it.

[0263] Step 4:

[0264] Terminal: The camera captures image data and stores it in memory.

[0265] Input: Camera image data

[0266] Output: Saved image data

[0267] How it works: The image sensor captures high-resolution image data and temporarily stores it in the smartphone's memory.

[0268] Step 5:

[0269] Terminal: The stored image data is compressed and sent to the server.

[0270] Input: Saved image data

[0271] Output: Compressed image data uploaded to the server

[0272] Specific operation: The smartphone uses an image compression algorithm (e.g., JPEG compression) to compress the image data and send it to the server over the network.

[0273] Step 6:

[0274] Server: Stores the received image data for analysis and performs preprocessing such as noise removal.

[0275] Input: Compressed image data uploaded to the server

[0276] Output: Preprocessed image data

[0277] What happens: The server decodes the image and applies a noise reduction filter (e.g., median filter) to improve the image quality.

[0278] Step 7:

[0279] Server: Analyzes image data and converts Braille into text data using a generative AI model that identifies Braille.

[0280] Input: Preprocessed image data

[0281] Output: Converted text data

[0282] What it does: Uses a generative AI model (e.g., Tesseract OCR or a custom-trained model) to identify Braille characters from pre-processed images and generate corresponding text data.

[0283] Step 8:

[0284] Server: Sends text data to the device.

[0285] Input: Converted text data

[0286] Output: Text data sent to the terminal

[0287] Specific operation: The server encodes the text data and sends it to the smartphone via the network.

[0288] Step 9:

[0289] Terminal: Converts received text data into speech and reads it to the user.

[0290] Input: Text data sent to the terminal

[0291] Output: Audio data

[0292] Specific operation: The smartphone uses a speech synthesis engine (e.g., gTTS) to convert the text data into audio data and play it back through the speaker.

[0293] Step 10:

[0294] User: Listen to what is read aloud and see what is in braille.

[0295] Input: Audio data

[0296] Output: Security-related information understood

[0297] Specific operation: The user listens to the audio from the smartphone speaker and obtains the information, which allows them to accurately understand the content of the Braille and take the necessary action.

[0298] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0299] The present invention provides a system that helps visually impaired people understand Braille by combining it with an emotion engine that recognizes the user's emotions, making it easier to use and more user-friendly. Specific embodiments of this system are described below.

[0300] System configuration

[0301] This system consists of a smartphone (terminal) operated by the user, a server that processes data, and an application that includes an emotion engine. The application runs on the smartphone and is activated by voice commands. A camera is used to capture an object with Braille printed on it, and the acquired image data is sent to the server. On the server, a generative AI analyzes the Braille from the image data and converts the recognized Braille into text data. Finally, the smartphone converts the text data into speech and reads it to the user. The emotion engine also has the function of recognizing the user's emotions and adjusting the voice output based on those emotions.

[0302] Program processing

[0303] The specific program processing executed by each device will be explained in natural language below.

[0304] Launching the app

[0305] User: Says the voice command to their smartphone: "Launch the Braille app."

[0306] Device: The voice recognition engine analyzes the voice command and identifies the corresponding application. The application launch command is executed, and the Braille reading application is launched.

[0307] Braille capture and transmission

[0308] User: Point the smartphone camera at an object with Braille on it.

[0309] Terminal: The camera acquires image data, stores it in memory, prepares it for transmission to the server, compresses it, and uploads it to the server.

[0310] Image data analysis

[0311] Server: Stores the received image data for analysis. Analysis begins and performs preprocessing such as noise removal. The generation AI inputs the preprocessed image data and analyzes the Braille patterns. The recognized Braille is converted into text data and sent to the device.

[0312] Converting text data into speech

[0313] Device: The device receives the text data and passes it to the speech synthesis engine. The speech synthesis engine converts the text data into voice data, which is then played back from the speaker.

[0314] Emotion engine processing

[0315] On the device: The emotion engine analyzes the user's voice and facial expressions to recognize their current emotional state. For example, it identifies the emotion if the user is anxious.

[0316] On the device: Adjust the tone and speed of the voice output based on the emotions recognized by the emotion engine, providing just-in-time feedback to optimize the voice output for the user.

[0317] Specific examples

[0318] Scenario: Reading Braille on a station sign

[0319] 1. User: Say to the smartphone, "Launch the Braille app."

[0320] 2. Device: The app will launch and the main screen will appear.

[0321] 3. User: Point the camera at the Braille on the station sign.

[0322] 4. Terminal: Operate the camera to capture Braille and obtain image data.

[0323] 5. Terminal: Compresses the image data and sends it to the server.

[0324] 6. Server: Receives the image data and converts the Braille into text using the generation AI. The generated text is, "Platform 1 goes to Tokyo, and platform 2 goes to Shinjuku."

[0325] 7. Server: Sends text data to the device.

[0326] 8. Terminal: Converts the received text data into speech and reads out loud, "Platform 1 goes to Tokyo, and platform 2 goes to Shinjuku."

[0327] 9. On the device: The emotion engine analyzes the user's emotions and adjusts the tone and speed of the voice based on the recognized emotion. For example, if the user is anxious, the voice will be played in a calming tone to give a sense of security.

[0328] 10. User: Listen to the audio and understand the instructions.

[0329] Combining emotion engines in this way will enable visually impaired people to obtain braille information more comfortably, improving the user experience. This is expected to further reduce barriers to obtaining information in daily life and expand the range of activities available to visually impaired people.

[0330] The processing flow will be explained below.

[0331] Step 1:

[0332] User: Says the voice command to their smartphone: "Launch the Braille app."

[0333] A voice recognition engine receives a user's voice command and analyzes the voice data.

[0334] Based on the recognized command, it determines whether to launch a Braille reading application.

[0335] Step 2:

[0336] Terminal: Execute the application launch command to launch the Braille reading app.

[0337] The app will start and the main screen will be displayed.

[0338] The camera control module is ready and waiting to capture braille.

[0339] Step 3:

[0340] User: Point the smartphone camera at an object with Braille on it.

[0341] Refer to the on-screen guidelines and audio guide to point the camera in the correct position.

[0342] Give a voice command for the camera button to prepare for capturing Braille.

[0343] Step 4:

[0344] Terminal: Braille image data is acquired using a camera.

[0345] The camera captures the image in the frame and stores the image data in memory.

[0346] The stored image data undergoes a compression process to prepare it for transmission to the server.

[0347] Step 5:

[0348] Terminal: Sends compressed image data to a server via the Internet.

[0349] Upload image data to the server's API endpoint.

[0350] Monitor the transmission status to ensure that the data was sent successfully.

[0351] Step 6:

[0352] Server: Initiates the process of analyzing the received image data.

[0353] The reception of the data is confirmed and the image data is stored in the database.

[0354] Generative AI performs pre-processing, noise removal, and image enhancement.

[0355] Step 7:

[0356] Server: Preprocessed image data is passed to the generation AI, which then identifies the Braille characters.

[0357] A generative AI model analyzes the image data and recognizes Braille patterns.

[0358] The recognized braille pattern is converted into text data.

[0359] Step 8:

[0360] Server: Sends the generated text data to the device.

[0361] Converts text data into an optimal format and sends it to the device's API endpoint.

[0362] Step 9:

[0363] Terminal: Converts the received text data into audio data and prepares it to be read to the user.

[0364] Check the integrity of the text data and pass it to the speech synthesis engine.

[0365] A speech synthesis engine converts the text data into speech data.

[0366] Step 10:

[0367] On the device: The emotion engine captures voice and facial expression data to analyze the user's emotions.

[0368] It uses voice input and a camera to collect user emotional data.

[0369] The emotion engine analyzes the collected data to identify the current emotional state.

[0370] Step 11:

[0371] Device: Adjusts voice output based on emotion data obtained from the emotion engine.

[0372] If the user seems anxious, calm the voice tone.

[0373] If the user is excited, adjust the speed of the speech output.

[0374] Step 12:

[0375] Device: The generated audio data is played through the device's speaker.

[0376] The audio data is played and read to the user.

[0377] Step 13:

[0378] User: Check the braille content through audio from the device.

[0379] Listen to the audio and get the information you need.

[0380] Example 2

[0381] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0382] Existing technologies for visually impaired people to understand Braille can sometimes make it difficult to read the content of Braille efficiently and accurately. Furthermore, they are unable to provide speech output that takes into account the user's emotional state, which can lead to a decline in the comprehension of information and the quality of the user experience. There is a need to provide a system that can solve this issue and enable visually impaired people to more comfortably acquire and understand Braille information.

[0383] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0384] In this invention, the server includes a means including a generative AI model that analyzes image data and identifies Braille, a means for converting the Braille identified by the generative AI model into text information, and a means for transmitting the converted text information to a terminal. This allows visually impaired people to capture images of Braille with a camera and hear its contents audibly. The terminal also includes a means for analyzing the user's emotional state using an emotion recognition engine and a means for adjusting the tone and speed of the voice based on the analyzed emotional state. This provides optimal audio output for the user, improving the comprehension of information and the quality of the user experience.

[0385] A "voice command" is a method by which a user issues instructions to a system using voice, which is processed using voice recognition technology.

[0386] "Image capture device" refers to a device, such as a camera, that captures visual information as digital image data.

[0387] "Data processing device" refers to a computer system such as a server that analyzes acquired data and converts it into necessary information.

[0388] A "generative AI model" refers to a model that uses artificial intelligence techniques to analyze and recognize data, in this case, to identify Braille.

[0389] "Text information" refers to data that has been converted from coded data such as Braille into a text format that is easily understandable by humans.

[0390] "Terminal" refers to a device that is directly operated by a user, such as a smartphone or tablet.

[0391] A "voice synthesis engine" refers to a technology that converts text data into voice data and provides it to the user as voice information.

[0392] An "emotion recognition engine" refers to a system that uses artificial intelligence technology to analyze and identify a user's emotional state from their voice and facial expressions.

[0393] "Tone" refers to the pitch and intensity of a voice, and is an element that adjusts the voice output according to the user's emotional state.

[0394] "Speed" refers to the speed at which the audio is played back, and is a factor that speeds up or slows down the audio output depending on the user's emotional state.

[0395] This system, which helps visually impaired people understand Braille, is composed as follows: The system consists of a terminal operated by the user, a server that processes data, and an application that includes an emotion recognition engine. The application runs on the terminal and is activated by accepting voice commands.

[0396] When a user issues a voice command to their smartphone (hereinafter referred to as the "device"), such as "Launch the Braille app," a speech recognition engine (e.g., Google Speech-to-Text) analyzes the voice and identifies the application to be launched. Once the application is launched, the main screen is displayed.

[0397] Next, the user points the device's camera at the object with the Braille. The camera module (e.g., Android's Camera API) acquires the image data and temporarily stores it in memory. The stored image data is compressed using a compression algorithm (e.g., JPEG compression) and sent to the server via a network module (e.g., the Retrofit library). The progress of the transmission is displayed on the device.

[0398] The server stores the received image data in memory and preprocesses it using a noise reduction algorithm (e.g., Gaussian filter). A generative AI model (e.g., TensorFlow's OCR engine) analyzes the preprocessed image data and recognizes Braille patterns with high accuracy. The recognized Braille information is then converted into text and sent to the device.

[0399] The device receives the text information and inputs it into a speech synthesis engine (e.g., Google Text-to-Speech). The speech synthesis engine converts the text information into audio data, which is then read aloud to the user through the speaker.

[0400] Furthermore, an emotion recognition engine (e.g., Affectiva SDK) analyzes the user's voice and facial expressions to identify their current emotional state. For example, if the user is anxious, it will identify that emotion. Based on this recognition result, the tone and speed of the voice output are adjusted. For example, an anxious user will be provided with a calming voice tone.

[0401] As a concrete example, consider a scenario in which a user is reading a Braille sign at a train station. When the user says to their smartphone, "Launch the Braille app," the application launches and the main screen appears. Next, the user points the camera at the Braille on the station sign, and the device's camera captures image data and sends it to the server. The server analyzes the Braille and generates text information such as "Platform 1 goes to Tokyo, Platform 2 goes to Shinjuku," which is then sent to the device. The device converts this information into audio data and reads it out loud as "Platform 1 goes to Tokyo, Platform 2 goes to Shinjuku." At this time, an emotion recognition engine analyzes the user's emotional state, and anxious users are provided with information in a calming tone.

[0402] In this way, this system is designed to enable visually impaired people to obtain Braille information comfortably and easily.

[0403] Prompt Sentence Examples

[0404] Below are some examples of prompt sentences used in this system.

[0405] "Launch the Braille app."

[0406] "Point the camera at the Braille and capture the image."

[0407] "Convert Braille to text"

[0408] "Read it aloud"

[0409] "Analyze emotions and adjust tone of voice"

[0410] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0411] Step 1:

[0412] The user issues a voice command to the smartphone, saying, "Launch the Braille app." The input is the user's voice, and the output is the voice data received by the smartphone.

[0413] Step 2:

[0414] The device uses a speech recognition engine (e.g., Google Speech-to-Text) to analyze the voice data and identify the command to "launch a Braille app." The input is the voice data, and the output is the analyzed text data and a launch command. Based on this, the application manager launches the Braille reading app, and the main screen is displayed.

[0415] Step 3:

[0416] The user points the smartphone camera at an object with Braille and determines its position. The input is the user's camera operation, and the output is the adjustment of the camera angle and focus.

[0417] Step 4:

[0418] The device acquires image data using a camera module (e.g., Android's Camera API) and stores the data in temporary memory. The input is the image captured by the camera, and the output is the stored image data.

[0419] Step 5:

[0420] The image data acquired by the terminal is compressed using a compression algorithm (e.g., JPEG compression). The input is uncompressed image data, and the output is compressed image data.

[0421] Step 6:

[0422] The device sends compressed image data to the server using a network module (e.g., Retrofit library). The input is the compressed image data, and the output is the upload to the server.

[0423] Step 7:

[0424] The server stores the received image data in memory. The input is the uploaded image data, and the output is the stored data in the server.

[0425] Step 8:

[0426] The server pre-processes the images with a noise reduction algorithm (e.g., Gaussian filter). The input is the stored image data, and the output is the pre-processed image data.

[0427] Step 9:

[0428] The server inputs the preprocessed image data into a generative AI model (e.g., TensorFlow's OCR engine) to analyze the Braille patterns. The input is the preprocessed image data, and the output is the Braille recognition data.

[0429] Step 10:

[0430] The server converts the recognized Braille information into text information and sends the text information to the terminal. The input is the Braille recognition data, and the output is the transmission of text information to the terminal.

[0431] Step 11:

[0432] The text information received by the device from the server is input into a speech synthesis engine (e.g., Google Text-to-Speech) and converted into voice data. The input is text information, and the output is the generated voice data.

[0433] Step 12:

[0434] The device plays the generated audio data from a speaker and reads the braille content aloud to the user. The input is audio data, and the output is audio playback.

[0435] Step 13:

[0436] The device uses an emotion recognition engine (e.g., Affectiva SDK) to analyze the user's voice and facial expressions to determine their current emotional state. The input is the user's facial expression and voice data, and the output is the analyzed emotional state.

[0437] Step 14:

[0438] The device adjusts the tone and rate of the audio output based on the recognized emotional state. The input is the analyzed emotional state, and the output is the adjusted audio output. For example, if the user appears anxious, audio will be played in a calming tone.

[0439] (Application example 2)

[0440] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0441] Conventional systems for visually impaired people to understand Braille have the problem of limited user experience because they do not take into account the user's emotions when reading Braille. Also, because the voice output is uniform, there is an issue that it is not possible to respond appropriately according to the user's mental state or emotions. As a result, ease of use for visually impaired people has not been sufficiently considered.

[0442] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for launching an application using a voice command, means for acquiring Braille image data using a camera, means for transmitting the acquired image data to the server, means including a generation AI on the server side for analyzing the image data and identifying Braille, means for converting the Braille identified by the generation AI into text data, means for transmitting the converted text data to the terminal, means for converting the received text data on the terminal side into voice and reading it aloud, and means for analyzing the user's emotions using an emotion engine and adjusting the voice output. This makes it possible for visually impaired people to acquire Braille information more comfortably and provides a user-friendly system.

[0443] "Visually impaired people" refers to people whose visual function is impaired and who have difficulty obtaining information through their vision.

[0444] Braille is a special arrangement of dots that allows visually impaired people to understand letters using their sense of touch.

[0445] A "system" is a collection of components that enable multiple devices and software to work together to achieve a specific function.

[0446] "Voice command" means an instruction given by a user to operate a device or application by speaking.

[0447] "Application" means a software program that runs on a computer or device to provide a particular function or service.

[0448] A "camera" is an optical device for capturing images or video.

[0449] "Image data" means image information represented in digital form.

[0450] A "server" is a computer system that provides services to other computers over a network.

[0451] "Generative AI" refers to artificial intelligence that uses machine learning techniques to generate specific information from data.

[0452] "Text data" means data that represents character information in a digital format.

[0453] An "emotion engine" is a system that analyzes a user's voice and facial expressions to recognize their current emotional state and generate appropriate feedback based on that.

[0454] "Adjusting the audio output" means changing the tone and rate of the audio depending on the user's emotional state.

[0455] "Noise reduction" refers to the process of removing unnecessary information and noise from image or audio data.

[0456] "Preprocessing" refers to processing to prepare data before the main processing is performed.

[0457] "Speech synthesis engine" refers to a software component that converts text data into speech.

[0458] This invention provides a system for visually impaired people to understand Braille, which is easier to use and more user-friendly by recognizing the user's emotions and adjusting the voice output. This system includes the following processes: launching an application via voice command, acquiring Braille image data using a camera, transmitting the image data to a server, identifying the Braille using AI generated on the server, converting it to text data, transmitting the text data to a terminal, converting it to voice and reading it out loud, and analyzing the user's emotions and adjusting the voice output using an emotion engine. Specific embodiments of this system are described below.

[0459] Hardware and software used

[0460] Hardware: Smartphone (with camera)

[0461] Software: Android / iOS app, server (cloud server), generative AI (generative AI model), speech synthesis engine (speech synthesis software), emotion engine (emotion analysis software)

[0462] System configuration

[0463] Launching the application

[0464] The user issues a voice command to their smartphone, saying, "Launch the Braille app." The device's voice recognition engine analyzes this voice command and identifies the corresponding application. The launch command for the identified application is executed, and the Braille reading app launches.

[0465] Braille capture and transmission

[0466] When a user points the smartphone camera at an object with Braille, the camera captures the image data and stores it in memory, where it is compressed and uploaded to the server.

[0467] Image data analysis

[0468] The server analyzes the received image data and performs preprocessing such as noise removal. The generation AI then inputs the preprocessed image data and analyzes the Braille patterns. The Braille recognized as a result of the analysis is converted into text data and sent to the device.

[0469] Converting text data into speech

[0470] The device receives the text data and passes it to a speech synthesis engine. The speech synthesis engine converts the text data into audio data, which is then played back through the device's speaker. This allows the user to hear the Braille content aloud.

[0471] Emotion engine processing

[0472] The device's emotion engine analyzes the user's voice and facial expressions to recognize their current emotional state. For example, if the user is feeling anxious, it will identify that emotion. Based on the emotion recognized by the emotion engine, the tone and speed of the voice output are adjusted in real time, providing the most appropriate voice output for the user.

[0473] Examples of concrete examples and prompts

[0474] Scenario: Reading Braille on a station sign

[0475] 1. User: Says to the smartphone, "Launch the Braille app."

[0476] 2. Device: The app will launch and the main screen will appear.

[0477] 3. User: Point the camera at the Braille on the station sign.

[0478] 4. Terminal: Operate the camera to capture Braille and obtain image data.

[0479] 5. Terminal: Compresses the image data and sends it to the server.

[0480] 6. Server: Receives the image data and converts the Braille into text using a generation AI. This generates text data such as "Platform 1 goes to Tokyo, and platform 2 goes to Shinjuku."

[0481] 7. Server: Sends text data to the device.

[0482] 8. Terminal: Converts the received text data into speech and reads out loud, "Platform 1 goes to Tokyo, and platform 2 goes to Shinjuku."

[0483] 9. On the device: The emotion engine analyzes the user's emotions and adjusts the tone and speed of the voice based on the recognized emotion. For example, if the user is anxious, the voice will be played in a calming tone to give a sense of security.

[0484] 10. User: Listen to the audio and understand the instructions.

[0485] Prompt Sentence Examples

[0486] Speech prompt: "Launch the Braille app"

[0487] Generative AI prompt: "Convert this Braille image into text."

[0488] Text-to-speech prompt: "Convert this text to speech: 'This product is coffee'"

[0489] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0490] Step 1:

[0491] The user issues a voice command to their smartphone saying, "Launch the Braille app." The device's voice recognition engine analyzes this voice command, identifies the corresponding application, and executes the launch command. This launches the Braille reading app. The input is the user's voice command, and the output is the launch of the application.

[0492] Step 2:

[0493] When a user points a smartphone camera at an object with Braille, the device's camera captures image data and stores it in memory. The input is the camera image, and the output is Braille image data.

[0494] Step 3:

[0495] The device compresses the acquired image data and prepares it for uploading to the server. The data is sent to the server and analysis begins. The input is Braille image data, and the output is compressed image data sent to the server.

[0496] Step 4:

[0497] The server analyzes the received image data and performs preprocessing using generative AI to identify Braille, including noise removal. The input is the image data sent to the server, and the output is the preprocessed image data.

[0498] Step 5:

[0499] The server passes the preprocessed image data to the generation AI, which analyzes the Braille patterns. The Braille recognized as a result of this analysis is converted into text data. The input is preprocessed image data, and the output is text data.

[0500] Step 6:

[0501] The server sends the converted text data to the terminal. The input is the text data generated by the server, and the output is the text data sent to the terminal.

[0502] Step 7:

[0503] The device passes the received text data to the speech synthesis engine. The speech synthesis engine converts this text data into voice data, and the generated voice data is played back from the device's speaker. The input is text data, and the output is voice data.

[0504] Step 8:

[0505] The device's emotion engine analyzes the user's voice and facial expressions to recognize their current emotional state. For example, if the user looks anxious, it identifies that emotion. The input is the user's voice and facial expression data, and the output is the recognized emotion.

[0506] Step 9:

[0507] The tone and speed of the voice output are adjusted in real time based on the emotion recognized by the emotion engine, providing the optimal voice output for the user. The input is the recognized emotion, and the output is the adjusted voice output.

[0508] These steps will enable visually impaired people to obtain braille information more comfortably and create a user-friendly system.

[0509] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0510] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0511] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0512] [Second embodiment]

[0513] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0514] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0515] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0516] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0517] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0518] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0519] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0520] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0521] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0522] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0523] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0524] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0525] The present invention is a system for visually impaired people to understand Braille, which uses a smartphone camera and generation AI to read Braille aloud. A specific embodiment of this system is described below.

[0526] System configuration

[0527] This system consists of a smartphone (terminal) operated by the user, a server that processes data, and an application that runs between them. The application runs on the smartphone and is activated by voice commands. A camera is used to capture an object with Braille printed on it, and the acquired image data is sent to the server. On the server, a generative AI analyzes the Braille from the image data and converts the recognized Braille into text data. Finally, the smartphone converts the text data into speech and reads it out loud to the user.

[0528] Program processing

[0529] The specific program processing executed by each device will be explained in natural language below.

[0530] Launching the app

[0531] User: Says the voice command to their smartphone: "Launch the Braille app."

[0532] Device: The voice recognition engine analyzes the voice command and identifies the corresponding application. The application launch command is executed, and the Braille reading application is launched.

[0533] Braille capture and transmission

[0534] User: Point the smartphone camera at an object with Braille on it.

[0535] Terminal: The camera acquires image data, stores it in memory, prepares it for transmission to the server, compresses it, and uploads it to the server.

[0536] Image data analysis

[0537] Server: Stores the received image data for analysis. Analysis begins and performs preprocessing such as noise removal. The generation AI inputs the preprocessed image data and analyzes the Braille patterns. The recognized Braille is converted into text data and sent to the device.

[0538] Converting text data into speech

[0539] Device: The device receives the text data and passes it to the speech synthesis engine. The speech synthesis engine converts the text data into voice data, which is then played back from the speaker.

[0540] User: Checks the content of Braille through audio data. This allows visually impaired people to obtain the content through audio without having to read Braille.

[0541] Specific examples

[0542] Scenario: Reading Braille on a station sign

[0543] 1. User: Say to the smartphone, "Launch the Braille app."

[0544] 2. Device: The app will launch and the main screen will appear.

[0545] 3. User: Point the camera at the Braille on the station sign.

[0546] 4. Terminal: Operate the camera to capture Braille and obtain image data.

[0547] 5. Terminal: Compresses the image data and sends it to the server.

[0548] 6. Server: Receives the image data and converts the Braille into text using the generation AI. The generated text is, "Platform 1 goes to Tokyo, and platform 2 goes to Shinjuku."

[0549] 7. Server: Sends text data to the device.

[0550] 8. Terminal: Converts the received text data into speech and reads out loud, "Platform 1 goes to Tokyo, and platform 2 goes to Shinjuku."

[0551] 9. User: Listen to the audio and understand the instructions.

[0552] Such an embodiment of the system will enable visually impaired people to easily obtain Braille information, which is expected to reduce barriers to obtaining information in daily life and expand the range of activities available to visually impaired people.

[0553] The processing flow will be explained below.

[0554] Step 1:

[0555] User: Says the voice command to their smartphone: "Launch the Braille app."

[0556] A voice recognition engine receives a user's voice command and analyzes the voice data.

[0557] Based on the recognized command, it determines whether to launch a Braille reading application.

[0558] Step 2:

[0559] Terminal: Execute the application launch command to launch the Braille reading app.

[0560] The app will start and the main screen will be displayed.

[0561] The camera control module is ready and waiting to capture braille.

[0562] Step 3:

[0563] User: Point the smartphone camera at an object with Braille on it.

[0564] Refer to the on-screen guidelines and audio guide to point the camera in the correct position.

[0565] Give a voice command for the camera button to prepare for capturing Braille.

[0566] Step 4:

[0567] Terminal: Braille image data is acquired using a camera.

[0568] The camera captures the image in the frame and stores the image data in memory.

[0569] The stored image data undergoes a compression process to prepare it for transmission to the server.

[0570] Step 5:

[0571] Terminal: Sends compressed image data to a server via the Internet.

[0572] Upload image data to the server's API endpoint.

[0573] Monitor the transmission status to ensure that the data was sent successfully.

[0574] Step 6:

[0575] Server: Initiates the process of analyzing the received image data.

[0576] The reception of the data is confirmed and the image data is stored in the database.

[0577] Generative AI performs pre-processing, noise removal, and image enhancement.

[0578] Step 7:

[0579] Server: Preprocessed image data is passed to the generation AI, which then identifies the Braille characters.

[0580] A generative AI model analyzes the image data and recognizes Braille patterns.

[0581] The recognized braille pattern is converted into text data.

[0582] Step 8:

[0583] Server: Sends the generated text data to the device.

[0584] Converts text data into an optimal format and sends it to the device's API endpoint.

[0585] Step 9:

[0586] Terminal: Converts the received text data into audio data and prepares it to be read to the user.

[0587] Check the integrity of the text data and pass it to the speech synthesis engine.

[0588] A speech synthesis engine converts the text data into speech data.

[0589] Step 10:

[0590] Device: The generated audio data is played through the device's speaker.

[0591] The audio data is played and read to the user.

[0592] Step 11:

[0593] User: Check the braille content through audio from the device.

[0594] Listen to the audio and get the information you need.

[0595] Example 1

[0596] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0597] It is extremely difficult for visually impaired people to understand Braille in their daily lives. In particular, there is a lack of means to quickly and accurately understand the Braille on information and guide signs provided in public places, which poses a major challenge for visually impaired people, as it takes a lot of time and effort for them to obtain the information they need.

[0598] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0599] In this invention, the server includes a means for processing image data acquired by an image acquisition device and identifying Braille, a means for converting Braille into character data using a generative AI model, and a means for transmitting the converted character data to a terminal, thereby enabling visually impaired people to efficiently and quickly understand Braille by voice using a terminal such as a smartphone.

[0600] "Visually impaired persons" refer to people who have impaired vision and are unable to recognize letters or figures using normal visual methods.

[0601] "Braille" refers to a writing system made up of raised and recessed patterns that is used by visually impaired people to understand written information using their sense of touch.

[0602] A "system" refers to a set of mechanisms in which multiple devices and software work together.

[0603] "Voice command" refers to a means by which a user gives instructions to electronic devices or software using voice.

[0604] "Application" means a software program designed to provide a particular function or service.

[0605] "Image capture device" refers to equipment that uses a device such as a camera to capture visual information as a digital image.

[0606] "Data processing device" refers to a computer or server system for storing and processing acquired data.

[0607] "Analysis" refers to the process of using algorithms and programs to decipher acquired data and extract meaningful information.

[0608] A "generative AI model" refers to an artificial intelligence model that is trained to perform a specific task using techniques such as machine learning and deep learning.

[0609] "Character data" refers to character information represented in digital form.

[0610] "Terminal" refers to an electronic device that can be directly operated by a user, such as a smartphone or tablet.

[0611] "Means for converting into audio" refers to the technology or software for converting text data into audio data and playing it back through an audio output device.

[0612] The present invention is a system for visually impaired people to understand Braille. An application is started using a voice command, Braille image data acquired by an image acquisition device is sent to a data processing device, and the Braille is converted into character data using a generative AI model, which is then read aloud. Specific embodiments of this system are described below.

[0613] System configuration

[0614] This system consists of a terminal operated by the user, a server that processes data, and an application that runs between them. The application runs on the terminal and is activated by voice commands. An image capture device (e.g., a smartphone camera) is used to capture an object with Braille printed on it, and the captured image data is sent to the server. On the server, a generative AI model analyzes the Braille from the image data and converts the recognized Braille into text data. Finally, the terminal converts the text data into audio and reads it to the user.

[0615] Application Behavior

[0616] Launching the app

[0617] The user issues a voice command to the device, saying, "Launch the Braille app." This voice command is analyzed by the device's voice recognition engine. The voice recognition engine uses technology such as Google Assistant or Siri to identify the corresponding application. The application launch command is executed, and the Braille reading app launches.

[0618] Braille capture and transmission

[0619] The user points the device's camera at an object with Braille. The camera captures image data, which is then stored in the device's memory. The stored image data is then compressed and uploaded to the server.

[0620] Image data analysis

[0621] The server stores the received image data. It then begins analyzing it and performs preprocessing such as noise removal using an image processing library such as OpenCV. The generative AI model uses the preprocessed image data as input and analyzes the Braille patterns. The recognized Braille characters are converted into character data and sent to the device.

[0622] Converting text data into speech

[0623] The device receives the text data and passes it to a speech synthesis engine such as Google Text-to-Speech or Apple's VoiceOver function. The speech synthesis engine converts the text data into audio data, which is then played back through the device's speaker. The user can then confirm the braille content through the audio data.

[0624] Specific examples

[0625] Scenario: Reading Braille on a station sign

[0626] 1. User: Say to the smartphone, "Launch the Braille app."

[0627] 2. Device: The app will launch and the main screen will appear.

[0628] 3. User: Point the camera at the Braille on the station sign.

[0629] 4. Terminal: Operate the camera to capture Braille and obtain image data.

[0630] 5. Terminal: Compresses the image data and sends it to the server.

[0631] 6. Server: Receives the image data and converts the Braille into text using a generative AI model. This generates text such as "Platform 1 goes to Tokyo, and platform 2 goes to Shinjuku."

[0632] 7. Server: Sends character data to the terminal.

[0633] 8. Terminal: Converts the received text data into speech and reads out loud, "Platform 1 goes to Tokyo, and platform 2 goes to Shinjuku."

[0634] 9. User: Listen to the audio and understand the instructions.

[0635] Examples of prompt statements

[0636] Using the voice command "Launch the Braille app" as input, the prompt sentence can be designed as follows:

[0637] "Just tell your smartphone to 'launch the Braille app' and point the camera at the Braille. The app will then capture the image and send it to the server. On the server side, a generative AI model will convert the image data from Braille to text data. Finally, the text data will be communicated to the user as an audio alert."

[0638] By inputting this prompt into the generative AI model, the entire process of reading Braille can be explained to visually impaired people, allowing them to understand the specific steps to take.

[0639] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0640] Step 1: Launch the app

[0641] User: Says the voice command to their smartphone: "Launch the Braille app."

[0642] Input: The user's voice command.

[0643] Output: Text data of the voice command.

[0644] Specific actions: The user holds the smartphone close to the microphone and speaks.

[0645] On the device: The voice recognition engine analyzes the voice command and converts it into text data. The converted text data is used to identify the corresponding application and execute the command to launch the application.

[0646] Input: Text data of the voice command.

[0647] Output: Launches a Braille reading app.

[0648] Specific operation: The device's internal voice recognition engine converts the voice into text and searches for and launches the target app.

[0649] Step 2: Capturing Braille

[0650] User: Point the smartphone camera at an object with Braille on it.

[0651] Input: An object with Braille inscribed on it.

[0652] Output: A focused image from the camera.

[0653] Specific actions: The user focuses the camera on the Braille and sees the Braille on the app screen.

[0654] Terminal: The camera captures image data and stores it in memory.

[0655] Input: An image captured by the camera.

[0656] Output: Image data stored in memory.

[0657] Specific operation: The device's camera app takes a photo and stores the data in memory.

[0658] Step 3: Sending image data

[0659] Terminal: Compresses the stored image data and prepares it for transmission to the data processing device (server). Compresses the data and uploads it to the server.

[0660] Input: Image data stored in memory.

[0661] Output: Send compressed image data to the server.

[0662] Specific operation: Data compression and uploading is performed using the communication module inside the terminal.

[0663] Step 4: Analyzing the image data

[0664] Server: Stores the received image data and begins analysis. Performs preprocessing such as noise reduction using an image processing library (e.g., OpenCV).

[0665] Input: Compressed image data.

[0666] Output: Preprocessed image data.

[0667] Specific operation: Image data is stored in the server storage and noise is removed using an image processing library.

[0668] Server: The preprocessed image data is input into the generative AI model, which analyzes the Braille patterns. The generative AI model converts the Braille into character data.

[0669] Input: Preprocessed image data.

[0670] Output: Character data.

[0671] Specific operation: Image data is input into the generative AI model, and Braille analysis and character conversion are performed.

[0672] Step 5: Sending text data

[0673] Server: Sends the generated character data to the terminal.

[0674] Input: Character data.

[0675] Output: Sending character data to the terminal.

[0676] Specific operation: Sends character data to the terminal via the network.

[0677] Step 6: Converting text data into speech

[0678] Terminal: Receives the received text data and passes it to the speech synthesis engine, which converts it into voice data.

[0679] Input: Character data.

[0680] Output: Audio data.

[0681] Specific operation: Converts text data into audio data using Google Text-to-Speech or Apple's VoiceOver function.

[0682] Device: Plays the generated audio data through the speaker.

[0683] Input: Audio data.

[0684] Output: The audio played through the speakers.

[0685] Specific operation: Plays audio data through the device's speaker.

[0686] User: Check the braille content through audio data.

[0687] Input: The audio coming from the device's speaker.

[0688] Output: Information comprehension.

[0689] What happens: The user is near the smartphone and hears a voice notification.

[0690] (Application example 1)

[0691] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0692] Currently, there are limited means for visually impaired people to safely and quickly check security-related information. Even if they can read Braille, they may not be able to understand or communicate it properly. Therefore, there is a need for a system that allows visually impaired people to obtain information accurately and quickly through Braille.

[0693] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0694] In this invention, the server includes a means for pre-processing image data acquired by the camera, such as noise removal, a means including a generation AI for analyzing the image data, identifying Braille, and converting it into text data, and a means for recognizing security-related information based on Braille and notifying it by voice, thereby enabling visually impaired people to accurately and quickly obtain security-related information by voice.

[0695] "Visually impaired persons" refers to people with visual impairments, including those who have difficulty obtaining visual information on their own.

[0696] Braille is a writing system that consists of special symbols that allow visually impaired people to read written information using their sense of touch.

[0697] A "system" refers to a complex of multiple components that interact with each other to achieve a specific function.

[0698] "Voice command" refers to a voice input means for recognizing a user's speech and operating the system based on that instruction.

[0699] "Application" refers to computer software designed to provide a specific function.

[0700] "Camera" refers to an optical device for acquiring image data, and in this context refers specifically to a camera built into a mobile device such as a smartphone.

[0701] "Image data" refers to data obtained by converting visual information acquired by an imaging device such as a camera into digital values.

[0702] A "server" refers to a computer system that provides services such as data processing, storage, and use over a network.

[0703] "Generative AI" refers to algorithms or models that use artificial intelligence techniques to analyze, generate, or transform data.

[0704] "Text data" refers to digital data consisting of characters.

[0705] Converting to "audio" and "reading aloud" refers to the act of converting text data into an audio signal using speech synthesis technology and playing it back through an output device.

[0706] "Security-related information" refers to confidential information such as personal information, authentication codes, and PIN codes, and includes information that must be verified in a secure manner.

[0707] This invention is a system that enables visually impaired people to understand Braille and safely check security-related information. This system uses a smartphone camera and generation AI to allow visually impaired people to understand Braille through audio. Specific embodiments of this system are described below.

[0708] This system consists of a terminal operated by a user, a server that processes data, and an application that runs between them.

[0709] System configuration

[0710] Device:

[0711] It is a smartphone operated by the user, and applications are launched using voice commands. By saying "Launch the Braille app," the voice recognition engine analyzes the voice command and identifies the corresponding application, which then launches the Braille reading app.

[0712] camera:

[0713] The system uses the smartphone's built-in camera: when the user points the camera at an object with Braille, the camera captures the image data and stores it in memory.

[0714] server:

[0715] The acquired image data is sent to the server, where it undergoes preprocessing such as noise removal. The preprocessed image data is then input into a generative AI model, which analyzes the Braille patterns. The recognized Braille characters are converted into text data, which is then sent back to the device.

[0716] Text to speech conversion:

[0717] The received text data is passed to a speech synthesis engine on the device. The speech synthesis engine (for example, gTTS: Google Text-to-Speech) converts the text data into voice data. The generated voice data is played back through the device's speaker.

[0718] Specific examples

[0719] Scenario: Checking the Braille PIN code on your cash card at a bank ATM

[0720] 1. User: Say to the smartphone, "Launch the Braille app."

[0721] 2. Device: The app will launch and display the main screen, with the camera activated and ready to take a photo of an object with Braille.

[0722] 3. User: Point the smartphone camera at the card with the PIN code written on it.

[0723] 4. Terminal: The camera takes a picture of the card.

[0724] 5. Terminal: The acquired image data is compressed and sent to the server.

[0725] 6. Server: Receives the image data and performs preprocessing such as noise removal. Then, the generative AI model analyzes the Braille and generates text data. The text "1234" is generated.

[0726] 7. Server: Sends text data to the device.

[0727] 8. Device: Converts the received text data into speech and reads out "1234."

[0728] 9. User: Listen to the audio and confirm the PIN code.

[0729] This system allows visually impaired people to easily access Braille information, and is particularly important for enabling them to check security-related information safely and quickly.

[0730] Prompt Sentence Examples

[0731] Say "Launch the Braille app." Then point the camera at the Braille. The app will launch and the camera will capture the image. The Braille will be analyzed and the results will be read aloud.

[0732] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0733] Step 1:

[0734] User: Says to the smartphone, "Launch the Braille app." Requests the launch of the application via voice command.

[0735] Input: User's voice command

[0736] Output: Text data of voice commands

[0737] What happens: The smartphone uses a speech recognition engine (e.g., Google Speech Recognition API) to convert the voice command into text and identify the phrase "Launch Braille app."

[0738] Step 2:

[0739] Device: Analyzes voice commands, identifies the corresponding application, and launches it.

[0740] Input: Text data of voice command

[0741] Output: Launching a Braille reading application

[0742] Specific operation: Based on the text data, the smartphone's operating system launches the Braille reading application, the application's main screen is displayed, and the camera becomes available for use.

[0743] Step 3:

[0744] User: Point the smartphone camera at an object with Braille on it.

[0745] Input: An object with braille on it

[0746] Output: Camera image data

[0747] What it does: The user positions the camera in the right position, and the smartphone camera captures an image of the object with Braille on it.

[0748] Step 4:

[0749] Terminal: The camera captures image data and stores it in memory.

[0750] Input: Camera image data

[0751] Output: Saved image data

[0752] How it works: The image sensor captures high-resolution image data and temporarily stores it in the smartphone's memory.

[0753] Step 5:

[0754] Terminal: The stored image data is compressed and sent to the server.

[0755] Input: Saved image data

[0756] Output: Compressed image data uploaded to the server

[0757] Specific operation: The smartphone uses an image compression algorithm (e.g., JPEG compression) to compress the image data and send it to the server over the network.

[0758] Step 6:

[0759] Server: Stores the received image data for analysis and performs preprocessing such as noise removal.

[0760] Input: Compressed image data uploaded to the server

[0761] Output: Preprocessed image data

[0762] What happens: The server decodes the image and applies a noise reduction filter (e.g., median filter) to improve the image quality.

[0763] Step 7:

[0764] Server: Analyzes image data and converts Braille into text data using a generative AI model that identifies Braille.

[0765] Input: Preprocessed image data

[0766] Output: Converted text data

[0767] What it does: Uses a generative AI model (e.g., Tesseract OCR or a custom-trained model) to identify Braille characters from pre-processed images and generate corresponding text data.

[0768] Step 8:

[0769] Server: Sends text data to the device.

[0770] Input: Converted text data

[0771] Output: Text data sent to the terminal

[0772] Specific operation: The server encodes the text data and sends it to the smartphone via the network.

[0773] Step 9:

[0774] Terminal: Converts received text data into speech and reads it to the user.

[0775] Input: Text data sent to the terminal

[0776] Output: Audio data

[0777] Specific operation: The smartphone uses a speech synthesis engine (e.g., gTTS) to convert the text data into audio data and play it back through the speaker.

[0778] Step 10:

[0779] User: Listen to what is read aloud and see what is in braille.

[0780] Input: Audio data

[0781] Output: Security-related information understood

[0782] Specific operation: The user listens to the audio from the smartphone speaker and obtains the information, which allows them to accurately understand the content of the Braille and take the necessary action.

[0783] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0784] The present invention provides a system that helps visually impaired people understand Braille by combining it with an emotion engine that recognizes the user's emotions, making it easier to use and more user-friendly. Specific embodiments of this system are described below.

[0785] System configuration

[0786] This system consists of a smartphone (terminal) operated by the user, a server that processes data, and an application that includes an emotion engine. The application runs on the smartphone and is activated by voice commands. A camera is used to capture an object with Braille printed on it, and the acquired image data is sent to the server. On the server, a generative AI analyzes the Braille from the image data and converts the recognized Braille into text data. Finally, the smartphone converts the text data into speech and reads it to the user. The emotion engine also has the function of recognizing the user's emotions and adjusting the voice output based on those emotions.

[0787] Program processing

[0788] The specific program processing executed by each device will be explained in natural language below.

[0789] Launching the app

[0790] User: Says the voice command to their smartphone: "Launch the Braille app."

[0791] Device: The voice recognition engine analyzes the voice command and identifies the corresponding application. The application launch command is executed, and the Braille reading application is launched.

[0792] Braille capture and transmission

[0793] User: Point the smartphone camera at an object with Braille on it.

[0794] Terminal: The camera acquires image data, stores it in memory, prepares it for transmission to the server, compresses it, and uploads it to the server.

[0795] Image data analysis

[0796] Server: Stores the received image data for analysis. Analysis begins and performs preprocessing such as noise removal. The generation AI inputs the preprocessed image data and analyzes the Braille patterns. The recognized Braille is converted into text data and sent to the device.

[0797] Converting text data into speech

[0798] Device: The device receives the text data and passes it to the speech synthesis engine. The speech synthesis engine converts the text data into voice data, which is then played back from the speaker.

[0799] Emotion engine processing

[0800] On the device: The emotion engine analyzes the user's voice and facial expressions to recognize their current emotional state. For example, it identifies the emotion if the user is anxious.

[0801] On the device: Adjust the tone and speed of the voice output based on the emotions recognized by the emotion engine, providing just-in-time feedback to optimize the voice output for the user.

[0802] Specific examples

[0803] Scenario: Reading Braille on a station sign

[0804] 1. User: Say to the smartphone, "Launch the Braille app."

[0805] 2. Device: The app will launch and the main screen will appear.

[0806] 3. User: Point the camera at the Braille on the station sign.

[0807] 4. Terminal: Operate the camera to capture Braille and obtain image data.

[0808] 5. Terminal: Compresses the image data and sends it to the server.

[0809] 6. Server: Receives the image data and converts the Braille into text using the generation AI. The generated text is, "Platform 1 goes to Tokyo, and platform 2 goes to Shinjuku."

[0810] 7. Server: Sends text data to the device.

[0811] 8. Terminal: Converts the received text data into speech and reads out loud, "Platform 1 goes to Tokyo, and platform 2 goes to Shinjuku."

[0812] 9. On the device: The emotion engine analyzes the user's emotions and adjusts the tone and speed of the voice based on the recognized emotion. For example, if the user is anxious, the voice will be played in a calming tone to give a sense of security.

[0813] 10. User: Listen to the audio and understand the instructions.

[0814] Combining emotion engines in this way will enable visually impaired people to obtain braille information more comfortably, improving the user experience. This is expected to further reduce barriers to obtaining information in daily life and expand the range of activities available to visually impaired people.

[0815] The processing flow will be explained below.

[0816] Step 1:

[0817] User: Says the voice command to their smartphone: "Launch the Braille app."

[0818] A voice recognition engine receives a user's voice command and analyzes the voice data.

[0819] Based on the recognized command, it determines whether to launch a Braille reading application.

[0820] Step 2:

[0821] Terminal: Execute the application launch command to launch the Braille reading app.

[0822] The app will start and the main screen will be displayed.

[0823] The camera control module is ready and waiting to capture braille.

[0824] Step 3:

[0825] User: Point the smartphone camera at an object with Braille on it.

[0826] Refer to the on-screen guidelines and audio guide to point the camera in the correct position.

[0827] Give a voice command for the camera button to prepare for capturing Braille.

[0828] Step 4:

[0829] Terminal: Braille image data is acquired using a camera.

[0830] The camera captures the image in the frame and stores the image data in memory.

[0831] The stored image data undergoes a compression process to prepare it for transmission to the server.

[0832] Step 5:

[0833] Terminal: Sends compressed image data to a server via the Internet.

[0834] Upload image data to the server's API endpoint.

[0835] Monitor the transmission status to ensure that the data was sent successfully.

[0836] Step 6:

[0837] Server: Initiates the process of analyzing the received image data.

[0838] The reception of the data is confirmed and the image data is stored in the database.

[0839] Generative AI performs pre-processing, noise removal, and image enhancement.

[0840] Step 7:

[0841] Server: Preprocessed image data is passed to the generation AI, which then identifies the Braille characters.

[0842] A generative AI model analyzes the image data and recognizes Braille patterns.

[0843] The recognized braille pattern is converted into text data.

[0844] Step 8:

[0845] Server: Sends the generated text data to the device.

[0846] Converts text data into an optimal format and sends it to the device's API endpoint.

[0847] Step 9:

[0848] Terminal: Converts the received text data into audio data and prepares it to be read to the user.

[0849] Check the integrity of the text data and pass it to the speech synthesis engine.

[0850] A speech synthesis engine converts the text data into speech data.

[0851] Step 10:

[0852] On the device: The emotion engine captures voice and facial expression data to analyze the user's emotions.

[0853] It uses voice input and a camera to collect user emotional data.

[0854] The emotion engine analyzes the collected data to identify the current emotional state.

[0855] Step 11:

[0856] Device: Adjusts voice output based on emotion data obtained from the emotion engine.

[0857] If the user seems anxious, calm the voice tone.

[0858] If the user is excited, adjust the speed of the speech output.

[0859] Step 12:

[0860] Device: The generated audio data is played through the device's speaker.

[0861] The audio data is played and read to the user.

[0862] Step 13:

[0863] User: Check the braille content through audio from the device.

[0864] Listen to the audio and get the information you need.

[0865] Example 2

[0866] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0867] Existing technologies for visually impaired people to understand Braille can sometimes make it difficult to read the content of Braille efficiently and accurately. Furthermore, they are unable to provide speech output that takes into account the user's emotional state, which can lead to a decline in the comprehension of information and the quality of the user experience. There is a need to provide a system that can solve this issue and enable visually impaired people to more comfortably acquire and understand Braille information.

[0868] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0869] In this invention, the server includes a means including a generative AI model that analyzes image data and identifies Braille, a means for converting the Braille identified by the generative AI model into text information, and a means for transmitting the converted text information to a terminal. This allows visually impaired people to capture images of Braille with a camera and hear its contents audibly. The terminal also includes a means for analyzing the user's emotional state using an emotion recognition engine and a means for adjusting the tone and speed of the voice based on the analyzed emotional state. This provides optimal audio output for the user, improving the comprehension of information and the quality of the user experience.

[0870] A "voice command" is a method by which a user issues instructions to a system using voice, which is processed using voice recognition technology.

[0871] "Image capture device" refers to a device, such as a camera, that captures visual information as digital image data.

[0872] "Data processing device" refers to a computer system such as a server that analyzes acquired data and converts it into necessary information.

[0873] A "generative AI model" refers to a model that uses artificial intelligence techniques to analyze and recognize data, in this case, to identify Braille.

[0874] "Text information" refers to data that has been converted from coded data such as Braille into a text format that is easily understandable by humans.

[0875] "Terminal" refers to a device that is directly operated by a user, such as a smartphone or tablet.

[0876] A "voice synthesis engine" refers to a technology that converts text data into voice data and provides it to the user as voice information.

[0877] An "emotion recognition engine" refers to a system that uses artificial intelligence technology to analyze and identify a user's emotional state from their voice and facial expressions.

[0878] "Tone" refers to the pitch and intensity of a voice, and is an element that adjusts the voice output according to the user's emotional state.

[0879] "Speed" refers to the speed at which the audio is played back, and is a factor that speeds up or slows down the audio output depending on the user's emotional state.

[0880] This system, which helps visually impaired people understand Braille, is composed as follows: The system consists of a terminal operated by the user, a server that processes data, and an application that includes an emotion recognition engine. The application runs on the terminal and is activated by accepting voice commands.

[0881] When a user issues a voice command to their smartphone (hereinafter referred to as the "device"), such as "Launch the Braille app," a speech recognition engine (e.g., Google Speech-to-Text) analyzes the voice and identifies the application to be launched. Once the application is launched, the main screen is displayed.

[0882] Next, the user points the device's camera at the object with the Braille. The camera module (e.g., Android's Camera API) acquires the image data and temporarily stores it in memory. The stored image data is compressed using a compression algorithm (e.g., JPEG compression) and sent to the server via a network module (e.g., the Retrofit library). The progress of the transmission is displayed on the device.

[0883] The server stores the received image data in memory and preprocesses it using a noise reduction algorithm (e.g., Gaussian filter). A generative AI model (e.g., TensorFlow's OCR engine) analyzes the preprocessed image data and recognizes Braille patterns with high accuracy. The recognized Braille information is then converted into text and sent to the device.

[0884] The device receives the text information and inputs it into a speech synthesis engine (e.g., Google Text-to-Speech). The speech synthesis engine converts the text information into audio data, which is then read aloud to the user through the speaker.

[0885] Furthermore, an emotion recognition engine (e.g., Affectiva SDK) analyzes the user's voice and facial expressions to identify their current emotional state. For example, if the user is anxious, it will identify that emotion. Based on this recognition result, the tone and speed of the voice output are adjusted. For example, an anxious user will be provided with a calming voice tone.

[0886] As a concrete example, consider a scenario in which a user is reading a Braille sign at a train station. When the user says to their smartphone, "Launch the Braille app," the application launches and the main screen appears. Next, the user points the camera at the Braille on the station sign, and the device's camera captures image data and sends it to the server. The server analyzes the Braille and generates text information such as "Platform 1 goes to Tokyo, Platform 2 goes to Shinjuku," which is then sent to the device. The device converts this information into audio data and reads it out loud as "Platform 1 goes to Tokyo, Platform 2 goes to Shinjuku." At this time, an emotion recognition engine analyzes the user's emotional state, and anxious users are provided with information in a calming tone.

[0887] In this way, this system is designed to enable visually impaired people to obtain Braille information comfortably and easily.

[0888] Prompt Sentence Examples

[0889] Below are some examples of prompt sentences used in this system.

[0890] "Launch the Braille app."

[0891] "Point the camera at the Braille and capture the image."

[0892] "Convert Braille to text"

[0893] "Read it aloud"

[0894] "Analyze emotions and adjust tone of voice"

[0895] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0896] Step 1:

[0897] The user issues a voice command to the smartphone, saying, "Launch the Braille app." The input is the user's voice, and the output is the voice data received by the smartphone.

[0898] Step 2:

[0899] The device uses a speech recognition engine (e.g., Google Speech-to-Text) to analyze the voice data and identify the command to "launch a Braille app." The input is the voice data, and the output is the analyzed text data and a launch command. Based on this, the application manager launches the Braille reading app, and the main screen is displayed.

[0900] Step 3:

[0901] The user points the smartphone camera at an object with Braille and determines its position. The input is the user's camera operation, and the output is the adjustment of the camera angle and focus.

[0902] Step 4:

[0903] The device acquires image data using a camera module (e.g., Android's Camera API) and stores the data in temporary memory. The input is the image captured by the camera, and the output is the stored image data.

[0904] Step 5:

[0905] The image data acquired by the terminal is compressed using a compression algorithm (e.g., JPEG compression). The input is uncompressed image data, and the output is compressed image data.

[0906] Step 6:

[0907] The device sends compressed image data to the server using a network module (e.g., Retrofit library). The input is the compressed image data, and the output is the upload to the server.

[0908] Step 7:

[0909] The server stores the received image data in memory. The input is the uploaded image data, and the output is the stored data in the server.

[0910] Step 8:

[0911] The server pre-processes the images with a noise reduction algorithm (e.g., Gaussian filter). The input is the stored image data, and the output is the pre-processed image data.

[0912] Step 9:

[0913] The server inputs the preprocessed image data into a generative AI model (e.g., TensorFlow's OCR engine) to analyze the Braille patterns. The input is the preprocessed image data, and the output is the Braille recognition data.

[0914] Step 10:

[0915] The server converts the recognized Braille information into text information and sends the text information to the terminal. The input is the Braille recognition data, and the output is the transmission of text information to the terminal.

[0916] Step 11:

[0917] The text information received by the device from the server is input into a speech synthesis engine (e.g., Google Text-to-Speech) and converted into voice data. The input is text information, and the output is the generated voice data.

[0918] Step 12:

[0919] The device plays the generated audio data from a speaker and reads the braille content aloud to the user. The input is audio data, and the output is audio playback.

[0920] Step 13:

[0921] The device uses an emotion recognition engine (e.g., Affectiva SDK) to analyze the user's voice and facial expressions to determine their current emotional state. The input is the user's facial expression and voice data, and the output is the analyzed emotional state.

[0922] Step 14:

[0923] The device adjusts the tone and rate of the audio output based on the recognized emotional state. The input is the analyzed emotional state, and the output is the adjusted audio output. For example, if the user appears anxious, audio will be played in a calming tone.

[0924] (Application example 2)

[0925] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0926] Conventional systems for visually impaired people to understand Braille have the problem of limited user experience because they do not take into account the user's emotions when reading Braille. Also, because the voice output is uniform, there is an issue that it is not possible to respond appropriately according to the user's mental state or emotions. As a result, ease of use for visually impaired people has not been sufficiently considered.

[0927] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for launching an application using a voice command, means for acquiring Braille image data using a camera, means for transmitting the acquired image data to the server, means including a generation AI on the server side for analyzing the image data and identifying Braille, means for converting the Braille identified by the generation AI into text data, means for transmitting the converted text data to the terminal, means for converting the received text data on the terminal side into voice and reading it aloud, and means for analyzing the user's emotions using an emotion engine and adjusting the voice output. This makes it possible for visually impaired people to acquire Braille information more comfortably and provides a user-friendly system.

[0928] "Visually impaired people" refers to people whose visual function is impaired and who have difficulty obtaining information through their vision.

[0929] Braille is a special arrangement of dots that allows visually impaired people to understand letters using their sense of touch.

[0930] A "system" is a collection of components that enable multiple devices and software to work together to achieve a specific function.

[0931] "Voice command" means an instruction given by a user to operate a device or application by speaking.

[0932] "Application" means a software program that runs on a computer or device to provide a particular function or service.

[0933] A "camera" is an optical device for capturing images or video.

[0934] "Image data" means image information represented in digital form.

[0935] A "server" is a computer system that provides services to other computers over a network.

[0936] "Generative AI" refers to artificial intelligence that uses machine learning techniques to generate specific information from data.

[0937] "Text data" means data that represents character information in a digital format.

[0938] An "emotion engine" is a system that analyzes a user's voice and facial expressions to recognize their current emotional state and generate appropriate feedback based on that.

[0939] "Adjusting the audio output" means changing the tone and rate of the audio depending on the user's emotional state.

[0940] "Noise reduction" refers to the process of removing unnecessary information and noise from image or audio data.

[0941] "Preprocessing" refers to processing to prepare data before the main processing is performed.

[0942] "Speech synthesis engine" refers to a software component that converts text data into speech.

[0943] This invention provides a system for visually impaired people to understand Braille, which is easier to use and more user-friendly by recognizing the user's emotions and adjusting the voice output. This system includes the following processes: launching an application via voice command, acquiring Braille image data using a camera, transmitting the image data to a server, identifying the Braille using AI generated on the server, converting it to text data, transmitting the text data to a terminal, converting it to voice and reading it out loud, and analyzing the user's emotions and adjusting the voice output using an emotion engine. Specific embodiments of this system are described below.

[0944] Hardware and software used

[0945] Hardware: Smartphone (with camera)

[0946] Software: Android / iOS app, server (cloud server), generative AI (generative AI model), speech synthesis engine (speech synthesis software), emotion engine (emotion analysis software)

[0947] System configuration

[0948] Launching the application

[0949] The user issues a voice command to their smartphone, saying, "Launch the Braille app." The device's voice recognition engine analyzes this voice command and identifies the corresponding application. The launch command for the identified application is executed, and the Braille reading app launches.

[0950] Braille capture and transmission

[0951] When a user points the smartphone camera at an object with Braille, the camera captures the image data and stores it in memory, where it is compressed and uploaded to the server.

[0952] Image data analysis

[0953] The server analyzes the received image data and performs preprocessing such as noise removal. The generation AI then inputs the preprocessed image data and analyzes the Braille patterns. The Braille recognized as a result of the analysis is converted into text data and sent to the device.

[0954] Converting text data into speech

[0955] The device receives the text data and passes it to a speech synthesis engine. The speech synthesis engine converts the text data into audio data, which is then played back through the device's speaker. This allows the user to hear the Braille content aloud.

[0956] Emotion engine processing

[0957] The device's emotion engine analyzes the user's voice and facial expressions to recognize their current emotional state. For example, if the user is feeling anxious, it will identify that emotion. Based on the emotion recognized by the emotion engine, the tone and speed of the voice output are adjusted in real time, providing the most appropriate voice output for the user.

[0958] Examples of concrete examples and prompts

[0959] Scenario: Reading Braille on a station sign

[0960] 1. User: Says to the smartphone, "Launch the Braille app."

[0961] 2. Device: The app will launch and the main screen will appear.

[0962] 3. User: Point the camera at the Braille on the station sign.

[0963] 4. Terminal: Operate the camera to capture Braille and obtain image data.

[0964] 5. Terminal: Compresses the image data and sends it to the server.

[0965] 6. Server: Receives the image data and converts the Braille into text using a generation AI. This generates text data such as "Platform 1 goes to Tokyo, and platform 2 goes to Shinjuku."

[0966] 7. Server: Sends text data to the device.

[0967] 8. Terminal: Converts the received text data into speech and reads out loud, "Platform 1 goes to Tokyo, and platform 2 goes to Shinjuku."

[0968] 9. On the device: The emotion engine analyzes the user's emotions and adjusts the tone and speed of the voice based on the recognized emotion. For example, if the user is anxious, the voice will be played in a calming tone to give a sense of security.

[0969] 10. User: Listen to the audio and understand the instructions.

[0970] Prompt Sentence Examples

[0971] Speech prompt: "Launch the Braille app"

[0972] Generative AI prompt: "Convert this Braille image into text."

[0973] Text-to-speech prompt: "Convert this text to speech: 'This product is coffee'"

[0974] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0975] Step 1:

[0976] The user issues a voice command to their smartphone saying, "Launch the Braille app." The device's voice recognition engine analyzes this voice command, identifies the corresponding application, and executes the launch command. This launches the Braille reading app. The input is the user's voice command, and the output is the launch of the application.

[0977] Step 2:

[0978] When a user points a smartphone camera at an object with Braille, the device's camera captures image data and stores it in memory. The input is the camera image, and the output is Braille image data.

[0979] Step 3:

[0980] The device compresses the acquired image data and prepares it for uploading to the server. The data is sent to the server and analysis begins. The input is Braille image data, and the output is compressed image data sent to the server.

[0981] Step 4:

[0982] The server analyzes the received image data and performs preprocessing using generative AI to identify Braille, including noise removal. The input is the image data sent to the server, and the output is the preprocessed image data.

[0983] Step 5:

[0984] The server passes the preprocessed image data to the generation AI, which analyzes the Braille patterns. The Braille recognized as a result of this analysis is converted into text data. The input is preprocessed image data, and the output is text data.

[0985] Step 6:

[0986] The server sends the converted text data to the terminal. The input is the text data generated by the server, and the output is the text data sent to the terminal.

[0987] Step 7:

[0988] The device passes the received text data to the speech synthesis engine. The speech synthesis engine converts this text data into voice data, and the generated voice data is played back from the device's speaker. The input is text data, and the output is voice data.

[0989] Step 8:

[0990] The device's emotion engine analyzes the user's voice and facial expressions to recognize their current emotional state. For example, if the user looks anxious, it identifies that emotion. The input is the user's voice and facial expression data, and the output is the recognized emotion.

[0991] Step 9:

[0992] The tone and speed of the voice output are adjusted in real time based on the emotion recognized by the emotion engine, providing the optimal voice output for the user. The input is the recognized emotion, and the output is the adjusted voice output.

[0993] These steps will enable visually impaired people to obtain braille information more comfortably and create a user-friendly system.

[0994] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0995] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0996] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0997] [Third embodiment]

[0998] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0999] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1000] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1001] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1002] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1003] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1004] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1005] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1006] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1007] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1008] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1009] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1010] The present invention is a system for visually impaired people to understand Braille, which uses a smartphone camera and generation AI to read Braille aloud. A specific embodiment of this system is described below.

[1011] System configuration

[1012] This system consists of a smartphone (terminal) operated by the user, a server that processes data, and an application that runs between them. The application runs on the smartphone and is activated by voice commands. A camera is used to capture an object with Braille printed on it, and the acquired image data is sent to the server. On the server, a generative AI analyzes the Braille from the image data and converts the recognized Braille into text data. Finally, the smartphone converts the text data into speech and reads it out loud to the user.

[1013] Program processing

[1014] The specific program processing executed by each device will be explained in natural language below.

[1015] Launching the app

[1016] User: Says the voice command to their smartphone: "Launch the Braille app."

[1017] Device: The voice recognition engine analyzes the voice command and identifies the corresponding application. The application launch command is executed, and the Braille reading application is launched.

[1018] Braille capture and transmission

[1019] User: Point the smartphone camera at an object with Braille on it.

[1020] Terminal: The camera acquires image data, stores it in memory, prepares it for transmission to the server, compresses it, and uploads it to the server.

[1021] Image data analysis

[1022] Server: Stores the received image data for analysis. Analysis begins and performs preprocessing such as noise removal. The generation AI inputs the preprocessed image data and analyzes the Braille patterns. The recognized Braille is converted into text data and sent to the device.

[1023] Converting text data into speech

[1024] Device: The device receives the text data and passes it to the speech synthesis engine. The speech synthesis engine converts the text data into voice data, which is then played back from the speaker.

[1025] User: Checks the content of Braille through audio data. This allows visually impaired people to obtain the content through audio without having to read Braille.

[1026] Specific examples

[1027] Scenario: Reading Braille on a station sign

[1028] 1. User: Say to the smartphone, "Launch the Braille app."

[1029] 2. Device: The app will launch and the main screen will appear.

[1030] 3. User: Point the camera at the Braille on the station sign.

[1031] 4. Terminal: Operate the camera to capture Braille and obtain image data.

[1032] 5. Terminal: Compresses the image data and sends it to the server.

[1033] 6. Server: Receives the image data and converts the Braille into text using the generation AI. The generated text is, "Platform 1 goes to Tokyo, and platform 2 goes to Shinjuku."

[1034] 7. Server: Sends text data to the device.

[1035] 8. Terminal: Converts the received text data into speech and reads out loud, "Platform 1 goes to Tokyo, and platform 2 goes to Shinjuku."

[1036] 9. User: Listen to the audio and understand the instructions.

[1037] Such an embodiment of the system will enable visually impaired people to easily obtain Braille information, which is expected to reduce barriers to obtaining information in daily life and expand the range of activities available to visually impaired people.

[1038] The processing flow will be explained below.

[1039] Step 1:

[1040] User: Says the voice command to their smartphone: "Launch the Braille app."

[1041] A voice recognition engine receives a user's voice command and analyzes the voice data.

[1042] Based on the recognized command, it determines whether to launch a Braille reading application.

[1043] Step 2:

[1044] Terminal: Execute the application launch command to launch the Braille reading app.

[1045] The app will start and the main screen will be displayed.

[1046] The camera control module is ready and waiting to capture braille.

[1047] Step 3:

[1048] User: Point the smartphone camera at an object with Braille on it.

[1049] Refer to the on-screen guidelines and audio guide to point the camera in the correct position.

[1050] Give a voice command for the camera button to prepare for capturing Braille.

[1051] Step 4:

[1052] Terminal: Braille image data is acquired using a camera.

[1053] The camera captures the image in the frame and stores the image data in memory.

[1054] The stored image data undergoes a compression process to prepare it for transmission to the server.

[1055] Step 5:

[1056] Terminal: Sends compressed image data to a server via the Internet.

[1057] Upload image data to the server's API endpoint.

[1058] Monitor the transmission status to ensure that the data was sent successfully.

[1059] Step 6:

[1060] Server: Initiates the process of analyzing the received image data.

[1061] The reception of the data is confirmed and the image data is stored in the database.

[1062] Generative AI performs pre-processing, noise removal, and image enhancement.

[1063] Step 7:

[1064] Server: Preprocessed image data is passed to the generation AI, which then identifies the Braille characters.

[1065] A generative AI model analyzes the image data and recognizes Braille patterns.

[1066] The recognized braille pattern is converted into text data.

[1067] Step 8:

[1068] Server: Sends the generated text data to the device.

[1069] Converts text data into an optimal format and sends it to the device's API endpoint.

[1070] Step 9:

[1071] Terminal: Converts the received text data into audio data and prepares it to be read to the user.

[1072] Check the integrity of the text data and pass it to the speech synthesis engine.

[1073] A speech synthesis engine converts the text data into speech data.

[1074] Step 10:

[1075] Device: The generated audio data is played through the device's speaker.

[1076] The audio data is played and read to the user.

[1077] Step 11:

[1078] User: Check the braille content through audio from the device.

[1079] Listen to the audio and get the information you need.

[1080] Example 1

[1081] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1082] It is extremely difficult for visually impaired people to understand Braille in their daily lives. In particular, there is a lack of means to quickly and accurately understand the Braille on information and guide signs provided in public places, which poses a major challenge for visually impaired people, as it takes a lot of time and effort for them to obtain the information they need.

[1083] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1084] In this invention, the server includes a means for processing image data acquired by an image acquisition device and identifying Braille, a means for converting Braille into character data using a generative AI model, and a means for transmitting the converted character data to a terminal, thereby enabling visually impaired people to efficiently and quickly understand Braille by voice using a terminal such as a smartphone.

[1085] "Visually impaired persons" refer to people who have impaired vision and are unable to recognize letters or figures using normal visual methods.

[1086] "Braille" refers to a writing system made up of raised and recessed patterns that is used by visually impaired people to understand written information using their sense of touch.

[1087] A "system" refers to a set of mechanisms in which multiple devices and software work together.

[1088] "Voice command" refers to a means by which a user gives instructions to electronic devices or software using voice.

[1089] "Application" means a software program designed to provide a particular function or service.

[1090] "Image capture device" refers to equipment that uses a device such as a camera to capture visual information as a digital image.

[1091] "Data processing device" refers to a computer or server system for storing and processing acquired data.

[1092] "Analysis" refers to the process of using algorithms and programs to decipher acquired data and extract meaningful information.

[1093] A "generative AI model" refers to an artificial intelligence model that is trained to perform a specific task using techniques such as machine learning and deep learning.

[1094] "Character data" refers to character information represented in digital form.

[1095] "Terminal" refers to an electronic device that can be directly operated by a user, such as a smartphone or tablet.

[1096] "Means for converting into audio" refers to the technology or software for converting text data into audio data and playing it back through an audio output device.

[1097] The present invention is a system for visually impaired people to understand Braille. An application is started using a voice command, Braille image data acquired by an image acquisition device is sent to a data processing device, and the Braille is converted into character data using a generative AI model, which is then read aloud. Specific embodiments of this system are described below.

[1098] System configuration

[1099] This system consists of a terminal operated by the user, a server that processes data, and an application that runs between them. The application runs on the terminal and is activated by voice commands. An image capture device (e.g., a smartphone camera) is used to capture an object with Braille printed on it, and the captured image data is sent to the server. On the server, a generative AI model analyzes the Braille from the image data and converts the recognized Braille into text data. Finally, the terminal converts the text data into audio and reads it to the user.

[1100] Application Behavior

[1101] Launching the app

[1102] The user issues a voice command to the device, saying, "Launch the Braille app." This voice command is analyzed by the device's voice recognition engine. The voice recognition engine uses technology such as Google Assistant or Siri to identify the corresponding application. The application launch command is executed, and the Braille reading app launches.

[1103] Braille capture and transmission

[1104] The user points the device's camera at an object with Braille. The camera captures image data, which is then stored in the device's memory. The stored image data is then compressed and uploaded to the server.

[1105] Image data analysis

[1106] The server stores the received image data. It then begins analyzing it and performs preprocessing such as noise removal using an image processing library such as OpenCV. The generative AI model uses the preprocessed image data as input and analyzes the Braille patterns. The recognized Braille characters are converted into character data and sent to the device.

[1107] Converting text data into speech

[1108] The device receives the text data and passes it to a speech synthesis engine such as Google Text-to-Speech or Apple's VoiceOver function. The speech synthesis engine converts the text data into audio data, which is then played back through the device's speaker. The user can then confirm the braille content through the audio data.

[1109] Specific examples

[1110] Scenario: Reading Braille on a station sign

[1111] 1. User: Say to the smartphone, "Launch the Braille app."

[1112] 2. Device: The app will launch and the main screen will appear.

[1113] 3. User: Point the camera at the Braille on the station sign.

[1114] 4. Terminal: Operate the camera to capture Braille and obtain image data.

[1115] 5. Terminal: Compresses the image data and sends it to the server.

[1116] 6. Server: Receives the image data and converts the Braille into text using a generative AI model. This generates text such as "Platform 1 goes to Tokyo, and platform 2 goes to Shinjuku."

[1117] 7. Server: Sends character data to the terminal.

[1118] 8. Terminal: Converts the received text data into speech and reads out loud, "Platform 1 goes to Tokyo, and platform 2 goes to Shinjuku."

[1119] 9. User: Listen to the audio and understand the instructions.

[1120] Examples of prompt statements

[1121] Using the voice command "Launch the Braille app" as input, the prompt sentence can be designed as follows:

[1122] "Just tell your smartphone to 'launch the Braille app' and point the camera at the Braille. The app will then capture the image and send it to the server. On the server side, a generative AI model will convert the image data from Braille to text data. Finally, the text data will be communicated to the user as an audio alert."

[1123] By inputting this prompt into the generative AI model, the entire process of reading Braille can be explained to visually impaired people, allowing them to understand the specific steps to take.

[1124] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1125] Step 1: Launch the app

[1126] User: Says the voice command to their smartphone: "Launch the Braille app."

[1127] Input: The user's voice command.

[1128] Output: Text data of the voice command.

[1129] Specific actions: The user holds the smartphone close to the microphone and speaks.

[1130] On the device: The voice recognition engine analyzes the voice command and converts it into text data. The converted text data is used to identify the corresponding application and execute the command to launch the application.

[1131] Input: Text data of the voice command.

[1132] Output: Launches a Braille reading app.

[1133] Specific operation: The device's internal voice recognition engine converts the voice into text and searches for and launches the target app.

[1134] Step 2: Capturing Braille

[1135] User: Point the smartphone camera at an object with Braille on it.

[1136] Input: An object with Braille inscribed on it.

[1137] Output: A focused image from the camera.

[1138] Specific actions: The user focuses the camera on the Braille and sees the Braille on the app screen.

[1139] Terminal: The camera captures image data and stores it in memory.

[1140] Input: An image captured by the camera.

[1141] Output: Image data stored in memory.

[1142] Specific operation: The device's camera app takes a photo and stores the data in memory.

[1143] Step 3: Sending image data

[1144] Terminal: Compresses the stored image data and prepares it for transmission to the data processing device (server). Compresses the data and uploads it to the server.

[1145] Input: Image data stored in memory.

[1146] Output: Send compressed image data to the server.

[1147] Specific operation: Data compression and uploading is performed using the communication module inside the terminal.

[1148] Step 4: Analyzing the image data

[1149] Server: Stores the received image data and begins analysis. Performs preprocessing such as noise reduction using an image processing library (e.g., OpenCV).

[1150] Input: Compressed image data.

[1151] Output: Preprocessed image data.

[1152] Specific operation: Image data is stored in the server storage and noise is removed using an image processing library.

[1153] Server: The preprocessed image data is input into the generative AI model, which analyzes the Braille patterns. The generative AI model converts the Braille into character data.

[1154] Input: Preprocessed image data.

[1155] Output: Character data.

[1156] Specific operation: Image data is input into the generative AI model, and Braille analysis and character conversion are performed.

[1157] Step 5: Sending text data

[1158] Server: Sends the generated character data to the terminal.

[1159] Input: Character data.

[1160] Output: Sending character data to the terminal.

[1161] Specific operation: Sends character data to the terminal via the network.

[1162] Step 6: Converting text data into speech

[1163] Terminal: Receives the received text data and passes it to the speech synthesis engine, which converts it into voice data.

[1164] Input: Character data.

[1165] Output: Audio data.

[1166] Specific operation: Converts text data into audio data using Google Text-to-Speech or Apple's VoiceOver function.

[1167] Device: Plays the generated audio data through the speaker.

[1168] Input: Audio data.

[1169] Output: The audio played through the speakers.

[1170] Specific operation: Plays audio data through the device's speaker.

[1171] User: Check the braille content through audio data.

[1172] Input: The audio coming from the device's speaker.

[1173] Output: Information comprehension.

[1174] What happens: The user is near the smartphone and hears a voice notification.

[1175] (Application example 1)

[1176] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1177] Currently, there are limited means for visually impaired people to safely and quickly check security-related information. Even if they can read Braille, they may not be able to understand or communicate it properly. Therefore, there is a need for a system that allows visually impaired people to obtain information accurately and quickly through Braille.

[1178] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1179] In this invention, the server includes a means for pre-processing image data acquired by the camera, such as noise removal, a means including a generation AI for analyzing the image data, identifying Braille, and converting it into text data, and a means for recognizing security-related information based on Braille and notifying it by voice, thereby enabling visually impaired people to accurately and quickly obtain security-related information by voice.

[1180] "Visually impaired persons" refers to people with visual impairments, including those who have difficulty obtaining visual information on their own.

[1181] Braille is a writing system that consists of special symbols that allow visually impaired people to read written information using their sense of touch.

[1182] A "system" refers to a complex of multiple components that interact with each other to achieve a specific function.

[1183] "Voice command" refers to a voice input means for recognizing a user's speech and operating the system based on that instruction.

[1184] "Application" refers to computer software designed to provide a specific function.

[1185] "Camera" refers to an optical device for acquiring image data, and in this context refers specifically to a camera built into a mobile device such as a smartphone.

[1186] "Image data" refers to data obtained by converting visual information acquired by an imaging device such as a camera into digital values.

[1187] A "server" refers to a computer system that provides services such as data processing, storage, and use over a network.

[1188] "Generative AI" refers to algorithms or models that use artificial intelligence techniques to analyze, generate, or transform data.

[1189] "Text data" refers to digital data consisting of characters.

[1190] Converting to "audio" and "reading aloud" refers to the act of converting text data into an audio signal using speech synthesis technology and playing it back through an output device.

[1191] "Security-related information" refers to confidential information such as personal information, authentication codes, and PIN codes, and includes information that must be verified in a secure manner.

[1192] This invention is a system that enables visually impaired people to understand Braille and safely check security-related information. This system uses a smartphone camera and generation AI to allow visually impaired people to understand Braille through audio. Specific embodiments of this system are described below.

[1193] This system consists of a terminal operated by a user, a server that processes data, and an application that runs between them.

[1194] System configuration

[1195] Device:

[1196] It is a smartphone operated by the user, and applications are launched using voice commands. By saying "Launch the Braille app," the voice recognition engine analyzes the voice command and identifies the corresponding application, which then launches the Braille reading app.

[1197] camera:

[1198] The system uses the smartphone's built-in camera: when the user points the camera at an object with Braille, the camera captures the image data and stores it in memory.

[1199] server:

[1200] The acquired image data is sent to the server, where it undergoes preprocessing such as noise removal. The preprocessed image data is then input into a generative AI model, which analyzes the Braille patterns. The recognized Braille characters are converted into text data, which is then sent back to the device.

[1201] Text to speech conversion:

[1202] The received text data is passed to a speech synthesis engine on the device. The speech synthesis engine (for example, gTTS: Google Text-to-Speech) converts the text data into voice data. The generated voice data is played back through the device's speaker.

[1203] Specific examples

[1204] Scenario: Checking the Braille PIN code on your cash card at a bank ATM

[1205] 1. User: Say to the smartphone, "Launch the Braille app."

[1206] 2. Device: The app will launch and display the main screen, with the camera activated and ready to take a photo of an object with Braille.

[1207] 3. User: Point the smartphone camera at the card with the PIN code written on it.

[1208] 4. Terminal: The camera takes a picture of the card.

[1209] 5. Terminal: The acquired image data is compressed and sent to the server.

[1210] 6. Server: Receives the image data and performs preprocessing such as noise removal. Then, the generative AI model analyzes the Braille and generates text data. The text "1234" is generated.

[1211] 7. Server: Sends text data to the device.

[1212] 8. Device: Converts the received text data into speech and reads out "1234."

[1213] 9. User: Listen to the audio and confirm the PIN code.

[1214] This system allows visually impaired people to easily access Braille information, and is particularly important for enabling them to check security-related information safely and quickly.

[1215] Prompt Sentence Examples

[1216] Say "Launch the Braille app." Then point the camera at the Braille. The app will launch and the camera will capture the image. The Braille will be analyzed and the results will be read aloud.

[1217] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1218] Step 1:

[1219] User: Says to the smartphone, "Launch the Braille app." Requests the launch of the application via voice command.

[1220] Input: User's voice command

[1221] Output: Text data of voice commands

[1222] What happens: The smartphone uses a speech recognition engine (e.g., Google Speech Recognition API) to convert the voice command into text and identify the phrase "Launch Braille app."

[1223] Step 2:

[1224] Device: Analyzes voice commands, identifies the corresponding application, and launches it.

[1225] Input: Text data of voice command

[1226] Output: Launching a Braille reading application

[1227] Specific operation: Based on the text data, the smartphone's operating system launches the Braille reading application, the application's main screen is displayed, and the camera becomes available for use.

[1228] Step 3:

[1229] User: Point the smartphone camera at an object with Braille on it.

[1230] Input: An object with braille on it

[1231] Output: Camera image data

[1232] What it does: The user positions the camera in the right position, and the smartphone camera captures an image of the object with Braille on it.

[1233] Step 4:

[1234] Terminal: The camera captures image data and stores it in memory.

[1235] Input: Camera image data

[1236] Output: Saved image data

[1237] How it works: The image sensor captures high-resolution image data and temporarily stores it in the smartphone's memory.

[1238] Step 5:

[1239] Terminal: The stored image data is compressed and sent to the server.

[1240] Input: Saved image data

[1241] Output: Compressed image data uploaded to the server

[1242] Specific operation: The smartphone uses an image compression algorithm (e.g., JPEG compression) to compress the image data and send it to the server over the network.

[1243] Step 6:

[1244] Server: Stores the received image data for analysis and performs preprocessing such as noise removal.

[1245] Input: Compressed image data uploaded to the server

[1246] Output: Preprocessed image data

[1247] What happens: The server decodes the image and applies a noise reduction filter (e.g., median filter) to improve the image quality.

[1248] Step 7:

[1249] Server: Analyzes image data and converts Braille into text data using a generative AI model that identifies Braille.

[1250] Input: Preprocessed image data

[1251] Output: Converted text data

[1252] What it does: Uses a generative AI model (e.g., Tesseract OCR or a custom-trained model) to identify Braille characters from pre-processed images and generate corresponding text data.

[1253] Step 8:

[1254] Server: Sends text data to the device.

[1255] Input: Converted text data

[1256] Output: Text data sent to the terminal

[1257] Specific operation: The server encodes the text data and sends it to the smartphone via the network.

[1258] Step 9:

[1259] Terminal: Converts received text data into speech and reads it to the user.

[1260] Input: Text data sent to the terminal

[1261] Output: Audio data

[1262] Specific operation: The smartphone uses a speech synthesis engine (e.g., gTTS) to convert the text data into audio data and play it back through the speaker.

[1263] Step 10:

[1264] User: Listen to what is read aloud and see what is in braille.

[1265] Input: Audio data

[1266] Output: Security-related information understood

[1267] Specific operation: The user listens to the audio from the smartphone speaker and obtains the information, which allows them to accurately understand the content of the Braille and take the necessary action.

[1268] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1269] The present invention provides a system that helps visually impaired people understand Braille by combining it with an emotion engine that recognizes the user's emotions, making it easier to use and more user-friendly. Specific embodiments of this system are described below.

[1270] System configuration

[1271] This system consists of a smartphone (terminal) operated by the user, a server that processes data, and an application that includes an emotion engine. The application runs on the smartphone and is activated by voice commands. A camera is used to capture an object with Braille printed on it, and the acquired image data is sent to the server. On the server, a generative AI analyzes the Braille from the image data and converts the recognized Braille into text data. Finally, the smartphone converts the text data into speech and reads it to the user. The emotion engine also has the function of recognizing the user's emotions and adjusting the voice output based on those emotions.

[1272] Program processing

[1273] The specific program processing executed by each device will be explained in natural language below.

[1274] Launching the app

[1275] User: Says the voice command to their smartphone: "Launch the Braille app."

[1276] Device: The voice recognition engine analyzes the voice command and identifies the corresponding application. The application launch command is executed, and the Braille reading application is launched.

[1277] Braille capture and transmission

[1278] User: Point the smartphone camera at an object with Braille on it.

[1279] Terminal: The camera acquires image data, stores it in memory, prepares it for transmission to the server, compresses it, and uploads it to the server.

[1280] Image data analysis

[1281] Server: Stores the received image data for analysis. Analysis begins and performs preprocessing such as noise removal. The generation AI inputs the preprocessed image data and analyzes the Braille patterns. The recognized Braille is converted into text data and sent to the device.

[1282] Converting text data into speech

[1283] Device: The device receives the text data and passes it to the speech synthesis engine. The speech synthesis engine converts the text data into voice data, which is then played back from the speaker.

[1284] Emotion engine processing

[1285] On the device: The emotion engine analyzes the user's voice and facial expressions to recognize their current emotional state. For example, it identifies the emotion if the user is anxious.

[1286] On the device: Adjust the tone and speed of the voice output based on the emotions recognized by the emotion engine, providing just-in-time feedback to optimize the voice output for the user.

[1287] Specific examples

[1288] Scenario: Reading Braille on a station sign

[1289] 1. User: Say to the smartphone, "Launch the Braille app."

[1290] 2. Device: The app will launch and the main screen will appear.

[1291] 3. User: Point the camera at the Braille on the station sign.

[1292] 4. Terminal: Operate the camera to capture Braille and obtain image data.

[1293] 5. Terminal: Compresses the image data and sends it to the server.

[1294] 6. Server: Receives the image data and converts the Braille into text using the generation AI. The generated text is, "Platform 1 goes to Tokyo, and platform 2 goes to Shinjuku."

[1295] 7. Server: Sends text data to the device.

[1296] 8. Terminal: Converts the received text data into speech and reads out loud, "Platform 1 goes to Tokyo, and platform 2 goes to Shinjuku."

[1297] 9. On the device: The emotion engine analyzes the user's emotions and adjusts the tone and speed of the voice based on the recognized emotion. For example, if the user is anxious, the voice will be played in a calming tone to give a sense of security.

[1298] 10. User: Listen to the audio and understand the instructions.

[1299] Combining emotion engines in this way will enable visually impaired people to obtain braille information more comfortably, improving the user experience. This is expected to further reduce barriers to obtaining information in daily life and expand the range of activities available to visually impaired people.

[1300] The processing flow will be explained below.

[1301] Step 1:

[1302] User: Says the voice command to their smartphone: "Launch the Braille app."

[1303] A voice recognition engine receives a user's voice command and analyzes the voice data.

[1304] Based on the recognized command, it determines whether to launch a Braille reading application.

[1305] Step 2:

[1306] Terminal: Execute the application launch command to launch the Braille reading app.

[1307] The app will start and the main screen will be displayed.

[1308] The camera control module is ready and waiting to capture braille.

[1309] Step 3:

[1310] User: Point the smartphone camera at an object with Braille on it.

[1311] Refer to the on-screen guidelines and audio guide to point the camera in the correct position.

[1312] Give a voice command for the camera button to prepare for capturing Braille.

[1313] Step 4:

[1314] Terminal: Braille image data is acquired using a camera.

[1315] The camera captures the image in the frame and stores the image data in memory.

[1316] The stored image data undergoes a compression process to prepare it for transmission to the server.

[1317] Step 5:

[1318] Terminal: Sends compressed image data to a server via the Internet.

[1319] Upload image data to the server's API endpoint.

[1320] Monitor the transmission status to ensure that the data was sent successfully.

[1321] Step 6:

[1322] Server: Initiates the process of analyzing the received image data.

[1323] The reception of the data is confirmed and the image data is stored in the database.

[1324] Generative AI performs pre-processing, noise removal, and image enhancement.

[1325] Step 7:

[1326] Server: Preprocessed image data is passed to the generation AI, which then identifies the Braille characters.

[1327] A generative AI model analyzes the image data and recognizes Braille patterns.

[1328] The recognized braille pattern is converted into text data.

[1329] Step 8:

[1330] Server: Sends the generated text data to the device.

[1331] Converts text data into an optimal format and sends it to the device's API endpoint.

[1332] Step 9:

[1333] Terminal: Converts the received text data into audio data and prepares it to be read to the user.

[1334] Check the integrity of the text data and pass it to the speech synthesis engine.

[1335] A speech synthesis engine converts the text data into speech data.

[1336] Step 10:

[1337] On the device: The emotion engine captures voice and facial expression data to analyze the user's emotions.

[1338] It uses voice input and a camera to collect user emotional data.

[1339] The emotion engine analyzes the collected data to identify the current emotional state.

[1340] Step 11:

[1341] Device: Adjusts voice output based on emotion data obtained from the emotion engine.

[1342] If the user seems anxious, calm the voice tone.

[1343] If the user is excited, adjust the speed of the speech output.

[1344] Step 12:

[1345] Device: The generated audio data is played through the device's speaker.

[1346] The audio data is played and read to the user.

[1347] Step 13:

[1348] User: Check the braille content through audio from the device.

[1349] Listen to the audio and get the information you need.

[1350] Example 2

[1351] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1352] Existing technologies for visually impaired people to understand Braille can sometimes make it difficult to read the content of Braille efficiently and accurately. Furthermore, they are unable to provide speech output that takes into account the user's emotional state, which can lead to a decline in the comprehension of information and the quality of the user experience. There is a need to provide a system that can solve this issue and enable visually impaired people to more comfortably acquire and understand Braille information.

[1353] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1354] In this invention, the server includes a means including a generative AI model that analyzes image data and identifies Braille, a means for converting the Braille identified by the generative AI model into text information, and a means for transmitting the converted text information to a terminal. This allows visually impaired people to capture images of Braille with a camera and hear its contents audibly. The terminal also includes a means for analyzing the user's emotional state using an emotion recognition engine and a means for adjusting the tone and speed of the voice based on the analyzed emotional state. This provides optimal audio output for the user, improving the comprehension of information and the quality of the user experience.

[1355] A "voice command" is a method by which a user issues instructions to a system using voice, which is processed using voice recognition technology.

[1356] "Image capture device" refers to a device, such as a camera, that captures visual information as digital image data.

[1357] "Data processing device" refers to a computer system such as a server that analyzes acquired data and converts it into necessary information.

[1358] A "generative AI model" refers to a model that uses artificial intelligence techniques to analyze and recognize data, in this case, to identify Braille.

[1359] "Text information" refers to data that has been converted from coded data such as Braille into a text format that is easily understandable by humans.

[1360] "Terminal" refers to a device that is directly operated by a user, such as a smartphone or tablet.

[1361] A "voice synthesis engine" refers to a technology that converts text data into voice data and provides it to the user as voice information.

[1362] An "emotion recognition engine" refers to a system that uses artificial intelligence technology to analyze and identify a user's emotional state from their voice and facial expressions.

[1363] "Tone" refers to the pitch and intensity of a voice, and is an element that adjusts the voice output according to the user's emotional state.

[1364] "Speed" refers to the speed at which the audio is played back, and is a factor that speeds up or slows down the audio output depending on the user's emotional state.

[1365] This system, which helps visually impaired people understand Braille, is composed as follows: The system consists of a terminal operated by the user, a server that processes data, and an application that includes an emotion recognition engine. The application runs on the terminal and is activated by accepting voice commands.

[1366] When a user issues a voice command to their smartphone (hereinafter referred to as the "device"), such as "Launch the Braille app," a speech recognition engine (e.g., Google Speech-to-Text) analyzes the voice and identifies the application to be launched. Once the application is launched, the main screen is displayed.

[1367] Next, the user points the device's camera at the object with the Braille. The camera module (e.g., Android's Camera API) acquires the image data and temporarily stores it in memory. The stored image data is compressed using a compression algorithm (e.g., JPEG compression) and sent to the server via a network module (e.g., the Retrofit library). The progress of the transmission is displayed on the device.

[1368] The server stores the received image data in memory and preprocesses it using a noise reduction algorithm (e.g., Gaussian filter). A generative AI model (e.g., TensorFlow's OCR engine) analyzes the preprocessed image data and recognizes Braille patterns with high accuracy. The recognized Braille information is then converted into text and sent to the device.

[1369] The device receives the text information and inputs it into a speech synthesis engine (e.g., Google Text-to-Speech). The speech synthesis engine converts the text information into audio data, which is then read aloud to the user through the speaker.

[1370] Furthermore, an emotion recognition engine (e.g., Affectiva SDK) analyzes the user's voice and facial expressions to identify their current emotional state. For example, if the user is anxious, it will identify that emotion. Based on this recognition result, the tone and speed of the voice output are adjusted. For example, an anxious user will be provided with a calming voice tone.

[1371] As a concrete example, consider a scenario in which a user is reading a Braille sign at a train station. When the user says to their smartphone, "Launch the Braille app," the application launches and the main screen appears. Next, the user points the camera at the Braille on the station sign, and the device's camera captures image data and sends it to the server. The server analyzes the Braille and generates text information such as "Platform 1 goes to Tokyo, Platform 2 goes to Shinjuku," which is then sent to the device. The device converts this information into audio data and reads it out loud as "Platform 1 goes to Tokyo, Platform 2 goes to Shinjuku." At this time, an emotion recognition engine analyzes the user's emotional state, and anxious users are provided with information in a calming tone.

[1372] In this way, this system is designed to enable visually impaired people to obtain Braille information comfortably and easily.

[1373] Prompt Sentence Examples

[1374] Below are some examples of prompt sentences used in this system.

[1375] "Launch the Braille app."

[1376] "Point the camera at the Braille and capture the image."

[1377] "Convert Braille to text"

[1378] "Read it aloud"

[1379] "Analyze emotions and adjust tone of voice"

[1380] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1381] Step 1:

[1382] The user issues a voice command to the smartphone, saying, "Launch the Braille app." The input is the user's voice, and the output is the voice data received by the smartphone.

[1383] Step 2:

[1384] The device uses a speech recognition engine (e.g., Google Speech-to-Text) to analyze the voice data and identify the command to "launch a Braille app." The input is the voice data, and the output is the analyzed text data and a launch command. Based on this, the application manager launches the Braille reading app, and the main screen is displayed.

[1385] Step 3:

[1386] The user points the smartphone camera at an object with Braille and determines its position. The input is the user's camera operation, and the output is the adjustment of the camera angle and focus.

[1387] Step 4:

[1388] The device acquires image data using a camera module (e.g., Android's Camera API) and stores the data in temporary memory. The input is the image captured by the camera, and the output is the stored image data.

[1389] Step 5:

[1390] The image data acquired by the terminal is compressed using a compression algorithm (e.g., JPEG compression). The input is uncompressed image data, and the output is compressed image data.

[1391] Step 6:

[1392] The device sends compressed image data to the server using a network module (e.g., Retrofit library). The input is the compressed image data, and the output is the upload to the server.

[1393] Step 7:

[1394] The server stores the received image data in memory. The input is the uploaded image data, and the output is the stored data in the server.

[1395] Step 8:

[1396] The server pre-processes the images with a noise reduction algorithm (e.g., Gaussian filter). The input is the stored image data, and the output is the pre-processed image data.

[1397] Step 9:

[1398] The server inputs the preprocessed image data into a generative AI model (e.g., TensorFlow's OCR engine) to analyze the Braille patterns. The input is the preprocessed image data, and the output is the Braille recognition data.

[1399] Step 10:

[1400] The server converts the recognized Braille information into text information and sends the text information to the terminal. The input is the Braille recognition data, and the output is the transmission of text information to the terminal.

[1401] Step 11:

[1402] The text information received by the device from the server is input into a speech synthesis engine (e.g., Google Text-to-Speech) and converted into voice data. The input is text information, and the output is the generated voice data.

[1403] Step 12:

[1404] The device plays the generated audio data from a speaker and reads the braille content aloud to the user. The input is audio data, and the output is audio playback.

[1405] Step 13:

[1406] The device uses an emotion recognition engine (e.g., Affectiva SDK) to analyze the user's voice and facial expressions to determine their current emotional state. The input is the user's facial expression and voice data, and the output is the analyzed emotional state.

[1407] Step 14:

[1408] The device adjusts the tone and rate of the audio output based on the recognized emotional state. The input is the analyzed emotional state, and the output is the adjusted audio output. For example, if the user appears anxious, audio will be played in a calming tone.

[1409] (Application example 2)

[1410] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1411] Conventional systems for visually impaired people to understand Braille have the problem of limited user experience because they do not take into account the user's emotions when reading Braille. Also, because the voice output is uniform, there is an issue that it is not possible to respond appropriately according to the user's mental state or emotions. As a result, ease of use for visually impaired people has not been sufficiently considered.

[1412] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for launching an application using a voice command, means for acquiring Braille image data using a camera, means for transmitting the acquired image data to the server, means including a generation AI on the server side for analyzing the image data and identifying Braille, means for converting the Braille identified by the generation AI into text data, means for transmitting the converted text data to the terminal, means for converting the received text data on the terminal side into voice and reading it aloud, and means for analyzing the user's emotions using an emotion engine and adjusting the voice output. This makes it possible for visually impaired people to acquire Braille information more comfortably and provides a user-friendly system.

[1413] "Visually impaired people" refers to people whose visual function is impaired and who have difficulty obtaining information through their vision.

[1414] Braille is a special arrangement of dots that allows visually impaired people to understand letters using their sense of touch.

[1415] A "system" is a collection of components that enable multiple devices and software to work together to achieve a specific function.

[1416] "Voice command" means an instruction given by a user to operate a device or application by speaking.

[1417] "Application" means a software program that runs on a computer or device to provide a particular function or service.

[1418] A "camera" is an optical device for capturing images or video.

[1419] "Image data" means image information represented in digital form.

[1420] A "server" is a computer system that provides services to other computers over a network.

[1421] "Generative AI" refers to artificial intelligence that uses machine learning techniques to generate specific information from data.

[1422] "Text data" means data that represents character information in a digital format.

[1423] An "emotion engine" is a system that analyzes a user's voice and facial expressions to recognize their current emotional state and generate appropriate feedback based on that.

[1424] "Adjusting the audio output" means changing the tone and rate of the audio depending on the user's emotional state.

[1425] "Noise reduction" refers to the process of removing unnecessary information and noise from image or audio data.

[1426] "Preprocessing" refers to processing to prepare data before the main processing is performed.

[1427] "Speech synthesis engine" refers to a software component that converts text data into speech.

[1428] This invention provides a system for visually impaired people to understand Braille, which is easier to use and more user-friendly by recognizing the user's emotions and adjusting the voice output. This system includes the following processes: launching an application via voice command, acquiring Braille image data using a camera, transmitting the image data to a server, identifying the Braille using AI generated on the server, converting it to text data, transmitting the text data to a terminal, converting it to voice and reading it out loud, and analyzing the user's emotions and adjusting the voice output using an emotion engine. Specific embodiments of this system are described below.

[1429] Hardware and software used

[1430] Hardware: Smartphone (with camera)

[1431] Software: Android / iOS app, server (cloud server), generative AI (generative AI model), speech synthesis engine (speech synthesis software), emotion engine (emotion analysis software)

[1432] System configuration

[1433] Launching the application

[1434] The user issues a voice command to their smartphone, saying, "Launch the Braille app." The device's voice recognition engine analyzes this voice command and identifies the corresponding application. The launch command for the identified application is executed, and the Braille reading app launches.

[1435] Braille capture and transmission

[1436] When a user points the smartphone camera at an object with Braille, the camera captures the image data and stores it in memory, where it is compressed and uploaded to the server.

[1437] Image data analysis

[1438] The server analyzes the received image data and performs preprocessing such as noise removal. The generation AI then inputs the preprocessed image data and analyzes the Braille patterns. The Braille recognized as a result of the analysis is converted into text data and sent to the device.

[1439] Converting text data into speech

[1440] The device receives the text data and passes it to a speech synthesis engine. The speech synthesis engine converts the text data into audio data, which is then played back through the device's speaker. This allows the user to hear the Braille content aloud.

[1441] Emotion engine processing

[1442] The device's emotion engine analyzes the user's voice and facial expressions to recognize their current emotional state. For example, if the user is feeling anxious, it will identify that emotion. Based on the emotion recognized by the emotion engine, the tone and speed of the voice output are adjusted in real time, providing the most appropriate voice output for the user.

[1443] Examples of concrete examples and prompts

[1444] Scenario: Reading Braille on a station sign

[1445] 1. User: Says to the smartphone, "Launch the Braille app."

[1446] 2. Device: The app will launch and the main screen will appear.

[1447] 3. User: Point the camera at the Braille on the station sign.

[1448] 4. Terminal: Operate the camera to capture Braille and obtain image data.

[1449] 5. Terminal: Compresses the image data and sends it to the server.

[1450] 6. Server: Receives the image data and converts the Braille into text using a generation AI. This generates text data such as "Platform 1 goes to Tokyo, and platform 2 goes to Shinjuku."

[1451] 7. Server: Sends text data to the device.

[1452] 8. Terminal: Converts the received text data into speech and reads out loud, "Platform 1 goes to Tokyo, and platform 2 goes to Shinjuku."

[1453] 9. On the device: The emotion engine analyzes the user's emotions and adjusts the tone and speed of the voice based on the recognized emotion. For example, if the user is anxious, the voice will be played in a calming tone to give a sense of security.

[1454] 10. User: Listen to the audio and understand the instructions.

[1455] Prompt Sentence Examples

[1456] Speech prompt: "Launch the Braille app"

[1457] Generative AI prompt: "Convert this Braille image into text."

[1458] Text-to-speech prompt: "Convert this text to speech: 'This product is coffee'"

[1459] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1460] Step 1:

[1461] The user issues a voice command to their smartphone saying, "Launch the Braille app." The device's voice recognition engine analyzes this voice command, identifies the corresponding application, and executes the launch command. This launches the Braille reading app. The input is the user's voice command, and the output is the launch of the application.

[1462] Step 2:

[1463] When a user points a smartphone camera at an object with Braille, the device's camera captures image data and stores it in memory. The input is the camera image, and the output is Braille image data.

[1464] Step 3:

[1465] The device compresses the acquired image data and prepares it for uploading to the server. The data is sent to the server and analysis begins. The input is Braille image data, and the output is compressed image data sent to the server.

[1466] Step 4:

[1467] The server analyzes the received image data and performs preprocessing using generative AI to identify Braille, including noise removal. The input is the image data sent to the server, and the output is the preprocessed image data.

[1468] Step 5:

[1469] The server passes the preprocessed image data to the generation AI, which analyzes the Braille patterns. The Braille recognized as a result of this analysis is converted into text data. The input is preprocessed image data, and the output is text data.

[1470] Step 6:

[1471] The server sends the converted text data to the terminal. The input is the text data generated by the server, and the output is the text data sent to the terminal.

[1472] Step 7:

[1473] The device passes the received text data to the speech synthesis engine. The speech synthesis engine converts this text data into voice data, and the generated voice data is played back from the device's speaker. The input is text data, and the output is voice data.

[1474] Step 8:

[1475] The device's emotion engine analyzes the user's voice and facial expressions to recognize their current emotional state. For example, if the user looks anxious, it identifies that emotion. The input is the user's voice and facial expression data, and the output is the recognized emotion.

[1476] Step 9:

[1477] The tone and speed of the voice output are adjusted in real time based on the emotion recognized by the emotion engine, providing the optimal voice output for the user. The input is the recognized emotion, and the output is the adjusted voice output.

[1478] These steps will enable visually impaired people to obtain braille information more comfortably and create a user-friendly system.

[1479] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1480] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1481] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1482] [Fourth embodiment]

[1483] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1484] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1485] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1486] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1487] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1488] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1489] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1490] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1491] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1492] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1493] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1494] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1495] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1496] The present invention is a system for visually impaired people to understand Braille, which uses a smartphone camera and generation AI to read Braille aloud. A specific embodiment of this system is described below.

[1497] System configuration

[1498] This system consists of a smartphone (terminal) operated by the user, a server that processes data, and an application that runs between them. The application runs on the smartphone and is activated by voice commands. A camera is used to capture an object with Braille printed on it, and the acquired image data is sent to the server. On the server, a generative AI analyzes the Braille from the image data and converts the recognized Braille into text data. Finally, the smartphone converts the text data into speech and reads it out loud to the user.

[1499] Program processing

[1500] The specific program processing executed by each device will be explained in natural language below.

[1501] Launching the app

[1502] User: Says the voice command to their smartphone: "Launch the Braille app."

[1503] Device: The voice recognition engine analyzes the voice command and identifies the corresponding application. The application launch command is executed, and the Braille reading application is launched.

[1504] Braille capture and transmission

[1505] User: Point the smartphone camera at an object with Braille on it.

[1506] Terminal: The camera acquires image data, stores it in memory, prepares it for transmission to the server, compresses it, and uploads it to the server.

[1507] Image data analysis

[1508] Server: Stores the received image data for analysis. Analysis begins and performs preprocessing such as noise removal. The generation AI inputs the preprocessed image data and analyzes the Braille patterns. The recognized Braille is converted into text data and sent to the device.

[1509] Converting text data into speech

[1510] Device: The device receives the text data and passes it to the speech synthesis engine. The speech synthesis engine converts the text data into voice data, which is then played back from the speaker.

[1511] User: Checks the content of Braille through audio data. This allows visually impaired people to obtain the content through audio without having to read Braille.

[1512] Specific examples

[1513] Scenario: Reading Braille on a station sign

[1514] 1. User: Say to the smartphone, "Launch the Braille app."

[1515] 2. Device: The app will launch and the main screen will appear.

[1516] 3. User: Point the camera at the Braille on the station sign.

[1517] 4. Terminal: Operate the camera to capture Braille and obtain image data.

[1518] 5. Terminal: Compresses the image data and sends it to the server.

[1519] 6. Server: Receives the image data and converts the Braille into text using the generation AI. The generated text is, "Platform 1 goes to Tokyo, and platform 2 goes to Shinjuku."

[1520] 7. Server: Sends text data to the device.

[1521] 8. Terminal: Converts the received text data into speech and reads out loud, "Platform 1 goes to Tokyo, and platform 2 goes to Shinjuku."

[1522] 9. User: Listen to the audio and understand the instructions.

[1523] Such an embodiment of the system will enable visually impaired people to easily obtain Braille information, which is expected to reduce barriers to obtaining information in daily life and expand the range of activities available to visually impaired people.

[1524] The processing flow will be explained below.

[1525] Step 1:

[1526] User: Says the voice command to their smartphone: "Launch the Braille app."

[1527] A voice recognition engine receives a user's voice command and analyzes the voice data.

[1528] Based on the recognized command, it determines whether to launch a Braille reading application.

[1529] Step 2:

[1530] Terminal: Execute the application launch command to launch the Braille reading app.

[1531] The app will start and the main screen will be displayed.

[1532] The camera control module is ready and waiting to capture braille.

[1533] Step 3:

[1534] User: Point the smartphone camera at an object with Braille on it.

[1535] Refer to the on-screen guidelines and audio guide to point the camera in the correct position.

[1536] Give a voice command for the camera button to prepare for capturing Braille.

[1537] Step 4:

[1538] Terminal: Braille image data is acquired using a camera.

[1539] The camera captures the image in the frame and stores the image data in memory.

[1540] The stored image data undergoes a compression process to prepare it for transmission to the server.

[1541] Step 5:

[1542] Terminal: Sends compressed image data to a server via the Internet.

[1543] Upload image data to the server's API endpoint.

[1544] Monitor the transmission status to ensure that the data was sent successfully.

[1545] Step 6:

[1546] Server: Initiates the process of analyzing the received image data.

[1547] The reception of the data is confirmed and the image data is stored in the database.

[1548] Generative AI performs pre-processing, noise removal, and image enhancement.

[1549] Step 7:

[1550] Server: Preprocessed image data is passed to the generation AI, which then identifies the Braille characters.

[1551] A generative AI model analyzes the image data and recognizes Braille patterns.

[1552] The recognized braille pattern is converted into text data.

[1553] Step 8:

[1554] Server: Sends the generated text data to the device.

[1555] Converts text data into an optimal format and sends it to the device's API endpoint.

[1556] Step 9:

[1557] Terminal: Converts the received text data into audio data and prepares it to be read to the user.

[1558] Check the integrity of the text data and pass it to the speech synthesis engine.

[1559] A speech synthesis engine converts the text data into speech data.

[1560] Step 10:

[1561] Device: The generated audio data is played through the device's speaker.

[1562] The audio data is played and read to the user.

[1563] Step 11:

[1564] User: Check the braille content through audio from the device.

[1565] Listen to the audio and get the information you need.

[1566] Example 1

[1567] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1568] It is extremely difficult for visually impaired people to understand Braille in their daily lives. In particular, there is a lack of means to quickly and accurately understand the Braille on information and guide signs provided in public places, which poses a major challenge for visually impaired people, as it takes a lot of time and effort for them to obtain the information they need.

[1569] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1570] In this invention, the server includes a means for processing image data acquired by an image acquisition device and identifying Braille, a means for converting Braille into character data using a generative AI model, and a means for transmitting the converted character data to a terminal, thereby enabling visually impaired people to efficiently and quickly understand Braille by voice using a terminal such as a smartphone.

[1571] "Visually impaired persons" refer to people who have impaired vision and are unable to recognize letters or figures using normal visual methods.

[1572] "Braille" refers to a writing system made up of raised and recessed patterns that is used by visually impaired people to understand written information using their sense of touch.

[1573] A "system" refers to a set of mechanisms in which multiple devices and software work together.

[1574] "Voice command" refers to a means by which a user gives instructions to electronic devices or software using voice.

[1575] "Application" means a software program designed to provide a particular function or service.

[1576] "Image capture device" refers to equipment that uses a device such as a camera to capture visual information as a digital image.

[1577] "Data processing device" refers to a computer or server system for storing and processing acquired data.

[1578] "Analysis" refers to the process of using algorithms and programs to decipher acquired data and extract meaningful information.

[1579] A "generative AI model" refers to an artificial intelligence model that is trained to perform a specific task using techniques such as machine learning and deep learning.

[1580] "Character data" refers to character information represented in digital form.

[1581] "Terminal" refers to an electronic device that can be directly operated by a user, such as a smartphone or tablet.

[1582] "Means for converting into audio" refers to the technology or software for converting text data into audio data and playing it back through an audio output device.

[1583] The present invention is a system for visually impaired people to understand Braille. An application is started using a voice command, Braille image data acquired by an image acquisition device is sent to a data processing device, and the Braille is converted into character data using a generative AI model, which is then read aloud. Specific embodiments of this system are described below.

[1584] System configuration

[1585] This system consists of a terminal operated by the user, a server that processes data, and an application that runs between them. The application runs on the terminal and is activated by voice commands. An image capture device (e.g., a smartphone camera) is used to capture an object with Braille printed on it, and the captured image data is sent to the server. On the server, a generative AI model analyzes the Braille from the image data and converts the recognized Braille into text data. Finally, the terminal converts the text data into audio and reads it to the user.

[1586] Application Behavior

[1587] Launching the app

[1588] The user issues a voice command to the device, saying, "Launch the Braille app." This voice command is analyzed by the device's voice recognition engine. The voice recognition engine uses technology such as Google Assistant or Siri to identify the corresponding application. The application launch command is executed, and the Braille reading app launches.

[1589] Braille capture and transmission

[1590] The user points the device's camera at an object with Braille. The camera captures image data, which is then stored in the device's memory. The stored image data is then compressed and uploaded to the server.

[1591] Image data analysis

[1592] The server stores the received image data. It then begins analyzing it and performs preprocessing such as noise removal using an image processing library such as OpenCV. The generative AI model uses the preprocessed image data as input and analyzes the Braille patterns. The recognized Braille characters are converted into character data and sent to the device.

[1593] Converting text data into speech

[1594] The device receives the text data and passes it to a speech synthesis engine such as Google Text-to-Speech or Apple's VoiceOver function. The speech synthesis engine converts the text data into audio data, which is then played back through the device's speaker. The user can then confirm the braille content through the audio data.

[1595] Specific examples

[1596] Scenario: Reading Braille on a station sign

[1597] 1. User: Say to the smartphone, "Launch the Braille app."

[1598] 2. Device: The app will launch and the main screen will appear.

[1599] 3. User: Point the camera at the Braille on the station sign.

[1600] 4. Terminal: Operate the camera to capture Braille and obtain image data.

[1601] 5. Terminal: Compresses the image data and sends it to the server.

[1602] 6. Server: Receives the image data and converts the Braille into text using a generative AI model. This generates text such as "Platform 1 goes to Tokyo, and platform 2 goes to Shinjuku."

[1603] 7. Server: Sends character data to the terminal.

[1604] 8. Terminal: Converts the received text data into speech and reads out loud, "Platform 1 goes to Tokyo, and platform 2 goes to Shinjuku."

[1605] 9. User: Listen to the audio and understand the instructions.

[1606] Examples of prompt statements

[1607] Using the voice command "Launch the Braille app" as input, the prompt sentence can be designed as follows:

[1608] "Just tell your smartphone to 'launch the Braille app' and point the camera at the Braille. The app will then capture the image and send it to the server. On the server side, a generative AI model will convert the image data from Braille to text data. Finally, the text data will be communicated to the user as an audio alert."

[1609] By inputting this prompt into the generative AI model, the entire process of reading Braille can be explained to visually impaired people, allowing them to understand the specific steps to take.

[1610] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1611] Step 1: Launch the app

[1612] User: Says the voice command to their smartphone: "Launch the Braille app."

[1613] Input: The user's voice command.

[1614] Output: Text data of the voice command.

[1615] Specific actions: The user holds the smartphone close to the microphone and speaks.

[1616] On the device: The voice recognition engine analyzes the voice command and converts it into text data. The converted text data is used to identify the corresponding application and execute the command to launch the application.

[1617] Input: Text data of the voice command.

[1618] Output: Launches a Braille reading app.

[1619] Specific operation: The device's internal voice recognition engine converts the voice into text and searches for and launches the target app.

[1620] Step 2: Capturing Braille

[1621] User: Point the smartphone camera at an object with Braille on it.

[1622] Input: An object with Braille inscribed on it.

[1623] Output: A focused image from the camera.

[1624] Specific actions: The user focuses the camera on the Braille and sees the Braille on the app screen.

[1625] Terminal: The camera captures image data and stores it in memory.

[1626] Input: An image captured by the camera.

[1627] Output: Image data stored in memory.

[1628] Specific operation: The device's camera app takes a photo and stores the data in memory.

[1629] Step 3: Sending image data

[1630] Terminal: Compresses the stored image data and prepares it for transmission to the data processing device (server). Compresses the data and uploads it to the server.

[1631] Input: Image data stored in memory.

[1632] Output: Send compressed image data to the server.

[1633] Specific operation: Data compression and uploading is performed using the communication module inside the terminal.

[1634] Step 4: Analyzing the image data

[1635] Server: Stores the received image data and begins analysis. Performs preprocessing such as noise reduction using an image processing library (e.g., OpenCV).

[1636] Input: Compressed image data.

[1637] Output: Preprocessed image data.

[1638] Specific operation: Image data is stored in the server storage and noise is removed using an image processing library.

[1639] Server: The preprocessed image data is input into the generative AI model, which analyzes the Braille patterns. The generative AI model converts the Braille into character data.

[1640] Input: Preprocessed image data.

[1641] Output: Character data.

[1642] Specific operation: Image data is input into the generative AI model, and Braille analysis and character conversion are performed.

[1643] Step 5: Sending text data

[1644] Server: Sends the generated character data to the terminal.

[1645] Input: Character data.

[1646] Output: Sending character data to the terminal.

[1647] Specific operation: Sends character data to the terminal via the network.

[1648] Step 6: Converting text data into speech

[1649] Terminal: Receives the received text data and passes it to the speech synthesis engine, which converts it into voice data.

[1650] Input: Character data.

[1651] Output: Audio data.

[1652] Specific operation: Converts text data into audio data using Google Text-to-Speech or Apple's VoiceOver function.

[1653] Device: Plays the generated audio data through the speaker.

[1654] Input: Audio data.

[1655] Output: The audio played through the speakers.

[1656] Specific operation: Plays audio data through the device's speaker.

[1657] User: Check the braille content through audio data.

[1658] Input: The audio coming from the device's speaker.

[1659] Output: Information comprehension.

[1660] What happens: The user is near the smartphone and hears a voice notification.

[1661] (Application example 1)

[1662] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1663] Currently, there are limited means for visually impaired people to safely and quickly check security-related information. Even if they can read Braille, they may not be able to understand or communicate it properly. Therefore, there is a need for a system that allows visually impaired people to obtain information accurately and quickly through Braille.

[1664] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1665] In this invention, the server includes a means for pre-processing image data acquired by the camera, such as noise removal, a means including a generation AI for analyzing the image data, identifying Braille, and converting it into text data, and a means for recognizing security-related information based on Braille and notifying it by voice, thereby enabling visually impaired people to accurately and quickly obtain security-related information by voice.

[1666] "Visually impaired persons" refers to people with visual impairments, including those who have difficulty obtaining visual information on their own.

[1667] Braille is a writing system that consists of special symbols that allow visually impaired people to read written information using their sense of touch.

[1668] A "system" refers to a complex of multiple components that interact with each other to achieve a specific function.

[1669] "Voice command" refers to a voice input means for recognizing a user's speech and operating the system based on that instruction.

[1670] "Application" refers to computer software designed to provide a specific function.

[1671] "Camera" refers to an optical device for acquiring image data, and in this context refers specifically to a camera built into a mobile device such as a smartphone.

[1672] "Image data" refers to data obtained by converting visual information acquired by an imaging device such as a camera into digital values.

[1673] A "server" refers to a computer system that provides services such as data processing, storage, and use over a network.

[1674] "Generative AI" refers to algorithms or models that use artificial intelligence techniques to analyze, generate, or transform data.

[1675] "Text data" refers to digital data consisting of characters.

[1676] Converting to "audio" and "reading aloud" refers to the act of converting text data into an audio signal using speech synthesis technology and playing it back through an output device.

[1677] "Security-related information" refers to confidential information such as personal information, authentication codes, and PIN codes, and includes information that must be verified in a secure manner.

[1678] This invention is a system that enables visually impaired people to understand Braille and safely check security-related information. This system uses a smartphone camera and generation AI to allow visually impaired people to understand Braille through audio. Specific embodiments of this system are described below.

[1679] This system consists of a terminal operated by a user, a server that processes data, and an application that runs between them.

[1680] System configuration

[1681] Device:

[1682] It is a smartphone operated by the user, and applications are launched using voice commands. By saying "Launch the Braille app," the voice recognition engine analyzes the voice command and identifies the corresponding application, which then launches the Braille reading app.

[1683] camera:

[1684] The system uses the smartphone's built-in camera: when the user points the camera at an object with Braille, the camera captures the image data and stores it in memory.

[1685] server:

[1686] The acquired image data is sent to the server, where it undergoes preprocessing such as noise removal. The preprocessed image data is then input into a generative AI model, which analyzes the Braille patterns. The recognized Braille characters are converted into text data, which is then sent back to the device.

[1687] Text to speech conversion:

[1688] The received text data is passed to a speech synthesis engine on the device. The speech synthesis engine (for example, gTTS: Google Text-to-Speech) converts the text data into voice data. The generated voice data is played back through the device's speaker.

[1689] Specific examples

[1690] Scenario: Checking the Braille PIN code on your cash card at a bank ATM

[1691] 1. User: Say to the smartphone, "Launch the Braille app."

[1692] 2. Device: The app will launch and display the main screen, with the camera activated and ready to take a photo of an object with Braille.

[1693] 3. User: Point the smartphone camera at the card with the PIN code written on it.

[1694] 4. Terminal: The camera takes a picture of the card.

[1695] 5. Terminal: The acquired image data is compressed and sent to the server.

[1696] 6. Server: Receives the image data and performs preprocessing such as noise removal. Then, the generative AI model analyzes the Braille and generates text data. The text "1234" is generated.

[1697] 7. Server: Sends text data to the device.

[1698] 8. Device: Converts the received text data into speech and reads out "1234."

[1699] 9. User: Listen to the audio and confirm the PIN code.

[1700] This system allows visually impaired people to easily access Braille information, and is particularly important for enabling them to check security-related information safely and quickly.

[1701] Prompt Sentence Examples

[1702] Say "Launch the Braille app." Then point the camera at the Braille. The app will launch and the camera will capture the image. The Braille will be analyzed and the results will be read aloud.

[1703] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1704] Step 1:

[1705] User: Says to the smartphone, "Launch the Braille app." Requests the launch of the application via voice command.

[1706] Input: User's voice command

[1707] Output: Text data of voice commands

[1708] What happens: The smartphone uses a speech recognition engine (e.g., Google Speech Recognition API) to convert the voice command into text and identify the phrase "Launch Braille app."

[1709] Step 2:

[1710] Device: Analyzes voice commands, identifies the corresponding application, and launches it.

[1711] Input: Text data of voice command

[1712] Output: Launching a Braille reading application

[1713] Specific operation: Based on the text data, the smartphone's operating system launches the Braille reading application, the application's main screen is displayed, and the camera becomes available for use.

[1714] Step 3:

[1715] User: Point the smartphone camera at an object with Braille on it.

[1716] Input: An object with braille on it

[1717] Output: Camera image data

[1718] What it does: The user positions the camera in the right position, and the smartphone camera captures an image of the object with Braille on it.

[1719] Step 4:

[1720] Terminal: The camera captures image data and stores it in memory.

[1721] Input: Camera image data

[1722] Output: Saved image data

[1723] How it works: The image sensor captures high-resolution image data and temporarily stores it in the smartphone's memory.

[1724] Step 5:

[1725] Terminal: The stored image data is compressed and sent to the server.

[1726] Input: Saved image data

[1727] Output: Compressed image data uploaded to the server

[1728] Specific operation: The smartphone uses an image compression algorithm (e.g., JPEG compression) to compress the image data and send it to the server over the network.

[1729] Step 6:

[1730] Server: Stores the received image data for analysis and performs preprocessing such as noise removal.

[1731] Input: Compressed image data uploaded to the server

[1732] Output: Preprocessed image data

[1733] What happens: The server decodes the image and applies a noise reduction filter (e.g., median filter) to improve the image quality.

[1734] Step 7:

[1735] Server: Analyzes image data and converts Braille into text data using a generative AI model that identifies Braille.

[1736] Input: Preprocessed image data

[1737] Output: Converted text data

[1738] What it does: Uses a generative AI model (e.g., Tesseract OCR or a custom-trained model) to identify Braille characters from pre-processed images and generate corresponding text data.

[1739] Step 8:

[1740] Server: Sends text data to the device.

[1741] Input: Converted text data

[1742] Output: Text data sent to the terminal

[1743] Specific operation: The server encodes the text data and sends it to the smartphone via the network.

[1744] Step 9:

[1745] Terminal: Converts received text data into speech and reads it to the user.

[1746] Input: Text data sent to the terminal

[1747] Output: Audio data

[1748] Specific operation: The smartphone uses a speech synthesis engine (e.g., gTTS) to convert the text data into audio data and play it back through the speaker.

[1749] Step 10:

[1750] User: Listen to what is read aloud and see what is in braille.

[1751] Input: Audio data

[1752] Output: Security-related information understood

[1753] Specific operation: The user listens to the audio from the smartphone speaker and obtains the information, which allows them to accurately understand the content of the Braille and take the necessary action.

[1754] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1755] The present invention provides a system that helps visually impaired people understand Braille by combining it with an emotion engine that recognizes the user's emotions, making it easier to use and more user-friendly. Specific embodiments of this system are described below.

[1756] System configuration

[1757] This system consists of a smartphone (terminal) operated by the user, a server that processes data, and an application that includes an emotion engine. The application runs on the smartphone and is activated by voice commands. A camera is used to capture an object with Braille printed on it, and the acquired image data is sent to the server. On the server, a generative AI analyzes the Braille from the image data and converts the recognized Braille into text data. Finally, the smartphone converts the text data into speech and reads it to the user. The emotion engine also has the function of recognizing the user's emotions and adjusting the voice output based on those emotions.

[1758] Program processing

[1759] The specific program processing executed by each device will be explained in natural language below.

[1760] Launching the app

[1761] User: Says the voice command to their smartphone: "Launch the Braille app."

[1762] Device: The voice recognition engine analyzes the voice command and identifies the corresponding application. The application launch command is executed, and the Braille reading application is launched.

[1763] Braille capture and transmission

[1764] User: Point the smartphone camera at an object with Braille on it.

[1765] Terminal: The camera acquires image data, stores it in memory, prepares it for transmission to the server, compresses it, and uploads it to the server.

[1766] Image data analysis

[1767] Server: Stores the received image data for analysis. Analysis begins and performs preprocessing such as noise removal. The generation AI inputs the preprocessed image data and analyzes the Braille patterns. The recognized Braille is converted into text data and sent to the device.

[1768] Converting text data into speech

[1769] Device: The device receives the text data and passes it to the speech synthesis engine. The speech synthesis engine converts the text data into voice data, which is then played back from the speaker.

[1770] Emotion engine processing

[1771] On the device: The emotion engine analyzes the user's voice and facial expressions to recognize their current emotional state. For example, it identifies the emotion if the user is anxious.

[1772] On the device: Adjust the tone and speed of the voice output based on the emotions recognized by the emotion engine, providing just-in-time feedback to optimize the voice output for the user.

[1773] Specific examples

[1774] Scenario: Reading Braille on a station sign

[1775] 1. User: Say to the smartphone, "Launch the Braille app."

[1776] 2. Device: The app will launch and the main screen will appear.

[1777] 3. User: Point the camera at the Braille on the station sign.

[1778] 4. Terminal: Operate the camera to capture Braille and obtain image data.

[1779] 5. Terminal: Compresses the image data and sends it to the server.

[1780] 6. Server: Receives the image data and converts the Braille into text using the generation AI. The generated text is, "Platform 1 goes to Tokyo, and platform 2 goes to Shinjuku."

[1781] 7. Server: Sends text data to the device.

[1782] 8. Terminal: Converts the received text data into speech and reads out loud, "Platform 1 goes to Tokyo, and platform 2 goes to Shinjuku."

[1783] 9. On the device: The emotion engine analyzes the user's emotions and adjusts the tone and speed of the voice based on the recognized emotion. For example, if the user is anxious, the voice will be played in a calming tone to give a sense of security.

[1784] 10. User: Listen to the audio and understand the instructions.

[1785] Combining emotion engines in this way will enable visually impaired people to obtain braille information more comfortably, improving the user experience. This is expected to further reduce barriers to obtaining information in daily life and expand the range of activities available to visually impaired people.

[1786] The processing flow will be explained below.

[1787] Step 1:

[1788] User: Says the voice command to their smartphone: "Launch the Braille app."

[1789] A voice recognition engine receives a user's voice command and analyzes the voice data.

[1790] Based on the recognized command, it determines whether to launch a Braille reading application.

[1791] Step 2:

[1792] Terminal: Execute the application launch command to launch the Braille reading app.

[1793] The app will start and the main screen will be displayed.

[1794] The camera control module is ready and waiting to capture braille.

[1795] Step 3:

[1796] User: Point the smartphone camera at an object with Braille on it.

[1797] Refer to the on-screen guidelines and audio guide to point the camera in the correct position.

[1798] Give a voice command for the camera button to prepare for capturing Braille.

[1799] Step 4:

[1800] Terminal: Braille image data is acquired using a camera.

[1801] The camera captures the image in the frame and stores the image data in memory.

[1802] The stored image data undergoes a compression process to prepare it for transmission to the server.

[1803] Step 5:

[1804] Terminal: Sends compressed image data to a server via the Internet.

[1805] Upload image data to the server's API endpoint.

[1806] Monitor the transmission status to ensure that the data was sent successfully.

[1807] Step 6:

[1808] Server: Initiates the process of analyzing the received image data.

[1809] The reception of the data is confirmed and the image data is stored in the database.

[1810] Generative AI performs pre-processing, noise removal, and image enhancement.

[1811] Step 7:

[1812] Server: Preprocessed image data is passed to the generation AI, which then identifies the Braille characters.

[1813] A generative AI model analyzes the image data and recognizes Braille patterns.

[1814] The recognized braille pattern is converted into text data.

[1815] Step 8:

[1816] Server: Sends the generated text data to the device.

[1817] Converts text data into an optimal format and sends it to the device's API endpoint.

[1818] Step 9:

[1819] Terminal: Converts the received text data into audio data and prepares it to be read to the user.

[1820] Check the integrity of the text data and pass it to the speech synthesis engine.

[1821] A speech synthesis engine converts the text data into speech data.

[1822] Step 10:

[1823] On the device: The emotion engine captures voice and facial expression data to analyze the user's emotions.

[1824] It uses voice input and a camera to collect user emotional data.

[1825] The emotion engine analyzes the collected data to identify the current emotional state.

[1826] Step 11:

[1827] Device: Adjusts voice output based on emotion data obtained from the emotion engine.

[1828] If the user seems anxious, calm the voice tone.

[1829] If the user is excited, adjust the speed of the speech output.

[1830] Step 12:

[1831] Device: The generated audio data is played through the device's speaker.

[1832] The audio data is played and read to the user.

[1833] Step 13:

[1834] User: Check the braille content through audio from the device.

[1835] Listen to the audio and get the information you need.

[1836] Example 2

[1837] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1838] Existing technologies for visually impaired people to understand Braille can sometimes make it difficult to read the content of Braille efficiently and accurately. Furthermore, they are unable to provide speech output that takes into account the user's emotional state, which can lead to a decline in the comprehension of information and the quality of the user experience. There is a need to provide a system that can solve this issue and enable visually impaired people to more comfortably acquire and understand Braille information.

[1839] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1840] In this invention, the server includes a means including a generative AI model that analyzes image data and identifies Braille, a means for converting the Braille identified by the generative AI model into text information, and a means for transmitting the converted text information to a terminal. This allows visually impaired people to capture images of Braille with a camera and hear its contents audibly. The terminal also includes a means for analyzing the user's emotional state using an emotion recognition engine and a means for adjusting the tone and speed of the voice based on the analyzed emotional state. This provides optimal audio output for the user, improving the comprehension of information and the quality of the user experience.

[1841] A "voice command" is a method by which a user issues instructions to a system using voice, which is processed using voice recognition technology.

[1842] "Image capture device" refers to a device, such as a camera, that captures visual information as digital image data.

[1843] "Data processing device" refers to a computer system such as a server that analyzes acquired data and converts it into necessary information.

[1844] A "generative AI model" refers to a model that uses artificial intelligence techniques to analyze and recognize data, in this case, to identify Braille.

[1845] "Text information" refers to data that has been converted from coded data such as Braille into a text format that is easily understandable by humans.

[1846] "Terminal" refers to a device that is directly operated by a user, such as a smartphone or tablet.

[1847] A "voice synthesis engine" refers to a technology that converts text data into voice data and provides it to the user as voice information.

[1848] An "emotion recognition engine" refers to a system that uses artificial intelligence technology to analyze and identify a user's emotional state from their voice and facial expressions.

[1849] "Tone" refers to the pitch and intensity of a voice, and is an element that adjusts the voice output according to the user's emotional state.

[1850] "Speed" refers to the speed at which the audio is played back, and is a factor that speeds up or slows down the audio output depending on the user's emotional state.

[1851] This system, which helps visually impaired people understand Braille, is composed as follows: The system consists of a terminal operated by the user, a server that processes data, and an application that includes an emotion recognition engine. The application runs on the terminal and is activated by accepting voice commands.

[1852] When a user issues a voice command to their smartphone (hereinafter referred to as the "device"), such as "Launch the Braille app," a speech recognition engine (e.g., Google Speech-to-Text) analyzes the voice and identifies the application to be launched. Once the application is launched, the main screen is displayed.

[1853] Next, the user points the device's camera at the object with the Braille. The camera module (e.g., Android's Camera API) acquires the image data and temporarily stores it in memory. The stored image data is compressed using a compression algorithm (e.g., JPEG compression) and sent to the server via a network module (e.g., the Retrofit library). The progress of the transmission is displayed on the device.

[1854] The server stores the received image data in memory and preprocesses it using a noise reduction algorithm (e.g., Gaussian filter). A generative AI model (e.g., TensorFlow's OCR engine) analyzes the preprocessed image data and recognizes Braille patterns with high accuracy. The recognized Braille information is then converted into text and sent to the device.

[1855] The device receives the text information and inputs it into a speech synthesis engine (e.g., Google Text-to-Speech). The speech synthesis engine converts the text information into audio data, which is then read aloud to the user through the speaker.

[1856] Furthermore, an emotion recognition engine (e.g., Affectiva SDK) analyzes the user's voice and facial expressions to identify their current emotional state. For example, if the user is anxious, it will identify that emotion. Based on this recognition result, the tone and speed of the voice output are adjusted. For example, an anxious user will be provided with a calming voice tone.

[1857] As a concrete example, consider a scenario in which a user is reading a Braille sign at a train station. When the user says to their smartphone, "Launch the Braille app," the application launches and the main screen appears. Next, the user points the camera at the Braille on the station sign, and the device's camera captures image data and sends it to the server. The server analyzes the Braille and generates text information such as "Platform 1 goes to Tokyo, Platform 2 goes to Shinjuku," which is then sent to the device. The device converts this information into audio data and reads it out loud as "Platform 1 goes to Tokyo, Platform 2 goes to Shinjuku." At this time, an emotion recognition engine analyzes the user's emotional state, and anxious users are provided with information in a calming tone.

[1858] In this way, this system is designed to enable visually impaired people to obtain Braille information comfortably and easily.

[1859] Prompt Sentence Examples

[1860] Below are some examples of prompt sentences used in this system.

[1861] "Launch the Braille app."

[1862] "Point the camera at the Braille and capture the image."

[1863] "Convert Braille to text"

[1864] "Read it aloud"

[1865] "Analyze emotions and adjust tone of voice"

[1866] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1867] Step 1:

[1868] The user issues a voice command to the smartphone, saying, "Launch the Braille app." The input is the user's voice, and the output is the voice data received by the smartphone.

[1869] Step 2:

[1870] The device uses a speech recognition engine (e.g., Google Speech-to-Text) to analyze the voice data and identify the command to "launch a Braille app." The input is the voice data, and the output is the analyzed text data and a launch command. Based on this, the application manager launches the Braille reading app, and the main screen is displayed.

[1871] Step 3:

[1872] The user points the smartphone camera at an object with Braille and determines its position. The input is the user's camera operation, and the output is the adjustment of the camera angle and focus.

[1873] Step 4:

[1874] The device acquires image data using a camera module (e.g., Android's Camera API) and stores the data in temporary memory. The input is the image captured by the camera, and the output is the stored image data.

[1875] Step 5:

[1876] The image data acquired by the terminal is compressed using a compression algorithm (e.g., JPEG compression). The input is uncompressed image data, and the output is compressed image data.

[1877] Step 6:

[1878] The device sends compressed image data to the server using a network module (e.g., Retrofit library). The input is the compressed image data, and the output is the upload to the server.

[1879] Step 7:

[1880] The server stores the received image data in memory. The input is the uploaded image data, and the output is the stored data in the server.

[1881] Step 8:

[1882] The server pre-processes the images with a noise reduction algorithm (e.g., Gaussian filter). The input is the stored image data, and the output is the pre-processed image data.

[1883] Step 9:

[1884] The server inputs the preprocessed image data into a generative AI model (e.g., TensorFlow's OCR engine) to analyze the Braille patterns. The input is the preprocessed image data, and the output is the Braille recognition data.

[1885] Step 10:

[1886] The server converts the recognized Braille information into text information and sends the text information to the terminal. The input is the Braille recognition data, and the output is the transmission of text information to the terminal.

[1887] Step 11:

[1888] The text information received by the device from the server is input into a speech synthesis engine (e.g., Google Text-to-Speech) and converted into voice data. The input is text information, and the output is the generated voice data.

[1889] Step 12:

[1890] The device plays the generated audio data from a speaker and reads the braille content aloud to the user. The input is audio data, and the output is audio playback.

[1891] Step 13:

[1892] The device uses an emotion recognition engine (e.g., Affectiva SDK) to analyze the user's voice and facial expressions to determine their current emotional state. The input is the user's facial expression and voice data, and the output is the analyzed emotional state.

[1893] Step 14:

[1894] The device adjusts the tone and rate of the audio output based on the recognized emotional state. The input is the analyzed emotional state, and the output is the adjusted audio output. For example, if the user appears anxious, audio will be played in a calming tone.

[1895] (Application example 2)

[1896] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1897] Conventional systems for visually impaired people to understand Braille have the problem of limited user experience because they do not take into account the user's emotions when reading Braille. Also, because the voice output is uniform, there is an issue that it is not possible to respond appropriately according to the user's mental state or emotions. As a result, ease of use for visually impaired people has not been sufficiently considered.

[1898] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for launching an application using a voice command, means for acquiring Braille image data using a camera, means for transmitting the acquired image data to the server, means including a generation AI on the server side for analyzing the image data and identifying Braille, means for converting the Braille identified by the generation AI into text data, means for transmitting the converted text data to the terminal, means for converting the received text data on the terminal side into voice and reading it aloud, and means for analyzing the user's emotions using an emotion engine and adjusting the voice output. This makes it possible for visually impaired people to acquire Braille information more comfortably and provides a user-friendly system.

[1899] "Visually impaired people" refers to people whose visual function is impaired and who have difficulty obtaining information through their vision.

[1900] Braille is a special arrangement of dots that allows visually impaired people to understand letters using their sense of touch.

[1901] A "system" is a collection of components that enable multiple devices and software to work together to achieve a specific function.

[1902] "Voice command" means an instruction given by a user to operate a device or application by speaking.

[1903] "Application" means a software program that runs on a computer or device to provide a particular function or service.

[1904] A "camera" is an optical device for capturing images or video.

[1905] "Image data" means image information represented in digital form.

[1906] A "server" is a computer system that provides services to other computers over a network.

[1907] "Generative AI" refers to artificial intelligence that uses machine learning techniques to generate specific information from data.

[1908] "Text data" means data that represents character information in a digital format.

[1909] An "emotion engine" is a system that analyzes a user's voice and facial expressions to recognize their current emotional state and generate appropriate feedback based on that.

[1910] "Adjusting the audio output" means changing the tone and rate of the audio depending on the user's emotional state.

[1911] "Noise reduction" refers to the process of removing unnecessary information and noise from image or audio data.

[1912] "Preprocessing" refers to processing to prepare data before the main processing is performed.

[1913] "Speech synthesis engine" refers to a software component that converts text data into speech.

[1914] This invention provides a system for visually impaired people to understand Braille, which is easier to use and more user-friendly by recognizing the user's emotions and adjusting the voice output. This system includes the following processes: launching an application via voice command, acquiring Braille image data using a camera, transmitting the image data to a server, identifying the Braille using AI generated on the server, converting it to text data, transmitting the text data to a terminal, converting it to voice and reading it out loud, and analyzing the user's emotions and adjusting the voice output using an emotion engine. Specific embodiments of this system are described below.

[1915] Hardware and software used

[1916] Hardware: Smartphone (with camera)

[1917] Software: Android / iOS app, server (cloud server), generative AI (generative AI model), speech synthesis engine (speech synthesis software), emotion engine (emotion analysis software)

[1918] System configuration

[1919] Launching the application

[1920] The user issues a voice command to their smartphone, saying, "Launch the Braille app." The device's voice recognition engine analyzes this voice command and identifies the corresponding application. The launch command for the identified application is executed, and the Braille reading app launches.

[1921] Braille capture and transmission

[1922] When a user points the smartphone camera at an object with Braille, the camera captures the image data and stores it in memory, where it is compressed and uploaded to the server.

[1923] Image data analysis

[1924] The server analyzes the received image data and performs preprocessing such as noise removal. The generation AI then inputs the preprocessed image data and analyzes the Braille patterns. The Braille recognized as a result of the analysis is converted into text data and sent to the device.

[1925] Converting text data into speech

[1926] The device receives the text data and passes it to a speech synthesis engine. The speech synthesis engine converts the text data into audio data, which is then played back through the device's speaker. This allows the user to hear the Braille content aloud.

[1927] Emotion engine processing

[1928] The device's emotion engine analyzes the user's voice and facial expressions to recognize their current emotional state. For example, if the user is feeling anxious, it will identify that emotion. Based on the emotion recognized by the emotion engine, the tone and speed of the voice output are adjusted in real time, providing the most appropriate voice output for the user.

[1929] Examples of concrete examples and prompts

[1930] Scenario: Reading Braille on a station sign

[1931] 1. User: Says to the smartphone, "Launch the Braille app."

[1932] 2. Device: The app will launch and the main screen will appear.

[1933] 3. User: Point the camera at the Braille on the station sign.

[1934] 4. Terminal: Operate the camera to capture Braille and obtain image data.

[1935] 5. Terminal: Compresses the image data and sends it to the server.

[1936] 6. Server: Receives the image data and converts the Braille into text using a generation AI. This generates text data such as "Platform 1 goes to Tokyo, and platform 2 goes to Shinjuku."

[1937] 7. Server: Sends text data to the device.

[1938] 8. Terminal: Converts the received text data into speech and reads out loud, "Platform 1 goes to Tokyo, and platform 2 goes to Shinjuku."

[1939] 9. On the device: The emotion engine analyzes the user's emotions and adjusts the tone and speed of the voice based on the recognized emotion. For example, if the user is anxious, the voice will be played in a calming tone to give a sense of security.

[1940] 10. User: Listen to the audio and understand the instructions.

[1941] Prompt Sentence Examples

[1942] Speech prompt: "Launch the Braille app"

[1943] Generative AI prompt: "Convert this Braille image into text."

[1944] Text-to-speech prompt: "Convert this text to speech: 'This product is coffee'"

[1945] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1946] Step 1:

[1947] The user issues a voice command to their smartphone saying, "Launch the Braille app." The device's voice recognition engine analyzes this voice command, identifies the corresponding application, and executes the launch command. This launches the Braille reading app. The input is the user's voice command, and the output is the launch of the application.

[1948] Step 2:

[1949] When a user points a smartphone camera at an object with Braille, the device's camera captures image data and stores it in memory. The input is the camera image, and the output is Braille image data.

[1950] Step 3:

[1951] The device compresses the acquired image data and prepares it for uploading to the server. The data is sent to the server and analysis begins. The input is Braille image data, and the output is compressed image data sent to the server.

[1952] Step 4:

[1953] The server analyzes the received image data and performs preprocessing using generative AI to identify Braille, including noise removal. The input is the image data sent to the server, and the output is the preprocessed image data.

[1954] Step 5:

[1955] The server passes the preprocessed image data to the generation AI, which analyzes the Braille patterns. The Braille recognized as a result of this analysis is converted into text data. The input is preprocessed image data, and the output is text data.

[1956] Step 6:

[1957] The server sends the converted text data to the terminal. The input is the text data generated by the server, and the output is the text data sent to the terminal.

[1958] Step 7:

[1959] The device passes the received text data to the speech synthesis engine. The speech synthesis engine converts this text data into voice data, and the generated voice data is played back from the device's speaker. The input is text data, and the output is voice data.

[1960] Step 8:

[1961] The device's emotion engine analyzes the user's voice and facial expressions to recognize their current emotional state. For example, if the user looks anxious, it identifies that emotion. The input is the user's voice and facial expression data, and the output is the recognized emotion.

[1962] Step 9:

[1963] The tone and speed of the voice output are adjusted in real time based on the emotion recognized by the emotion engine, providing the optimal voice output for the user. The input is the recognized emotion, and the output is the adjusted voice output.

[1964] These steps will enable visually impaired people to obtain braille information more comfortably and create a user-friendly system.

[1965] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1966] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1967] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1968] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1969] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1970] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1971] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1972] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1973] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1974] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1975] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1976] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1977] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1978] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1979] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1980] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1981] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1982] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1983] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1984] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1985] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1986] The following is further disclosed regarding the above embodiment.

[1987] (Claim 1)

[1988] A system for visually impaired people to understand Braille,

[1989] means for launching an application using a voice command;

[1990] means for acquiring image data of Braille using a camera;

[1991] means for transmitting the acquired image data to a server;

[1992] A means including a generating AI that analyzes image data on the server side and identifies Braille;

[1993] A means for converting the Braille identified by the generation AI into text data;

[1994] means for transmitting the converted text data to a terminal;

[1995] A means for converting the text data received on the terminal into voice and reading it out loud;

[1996] A system including:

[1997] (Claim 2)

[1998] 10. The system of claim 1, further comprising means for pre-processing image data acquired by the camera, such as noise reduction.

[1999] (Claim 3)

[2000] 10. The system of claim 1, further comprising means for accepting an operation from a user by a voice command.

[2001] "Example 1"

[2002] (Claim 1)

[2003] A system for visually impaired people to understand Braille,

[2004] means for launching an application using a voice command;

[2005] means for acquiring image data of Braille using an image acquisition device;

[2006] means for transmitting the acquired image data to a data processing device;

[2007] A means including a generative AI model for analyzing image data and identifying Braille characters on the data processing device side;

[2008] A means for converting the Braille identified by the generative AI model into character data;

[2009] means for transmitting the converted character data to a terminal;

[2010] A means for converting the received text data into voice and reading it out loud on the terminal side;

[2011] A system including:

[2012] (Claim 2)

[2013] 10. The system of claim 1, further comprising means for pre-processing the image data acquired by the image acquisition device, such as noise removal.

[2014] (Claim 3)

[2015] 10. The system of claim 1, further comprising means for accepting an operation from a user by a voice command.

[2016] "Application Example 1"

[2017] (Claim 1)

[2018] A system for visually impaired people to understand Braille,

[2019] means for launching an application using a voice command;

[2020] means for acquiring image data of Braille using a camera;

[2021] means for transmitting the acquired image data to a server;

[2022] A means including a generating AI that analyzes image data on the server side and identifies Braille;

[2023] A means for converting the Braille identified by the generation AI into text data;

[2024] means for transmitting the converted text data to a terminal;

[2025] A means for converting the text data received on the terminal into voice and reading it out loud;

[2026] means for recognizing and audibly announcing braille-based security-related information;

[2027] A system including:

[2028] (Claim 2)

[2029] 10. The system of claim 1, further comprising means for pre-processing image data acquired by the camera, such as noise reduction.

[2030] (Claim 3)

[2031] 10. The system of claim 1, further comprising means for accepting an operation from a user by a voice command.

[2032] "Example 2: Combining Emotion Engines"

[2033] (Claim 1)

[2034] A system for visually impaired people to understand Braille,

[2035] means for launching an application using a voice command;

[2036] means for acquiring image data of Braille using an image acquisition device;

[2037] means for transmitting the acquired image data to a data processing device;

[2038] A means including a generative AI model for analyzing image data and identifying Braille characters on the data processing device side;

[2039] A means for converting the Braille identified by the generative AI model into text information;

[2040] means for transmitting the converted character information to a terminal;

[2041] A means for converting the received text information into voice and reading it out loud on the terminal side;

[2042] means for analyzing a user's emotional state using an emotion recognition engine;

[2043] means for adjusting the tone and rate of the speech based on the analyzed emotional state;

[2044] A system including:

[2045] (Claim 2)

[2046] 10. The system of claim 1, further comprising means for pre-processing the image data acquired by the image acquisition device, such as noise removal.

[2047] (Claim 3)

[2048] 10. The system of claim 1, further comprising means for accepting an operation from a user by a voice command.

[2049] "Application example 2 when combining emotion engines"

[2050] (Claim 1)

[2051] A system for visually impaired people to understand Braille,

[2052] means for launching an application using a voice command;

[2053] means for acquiring image data of Braille using a camera;

[2054] means for transmitting the acquired image data to a server;

[2055] A means including a generating AI that analyzes image data on the server side and identifies Braille;

[2056] A means for converting the Braille identified by the generation AI into text data;

[2057] means for transmitting the converted text data to a terminal;

[2058] A means for converting the text data received on the terminal into voice and reading it out loud;

[2059] means for analyzing a user's emotions with an emotion engine and adjusting the voice output;

[2060] A system including:

[2061] (Claim 2)

[2062] 10. The system of claim 1, further comprising means for pre-processing image data acquired by the camera, such as noise reduction.

[2063] (Claim 3)

[2064] 10. The system of claim 1, further comprising means for accepting an operation from a user by a voice command. [Explanation of symbols]

[2065] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A system for visually impaired people to understand Braille, means for launching an application using a voice command; means for acquiring image data of Braille using a camera; means for transmitting the acquired image data to a server; A means including a generating AI that analyzes image data on the server side and identifies Braille; A means for converting the Braille identified by the generation AI into text data; means for transmitting the converted text data to a terminal; A means for converting the text data received on the terminal into voice and reading it out loud; A system including:

2. The system of claim 1 , further comprising means for pre-processing image data acquired by the camera, such as noise reduction.

3. The system of claim 1 , further comprising means for accepting an operation from a user by a voice command.

Images