System

Abstract

A system is provided.SOLUTION: A visual aid device worn by a user, the system comprising: means for capturing an image containing characters; means for extracting the characters from the captured image; means for transmitting the extracted characters to a remote computing device for translation; means for receiving translation results from the remote computing device; and means for audibly guiding the user through the received translation results.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In today's globalized society, communication between people who speak different languages ​​is important. However, many people face language barriers, which hinder information acquisition, especially when traveling or doing business abroad. For the visually impaired, acquiring information in daily life and while traveling is even more difficult. To address these challenges, there is a need for the development of visual aid devices that combine language translation and visual assistance. [Means for solving the problem]

[0005] One aspect of the present invention is a visual aid device worn by a user. The device is a system including means for capturing an image containing text, means for extracting text from the captured image, means for transmitting the extracted text to a remote computing device for translation, means for receiving a translation result from the remote computing device, and means for providing an audio guide to the user of the received translation result. The system also includes means for detecting spoken words and capturing audio data, means for transmitting the captured audio data to the remote computing device, means for receiving translated audio data from the remote computing device, and means for providing an audio guide to the user of the received translation result. The system also includes a setting means for changing the audio guide of the translation result to the voice of a celebrity selected by the user. This configuration not only overcomes language barriers and facilitates communication between people who speak different languages, but also provides support for the visually impaired.

[0006] A "visual aid device" is an electronic device worn by a user to assist with visual information and provide visual translation and guidance.

[0007] "Means for capturing images" refers to a function that uses a camera or sensor to obtain visual information about the surroundings as a digital image.

[0008] The "means for extracting characters" is a function that recognizes character information from a captured image and generates text data.

[0009] A "remote computing device" is an external server or computer connected via the Internet to receive, process, or translate data from the device.

[0010] The "means for receiving the translation result" is a communication function for receiving the translated data sent from the remote computing device.

[0011] The "means for providing guidance to the user by voice" refers to an audio output device such as a speaker or earphones that synthesizes the translation results and other information into voice and provides the user with the audio.

[0012] "Means for capturing audio data" refers to a function that records surrounding sounds and conversations using a microphone or the like and acquires them as digital audio data.

[0013] The "means for presenting translated audio to the user" is a function for playing back the translated audio data transmitted from the remote computing device and audibly providing it to the user.

[0014] The "setting means for changing to a famous person's voice" is an option setting function that allows the user to change the voice guidance to the voice of a specific famous voice actor or actor.

[0015] A "network" is a communications infrastructure, including the Internet, that devices and remote computing devices use to exchange data.

[0016] "Digital voice data" refers to data in which analog voice is recorded in digital format and is used for translation and voice guidance.

[0017] "Speech synthesis" is a technology that generates synthetic speech based on text data and plays it back as a natural voice. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4]FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0019] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0020] First, the terms used in the following description will be explained.

[0021] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0022] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0023] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0024] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0026] [First embodiment]

[0027] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0028] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0031] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0034] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0038] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0039] This invention is a system that automatically translates different languages ​​and provides audio guidance as a visual aid device worn by the user. This system can acquire visual information, translate it in real time, and provide audio guidance. It also has the function of translating spoken language and providing audio guidance.

[0040] This visual assistance device (hereinafter referred to as the device) is in the form of glasses worn by the user and contains a camera, microphone, speaker, communication module, and processor. The device supplements its processing capabilities by communicating with a remote computing device (hereinafter referred to as the server).

[0041] 1. Visual translation and audio guidance

[0042] Camera and character extraction

[0043] User:

[0044] The device recognizes objects with text information, such as signs and menus, in the user's field of vision.

[0045] Device:

[0046] The device's camera periodically captures images of its surroundings, which are then analyzed by a processor to extract textual information.

[0047] Sending translation requests and receiving translation results

[0048] Device:

[0049] The extracted text information is sent to the server via the communication module, and the data sent also includes information about the language to be translated.

[0050] server:

[0051] The server analyzes the received text and translates it into the specified language. After generating the translation results, it constructs them as text for voice guidance. It then converts the text into audio data using the voice of a celebrity selected by the user and sends it to the device.

[0052] Device:

[0053] The received audio data is played back and the translation results are presented to the user.

[0054] Specific examples

[0055] User:

[0056] Suppose a Japanese-speaking user is visiting France and looking for a restaurant.

[0057] Device:

[0058] A camera captures a sign that says "Restaurant," and a character recognition process extracts the word "Restaurant." The extracted character data is sent to a server.

[0059] server:

[0060] The server receives the character "restaurant" and translates it into English as "Restaurant." Based on the translation, it generates a narration saying "This place is a restaurant. Recommended dishes are pasta and steak." This is converted into audio data and sent to the device.

[0061] Device:

[0062] The voice guidance played back informs the user that "This is a restaurant. Recommended dishes are pasta and steak."

[0063] 2. Spoken language translation and audio guidance

[0064] Microphone and voice data capture

[0065] User:

[0066] Imagine a situation where a user is communicating with local people in a foreign market.

[0067] Device:

[0068] The device's microphone constantly monitors the surrounding sound and captures audio data, such as conversations, which are then stored as audio data.

[0069] Sending translation requests and receiving translation results

[0070] Device:

[0071] The stored voice data is transmitted to a server via a communication module.

[0072] server:

[0073] The server receives the voice data, converts it into text using speech recognition technology, then translates the text into the specified language and generates the translation as voice data, which is then sent to the device.

[0074] Device:

[0075] The received audio data is played back and the translation results are presented to the user.

[0076] Specific examples

[0077] User:

[0078] Consider a scenario in which a Japanese-speaking user in an American market asks "Where is the nearest station?" to get directions.

[0079] Device:

[0080] A microphone captures the user's question and transmits the audio data to the server.

[0081] server:

[0082] The server receives the speech data "Where is the nearest station?" and translates it into "Where is the nearest station?". It generates the translated speech data and sends it to the device.

[0083] Device:

[0084] The audio guide played allows the user to ask local people, "Where is the nearest station?"

[0085] In this way, the present invention allows users to translate visual and audio information in real time and provide appropriate guidance, thereby enabling smooth communication even in environments where different languages ​​are spoken. Furthermore, by providing audio guidance in the voice of a celebrity selected by the user, a more friendly service can be provided.

[0086] The processing flow will be explained below.

[0087] Processing flow for translation of visual information and voice guidance

[0088] Visual translation

[0089] Step 1:

[0090] User:

[0091] A user wearing the device looks at visual information, such as a sign or a menu.

[0092] Step 2:

[0093] Device:

[0094] The device's camera periodically captures images of its surroundings, which are then sent to a processor.

[0095] Step 3:

[0096] Device:

[0097] The processor uses image analysis algorithms to extract text information from the captured image, for example, using optical character recognition (OCR) technology to convert the characters into text data.

[0098] Step 4:

[0099] Device:

[0100] A request is generated to transmit the extracted character data and the linguistic information to be translated to a remote computing device (server).

[0101] Step 5:

[0102] Device:

[0103] The generated request is sent to the server through a communication module.

[0104] Step 6:

[0105] server:

[0106] The server receives the request, parses the text, and feeds the text into the corresponding translation model to perform translation into the specified language.

[0107] Step 7:

[0108] server:

[0109] Once the translation results are generated, a narration is created based on the translated text. For example, it generates a sentence like, "This is a cafe. The recommended menu item is latte."

[0110] Step 8:

[0111] server:

[0112] To convert the narration text into audio data, text-to-speech synthesis (TTS) is performed using the voice of a celebrity selected by the user.

[0113] Step 9:

[0114] server:

[0115] The generated voice data is transmitted to the terminal.

[0116] Step 10:

[0117] Device:

[0118] The terminal plays back the audio data received from the server and provides the user with the translation results by voice.

[0119] Specific examples

[0120] User:

[0121] A Japanese speaking user sees a sign in French.

[0122] Device:

[0123] The camera captures the sign and uses OCR technology to extract the word "restaurant."

[0124] server:

[0125] The server receives "restaurant" and translates it as "restaurant." It then generates a narration saying, "This is a restaurant. The recommended dish is pasta," converts it into voice data, and sends it to the device.

[0126] Device:

[0127] The device plays the audio data and informs the user of the translation results. The voice will say in Japanese, "This is a restaurant. The recommended dish is pasta."

[0128] ---

[0129] Spoken language translation and voice guidance processing flow

[0130] Audio translation

[0131] Step 1:

[0132] User:

[0133] The user puts on the device and attempts to converse with local people.

[0134] Step 2:

[0135] Device:

[0136] The device's microphone constantly monitors the surrounding sounds and captures what is being said.

[0137] Step 3:

[0138] Device:

[0139] The captured audio data is sent to a processor.

[0140] Step 4:

[0141] Device:

[0142] A processor temporarily stores the audio data and generates a request to send it to a server via a communication module.

[0143] Step 5:

[0144] Device:

[0145] The generated request is sent to the server through a communication module.

[0146] Step 6:

[0147] server:

[0148] The server receives the voice data and converts it into text using speech recognition technology, for example, "Where is the nearest station?"

[0149] Step 7:

[0150] server:

[0151] Translate the recognized text into the specified language, for example, "Where is the nearest station?" into English.

[0152] Step 8:

[0153] server:

[0154] To generate audio data based on the translated text, TTS is used to create audio data in a voice selected by the user.

[0155] Step 9:

[0156] server:

[0157] The generated voice data is transmitted to the terminal.

[0158] Step 10:

[0159] Device:

[0160] The terminal plays back the audio data received from the server and provides the user with the translation results by voice.

[0161] Specific examples

[0162] User:

[0163] A Japanese speaking user wants to ask for directions in the United States.

[0164] Device:

[0165] The microphone captures the user's question, "Where is the nearest station?" and sends the voice data to the server.

[0166] server:

[0167] The server receives the voice data, converts it into text, and translates it. The text "Where is the nearest station?" is converted into voice data and sent to the device.

[0168] Device:

[0169] The device will play the translated audio and the user can ask a local person, "Where is the nearest station?"

[0170] The above is the specific processing flow of this system, and the operation at each step is explained in detail.

[0171] Example 1

[0172] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0173] In modern society, people who speak different languages ​​coexist, and smooth communication is essential. However, language barriers make it difficult to understand signs and menus, or to converse smoothly with local people, both in daily life and while traveling. To solve these problems, a visual aid device is needed that can translate in real time and provide audio guidance.

[0174] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0175] In this invention, the server includes means for capturing an image containing text, means for extracting text from the captured image, means for translating the extracted text, means for receiving a translation result from a remote computing device, means for providing audible guidance to a user about the received translation result, and means for converting the received translation result into audio data in a specified voice, thereby enabling real-time translation of text and conversations displayed in different languages ​​and audio guidance.

[0176] 1. "Visual assistance device" means a device worn by a user that has the function of acquiring, processing, and providing visual information.

[0177] 2. "Capturing means" is a collective term for the hardware and software that a device uses to capture images and audio.

[0178] 3. "Text extraction means" means a process or device that has the ability to identify text information from a captured image and extract it as data.

[0179] 4. "Remote Computing Device" means a device that provides remotely located computing resources for processing data transmitted from a Device.

[0180] 5. "Translation means" means software and hardware functions for converting written or spoken information in one language into another language.

[0181] 6. "Means for receiving translation results" means a device that has the function of receiving translation data sent from a remote computing device on the device side.

[0182] 7. "Means for providing audio guidance to the user" refers to the device's function of converting the received translation results into audio format and notifying the user by audio.

[0183] 8. "Means for converting into voice data using a specified voice" means a device that has the function of converting the translation results into voice data using a specific voice that the user has specified in advance.

[0184] 9. "Audio data capture means" means a device or process that captures a user's speech or surrounding sounds and stores them as digital audio data.

[0185] 10. "Means for receiving translated voice data" means a device that has the capability to receive translated voice data sent from a remote computing device.

[0186] 11. "Means for providing information to the user" refers to functions or devices for providing acquired or received information to the user in an easy-to-understand manner.

[0187] 12. "Configuration means" means the interface and software functionality that allows a user to customize the operation and functionality of a device to suit the user's preferences.

[0188] This invention provides a multilingual visual aid device, enabling users to translate visual and audio information in real time and provide audio guidance in different language environments. This invention achieves this by using a capture device, a remote server, speech synthesis technology, and a communication module.

[0189] Hardware and software configurations used

[0190] 1. Wearable visual aids:

[0191] Camera: Captures visual information about the surroundings.

[0192] Microphone: Captures surrounding sounds.

[0193] Speaker: A means of providing the translation results to the user aloud.

[0194] Communications Module: A means for sending and receiving data to and from a remote computing device.

[0195] Processor: Processes image and audio data.

[0196] 2. Remote Computing Device (Server):

[0197] Character recognition software: Extracts text information from images (e.g., Tesseract OCR).

[0198] Speech recognition technology: Converts captured voice data into text (e.g., Google Speech-to-Text).

[0199] Translation engine: Translates text from one language to another (e.g., Google Translate API, DeepL).

[0200] Speech synthesis engine: Converts text to speech (e.g., Amazon Polly, Google Text-to-Speech).

[0201] System Operation

[0202] Visual translation and audio guidance

[0203] A visual assistance device worn by a user captures visual information from the surroundings using a camera. For example, the camera may capture a photo of a sign for a French restaurant. The processor of the terminal (device) analyzes the captured image and uses character recognition software to extract the text information "restaurant." This text information is then sent to a server via a communications module. The server analyzes the received text data, translates it into the specified language (e.g., from French to English), and outputs the translation as "Restaurant." Based on the translation, the server then generates a narration such as "This place is a restaurant. Recommended dishes are pasta and steak." This is converted into audio data using a speech synthesis engine and sent to the device. The terminal (device) receives the audio data and plays it through a speaker, so the user is informed that "This place is a restaurant. Recommended dishes are pasta and steak."

[0204] Spoken language translation and audio guidance

[0205] Consider a scenario in which a user in a foreign market asks, "Where is the nearest station?" A microphone installed on the terminal (device) captures surrounding sounds and sends the voice data to a server. The server receives the voice data and converts it into text using voice recognition technology. It then translates the text into a specified language (e.g., Japanese to English), generates voice data based on the translation results, and sends it to the device. The terminal (device) receives the voice data and asks the user, "Where is the nearest station?"

[0206] Examples of concrete examples and prompts

[0207] Specific examples

[0208] A Japanese-speaking user visits France and a camera captures a sign that says "Restaurant." The text data "Restaurant" is sent to the server. The server translates "Restaurant" into "Restaurant," generates a narration saying, "This place is a restaurant. Recommended dishes are pasta and steak.", and sends this voice data to the device. By listening to this voice guidance, the user learns, "This place is a restaurant. Recommended dishes are pasta and steak."

[0209] Prompt Sentence Examples

[0210] "Translate the text written on the sign. Generate a prompt to translate from French to English: 'Extract text from captured image and translate it into the specified language.'"

[0211] "Generate a prompt that translates the Japanese question 'Where is the nearest station?' into English: 'Use speech recognition technology to convert captured speech into text, and then translate that text into the specified language.'"

[0212] In this way, the present invention is a system that can translate visual and audio information in real time in a multilingual environment and provide audio guidance to the user.

[0213] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0214] Visual translation and audio guidance

[0215] Step 1: Capture visual information

[0216] Device:

[0217] While a user is wearing a visual aid device, the device's camera periodically captures visual information about the surroundings.

[0218] Input: Objects in the user's field of view (e.g., signs, menus)

[0219] Data processing: The image capture process is activated and the camera acquires the image data.

[0220] Output: Acquired raw image data

[0221] What happens: The device's camera takes a picture every second.

[0222] Step 2: Extracting characters

[0223] Device:

[0224] The processor analyzes the captured image data in real time and extracts text information using optical character recognition software (OCR).

[0225] Input: Captured image data

[0226] Data processing: Extracting text data from images using OCR algorithms (e.g., Tesseract).

[0227] Output: Extracted text data (e.g., "restaurant")

[0228] What it does: The processor recognizes characters in the image and generates data in text format.

[0229] Step 3: Submit a translation request

[0230] Device:

[0231] The extracted text data and the language information to be translated are sent to the server via the communication module.

[0232] Input: Extracted text data, language information to be translated (e.g. French to English)

[0233] Data processing: Constructing data packets and sending them to the server using a communication protocol (e.g., HTTP).

[0234] Output: Data packet sent to the server

[0235] What happens: The device sends the text "restaurant" and a translation request from French to English.

[0236] Step 4: Translation and Narration Generation

[0237] server:

[0238] The received text data is analyzed and translated into the specified language, and a narration is generated based on the translation results.

[0239] Input: Text data (e.g., "restaurant"), language information to be translated

[0240] Data processing: Translate to "Restaurant" using a translation engine (e.g., Google Translate API). Then generate a voiceover sentence: "This place is a restaurant. Recommended dishes are pasta and steak."

[0241] Output: Translated text and narration

[0242] Specific operation: The server translates "restaurant" to "Restaurant" and constructs a guide text.

[0243] Step 5: Generate audio data

[0244] server:

[0245] The narration sentences are converted into voice data using a voice synthesis engine.

[0246] Input: Narration (e.g., "This place is a restaurant. Recommended dishes are pasta and steak.")

[0247] Data processing: Generate voice data using a speech synthesis engine (e.g., Amazon Polly).

[0248] Output: Audio data

[0249] Specific operation: The server generates audio based on the narration text.

[0250] Step 6: Sending voice data and guidance

[0251] Device:

[0252] The audio data is received from the server and played back to the user through the speaker.

[0253] Input: Audio data received from the server

[0254] Data processing: Decodes the received data and plays it on the speaker.

[0255] Output: A voice prompt to the user

[0256] What happens: The speaker plays a voice saying, "This place is a restaurant. Recommended dishes are pasta and steak."

[0257] ---

[0258] Spoken language translation and audio guidance

[0259] Step 1: Capture audio data

[0260] Device:

[0261] The microphone captures the surrounding sounds.

[0262] Input: User speech and surrounding audio (e.g., "Where is the nearest station?")

[0263] Data processing: Recording audio data and converting it into digital format.

[0264] Output: Captured audio data

[0265] What it does: A microphone records the user's question.

[0266] Step 2: Sending audio data

[0267] Device:

[0268] The captured audio data is transmitted to a server via a communication module.

[0269] Input: Audio data

[0270] Data processing: Constructing data packets and sending them to the server using a communication protocol.

[0271] Output: Audio data sent to the server

[0272] Specific operation: The device sends voice data saying "Where is the nearest station?"

[0273] Step 3: Transcribe and translate the audio data

[0274] server:

[0275] The received voice data is converted into text using voice recognition technology and then translated.

[0276] Input: Speech data (e.g., "Where is the nearest station?")

[0277] Data processing: Use speech recognition technology (e.g., Google Speech-to-Text) to convert the speech to text, then use a translation engine to translate it into "Where is the nearest station?"

[0278] Output: Translated text data

[0279] What happens: The server converts the voice data into text and translates it into English.

[0280] Step 4: Generate translated audio data

[0281] server:

[0282] The translated text data is converted into voice data using a voice synthesis engine.

[0283] Input: Translated text data (e.g., "Where is the nearest station?")

[0284] Data processing: Generate voice data using a speech synthesis engine.

[0285] Output: Audio data

[0286] Specific operation: The server generates audio data based on the translation results.

[0287] Step 5: Voice prompts

[0288] Device:

[0289] The generated audio data is received and played back to the user through a speaker.

[0290] Input: Audio data

[0291] Data processing: Decodes the received data and plays it on the speaker.

[0292] Output: A voice prompt to the user

[0293] Specific action: The speaker will play a voice saying "Where is the nearest station?"

[0294] In this way, the system captures visual and audio information in real time, translates it, and provides audio guidance, thereby providing smooth communication that transcends language barriers.

[0295] Examples of concrete examples and prompts

[0296] Specific examples

[0297] A Japanese-speaking user uses a camera to capture a sign that says "restaurant" in a French restaurant, and the text data "restaurant" is sent to the server. The server translates "restaurant" into "Restaurant" and generates a narration saying, "This place is a restaurant. Recommended dishes are pasta and steak." This is sent as audio data to the device. After listening to the audio guidance, the user learns, "This place is a restaurant. Recommended dishes are pasta and steak."

[0298] A Japanese-speaking user in an American market asks, "Where is the nearest station?" The device captures the speech and sends it to the server. The server translates "Where is the nearest station?" to "Where is the nearest station?" and sends it as voice data to the device. The device plays the speech and tells the user, "Where is the nearest station?"

[0299] Prompt Sentence Examples

[0300] "Translate the text written on the sign. Generate a prompt to translate from French to English: 'Extract text from captured image and translate it into the specified language.'"

[0301] "Generate a prompt that translates the Japanese question 'Where is the nearest station?' into English: 'Use speech recognition technology to convert captured speech into text, and then translate that text into the specified language.'"

[0302] This system is designed to support smooth information acquisition and communication in a multilingual environment.

[0303] (Application example 1)

[0304] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0305] There is a problem in that users who speak different languages ​​cannot communicate smoothly in autonomous vehicles (e.g., self-driving vehicles). It is also difficult to understand visual information such as signs and road signs written in different languages, which can make travel in a self-driving vehicle unstable. Furthermore, local guidance and responses to questions cannot be provided immediately, which causes great stress for tourists and foreign users.

[0306] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0307] In this invention, the server includes a processing device for translating visual information, a processing device for translating audio information, and a processing device for converting received translation results into audio data. This enables real-time translation of visual and audio information in different languages ​​within an autonomous mobile body. The translation results are provided to the user via audio guidance via the visual assistance system, allowing the user to quickly understand route guidance to their destination and emergency information. In addition, user-selected audio guidance is provided in a user-friendly format.

[0308] A "visual aid system" is a device worn by a user that captures visual information and provides the information to the user visually or audibly.

[0309] "Visual information" refers to information such as characters and figures that are included in the user's field of vision.

[0310] A "remote processor" is a remotely located computing device that processes data sent from a visual aid system and returns the results.

[0311] An "autonomous mobile object" is a vehicle or device that can move autonomously and reach its destination using artificial intelligence and sensor technology, such as a self-driving vehicle.

[0312] "Audio data" refers to audio signals captured by an audio input device such as a microphone and converted into digital data.

[0313] "Translation result" is text or audio data that has been converted into a different language by a remote processor.

[0314] "Famous person's voice" is voice data of a commonly recognized person selected by the user.

[0315] "Guidance means" refers to a mechanism or device that allows the visual aid system to provide information to the user.

[0316] A "processing device" is a computer or hardware for processing and analyzing data.

[0317] "Friendly voice guidance" refers to voice guidance provided in a format that is easy for the user to hear and understand.

[0318] This invention utilizes a visual aid system worn by a user to translate visual and audio information in different languages ​​in real time within an autonomous vehicle, and provides audio guidance. The system of the present invention includes a visual aid system, a remote processor, and means used within the autonomous vehicle.

[0319] Hardware and Software

[0320] Visual aid systems:

[0321] The smart glasses are worn by the user and contain a camera, microphone, speaker, communication module, and processor. The system is responsible for capturing visual and audio information and transmitting that data to a remote processor.

[0322] Remote Processor:

[0323] Extract text from visual information (e.g., signs and road signs) and translate it into different languages. Also convert audio data to text, translate it, and convert it back to audio. Use software such as Google Translate API, SpeechRecognition, and gTTS (Google Text-to-Speech).

[0324] Autonomous mobile:

[0325] This includes autonomous vehicles, which are equipped with smart glasses worn by the user and have a communication module that connects to a remote processor via a network.

[0326] Data Processing Overview

[0327] Visual information processing:

[0328] 1. Server: Receives the surrounding images sent from the smart glasses and extracts the text data using character recognition technology.

[0329] 2. Server: Use the Google Translate API to translate the extracted text data into the specified language.

[0330] 3. Server: Uses gTTS to convert the translation results into audio data and sends it to the smart glasses.

[0331] 4. Terminal: Plays back the received audio data and displays the translation results to the user.

[0332] Processing of audio information:

[0333] 1. Device: The microphone in the smart glasses captures the surrounding sounds and sends the audio data to the server.

[0334] 2. Server: Use SpeechRecognition to convert the received audio data into text data.

[0335] 3. Server: Uses Google Translate API and gTTS to translate text data into the specified language and convert it back to audio data.

[0336] 4. Terminal: Plays back the received audio data and displays the translation results to the user.

[0337] Specific examples

[0338] Example 1:

[0339] Consider a situation where a traveler is using an autonomous taxi. To understand a tourist information board they come across on the way to their destination, the smart glasses capture the text information on the sign and send it to a server. The server returns the translated results as audio data and provides audio guidance to the user through the smart glasses. This allows the traveler to instantly understand information about their destination and tourist spots.

[0340] Example 2:

[0341] Imagine a situation where a traveler communicates with local people in an autonomous taxi. When the user asks a question using the microphone in the smart glasses, the voice data is sent to the server and translated. The translated result is played back as voice data, allowing the traveler to communicate smoothly with local people.

[0342] Example prompt for a generative AI model:

[0343] Input: "Describe how you would implement a system that uses smart glasses in a self-driving taxi to translate destination signs and provide voice directions."

[0344] Output: "This implementation utilizes the camera, microphone, and speaker built into the smart glasses. The camera captures visual information and extracts text data using character recognition technology. The extracted text is translated in real time and audio guidance is provided to the user through the smart glasses' speaker. The microphone also captures audio, which is converted into text using speech recognition technology. This is then translated and audio guidance is provided through the speaker."

[0345] In this way, the system of the present invention enables users who speak different languages ​​to understand visual and audio information in different languages ​​within an autonomous mobile body and to communicate smoothly.

[0346] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0347] Step 1:

[0348] Terminal: The camera in the smart glasses captures visual information from the surroundings. The captured image data is sent to the processor. Specifically, the camera periodically acquires images, which are then processed by the internal processor. The input is the captured visual information, and the output is the image data passed to the processor.

[0349] Step 2:

[0350] Terminal: The processor extracts character information from the captured image data. It uses OCR (Optical Character Recognition) technology to convert the characters in the image into text data. The input is the image data from step 1, and the output is the extracted character data.

[0351] Step 3:

[0352] Terminal: The extracted character data is sent to the remote processor via the communication module, where the sent data is passed to the server through the network. The input is the character data, and the output is the data sent to the remote processor.

[0353] Step 4:

[0354] Server: Translates received text data into a specified language. Uses the Google Translate API to convert text data into a different language. The input is the text data to be translated, and the output is the translation result.

[0355] Step 5:

[0356] Server: Converts the translation results into audio data. Using gTTS (Google Text-to-Speech), converts text data into an audio file. The input is the translated text data, and the output is audio data.

[0357] Step 6:

[0358] Server: Transmits the converted voice data to the smart glasses. The voice data is sent to the terminal via the communication module. The input is the voice data, and the output is the voice data sent to the smart glasses.

[0359] Step 7:

[0360] Terminal: The received audio data is played on the smart glasses' speaker, providing audio guidance to the user. Specifically, the audio data is decoded and played as audio through the speaker. The input is audio data, and the output is audio guidance provided to the user.

[0361] Step 8:

[0362] Terminal: The microphone in the smart glasses captures the surrounding sound. The captured sound data is sent to the processor. The input is the captured sound data, and the output is the sound data passed to the processor.

[0363] Step 9:

[0364] Terminal: Transmits the captured audio data to a remote processor via a communication module, where the transmitted data is passed to the server via the network. The input is the audio data, and the output is the data transmitted to the remote processor.

[0365] Step 10:

[0366] Server: Converts received voice data to text. Using SpeechRecognition, converts voice data to text data. The input is voice data, and the output is text data.

[0367] Step 11:

[0368] Server: Translates the converted text data into the specified language. The input is the text data, and the output is the translated text data.

[0369] Step 12:

[0370] Server: The translated text data is converted back into audio data using gTTS. The input is the translated text data, and the output is the audio data.

[0371] Step 13:

[0372] Server: Transmits the converted voice data to the smart glasses. The voice data is sent to the terminal via the communication module. The input is the voice data, and the output is the voice data sent to the smart glasses.

[0373] Step 14:

[0374] Terminal: The received audio data is played on the smart glasses' speaker, providing audio guidance to the user. Specifically, the audio data is decoded and played as audio through the speaker. The input is audio data, and the output is audio guidance provided to the user.

[0375] Through these processing steps, the visual aid system translates visual and audio information in different languages ​​in real time within the autonomous vehicle and provides audio guidance to the user.

[0376] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0377] This invention is a system that automatically translates different languages ​​and provides audio guidance as a visual aid device worn by the user. This system can acquire visual and audio information, translate it in real time, and provide audio guidance. It also has the ability to recognize the user's emotions and adjust the guidance method based on the user's emotional state.

[0378] System configuration

[0379] The visual assistance device (hereafter referred to as the device) is a pair of glasses worn by the user, and contains a camera, microphone, speaker, communication module, processor, and emotion engine. The device supplements its processing capabilities by communicating with a remote computing device (hereafter referred to as the server).

[0380] Visual translation and audio guidance

[0381] Camera and character extraction

[0382] User:

[0383] The user wearing the device can recognize objects that contain text information, such as signs and menus.

[0384] Device:

[0385] The device's camera periodically captures images of its surroundings, which are then analyzed by a processor to extract textual information.

[0386] server:

[0387] The server analyzes the received text and translates it into the specified language. After generating the translation results, it constructs them as text for voice guidance. It then converts the text into audio data using the voice of a celebrity selected by the user and sends it to the device.

[0388] Device:

[0389] The received audio data is played back and the translation results are presented to the user.

[0390] Specific examples

[0391] User:

[0392] Suppose a Japanese-speaking user is visiting France and looking for a restaurant.

[0393] Device:

[0394] A camera captures a sign that says "Restaurant," and a character recognition process extracts the word "Restaurant." The extracted character data is sent to a server.

[0395] server:

[0396] The server receives "restaurant" and translates it into English as "Restaurant." It then generates a narration saying, "This place is a restaurant. Recommended dishes are pasta and steak.", converts it into voice data, and sends it to the device.

[0397] Device:

[0398] The voice guidance played back informs the user that "This is a restaurant. Recommended dishes are pasta and steak."

[0399] Spoken language translation and audio guidance

[0400] Microphone and voice data capture

[0401] User:

[0402] Imagine a situation where a user is communicating with local people in a foreign market.

[0403] Device:

[0404] The device's microphone constantly monitors the surrounding sound and captures audio data, such as conversations, which are then stored as audio data.

[0405] server:

[0406] The server receives the voice data, converts it into text using speech recognition technology, then translates the text into the specified language and generates the translation as voice data, which is then sent to the device.

[0407] Device:

[0408] The received audio data is played back and the translation results are presented to the user.

[0409] Specific examples

[0410] User:

[0411] Consider a scenario where a Japanese-speaking user is asking for directions in an American market.

[0412] Device:

[0413] The microphone captures the user's question, "Where is the nearest station?" and sends the voice data to the server.

[0414] server:

[0415] The server receives the voice data, converts it into text, and translates it. The text "Where is the nearest station?" is generated, converted into voice data, and sent to the device.

[0416] Device:

[0417] The audio guide played allows the user to ask local people, "Where is the nearest station?"

[0418] Emotion Engine Functions

[0419] Emotion recognition and guidance adjustment

[0420] Device:

[0421] It has a built-in emotion engine that analyzes the user's facial expressions and tone of voice captured by the camera and microphone, recognizing the user's emotional state in real time and adjusting the way translation results are presented based on that information.

[0422] Server or device:

[0423] The emotion engine analyzes the user's current emotional state (e.g., happy, sad, anger, surprise) and adjusts the tone and content of the announcements accordingly. For example, if the user is feeling stressed, a calming voice announcement setting will be automatically selected.

[0424] Specific examples

[0425] User:

[0426] A user may feel stressed while trying to find a restaurant in a foreign country.

[0427] Device:

[0428] The emotion engine detects the user's facial expressions and tone of voice and recognizes when the user is feeling stressed.

[0429] Server or device:

[0430] The tone of the translated voice guidance is softened to reflect the user's emotional state. For example, a message such as "This is a cafe. The recommended dish is latte" will be played in a softer voice.

[0431] This invention not only translates the user's visual and audio information in real time and provides audio guidance based on that information, but also grasps the user's emotional state and adjusts the guidance method accordingly, thereby realizing more user-friendly information provision.

[0432] The processing flow will be explained below.

[0433] Processing flow for translation of visual information and voice guidance

[0434] Visual translation

[0435] Step 1:

[0436] User:

[0437] A user wearing the device looks at visual information, such as a sign or a menu.

[0438] Step 2:

[0439] Device:

[0440] The device's camera periodically captures images of its surroundings, which are then sent to a processor.

[0441] Step 3:

[0442] Device:

[0443] The processor uses image analysis algorithms to extract text information from the captured image, for example, using optical character recognition (OCR) technology to convert the characters into text data.

[0444] Step 4:

[0445] Device:

[0446] A request is generated to transmit the extracted character data and the linguistic information to be translated to a remote computing device (server).

[0447] Step 5:

[0448] Device:

[0449] The generated request is sent to the server through a communication module.

[0450] Step 6:

[0451] server:

[0452] The server receives the request, parses the text, and feeds the text into the corresponding translation model to perform translation into the specified language.

[0453] Step 7:

[0454] server:

[0455] Once the translation results are generated, a narration is created based on the translated text. For example, it generates a sentence like, "This is a cafe. The recommended menu item is latte."

[0456] Step 8:

[0457] server:

[0458] To convert the narration text into audio data, text-to-speech synthesis (TTS) is performed using the voice of a celebrity selected by the user.

[0459] Step 9:

[0460] server:

[0461] The generated voice data is transmitted to the terminal.

[0462] Step 10:

[0463] Device:

[0464] The terminal plays back the audio data received from the server and provides the user with the translation results by voice.

[0465] Specific examples

[0466] User:

[0467] A Japanese speaking user sees a sign in French.

[0468] Device:

[0469] The camera captures the sign and uses OCR technology to extract the word "restaurant."

[0470] server:

[0471] The server receives "restaurant" and translates it as "restaurant." It then generates a narration saying, "This is a restaurant. The recommended dish is pasta," converts it into voice data, and sends it to the device.

[0472] Device:

[0473] The terminal plays back the audio data and provides the user with a voice guide saying, "This is a restaurant. The recommended dish is pasta."

[0474] ---

[0475] Spoken language translation and voice guidance processing flow

[0476] Audio translation

[0477] Step 1:

[0478] User:

[0479] The user attempts to converse with local people through a device worn by the user.

[0480] Step 2:

[0481] Device:

[0482] The device's microphone constantly monitors the surrounding sounds and captures what is being said.

[0483] Step 3:

[0484] Device:

[0485] The captured audio data is sent to a processor.

[0486] Step 4:

[0487] Device:

[0488] A processor temporarily stores the audio data and generates a request to send it to a server via a communication module.

[0489] Step 5:

[0490] Device:

[0491] The generated request is sent to the server through a communication module.

[0492] Step 6:

[0493] server:

[0494] The server receives the voice data and converts it into text using speech recognition technology, for example, "Where is the nearest station?"

[0495] Step 7:

[0496] server:

[0497] Translate the recognized text into the specified language, for example, "Where is the nearest station?" into English.

[0498] Step 8:

[0499] server:

[0500] To generate audio data based on the translated text, TTS is used to create audio data in a voice selected by the user.

[0501] Step 9:

[0502] server:

[0503] The generated voice data is transmitted to the terminal.

[0504] Step 10:

[0505] Device:

[0506] The terminal plays back the audio data received from the server and provides the user with the translation results by voice.

[0507] Specific examples

[0508] User:

[0509] A Japanese speaking user wants to ask for directions in the United States.

[0510] Device:

[0511] The microphone captures the user's question, "Where is the nearest station?" and sends the voice data to the server.

[0512] server:

[0513] The server receives the voice data, converts it into text, and translates it. The text "Where is the nearest station?" is converted into voice data and sent to the device.

[0514] Device:

[0515] The device will play the translated audio and the user can ask a local person, "Where is the nearest station?"

[0516] ---

[0517] Emotion Engine Processing Flow

[0518] Emotion recognition and guidance adjustment

[0519] Step 1:

[0520] User:

[0521] The user wears the device and displays changes in stress and emotions during normal use.

[0522] Step 2:

[0523] Device:

[0524] The device's built-in emotion engine uses a camera and microphone to monitor the user's facial expressions and tone of voice in real time.

[0525] Step 3:

[0526] Device:

[0527] The emotion engine analyzes the user's emotional state and detects, for example, stress, joy, and anger.

[0528] Step 4:

[0529] Device:

[0530] The emotion engine generates data based on the sensed emotional state and sends it to an internal processor.

[0531] Step 5:

[0532] Terminal (device) or server:

[0533] The system adjusts the way translation results are presented based on the user's emotional state. For example, if the user is feeling stressed, the system will soften the tone of the voice guidance.

[0534] Step 6:

[0535] Device:

[0536] The adjusted voice guidance is played back to guide the user in a manner that is adaptive to the user.

[0537] Specific examples

[0538] User:

[0539] A user is frustrated trying to find a restaurant in a foreign city.

[0540] Device:

[0541] The emotion engine analyzes the user's facial expressions and tone of voice to recognize stress.

[0542] Server or device:

[0543] The tone of the voice guidance is softened based on the emotional state.

[0544] Device:

[0545] In reality, a gentle voice announces, "This is a cafe. Our recommended dish is latte."

[0546] The above is the specific processing flow of this system. By explaining the operation at each step in detail, we will realize a system that is convenient and highly applicable for users.

[0547] Example 2

[0548] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0549] In recent years, communication between people who speak different languages ​​has become increasingly important. Language barriers are a major obstacle, especially when traveling or traveling to foreign countries for business. However, existing translation devices and applications lack the ability to translate visual and audio information in real time and provide guidance. Furthermore, they are unable to provide guidance that takes into account the user's emotional state, which can cause stress and anxiety. In response to this, the present invention aims to provide user-friendly information by translating the user's visual and audio information in real time and adjusting the guidance method to adapt to the user's emotional state.

[0550] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0551] In this invention, the server includes means for analyzing text information and translating it into a specified language, means for converting it into voice data and sending it to the terminal, and means for analyzing the user's emotional state and adjusting the guidance method, which not only translates the user's visual and voice information in real time and provides guidance, but also makes it possible to provide appropriate guidance according to the user's emotional state.

[0552] A "visual assistance device" is a device worn by a user and has the function of acquiring visual information about the surroundings and providing the information to the user.

[0553] A "remote computing device" is a computer device that is remotely located over the Internet and performs data analysis and calculations.

[0554] "Means for capturing" refers to the function of acquiring visual and audio information using a camera, microphone, etc.

[0555] "Means for extracting characters" refers to a function for detecting and extracting character information from visual information.

[0556] "Means for receiving a translation result" refers to a function by which the terminal receives a translation result sent from a remote computing device.

[0557] "Means for generating voice data" refers to a function for converting text information into voice.

[0558] "Means for recognizing emotional state" refers to a function that uses a camera or microphone to analyze the user's facial expressions and tone of voice to identify the user's emotions.

[0559] "Means for adjusting the guidance method" refers to a function that changes the content and tone of the guidance depending on the user's emotional state.

[0560] "Means for detecting spoken language" refers to the function of detecting surrounding conversations and sounds and acquiring them as audio data.

[0561] The "setting means that can be changed based on a voice profile" refers to a function that allows the tone and style of voice guidance to be changed based on voice characteristics selected by the user.

[0562] MODE FOR CARRYING OUT THE INVENTION

[0563] This invention is a system that uses a user-worn visual aid device to automatically translate different languages ​​and provide audio guidance. The visual aid device has a built-in camera, microphone, speaker, communication module, processor, and emotion engine, and complements its processing capabilities by communicating with a remote computing device (server).

[0564] System configuration

[0565] The visual assistance device is a glasses-type device worn by the user that contains a camera, microphone, speaker, communication module, processor, and emotion engine. The device communicates with a remote server to perform advanced processing.

[0566] System Operation

[0567] Visual translation and audio guidance

[0568] The device periodically captures images of its surroundings using a camera and extracts text information from them. The text information is extracted using OCR (Optical Character Recognition) technology. The text information extracted from the captured images is sent to a server via a wireless communication module.

[0569] After receiving the text information, the server translates it into the specified language using a translation model (e.g., a general translation API). The translated text is then converted into voice data using a speech synthesis tool (e.g., a general speech synthesis API). The generated voice data is then sent to the terminal (device) via a communications network.

[0570] The terminal (device) plays the received voice data through a speaker and provides the user with audible guidance on the translation results.

[0571] As a concrete example, consider a situation where a Japanese-speaking user is visiting France and looking for a restaurant. The user wears a device and sees a sign that says "restaurant." The device's camera captures the sign, and a character recognition process extracts the word "restaurant." This data is sent to a server, which generates the English translation of "restaurant," "Restaurant," and then generates a voice prompt saying, "This place is a restaurant. Recommended dishes are pasta and steak." The terminal (device) plays the received voice data and provides the user with instructions.

[0572] Spoken language translation and audio guidance

[0573] The device captures the surrounding sound using a built-in microphone, and the captured sound data is sent directly to the server.

[0574] After receiving the voice data, the server converts it into text using voice recognition technology (e.g., a general voice recognition API), then translates the converted text into the specified language using a translation model, and generates voice data again using a voice synthesis tool. The generated voice data is then sent to the device.

[0575] The terminal (device) plays back the received audio data again and presents the translation results to the user.

[0576] As a concrete example, consider a situation where a Japanese-speaking user in an American market asks, "Where is the nearest station?" The device's microphone captures the question and sends the data to a server. The server translates the voice data into "Where is the nearest station?" and generates voice data that is sent to the terminal (device). The user can then use this voice data to ask local people.

[0577] Emotion Engine Functions

[0578] The device uses a built-in camera and microphone to analyze the user's facial expressions and tone of voice, recognizing their emotional state in real time.

[0579] The server or terminal (device) adjusts the guidance method based on the user's emotional state. For example, if the user is feeling stressed, the tone of the voice guidance is set to a gentler tone. This allows for flexible guidance tailored to the user's emotional state.

[0580] For example, consider a case where a user is searching for a restaurant in a foreign country but is feeling stressed. The device can sense the user's stress and guide them in a calmer tone, saying, "This is a cafe. The recommended dish is a latte."

[0581] This invention enables the translation of a user's visual and audio information in real time, and also provides appropriate guidance according to the user's emotional state. Based on the above explanation, we have shown what hardware and software are used to specifically implement this system.

[0582] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0583] Visual translation and audio guidance

[0584] Step 1:

[0585] User: Wears the device and acquires visual information. Specifically, the user looks at an object with text information, such as a sign or menu.

[0586] Input: Images containing text information, such as signs and menus

[0587] Output: Visual capture request

[0588] Step 2:

[0589] Device: Uses the camera to capture images of the surroundings.

[0590] Input: Captcha request

[0591] Output: Captured image data

[0592] Step 3:

[0593] Terminal (device): The processor analyzes the captured image data and extracts text information using OCR technology.

[0594] Input: Captured image data

[0595] Output: Extracted text information

[0596] Step 4:

[0597] Terminal (device): The extracted text information is sent to the server using a communication module.

[0598] Input: Extracted text information

[0599] Output: Request to send text information

[0600] Step 5:

[0601] Server: Analyzes the received text information and translates it into the specified language using a generative AI model.

[0602] Input: Extracted text information

[0603] Output: The translated text

[0604] Step 6:

[0605] Server: Converts the translated text into audio data using a speech synthesis tool, possibly using a voice profile selected by the user.

[0606] Input: Translated text

[0607] Output: Audio data

[0608] Step 7:

[0609] Server: Generates a request to send the generated audio data to the device.

[0610] Input: Audio data

[0611] Output: Audio data transmission request

[0612] Step 8:

[0613] Device: Plays the received audio data through a speaker.

[0614] Input: Audio data

[0615] Output: Play voice prompts

[0616] Spoken language translation and audio guidance

[0617] Step 1:

[0618] User: Start a conversation with a foreigner, specifically asking a question such as "Where is the nearest station?"

[0619] Input: User conversation

[0620] Output: Audio data capture request

[0621] Step 2:

[0622] Device: Uses a microphone to capture surrounding sounds.

[0623] Input: Audio data capture request

[0624] Output: Captured audio data

[0625] Step 3:

[0626] Device: Sends captured audio data to the server using a communication module.

[0627] Input: Captured audio data

[0628] output: Request to send audio data

[0629] Step 4:

[0630] Server: Analyzes the received voice data and converts it into text using voice recognition technology.

[0631] Input: Captured audio data

[0632] Output: Speech-to-text data

[0633] Step 5:

[0634] Server: Translates the converted text into the specified language using a generative AI model.

[0635] Input: Speech-to-text data

[0636] Output: The translated text

[0637] Step 6:

[0638] Server: The translated text is converted into audio data using a speech synthesis tool.

[0639] Input: Translated text

[0640] Output: Audio data

[0641] Step 7:

[0642] Server: Generates a request to send the generated audio data to the device.

[0643] Input: Audio data

[0644] Output: Audio data transmission request

[0645] Step 8:

[0646] Device: Plays the received audio data through a speaker.

[0647] Input: Audio data

[0648] Output: Play voice prompts

[0649] Emotion Engine Functions

[0650] Step 1:

[0651] Device: Analyzes the user's facial expressions and tone of voice using the built-in camera and microphone.

[0652] Input: User's facial expression, tone of voice

[0653] Output: Emotional state analysis request

[0654] Step 2:

[0655] Terminal (device) or server: Recognizes emotional state based on analyzed data.

[0656] Input: Emotional state analysis request

[0657] Output: Emotional state data

[0658] Step 3:

[0659] Terminal (device) or server: Adjust the tone and content of the announcement depending on the user's emotional state.

[0660] Input: Emotional state data

[0661] Output: Adjusted voice prompts

[0662] Step 4:

[0663] Device: Plays tailored voice prompts through the speaker.

[0664] Input: Adjusted voice prompts

[0665] Output: Play a gentle voice prompt

[0666] (Application example 2)

[0667] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0668] Conventional visual assistance devices are unable to adapt flexibly to the user's emotional state when receiving translation results. Furthermore, language barriers can hinder the smooth shopping experience in physical stores. There is a need for devices that can provide appropriate support even when users feel stressed or confused.

[0669] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for capturing an image including text, means for extracting text from the captured image, means for transmitting the extracted text to a remote computing device for translation, means for receiving a translation result from the remote computing device, means for providing audible guidance to the user of the received translation result, means for recognizing the user's emotional state and adjusting the guidance method, means for adjusting the guidance method of the translated audio based on the emotion recognition, and setting means for changing the audio guidance of the translation result to the voice of a celebrity selected by the user. This enables flexible response according to the user's emotions, making it possible to have a smooth and stress-free shopping experience in a physical store.

[0670] A "visual assistance device" is a device worn by a user to acquire visual information and assist the user's vision.

[0671] "Images containing text" refers to images with text on them, such as signs or product labels.

[0672] "Capturing means" refers to the method of acquiring an image using a camera or other photographic device.

[0673] The "means for extracting characters" is a method for analyzing character information from a captured image and extracting the characters.

[0674] A "remote computing device" is a server or computer that receives data over a network and performs complex calculations or processing.

[0675] The "means for receiving the translation result" is a method for obtaining the translated data from the remote computing device.

[0676] "Means for providing audio guidance to the user" refers to a method for converting the received translation results into audio and conveying it to the user.

[0677] The "means for recognizing the user's emotional state and adjusting the guidance method" is a method for analyzing the user's facial expression and tone of voice and changing the guidance method according to the user's emotions.

[0678] The "means for detecting spoken words and capturing audio data" is a method for detecting spoken words using an audio input device such as a microphone and acquiring the audio as data.

[0679] The "means for adjusting the guidance method of the translated voice based on emotion recognition" is a method for analyzing the user's emotions through emotion recognition and changing the tone and content of the translated voice accordingly.

[0680] The "setting means for changing to a celebrity's voice" is a method for changing the settings so that guidance is given using the voice of a celebrity selected by the user.

[0681] This invention is a system that supports the shopping experience in a physical store as a visual aid device worn by a user. The system is composed of smart glasses worn by a user and a remote computing device (server).

[0682] Hardware and Software

[0683] Hardware: The smart glasses contain a camera, microphone, speaker, processor, and communication module. The camera captures images containing text information, and the microphone captures audio data. The speaker is used to provide the user with audible translation results.

[0684] software:

[0685] Image processing library (OpenCV): Used to extract characters from captured images.

[0686] Speech Recognition Library (Google Speech-to-Text API): Used to convert captured voice data into text.

[0687] Translation API (Google Translate API): Used to translate the extracted text into a specified language.

[0688] Speech synthesis library (Google Text-to-Speech API): Used to convert the translation text into audio data.

[0689] Emotion Recognition Engine (Emotion API): Used to recognize the user's emotional state and adjust guidance methods.

[0690] System Operation

[0691] User interaction: A user puts on the smart glasses and moves around a physical store. When the user wants to check the price label or description of a product, the camera captures the text.

[0692] Terminal (smart glasses) processing:

[0693] 1. The processor analyzes the image captured by the camera and extracts text information.

[0694] 2. The extracted text information is sent to the server.

[0695] Server Action:

[0696] 1. The server uses a translation API to translate the text received into the specified language.

[0697] 2. Convert the translated text into audio data using a speech synthesis library.

[0698] 3. Use an emotion recognition engine to analyze the user's emotional state and adjust the tone of the voice prompts if the user is feeling stressed.

[0699] Terminal (smart glasses) processing:

[0700] 1. Receives voice data sent from the server and provides voice guidance through the speaker.

[0701] Specific examples

[0702] Consider shopping in a brick-and-mortar store. For example, a foreign tourist wearing smart glasses while shopping in a Japanese department store. The camera captures information about the shelves and store directions, and the system translates the text in real time and provides voice guidance. If the user becomes lost or stressed, the tone of the guidance changes to a gentler tone, providing a sense of security.

[0703] Example prompt sentence:

[0704] A user wears smart glasses and looks at shelves in a department store. The system captures information about the shelves, translates it into the user's language in real time, and provides voice guidance. If the user feels stressed, the tone of the guidance changes to a gentler tone to provide reassurance. Please explain how this system is implemented and provide specific examples of API usage.

[0705] In this way, the visual aid device adapts to the user's emotional state and helps facilitate a smooth shopping experience in a physical store.

[0706] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0707] Step 1:

[0708] A user wears smart glasses and moves around a physical store. When the user wants to check the price label or description of a product, the camera captures the text information. Here, the input is the captured image, and the output is an image file containing the text information.

[0709] Step 2:

[0710] The processor in the device (smart glasses) analyzes the captured image and extracts text information. Specifically, it uses an image processing library (OpenCV) to identify the text and then uses optical character recognition (OCR) technology to convert the text into text data. Here, the input is an image file and the output is the extracted text information (text data).

[0711] Step 3:

[0712] The terminal (smart glasses) transmits the extracted text information to a remote computing device (server). A communication module is used to send the text data to the server. The input of this step is the text data, and the output is the data transmitted to the server.

[0713] Step 4:

[0714] The server translates the received text into the specified language using a translation API (Google Translate API). Here, the server first inputs the text data, processes it through the translation API, and obtains the translation result. The input is the received text information, and the output is the translated text data.

[0715] Step 5:

[0716] The server converts the translated text data into audio data using a speech synthesis library (Google Text-to-Speech API). Specifically, the translated text is input into the speech synthesis API to generate an audio file. The input is the translated text data, and the output is an audio file.

[0717] Step 6:

[0718] Here, the server uses an emotion recognition engine (Emotion API) to analyze the user's emotional state. The camera and microphone capture the user's facial expressions and tone of voice, and send the data to the server. The input is facial expression data and voice data, and the output is the user's emotional state (e.g., stress, joy, surprise).

[0719] Step 7:

[0720] The server adjusts the tone of the voice guidance based on the emotion recognition results. For example, if the user is feeling stressed, the tone of the voice guidance is set to be gentler. The input is the emotional state and the audio file, and the output is the adjusted audio file.

[0721] Step 8:

[0722] The server sends the adjusted audio file to the device (smart glasses). The input of this step is the adjusted audio file, and the output is the data sent to the device.

[0723] Step 9:

[0724] The speaker on the device (smart glasses) plays the received audio data and provides the translation results to the user by voice. The input is the received audio data, and the output is the audio guidance the user hears.

[0725] In this way, visual aid devices can help users have a smoother shopping experience in brick-and-mortar stores. Furthermore, by providing flexible guidance that responds to the user's emotional state, it can provide more user-friendly information.

[0726] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0727] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0728] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0729] [Second embodiment]

[0730] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0731] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0732] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0733] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0734] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0735] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0736] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0737] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0738] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0739] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0740] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0741] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0742] This invention is a system that automatically translates different languages ​​and provides audio guidance as a visual aid device worn by the user. This system can acquire visual information, translate it in real time, and provide audio guidance. It also has the function of translating spoken language and providing audio guidance.

[0743] This visual assistance device (hereinafter referred to as the device) is in the form of glasses worn by the user and contains a camera, microphone, speaker, communication module, and processor. The device supplements its processing capabilities by communicating with a remote computing device (hereinafter referred to as the server).

[0744] 1. Visual translation and audio guidance

[0745] Camera and character extraction

[0746] User:

[0747] The device recognizes objects with text information, such as signs and menus, in the user's field of vision.

[0748] Device:

[0749] The device's camera periodically captures images of its surroundings, which are then analyzed by a processor to extract textual information.

[0750] Sending translation requests and receiving translation results

[0751] Device:

[0752] The extracted text information is sent to the server via the communication module, and the data sent also includes information about the language to be translated.

[0753] server:

[0754] The server analyzes the received text and translates it into the specified language. After generating the translation results, it constructs them as text for voice guidance. It then converts the text into audio data using the voice of a celebrity selected by the user and sends it to the device.

[0755] Device:

[0756] The received audio data is played back and the translation results are presented to the user.

[0757] Specific examples

[0758] User:

[0759] Suppose a Japanese-speaking user is visiting France and looking for a restaurant.

[0760] Device:

[0761] A camera captures a sign that says "Restaurant," and a character recognition process extracts the word "Restaurant." The extracted character data is sent to a server.

[0762] server:

[0763] The server receives the character "restaurant" and translates it into English as "Restaurant." Based on the translation, it generates a narration saying "This place is a restaurant. Recommended dishes are pasta and steak." This is converted into audio data and sent to the device.

[0764] Device:

[0765] The voice guidance played back informs the user that "This is a restaurant. Recommended dishes are pasta and steak."

[0766] 2. Spoken language translation and audio guidance

[0767] Microphone and voice data capture

[0768] User:

[0769] Imagine a situation where a user is communicating with local people in a foreign market.

[0770] Device:

[0771] The device's microphone constantly monitors the surrounding sound and captures audio data, such as conversations, which are then stored as audio data.

[0772] Sending translation requests and receiving translation results

[0773] Device:

[0774] The stored voice data is transmitted to a server via a communication module.

[0775] server:

[0776] The server receives the voice data, converts it into text using speech recognition technology, then translates the text into the specified language and generates the translation as voice data, which is then sent to the device.

[0777] Device:

[0778] The received audio data is played back and the translation results are presented to the user.

[0779] Specific examples

[0780] User:

[0781] Consider a scenario in which a Japanese-speaking user in an American market asks "Where is the nearest station?" to get directions.

[0782] Device:

[0783] A microphone captures the user's question and transmits the audio data to the server.

[0784] server:

[0785] The server receives the speech data "Where is the nearest station?" and translates it into "Where is the nearest station?". It generates the translated speech data and sends it to the device.

[0786] Device:

[0787] The audio guide played allows the user to ask local people, "Where is the nearest station?"

[0788] In this way, the present invention allows users to translate visual and audio information in real time and provide appropriate guidance, thereby enabling smooth communication even in environments where different languages ​​are spoken. Furthermore, by providing audio guidance in the voice of a celebrity selected by the user, a more friendly service can be provided.

[0789] The processing flow will be explained below.

[0790] Processing flow for translation of visual information and voice guidance

[0791] Visual translation

[0792] Step 1:

[0793] User:

[0794] A user wearing the device looks at visual information, such as a sign or a menu.

[0795] Step 2:

[0796] Device:

[0797] The device's camera periodically captures images of its surroundings, which are then sent to a processor.

[0798] Step 3:

[0799] Device:

[0800] The processor uses image analysis algorithms to extract text information from the captured image, for example, using optical character recognition (OCR) technology to convert the characters into text data.

[0801] Step 4:

[0802] Device:

[0803] A request is generated to transmit the extracted character data and the linguistic information to be translated to a remote computing device (server).

[0804] Step 5:

[0805] Device:

[0806] The generated request is sent to the server through a communication module.

[0807] Step 6:

[0808] server:

[0809] The server receives the request, parses the text, and feeds the text into the corresponding translation model to perform translation into the specified language.

[0810] Step 7:

[0811] server:

[0812] Once the translation results are generated, a narration is created based on the translated text. For example, it generates a sentence like, "This is a cafe. The recommended menu item is latte."

[0813] Step 8:

[0814] server:

[0815] To convert the narration text into audio data, text-to-speech synthesis (TTS) is performed using the voice of a celebrity selected by the user.

[0816] Step 9:

[0817] server:

[0818] The generated voice data is transmitted to the terminal.

[0819] Step 10:

[0820] Device:

[0821] The terminal plays back the audio data received from the server and provides the user with the translation results by voice.

[0822] Specific examples

[0823] User:

[0824] A Japanese speaking user sees a sign in French.

[0825] Device:

[0826] The camera captures the sign and uses OCR technology to extract the word "restaurant."

[0827] server:

[0828] The server receives "restaurant" and translates it as "restaurant." It then generates a narration saying, "This is a restaurant. The recommended dish is pasta," converts it into voice data, and sends it to the device.

[0829] Device:

[0830] The device plays the audio data and informs the user of the translation results. The voice will say in Japanese, "This is a restaurant. The recommended dish is pasta."

[0831] ---

[0832] Spoken language translation and voice guidance processing flow

[0833] Audio translation

[0834] Step 1:

[0835] User:

[0836] The user puts on the device and attempts to converse with local people.

[0837] Step 2:

[0838] Device:

[0839] The device's microphone constantly monitors the surrounding sounds and captures what is being said.

[0840] Step 3:

[0841] Device:

[0842] The captured audio data is sent to a processor.

[0843] Step 4:

[0844] Device:

[0845] A processor temporarily stores the audio data and generates a request to send it to a server via a communication module.

[0846] Step 5:

[0847] Device:

[0848] The generated request is sent to the server through a communication module.

[0849] Step 6:

[0850] server:

[0851] The server receives the voice data and converts it into text using speech recognition technology, for example, "Where is the nearest station?"

[0852] Step 7:

[0853] server:

[0854] Translate the recognized text into the specified language, for example, "Where is the nearest station?" into English.

[0855] Step 8:

[0856] server:

[0857] To generate audio data based on the translated text, TTS is used to create audio data in a voice selected by the user.

[0858] Step 9:

[0859] server:

[0860] The generated voice data is transmitted to the terminal.

[0861] Step 10:

[0862] Device:

[0863] The terminal plays back the audio data received from the server and provides the user with the translation results by voice.

[0864] Specific examples

[0865] User:

[0866] A Japanese speaking user wants to ask for directions in the United States.

[0867] Device:

[0868] The microphone captures the user's question, "Where is the nearest station?" and sends the voice data to the server.

[0869] server:

[0870] The server receives the voice data, converts it into text, and translates it. The text "Where is the nearest station?" is converted into voice data and sent to the device.

[0871] Device:

[0872] The device will play the translated audio and the user can ask a local person, "Where is the nearest station?"

[0873] The above is the specific processing flow of this system, and the operation at each step is explained in detail.

[0874] Example 1

[0875] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0876] In modern society, people who speak different languages ​​coexist, and smooth communication is essential. However, language barriers make it difficult to understand signs and menus, or to converse smoothly with local people, both in daily life and while traveling. To solve these problems, a visual aid device is needed that can translate in real time and provide audio guidance.

[0877] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0878] In this invention, the server includes means for capturing an image containing text, means for extracting text from the captured image, means for translating the extracted text, means for receiving a translation result from a remote computing device, means for providing audible guidance to a user about the received translation result, and means for converting the received translation result into audio data in a specified voice, thereby enabling real-time translation of text and conversations displayed in different languages ​​and audio guidance.

[0879] 1. "Visual assistance device" means a device worn by a user that has the function of acquiring, processing, and providing visual information.

[0880] 2. "Capturing means" is a collective term for the hardware and software that a device uses to capture images and audio.

[0881] 3. "Text extraction means" means a process or device that has the ability to identify text information from a captured image and extract it as data.

[0882] 4. "Remote Computing Device" means a device that provides remotely located computing resources for processing data transmitted from a Device.

[0883] 5. "Translation means" means software and hardware functions for converting written or spoken information in one language into another language.

[0884] 6. "Means for receiving translation results" means a device that has the function of receiving translation data sent from a remote computing device on the device side.

[0885] 7. "Means for providing audio guidance to the user" refers to the device's function of converting the received translation results into audio format and notifying the user by audio.

[0886] 8. "Means for converting into voice data using a specified voice" means a device that has the function of converting the translation results into voice data using a specific voice that the user has specified in advance.

[0887] 9. "Audio data capture means" means a device or process that captures a user's speech or surrounding sounds and stores them as digital audio data.

[0888] 10. "Means for receiving translated voice data" means a device that has the capability to receive translated voice data sent from a remote computing device.

[0889] 11. "Means for providing information to the user" refers to functions or devices for providing acquired or received information to the user in an easy-to-understand manner.

[0890] 12. "Configuration means" means the interface and software functionality that allows a user to customize the operation and functionality of a device to suit the user's preferences.

[0891] This invention provides a multilingual visual aid device, enabling users to translate visual and audio information in real time and provide audio guidance in different language environments. This invention achieves this by using a capture device, a remote server, speech synthesis technology, and a communication module.

[0892] Hardware and software configurations used

[0893] 1. Wearable visual aids:

[0894] Camera: Captures visual information about the surroundings.

[0895] Microphone: Captures surrounding sounds.

[0896] Speaker: A means of providing the translation results to the user aloud.

[0897] Communications Module: A means for sending and receiving data to and from a remote computing device.

[0898] Processor: Processes image and audio data.

[0899] 2. Remote Computing Device (Server):

[0900] Character recognition software: Extracts text information from images (e.g., Tesseract OCR).

[0901] Speech recognition technology: Converts captured voice data into text (e.g., Google Speech-to-Text).

[0902] Translation engine: Translates text from one language to another (e.g., Google Translate API, DeepL).

[0903] Speech synthesis engine: Converts text to speech (e.g., Amazon Polly, Google Text-to-Speech).

[0904] System Operation

[0905] Visual translation and audio guidance

[0906] A visual assistance device worn by a user captures visual information from the surroundings using a camera. For example, the camera may capture a photo of a sign for a French restaurant. The processor of the terminal (device) analyzes the captured image and uses character recognition software to extract the text information "restaurant." This text information is then sent to a server via a communications module. The server analyzes the received text data, translates it into the specified language (e.g., from French to English), and outputs the translation as "Restaurant." Based on the translation, the server then generates a narration such as "This place is a restaurant. Recommended dishes are pasta and steak." This is converted into audio data using a speech synthesis engine and sent to the device. The terminal (device) receives the audio data and plays it through a speaker, so the user is informed that "This place is a restaurant. Recommended dishes are pasta and steak."

[0907] Spoken language translation and audio guidance

[0908] Consider a scenario in which a user in a foreign market asks, "Where is the nearest station?" A microphone installed on the terminal (device) captures surrounding sounds and sends the voice data to a server. The server receives the voice data and converts it into text using voice recognition technology. It then translates the text into a specified language (e.g., Japanese to English), generates voice data based on the translation results, and sends it to the device. The terminal (device) receives the voice data and asks the user, "Where is the nearest station?"

[0909] Examples of concrete examples and prompts

[0910] Specific examples

[0911] A Japanese-speaking user visits France and a camera captures a sign that says "Restaurant." The text data "Restaurant" is sent to the server. The server translates "Restaurant" into "Restaurant," generates a narration saying, "This place is a restaurant. Recommended dishes are pasta and steak.", and sends this voice data to the device. By listening to this voice guidance, the user learns, "This place is a restaurant. Recommended dishes are pasta and steak."

[0912] Prompt Sentence Examples

[0913] "Translate the text written on the sign. Generate a prompt to translate from French to English: 'Extract text from captured image and translate it into the specified language.'"

[0914] "Generate a prompt that translates the Japanese question 'Where is the nearest station?' into English: 'Use speech recognition technology to convert captured speech into text, and then translate that text into the specified language.'"

[0915] In this way, the present invention is a system that can translate visual and audio information in real time in a multilingual environment and provide audio guidance to the user.

[0916] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0917] Visual translation and audio guidance

[0918] Step 1: Capture visual information

[0919] Device:

[0920] While a user is wearing a visual aid device, the device's camera periodically captures visual information about the surroundings.

[0921] Input: Objects in the user's field of view (e.g., signs, menus)

[0922] Data processing: The image capture process is activated and the camera acquires the image data.

[0923] Output: Acquired raw image data

[0924] What happens: The device's camera takes a picture every second.

[0925] Step 2: Extracting characters

[0926] Device:

[0927] The processor analyzes the captured image data in real time and extracts text information using optical character recognition software (OCR).

[0928] Input: Captured image data

[0929] Data processing: Extracting text data from images using OCR algorithms (e.g., Tesseract).

[0930] Output: Extracted text data (e.g., "restaurant")

[0931] What it does: The processor recognizes characters in the image and generates data in text format.

[0932] Step 3: Submit a translation request

[0933] Device:

[0934] The extracted text data and the language information to be translated are sent to the server via the communication module.

[0935] Input: Extracted text data, language information to be translated (e.g. French to English)

[0936] Data processing: Constructing data packets and sending them to the server using a communication protocol (e.g., HTTP).

[0937] Output: Data packet sent to the server

[0938] What happens: The device sends the text "restaurant" and a translation request from French to English.

[0939] Step 4: Translation and Narration Generation

[0940] server:

[0941] The received text data is analyzed and translated into the specified language, and a narration is generated based on the translation results.

[0942] Input: Text data (e.g., "restaurant"), language information to be translated

[0943] Data processing: Translate to "Restaurant" using a translation engine (e.g., Google Translate API). Then generate a voiceover sentence: "This place is a restaurant. Recommended dishes are pasta and steak."

[0944] Output: Translated text and narration

[0945] Specific operation: The server translates "restaurant" to "Restaurant" and constructs a guide text.

[0946] Step 5: Generate audio data

[0947] server:

[0948] The narration sentences are converted into voice data using a voice synthesis engine.

[0949] Input: Narration (e.g., "This place is a restaurant. Recommended dishes are pasta and steak.")

[0950] Data processing: Generate voice data using a speech synthesis engine (e.g., Amazon Polly).

[0951] Output: Audio data

[0952] Specific operation: The server generates audio based on the narration text.

[0953] Step 6: Sending voice data and guidance

[0954] Device:

[0955] The audio data is received from the server and played back to the user through the speaker.

[0956] Input: Audio data received from the server

[0957] Data processing: Decodes the received data and plays it on the speaker.

[0958] Output: A voice prompt to the user

[0959] What happens: The speaker plays a voice saying, "This place is a restaurant. Recommended dishes are pasta and steak."

[0960] ---

[0961] Spoken language translation and audio guidance

[0962] Step 1: Capture audio data

[0963] Device:

[0964] The microphone captures the surrounding sounds.

[0965] Input: User speech and surrounding audio (e.g., "Where is the nearest station?")

[0966] Data processing: Recording audio data and converting it into digital format.

[0967] Output: Captured audio data

[0968] What it does: A microphone records the user's question.

[0969] Step 2: Sending audio data

[0970] Device:

[0971] The captured audio data is transmitted to a server via a communication module.

[0972] Input: Audio data

[0973] Data processing: Constructing data packets and sending them to the server using a communication protocol.

[0974] Output: Audio data sent to the server

[0975] Specific operation: The device sends voice data saying "Where is the nearest station?"

[0976] Step 3: Transcribe and translate the audio data

[0977] server:

[0978] The received voice data is converted into text using voice recognition technology and then translated.

[0979] Input: Speech data (e.g., "Where is the nearest station?")

[0980] Data processing: Use speech recognition technology (e.g., Google Speech-to-Text) to convert the speech to text, then use a translation engine to translate it into "Where is the nearest station?"

[0981] Output: Translated text data

[0982] What happens: The server converts the voice data into text and translates it into English.

[0983] Step 4: Generate translated audio data

[0984] server:

[0985] The translated text data is converted into voice data using a voice synthesis engine.

[0986] Input: Translated text data (e.g., "Where is the nearest station?")

[0987] Data processing: Generate voice data using a speech synthesis engine.

[0988] Output: Audio data

[0989] Specific operation: The server generates audio data based on the translation results.

[0990] Step 5: Voice prompts

[0991] Device:

[0992] The generated audio data is received and played back to the user through a speaker.

[0993] Input: Audio data

[0994] Data processing: Decodes the received data and plays it on the speaker.

[0995] Output: A voice prompt to the user

[0996] Specific action: The speaker will play a voice saying "Where is the nearest station?"

[0997] In this way, the system captures visual and audio information in real time, translates it, and provides audio guidance, thereby providing smooth communication that transcends language barriers.

[0998] Examples of concrete examples and prompts

[0999] Specific examples

[1000] A Japanese-speaking user uses a camera to capture a sign that says "restaurant" in a French restaurant, and the text data "restaurant" is sent to the server. The server translates "restaurant" into "Restaurant" and generates a narration saying, "This place is a restaurant. Recommended dishes are pasta and steak." This is sent as audio data to the device. After listening to the audio guidance, the user learns, "This place is a restaurant. Recommended dishes are pasta and steak."

[1001] A Japanese-speaking user in an American market asks, "Where is the nearest station?" The device captures the speech and sends it to the server. The server translates "Where is the nearest station?" to "Where is the nearest station?" and sends it as voice data to the device. The device plays the speech and tells the user, "Where is the nearest station?"

[1002] Prompt Sentence Examples

[1003] "Translate the text written on the sign. Generate a prompt to translate from French to English: 'Extract text from captured image and translate it into the specified language.'"

[1004] "Generate a prompt that translates the Japanese question 'Where is the nearest station?' into English: 'Use speech recognition technology to convert captured speech into text, and then translate that text into the specified language.'"

[1005] This system is designed to support smooth information acquisition and communication in a multilingual environment.

[1006] (Application example 1)

[1007] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1008] There is a problem in that users who speak different languages ​​cannot communicate smoothly in autonomous vehicles (e.g., self-driving vehicles). It is also difficult to understand visual information such as signs and road signs written in different languages, which can make travel in a self-driving vehicle unstable. Furthermore, local guidance and responses to questions cannot be provided immediately, which causes great stress for tourists and foreign users.

[1009] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1010] In this invention, the server includes a processing device for translating visual information, a processing device for translating audio information, and a processing device for converting received translation results into audio data. This enables real-time translation of visual and audio information in different languages ​​within an autonomous mobile body. The translation results are provided to the user via audio guidance via the visual assistance system, allowing the user to quickly understand route guidance to their destination and emergency information. In addition, user-selected audio guidance is provided in a user-friendly format.

[1011] A "visual aid system" is a device worn by a user that captures visual information and provides the information to the user visually or audibly.

[1012] "Visual information" refers to information such as characters and figures that are included in the user's field of vision.

[1013] A "remote processor" is a remotely located computing device that processes data sent from a visual aid system and returns the results.

[1014] An "autonomous mobile object" is a vehicle or device that can move autonomously and reach its destination using artificial intelligence and sensor technology, such as a self-driving vehicle.

[1015] "Audio data" refers to audio signals captured by an audio input device such as a microphone and converted into digital data.

[1016] "Translation result" is text or audio data that has been converted into a different language by a remote processor.

[1017] "Famous person's voice" is voice data of a commonly recognized person selected by the user.

[1018] "Guidance means" refers to a mechanism or device that allows the visual aid system to provide information to the user.

[1019] A "processing device" is a computer or hardware for processing and analyzing data.

[1020] "Friendly voice guidance" refers to voice guidance provided in a format that is easy for the user to hear and understand.

[1021] This invention utilizes a visual aid system worn by a user to translate visual and audio information in different languages ​​in real time within an autonomous vehicle, and provides audio guidance. The system of the present invention includes a visual aid system, a remote processor, and means used within the autonomous vehicle.

[1022] Hardware and Software

[1023] Visual aid systems:

[1024] The smart glasses are worn by the user and contain a camera, microphone, speaker, communication module, and processor. The system is responsible for capturing visual and audio information and transmitting that data to a remote processor.

[1025] Remote Processor:

[1026] Extract text from visual information (e.g., signs and road signs) and translate it into different languages. Also convert audio data to text, translate it, and convert it back to audio. Use software such as Google Translate API, SpeechRecognition, and gTTS (Google Text-to-Speech).

[1027] Autonomous mobile:

[1028] This includes autonomous vehicles, which are equipped with smart glasses worn by the user and have a communication module that connects to a remote processor via a network.

[1029] Data Processing Overview

[1030] Visual information processing:

[1031] 1. Server: Receives the surrounding images sent from the smart glasses and extracts the text data using character recognition technology.

[1032] 2. Server: Use the Google Translate API to translate the extracted text data into the specified language.

[1033] 3. Server: Uses gTTS to convert the translation results into audio data and sends it to the smart glasses.

[1034] 4. Terminal: Plays back the received audio data and displays the translation results to the user.

[1035] Processing of audio information:

[1036] 1. Device: The microphone in the smart glasses captures the surrounding sounds and sends the audio data to the server.

[1037] 2. Server: Use SpeechRecognition to convert the received audio data into text data.

[1038] 3. Server: Uses Google Translate API and gTTS to translate text data into the specified language and convert it back to audio data.

[1039] 4. Terminal: Plays back the received audio data and displays the translation results to the user.

[1040] Specific examples

[1041] Example 1:

[1042] Consider a situation where a traveler is using an autonomous taxi. To understand a tourist information board they come across on the way to their destination, the smart glasses capture the text information on the sign and send it to a server. The server returns the translated results as audio data and provides audio guidance to the user through the smart glasses. This allows the traveler to instantly understand information about their destination and tourist spots.

[1043] Example 2:

[1044] Imagine a situation where a traveler communicates with local people in an autonomous taxi. When the user asks a question using the microphone in the smart glasses, the voice data is sent to the server and translated. The translated result is played back as voice data, allowing the traveler to communicate smoothly with local people.

[1045] Example prompt for a generative AI model:

[1046] Input: "Describe how you would implement a system that uses smart glasses in a self-driving taxi to translate destination signs and provide voice directions."

[1047] Output: "This implementation utilizes the camera, microphone, and speaker built into the smart glasses. The camera captures visual information and extracts text data using character recognition technology. The extracted text is translated in real time and audio guidance is provided to the user through the smart glasses' speaker. The microphone also captures audio, which is converted into text using speech recognition technology. This is then translated and audio guidance is provided through the speaker."

[1048] In this way, the system of the present invention enables users who speak different languages ​​to understand visual and audio information in different languages ​​within an autonomous mobile body and to communicate smoothly.

[1049] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1050] Step 1:

[1051] Terminal: The camera in the smart glasses captures visual information from the surroundings. The captured image data is sent to the processor. Specifically, the camera periodically acquires images, which are then processed by the internal processor. The input is the captured visual information, and the output is the image data passed to the processor.

[1052] Step 2:

[1053] Terminal: The processor extracts character information from the captured image data. It uses OCR (Optical Character Recognition) technology to convert the characters in the image into text data. The input is the image data from step 1, and the output is the extracted character data.

[1054] Step 3:

[1055] Terminal: The extracted character data is sent to the remote processor via the communication module, where the sent data is passed to the server through the network. The input is the character data, and the output is the data sent to the remote processor.

[1056] Step 4:

[1057] Server: Translates received text data into a specified language. Uses the Google Translate API to convert text data into a different language. The input is the text data to be translated, and the output is the translation result.

[1058] Step 5:

[1059] Server: Converts the translation results into audio data. Using gTTS (Google Text-to-Speech), converts text data into an audio file. The input is the translated text data, and the output is audio data.

[1060] Step 6:

[1061] Server: Transmits the converted voice data to the smart glasses. The voice data is sent to the terminal via the communication module. The input is the voice data, and the output is the voice data sent to the smart glasses.

[1062] Step 7:

[1063] Terminal: The received audio data is played on the smart glasses' speaker, providing audio guidance to the user. Specifically, the audio data is decoded and played as audio through the speaker. The input is audio data, and the output is audio guidance provided to the user.

[1064] Step 8:

[1065] Terminal: The microphone in the smart glasses captures the surrounding sound. The captured sound data is sent to the processor. The input is the captured sound data, and the output is the sound data passed to the processor.

[1066] Step 9:

[1067] Terminal: Transmits the captured audio data to a remote processor via a communication module, where the transmitted data is passed to the server via the network. The input is the audio data, and the output is the data transmitted to the remote processor.

[1068] Step 10:

[1069] Server: Converts received voice data to text. Using SpeechRecognition, converts voice data to text data. The input is voice data, and the output is text data.

[1070] Step 11:

[1071] Server: Translates the converted text data into the specified language. The input is the text data, and the output is the translated text data.

[1072] Step 12:

[1073] Server: The translated text data is converted back into audio data using gTTS. The input is the translated text data, and the output is the audio data.

[1074] Step 13:

[1075] Server: Transmits the converted voice data to the smart glasses. The voice data is sent to the terminal via the communication module. The input is the voice data, and the output is the voice data sent to the smart glasses.

[1076] Step 14:

[1077] Terminal: The received audio data is played on the smart glasses' speaker, providing audio guidance to the user. Specifically, the audio data is decoded and played as audio through the speaker. The input is audio data, and the output is audio guidance provided to the user.

[1078] Through these processing steps, the visual aid system translates visual and audio information in different languages ​​in real time within the autonomous vehicle and provides audio guidance to the user.

[1079] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1080] This invention is a system that automatically translates different languages ​​and provides audio guidance as a visual aid device worn by the user. This system can acquire visual and audio information, translate it in real time, and provide audio guidance. It also has the ability to recognize the user's emotions and adjust the guidance method based on the user's emotional state.

[1081] System configuration

[1082] The visual assistance device (hereafter referred to as the device) is a pair of glasses worn by the user, and contains a camera, microphone, speaker, communication module, processor, and emotion engine. The device supplements its processing capabilities by communicating with a remote computing device (hereafter referred to as the server).

[1083] Visual translation and audio guidance

[1084] Camera and character extraction

[1085] User:

[1086] The user wearing the device can recognize objects that contain text information, such as signs and menus.

[1087] Device:

[1088] The device's camera periodically captures images of its surroundings, which are then analyzed by a processor to extract textual information.

[1089] server:

[1090] The server analyzes the received text and translates it into the specified language. After generating the translation results, it constructs them as text for voice guidance. It then converts the text into audio data using the voice of a celebrity selected by the user and sends it to the device.

[1091] Device:

[1092] The received audio data is played back and the translation results are presented to the user.

[1093] Specific examples

[1094] User:

[1095] Suppose a Japanese-speaking user is visiting France and looking for a restaurant.

[1096] Device:

[1097] A camera captures a sign that says "Restaurant," and a character recognition process extracts the word "Restaurant." The extracted character data is sent to a server.

[1098] server:

[1099] The server receives "restaurant" and translates it into English as "Restaurant." It then generates a narration saying, "This place is a restaurant. Recommended dishes are pasta and steak.", converts it into voice data, and sends it to the device.

[1100] Device:

[1101] The voice guidance played back informs the user that "This is a restaurant. Recommended dishes are pasta and steak."

[1102] Spoken language translation and audio guidance

[1103] Microphone and voice data capture

[1104] User:

[1105] Imagine a situation where a user is communicating with local people in a foreign market.

[1106] Device:

[1107] The device's microphone constantly monitors the surrounding sound and captures audio data, such as conversations, which are then stored as audio data.

[1108] server:

[1109] The server receives the voice data, converts it into text using speech recognition technology, then translates the text into the specified language and generates the translation as voice data, which is then sent to the device.

[1110] Device:

[1111] The received audio data is played back and the translation results are presented to the user.

[1112] Specific examples

[1113] User:

[1114] Consider a scenario where a Japanese-speaking user is asking for directions in an American market.

[1115] Device:

[1116] The microphone captures the user's question, "Where is the nearest station?" and sends the voice data to the server.

[1117] server:

[1118] The server receives the voice data, converts it into text, and translates it. The text "Where is the nearest station?" is generated, converted into voice data, and sent to the device.

[1119] Device:

[1120] The audio guide played allows the user to ask local people, "Where is the nearest station?"

[1121] Emotion Engine Functions

[1122] Emotion recognition and guidance adjustment

[1123] Device:

[1124] It has a built-in emotion engine that analyzes the user's facial expressions and tone of voice captured by the camera and microphone, recognizing the user's emotional state in real time and adjusting the way translation results are presented based on that information.

[1125] Server or device:

[1126] The emotion engine analyzes the user's current emotional state (e.g., happy, sad, anger, surprise) and adjusts the tone and content of the announcements accordingly. For example, if the user is feeling stressed, a calming voice announcement setting will be automatically selected.

[1127] Specific examples

[1128] User:

[1129] A user may feel stressed while trying to find a restaurant in a foreign country.

[1130] Device:

[1131] The emotion engine detects the user's facial expressions and tone of voice and recognizes when the user is feeling stressed.

[1132] Server or device:

[1133] The tone of the translated voice guidance is softened to reflect the user's emotional state. For example, a message such as "This is a cafe. The recommended dish is latte" will be played in a softer voice.

[1134] This invention not only translates the user's visual and audio information in real time and provides audio guidance based on that information, but also grasps the user's emotional state and adjusts the guidance method accordingly, thereby realizing more user-friendly information provision.

[1135] The processing flow will be explained below.

[1136] Processing flow for translation of visual information and voice guidance

[1137] Visual translation

[1138] Step 1:

[1139] User:

[1140] A user wearing the device looks at visual information, such as a sign or a menu.

[1141] Step 2:

[1142] Device:

[1143] The device's camera periodically captures images of its surroundings, which are then sent to a processor.

[1144] Step 3:

[1145] Device:

[1146] The processor uses image analysis algorithms to extract text information from the captured image, for example, using optical character recognition (OCR) technology to convert the characters into text data.

[1147] Step 4:

[1148] Device:

[1149] A request is generated to transmit the extracted character data and the linguistic information to be translated to a remote computing device (server).

[1150] Step 5:

[1151] Device:

[1152] The generated request is sent to the server through a communication module.

[1153] Step 6:

[1154] server:

[1155] The server receives the request, parses the text, and feeds the text into the corresponding translation model to perform translation into the specified language.

[1156] Step 7:

[1157] server:

[1158] Once the translation results are generated, a narration is created based on the translated text. For example, it generates a sentence like, "This is a cafe. The recommended menu item is latte."

[1159] Step 8:

[1160] server:

[1161] To convert the narration text into audio data, text-to-speech synthesis (TTS) is performed using the voice of a celebrity selected by the user.

[1162] Step 9:

[1163] server:

[1164] The generated voice data is transmitted to the terminal.

[1165] Step 10:

[1166] Device:

[1167] The terminal plays back the audio data received from the server and provides the user with the translation results by voice.

[1168] Specific examples

[1169] User:

[1170] A Japanese speaking user sees a sign in French.

[1171] Device:

[1172] The camera captures the sign and uses OCR technology to extract the word "restaurant."

[1173] server:

[1174] The server receives "restaurant" and translates it as "restaurant." It then generates a narration saying, "This is a restaurant. The recommended dish is pasta," converts it into voice data, and sends it to the device.

[1175] Device:

[1176] The terminal plays back the audio data and provides the user with a voice guide saying, "This is a restaurant. The recommended dish is pasta."

[1177] ---

[1178] Spoken language translation and voice guidance processing flow

[1179] Audio translation

[1180] Step 1:

[1181] User:

[1182] The user attempts to converse with local people through a device worn by the user.

[1183] Step 2:

[1184] Device:

[1185] The device's microphone constantly monitors the surrounding sounds and captures what is being said.

[1186] Step 3:

[1187] Device:

[1188] The captured audio data is sent to a processor.

[1189] Step 4:

[1190] Device:

[1191] A processor temporarily stores the audio data and generates a request to send it to a server via a communication module.

[1192] Step 5:

[1193] Device:

[1194] The generated request is sent to the server through a communication module.

[1195] Step 6:

[1196] server:

[1197] The server receives the voice data and converts it into text using speech recognition technology, for example, "Where is the nearest station?"

[1198] Step 7:

[1199] server:

[1200] Translate the recognized text into the specified language, for example, "Where is the nearest station?" into English.

[1201] Step 8:

[1202] server:

[1203] To generate audio data based on the translated text, TTS is used to create audio data in a voice selected by the user.

[1204] Step 9:

[1205] server:

[1206] The generated voice data is transmitted to the terminal.

[1207] Step 10:

[1208] Device:

[1209] The terminal plays back the audio data received from the server and provides the user with the translation results by voice.

[1210] Specific examples

[1211] User:

[1212] A Japanese speaking user wants to ask for directions in the United States.

[1213] Device:

[1214] The microphone captures the user's question, "Where is the nearest station?" and sends the voice data to the server.

[1215] server:

[1216] The server receives the voice data, converts it into text, and translates it. The text "Where is the nearest station?" is converted into voice data and sent to the device.

[1217] Device:

[1218] The device will play the translated audio and the user can ask a local person, "Where is the nearest station?"

[1219] ---

[1220] Emotion Engine Processing Flow

[1221] Emotion recognition and guidance adjustment

[1222] Step 1:

[1223] User:

[1224] The user wears the device and displays changes in stress and emotions during normal use.

[1225] Step 2:

[1226] Device:

[1227] The device's built-in emotion engine uses a camera and microphone to monitor the user's facial expressions and tone of voice in real time.

[1228] Step 3:

[1229] Device:

[1230] The emotion engine analyzes the user's emotional state and detects, for example, stress, joy, and anger.

[1231] Step 4:

[1232] Device:

[1233] The emotion engine generates data based on the sensed emotional state and sends it to an internal processor.

[1234] Step 5:

[1235] Terminal (device) or server:

[1236] The system adjusts the way translation results are presented based on the user's emotional state. For example, if the user is feeling stressed, the system will soften the tone of the voice guidance.

[1237] Step 6:

[1238] Device:

[1239] The adjusted voice guidance is played back to guide the user in a manner that is adaptive to the user.

[1240] Specific examples

[1241] User:

[1242] A user is frustrated trying to find a restaurant in a foreign city.

[1243] Device:

[1244] The emotion engine analyzes the user's facial expressions and tone of voice to recognize stress.

[1245] Server or device:

[1246] The tone of the voice guidance is softened based on the emotional state.

[1247] Device:

[1248] In reality, a gentle voice announces, "This is a cafe. Our recommended dish is latte."

[1249] The above is the specific processing flow of this system. By explaining the operation at each step in detail, we will realize a system that is convenient and highly applicable for users.

[1250] Example 2

[1251] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1252] In recent years, communication between people who speak different languages ​​has become increasingly important. Language barriers are a major obstacle, especially when traveling or traveling to foreign countries for business. However, existing translation devices and applications lack the ability to translate visual and audio information in real time and provide guidance. Furthermore, they are unable to provide guidance that takes into account the user's emotional state, which can cause stress and anxiety. In response to this, the present invention aims to provide user-friendly information by translating the user's visual and audio information in real time and adjusting the guidance method to adapt to the user's emotional state.

[1253] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1254] In this invention, the server includes means for analyzing text information and translating it into a specified language, means for converting it into voice data and sending it to the terminal, and means for analyzing the user's emotional state and adjusting the guidance method, which not only translates the user's visual and voice information in real time and provides guidance, but also makes it possible to provide appropriate guidance according to the user's emotional state.

[1255] A "visual assistance device" is a device worn by a user and has the function of acquiring visual information about the surroundings and providing the information to the user.

[1256] A "remote computing device" is a computer device that is remotely located over the Internet and performs data analysis and calculations.

[1257] "Means for capturing" refers to the function of acquiring visual and audio information using a camera, microphone, etc.

[1258] "Means for extracting characters" refers to a function for detecting and extracting character information from visual information.

[1259] "Means for receiving a translation result" refers to a function by which the terminal receives a translation result sent from a remote computing device.

[1260] "Means for generating voice data" refers to a function for converting text information into voice.

[1261] "Means for recognizing emotional state" refers to a function that uses a camera or microphone to analyze the user's facial expressions and tone of voice to identify the user's emotions.

[1262] "Means for adjusting the guidance method" refers to a function that changes the content and tone of the guidance depending on the user's emotional state.

[1263] "Means for detecting spoken language" refers to the function of detecting surrounding conversations and sounds and acquiring them as audio data.

[1264] The "setting means that can be changed based on a voice profile" refers to a function that allows the tone and style of voice guidance to be changed based on voice characteristics selected by the user.

[1265] MODE FOR CARRYING OUT THE INVENTION

[1266] This invention is a system that uses a user-worn visual aid device to automatically translate different languages ​​and provide audio guidance. The visual aid device has a built-in camera, microphone, speaker, communication module, processor, and emotion engine, and complements its processing capabilities by communicating with a remote computing device (server).

[1267] System configuration

[1268] The visual assistance device is a glasses-type device worn by the user that contains a camera, microphone, speaker, communication module, processor, and emotion engine. The device communicates with a remote server to perform advanced processing.

[1269] System Operation

[1270] Visual translation and audio guidance

[1271] The device periodically captures images of its surroundings using a camera and extracts text information from them. The text information is extracted using OCR (Optical Character Recognition) technology. The text information extracted from the captured images is sent to a server via a wireless communication module.

[1272] After receiving the text information, the server translates it into the specified language using a translation model (e.g., a general translation API). The translated text is then converted into voice data using a speech synthesis tool (e.g., a general speech synthesis API). The generated voice data is then sent to the terminal (device) via a communications network.

[1273] The terminal (device) plays the received voice data through a speaker and provides the user with audible guidance on the translation results.

[1274] As a concrete example, consider a situation where a Japanese-speaking user is visiting France and looking for a restaurant. The user wears a device and sees a sign that says "restaurant." The device's camera captures the sign, and a character recognition process extracts the word "restaurant." This data is sent to a server, which generates the English translation of "restaurant," "Restaurant," and then generates a voice prompt saying, "This place is a restaurant. Recommended dishes are pasta and steak." The terminal (device) plays the received voice data and provides the user with instructions.

[1275] Spoken language translation and audio guidance

[1276] The device captures the surrounding sound using a built-in microphone, and the captured sound data is sent directly to the server.

[1277] After receiving the voice data, the server converts it into text using voice recognition technology (e.g., a general voice recognition API), then translates the converted text into the specified language using a translation model, and generates voice data again using a voice synthesis tool. The generated voice data is then sent to the device.

[1278] The terminal (device) plays back the received audio data again and presents the translation results to the user.

[1279] As a concrete example, consider a situation where a Japanese-speaking user in an American market asks, "Where is the nearest station?" The device's microphone captures the question and sends the data to a server. The server translates the voice data into "Where is the nearest station?" and generates voice data that is sent to the terminal (device). The user can then use this voice data to ask local people.

[1280] Emotion Engine Functions

[1281] The device uses a built-in camera and microphone to analyze the user's facial expressions and tone of voice, recognizing their emotional state in real time.

[1282] The server or terminal (device) adjusts the guidance method based on the user's emotional state. For example, if the user is feeling stressed, the tone of the voice guidance is set to a gentler tone. This allows for flexible guidance tailored to the user's emotional state.

[1283] For example, consider a case where a user is searching for a restaurant in a foreign country but is feeling stressed. The device can sense the user's stress and guide them in a calmer tone, saying, "This is a cafe. The recommended dish is a latte."

[1284] This invention enables the translation of a user's visual and audio information in real time, and also provides appropriate guidance according to the user's emotional state. Based on the above explanation, we have shown what hardware and software are used to specifically implement this system.

[1285] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1286] Visual translation and audio guidance

[1287] Step 1:

[1288] User: Wears the device and acquires visual information. Specifically, the user looks at an object with text information, such as a sign or menu.

[1289] Input: Images containing text information, such as signs and menus

[1290] Output: Visual capture request

[1291] Step 2:

[1292] Device: Uses the camera to capture images of the surroundings.

[1293] Input: Captcha request

[1294] Output: Captured image data

[1295] Step 3:

[1296] Terminal (device): The processor analyzes the captured image data and extracts text information using OCR technology.

[1297] Input: Captured image data

[1298] Output: Extracted text information

[1299] Step 4:

[1300] Terminal (device): The extracted text information is sent to the server using a communication module.

[1301] Input: Extracted text information

[1302] Output: Request to send text information

[1303] Step 5:

[1304] Server: Analyzes the received text information and translates it into the specified language using a generative AI model.

[1305] Input: Extracted text information

[1306] Output: The translated text

[1307] Step 6:

[1308] Server: Converts the translated text into audio data using a speech synthesis tool, possibly using a voice profile selected by the user.

[1309] Input: Translated text

[1310] Output: Audio data

[1311] Step 7:

[1312] Server: Generates a request to send the generated audio data to the device.

[1313] Input: Audio data

[1314] Output: Audio data transmission request

[1315] Step 8:

[1316] Device: Plays the received audio data through a speaker.

[1317] Input: Audio data

[1318] Output: Play voice prompts

[1319] Spoken language translation and audio guidance

[1320] Step 1:

[1321] User: Start a conversation with a foreigner, specifically asking a question such as "Where is the nearest station?"

[1322] Input: User conversation

[1323] Output: Audio data capture request

[1324] Step 2:

[1325] Device: Uses a microphone to capture surrounding sounds.

[1326] Input: Audio data capture request

[1327] Output: Captured audio data

[1328] Step 3:

[1329] Device: Sends captured audio data to the server using a communication module.

[1330] Input: Captured audio data

[1331] output: Request to send audio data

[1332] Step 4:

[1333] Server: Analyzes the received voice data and converts it into text using voice recognition technology.

[1334] Input: Captured audio data

[1335] Output: Speech-to-text data

[1336] Step 5:

[1337] Server: Translates the converted text into the specified language using a generative AI model.

[1338] Input: Speech-to-text data

[1339] Output: The translated text

[1340] Step 6:

[1341] Server: The translated text is converted into audio data using a speech synthesis tool.

[1342] Input: Translated text

[1343] Output: Audio data

[1344] Step 7:

[1345] Server: Generates a request to send the generated audio data to the device.

[1346] Input: Audio data

[1347] Output: Audio data transmission request

[1348] Step 8:

[1349] Device: Plays the received audio data through a speaker.

[1350] Input: Audio data

[1351] Output: Play voice prompts

[1352] Emotion Engine Functions

[1353] Step 1:

[1354] Device: Analyzes the user's facial expressions and tone of voice using the built-in camera and microphone.

[1355] Input: User's facial expression, tone of voice

[1356] Output: Emotional state analysis request

[1357] Step 2:

[1358] Terminal (device) or server: Recognizes emotional state based on analyzed data.

[1359] Input: Emotional state analysis request

[1360] Output: Emotional state data

[1361] Step 3:

[1362] Terminal (device) or server: Adjust the tone and content of the announcement depending on the user's emotional state.

[1363] Input: Emotional state data

[1364] Output: Adjusted voice prompts

[1365] Step 4:

[1366] Device: Plays tailored voice prompts through the speaker.

[1367] Input: Adjusted voice prompts

[1368] Output: Play a gentle voice prompt

[1369] (Application example 2)

[1370] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1371] Conventional visual assistance devices are unable to adapt flexibly to the user's emotional state when receiving translation results. Furthermore, language barriers can hinder the smooth shopping experience in physical stores. There is a need for devices that can provide appropriate support even when users feel stressed or confused.

[1372] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for capturing an image including text, means for extracting text from the captured image, means for transmitting the extracted text to a remote computing device for translation, means for receiving a translation result from the remote computing device, means for providing audible guidance to the user of the received translation result, means for recognizing the user's emotional state and adjusting the guidance method, means for adjusting the guidance method of the translated audio based on the emotion recognition, and setting means for changing the audio guidance of the translation result to the voice of a celebrity selected by the user. This enables flexible response according to the user's emotions, making it possible to have a smooth and stress-free shopping experience in a physical store.

[1373] A "visual assistance device" is a device worn by a user to acquire visual information and assist the user's vision.

[1374] "Images containing text" refers to images with text on them, such as signs or product labels.

[1375] "Capturing means" refers to the method of acquiring an image using a camera or other photographic device.

[1376] The "means for extracting characters" is a method for analyzing character information from a captured image and extracting the characters.

[1377] A "remote computing device" is a server or computer that receives data over a network and performs complex calculations or processing.

[1378] The "means for receiving the translation result" is a method for obtaining the translated data from the remote computing device.

[1379] "Means for providing audio guidance to the user" refers to a method for converting the received translation results into audio and conveying it to the user.

[1380] The "means for recognizing the user's emotional state and adjusting the guidance method" is a method for analyzing the user's facial expression and tone of voice and changing the guidance method according to the user's emotions.

[1381] The "means for detecting spoken words and capturing audio data" is a method for detecting spoken words using an audio input device such as a microphone and acquiring the audio as data.

[1382] The "means for adjusting the guidance method of the translated voice based on emotion recognition" is a method for analyzing the user's emotions through emotion recognition and changing the tone and content of the translated voice accordingly.

[1383] The "setting means for changing to a celebrity's voice" is a method for changing the settings so that guidance is given using the voice of a celebrity selected by the user.

[1384] This invention is a system that supports the shopping experience in a physical store as a visual aid device worn by a user. The system is composed of smart glasses worn by a user and a remote computing device (server).

[1385] Hardware and Software

[1386] Hardware: The smart glasses contain a camera, microphone, speaker, processor, and communication module. The camera captures images containing text information, and the microphone captures audio data. The speaker is used to provide the user with audible translation results.

[1387] software:

[1388] Image processing library (OpenCV): Used to extract characters from captured images.

[1389] Speech Recognition Library (Google Speech-to-Text API): Used to convert captured voice data into text.

[1390] Translation API (Google Translate API): Used to translate the extracted text into a specified language.

[1391] Speech synthesis library (Google Text-to-Speech API): Used to convert the translation text into audio data.

[1392] Emotion Recognition Engine (Emotion API): Used to recognize the user's emotional state and adjust guidance methods.

[1393] System Operation

[1394] User interaction: A user puts on the smart glasses and moves around a physical store. When the user wants to check the price label or description of a product, the camera captures the text.

[1395] Terminal (smart glasses) processing:

[1396] 1. The processor analyzes the image captured by the camera and extracts text information.

[1397] 2. The extracted text information is sent to the server.

[1398] Server Action:

[1399] 1. The server uses a translation API to translate the text received into the specified language.

[1400] 2. Convert the translated text into audio data using a speech synthesis library.

[1401] 3. Use an emotion recognition engine to analyze the user's emotional state and adjust the tone of the voice prompts if the user is feeling stressed.

[1402] Terminal (smart glasses) processing:

[1403] 1. Receives voice data sent from the server and provides voice guidance through the speaker.

[1404] Specific examples

[1405] Consider shopping in a brick-and-mortar store. For example, a foreign tourist wearing smart glasses while shopping in a Japanese department store. The camera captures information about the shelves and store directions, and the system translates the text in real time and provides voice guidance. If the user becomes lost or stressed, the tone of the guidance changes to a gentler tone, providing a sense of security.

[1406] Example prompt sentence:

[1407] A user wears smart glasses and looks at shelves in a department store. The system captures information about the shelves, translates it into the user's language in real time, and provides voice guidance. If the user feels stressed, the tone of the guidance changes to a gentler tone to provide reassurance. Please explain how this system is implemented and provide specific examples of API usage.

[1408] In this way, the visual aid device adapts to the user's emotional state and helps facilitate a smooth shopping experience in a physical store.

[1409] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1410] Step 1:

[1411] A user wears smart glasses and moves around a physical store. When the user wants to check the price label or description of a product, the camera captures the text information. Here, the input is the captured image, and the output is an image file containing the text information.

[1412] Step 2:

[1413] The processor in the device (smart glasses) analyzes the captured image and extracts text information. Specifically, it uses an image processing library (OpenCV) to identify the text and then uses optical character recognition (OCR) technology to convert the text into text data. Here, the input is an image file and the output is the extracted text information (text data).

[1414] Step 3:

[1415] The terminal (smart glasses) transmits the extracted text information to a remote computing device (server). A communication module is used to send the text data to the server. The input of this step is the text data, and the output is the data transmitted to the server.

[1416] Step 4:

[1417] The server translates the received text into the specified language using a translation API (Google Translate API). Here, the server first inputs the text data, processes it through the translation API, and obtains the translation result. The input is the received text information, and the output is the translated text data.

[1418] Step 5:

[1419] The server converts the translated text data into audio data using a speech synthesis library (Google Text-to-Speech API). Specifically, the translated text is input into the speech synthesis API to generate an audio file. The input is the translated text data, and the output is an audio file.

[1420] Step 6:

[1421] Here, the server uses an emotion recognition engine (Emotion API) to analyze the user's emotional state. The camera and microphone capture the user's facial expressions and tone of voice, and send the data to the server. The input is facial expression data and voice data, and the output is the user's emotional state (e.g., stress, joy, surprise).

[1422] Step 7:

[1423] The server adjusts the tone of the voice guidance based on the emotion recognition results. For example, if the user is feeling stressed, the tone of the voice guidance is set to be gentler. The input is the emotional state and the audio file, and the output is the adjusted audio file.

[1424] Step 8:

[1425] The server sends the adjusted audio file to the device (smart glasses). The input of this step is the adjusted audio file, and the output is the data sent to the device.

[1426] Step 9:

[1427] The speaker on the device (smart glasses) plays the received audio data and provides the translation results to the user by voice. The input is the received audio data, and the output is the audio guidance the user hears.

[1428] In this way, visual aid devices can help users have a smoother shopping experience in brick-and-mortar stores. Furthermore, by providing flexible guidance that responds to the user's emotional state, it can provide more user-friendly information.

[1429] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1430] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1431] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1432] [Third embodiment]

[1433] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1434] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1435] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1436] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1437] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1438] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1439] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1440] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1441] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1442] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1443] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1444] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1445] This invention is a system that automatically translates different languages ​​and provides audio guidance as a visual aid device worn by the user. This system can acquire visual information, translate it in real time, and provide audio guidance. It also has the function of translating spoken language and providing audio guidance.

[1446] This visual assistance device (hereinafter referred to as the device) is in the form of glasses worn by the user and contains a camera, microphone, speaker, communication module, and processor. The device supplements its processing capabilities by communicating with a remote computing device (hereinafter referred to as the server).

[1447] 1. Visual translation and audio guidance

[1448] Camera and character extraction

[1449] User:

[1450] The device recognizes objects with text information, such as signs and menus, in the user's field of vision.

[1451] Device:

[1452] The device's camera periodically captures images of its surroundings, which are then analyzed by a processor to extract textual information.

[1453] Sending translation requests and receiving translation results

[1454] Device:

[1455] The extracted text information is sent to the server via the communication module, and the data sent also includes information about the language to be translated.

[1456] server:

[1457] The server analyzes the received text and translates it into the specified language. After generating the translation results, it constructs them as text for voice guidance. It then converts the text into audio data using the voice of a celebrity selected by the user and sends it to the device.

[1458] Device:

[1459] The received audio data is played back and the translation results are presented to the user.

[1460] Specific examples

[1461] User:

[1462] Suppose a Japanese-speaking user is visiting France and looking for a restaurant.

[1463] Device:

[1464] A camera captures a sign that says "Restaurant," and a character recognition process extracts the word "Restaurant." The extracted character data is sent to a server.

[1465] server:

[1466] The server receives the character "restaurant" and translates it into English as "Restaurant." Based on the translation, it generates a narration saying "This place is a restaurant. Recommended dishes are pasta and steak." This is converted into audio data and sent to the device.

[1467] Device:

[1468] The voice guidance played back informs the user that "This is a restaurant. Recommended dishes are pasta and steak."

[1469] 2. Spoken language translation and audio guidance

[1470] Microphone and voice data capture

[1471] User:

[1472] Imagine a situation where a user is communicating with local people in a foreign market.

[1473] Device:

[1474] The device's microphone constantly monitors the surrounding sound and captures audio data, such as conversations, which are then stored as audio data.

[1475] Sending translation requests and receiving translation results

[1476] Device:

[1477] The stored voice data is transmitted to a server via a communication module.

[1478] server:

[1479] The server receives the voice data, converts it into text using speech recognition technology, then translates the text into the specified language and generates the translation as voice data, which is then sent to the device.

[1480] Device:

[1481] The received audio data is played back and the translation results are presented to the user.

[1482] Specific examples

[1483] User:

[1484] Consider a scenario in which a Japanese-speaking user in an American market asks "Where is the nearest station?" to get directions.

[1485] Device:

[1486] A microphone captures the user's question and transmits the audio data to the server.

[1487] server:

[1488] The server receives the speech data "Where is the nearest station?" and translates it into "Where is the nearest station?". It generates the translated speech data and sends it to the device.

[1489] Device:

[1490] The audio guide played allows the user to ask local people, "Where is the nearest station?"

[1491] In this way, the present invention allows users to translate visual and audio information in real time and provide appropriate guidance, thereby enabling smooth communication even in environments where different languages ​​are spoken. Furthermore, by providing audio guidance in the voice of a celebrity selected by the user, a more friendly service can be provided.

[1492] The processing flow will be explained below.

[1493] Processing flow for translation of visual information and voice guidance

[1494] Visual translation

[1495] Step 1:

[1496] User:

[1497] A user wearing the device looks at visual information, such as a sign or a menu.

[1498] Step 2:

[1499] Device:

[1500] The device's camera periodically captures images of its surroundings, which are then sent to a processor.

[1501] Step 3:

[1502] Device:

[1503] The processor uses image analysis algorithms to extract text information from the captured image, for example, using optical character recognition (OCR) technology to convert the characters into text data.

[1504] Step 4:

[1505] Device:

[1506] A request is generated to transmit the extracted character data and the linguistic information to be translated to a remote computing device (server).

[1507] Step 5:

[1508] Device:

[1509] The generated request is sent to the server through a communication module.

[1510] Step 6:

[1511] server:

[1512] The server receives the request, parses the text, and feeds the text into the corresponding translation model to perform translation into the specified language.

[1513] Step 7:

[1514] server:

[1515] Once the translation results are generated, a narration is created based on the translated text. For example, it generates a sentence like, "This is a cafe. The recommended menu item is latte."

[1516] Step 8:

[1517] server:

[1518] To convert the narration text into audio data, text-to-speech synthesis (TTS) is performed using the voice of a celebrity selected by the user.

[1519] Step 9:

[1520] server:

[1521] The generated voice data is transmitted to the terminal.

[1522] Step 10:

[1523] Device:

[1524] The terminal plays back the audio data received from the server and provides the user with the translation results by voice.

[1525] Specific examples

[1526] User:

[1527] A Japanese speaking user sees a sign in French.

[1528] Device:

[1529] The camera captures the sign and uses OCR technology to extract the word "restaurant."

[1530] server:

[1531] The server receives "restaurant" and translates it as "restaurant." It then generates a narration saying, "This is a restaurant. The recommended dish is pasta," converts it into voice data, and sends it to the device.

[1532] Device:

[1533] The device plays the audio data and informs the user of the translation results. The voice will say in Japanese, "This is a restaurant. The recommended dish is pasta."

[1534] ---

[1535] Spoken language translation and voice guidance processing flow

[1536] Audio translation

[1537] Step 1:

[1538] User:

[1539] The user puts on the device and attempts to converse with local people.

[1540] Step 2:

[1541] Device:

[1542] The device's microphone constantly monitors the surrounding sounds and captures what is being said.

[1543] Step 3:

[1544] Device:

[1545] The captured audio data is sent to a processor.

[1546] Step 4:

[1547] Device:

[1548] A processor temporarily stores the audio data and generates a request to send it to a server via a communication module.

[1549] Step 5:

[1550] Device:

[1551] The generated request is sent to the server through a communication module.

[1552] Step 6:

[1553] server:

[1554] The server receives the voice data and converts it into text using speech recognition technology, for example, "Where is the nearest station?"

[1555] Step 7:

[1556] server:

[1557] Translate the recognized text into the specified language, for example, "Where is the nearest station?" into English.

[1558] Step 8:

[1559] server:

[1560] To generate audio data based on the translated text, TTS is used to create audio data in a voice selected by the user.

[1561] Step 9:

[1562] server:

[1563] The generated voice data is transmitted to the terminal.

[1564] Step 10:

[1565] Device:

[1566] The terminal plays back the audio data received from the server and provides the user with the translation results by voice.

[1567] Specific examples

[1568] User:

[1569] A Japanese speaking user wants to ask for directions in the United States.

[1570] Device:

[1571] The microphone captures the user's question, "Where is the nearest station?" and sends the voice data to the server.

[1572] server:

[1573] The server receives the voice data, converts it into text, and translates it. The text "Where is the nearest station?" is converted into voice data and sent to the device.

[1574] Device:

[1575] The device will play the translated audio and the user can ask a local person, "Where is the nearest station?"

[1576] The above is the specific processing flow of this system, and the operation at each step is explained in detail.

[1577] Example 1

[1578] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1579] In modern society, people who speak different languages ​​coexist, and smooth communication is essential. However, language barriers make it difficult to understand signs and menus, or to converse smoothly with local people, both in daily life and while traveling. To solve these problems, a visual aid device is needed that can translate in real time and provide audio guidance.

[1580] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1581] In this invention, the server includes means for capturing an image containing text, means for extracting text from the captured image, means for translating the extracted text, means for receiving a translation result from a remote computing device, means for providing audible guidance to a user about the received translation result, and means for converting the received translation result into audio data in a specified voice, thereby enabling real-time translation of text and conversations displayed in different languages ​​and audio guidance.

[1582] 1. "Visual assistance device" means a device worn by a user that has the function of acquiring, processing, and providing visual information.

[1583] 2. "Capturing means" is a collective term for the hardware and software that a device uses to capture images and audio.

[1584] 3. "Text extraction means" means a process or device that has the ability to identify text information from a captured image and extract it as data.

[1585] 4. "Remote Computing Device" means a device that provides remotely located computing resources for processing data transmitted from a Device.

[1586] 5. "Translation means" means software and hardware functions for converting written or spoken information in one language into another language.

[1587] 6. "Means for receiving translation results" means a device that has the function of receiving translation data sent from a remote computing device on the device side.

[1588] 7. "Means for providing audio guidance to the user" refers to the device's function of converting the received translation results into audio format and notifying the user by audio.

[1589] 8. "Means for converting into voice data using a specified voice" means a device that has the function of converting the translation results into voice data using a specific voice that the user has specified in advance.

[1590] 9. "Audio data capture means" means a device or process that captures a user's speech or surrounding sounds and stores them as digital audio data.

[1591] 10. "Means for receiving translated voice data" means a device that has the capability to receive translated voice data sent from a remote computing device.

[1592] 11. "Means for providing information to the user" refers to functions or devices for providing acquired or received information to the user in an easy-to-understand manner.

[1593] 12. "Configuration means" means the interface and software functionality that allows a user to customize the operation and functionality of a device to suit the user's preferences.

[1594] This invention provides a multilingual visual aid device, enabling users to translate visual and audio information in real time and provide audio guidance in different language environments. This invention achieves this by using a capture device, a remote server, speech synthesis technology, and a communication module.

[1595] Hardware and software configurations used

[1596] 1. Wearable visual aids:

[1597] Camera: Captures visual information about the surroundings.

[1598] Microphone: Captures surrounding sounds.

[1599] Speaker: A means of providing the translation results to the user aloud.

[1600] Communications Module: A means for sending and receiving data to and from a remote computing device.

[1601] Processor: Processes image and audio data.

[1602] 2. Remote Computing Device (Server):

[1603] Character recognition software: Extracts text information from images (e.g., Tesseract OCR).

[1604] Speech recognition technology: Converts captured voice data into text (e.g., Google Speech-to-Text).

[1605] Translation engine: Translates text from one language to another (e.g., Google Translate API, DeepL).

[1606] Speech synthesis engine: Converts text to speech (e.g., Amazon Polly, Google Text-to-Speech).

[1607] System Operation

[1608] Visual translation and audio guidance

[1609] A visual assistance device worn by a user captures visual information from the surroundings using a camera. For example, the camera may capture a photo of a sign for a French restaurant. The processor of the terminal (device) analyzes the captured image and uses character recognition software to extract the text information "restaurant." This text information is then sent to a server via a communications module. The server analyzes the received text data, translates it into the specified language (e.g., from French to English), and outputs the translation as "Restaurant." Based on the translation, the server then generates a narration such as "This place is a restaurant. Recommended dishes are pasta and steak." This is converted into audio data using a speech synthesis engine and sent to the device. The terminal (device) receives the audio data and plays it through a speaker, so the user is informed that "This place is a restaurant. Recommended dishes are pasta and steak."

[1610] Spoken language translation and audio guidance

[1611] Consider a scenario in which a user in a foreign market asks, "Where is the nearest station?" A microphone installed on the terminal (device) captures surrounding sounds and sends the voice data to a server. The server receives the voice data and converts it into text using voice recognition technology. It then translates the text into a specified language (e.g., Japanese to English), generates voice data based on the translation results, and sends it to the device. The terminal (device) receives the voice data and asks the user, "Where is the nearest station?"

[1612] Examples of concrete examples and prompts

[1613] Specific examples

[1614] A Japanese-speaking user visits France and a camera captures a sign that says "Restaurant." The text data "Restaurant" is sent to the server. The server translates "Restaurant" into "Restaurant," generates a narration saying, "This place is a restaurant. Recommended dishes are pasta and steak.", and sends this voice data to the device. By listening to this voice guidance, the user learns, "This place is a restaurant. Recommended dishes are pasta and steak."

[1615] Prompt Sentence Examples

[1616] "Translate the text written on the sign. Generate a prompt to translate from French to English: 'Extract text from captured image and translate it into the specified language.'"

[1617] "Generate a prompt that translates the Japanese question 'Where is the nearest station?' into English: 'Use speech recognition technology to convert captured speech into text, and then translate that text into the specified language.'"

[1618] In this way, the present invention is a system that can translate visual and audio information in real time in a multilingual environment and provide audio guidance to the user.

[1619] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1620] Visual translation and audio guidance

[1621] Step 1: Capture visual information

[1622] Device:

[1623] While a user is wearing a visual aid device, the device's camera periodically captures visual information about the surroundings.

[1624] Input: Objects in the user's field of view (e.g., signs, menus)

[1625] Data processing: The image capture process is activated and the camera acquires the image data.

[1626] Output: Acquired raw image data

[1627] What happens: The device's camera takes a picture every second.

[1628] Step 2: Extracting characters

[1629] Device:

[1630] The processor analyzes the captured image data in real time and extracts text information using optical character recognition software (OCR).

[1631] Input: Captured image data

[1632] Data processing: Extracting text data from images using OCR algorithms (e.g., Tesseract).

[1633] Output: Extracted text data (e.g., "restaurant")

[1634] What it does: The processor recognizes characters in the image and generates data in text format.

[1635] Step 3: Submit a translation request

[1636] Device:

[1637] The extracted text data and the language information to be translated are sent to the server via the communication module.

[1638] Input: Extracted text data, language information to be translated (e.g. French to English)

[1639] Data processing: Constructing data packets and sending them to the server using a communication protocol (e.g., HTTP).

[1640] Output: Data packet sent to the server

[1641] What happens: The device sends the text "restaurant" and a translation request from French to English.

[1642] Step 4: Translation and Narration Generation

[1643] server:

[1644] The received text data is analyzed and translated into the specified language, and a narration is generated based on the translation results.

[1645] Input: Text data (e.g., "restaurant"), language information to be translated

[1646] Data processing: Translate to "Restaurant" using a translation engine (e.g., Google Translate API). Then generate a voiceover sentence: "This place is a restaurant. Recommended dishes are pasta and steak."

[1647] Output: Translated text and narration

[1648] Specific operation: The server translates "restaurant" to "Restaurant" and constructs a guide text.

[1649] Step 5: Generate audio data

[1650] server:

[1651] The narration sentences are converted into voice data using a voice synthesis engine.

[1652] Input: Narration (e.g., "This place is a restaurant. Recommended dishes are pasta and steak.")

[1653] Data processing: Generate voice data using a speech synthesis engine (e.g., Amazon Polly).

[1654] Output: Audio data

[1655] Specific operation: The server generates audio based on the narration text.

[1656] Step 6: Sending voice data and guidance

[1657] Device:

[1658] The audio data is received from the server and played back to the user through the speaker.

[1659] Input: Audio data received from the server

[1660] Data processing: Decodes the received data and plays it on the speaker.

[1661] Output: A voice prompt to the user

[1662] What happens: The speaker plays a voice saying, "This place is a restaurant. Recommended dishes are pasta and steak."

[1663] ---

[1664] Spoken language translation and audio guidance

[1665] Step 1: Capture audio data

[1666] Device:

[1667] The microphone captures the surrounding sounds.

[1668] Input: User speech and surrounding audio (e.g., "Where is the nearest station?")

[1669] Data processing: Recording audio data and converting it into digital format.

[1670] Output: Captured audio data

[1671] What it does: A microphone records the user's question.

[1672] Step 2: Sending audio data

[1673] Device:

[1674] The captured audio data is transmitted to a server via a communication module.

[1675] Input: Audio data

[1676] Data processing: Constructing data packets and sending them to the server using a communication protocol.

[1677] Output: Audio data sent to the server

[1678] Specific operation: The device sends voice data saying "Where is the nearest station?"

[1679] Step 3: Transcribe and translate the audio data

[1680] server:

[1681] The received voice data is converted into text using voice recognition technology and then translated.

[1682] Input: Speech data (e.g., "Where is the nearest station?")

[1683] Data processing: Use speech recognition technology (e.g., Google Speech-to-Text) to convert the speech to text, then use a translation engine to translate it into "Where is the nearest station?"

[1684] Output: Translated text data

[1685] What happens: The server converts the voice data into text and translates it into English.

[1686] Step 4: Generate translated audio data

[1687] server:

[1688] The translated text data is converted into voice data using a voice synthesis engine.

[1689] Input: Translated text data (e.g., "Where is the nearest station?")

[1690] Data processing: Generate voice data using a speech synthesis engine.

[1691] Output: Audio data

[1692] Specific operation: The server generates audio data based on the translation results.

[1693] Step 5: Voice prompts

[1694] Device:

[1695] The generated audio data is received and played back to the user through a speaker.

[1696] Input: Audio data

[1697] Data processing: Decodes the received data and plays it on the speaker.

[1698] Output: A voice prompt to the user

[1699] Specific action: The speaker will play a voice saying "Where is the nearest station?"

[1700] In this way, the system captures visual and audio information in real time, translates it, and provides audio guidance, thereby providing smooth communication that transcends language barriers.

[1701] Examples of concrete examples and prompts

[1702] Specific examples

[1703] A Japanese-speaking user uses a camera to capture a sign that says "restaurant" in a French restaurant, and the text data "restaurant" is sent to the server. The server translates "restaurant" into "Restaurant" and generates a narration saying, "This place is a restaurant. Recommended dishes are pasta and steak." This is sent as audio data to the device. After listening to the audio guidance, the user learns, "This place is a restaurant. Recommended dishes are pasta and steak."

[1704] A Japanese-speaking user in an American market asks, "Where is the nearest station?" The device captures the speech and sends it to the server. The server translates "Where is the nearest station?" to "Where is the nearest station?" and sends it as voice data to the device. The device plays the speech and tells the user, "Where is the nearest station?"

[1705] Prompt Sentence Examples

[1706] "Translate the text written on the sign. Generate a prompt to translate from French to English: 'Extract text from captured image and translate it into the specified language.'"

[1707] "Generate a prompt that translates the Japanese question 'Where is the nearest station?' into English: 'Use speech recognition technology to convert captured speech into text, and then translate that text into the specified language.'"

[1708] This system is designed to support smooth information acquisition and communication in a multilingual environment.

[1709] (Application example 1)

[1710] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1711] There is a problem in that users who speak different languages ​​cannot communicate smoothly in autonomous vehicles (e.g., self-driving vehicles). It is also difficult to understand visual information such as signs and road signs written in different languages, which can make travel in a self-driving vehicle unstable. Furthermore, local guidance and responses to questions cannot be provided immediately, which causes great stress for tourists and foreign users.

[1712] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1713] In this invention, the server includes a processing device for translating visual information, a processing device for translating audio information, and a processing device for converting received translation results into audio data. This enables real-time translation of visual and audio information in different languages ​​within an autonomous mobile body. The translation results are provided to the user via audio guidance via the visual assistance system, allowing the user to quickly understand route guidance to their destination and emergency information. In addition, user-selected audio guidance is provided in a user-friendly format.

[1714] A "visual aid system" is a device worn by a user that captures visual information and provides the information to the user visually or audibly.

[1715] "Visual information" refers to information such as characters and figures that are included in the user's field of vision.

[1716] A "remote processor" is a remotely located computing device that processes data sent from a visual aid system and returns the results.

[1717] An "autonomous mobile object" is a vehicle or device that can move autonomously and reach its destination using artificial intelligence and sensor technology, such as a self-driving vehicle.

[1718] "Audio data" refers to audio signals captured by an audio input device such as a microphone and converted into digital data.

[1719] "Translation result" is text or audio data that has been converted into a different language by a remote processor.

[1720] "Famous person's voice" is voice data of a commonly recognized person selected by the user.

[1721] "Guidance means" refers to a mechanism or device that allows the visual aid system to provide information to the user.

[1722] A "processing device" is a computer or hardware for processing and analyzing data.

[1723] "Friendly voice guidance" refers to voice guidance provided in a format that is easy for the user to hear and understand.

[1724] This invention utilizes a visual aid system worn by a user to translate visual and audio information in different languages ​​in real time within an autonomous vehicle, and provides audio guidance. The system of the present invention includes a visual aid system, a remote processor, and means used within the autonomous vehicle.

[1725] Hardware and Software

[1726] Visual aid systems:

[1727] The smart glasses are worn by the user and contain a camera, microphone, speaker, communication module, and processor. The system is responsible for capturing visual and audio information and transmitting that data to a remote processor.

[1728] Remote Processor:

[1729] Extract text from visual information (e.g., signs and road signs) and translate it into different languages. Also convert audio data to text, translate it, and convert it back to audio. Use software such as Google Translate API, SpeechRecognition, and gTTS (Google Text-to-Speech).

[1730] Autonomous mobile:

[1731] This includes autonomous vehicles, which are equipped with smart glasses worn by the user and have a communication module that connects to a remote processor via a network.

[1732] Data Processing Overview

[1733] Visual information processing:

[1734] 1. Server: Receives the surrounding images sent from the smart glasses and extracts the text data using character recognition technology.

[1735] 2. Server: Use the Google Translate API to translate the extracted text data into the specified language.

[1736] 3. Server: Uses gTTS to convert the translation results into audio data and sends it to the smart glasses.

[1737] 4. Terminal: Plays back the received audio data and displays the translation results to the user.

[1738] Processing of audio information:

[1739] 1. Device: The microphone in the smart glasses captures the surrounding sounds and sends the audio data to the server.

[1740] 2. Server: Use SpeechRecognition to convert the received audio data into text data.

[1741] 3. Server: Uses Google Translate API and gTTS to translate text data into the specified language and convert it back to audio data.

[1742] 4. Terminal: Plays back the received audio data and displays the translation results to the user.

[1743] Specific examples

[1744] Example 1:

[1745] Consider a situation where a traveler is using an autonomous taxi. To understand a tourist information board they come across on the way to their destination, the smart glasses capture the text information on the sign and send it to a server. The server returns the translated results as audio data and provides audio guidance to the user through the smart glasses. This allows the traveler to instantly understand information about their destination and tourist spots.

[1746] Example 2:

[1747] Imagine a situation where a traveler communicates with local people in an autonomous taxi. When the user asks a question using the microphone in the smart glasses, the voice data is sent to the server and translated. The translated result is played back as voice data, allowing the traveler to communicate smoothly with local people.

[1748] Example prompt for a generative AI model:

[1749] Input: "Describe how you would implement a system that uses smart glasses in a self-driving taxi to translate destination signs and provide voice directions."

[1750] Output: "This implementation utilizes the camera, microphone, and speaker built into the smart glasses. The camera captures visual information and extracts text data using character recognition technology. The extracted text is translated in real time and audio guidance is provided to the user through the smart glasses' speaker. The microphone also captures audio, which is converted into text using speech recognition technology. This is then translated and audio guidance is provided through the speaker."

[1751] In this way, the system of the present invention enables users who speak different languages ​​to understand visual and audio information in different languages ​​within an autonomous mobile body and to communicate smoothly.

[1752] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1753] Step 1:

[1754] Terminal: The camera in the smart glasses captures visual information from the surroundings. The captured image data is sent to the processor. Specifically, the camera periodically acquires images, which are then processed by the internal processor. The input is the captured visual information, and the output is the image data passed to the processor.

[1755] Step 2:

[1756] Terminal: The processor extracts character information from the captured image data. It uses OCR (Optical Character Recognition) technology to convert the characters in the image into text data. The input is the image data from step 1, and the output is the extracted character data.

[1757] Step 3:

[1758] Terminal: The extracted character data is sent to the remote processor via the communication module, where the sent data is passed to the server through the network. The input is the character data, and the output is the data sent to the remote processor.

[1759] Step 4:

[1760] Server: Translates received text data into a specified language. Uses the Google Translate API to convert text data into a different language. The input is the text data to be translated, and the output is the translation result.

[1761] Step 5:

[1762] Server: Converts the translation results into audio data. Using gTTS (Google Text-to-Speech), converts text data into an audio file. The input is the translated text data, and the output is audio data.

[1763] Step 6:

[1764] Server: Transmits the converted voice data to the smart glasses. The voice data is sent to the terminal via the communication module. The input is the voice data, and the output is the voice data sent to the smart glasses.

[1765] Step 7:

[1766] Terminal: The received audio data is played on the smart glasses' speaker, providing audio guidance to the user. Specifically, the audio data is decoded and played as audio through the speaker. The input is audio data, and the output is audio guidance provided to the user.

[1767] Step 8:

[1768] Terminal: The microphone in the smart glasses captures the surrounding sound. The captured sound data is sent to the processor. The input is the captured sound data, and the output is the sound data passed to the processor.

[1769] Step 9:

[1770] Terminal: Transmits the captured audio data to a remote processor via a communication module, where the transmitted data is passed to the server via the network. The input is the audio data, and the output is the data transmitted to the remote processor.

[1771] Step 10:

[1772] Server: Converts received voice data to text. Using SpeechRecognition, converts voice data to text data. The input is voice data, and the output is text data.

[1773] Step 11:

[1774] Server: Translates the converted text data into the specified language. The input is the text data, and the output is the translated text data.

[1775] Step 12:

[1776] Server: The translated text data is converted back into audio data using gTTS. The input is the translated text data, and the output is the audio data.

[1777] Step 13:

[1778] Server: Transmits the converted voice data to the smart glasses. The voice data is sent to the terminal via the communication module. The input is the voice data, and the output is the voice data sent to the smart glasses.

[1779] Step 14:

[1780] Terminal: The received audio data is played on the smart glasses' speaker, providing audio guidance to the user. Specifically, the audio data is decoded and played as audio through the speaker. The input is audio data, and the output is audio guidance provided to the user.

[1781] Through these processing steps, the visual aid system translates visual and audio information in different languages ​​in real time within the autonomous vehicle and provides audio guidance to the user.

[1782] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1783] This invention is a system that automatically translates different languages ​​and provides audio guidance as a visual aid device worn by the user. This system can acquire visual and audio information, translate it in real time, and provide audio guidance. It also has the ability to recognize the user's emotions and adjust the guidance method based on the user's emotional state.

[1784] System configuration

[1785] The visual assistance device (hereafter referred to as the device) is a pair of glasses worn by the user, and contains a camera, microphone, speaker, communication module, processor, and emotion engine. The device supplements its processing capabilities by communicating with a remote computing device (hereafter referred to as the server).

[1786] Visual translation and audio guidance

[1787] Camera and character extraction

[1788] User:

[1789] The user wearing the device can recognize objects that contain text information, such as signs and menus.

[1790] Device:

[1791] The device's camera periodically captures images of its surroundings, which are then analyzed by a processor to extract textual information.

[1792] server:

[1793] The server analyzes the received text and translates it into the specified language. After generating the translation results, it constructs them as text for voice guidance. It then converts the text into audio data using the voice of a celebrity selected by the user and sends it to the device.

[1794] Device:

[1795] The received audio data is played back and the translation results are presented to the user.

[1796] Specific examples

[1797] User:

[1798] Suppose a Japanese-speaking user is visiting France and looking for a restaurant.

[1799] Device:

[1800] A camera captures a sign that says "Restaurant," and a character recognition process extracts the word "Restaurant." The extracted character data is sent to a server.

[1801] server:

[1802] The server receives "restaurant" and translates it into English as "Restaurant." It then generates a narration saying, "This place is a restaurant. Recommended dishes are pasta and steak.", converts it into voice data, and sends it to the device.

[1803] Device:

[1804] The voice guidance played back informs the user that "This is a restaurant. Recommended dishes are pasta and steak."

[1805] Spoken language translation and audio guidance

[1806] Microphone and voice data capture

[1807] User:

[1808] Imagine a situation where a user is communicating with local people in a foreign market.

[1809] Device:

[1810] The device's microphone constantly monitors the surrounding sound and captures audio data, such as conversations, which are then stored as audio data.

[1811] server:

[1812] The server receives the voice data, converts it into text using speech recognition technology, then translates the text into the specified language and generates the translation as voice data, which is then sent to the device.

[1813] Device:

[1814] The received audio data is played back and the translation results are presented to the user.

[1815] Specific examples

[1816] User:

[1817] Consider a scenario where a Japanese-speaking user is asking for directions in an American market.

[1818] Device:

[1819] The microphone captures the user's question, "Where is the nearest station?" and sends the voice data to the server.

[1820] server:

[1821] The server receives the voice data, converts it into text, and translates it. The text "Where is the nearest station?" is generated, converted into voice data, and sent to the device.

[1822] Device:

[1823] The audio guide played allows the user to ask local people, "Where is the nearest station?"

[1824] Emotion Engine Functions

[1825] Emotion recognition and guidance adjustment

[1826] Device:

[1827] It has a built-in emotion engine that analyzes the user's facial expressions and tone of voice captured by the camera and microphone, recognizing the user's emotional state in real time and adjusting the way translation results are presented based on that information.

[1828] Server or device:

[1829] The emotion engine analyzes the user's current emotional state (e.g., happy, sad, anger, surprise) and adjusts the tone and content of the announcements accordingly. For example, if the user is feeling stressed, a calming voice announcement setting will be automatically selected.

[1830] Specific examples

[1831] User:

[1832] A user may feel stressed while trying to find a restaurant in a foreign country.

[1833] Device:

[1834] The emotion engine detects the user's facial expressions and tone of voice and recognizes when the user is feeling stressed.

[1835] Server or device:

[1836] The tone of the translated voice guidance is softened to reflect the user's emotional state. For example, a message such as "This is a cafe. The recommended dish is latte" will be played in a softer voice.

[1837] This invention not only translates the user's visual and audio information in real time and provides audio guidance based on that information, but also grasps the user's emotional state and adjusts the guidance method accordingly, thereby realizing more user-friendly information provision.

[1838] The processing flow will be explained below.

[1839] Processing flow for translation of visual information and voice guidance

[1840] Visual translation

[1841] Step 1:

[1842] User:

[1843] A user wearing the device looks at visual information, such as a sign or a menu.

[1844] Step 2:

[1845] Device:

[1846] The device's camera periodically captures images of its surroundings, which are then sent to a processor.

[1847] Step 3:

[1848] Device:

[1849] The processor uses image analysis algorithms to extract text information from the captured image, for example, using optical character recognition (OCR) technology to convert the characters into text data.

[1850] Step 4:

[1851] Device:

[1852] A request is generated to transmit the extracted character data and the linguistic information to be translated to a remote computing device (server).

[1853] Step 5:

[1854] Device:

[1855] The generated request is sent to the server through a communication module.

[1856] Step 6:

[1857] server:

[1858] The server receives the request, parses the text, and feeds the text into the corresponding translation model to perform translation into the specified language.

[1859] Step 7:

[1860] server:

[1861] Once the translation results are generated, a narration is created based on the translated text. For example, it generates a sentence like, "This is a cafe. The recommended menu item is latte."

[1862] Step 8:

[1863] server:

[1864] To convert the narration text into audio data, text-to-speech synthesis (TTS) is performed using the voice of a celebrity selected by the user.

[1865] Step 9:

[1866] server:

[1867] The generated voice data is transmitted to the terminal.

[1868] Step 10:

[1869] Device:

[1870] The terminal plays back the audio data received from the server and provides the user with the translation results by voice.

[1871] Specific examples

[1872] User:

[1873] A Japanese speaking user sees a sign in French.

[1874] Device:

[1875] The camera captures the sign and uses OCR technology to extract the word "restaurant."

[1876] server:

[1877] The server receives "restaurant" and translates it as "restaurant." It then generates a narration saying, "This is a restaurant. The recommended dish is pasta," converts it into voice data, and sends it to the device.

[1878] Device:

[1879] The terminal plays back the audio data and provides the user with a voice guide saying, "This is a restaurant. The recommended dish is pasta."

[1880] ---

[1881] Spoken language translation and voice guidance processing flow

[1882] Audio translation

[1883] Step 1:

[1884] User:

[1885] The user attempts to converse with local people through a device worn by the user.

[1886] Step 2:

[1887] Device:

[1888] The device's microphone constantly monitors the surrounding sounds and captures what is being said.

[1889] Step 3:

[1890] Device:

[1891] The captured audio data is sent to a processor.

[1892] Step 4:

[1893] Device:

[1894] A processor temporarily stores the audio data and generates a request to send it to a server via a communication module.

[1895] Step 5:

[1896] Device:

[1897] The generated request is sent to the server through a communication module.

[1898] Step 6:

[1899] server:

[1900] The server receives the voice data and converts it into text using speech recognition technology, for example, "Where is the nearest station?"

[1901] Step 7:

[1902] server:

[1903] Translate the recognized text into the specified language, for example, "Where is the nearest station?" into English.

[1904] Step 8:

[1905] server:

[1906] To generate audio data based on the translated text, TTS is used to create audio data in a voice selected by the user.

[1907] Step 9:

[1908] server:

[1909] The generated voice data is transmitted to the terminal.

[1910] Step 10:

[1911] Device:

[1912] The terminal plays back the audio data received from the server and provides the user with the translation results by voice.

[1913] Specific examples

[1914] User:

[1915] A Japanese speaking user wants to ask for directions in the United States.

[1916] Device:

[1917] The microphone captures the user's question, "Where is the nearest station?" and sends the voice data to the server.

[1918] server:

[1919] The server receives the voice data, converts it into text, and translates it. The text "Where is the nearest station?" is converted into voice data and sent to the device.

[1920] Device:

[1921] The device will play the translated audio and the user can ask a local person, "Where is the nearest station?"

[1922] ---

[1923] Emotion Engine Processing Flow

[1924] Emotion recognition and guidance adjustment

[1925] Step 1:

[1926] User:

[1927] The user wears the device and displays changes in stress and emotions during normal use.

[1928] Step 2:

[1929] Device:

[1930] The device's built-in emotion engine uses a camera and microphone to monitor the user's facial expressions and tone of voice in real time.

[1931] Step 3:

[1932] Device:

[1933] The emotion engine analyzes the user's emotional state and detects, for example, stress, joy, and anger.

[1934] Step 4:

[1935] Device:

[1936] The emotion engine generates data based on the sensed emotional state and sends it to an internal processor.

[1937] Step 5:

[1938] Terminal (device) or server:

[1939] The system adjusts the way translation results are presented based on the user's emotional state. For example, if the user is feeling stressed, the system will soften the tone of the voice guidance.

[1940] Step 6:

[1941] Device:

[1942] The adjusted voice guidance is played back to guide the user in a manner that is adaptive to the user.

[1943] Specific examples

[1944] User:

[1945] A user is frustrated trying to find a restaurant in a foreign city.

[1946] Device:

[1947] The emotion engine analyzes the user's facial expressions and tone of voice to recognize stress.

[1948] Server or device:

[1949] The tone of the voice guidance is softened based on the emotional state.

[1950] Device:

[1951] In reality, a gentle voice announces, "This is a cafe. Our recommended dish is latte."

[1952] The above is the specific processing flow of this system. By explaining the operation at each step in detail, we will realize a system that is convenient and highly applicable for users.

[1953] Example 2

[1954] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1955] In recent years, communication between people who speak different languages ​​has become increasingly important. Language barriers are a major obstacle, especially when traveling or traveling to foreign countries for business. However, existing translation devices and applications lack the ability to translate visual and audio information in real time and provide guidance. Furthermore, they are unable to provide guidance that takes into account the user's emotional state, which can cause stress and anxiety. In response to this, the present invention aims to provide user-friendly information by translating the user's visual and audio information in real time and adjusting the guidance method to adapt to the user's emotional state.

[1956] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1957] In this invention, the server includes means for analyzing text information and translating it into a specified language, means for converting it into voice data and sending it to the terminal, and means for analyzing the user's emotional state and adjusting the guidance method, which not only translates the user's visual and voice information in real time and provides guidance, but also makes it possible to provide appropriate guidance according to the user's emotional state.

[1958] A "visual assistance device" is a device worn by a user and has the function of acquiring visual information about the surroundings and providing the information to the user.

[1959] A "remote computing device" is a computer device that is remotely located over the Internet and performs data analysis and calculations.

[1960] "Means for capturing" refers to the function of acquiring visual and audio information using a camera, microphone, etc.

[1961] "Means for extracting characters" refers to a function for detecting and extracting character information from visual information.

[1962] "Means for receiving a translation result" refers to a function by which the terminal receives a translation result sent from a remote computing device.

[1963] "Means for generating voice data" refers to a function for converting text information into voice.

[1964] "Means for recognizing emotional state" refers to a function that uses a camera or microphone to analyze the user's facial expressions and tone of voice to identify the user's emotions.

[1965] "Means for adjusting the guidance method" refers to a function that changes the content and tone of the guidance depending on the user's emotional state.

[1966] "Means for detecting spoken language" refers to the function of detecting surrounding conversations and sounds and acquiring them as audio data.

[1967] The "setting means that can be changed based on a voice profile" refers to a function that allows the tone and style of voice guidance to be changed based on voice characteristics selected by the user.

[1968] MODE FOR CARRYING OUT THE INVENTION

[1969] This invention is a system that uses a user-worn visual aid device to automatically translate different languages ​​and provide audio guidance. The visual aid device has a built-in camera, microphone, speaker, communication module, processor, and emotion engine, and complements its processing capabilities by communicating with a remote computing device (server).

[1970] System configuration

[1971] The visual assistance device is a glasses-type device worn by the user that contains a camera, microphone, speaker, communication module, processor, and emotion engine. The device communicates with a remote server to perform advanced processing.

[1972] System Operation

[1973] Visual translation and audio guidance

[1974] The device periodically captures images of its surroundings using a camera and extracts text information from them. The text information is extracted using OCR (Optical Character Recognition) technology. The text information extracted from the captured images is sent to a server via a wireless communication module.

[1975] After receiving the text information, the server translates it into the specified language using a translation model (e.g., a general translation API). The translated text is then converted into voice data using a speech synthesis tool (e.g., a general speech synthesis API). The generated voice data is then sent to the terminal (device) via a communications network.

[1976] The terminal (device) plays the received voice data through a speaker and provides the user with audible guidance on the translation results.

[1977] As a concrete example, consider a situation where a Japanese-speaking user is visiting France and looking for a restaurant. The user wears a device and sees a sign that says "restaurant." The device's camera captures the sign, and a character recognition process extracts the word "restaurant." This data is sent to a server, which generates the English translation of "restaurant," "Restaurant," and then generates a voice prompt saying, "This place is a restaurant. Recommended dishes are pasta and steak." The terminal (device) plays the received voice data and provides the user with instructions.

[1978] Spoken language translation and audio guidance

[1979] The device captures the surrounding sound using a built-in microphone, and the captured sound data is sent directly to the server.

[1980] After receiving the voice data, the server converts it into text using voice recognition technology (e.g., a general voice recognition API), then translates the converted text into the specified language using a translation model, and generates voice data again using a voice synthesis tool. The generated voice data is then sent to the device.

[1981] The terminal (device) plays back the received audio data again and presents the translation results to the user.

[1982] As a concrete example, consider a situation where a Japanese-speaking user in an American market asks, "Where is the nearest station?" The device's microphone captures the question and sends the data to a server. The server translates the voice data into "Where is the nearest station?" and generates voice data that is sent to the terminal (device). The user can then use this voice data to ask local people.

[1983] Emotion Engine Functions

[1984] The device uses a built-in camera and microphone to analyze the user's facial expressions and tone of voice, recognizing their emotional state in real time.

[1985] The server or terminal (device) adjusts the guidance method based on the user's emotional state. For example, if the user is feeling stressed, the tone of the voice guidance is set to a gentler tone. This allows for flexible guidance tailored to the user's emotional state.

[1986] For example, consider a case where a user is searching for a restaurant in a foreign country but is feeling stressed. The device can sense the user's stress and guide them in a calmer tone, saying, "This is a cafe. The recommended dish is a latte."

[1987] This invention enables the translation of a user's visual and audio information in real time, and also provides appropriate guidance according to the user's emotional state. Based on the above explanation, we have shown what hardware and software are used to specifically implement this system.

[1988] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1989] Visual translation and audio guidance

[1990] Step 1:

[1991] User: Wears the device and acquires visual information. Specifically, the user looks at an object with text information, such as a sign or menu.

[1992] Input: Images containing text information, such as signs and menus

[1993] Output: Visual capture request

[1994] Step 2:

[1995] Device: Uses the camera to capture images of the surroundings.

[1996] Input: Captcha request

[1997] Output: Captured image data

[1998] Step 3:

[1999] Terminal (device): The processor analyzes the captured image data and extracts text information using OCR technology.

[2000] Input: Captured image data

[2001] Output: Extracted text information

[2002] Step 4:

[2003] Terminal (device): The extracted text information is sent to the server using a communication module.

[2004] Input: Extracted text information

[2005] Output: Request to send text information

[2006] Step 5:

[2007] Server: Analyzes the received text information and translates it into the specified language using a generative AI model.

[2008] Input: Extracted text information

[2009] Output: The translated text

[2010] Step 6:

[2011] Server: Converts the translated text into audio data using a speech synthesis tool, possibly using a voice profile selected by the user.

[2012] Input: Translated text

[2013] Output: Audio data

[2014] Step 7:

[2015] Server: Generates a request to send the generated audio data to the device.

[2016] Input: Audio data

[2017] Output: Audio data transmission request

[2018] Step 8:

[2019] Device: Plays the received audio data through a speaker.

[2020] Input: Audio data

[2021] Output: Play voice prompts

[2022] Spoken language translation and audio guidance

[2023] Step 1:

[2024] User: Start a conversation with a foreigner, specifically asking a question such as "Where is the nearest station?"

[2025] Input: User conversation

[2026] Output: Audio data capture request

[2027] Step 2:

[2028] Device: Uses a microphone to capture surrounding sounds.

[2029] Input: Audio data capture request

[2030] Output: Captured audio data

[2031] Step 3:

[2032] Device: Sends captured audio data to the server using a communication module.

[2033] Input: Captured audio data

[2034] output: Request to send audio data

[2035] Step 4:

[2036] Server: Analyzes the received voice data and converts it into text using voice recognition technology.

[2037] Input: Captured audio data

[2038] Output: Speech-to-text data

[2039] Step 5:

[2040] Server: Translates the converted text into the specified language using a generative AI model.

[2041] Input: Speech-to-text data

[2042] Output: The translated text

[2043] Step 6:

[2044] Server: The translated text is converted into audio data using a speech synthesis tool.

[2045] Input: Translated text

[2046] Output: Audio data

[2047] Step 7:

[2048] Server: Generates a request to send the generated audio data to the device.

[2049] Input: Audio data

[2050] Output: Audio data transmission request

[2051] Step 8:

[2052] Device: Plays the received audio data through a speaker.

[2053] Input: Audio data

[2054] Output: Play voice prompts

[2055] Emotion Engine Functions

[2056] Step 1:

[2057] Device: Analyzes the user's facial expressions and tone of voice using the built-in camera and microphone.

[2058] Input: User's facial expression, tone of voice

[2059] Output: Emotional state analysis request

[2060] Step 2:

[2061] Terminal (device) or server: Recognizes emotional state based on analyzed data.

[2062] Input: Emotional state analysis request

[2063] Output: Emotional state data

[2064] Step 3:

[2065] Terminal (device) or server: Adjust the tone and content of the announcement depending on the user's emotional state.

[2066] Input: Emotional state data

[2067] Output: Adjusted voice prompts

[2068] Step 4:

[2069] Device: Plays tailored voice prompts through the speaker.

[2070] Input: Adjusted voice prompts

[2071] Output: Play a gentle voice prompt

[2072] (Application example 2)

[2073] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[2074] Conventional visual assistance devices are unable to adapt flexibly to the user's emotional state when receiving translation results. Furthermore, language barriers can hinder the smooth shopping experience in physical stores. There is a need for devices that can provide appropriate support even when users feel stressed or confused.

[2075] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for capturing an image including text, means for extracting text from the captured image, means for transmitting the extracted text to a remote computing device for translation, means for receiving a translation result from the remote computing device, means for providing audible guidance to the user of the received translation result, means for recognizing the user's emotional state and adjusting the guidance method, means for adjusting the guidance method of the translated audio based on the emotion recognition, and setting means for changing the audio guidance of the translation result to the voice of a celebrity selected by the user. This enables flexible response according to the user's emotions, making it possible to have a smooth and stress-free shopping experience in a physical store.

[2076] A "visual assistance device" is a device worn by a user to acquire visual information and assist the user's vision.

[2077] "Images containing text" refers to images with text on them, such as signs or product labels.

[2078] "Capturing means" refers to the method of acquiring an image using a camera or other photographic device.

[2079] The "means for extracting characters" is a method for analyzing character information from a captured image and extracting the characters.

[2080] A "remote computing device" is a server or computer that receives data over a network and performs complex calculations or processing.

[2081] The "means for receiving the translation result" is a method for obtaining the translated data from the remote computing device.

[2082] "Means for providing audio guidance to the user" refers to a method for converting the received translation results into audio and conveying it to the user.

[2083] The "means for recognizing the user's emotional state and adjusting the guidance method" is a method for analyzing the user's facial expression and tone of voice and changing the guidance method according to the user's emotions.

[2084] The "means for detecting spoken words and capturing audio data" is a method for detecting spoken words using an audio input device such as a microphone and acquiring the audio as data.

[2085] The "means for adjusting the guidance method of the translated voice based on emotion recognition" is a method for analyzing the user's emotions through emotion recognition and changing the tone and content of the translated voice accordingly.

[2086] The "setting means for changing to a celebrity's voice" is a method for changing the settings so that guidance is given using the voice of a celebrity selected by the user.

[2087] This invention is a system that supports the shopping experience in a physical store as a visual aid device worn by a user. The system is composed of smart glasses worn by a user and a remote computing device (server).

[2088] Hardware and Software

[2089] Hardware: The smart glasses contain a camera, microphone, speaker, processor, and communication module. The camera captures images containing text information, and the microphone captures audio data. The speaker is used to provide the user with audible translation results.

[2090] software:

[2091] Image processing library (OpenCV): Used to extract characters from captured images.

[2092] Speech Recognition Library (Google Speech-to-Text API): Used to convert captured voice data into text.

[2093] Translation API (Google Translate API): Used to translate the extracted text into a specified language.

[2094] Speech synthesis library (Google Text-to-Speech API): Used to convert the translation text into audio data.

[2095] Emotion Recognition Engine (Emotion API): Used to recognize the user's emotional state and adjust guidance methods.

[2096] System Operation

[2097] User interaction: A user puts on the smart glasses and moves around a physical store. When the user wants to check the price label or description of a product, the camera captures the text.

[2098] Terminal (smart glasses) processing:

[2099] 1. The processor analyzes the image captured by the camera and extracts text information.

[2100] 2. The extracted text information is sent to the server.

[2101] Server Action:

[2102] 1. The server uses a translation API to translate the text received into the specified language.

[2103] 2. Convert the translated text into audio data using a speech synthesis library.

[2104] 3. Use an emotion recognition engine to analyze the user's emotional state and adjust the tone of the voice prompts if the user is feeling stressed.

[2105] Terminal (smart glasses) processing:

[2106] 1. Receives voice data sent from the server and provides voice guidance through the speaker.

[2107] Specific examples

[2108] Consider shopping in a brick-and-mortar store. For example, a foreign tourist wearing smart glasses while shopping in a Japanese department store. The camera captures information about the shelves and store directions, and the system translates the text in real time and provides voice guidance. If the user becomes lost or stressed, the tone of the guidance changes to a gentler tone, providing a sense of security.

[2109] Example prompt sentence:

[2110] A user wears smart glasses and looks at shelves in a department store. The system captures information about the shelves, translates it into the user's language in real time, and provides voice guidance. If the user feels stressed, the tone of the guidance changes to a gentler tone to provide reassurance. Please explain how this system is implemented and provide specific examples of API usage.

[2111] In this way, the visual aid device adapts to the user's emotional state and helps facilitate a smooth shopping experience in a physical store.

[2112] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2113] Step 1:

[2114] A user wears smart glasses and moves around a physical store. When the user wants to check the price label or description of a product, the camera captures the text information. Here, the input is the captured image, and the output is an image file containing the text information.

[2115] Step 2:

[2116] The processor in the device (smart glasses) analyzes the captured image and extracts text information. Specifically, it uses an image processing library (OpenCV) to identify the text and then uses optical character recognition (OCR) technology to convert the text into text data. Here, the input is an image file and the output is the extracted text information (text data).

[2117] Step 3:

[2118] The terminal (smart glasses) transmits the extracted text information to a remote computing device (server). A communication module is used to send the text data to the server. The input of this step is the text data, and the output is the data transmitted to the server.

[2119] Step 4:

[2120] The server translates the received text into the specified language using a translation API (Google Translate API). Here, the server first inputs the text data, processes it through the translation API, and obtains the translation result. The input is the received text information, and the output is the translated text data.

[2121] Step 5:

[2122] The server converts the translated text data into audio data using a speech synthesis library (Google Text-to-Speech API). Specifically, the translated text is input into the speech synthesis API to generate an audio file. The input is the translated text data, and the output is an audio file.

[2123] Step 6:

[2124] Here, the server uses an emotion recognition engine (Emotion API) to analyze the user's emotional state. The camera and microphone capture the user's facial expressions and tone of voice, and send the data to the server. The input is facial expression data and voice data, and the output is the user's emotional state (e.g., stress, joy, surprise).

[2125] Step 7:

[2126] The server adjusts the tone of the voice guidance based on the emotion recognition results. For example, if the user is feeling stressed, the tone of the voice guidance is set to be gentler. The input is the emotional state and the audio file, and the output is the adjusted audio file.

[2127] Step 8:

[2128] The server sends the adjusted audio file to the device (smart glasses). The input of this step is the adjusted audio file, and the output is the data sent to the device.

[2129] Step 9:

[2130] The speaker on the device (smart glasses) plays the received audio data and provides the translation results to the user by voice. The input is the received audio data, and the output is the audio guidance the user hears.

[2131] In this way, visual aid devices can help users have a smoother shopping experience in brick-and-mortar stores. Furthermore, by providing flexible guidance that responds to the user's emotional state, it can provide more user-friendly information.

[2132] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[2133] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2134] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[2135] [Fourth embodiment]

[2136] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[2137] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[2138] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[2139] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[2140] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[2141] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[2142] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[2143] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[2144] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[2145] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[2146] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[2147] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[2148] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2149] This invention is a system that automatically translates different languages ​​and provides audio guidance as a visual aid device worn by the user. This system can acquire visual information, translate it in real time, and provide audio guidance. It also has the function of translating spoken language and providing audio guidance.

[2150] This visual assistance device (hereinafter referred to as the device) is in the form of glasses worn by the user and contains a camera, microphone, speaker, communication module, and processor. The device supplements its processing capabilities by communicating with a remote computing device (hereinafter referred to as the server).

[2151] 1. Visual translation and audio guidance

[2152] Camera and character extraction

[2153] User:

[2154] The device recognizes objects with text information, such as signs and menus, in the user's field of vision.

[2155] Device:

[2156] The device's camera periodically captures images of its surroundings, which are then analyzed by a processor to extract textual information.

[2157] Sending translation requests and receiving translation results

[2158] Device:

[2159] The extracted text information is sent to the server via the communication module, and the data sent also includes information about the language to be translated.

[2160] server:

[2161] The server analyzes the received text and translates it into the specified language. After generating the translation results, it constructs them as text for voice guidance. It then converts the text into audio data using the voice of a celebrity selected by the user and sends it to the device.

[2162] Device:

[2163] The received audio data is played back and the translation results are presented to the user.

[2164] Specific examples

[2165] User:

[2166] Suppose a Japanese-speaking user is visiting France and looking for a restaurant.

[2167] Device:

[2168] A camera captures a sign that says "Restaurant," and a character recognition process extracts the word "Restaurant." The extracted character data is sent to a server.

[2169] server:

[2170] The server receives the character "restaurant" and translates it into English as "Restaurant." Based on the translation, it generates a narration saying "This place is a restaurant. Recommended dishes are pasta and steak." This is converted into audio data and sent to the device.

[2171] Device:

[2172] The voice guidance played back informs the user that "This is a restaurant. Recommended dishes are pasta and steak."

[2173] 2. Spoken language translation and audio guidance

[2174] Microphone and voice data capture

[2175] User:

[2176] Imagine a situation where a user is communicating with local people in a foreign market.

[2177] Device:

[2178] The device's microphone constantly monitors the surrounding sound and captures audio data, such as conversations, which are then stored as audio data.

[2179] Sending translation requests and receiving translation results

[2180] Device:

[2181] The stored voice data is transmitted to a server via a communication module.

[2182] server:

[2183] The server receives the voice data, converts it into text using speech recognition technology, then translates the text into the specified language and generates the translation as voice data, which is then sent to the device.

[2184] Device:

[2185] The received audio data is played back and the translation results are presented to the user.

[2186] Specific examples

[2187] User:

[2188] Consider a scenario in which a Japanese-speaking user in an American market asks "Where is the nearest station?" to get directions.

[2189] Device:

[2190] A microphone captures the user's question and transmits the audio data to the server.

[2191] server:

[2192] The server receives the speech data "Where is the nearest station?" and translates it into "Where is the nearest station?". It generates the translated speech data and sends it to the device.

[2193] Device:

[2194] The audio guide played allows the user to ask local people, "Where is the nearest station?"

[2195] In this way, the present invention allows users to translate visual and audio information in real time and provide appropriate guidance, thereby enabling smooth communication even in environments where different languages ​​are spoken. Furthermore, by providing audio guidance in the voice of a celebrity selected by the user, a more friendly service can be provided.

[2196] The processing flow will be explained below.

[2197] Processing flow for translation of visual information and voice guidance

[2198] Visual translation

[2199] Step 1:

[2200] User:

[2201] A user wearing the device looks at visual information, such as a sign or a menu.

[2202] Step 2:

[2203] Device:

[2204] The device's camera periodically captures images of its surroundings, which are then sent to a processor.

[2205] Step 3:

[2206] Device:

[2207] The processor uses image analysis algorithms to extract text information from the captured image, for example, using optical character recognition (OCR) technology to convert the characters into text data.

[2208] Step 4:

[2209] Device:

[2210] A request is generated to transmit the extracted character data and the linguistic information to be translated to a remote computing device (server).

[2211] Step 5:

[2212] Device:

[2213] The generated request is sent to the server through a communication module.

[2214] Step 6:

[2215] server:

[2216] The server receives the request, parses the text, and feeds the text into the corresponding translation model to perform translation into the specified language.

[2217] Step 7:

[2218] server:

[2219] Once the translation results are generated, a narration is created based on the translated text. For example, it generates a sentence like, "This is a cafe. The recommended menu item is latte."

[2220] Step 8:

[2221] server:

[2222] To convert the narration text into audio data, text-to-speech synthesis (TTS) is performed using the voice of a celebrity selected by the user.

[2223] Step 9:

[2224] server:

[2225] The generated voice data is transmitted to the terminal.

[2226] Step 10:

[2227] Device:

[2228] The terminal plays back the audio data received from the server and provides the user with the translation results by voice.

[2229] Specific examples

[2230] User:

[2231] A Japanese speaking user sees a sign in French.

[2232] Device:

[2233] The camera captures the sign and uses OCR technology to extract the word "restaurant."

[2234] server:

[2235] The server receives "restaurant" and translates it as "restaurant." It then generates a narration saying, "This is a restaurant. The recommended dish is pasta," converts it into voice data, and sends it to the device.

[2236] Device:

[2237] The device plays the audio data and informs the user of the translation results. The voice will say in Japanese, "This is a restaurant. The recommended dish is pasta."

[2238] ---

[2239] Spoken language translation and voice guidance processing flow

[2240] Audio translation

[2241] Step 1:

[2242] User:

[2243] The user puts on the device and attempts to converse with local people.

[2244] Step 2:

[2245] Device:

[2246] The device's microphone constantly monitors the surrounding sounds and captures what is being said.

[2247] Step 3:

[2248] Device:

[2249] The captured audio data is sent to a processor.

[2250] Step 4:

[2251] Device:

[2252] A processor temporarily stores the audio data and generates a request to send it to a server via a communication module.

[2253] Step 5:

[2254] Device:

[2255] The generated request is sent to the server through a communication module.

[2256] Step 6:

[2257] server:

[2258] The server receives the voice data and converts it into text using speech recognition technology, for example, "Where is the nearest station?"

[2259] Step 7:

[2260] server:

[2261] Translate the recognized text into the specified language, for example, "Where is the nearest station?" into English.

[2262] Step 8:

[2263] server:

[2264] To generate audio data based on the translated text, TTS is used to create audio data in a voice selected by the user.

[2265] Step 9:

[2266] server:

[2267] The generated voice data is transmitted to the terminal.

[2268] Step 10:

[2269] Device:

[2270] The terminal plays back the audio data received from the server and provides the user with the translation results by voice.

[2271] Specific examples

[2272] User:

[2273] A Japanese speaking user wants to ask for directions in the United States.

[2274] Device:

[2275] The microphone captures the user's question, "Where is the nearest station?" and sends the voice data to the server.

[2276] server:

[2277] The server receives the voice data, converts it into text, and translates it. The text "Where is the nearest station?" is converted into voice data and sent to the device.

[2278] Device:

[2279] The device will play the translated audio and the user can ask a local person, "Where is the nearest station?"

[2280] The above is the specific processing flow of this system, and the operation at each step is explained in detail.

[2281] Example 1

[2282] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2283] In modern society, people who speak different languages ​​coexist, and smooth communication is essential. However, language barriers make it difficult to understand signs and menus, or to converse smoothly with local people, both in daily life and while traveling. To solve these problems, a visual aid device is needed that can translate in real time and provide audio guidance.

[2284] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[2285] In this invention, the server includes means for capturing an image containing text, means for extracting text from the captured image, means for translating the extracted text, means for receiving a translation result from a remote computing device, means for providing audible guidance to a user about the received translation result, and means for converting the received translation result into audio data in a specified voice, thereby enabling real-time translation of text and conversations displayed in different languages ​​and audio guidance.

[2286] 1. "Visual assistance device" means a device worn by a user that has the function of acquiring, processing, and providing visual information.

[2287] 2. "Capturing means" is a collective term for the hardware and software that a device uses to capture images and audio.

[2288] 3. "Text extraction means" means a process or device that has the ability to identify text information from a captured image and extract it as data.

[2289] 4. "Remote Computing Device" means a device that provides remotely located computing resources for processing data transmitted from a Device.

[2290] 5. "Translation means" means software and hardware functions for converting written or spoken information in one language into another language.

[2291] 6. "Means for receiving translation results" means a device that has the function of receiving translation data sent from a remote computing device on the device side.

[2292] 7. "Means for providing audio guidance to the user" refers to the device's function of converting the received translation results into audio format and notifying the user by audio.

[2293] 8. "Means for converting into voice data using a specified voice" means a device that has the function of converting the translation results into voice data using a specific voice that the user has specified in advance.

[2294] 9. "Audio data capture means" means a device or process that captures a user's speech or surrounding sounds and stores them as digital audio data.

[2295] 10. "Means for receiving translated voice data" means a device that has the capability to receive translated voice data sent from a remote computing device.

[2296] 11. "Means for providing information to the user" refers to functions or devices for providing acquired or received information to the user in an easy-to-understand manner.

[2297] 12. "Configuration means" means the interface and software functionality that allows a user to customize the operation and functionality of a device to suit the user's preferences.

[2298] This invention provides a multilingual visual aid device, enabling users to translate visual and audio information in real time and provide audio guidance in different language environments. This invention achieves this by using a capture device, a remote server, speech synthesis technology, and a communication module.

[2299] Hardware and software configurations used

[2300] 1. Wearable visual aids:

[2301] Camera: Captures visual information about the surroundings.

[2302] Microphone: Captures surrounding sounds.

[2303] Speaker: A means of providing the translation results to the user aloud.

[2304] Communications Module: A means for sending and receiving data to and from a remote computing device.

[2305] Processor: Processes image and audio data.

[2306] 2. Remote Computing Device (Server):

[2307] Character recognition software: Extracts text information from images (e.g., Tesseract OCR).

[2308] Speech recognition technology: Converts captured voice data into text (e.g., Google Speech-to-Text).

[2309] Translation engine: Translates text from one language to another (e.g., Google Translate API, DeepL).

[2310] Speech synthesis engine: Converts text to speech (e.g., Amazon Polly, Google Text-to-Speech).

[2311] System Operation

[2312] Visual translation and audio guidance

[2313] A visual assistance device worn by a user captures visual information from the surroundings using a camera. For example, the camera may capture a photo of a sign for a French restaurant. The processor of the terminal (device) analyzes the captured image and uses character recognition software to extract the text information "restaurant." This text information is then sent to a server via a communications module. The server analyzes the received text data, translates it into the specified language (e.g., from French to English), and outputs the translation as "Restaurant." Based on the translation, the server then generates a narration such as "This place is a restaurant. Recommended dishes are pasta and steak." This is converted into audio data using a speech synthesis engine and sent to the device. The terminal (device) receives the audio data and plays it through a speaker, so the user is informed that "This place is a restaurant. Recommended dishes are pasta and steak."

[2314] Spoken language translation and audio guidance

[2315] Consider a scenario in which a user in a foreign market asks, "Where is the nearest station?" A microphone installed on the terminal (device) captures surrounding sounds and sends the voice data to a server. The server receives the voice data and converts it into text using voice recognition technology. It then translates the text into a specified language (e.g., Japanese to English), generates voice data based on the translation results, and sends it to the device. The terminal (device) receives the voice data and asks the user, "Where is the nearest station?"

[2316] Examples of concrete examples and prompts

[2317] Specific examples

[2318] A Japanese-speaking user visits France and a camera captures a sign that says "Restaurant." The text data "Restaurant" is sent to the server. The server translates "Restaurant" into "Restaurant," generates a narration saying, "This place is a restaurant. Recommended dishes are pasta and steak.", and sends this voice data to the device. By listening to this voice guidance, the user learns, "This place is a restaurant. Recommended dishes are pasta and steak."

[2319] Prompt Sentence Examples

[2320] "Translate the text written on the sign. Generate a prompt to translate from French to English: 'Extract text from captured image and translate it into the specified language.'"

[2321] "Generate a prompt that translates the Japanese question 'Where is the nearest station?' into English: 'Use speech recognition technology to convert captured speech into text, and then translate that text into the specified language.'"

[2322] In this way, the present invention is a system that can translate visual and audio information in real time in a multilingual environment and provide audio guidance to the user.

[2323] The flow of the identification process in the first embodiment will be described with reference to FIG.

[2324] Visual translation and audio guidance

[2325] Step 1: Capture visual information

[2326] Device:

[2327] While a user is wearing a visual aid device, the device's camera periodically captures visual information about the surroundings.

[2328] Input: Objects in the user's field of view (e.g., signs, menus)

[2329] Data processing: The image capture process is activated and the camera acquires the image data.

[2330] Output: Acquired raw image data

[2331] What happens: The device's camera takes a picture every second.

[2332] Step 2: Extracting characters

[2333] Device:

[2334] The processor analyzes the captured image data in real time and extracts text information using optical character recognition software (OCR).

[2335] Input: Captured image data

[2336] Data processing: Extracting text data from images using OCR algorithms (e.g., Tesseract).

[2337] Output: Extracted text data (e.g., "restaurant")

[2338] What it does: The processor recognizes characters in the image and generates data in text format.

[2339] Step 3: Submit a translation request

[2340] Device:

[2341] The extracted text data and the language information to be translated are sent to the server via the communication module.

[2342] Input: Extracted text data, language information to be translated (e.g. French to English)

[2343] Data processing: Constructing data packets and sending them to the server using a communication protocol (e.g., HTTP).

[2344] Output: Data packet sent to the server

[2345] What happens: The device sends the text "restaurant" and a translation request from French to English.

[2346] Step 4: Translation and Narration Generation

[2347] server:

[2348] The received text data is analyzed and translated into the specified language, and a narration is generated based on the translation results.

[2349] Input: Text data (e.g., "restaurant"), language information to be translated

[2350] Data processing: Translate to "Restaurant" using a translation engine (e.g., Google Translate API). Then generate a voiceover sentence: "This place is a restaurant. Recommended dishes are pasta and steak."

[2351] Output: Translated text and narration

[2352] Specific operation: The server translates "restaurant" to "Restaurant" and constructs a guide text.

[2353] Step 5: Generate audio data

[2354] server:

[2355] The narration sentences are converted into voice data using a voice synthesis engine.

[2356] Input: Narration (e.g., "This place is a restaurant. Recommended dishes are pasta and steak.")

[2357] Data processing: Generate voice data using a speech synthesis engine (e.g., Amazon Polly).

[2358] Output: Audio data

[2359] Specific operation: The server generates audio based on the narration text.

[2360] Step 6: Sending voice data and guidance

[2361] Device:

[2362] The audio data is received from the server and played back to the user through the speaker.

[2363] Input: Audio data received from the server

[2364] Data processing: Decodes the received data and plays it on the speaker.

[2365] Output: A voice prompt to the user

[2366] What happens: The speaker plays a voice saying, "This place is a restaurant. Recommended dishes are pasta and steak."

[2367] ---

[2368] Spoken language translation and audio guidance

[2369] Step 1: Capture audio data

[2370] Device:

[2371] The microphone captures the surrounding sounds.

[2372] Input: User speech and surrounding audio (e.g., "Where is the nearest station?")

[2373] Data processing: Recording audio data and converting it into digital format.

[2374] Output: Captured audio data

[2375] What it does: A microphone records the user's question.

[2376] Step 2: Sending audio data

[2377] Device:

[2378] The captured audio data is transmitted to a server via a communication module.

[2379] Input: Audio data

[2380] Data processing: Constructing data packets and sending them to the server using a communication protocol.

[2381] Output: Audio data sent to the server

[2382] Specific operation: The device sends voice data saying "Where is the nearest station?"

[2383] Step 3: Transcribe and translate the audio data

[2384] server:

[2385] The received voice data is converted into text using voice recognition technology and then translated.

[2386] Input: Speech data (e.g., "Where is the nearest station?")

[2387] Data processing: Use speech recognition technology (e.g., Google Speech-to-Text) to convert the speech to text, then use a translation engine to translate it into "Where is the nearest station?"

[2388] Output: Translated text data

[2389] What happens: The server converts the voice data into text and translates it into English.

[2390] Step 4: Generate translated audio data

[2391] server:

[2392] The translated text data is converted into voice data using a voice synthesis engine.

[2393] Input: Translated text data (e.g., "Where is the nearest station?")

[2394] Data processing: Generate voice data using a speech synthesis engine.

[2395] Output: Audio data

[2396] Specific operation: The server generates audio data based on the translation results.

[2397] Step 5: Voice prompts

[2398] Device:

[2399] The generated audio data is received and played back to the user through a speaker.

[2400] Input: Audio data

[2401] Data processing: Decodes the received data and plays it on the speaker.

[2402] Output: A voice prompt to the user

[2403] Specific action: The speaker will play a voice saying "Where is the nearest station?"

[2404] In this way, the system captures visual and audio information in real time, translates it, and provides audio guidance, thereby providing smooth communication that transcends language barriers.

[2405] Examples of concrete examples and prompts

[2406] Specific examples

[2407] A Japanese-speaking user uses a camera to capture a sign that says "restaurant" in a French restaurant, and the text data "restaurant" is sent to the server. The server translates "restaurant" into "Restaurant" and generates a narration saying, "This place is a restaurant. Recommended dishes are pasta and steak." This is sent as audio data to the device. After listening to the audio guidance, the user learns, "This place is a restaurant. Recommended dishes are pasta and steak."

[2408] A Japanese-speaking user in an American market asks, "Where is the nearest station?" The device captures the speech and sends it to the server. The server translates "Where is the nearest station?" to "Where is the nearest station?" and sends it as voice data to the device. The device plays the speech and tells the user, "Where is the nearest station?"

[2409] Prompt Sentence Examples

[2410] "Translate the text written on the sign. Generate a prompt to translate from French to English: 'Extract text from captured image and translate it into the specified language.'"

[2411] "Generate a prompt that translates the Japanese question 'Where is the nearest station?' into English: 'Use speech recognition technology to convert captured speech into text, and then translate that text into the specified language.'"

[2412] This system is designed to support smooth information acquisition and communication in a multilingual environment.

[2413] (Application example 1)

[2414] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2415] There is a problem in that users who speak different languages ​​cannot communicate smoothly in autonomous vehicles (e.g., self-driving vehicles). It is also difficult to understand visual information such as signs and road signs written in different languages, which can make travel in a self-driving vehicle unstable. Furthermore, local guidance and responses to questions cannot be provided immediately, which causes great stress for tourists and foreign users.

[2416] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[2417] In this invention, the server includes a processing device for translating visual information, a processing device for translating audio information, and a processing device for converting received translation results into audio data. This enables real-time translation of visual and audio information in different languages ​​within an autonomous mobile body. The translation results are provided to the user via audio guidance via the visual assistance system, allowing the user to quickly understand route guidance to their destination and emergency information. In addition, user-selected audio guidance is provided in a user-friendly format.

[2418] A "visual aid system" is a device worn by a user that captures visual information and provides the information to the user visually or audibly.

[2419] "Visual information" refers to information such as characters and figures that are included in the user's field of vision.

[2420] A "remote processor" is a remotely located computing device that processes data sent from a visual aid system and returns the results.

[2421] An "autonomous mobile object" is a vehicle or device that can move autonomously and reach its destination using artificial intelligence and sensor technology, such as a self-driving vehicle.

[2422] "Audio data" refers to audio signals captured by an audio input device such as a microphone and converted into digital data.

[2423] "Translation result" is text or audio data that has been converted into a different language by a remote processor.

[2424] "Famous person's voice" is voice data of a commonly recognized person selected by the user.

[2425] "Guidance means" refers to a mechanism or device that allows the visual aid system to provide information to the user.

[2426] A "processing device" is a computer or hardware for processing and analyzing data.

[2427] "Friendly voice guidance" refers to voice guidance provided in a format that is easy for the user to hear and understand.

[2428] This invention utilizes a visual aid system worn by a user to translate visual and audio information in different languages ​​in real time within an autonomous vehicle, and provides audio guidance. The system of the present invention includes a visual aid system, a remote processor, and means used within the autonomous vehicle.

[2429] Hardware and Software

[2430] Visual aid systems:

[2431] The smart glasses are worn by the user and contain a camera, microphone, speaker, communication module, and processor. The system is responsible for capturing visual and audio information and transmitting that data to a remote processor.

[2432] Remote Processor:

[2433] Extract text from visual information (e.g., signs and road signs) and translate it into different languages. Also convert audio data to text, translate it, and convert it back to audio. Use software such as Google Translate API, SpeechRecognition, and gTTS (Google Text-to-Speech).

[2434] Autonomous mobile:

[2435] This includes autonomous vehicles, which are equipped with smart glasses worn by the user and have a communication module that connects to a remote processor via a network.

[2436] Data Processing Overview

[2437] Visual information processing:

[2438] 1. Server: Receives the surrounding images sent from the smart glasses and extracts the text data using character recognition technology.

[2439] 2. Server: Use the Google Translate API to translate the extracted text data into the specified language.

[2440] 3. Server: Uses gTTS to convert the translation results into audio data and sends it to the smart glasses.

[2441] 4. Terminal: Plays back the received audio data and displays the translation results to the user.

[2442] Processing of audio information:

[2443] 1. Device: The microphone in the smart glasses captures the surrounding sounds and sends the audio data to the server.

[2444] 2. Server: Use SpeechRecognition to convert the received audio data into text data.

[2445] 3. Server: Uses Google Translate API and gTTS to translate text data into the specified language and convert it back to audio data.

[2446] 4. Terminal: Plays back the received audio data and displays the translation results to the user.

[2447] Specific examples

[2448] Example 1:

[2449] Consider a situation where a traveler is using an autonomous taxi. To understand a tourist information board they come across on the way to their destination, the smart glasses capture the text information on the sign and send it to a server. The server returns the translated results as audio data and provides audio guidance to the user through the smart glasses. This allows the traveler to instantly understand information about their destination and tourist spots.

[2450] Example 2:

[2451] Imagine a situation where a traveler communicates with local people in an autonomous taxi. When the user asks a question using the microphone in the smart glasses, the voice data is sent to the server and translated. The translated result is played back as voice data, allowing the traveler to communicate smoothly with local people.

[2452] Example prompt for a generative AI model:

[2453] Input: "Describe how you would implement a system that uses smart glasses in a self-driving taxi to translate destination signs and provide voice directions."

[2454] Output: "This implementation utilizes the camera, microphone, and speaker built into the smart glasses. The camera captures visual information and extracts text data using character recognition technology. The extracted text is translated in real time and audio guidance is provided to the user through the smart glasses' speaker. The microphone also captures audio, which is converted into text using speech recognition technology. This is then translated and audio guidance is provided through the speaker."

[2455] In this way, the system of the present invention enables users who speak different languages ​​to understand visual and audio information in different languages ​​within an autonomous mobile body and to communicate smoothly.

[2456] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[2457] Step 1:

[2458] Terminal: The camera in the smart glasses captures visual information from the surroundings. The captured image data is sent to the processor. Specifically, the camera periodically acquires images, which are then processed by the internal processor. The input is the captured visual information, and the output is the image data passed to the processor.

[2459] Step 2:

[2460] Terminal: The processor extracts character information from the captured image data. It uses OCR (Optical Character Recognition) technology to convert the characters in the image into text data. The input is the image data from step 1, and the output is the extracted character data.

[2461] Step 3:

[2462] Terminal: The extracted character data is sent to the remote processor via the communication module, where the sent data is passed to the server through the network. The input is the character data, and the output is the data sent to the remote processor.

[2463] Step 4:

[2464] Server: Translates received text data into a specified language. Uses the Google Translate API to convert text data into a different language. The input is the text data to be translated, and the output is the translation result.

[2465] Step 5:

[2466] Server: Converts the translation results into audio data. Using gTTS (Google Text-to-Speech), converts text data into an audio file. The input is the translated text data, and the output is audio data.

[2467] Step 6:

[2468] Server: Transmits the converted voice data to the smart glasses. The voice data is sent to the terminal via the communication module. The input is the voice data, and the output is the voice data sent to the smart glasses.

[2469] Step 7:

[2470] Terminal: The received audio data is played on the smart glasses' speaker, providing audio guidance to the user. Specifically, the audio data is decoded and played as audio through the speaker. The input is audio data, and the output is audio guidance provided to the user.

[2471] Step 8:

[2472] Terminal: The microphone in the smart glasses captures the surrounding sound. The captured sound data is sent to the processor. The input is the captured sound data, and the output is the sound data passed to the processor.

[2473] Step 9:

[2474] Terminal: Transmits the captured audio data to a remote processor via a communication module, where the transmitted data is passed to the server via the network. The input is the audio data, and the output is the data transmitted to the remote processor.

[2475] Step 10:

[2476] Server: Converts received voice data to text. Using SpeechRecognition, converts voice data to text data. The input is voice data, and the output is text data.

[2477] Step 11:

[2478] Server: Translates the converted text data into the specified language. The input is the text data, and the output is the translated text data.

[2479] Step 12:

[2480] Server: The translated text data is converted back into audio data using gTTS. The input is the translated text data, and the output is the audio data.

[2481] Step 13:

[2482] Server: Transmits the converted voice data to the smart glasses. The voice data is sent to the terminal via the communication module. The input is the voice data, and the output is the voice data sent to the smart glasses.

[2483] Step 14:

[2484] Terminal: The received audio data is played on the smart glasses' speaker, providing audio guidance to the user. Specifically, the audio data is decoded and played as audio through the speaker. The input is audio data, and the output is audio guidance provided to the user.

[2485] Through these processing steps, the visual aid system translates visual and audio information in different languages ​​in real time within the autonomous vehicle and provides audio guidance to the user.

[2486] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2487] This invention is a system that automatically translates different languages ​​and provides audio guidance as a visual aid device worn by the user. This system can acquire visual and audio information, translate it in real time, and provide audio guidance. It also has the ability to recognize the user's emotions and adjust the guidance method based on the user's emotional state.

[2488] System configuration

[2489] The visual assistance device (hereafter referred to as the device) is a pair of glasses worn by the user, and contains a camera, microphone, speaker, communication module, processor, and emotion engine. The device supplements its processing capabilities by communicating with a remote computing device (hereafter referred to as the server).

[2490] Visual translation and audio guidance

[2491] Camera and character extraction

[2492] User:

[2493] The user wearing the device can recognize objects that contain text information, such as signs and menus.

[2494] Device:

[2495] The device's camera periodically captures images of its surroundings, which are then analyzed by a processor to extract textual information.

[2496] server:

[2497] The server analyzes the received text and translates it into the specified language. After generating the translation results, it constructs them as text for voice guidance. It then converts the text into audio data using the voice of a celebrity selected by the user and sends it to the device.

[2498] Device:

[2499] The received audio data is played back and the translation results are presented to the user.

[2500] Specific examples

[2501] User:

[2502] Suppose a Japanese-speaking user is visiting France and looking for a restaurant.

[2503] Device:

[2504] A camera captures a sign that says "Restaurant," and a character recognition process extracts the word "Restaurant." The extracted character data is sent to a server.

[2505] server:

[2506] The server receives "restaurant" and translates it into English as "Restaurant." It then generates a narration saying, "This place is a restaurant. Recommended dishes are pasta and steak.", converts it into voice data, and sends it to the device.

[2507] Device:

[2508] The voice guidance played back informs the user that "This is a restaurant. Recommended dishes are pasta and steak."

[2509] Spoken language translation and audio guidance

[2510] Microphone and voice data capture

[2511] User:

[2512] Imagine a situation where a user is communicating with local people in a foreign market.

[2513] Device:

[2514] The device's microphone constantly monitors the surrounding sound and captures audio data, such as conversations, which are then stored as audio data.

[2515] server:

[2516] The server receives the voice data, converts it into text using speech recognition technology, then translates the text into the specified language and generates the translation as voice data, which is then sent to the device.

[2517] Device:

[2518] The received audio data is played back and the translation results are presented to the user.

[2519] Specific examples

[2520] User:

[2521] Consider a scenario where a Japanese-speaking user is asking for directions in an American market.

[2522] Device:

[2523] The microphone captures the user's question, "Where is the nearest station?" and sends the voice data to the server.

[2524] server:

[2525] The server receives the voice data, converts it into text, and translates it. The text "Where is the nearest station?" is generated, converted into voice data, and sent to the device.

[2526] Device:

[2527] The audio guide played allows the user to ask local people, "Where is the nearest station?"

[2528] Emotion Engine Functions

[2529] Emotion recognition and guidance adjustment

[2530] Device:

[2531] It has a built-in emotion engine that analyzes the user's facial expressions and tone of voice captured by the camera and microphone, recognizing the user's emotional state in real time and adjusting the way translation results are presented based on that information.

[2532] Server or device:

[2533] The emotion engine analyzes the user's current emotional state (e.g., happy, sad, anger, surprise) and adjusts the tone and content of the announcements accordingly. For example, if the user is feeling stressed, a calming voice announcement setting will be automatically selected.

[2534] Specific examples

[2535] User:

[2536] A user may feel stressed while trying to find a restaurant in a foreign country.

[2537] Device:

[2538] The emotion engine detects the user's facial expressions and tone of voice and recognizes when the user is feeling stressed.

[2539] Server or device:

[2540] The tone of the translated voice guidance is softened to reflect the user's emotional state. For example, a message such as "This is a cafe. The recommended dish is latte" will be played in a softer voice.

[2541] This invention not only translates the user's visual and audio information in real time and provides audio guidance based on that information, but also grasps the user's emotional state and adjusts the guidance method accordingly, thereby realizing more user-friendly information provision.

[2542] The processing flow will be explained below.

[2543] Processing flow for translation of visual information and voice guidance

[2544] Visual translation

[2545] Step 1:

[2546] User:

[2547] A user wearing the device looks at visual information, such as a sign or a menu.

[2548] Step 2:

[2549] Device:

[2550] The device's camera periodically captures images of its surroundings, which are then sent to a processor.

[2551] Step 3:

[2552] Device:

[2553] The processor uses image analysis algorithms to extract text information from the captured image, for example, using optical character recognition (OCR) technology to convert the characters into text data.

[2554] Step 4:

[2555] Device:

[2556] A request is generated to transmit the extracted character data and the linguistic information to be translated to a remote computing device (server).

[2557] Step 5:

[2558] Device:

[2559] The generated request is sent to the server through a communication module.

[2560] Step 6:

[2561] server:

[2562] The server receives the request, parses the text, and feeds the text into the corresponding translation model to perform translation into the specified language.

[2563] Step 7:

[2564] server:

[2565] Once the translation results are generated, a narration is created based on the translated text. For example, it generates a sentence like, "This is a cafe. The recommended menu item is latte."

[2566] Step 8:

[2567] server:

[2568] To convert the narration text into audio data, text-to-speech synthesis (TTS) is performed using the voice of a celebrity selected by the user.

[2569] Step 9:

[2570] server:

[2571] The generated voice data is transmitted to the terminal.

[2572] Step 10:

[2573] Device:

[2574] The terminal plays back the audio data received from the server and provides the user with the translation results by voice.

[2575] Specific examples

[2576] User:

[2577] A Japanese speaking user sees a sign in French.

[2578] Device:

[2579] The camera captures the sign and uses OCR technology to extract the word "restaurant."

[2580] server:

[2581] The server receives "restaurant" and translates it as "restaurant." It then generates a narration saying, "This is a restaurant. The recommended dish is pasta," converts it into voice data, and sends it to the device.

[2582] Device:

[2583] The terminal plays back the audio data and provides the user with a voice guide saying, "This is a restaurant. The recommended dish is pasta."

[2584] ---

[2585] Spoken language translation and voice guidance processing flow

[2586] Audio translation

[2587] Step 1:

[2588] User:

[2589] The user attempts to converse with local people through a device worn by the user.

[2590] Step 2:

[2591] Device:

[2592] The device's microphone constantly monitors the surrounding sounds and captures what is being said.

[2593] Step 3:

[2594] Device:

[2595] The captured audio data is sent to a processor.

[2596] Step 4:

[2597] Device:

[2598] A processor temporarily stores the audio data and generates a request to send it to a server via a communication module.

[2599] Step 5:

[2600] Device:

[2601] The generated request is sent to the server through a communication module.

[2602] Step 6:

[2603] server:

[2604] The server receives the voice data and converts it into text using speech recognition technology, for example, "Where is the nearest station?"

[2605] Step 7:

[2606] server:

[2607] Translate the recognized text into the specified language, for example, "Where is the nearest station?" into English.

[2608] Step 8:

[2609] server:

[2610] To generate audio data based on the translated text, TTS is used to create audio data in a voice selected by the user.

[2611] Step 9:

[2612] server:

[2613] The generated voice data is transmitted to the terminal.

[2614] Step 10:

[2615] Device:

[2616] The terminal plays back the audio data received from the server and provides the user with the translation results by voice.

[2617] Specific examples

[2618] User:

[2619] A Japanese speaking user wants to ask for directions in the United States.

[2620] Device:

[2621] The microphone captures the user's question, "Where is the nearest station?" and sends the voice data to the server.

[2622] server:

[2623] The server receives the voice data, converts it into text, and translates it. The text "Where is the nearest station?" is converted into voice data and sent to the device.

[2624] Device:

[2625] The device will play the translated audio and the user can ask a local person, "Where is the nearest station?"

[2626] ---

[2627] Emotion Engine Processing Flow

[2628] Emotion recognition and guidance adjustment

[2629] Step 1:

[2630] User:

[2631] The user wears the device and displays changes in stress and emotions during normal use.

[2632] Step 2:

[2633] Device:

[2634] The device's built-in emotion engine uses a camera and microphone to monitor the user's facial expressions and tone of voice in real time.

[2635] Step 3:

[2636] Device:

[2637] The emotion engine analyzes the user's emotional state and detects, for example, stress, joy, and anger.

[2638] Step 4:

[2639] Device:

[2640] The emotion engine generates data based on the sensed emotional state and sends it to an internal processor.

[2641] Step 5:

[2642] Terminal (device) or server:

[2643] The system adjusts the way translation results are presented based on the user's emotional state. For example, if the user is feeling stressed, the system will soften the tone of the voice guidance.

[2644] Step 6:

[2645] Device:

[2646] The adjusted voice guidance is played back to guide the user in a manner that is adaptive to the user.

[2647] Specific examples

[2648] User:

[2649] A user is frustrated trying to find a restaurant in a foreign city.

[2650] Device:

[2651] The emotion engine analyzes the user's facial expressions and tone of voice to recognize stress.

[2652] Server or device:

[2653] The tone of the voice guidance is softened based on the emotional state.

[2654] Device:

[2655] In reality, a gentle voice announces, "This is a cafe. Our recommended dish is latte."

[2656] The above is the specific processing flow of this system. By explaining the operation at each step in detail, we will realize a system that is convenient and highly applicable for users.

[2657] Example 2

[2658] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2659] In recent years, communication between people who speak different languages ​​has become increasingly important. Language barriers are a major obstacle, especially when traveling or traveling to foreign countries for business. However, existing translation devices and applications lack the ability to translate visual and audio information in real time and provide guidance. Furthermore, they are unable to provide guidance that takes into account the user's emotional state, which can cause stress and anxiety. In response to this, the present invention aims to provide user-friendly information by translating the user's visual and audio information in real time and adjusting the guidance method to adapt to the user's emotional state.

[2660] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[2661] In this invention, the server includes means for analyzing text information and translating it into a specified language, means for converting it into voice data and sending it to the terminal, and means for analyzing the user's emotional state and adjusting the guidance method, which not only translates the user's visual and voice information in real time and provides guidance, but also makes it possible to provide appropriate guidance according to the user's emotional state.

[2662] A "visual assistance device" is a device worn by a user and has the function of acq...

Claims

1. A visual aid device worn by a user, comprising: means for capturing an image including text; means for extracting text from the captured image; means for transmitting the extracted characters to a remote computing device for translation; means for receiving a translation result from a remote computing device; means for providing audible guidance to the user on the received translation results; A system including:

2. means for detecting speech and capturing audio data; means for transmitting the captured audio data to a remote computing device; means for receiving the translated speech data from the remote computing device; means for presenting the received translated speech to a user; The system of claim 1 further comprising:

3. 2. The system according to claim 1, further comprising a setting means for changing the voice guidance of the translation result to the voice of a celebrity selected by the user.

Images