system

Abstract

A system is provided.SOLUTION: A system comprising: means for capturing visual images of a user, including a camera for monitoring the user's field of view; means for capturing text data from captured video frames, including an optical character recognition (OCR) engine for tracking the user's gaze and identifying the most visually focused text areas; means for transmitting the extracted text data to a translation server and making a translation request; and means for overlaying the translation results received from the translation server on the user's field of view.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Cross-border communication is becoming increasingly important in modern society. However, significant barriers still exist for communication between individuals who speak different languages. This challenge is even more pronounced in certain situations, such as during international travel, business meetings, exhibitions, and intercultural education. Furthermore, in today's world where real-time information access is required, using translation applications requires user interaction, making hands-free information acquisition difficult. This creates a need for effective solutions to remove language barriers and ensure smooth communication. [Means for solving the problem]

[0005] To solve the above-mentioned problems, the present invention provides a system equipped with a camera that monitors and captures a user's field of view. An optical character recognition (OCR) engine is used to extract text data from the captured video frames, and an eye-tracking sensor is used to identify the user's point of gaze. The identified text data is sent to a translation server, which issues a translation request. The translation results received from the translation server are then overlaid on the user's field of view. Furthermore, a speech synthesis engine is provided that plays back the translation results aloud, providing both visual and auditory support to the user, as needed. In this way, hands-free real-time text translation and display eliminates language barriers and enables smooth communication.

[0006] A "user" is an individual who wears the smart glasses and utilizes the functionality to translate text that comes into their field of vision.

[0007] "Field of view image" refers to an image of the scenery or objects that the user visually perceives, captured by the camera in the smart glasses.

[0008] A "camera" is a photographic device mounted on smart glasses to capture the user's field of view.

[0009] A "video frame" is a unit of still image captured by a camera at regular intervals.

[0010] An "Optical Character Recognition (OCR) engine" is a technology that recognizes text from captured video frames and extracts it as digital data.

[0011] "Text data" is character string information extracted from video frames by the OCR engine.

[0012] An "eye tracking sensor" is a device that detects the position of a user's gaze and identifies the point of gaze.

[0013] The "point of gaze" is a specific position where the user focuses their gaze most.

[0014] A "translation server" is a computer system that receives text data and translates that text into other languages.

[0015] A "translation request" is communication data that requests a translation server to translate specific text data.

[0016] "Translation result" refers to text data converted into another language that is generated and transmitted by a translation server.

[0017] "Overlay display" is a method of displaying new information in the user's field of view by superimposing it on the original field of view image.

[0018] A "speech synthesis engine" is a technology that converts text data into voice data and conveys it to the user as auditory information. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7]FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0020] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0021] First, the terms used in the following description will be explained.

[0022] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0023] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0024] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0025] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0026] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0027] [First embodiment]

[0028] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0029] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0030] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0031] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0032] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0033] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0034] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0035] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0036] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0037] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0038] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0039] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0040] The present invention relates to a smart glasses system worn by a user, which includes a camera, an optical character recognition (OCR) engine, an eye-tracking sensor, a translation server, a translation result overlay display function, and a speech synthesis engine.

[0041] Program processing explanation

[0042] Text Capture

[0043] The device constantly monitors the user's field of vision, capturing the scenery, signs, menus, and other things the user sees through the smart glasses. For example, if the user spots a sign written in a foreign language, the camera will capture that object.

[0044] OCR (Optical Character Recognition)

[0045] The image frames captured by the device are fed into an OCR engine to extract text data, identifying text segments written by the user in different languages ​​and extracting them as digital character data.

[0046] Eye tracking

[0047] The device uses an eye-tracking sensor to identify the area where the user is focusing their gaze, allowing it to prioritize the text the user is most focused on, even in situations where there are multiple text areas.

[0048] Submit a translation request

[0049] The device sends the extracted text data to a translation server. The translation request includes the original text data as well as the user's language preference. For example, a request is made to translate a Japanese sign into English.

[0050] Translation Processing

[0051] The server processes the received text data using a translation engine. The original text is converted into high-quality translated text based on the target language set by the user. The server then processes the text and returns the translation results to the device.

[0052] Sending translation results

[0053] The translation results generated by the server are sent to the terminal via the network. Data compression and caching techniques may be used to ensure user visibility.

[0054] Displaying translation results

[0055] The device receives the translation result and displays it as an overlay in the user's field of vision. The translation result is displayed visually on top of the original text, allowing the user to instantly understand the meaning. For example, the sign "Exit" that the user is looking at is displayed as the translation result "Exit."

[0056] Audio support (optional)

[0057] The device can optionally use a speech synthesis engine to play back the translation results aloud, allowing users to hear the information even when they cannot see.

[0058] Specific examples

[0059] As an example, consider the situation where a user is looking at a menu in a French restaurant.

[0060] 1. A user looks at a restaurant menu in French.

[0061] 2. The device uses its camera to capture a video frame of the menu.

[0062] 3. The device uses OCR to extract the text "Poulet rôti".

[0063] 4. The device uses an eye-tracking sensor to determine the user's point of gaze and focuses on "Poulet rôti."

[0064] 5. The device sends the text data "Poulet rôti" to the translation server.

[0065] 6. The server receives the request and uses an AI translation engine to translate "Poulet rôti" into English "Roast chicken."

[0066] 7. The server sends the translation results to the device.

[0067] 8. The device overlays the translation result "Roast chicken" on the user's field of view.

[0068] 9. If necessary, the device plays "Roast chicken" aloud, providing both visual and auditory confirmation of the information to the user.

[0069] In this way, the present invention provides a system that helps users understand text in different languages ​​in real time, thereby removing language barriers and enabling smooth communication and information access.

[0070] The processing flow will be explained below.

[0071] Step 1:

[0072] A user puts on the smart glasses and looks at text that comes into their field of vision. For example, the user sees a sign written in French.

[0073] Step 2:

[0074] The device activates its built-in camera and captures a video of the user's field of view, capturing a video frame of the sign the user is looking at.

[0075] Step 3:

[0076] The device inputs the captured video frame into the OCR engine, which extracts text information from the video. The text "Sortie" on the sign the user is looking at is extracted.

[0077] Step 4:

[0078] The device uses an eye-tracking sensor to track the user's gaze and identify the text area that receives the most attention. It detects that the gaze is focused on the "Sortie" section.

[0079] Step 5:

[0080] The device sends the extracted text data "Sortie" to the translation server, along with the user's language setting (e.g., French to English).

[0081] Step 6:

[0082] The server receives the translation request and passes the text data "Sortie" to the AI ​​translation engine, which translates "Sortie" as "Exit."

[0083] Step 7:

[0084] The server sends the generated translation result "Exit" to the terminal.

[0085] Step 8:

[0086] The translation result "Exit" received by the device is overlaid on the user's field of view, with "Exit" displayed over the original "Sortie" part.

[0087] Step 9:

[0088] (Optional) The device uses a speech synthesis engine to play the translation result "Exit" aloud, allowing the user to confirm the translation result both visually and audibly.

[0089] In this way, users can understand texts in different languages ​​in real time.

[0090] Example 1

[0091] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0092] In conventional smart glasses systems, the technology to translate and display visual information in real time is not fully developed. As a result, users often have difficulty understanding visual information when confronted with displays in a different language in a foreign country. Furthermore, existing systems do not identify gaze points using eye tracking or integrate visual information translation and voice support, making it difficult to achieve real-time information access and smooth communication.

[0093] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0094] In this invention, the server includes an image capture device for monitoring a user's field of view, a means for capturing an object visually perceived by the user, an optical character recognition (OCR) engine for extracting character data from the captured image frame, an eye-tracking sensor for tracking the user's gaze and identifying the most gazed character area, a means for transmitting the extracted character data to a translation processing device and making a translation request, and a means for overlaying the translation result received from the translation processing device on the user's field of view. This makes it easier for users to understand text written in different languages ​​in real time, and the integration of visual and audio information enables smooth communication and information access.

[0095] An "image capture device" is a device that monitors the user's field of view and captures the object that the user visually perceives.

[0096] An "Optical Character Recognition (OCR) engine" is an engine for extracting character data from captured video frames.

[0097] The "eye tracking sensor" is a sensor that tracks the user's gaze and identifies the text area that is most closely watched.

[0098] The "translation processing device" is a device that transmits extracted character data to a translation server and makes a translation request.

[0099] The "means for overlaying and displaying in the user's field of view" refers to a means for displaying the translation result received from the translation server in an overlaid manner in the user's field of view.

[0100] "User language setting information" is setting information related to the language used by the user and the language to be translated.

[0101] A "voice synthesizer" is a device that converts character data into voice data and plays it back aloud.

[0102] The present invention relates to a system that translates text in a different language that a user visually perceives in real time and overlays it on the user's field of vision. The system includes an image capture device, an optical character recognition (OCR) engine, an eye-tracking sensor, a translation processor, and a speech synthesizer.

[0103] Image acquisition device

[0104] The device constantly monitors the user's field of view. A high-resolution camera captures the scenery and text the user sees through the smart glasses in real time. This camera captures video frames and stores them in a buffer.

[0105] Optical Character Recognition (OCR) engine

[0106] The video frames captured by the device are input into an OCR engine (e.g., Tesseract OCR) to extract text data. The OCR engine analyzes the text data from the image data and extracts it as digital text data.

[0107] Eye tracking sensor

[0108] The device uses an eye-tracking sensor to track the user's gaze, which identifies the specific text area the user is looking at and prioritizes the most-looked-at text even when multiple pieces of text are present.

[0109] Translation Processing Device

[0110] The device sends the extracted text data and the user's language preference information to a translation processing device. The translation request is sent over the network, and the translation processing device receives the request. This device inputs the received text data into a translation engine (e.g., Google Translate API) and translates it into the specified language.

[0111] Overlay of translation results

[0112] The server sends the translation results to the terminal via the network. The terminal displays the received translation results as an overlay in the user's field of view. The overlay display allows the user to check the translation results while viewing the original text.

[0113] speech synthesizer

[0114] If necessary, the device will launch a speech synthesis engine (e.g., Google Text-to-Speech API) and play back the generated translation aloud. This speech playback feature allows users to hear information aloud even in situations where visual confirmation is difficult.

[0115] Specific examples

[0116] As an example, consider the situation where a user is looking at a menu in a French restaurant.

[0117] 1. A user looks at a restaurant menu in French.

[0118] 2. The device uses its camera to capture a video frame of the menu.

[0119] 3. The device uses OCR to extract the text "Poulet rôti".

[0120] 4. The device uses an eye-tracking sensor to determine the user's point of gaze and focuses on "Poulet rôti."

[0121] 5. The terminal sends the text data "Poulet rôti" to the translation processing device.

[0122] 6. The server receives the request and uses an AI translation engine to translate "Poulet rôti" into English "Roast chicken."

[0123] 7. The server sends the translation results to the device.

[0124] 8. The device overlays the translation result "Roast chicken" on the user's field of view.

[0125] 9. If necessary, the device plays "Roast chicken" aloud, providing both visual and auditory confirmation of the information to the user.

[0126] Example prompt sentence:

[0127] "The user looks at the menu," "The device activates the camera," "The OCR engine extracts the text," "The eye-tracking sensor identifies the gaze point," "The translation processor performs the translation," "An overlay is displayed," and "A speech synthesis device plays back the audio."

[0128] The present invention is a system that allows users to easily understand text written in different languages ​​in real time, and enables smooth communication and information access by integrating visual and audio information.

[0129] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0130] Step 1:

[0131] The user wears the smart glasses and looks around at what comes into view. The user moves to capture the text they see. The input is the user's line of sight. The output is the image of the view that comes into the camera's capture range.

[0132] Specific behavior:

[0133] The object the user wants to see is placed in the field of view of the smart glasses.

[0134] The device's built-in camera activates and captures the user's field of view.

[0135] Step 2:

[0136] The device captures the field of view with a high-resolution camera. The input is the field of view captured by the camera. The output is a video frame stored in a buffer.

[0137] Specific behavior:

[0138] The device continuously captures video at a specific frame rate (e.g., 30 frames per second).

[0139] The device stores the captured video frames in a buffer.

[0140] Step 3:

[0141] The device inputs video frames into an optical character recognition (OCR) engine to extract character data. The input is the video frames stored in a buffer. The output is the character data in digital form.

[0142] Specific behavior:

[0143] The device launches an OCR engine (e.g., Tesseract OCR) and provides the captured video frames as input data.

[0144] The OCR engine analyzes the text data from the image data, extracts the digital character data, and returns it to the terminal.

[0145] Step 4:

[0146] The device uses an eye-tracking sensor to track the user's gaze. The input is the user's gaze. The output is information about the specific text region being gazed at.

[0147] Specific behavior:

[0148] The device's built-in eye tracking sensor monitors the user's eye movements in real time.

[0149] The device analyzes the gaze data and identifies the text area the user is gazing at.

[0150] Step 5:

[0151] The terminal transmits the extracted text data and the user's language preference information to the translation processing device. The input is the digital text data and the language preference information. The output is the generation and transmission of a translation request.

[0152] Specific behavior:

[0153] The terminal sends data to the translation processing device in the form of an HTTP request (e.g., REST API).

[0154] The terminal generates a translation request including digital text data and the user's language setting information, and transmits the request over the network.

[0155] Step 6:

[0156] The server receives a translation request and translates the text data using a translation engine. The input is the data in the translation request. The output is the translated text.

[0157] Specific behavior:

[0158] The server parses the request data and passes it to a translation engine (e.g., Google Translate API).

[0159] The translation engine translates the original text into the specified language, generates the translation result, and returns it to the server.

[0160] Step 7:

[0161] The server generates translation results and sends them to the terminal. The input is the translated text data. The output is the translation results sent to the terminal.

[0162] Specific behavior:

[0163] The server packages the translation results in JSON format and sends them to the device.

[0164] The server uses data compression techniques (e.g., GZIP) to minimize the data transmitted.

[0165] Step 8:

[0166] The translation results received by the terminal are overlaid on the user's field of view. The input is the translation results received from the server. The output is the translation results displayed in the user's field of view.

[0167] Specific behavior:

[0168] After the device receives the translation result, it will be displayed by properly arranging it in the original text area.

[0169] The terminal dynamically renders the translated text displayed in the user's field of view.

[0170] Step 9:

[0171] The device activates a speech synthesis engine as needed and plays back the translation result aloud. The input is the text data of the translation result. The output is the audio that is played back to the user.

[0172] Specific behavior:

[0173] The device generates voice data using a speech synthesis engine (e.g., Google Text-to-Speech API).

[0174] The device plays the generated audio data from the speaker.

[0175] This series of processes enables users to understand text in different languages ​​in real time, and enables smooth communication and information access through the integration of visual and audio information.

[0176] (Application example 1)

[0177] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0178] Passengers in self-driving vehicles face challenges when traveling to different cultural regions, such as Japan, where rapid and accurate understanding of traffic signs and information boards in different languages ​​is a major challenge. In particular, not being able to understand place names and directions while driving or in a moving vehicle can lead to confusion and misunderstanding. Current technology is limited in its ability to translate text in real time and provide visual and auditory translations, creating a need for a system that can address this issue.

[0179] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0180] In this invention, the server includes a camera for monitoring the user's field of view, a means for capturing objects visually perceived by the user, an optical character recognition (OCR) engine for extracting text information from the captured video frames, an eye-tracking sensor for tracking the user's gaze and identifying the text area most gazed at, a means for sending the extracted text information to a translation service and making a translation request, a means for overlaying the translation result received from the translation service on the user's field of view, a speech synthesis engine for playing back the translation result as voice, and a means for recognizing, translating, and displaying signs in different languages ​​in real time while the user is in a moving vehicle. This enables passengers to quickly understand signs and information boards in different languages ​​both visually and audibly.

[0181] A "camera" is a device used to capture an image of a user's field of view.

[0182] An "Optical Character Recognition (OCR) engine" is software for extracting textual information from captured video frames.

[0183] An "eye tracking sensor" is a device that detects the user's gaze and identifies the area where the gaze is most focused.

[0184] A "translation service" is an online service or server that converts extracted text information into a specified language.

[0185] A "translation request" is a data transmission requesting a translation that includes the user's language setting and extracted text information.

[0186] "Overlay display" is a technology that displays translated text information superimposed on the user's field of vision.

[0187] A "speech synthesis engine" is software that converts text information into speech and provides it to the user audibly.

[0188] "Moving vehicle" means a vehicle, including automated public transport and other means of transportation.

[0189] "Signs" refer to guide signs and warning signs installed on roads and in public places.

[0190] This invention is a system using smart glasses worn by users that allows users to recognize signs and information boards in different languages ​​in real time and provides translation results both visually and audibly, making it easier for users to understand information in cross-cultural environments.

[0191] Hardware and software used

[0192] Smart glasses: Equipped with a camera, eye tracking sensor, and a see-through display.

[0193] Server: An online system for performing translation services.

[0194] Software: OCR engines (e.g., Tesseract OCR), translation APIs (e.g., Google Translate API), and speech synthesis engines.

[0195] Processing flow

[0196] The system performs the following steps:

[0197] Image Capture

[0198] While a user wears smart glasses and moves around in an autonomous vehicle, a camera captures signs and guidance displays that come into view. The camera monitors the user's field of view in real time.

[0199] OCR processing

[0200] The server inputs the captured video frames into an OCR engine to extract the text information. The optical character recognition (OCR) engine converts the text information in the captured frames into digital data with high accuracy.

[0201] Eye tracking

[0202] The server uses an eye-tracking sensor to detect the user's gaze and identify the text area where the gaze is most focused, allowing important information to be processed first even when there are multiple text areas.

[0203] Text Translation

[0204] The server sends the extracted text to a translation service, requesting a translation. The translation service also includes the user's language preference data and translates the text into the specified language, such as Japanese to English.

[0205] Display translation results and play audio

[0206] The user sees the translation results overlaid on the smart glasses' transparent display, and the translation results are also played back as audio by a speech synthesis engine, allowing the user to confirm the information both visually and audibly.

[0207] Specific examples

[0208] For example, consider a case where a user finds a Japanese sign indicating "next station" in a self-driving vehicle.

[0209] 1. The user puts on the smart glasses and brings the sign into view.

[0210] 2. The camera captures video of the sign.

[0211] 3. The server uses an OCR engine to extract the text "Next Station."

[0212] 4. The server uses an eye-tracking sensor to analyze the user's gaze and confirm that their gaze is focused on the "next station."

[0213] 5. The extracted text data is sent to a translation service, and a request is made to translate it into the user's preferred language (e.g., English).

[0214] 6. The translation service translates "next station" as "Next Station" and sends it back to the server.

[0215] 7. The server overlays the translation result, "Next Station," on the transparent display of the smart glasses, and simultaneously plays "Next Station" using the speech synthesis engine.

[0216] This allows the user to immediately understand visually and audibly that "next station" is "Next Station" in English.

[0217] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0218] Step 1:

[0219] The user wears smart glasses and uses a camera to capture signs and guide displays in their field of vision. The device's camera captures the image in the user's field of vision in real time and acquires the image frames. The input is the image frames captured by the camera, and the captured image data is obtained as the output.

[0220] Step 2:

[0221] The server inputs the captured video frame into an OCR engine to extract the text information. The server acquires the digital data of the video frame and uses an OCR engine (e.g., Tesseract OCR) to analyze the text information in the image. The input is the video data, and the output is the recognized text data.

[0222] Step 3:

[0223] The server uses an eye-tracking sensor to detect the user's gaze and identify the text area where the gaze is most focused. The server analyzes the gaze data obtained from the eye-tracking sensor and determines the location where the user is most focused. The input is the gaze data and recognized character data, and the output is the character area corresponding to the gaze point.

[0224] Step 4:

[0225] The server sends the extracted text information to a translation service and makes a translation request. Based on the user's language settings, the server sends a request to a translation API (e.g., Google Translate API) to translate the recognized text information into the target language. The input is the text data corresponding to the gaze point and language setting data, and the output is the completion of sending a translation request.

[0226] Step 5:

[0227] The translation service receives the request and returns the translation result into the specified language to the server. The server receives the translated data returned from the translation API. The input is the translation request, and the translated text data is returned to the server as the output.

[0228] Step 6:

[0229] The server sends the translation results to the smart glasses, which overlay the translation results on the user's field of view. The server then sends instructions to overlay the translated text data on the smart glasses' transparent display. The input is the translated text data, and the output is an overlay display on the user's field of view.

[0230] Step 7:

[0231] The server uses a speech synthesis engine to play back the translation results as audio. The server inputs the translated text data into the speech synthesis engine to generate audio data. The generated audio data is played back through the smart glasses' speakers. The input is the translated text data, and the output is audio that is generated and played back so that the user can hear it.

[0232] This series of processes allows users to understand signs and guidance displays in different languages ​​in real time.

[0233] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0234] The present invention relates to a smart glasses system worn by a user, which includes a camera, an optical character recognition (OCR) engine, an eye tracking sensor, a translation server, a translation result overlay display function, a speech synthesis engine, and an emotion engine.

[0235] Program processing explanation

[0236] Text Capture

[0237] The device constantly monitors the user's field of vision, capturing images of signs, menus, books, and other items the user is looking at through the smart glasses. For example, if the user comes across a menu written in a foreign language, the device will capture a video frame of that menu.

[0238] OCR (Optical Character Recognition)

[0239] The video frame captured by the device is input into the OCR engine to extract text data. The text "Poulet rôti" in the menu the user is viewing is extracted using OCR.

[0240] Eye tracking

[0241] The device uses an eye-tracking sensor to identify the area where the user's gaze is focused, allowing it to prioritize the text the user is most focused on, even when there are multiple text areas.

[0242] Submit a translation request

[0243] The terminal sends the extracted text data to a translation server. The translation request includes the original text data as well as the user's language preference, for example, a translation request from French to English.

[0244] Translation Processing

[0245] The server processes the received text data using an AI translation engine. The original text is converted into high-quality translated text based on the target language set by the user. The server then processes the text and sends the translation results back to the device.

[0246] Sending translation results

[0247] The translation results generated by the server are sent to the terminal via the network. Data compression and caching techniques may be used to ensure user visibility.

[0248] Displaying translation results

[0249] The device receives the translation result and displays it as an overlay in the user's field of view. The translation result is displayed visually on top of the original text, allowing the user to instantly understand the meaning. For example, "Poulet rôti" can be displayed as "Roast chicken."

[0250] Audio Support

[0251] The device can optionally use a speech synthesis engine to play back the translation results aloud, allowing users to hear the information even when they cannot see.

[0252] emotion recognition

[0253] The device uses the emotion engine to recognize emotions from the user's facial expressions, voice tone, heart rate, etc. For example, if the user is feeling stressed or anxious, this information will be detected by the emotion engine.

[0254] Adjusting the display method

[0255] The device will adjust how the translation results are displayed based on the perceived emotion: for example, if the user is feeling anxious, the translation results will be displayed in a larger font size and played aloud.

[0256] Specific examples

[0257] As an example, consider the situation where a user is looking at a menu in a French restaurant.

[0258] 1. A user looks at a restaurant menu in French.

[0259] 2. The device uses its camera to capture a video frame of the menu.

[0260] 3. The device uses OCR to extract the text "Poulet rôti".

[0261] 4. The device uses an eye-tracking sensor to determine the user's point of gaze and focuses on "Poulet rôti."

[0262] 5. The device sends the text data "Poulet rôti" to the translation server.

[0263] 6. The server receives the request and uses an AI translation engine to translate "Poulet rôti" into English "Roast chicken."

[0264] 7. The server sends the translation result "Roast chicken" to the terminal.

[0265] 8. The device overlays the translation result "Roast chicken" on the user's field of view.

[0266] 9. The device uses an emotion engine to recognize the user's emotions, and if they indicate anxiety, it displays the translation results in a larger font size and plays them aloud.

[0267] In this way, the present invention provides a system that helps users understand text in different languages ​​in real time, thereby removing language barriers and enabling smooth communication and information access.

[0268] The processing flow will be explained below.

[0269] Step 1:

[0270] A user puts on the smart glasses and looks at text that comes into their field of view. For example, the user sees a menu written in French.

[0271] Step 2:

[0272] The device activates the built-in camera and captures the user's field of view, capturing a frame of the menu.

[0273] Step 3:

[0274] The device inputs the captured video frame into the OCR engine to extract the text data. The text "Poulet rôti" is extracted by OCR.

[0275] Step 4:

[0276] The device uses an eye-tracking sensor to track the user's gaze and identify the text area that receives the most attention. It detects that the gaze is focused on the word "Poulet rôti."

[0277] Step 5:

[0278] The device sends the extracted text data "Poulet rôti" to the translation server, along with the user's language setting (from French to English).

[0279] Step 6:

[0280] The server receives the translation request and inputs the text data "Poulet rôti" into the AI ​​translation engine, which translates the text as "Roast chicken."

[0281] Step 7:

[0282] The server sends the generated translation result "Roast chicken" to the terminal.

[0283] Step 8:

[0284] The device receives the translation result "Roast chicken" and displays it as an overlay in the user's field of view, overlaying "Poulet rôti" with "Roast chicken."

[0285] Step 9:

[0286] The device uses the emotion engine to recognize emotions from the user's facial expressions, voice tone, heart rate, etc. For example, if the user is feeling anxious, the emotion engine will detect this.

[0287] Step 10:

[0288] The device adjusts how translation results are displayed based on the emotions recognized by the emotion engine: if the user is feeling anxious, the translation results will be displayed in a larger font size and played aloud.

[0289] Step 11:

[0290] (Optional) The device uses a speech synthesis engine to play the translation result "Roast chicken" aloud, allowing the user to confirm the translation both visually and audibly.

[0291] For example, consider a scenario where a user is ordering a menu item at a French restaurant. This process helps users understand text in different languages ​​in real time, providing them with peace of mind in situations where they may feel uneasy.

[0292] Example 2

[0293] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0294] Conventional translation systems require a lot of effort for users to visually understand information, making it difficult to understand text in different languages ​​in real time. Furthermore, because the display does not adjust according to the user's emotional state, they are unable to provide effective support when users are feeling stressed or anxious.

[0295] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a camera for monitoring the user's field of view, a means for capturing an object visually perceived by the user, an optical character recognition engine for extracting character data from the captured video frame, an eye-tracking device for tracking the user's gaze and identifying the most gazed character region, a means for transmitting the extracted character data to a translation device and making a translation request, a means for overlaying the translation result received from the translation device on the user's field of view, and an emotion recognition engine for recognizing the user's emotions and adjusting the display method based on the user's emotional state. This makes it easier to understand texts in different languages ​​in real time and enables effective support according to the user's emotional state.

[0296] "User" refers to a person who uses the system to obtain visual information and needs translation.

[0297] "Field of view image" refers to the image of the scenery or object seen by the user, captured through the camera in the smart glasses.

[0298] "Capture device" refers to a device such as a camera for capturing an image of a user's field of view.

[0299] "Optical character recognition engine" refers to software or hardware for extracting character data from captured video frames.

[0300] "Eye tracking device" refers to a sensor that monitors the movement of a user's eyes and identifies the area of ​​text that is most frequently looked at.

[0301] "Translation device" refers to a server or service for translating extracted character data into a specified target language.

[0302] "Overlay display" refers to a method of displaying the translation results superimposed on the user's field of vision, i.e., displaying them as additional information on the original field of vision image.

[0303] An "emotion recognition engine" is software or hardware for recognizing a user's emotional state, and refers to a device that analyzes emotions from facial expressions, vocal tone, heart rate, etc.

[0304] "Adjusting the display method" refers to the process of changing the display format of the translation result depending on the user's emotional state.

[0305] The present invention relates to a smart glasses system worn by a user, specifically a system that aims to translate and display text in different languages ​​in real time. The main hardware components of this system include a camera, an eye-tracking sensor, and a speech synthesis engine, and the software components include an optical character recognition (OCR) engine, a translation server, and an emotion recognition engine.

[0306] The device constantly monitors the user's field of vision using a camera, capturing signs, menus, books, and other items that come into the user's sight. For example, when a user looks at a restaurant menu, the device captures the menu frame by frame. The camera used here could be a versatile wearable camera.

[0307] The video frames captured by the device are input into an OCR engine to extract text data. Specifically, OCR engines such as Google Cloud Vision and Tesseract can be used. For example, the string "Poulet rôti" written in French can be extracted.

[0308] Next, the device uses an eye-tracking sensor to identify the user's gaze point. A specific example of an eye-tracking device is the Tobii Eyetracker. This allows the device to prioritize the text the user is looking at, even when there are multiple text areas. For example, it can detect when the user is looking at "Poulet rôti."

[0309] The device sends the extracted text data to a translation server. This translation request includes the original text data and the user's language setting (e.g., French to English). The server receives the translation request and processes it using an AI translation engine. For example, you can use the Google Translate API or Microsoft Translator API. The server translates "Poulet rôti" to "Roast chicken."

[0310] The translation results generated by the server are sent to the terminal via the network, and data compression and caching techniques may be used to maintain user visibility.

[0311] The device receives the translation result and displays it as an overlay in the user's field of view. It is displayed over the original text, so the user can visually recognize the translated content. For example, "Roast chicken" is displayed in the user's field of view.

[0312] Furthermore, the device can use a speech synthesis engine such as Amazon Polly or Google Text-to-Speech to play back the translation results as audio, if necessary, to support not only visual translation but also auditory comprehension.

[0313] The device uses an emotion engine to recognize the user's emotional state. For example, it uses technologies such as Affectiva and IBM Watson Emotional Analysis to determine the user's emotional state based on their facial expressions, heart rate, and voice tone. If the user is feeling stressed or anxious, this information will be detected by the emotion engine.

[0314] Finally, the device will adjust the way translation results are displayed based on the recognized emotion. For example, if the user is feeling anxious, the device will display the translation results in a larger font size and play a voice message, making the user feel more comfortable receiving the translation results.

[0315] Specific examples

[0316] Take the specific example of a user looking at a menu at a restaurant in France.

[0317] 1. A user looks at a restaurant menu in French.

[0318] 2. The device uses its camera to capture a video frame of the menu.

[0319] 3. The device uses OCR to extract the text "Poulet rôti".

[0320] 4. The device uses an eye-tracking sensor to determine the user's point of gaze and focuses on "Poulet rôti."

[0321] 5. The device sends the text data "Poulet rôti" to the translation server.

[0322] 6. The server receives the request and uses an AI translation engine to translate "Poulet rôti" into English "Roast chicken."

[0323] 7. The server sends the translation result "Roast chicken" to the terminal.

[0324] 8. The device overlays the translation result "Roast chicken" on the user's field of view.

[0325] 9. The device uses an emotion engine to recognize the user's emotions, and if they indicate anxiety, it displays the translation results in a larger font size and plays them aloud.

[0326] Example prompts for generative AI models

[0327] "Video Capture": The smart glasses' camera captures the video of the user's field of vision. The captured video is analyzed by the OCR engine to extract text data. The eye-tracking sensor identifies the user's point of gaze and sends the key text to the translation server. The server generates the translation results, which the device overlays in the user's field of vision. If necessary, the speech synthesis engine plays the translation results aloud. Additionally, the emotion engine recognizes the user's emotional state and adjusts the display accordingly.

[0328] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0329] Step 1: Capture a visual image

[0330] The device constantly monitors the user's field of view. As input, the camera captures live images of the scenery and objects the user is looking at. This captured image is sent to the next step, where it is processed for text extraction. Specifically, the camera mounted on the smart glasses captures images of what is in front of the user's eyes in real time.

[0331] Step 2: OCR processing

[0332] The device inputs the captured video frames into an OCR engine. The input is the captured video from step 1. The OCR engine (e.g., Google Cloud Vision or Tesseract) extracts text data from this video. Specifically, the OCR engine analyzes the image and converts the characters into digital text. The output of this step is the extracted text data (e.g., "Poulet rôti").

[0333] Step 3: Eye tracking

[0334] The device uses an eye-tracking sensor to identify the user's gaze focus point. The input requires data capturing the user's eye movements in real time. The eye-tracking device (e.g., Tobii Eyetracker) analyzes this data and identifies the text area the user is most focused on. Specifically, the eye-tracking sensor tracks the position and movement of the user's pupils, detecting the gaze direction and focus point. The output of this step is text data with the user's gaze point identified.

[0335] Step 4: Submit a translation request

[0336] The device sends the extracted text data to the translation server. The input includes the text data extracted in step 2 and the user's language preference data (e.g., French to English translation). The request is sent to the translation server, and the text data is ready for translation. Specifically, the device sends the text and language preference over the network to the translation server. The output of this step is the translation request data sent to the translation server.

[0337] Step 5: Translation process

[0338] The server processes the received text data with an AI translation engine. The input includes the text data sent in step 4 and the user's language preference data. The AI ​​translation engine (e.g., Google Translate API or Microsoft Translator API) converts the original text into the target language. Specifically, the AI ​​analyzes the text and generates the optimal translation result. The output of this step is the translated text data (e.g., "Roast chicken").

[0339] Step 6: Send the translation

[0340] The server sends the translation results it has generated to the device. The input includes the translated text data generated in step 5. The server sends this data to the device over the network. Specifically, the server converts the translation results into an appropriate format and uses data compression and caching techniques to transmit them efficiently. The output of this step is the translated text data received by the device.

[0341] Step 7: Overlaying the translation results

[0342] The device overlays the translation result in the user's field of view. The input includes the translated text data received in step 6. Specifically, the device overlays the translated text on the original field of view, allowing the user to visually recognize the translation result. The output of this step is the translated text (e.g., "Roast chicken") overlaid in the user's field of view.

[0343] Step 8: Playback by Text-to-Speech Engine

[0344] The device uses a speech synthesis engine, if necessary, to play back the translation result. The input includes the translation text data received in step 6. The speech synthesis engine (e.g., Amazon Polly or Google Text-to-Speech) converts the text data into speech data and plays it back. Specifically, this process generates speech data from the text and plays it back through a speaker. The output of this step is a speech playback of the translation result.

[0345] Step 9: Emotion Recognition

[0346] The device uses an emotion engine to recognize the user's emotional state. Inputs include data such as the user's facial expressions, heart rate, and voice tone. The emotion recognition engine (e.g., Affectiva or IBM Watson Emotional Analysis) analyzes this data to determine the user's emotional state. Specifically, the process involves analyzing data acquired by sensors to detect emotions such as stress or anxiety. The output of this step is the recognized user's emotional state.

[0347] Step 10: Adjust how the translation results are displayed

[0348] The device adjusts the display of the translation result based on the recognized emotion. The input includes the user's emotional state recognized in step 9 and the translated text data obtained in steps 6 or 7. Specific operations include displaying the translation result in a larger font size and adjusting the volume of the audio playback. The output of this step is a display of the translation result optimized for the user's emotional state.

[0349] (Application example 2)

[0350] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0351] Language barriers still exist in many situations today, posing a major problem when it comes to communication at tourist destinations and brick-and-mortar stores. Travelers and those who are not fluent in foreign languages ​​often have difficulty understanding information on local signs, menus, and guide boards. Furthermore, there is a lack of appropriate and quick ways to obtain information in emergencies and stressful situations.

[0352] The specification process by the specification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes an imaging device for monitoring the user's field of view, a means for capturing an object visually perceived by the user, an optical character recognition (OCR) engine for extracting character data from the captured video frame, an eye-tracking sensor for tracking the user's gaze and identifying the most gazed-at character area, a means for transmitting the extracted character data to a translation server and making a translation request, a means for overlaying the translation result received from the translation server on the user's field of view, a speech synthesis engine for playing the translation result aloud as needed, and an emotion engine for recognizing the user's emotional state and adjusting the display method. This makes it possible to understand information in different languages ​​in real time and provide the information in an optimal format depending on the user's emotional state.

[0353] The "user's visual field image" refers to the image range within the visual field that the user can perceive with normal vision.

[0354] An "imaging device" is an electronic device for capturing images, such as a camera.

[0355] A "capturing means" is a device or technology that records visually perceived objects in digital form.

[0356] An "optical character recognition (OCR) engine" is software for extracting text information from images and videos.

[0357] An "eye tracking sensor" is a sensor for tracking the direction and focus of a user's gaze.

[0358] "Character data" refers to the text information extracted by the OCR engine.

[0359] A "translation server" is a server that translates input character data into another specified language.

[0360] The "means for making a translation request" is a communication means for transmitting the captured character data and a translation request to the server.

[0361] "Means for overlay display" refers to a technique for displaying translation results superimposed within the user's field of vision.

[0362] A "speech synthesis engine" is software that converts text data into speech output.

[0363] An "emotion engine" is software that recognizes a user's emotional state from their facial expressions and biometric information.

[0364] "Tourist attractions and brick-and-mortar stores" refer to places that users actually visit and stores where sales activities are carried out.

[0365] The "function for adjusting the display method" is a technology for changing the display format of presented information depending on the emotional state of the user.

[0366] The present invention provides a system for removing language barriers at tourist spots and brick-and-mortar stores. The system operates based on smart glasses worn by users and includes the following main software and hardware components:

[0367] Specifically, the following hardware and software are used:

[0368] Smart glasses: imaging device, eye tracking sensor, display

[0369] OCR engine: Tesseract OCR

[0370] Translation server: Google Cloud Translation API

[0371] Speech synthesis engine: Amazon Polly

[0372] Emotion engine: Affectiva SDK

[0373] Program processing explanation

[0374] Camera Capture

[0375] When a user looks at tourist information, signs, menus, etc. through smart glasses, the camera built into the glasses captures images within their field of view, enhancing the user's visual experience and providing real-time information.

[0376] OCR processing

[0377] The captured video is analyzed by the Tesseract OCR engine to extract text data from the image, converting the visual information into digital text.

[0378] Eye tracking

[0379] An eye-tracking sensor monitors the user's gaze and identifies the text area the user is most focused on, allowing the user to prioritize the information that is most important to them, even when multiple pieces of text are present.

[0380] Translation Request

[0381] The extracted text data is sent to a translation server, where it is translated into the specified target language in real time using the Google Cloud Translation API. The request parameters include the source language, target language, and the extracted text data.

[0382] Displaying translation results

[0383] The translation results received from the translation server are overlaid on the smart glasses display, allowing users to simultaneously view the original text and the translated text, facilitating quick understanding.

[0384] Audio playback

[0385] If necessary, the translation result is synthesized into speech using Amazon Polly and played back to the user, allowing them to confirm the information audibly as well as visually.

[0386] emotion recognition

[0387] Using the Affectiva SDK, the app analyzes the user's facial expressions and biometrics to recognize their current emotional state. Based on this, the way translation results are displayed will be automatically adjusted. For example, if the user is stressed, the translation will be displayed in a larger font or played aloud.

[0388] Specific examples

[0389] For tourist information:

[0390] "Loads the description of a Universal Studios attraction, translates it from Japanese to English, and displays it on the smart glasses."

[0391] For restaurant menus:

[0392] "Italian restaurant menus can be translated from Italian to Japanese and displayed on smart glasses, with voice guidance available if necessary."

[0393] This allows users to smoothly understand information and enjoy a comfortable experience even in different language environments.

[0394] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0395] Step 1:

[0396] The user wears the smart glasses and can see objects within their field of view at tourist spots or brick-and-mortar stores.

[0397] Input: User's field of view (2D image)

[0398] Output: Visual field video data (image frames)

[0399] How it works: The imaging device built into the smart glasses captures the user's field of view in real time with a camera and saves it as image data.

[0400] Step 2:

[0401] The video frames captured by the device are input into an OCR engine, which extracts character data from the images.

[0402] Input: Video frame (image data)

[0403] Output: Character data (text)

[0404] How it works: The Tesseract OCR engine processes the captured video data, optically recognizing characters in the video and extracting them as text data.

[0405] Step 3:

[0406] The device uses an eye-tracking sensor to identify where the user's gaze is focused.

[0407] Input: Text data, user gaze information

[0408] Output: Character data (text) that the user is gazing at

[0409] Movement: The eye-tracking sensor monitors the user's gaze in real time and identifies areas of text that the user is particularly focused on. The text data in the identified areas is then filtered.

[0410] Step 4:

[0411] The terminal transmits the extracted and identified character data to a translation server and makes a translation request.

[0412] Input: Character data (text) that the user is gazing at, translation request (original text and language setting)

[0413] Output: Translated character data (text)

[0414] How it works: The device sends a translation request to the Google Cloud Translation API and gets the text result translated into the specified target language.

[0415] Step 5:

[0416] The translation results acquired by the server are sent to the terminal and displayed as an overlay in the user's field of view.

[0417] Input: Translated character data (text)

[0418] Output: Translation results displayed in the user's field of view (overlay text)

[0419] How it works: The translation results are received from the translation server and displayed on the smart glasses display, overlaid on the original text.

[0420] Step 6:

[0421] If necessary, the device will play back the translation results aloud using a speech synthesis engine.

[0422] Input: Translated character data (text)

[0423] Output: Audio output (audio of translation results)

[0424] How it works: Amazon Polly is used to convert the translated text into speech and play it through the smart glasses' speakers.

[0425] Step 7:

[0426] The device uses an emotion engine to recognize the user's emotional state and adjust how the translation results are displayed.

[0427] Input: User's facial expression data, biometric information, translation result (text)

[0428] Output: Adjusted display (font size, color, etc.), audio playback

[0429] How it works: The Affectiva SDK analyzes the user's facial expressions and heart rate to identify emotional states such as stress or anxiety. Based on the identified emotional state, the system adjusts the way translation results are displayed and the audio playback. For example, if the user is feeling stressed, the system will display the translation results in a larger font and play the audio to emphasize it.

[0430] This is the specific processing flow of the system, which allows users to smoothly obtain information even in a multilingual environment, providing a more comfortable experience.

[0431] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0432] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0433] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0434] [Second embodiment]

[0435] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0436] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0437] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0438] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0439] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0440] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0441] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0442] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0443] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0444] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0445] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0446] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0447] The present invention relates to a smart glasses system worn by a user, which includes a camera, an optical character recognition (OCR) engine, an eye-tracking sensor, a translation server, a translation result overlay display function, and a speech synthesis engine.

[0448] Program processing explanation

[0449] Text Capture

[0450] The device constantly monitors the user's field of vision, capturing the scenery, signs, menus, and other things the user sees through the smart glasses. For example, if the user spots a sign written in a foreign language, the camera will capture that object.

[0451] OCR (Optical Character Recognition)

[0452] The image frames captured by the device are fed into an OCR engine to extract text data, identifying text segments written by the user in different languages ​​and extracting them as digital character data.

[0453] Eye tracking

[0454] The device uses an eye-tracking sensor to identify the area where the user is focusing their gaze, allowing it to prioritize the text the user is most focused on, even in situations where there are multiple text areas.

[0455] Submit a translation request

[0456] The device sends the extracted text data to a translation server. The translation request includes the original text data as well as the user's language preference. For example, a request is made to translate a Japanese sign into English.

[0457] Translation Processing

[0458] The server processes the received text data using a translation engine. The original text is converted into high-quality translated text based on the target language set by the user. The server then processes the text and returns the translation results to the device.

[0459] Sending translation results

[0460] The translation results generated by the server are sent to the terminal via the network. Data compression and caching techniques may be used to ensure user visibility.

[0461] Displaying translation results

[0462] The device receives the translation result and displays it as an overlay in the user's field of vision. The translation result is displayed visually on top of the original text, allowing the user to instantly understand the meaning. For example, the sign "Exit" that the user is looking at is displayed as the translation result "Exit."

[0463] Audio support (optional)

[0464] The device can optionally use a speech synthesis engine to play back the translation results aloud, allowing users to hear the information even when they cannot see.

[0465] Specific examples

[0466] As an example, consider the situation where a user is looking at a menu in a French restaurant.

[0467] 1. A user looks at a restaurant menu in French.

[0468] 2. The device uses its camera to capture a video frame of the menu.

[0469] 3. The device uses OCR to extract the text "Poulet rôti".

[0470] 4. The device uses an eye-tracking sensor to determine the user's point of gaze and focuses on "Poulet rôti."

[0471] 5. The device sends the text data "Poulet rôti" to the translation server.

[0472] 6. The server receives the request and uses an AI translation engine to translate "Poulet rôti" into English "Roast chicken."

[0473] 7. The server sends the translation results to the device.

[0474] 8. The device overlays the translation result "Roast chicken" on the user's field of view.

[0475] 9. If necessary, the device plays "Roast chicken" aloud, providing both visual and auditory confirmation of the information to the user.

[0476] In this way, the present invention provides a system that helps users understand text in different languages ​​in real time, thereby removing language barriers and enabling smooth communication and information access.

[0477] The processing flow will be explained below.

[0478] Step 1:

[0479] A user puts on the smart glasses and looks at text that comes into their field of vision. For example, the user sees a sign written in French.

[0480] Step 2:

[0481] The device activates its built-in camera and captures a video of the user's field of view, capturing a video frame of the sign the user is looking at.

[0482] Step 3:

[0483] The device inputs the captured video frame into the OCR engine, which extracts text information from the video. The text "Sortie" on the sign the user is looking at is extracted.

[0484] Step 4:

[0485] The device uses an eye-tracking sensor to track the user's gaze and identify the text area that receives the most attention. It detects that the gaze is focused on the "Sortie" section.

[0486] Step 5:

[0487] The device sends the extracted text data "Sortie" to the translation server, along with the user's language setting (e.g., French to English).

[0488] Step 6:

[0489] The server receives the translation request and passes the text data "Sortie" to the AI ​​translation engine, which translates "Sortie" as "Exit."

[0490] Step 7:

[0491] The server sends the generated translation result "Exit" to the terminal.

[0492] Step 8:

[0493] The translation result "Exit" received by the device is overlaid on the user's field of view, with "Exit" displayed over the original "Sortie" part.

[0494] Step 9:

[0495] (Optional) The device uses a speech synthesis engine to play the translation result "Exit" aloud, allowing the user to confirm the translation result both visually and audibly.

[0496] In this way, users can understand texts in different languages ​​in real time.

[0497] Example 1

[0498] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0499] In conventional smart glasses systems, the technology to translate and display visual information in real time is not fully developed. As a result, users often have difficulty understanding visual information when confronted with displays in a different language in a foreign country. Furthermore, existing systems do not identify gaze points using eye tracking or integrate visual information translation and voice support, making it difficult to achieve real-time information access and smooth communication.

[0500] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0501] In this invention, the server includes an image capture device for monitoring a user's field of view, a means for capturing an object visually perceived by the user, an optical character recognition (OCR) engine for extracting character data from the captured image frame, an eye-tracking sensor for tracking the user's gaze and identifying the most gazed character area, a means for transmitting the extracted character data to a translation processing device and making a translation request, and a means for overlaying the translation result received from the translation processing device on the user's field of view. This makes it easier for users to understand text written in different languages ​​in real time, and the integration of visual and audio information enables smooth communication and information access.

[0502] An "image capture device" is a device that monitors the user's field of view and captures the object that the user visually perceives.

[0503] An "Optical Character Recognition (OCR) engine" is an engine for extracting character data from captured video frames.

[0504] The "eye tracking sensor" is a sensor that tracks the user's gaze and identifies the text area that is most closely watched.

[0505] The "translation processing device" is a device that transmits extracted character data to a translation server and makes a translation request.

[0506] The "means for overlaying and displaying in the user's field of view" refers to a means for displaying the translation result received from the translation server in an overlaid manner in the user's field of view.

[0507] "User language setting information" is setting information related to the language used by the user and the language to be translated.

[0508] A "voice synthesizer" is a device that converts character data into voice data and plays it back aloud.

[0509] The present invention relates to a system that translates text in a different language that a user visually perceives in real time and overlays it on the user's field of vision. The system includes an image capture device, an optical character recognition (OCR) engine, an eye-tracking sensor, a translation processor, and a speech synthesizer.

[0510] Image acquisition device

[0511] The device constantly monitors the user's field of view. A high-resolution camera captures the scenery and text the user sees through the smart glasses in real time. This camera captures video frames and stores them in a buffer.

[0512] Optical Character Recognition (OCR) engine

[0513] The video frames captured by the device are input into an OCR engine (e.g., Tesseract OCR) to extract text data. The OCR engine analyzes the text data from the image data and extracts it as digital text data.

[0514] Eye tracking sensor

[0515] The device uses an eye-tracking sensor to track the user's gaze, which identifies the specific text area the user is looking at and prioritizes the most-looked-at text even when multiple pieces of text are present.

[0516] Translation Processing Device

[0517] The device sends the extracted text data and the user's language preference information to a translation processing device. The translation request is sent over the network, and the translation processing device receives the request. This device inputs the received text data into a translation engine (e.g., Google Translate API) and translates it into the specified language.

[0518] Overlay of translation results

[0519] The server sends the translation results to the terminal via the network. The terminal displays the received translation results as an overlay in the user's field of view. The overlay display allows the user to check the translation results while viewing the original text.

[0520] speech synthesizer

[0521] If necessary, the device will launch a speech synthesis engine (e.g., Google Text-to-Speech API) and play back the generated translation aloud. This speech playback feature allows users to hear information aloud even in situations where visual confirmation is difficult.

[0522] Specific examples

[0523] As an example, consider the situation where a user is looking at a menu in a French restaurant.

[0524] 1. A user looks at a restaurant menu in French.

[0525] 2. The device uses its camera to capture a video frame of the menu.

[0526] 3. The device uses OCR to extract the text "Poulet rôti".

[0527] 4. The device uses an eye-tracking sensor to determine the user's point of gaze and focuses on "Poulet rôti."

[0528] 5. The terminal sends the text data "Poulet rôti" to the translation processing device.

[0529] 6. The server receives the request and uses an AI translation engine to translate "Poulet rôti" into English "Roast chicken."

[0530] 7. The server sends the translation results to the device.

[0531] 8. The device overlays the translation result "Roast chicken" on the user's field of view.

[0532] 9. If necessary, the device plays "Roast chicken" aloud, providing both visual and auditory confirmation of the information to the user.

[0533] Example prompt sentence:

[0534] "The user looks at the menu," "The device activates the camera," "The OCR engine extracts the text," "The eye-tracking sensor identifies the gaze point," "The translation processor performs the translation," "An overlay is displayed," and "A speech synthesis device plays back the audio."

[0535] The present invention is a system that allows users to easily understand text written in different languages ​​in real time, and enables smooth communication and information access by integrating visual and audio information.

[0536] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0537] Step 1:

[0538] The user wears the smart glasses and looks around at what comes into view. The user moves to capture the text they see. The input is the user's line of sight. The output is the image of the view that comes into the camera's capture range.

[0539] Specific behavior:

[0540] The object the user wants to see is placed in the field of view of the smart glasses.

[0541] The device's built-in camera activates and captures the user's field of view.

[0542] Step 2:

[0543] The device captures the field of view with a high-resolution camera. The input is the field of view captured by the camera. The output is a video frame stored in a buffer.

[0544] Specific behavior:

[0545] The device continuously captures video at a specific frame rate (e.g., 30 frames per second).

[0546] The device stores the captured video frames in a buffer.

[0547] Step 3:

[0548] The device inputs video frames into an optical character recognition (OCR) engine to extract character data. The input is the video frames stored in a buffer. The output is the character data in digital form.

[0549] Specific behavior:

[0550] The device launches an OCR engine (e.g., Tesseract OCR) and provides the captured video frames as input data.

[0551] The OCR engine analyzes the text data from the image data, extracts the digital character data, and returns it to the terminal.

[0552] Step 4:

[0553] The device uses an eye-tracking sensor to track the user's gaze. The input is the user's gaze. The output is information about the specific text region being gazed at.

[0554] Specific behavior:

[0555] The device's built-in eye tracking sensor monitors the user's eye movements in real time.

[0556] The device analyzes the gaze data and identifies the text area the user is gazing at.

[0557] Step 5:

[0558] The terminal transmits the extracted text data and the user's language preference information to the translation processing device. The input is the digital text data and the language preference information. The output is the generation and transmission of a translation request.

[0559] Specific behavior:

[0560] The terminal sends data to the translation processing device in the form of an HTTP request (e.g., REST API).

[0561] The terminal generates a translation request including digital text data and the user's language setting information, and transmits the request over the network.

[0562] Step 6:

[0563] The server receives a translation request and translates the text data using a translation engine. The input is the data in the translation request. The output is the translated text.

[0564] Specific behavior:

[0565] The server parses the request data and passes it to a translation engine (e.g., Google Translate API).

[0566] The translation engine translates the original text into the specified language, generates the translation result, and returns it to the server.

[0567] Step 7:

[0568] The server generates translation results and sends them to the terminal. The input is the translated text data. The output is the translation results sent to the terminal.

[0569] Specific behavior:

[0570] The server packages the translation results in JSON format and sends them to the device.

[0571] The server uses data compression techniques (e.g., GZIP) to minimize the data transmitted.

[0572] Step 8:

[0573] The translation results received by the terminal are overlaid on the user's field of view. The input is the translation results received from the server. The output is the translation results displayed in the user's field of view.

[0574] Specific behavior:

[0575] After the device receives the translation result, it will be displayed by properly arranging it in the original text area.

[0576] The terminal dynamically renders the translated text displayed in the user's field of view.

[0577] Step 9:

[0578] The device activates a speech synthesis engine as needed and plays back the translation result aloud. The input is the text data of the translation result. The output is the audio that is played back to the user.

[0579] Specific behavior:

[0580] The device generates voice data using a speech synthesis engine (e.g., Google Text-to-Speech API).

[0581] The device plays the generated audio data from the speaker.

[0582] This series of processes enables users to understand text in different languages ​​in real time, and enables smooth communication and information access through the integration of visual and audio information.

[0583] (Application example 1)

[0584] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0585] Passengers in self-driving vehicles face challenges when traveling to different cultural regions, such as Japan, where rapid and accurate understanding of traffic signs and information boards in different languages ​​is a major challenge. In particular, not being able to understand place names and directions while driving or in a moving vehicle can lead to confusion and misunderstanding. Current technology is limited in its ability to translate text in real time and provide visual and auditory translations, creating a need for a system that can address this issue.

[0586] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0587] In this invention, the server includes a camera for monitoring the user's field of view, a means for capturing objects visually perceived by the user, an optical character recognition (OCR) engine for extracting text information from the captured video frames, an eye-tracking sensor for tracking the user's gaze and identifying the text area most gazed at, a means for sending the extracted text information to a translation service and making a translation request, a means for overlaying the translation result received from the translation service on the user's field of view, a speech synthesis engine for playing back the translation result as voice, and a means for recognizing, translating, and displaying signs in different languages ​​in real time while the user is in a moving vehicle. This enables passengers to quickly understand signs and information boards in different languages ​​both visually and audibly.

[0588] A "camera" is a device used to capture an image of a user's field of view.

[0589] An "Optical Character Recognition (OCR) engine" is software for extracting textual information from captured video frames.

[0590] An "eye tracking sensor" is a device that detects the user's gaze and identifies the area where the gaze is most focused.

[0591] A "translation service" is an online service or server that converts extracted text information into a specified language.

[0592] A "translation request" is a data transmission requesting a translation that includes the user's language setting and extracted text information.

[0593] "Overlay display" is a technology that displays translated text information superimposed on the user's field of vision.

[0594] A "speech synthesis engine" is software that converts text information into speech and provides it to the user audibly.

[0595] "Moving vehicle" means a vehicle, including automated public transport and other means of transportation.

[0596] "Signs" refer to guide signs and warning signs installed on roads and in public places.

[0597] This invention is a system using smart glasses worn by users that allows users to recognize signs and information boards in different languages ​​in real time and provides translation results both visually and audibly, making it easier for users to understand information in cross-cultural environments.

[0598] Hardware and software used

[0599] Smart glasses: Equipped with a camera, eye tracking sensor, and a see-through display.

[0600] Server: An online system for performing translation services.

[0601] Software: OCR engines (e.g., Tesseract OCR), translation APIs (e.g., Google Translate API), and speech synthesis engines.

[0602] Processing flow

[0603] The system performs the following steps:

[0604] Image Capture

[0605] While a user wears smart glasses and moves around in an autonomous vehicle, a camera captures signs and guidance displays that come into view. The camera monitors the user's field of view in real time.

[0606] OCR processing

[0607] The server inputs the captured video frames into an OCR engine to extract the text information. The optical character recognition (OCR) engine converts the text information in the captured frames into digital data with high accuracy.

[0608] Eye tracking

[0609] The server uses an eye-tracking sensor to detect the user's gaze and identify the text area where the gaze is most focused, allowing important information to be processed first even when there are multiple text areas.

[0610] Text Translation

[0611] The server sends the extracted text to a translation service, requesting a translation. The translation service also includes the user's language preference data and translates the text into the specified language, such as Japanese to English.

[0612] Display translation results and play audio

[0613] The user sees the translation results overlaid on the smart glasses' transparent display, and the translation results are also played back as audio by a speech synthesis engine, allowing the user to confirm the information both visually and audibly.

[0614] Specific examples

[0615] For example, consider a case where a user finds a Japanese sign indicating "next station" in a self-driving vehicle.

[0616] 1. The user puts on the smart glasses and brings the sign into view.

[0617] 2. The camera captures video of the sign.

[0618] 3. The server uses an OCR engine to extract the text "Next Station."

[0619] 4. The server uses an eye-tracking sensor to analyze the user's gaze and confirm that their gaze is focused on the "next station."

[0620] 5. The extracted text data is sent to a translation service, and a request is made to translate it into the user's preferred language (e.g., English).

[0621] 6. The translation service translates "next station" as "Next Station" and sends it back to the server.

[0622] 7. The server overlays the translation result, "Next Station," on the transparent display of the smart glasses, and simultaneously plays "Next Station" using the speech synthesis engine.

[0623] This allows the user to immediately understand visually and audibly that "next station" is "Next Station" in English.

[0624] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0625] Step 1:

[0626] The user wears smart glasses and uses a camera to capture signs and guide displays in their field of vision. The device's camera captures the image in the user's field of vision in real time and acquires the image frames. The input is the image frames captured by the camera, and the captured image data is obtained as the output.

[0627] Step 2:

[0628] The server inputs the captured video frame into an OCR engine to extract the text information. The server acquires the digital data of the video frame and uses an OCR engine (e.g., Tesseract OCR) to analyze the text information in the image. The input is the video data, and the output is the recognized text data.

[0629] Step 3:

[0630] The server uses an eye-tracking sensor to detect the user's gaze and identify the text area where the gaze is most focused. The server analyzes the gaze data obtained from the eye-tracking sensor and determines the location where the user is most focused. The input is the gaze data and recognized character data, and the output is the character area corresponding to the gaze point.

[0631] Step 4:

[0632] The server sends the extracted text information to a translation service and makes a translation request. Based on the user's language settings, the server sends a request to a translation API (e.g., Google Translate API) to translate the recognized text information into the target language. The input is the text data corresponding to the gaze point and language setting data, and the output is the completion of sending a translation request.

[0633] Step 5:

[0634] The translation service receives the request and returns the translation result into the specified language to the server. The server receives the translated data returned from the translation API. The input is the translation request, and the translated text data is returned to the server as the output.

[0635] Step 6:

[0636] The server sends the translation results to the smart glasses, which overlay the translation results on the user's field of view. The server then sends instructions to overlay the translated text data on the smart glasses' transparent display. The input is the translated text data, and the output is an overlay display on the user's field of view.

[0637] Step 7:

[0638] The server uses a speech synthesis engine to play back the translation results as audio. The server inputs the translated text data into the speech synthesis engine to generate audio data. The generated audio data is played back through the smart glasses' speakers. The input is the translated text data, and the output is audio that is generated and played back so that the user can hear it.

[0639] This series of processes allows users to understand signs and guidance displays in different languages ​​in real time.

[0640] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0641] The present invention relates to a smart glasses system worn by a user, which includes a camera, an optical character recognition (OCR) engine, an eye tracking sensor, a translation server, a translation result overlay display function, a speech synthesis engine, and an emotion engine.

[0642] Program processing explanation

[0643] Text Capture

[0644] The device constantly monitors the user's field of vision, capturing images of signs, menus, books, and other items the user is looking at through the smart glasses. For example, if the user comes across a menu written in a foreign language, the device will capture a video frame of that menu.

[0645] OCR (Optical Character Recognition)

[0646] The video frame captured by the device is input into the OCR engine to extract text data. The text "Poulet rôti" in the menu the user is viewing is extracted using OCR.

[0647] Eye tracking

[0648] The device uses an eye-tracking sensor to identify the area where the user's gaze is focused, allowing it to prioritize the text the user is most focused on, even when there are multiple text areas.

[0649] Submit a translation request

[0650] The terminal sends the extracted text data to a translation server. The translation request includes the original text data as well as the user's language preference, for example, a translation request from French to English.

[0651] Translation Processing

[0652] The server processes the received text data using an AI translation engine. The original text is converted into high-quality translated text based on the target language set by the user. The server then processes the text and sends the translation results back to the device.

[0653] Sending translation results

[0654] The translation results generated by the server are sent to the terminal via the network. Data compression and caching techniques may be used to ensure user visibility.

[0655] Displaying translation results

[0656] The device receives the translation result and displays it as an overlay in the user's field of view. The translation result is displayed visually on top of the original text, allowing the user to instantly understand the meaning. For example, "Poulet rôti" can be displayed as "Roast chicken."

[0657] Audio Support

[0658] The device can optionally use a speech synthesis engine to play back the translation results aloud, allowing users to hear the information even when they cannot see.

[0659] emotion recognition

[0660] The device uses the emotion engine to recognize emotions from the user's facial expressions, voice tone, heart rate, etc. For example, if the user is feeling stressed or anxious, this information will be detected by the emotion engine.

[0661] Adjusting the display method

[0662] The device will adjust how the translation results are displayed based on the perceived emotion: for example, if the user is feeling anxious, the translation results will be displayed in a larger font size and played aloud.

[0663] Specific examples

[0664] As an example, consider the situation where a user is looking at a menu in a French restaurant.

[0665] 1. A user looks at a restaurant menu in French.

[0666] 2. The device uses its camera to capture a video frame of the menu.

[0667] 3. The device uses OCR to extract the text "Poulet rôti".

[0668] 4. The device uses an eye-tracking sensor to determine the user's point of gaze and focuses on "Poulet rôti."

[0669] 5. The device sends the text data "Poulet rôti" to the translation server.

[0670] 6. The server receives the request and uses an AI translation engine to translate "Poulet rôti" into English "Roast chicken."

[0671] 7. The server sends the translation result "Roast chicken" to the terminal.

[0672] 8. The device overlays the translation result "Roast chicken" on the user's field of view.

[0673] 9. The device uses an emotion engine to recognize the user's emotions, and if they indicate anxiety, it displays the translation results in a larger font size and plays them aloud.

[0674] In this way, the present invention provides a system that helps users understand text in different languages ​​in real time, thereby removing language barriers and enabling smooth communication and information access.

[0675] The processing flow will be explained below.

[0676] Step 1:

[0677] A user puts on the smart glasses and looks at text that comes into their field of view. For example, the user sees a menu written in French.

[0678] Step 2:

[0679] The device activates the built-in camera and captures the user's field of view, capturing a frame of the menu.

[0680] Step 3:

[0681] The device inputs the captured video frame into the OCR engine to extract the text data. The text "Poulet rôti" is extracted by OCR.

[0682] Step 4:

[0683] The device uses an eye-tracking sensor to track the user's gaze and identify the text area that receives the most attention. It detects that the gaze is focused on the word "Poulet rôti."

[0684] Step 5:

[0685] The device sends the extracted text data "Poulet rôti" to the translation server, along with the user's language setting (from French to English).

[0686] Step 6:

[0687] The server receives the translation request and inputs the text data "Poulet rôti" into the AI ​​translation engine, which translates the text as "Roast chicken."

[0688] Step 7:

[0689] The server sends the generated translation result "Roast chicken" to the terminal.

[0690] Step 8:

[0691] The device receives the translation result "Roast chicken" and displays it as an overlay in the user's field of view, overlaying "Poulet rôti" with "Roast chicken."

[0692] Step 9:

[0693] The device uses the emotion engine to recognize emotions from the user's facial expressions, voice tone, heart rate, etc. For example, if the user is feeling anxious, the emotion engine will detect this.

[0694] Step 10:

[0695] The device adjusts how translation results are displayed based on the emotions recognized by the emotion engine: if the user is feeling anxious, the translation results will be displayed in a larger font size and played aloud.

[0696] Step 11:

[0697] (Optional) The device uses a speech synthesis engine to play the translation result "Roast chicken" aloud, allowing the user to confirm the translation both visually and audibly.

[0698] For example, consider a scenario where a user is ordering a menu item at a French restaurant. This process helps users understand text in different languages ​​in real time, providing them with peace of mind in situations where they may feel uneasy.

[0699] Example 2

[0700] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0701] Conventional translation systems require a lot of effort for users to visually understand information, making it difficult to understand text in different languages ​​in real time. Furthermore, because the display does not adjust according to the user's emotional state, they are unable to provide effective support when users are feeling stressed or anxious.

[0702] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a camera for monitoring the user's field of view, a means for capturing an object visually perceived by the user, an optical character recognition engine for extracting character data from the captured video frame, an eye-tracking device for tracking the user's gaze and identifying the most gazed character region, a means for transmitting the extracted character data to a translation device and making a translation request, a means for overlaying the translation result received from the translation device on the user's field of view, and an emotion recognition engine for recognizing the user's emotions and adjusting the display method based on the user's emotional state. This makes it easier to understand texts in different languages ​​in real time and enables effective support according to the user's emotional state.

[0703] "User" refers to a person who uses the system to obtain visual information and needs translation.

[0704] "Field of view image" refers to the image of the scenery or object seen by the user, captured through the camera in the smart glasses.

[0705] "Capture device" refers to a device such as a camera for capturing an image of a user's field of view.

[0706] "Optical character recognition engine" refers to software or hardware for extracting character data from captured video frames.

[0707] "Eye tracking device" refers to a sensor that monitors the movement of a user's eyes and identifies the area of ​​text that is most frequently looked at.

[0708] "Translation device" refers to a server or service for translating extracted character data into a specified target language.

[0709] "Overlay display" refers to a method of displaying the translation results superimposed on the user's field of vision, i.e., displaying them as additional information on the original field of vision image.

[0710] An "emotion recognition engine" is software or hardware for recognizing a user's emotional state, and refers to a device that analyzes emotions from facial expressions, vocal tone, heart rate, etc.

[0711] "Adjusting the display method" refers to the process of changing the display format of the translation result depending on the user's emotional state.

[0712] The present invention relates to a smart glasses system worn by a user, specifically a system that aims to translate and display text in different languages ​​in real time. The main hardware components of this system include a camera, an eye-tracking sensor, and a speech synthesis engine, and the software components include an optical character recognition (OCR) engine, a translation server, and an emotion recognition engine.

[0713] The device constantly monitors the user's field of vision using a camera, capturing signs, menus, books, and other items that come into the user's sight. For example, when a user looks at a restaurant menu, the device captures the menu frame by frame. The camera used here could be a versatile wearable camera.

[0714] The video frames captured by the device are input into an OCR engine to extract text data. Specifically, OCR engines such as Google Cloud Vision and Tesseract can be used. For example, the string "Poulet rôti" written in French can be extracted.

[0715] Next, the device uses an eye-tracking sensor to identify the user's gaze point. A specific example of an eye-tracking device is the Tobii Eyetracker. This allows the device to prioritize the text the user is looking at, even when there are multiple text areas. For example, it can detect when the user is looking at "Poulet rôti."

[0716] The device sends the extracted text data to a translation server. This translation request includes the original text data and the user's language setting (e.g., French to English). The server receives the translation request and processes it using an AI translation engine. For example, you can use the Google Translate API or Microsoft Translator API. The server translates "Poulet rôti" to "Roast chicken."

[0717] The translation results generated by the server are sent to the terminal via the network, and data compression and caching techniques may be used to maintain user visibility.

[0718] The device receives the translation result and displays it as an overlay in the user's field of view. It is displayed over the original text, so the user can visually recognize the translated content. For example, "Roast chicken" is displayed in the user's field of view.

[0719] Furthermore, the device can use a speech synthesis engine such as Amazon Polly or Google Text-to-Speech to play back the translation results as audio, if necessary, to support not only visual translation but also auditory comprehension.

[0720] The device uses an emotion engine to recognize the user's emotional state. For example, it uses technologies such as Affectiva and IBM Watson Emotional Analysis to determine the user's emotional state based on their facial expressions, heart rate, and voice tone. If the user is feeling stressed or anxious, this information will be detected by the emotion engine.

[0721] Finally, the device will adjust the way translation results are displayed based on the recognized emotion. For example, if the user is feeling anxious, the device will display the translation results in a larger font size and play a voice message, making the user feel more comfortable receiving the translation results.

[0722] Specific examples

[0723] Take the specific example of a user looking at a menu at a restaurant in France.

[0724] 1. A user looks at a restaurant menu in French.

[0725] 2. The device uses its camera to capture a video frame of the menu.

[0726] 3. The device uses OCR to extract the text "Poulet rôti".

[0727] 4. The device uses an eye-tracking sensor to determine the user's point of gaze and focuses on "Poulet rôti."

[0728] 5. The device sends the text data "Poulet rôti" to the translation server.

[0729] 6. The server receives the request and uses an AI translation engine to translate "Poulet rôti" into English "Roast chicken."

[0730] 7. The server sends the translation result "Roast chicken" to the terminal.

[0731] 8. The device overlays the translation result "Roast chicken" on the user's field of view.

[0732] 9. The device uses an emotion engine to recognize the user's emotions, and if they indicate anxiety, it displays the translation results in a larger font size and plays them aloud.

[0733] Example prompts for generative AI models

[0734] "Video Capture": The smart glasses' camera captures the video of the user's field of vision. The captured video is analyzed by the OCR engine to extract text data. The eye-tracking sensor identifies the user's point of gaze and sends the key text to the translation server. The server generates the translation results, which the device overlays in the user's field of vision. If necessary, the speech synthesis engine plays the translation results aloud. Additionally, the emotion engine recognizes the user's emotional state and adjusts the display accordingly.

[0735] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0736] Step 1: Capture a visual image

[0737] The device constantly monitors the user's field of view. As input, the camera captures live images of the scenery and objects the user is looking at. This captured image is sent to the next step, where it is processed for text extraction. Specifically, the camera mounted on the smart glasses captures images of what is in front of the user's eyes in real time.

[0738] Step 2: OCR processing

[0739] The device inputs the captured video frames into an OCR engine. The input is the captured video from step 1. The OCR engine (e.g., Google Cloud Vision or Tesseract) extracts text data from this video. Specifically, the OCR engine analyzes the image and converts the characters into digital text. The output of this step is the extracted text data (e.g., "Poulet rôti").

[0740] Step 3: Eye tracking

[0741] The device uses an eye-tracking sensor to identify the user's gaze focus point. The input requires data capturing the user's eye movements in real time. The eye-tracking device (e.g., Tobii Eyetracker) analyzes this data and identifies the text area the user is most focused on. Specifically, the eye-tracking sensor tracks the position and movement of the user's pupils, detecting the gaze direction and focus point. The output of this step is text data with the user's gaze point identified.

[0742] Step 4: Submit a translation request

[0743] The device sends the extracted text data to the translation server. The input includes the text data extracted in step 2 and the user's language preference data (e.g., French to English translation). The request is sent to the translation server, and the text data is ready for translation. Specifically, the device sends the text and language preference over the network to the translation server. The output of this step is the translation request data sent to the translation server.

[0744] Step 5: Translation process

[0745] The server processes the received text data with an AI translation engine. The input includes the text data sent in step 4 and the user's language preference data. The AI ​​translation engine (e.g., Google Translate API or Microsoft Translator API) converts the original text into the target language. Specifically, the AI ​​analyzes the text and generates the optimal translation result. The output of this step is the translated text data (e.g., "Roast chicken").

[0746] Step 6: Send the translation

[0747] The server sends the translation results it has generated to the device. The input includes the translated text data generated in step 5. The server sends this data to the device over the network. Specifically, the server converts the translation results into an appropriate format and uses data compression and caching techniques to transmit them efficiently. The output of this step is the translated text data received by the device.

[0748] Step 7: Overlaying the translation results

[0749] The device overlays the translation result in the user's field of view. The input includes the translated text data received in step 6. Specifically, the device overlays the translated text on the original field of view, allowing the user to visually recognize the translation result. The output of this step is the translated text (e.g., "Roast chicken") overlaid in the user's field of view.

[0750] Step 8: Playback by Text-to-Speech Engine

[0751] The device uses a speech synthesis engine, if necessary, to play back the translation result. The input includes the translation text data received in step 6. The speech synthesis engine (e.g., Amazon Polly or Google Text-to-Speech) converts the text data into speech data and plays it back. Specifically, this process generates speech data from the text and plays it back through a speaker. The output of this step is a speech playback of the translation result.

[0752] Step 9: Emotion Recognition

[0753] The device uses an emotion engine to recognize the user's emotional state. Inputs include data such as the user's facial expressions, heart rate, and voice tone. The emotion recognition engine (e.g., Affectiva or IBM Watson Emotional Analysis) analyzes this data to determine the user's emotional state. Specifically, the process involves analyzing data acquired by sensors to detect emotions such as stress or anxiety. The output of this step is the recognized user's emotional state.

[0754] Step 10: Adjust how the translation results are displayed

[0755] The device adjusts the display of the translation result based on the recognized emotion. The input includes the user's emotional state recognized in step 9 and the translated text data obtained in steps 6 or 7. Specific operations include displaying the translation result in a larger font size and adjusting the volume of the audio playback. The output of this step is a display of the translation result optimized for the user's emotional state.

[0756] (Application example 2)

[0757] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0758] Language barriers still exist in many situations today, posing a major problem when it comes to communication at tourist destinations and brick-and-mortar stores. Travelers and those who are not fluent in foreign languages ​​often have difficulty understanding information on local signs, menus, and guide boards. Furthermore, there is a lack of appropriate and quick ways to obtain information in emergencies and stressful situations.

[0759] The specification process by the specification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes an imaging device for monitoring the user's field of view, a means for capturing an object visually perceived by the user, an optical character recognition (OCR) engine for extracting character data from the captured video frame, an eye-tracking sensor for tracking the user's gaze and identifying the most gazed-at character area, a means for transmitting the extracted character data to a translation server and making a translation request, a means for overlaying the translation result received from the translation server on the user's field of view, a speech synthesis engine for playing the translation result aloud as needed, and an emotion engine for recognizing the user's emotional state and adjusting the display method. This makes it possible to understand information in different languages ​​in real time and provide the information in an optimal format depending on the user's emotional state.

[0760] The "user's visual field image" refers to the image range within the visual field that the user can perceive with normal vision.

[0761] An "imaging device" is an electronic device for capturing images, such as a camera.

[0762] A "capturing means" is a device or technology that records visually perceived objects in digital form.

[0763] An "optical character recognition (OCR) engine" is software for extracting text information from images and videos.

[0764] An "eye tracking sensor" is a sensor for tracking the direction and focus of a user's gaze.

[0765] "Character data" refers to the text information extracted by the OCR engine.

[0766] A "translation server" is a server that translates input character data into another specified language.

[0767] The "means for making a translation request" is a communication means for transmitting the captured character data and a translation request to the server.

[0768] "Means for overlay display" refers to a technique for displaying translation results superimposed within the user's field of vision.

[0769] A "speech synthesis engine" is software that converts text data into speech output.

[0770] An "emotion engine" is software that recognizes a user's emotional state from their facial expressions and biometric information.

[0771] "Tourist attractions and brick-and-mortar stores" refer to places that users actually visit and stores where sales activities are carried out.

[0772] The "function for adjusting the display method" is a technology for changing the display format of presented information depending on the emotional state of the user.

[0773] The present invention provides a system for removing language barriers at tourist spots and brick-and-mortar stores. The system operates based on smart glasses worn by users and includes the following main software and hardware components:

[0774] Specifically, the following hardware and software are used:

[0775] Smart glasses: imaging device, eye tracking sensor, display

[0776] OCR engine: Tesseract OCR

[0777] Translation server: Google Cloud Translation API

[0778] Speech synthesis engine: Amazon Polly

[0779] Emotion engine: Affectiva SDK

[0780] Program processing explanation

[0781] Camera Capture

[0782] When a user looks at tourist information, signs, menus, etc. through smart glasses, the camera built into the glasses captures images within their field of view, enhancing the user's visual experience and providing real-time information.

[0783] OCR processing

[0784] The captured video is analyzed by the Tesseract OCR engine to extract text data from the image, converting the visual information into digital text.

[0785] Eye tracking

[0786] An eye-tracking sensor monitors the user's gaze and identifies the text area the user is most focused on, allowing the user to prioritize the information that is most important to them, even when multiple pieces of text are present.

[0787] Translation Request

[0788] The extracted text data is sent to a translation server, where it is translated into the specified target language in real time using the Google Cloud Translation API. The request parameters include the source language, target language, and the extracted text data.

[0789] Displaying translation results

[0790] The translation results received from the translation server are overlaid on the smart glasses display, allowing users to simultaneously view the original text and the translated text, facilitating quick understanding.

[0791] Audio playback

[0792] If necessary, the translation result is synthesized into speech using Amazon Polly and played back to the user, allowing them to confirm the information audibly as well as visually.

[0793] emotion recognition

[0794] Using the Affectiva SDK, the app analyzes the user's facial expressions and biometrics to recognize their current emotional state. Based on this, the way translation results are displayed will be automatically adjusted. For example, if the user is stressed, the translation will be displayed in a larger font or played aloud.

[0795] Specific examples

[0796] For tourist information:

[0797] "Loads the description of a Universal Studios attraction, translates it from Japanese to English, and displays it on the smart glasses."

[0798] For restaurant menus:

[0799] "Italian restaurant menus can be translated from Italian to Japanese and displayed on smart glasses, with voice guidance available if necessary."

[0800] This allows users to smoothly understand information and enjoy a comfortable experience even in different language environments.

[0801] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0802] Step 1:

[0803] The user wears the smart glasses and can see objects within their field of view at tourist spots or brick-and-mortar stores.

[0804] Input: User's field of view (2D image)

[0805] Output: Visual field video data (image frames)

[0806] How it works: The imaging device built into the smart glasses captures the user's field of view in real time with a camera and saves it as image data.

[0807] Step 2:

[0808] The video frames captured by the device are input into an OCR engine, which extracts character data from the images.

[0809] Input: Video frame (image data)

[0810] Output: Character data (text)

[0811] How it works: The Tesseract OCR engine processes the captured video data, optically recognizing characters in the video and extracting them as text data.

[0812] Step 3:

[0813] The device uses an eye-tracking sensor to identify where the user's gaze is focused.

[0814] Input: Text data, user gaze information

[0815] Output: Character data (text) that the user is gazing at

[0816] Movement: The eye-tracking sensor monitors the user's gaze in real time and identifies areas of text that the user is particularly focused on. The text data in the identified areas is then filtered.

[0817] Step 4:

[0818] The terminal transmits the extracted and identified character data to a translation server and makes a translation request.

[0819] Input: Character data (text) that the user is gazing at, translation request (original text and language setting)

[0820] Output: Translated character data (text)

[0821] How it works: The device sends a translation request to the Google Cloud Translation API and gets the text result translated into the specified target language.

[0822] Step 5:

[0823] The translation results acquired by the server are sent to the terminal and displayed as an overlay in the user's field of view.

[0824] Input: Translated character data (text)

[0825] Output: Translation results displayed in the user's field of view (overlay text)

[0826] How it works: The translation results are received from the translation server and displayed on the smart glasses display, overlaid on the original text.

[0827] Step 6:

[0828] If necessary, the device will play back the translation results aloud using a speech synthesis engine.

[0829] Input: Translated character data (text)

[0830] Output: Audio output (audio of translation results)

[0831] How it works: Amazon Polly is used to convert the translated text into speech and play it through the smart glasses' speakers.

[0832] Step 7:

[0833] The device uses an emotion engine to recognize the user's emotional state and adjust how the translation results are displayed.

[0834] Input: User's facial expression data, biometric information, translation result (text)

[0835] Output: Adjusted display (font size, color, etc.), audio playback

[0836] How it works: The Affectiva SDK analyzes the user's facial expressions and heart rate to identify emotional states such as stress or anxiety. Based on the identified emotional state, the system adjusts the way translation results are displayed and the audio playback. For example, if the user is feeling stressed, the system will display the translation results in a larger font and play the audio to emphasize it.

[0837] This is the specific processing flow of the system, which allows users to smoothly obtain information even in a multilingual environment, providing a more comfortable experience.

[0838] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0839] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0840] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0841] [Third embodiment]

[0842] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0843] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0844] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0845] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0846] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0847] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0848] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0849] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0850] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0851] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0852] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0853] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0854] The present invention relates to a smart glasses system worn by a user, which includes a camera, an optical character recognition (OCR) engine, an eye tracking sensor, a translation server, a translation result overlay display function, and a speech synthesis engine.

[0855] Program processing explanation

[0856] Text Capture

[0857] The device constantly monitors the user's field of vision, capturing the scenery, signs, menus, and other things the user sees through the smart glasses. For example, if the user spots a sign written in a foreign language, the camera will capture that object.

[0858] OCR (Optical Character Recognition)

[0859] The image frames captured by the device are fed into an OCR engine to extract text data, identifying text segments written by the user in different languages ​​and extracting them as digital character data.

[0860] Eye tracking

[0861] The device uses an eye-tracking sensor to identify the area where the user is focusing their gaze, allowing it to prioritize the text the user is most focused on, even in situations where there are multiple text areas.

[0862] Submit a translation request

[0863] The device sends the extracted text data to a translation server. The translation request includes the original text data as well as the user's language preference. For example, a request is made to translate a Japanese sign into English.

[0864] Translation Processing

[0865] The server processes the received text data using a translation engine. The original text is converted into high-quality translated text based on the target language set by the user. The server then processes the text and returns the translation results to the device.

[0866] Sending translation results

[0867] The translation results generated by the server are sent to the terminal via the network. Data compression and caching techniques may be used to ensure user visibility.

[0868] Displaying translation results

[0869] The device receives the translation result and displays it as an overlay in the user's field of vision. The translation result is displayed visually on top of the original text, allowing the user to instantly understand the meaning. For example, the sign "Exit" that the user is looking at is displayed as the translation result "Exit."

[0870] Audio support (optional)

[0871] The device can optionally use a speech synthesis engine to play back the translation results aloud, allowing users to hear the information even when they cannot see.

[0872] Specific examples

[0873] As an example, consider the situation where a user is looking at a menu in a French restaurant.

[0874] 1. A user looks at a restaurant menu in French.

[0875] 2. The device uses its camera to capture a video frame of the menu.

[0876] 3. The device uses OCR to extract the text "Poulet rôti".

[0877] 4. The device uses an eye-tracking sensor to determine the user's point of gaze and focuses on "Poulet rôti."

[0878] 5. The device sends the text data "Poulet rôti" to the translation server.

[0879] 6. The server receives the request and uses an AI translation engine to translate "Poulet rôti" into English "Roast chicken."

[0880] 7. The server sends the translation results to the device.

[0881] 8. The device overlays the translation result "Roast chicken" on the user's field of view.

[0882] 9. If necessary, the device plays "Roast chicken" aloud, providing both visual and auditory confirmation of the information to the user.

[0883] In this way, the present invention provides a system that helps users understand text in different languages ​​in real time, thereby removing language barriers and enabling smooth communication and information access.

[0884] The processing flow will be explained below.

[0885] Step 1:

[0886] A user puts on the smart glasses and looks at text that comes into their field of vision. For example, the user sees a sign written in French.

[0887] Step 2:

[0888] The device activates its built-in camera and captures a video of the user's field of view, capturing a video frame of the sign the user is looking at.

[0889] Step 3:

[0890] The device inputs the captured video frame into the OCR engine, which extracts text information from the video. The text "Sortie" on the sign the user is looking at is extracted.

[0891] Step 4:

[0892] The device uses an eye-tracking sensor to track the user's gaze and identify the text area that receives the most attention. It detects that the gaze is focused on the "Sortie" section.

[0893] Step 5:

[0894] The device sends the extracted text data "Sortie" to the translation server, along with the user's language setting (e.g., French to English).

[0895] Step 6:

[0896] The server receives the translation request and passes the text data "Sortie" to the AI ​​translation engine, which translates "Sortie" as "Exit."

[0897] Step 7:

[0898] The server sends the generated translation result "Exit" to the terminal.

[0899] Step 8:

[0900] The translation result "Exit" received by the device is overlaid on the user's field of view, with "Exit" displayed over the original "Sortie" part.

[0901] Step 9:

[0902] (Optional) The device uses a speech synthesis engine to play the translation result "Exit" aloud, allowing the user to confirm the translation result both visually and audibly.

[0903] In this way, users can understand texts in different languages ​​in real time.

[0904] Example 1

[0905] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0906] In conventional smart glasses systems, the technology to translate and display visual information in real time is not fully developed. As a result, users often have difficulty understanding visual information when confronted with displays in a different language in a foreign country. Furthermore, existing systems do not identify gaze points using eye tracking or integrate visual information translation and voice support, making it difficult to achieve real-time information access and smooth communication.

[0907] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0908] In this invention, the server includes an image capture device for monitoring a user's field of view, a means for capturing an object visually perceived by the user, an optical character recognition (OCR) engine for extracting character data from the captured image frame, an eye-tracking sensor for tracking the user's gaze and identifying the most gazed character area, a means for transmitting the extracted character data to a translation processing device and making a translation request, and a means for overlaying the translation result received from the translation processing device on the user's field of view. This makes it easier for users to understand text written in different languages ​​in real time, and the integration of visual and audio information enables smooth communication and information access.

[0909] An "image capture device" is a device that monitors the user's field of view and captures the object that the user visually perceives.

[0910] An "Optical Character Recognition (OCR) engine" is an engine for extracting character data from captured video frames.

[0911] The "eye tracking sensor" is a sensor that tracks the user's gaze and identifies the text area that is most closely watched.

[0912] The "translation processing device" is a device that transmits extracted character data to a translation server and makes a translation request.

[0913] The "means for overlaying and displaying in the user's field of view" refers to a means for displaying the translation result received from the translation server in an overlaid manner in the user's field of view.

[0914] "User language setting information" is setting information related to the language used by the user and the language to be translated.

[0915] A "voice synthesizer" is a device that converts character data into voice data and plays it back aloud.

[0916] The present invention relates to a system that translates text in a different language that a user visually perceives in real time and overlays it on the user's field of vision. The system includes an image capture device, an optical character recognition (OCR) engine, an eye-tracking sensor, a translation processor, and a speech synthesizer.

[0917] Image acquisition device

[0918] The device constantly monitors the user's field of view. A high-resolution camera captures the scenery and text the user sees through the smart glasses in real time. This camera captures video frames and stores them in a buffer.

[0919] Optical Character Recognition (OCR) engine

[0920] The video frames captured by the device are input into an OCR engine (e.g., Tesseract OCR) to extract text data. The OCR engine analyzes the text data from the image data and extracts it as digital text data.

[0921] Eye tracking sensor

[0922] The device uses an eye-tracking sensor to track the user's gaze, which identifies the specific text area the user is looking at and prioritizes the most-looked-at text even when multiple pieces of text are present.

[0923] Translation Processing Device

[0924] The device sends the extracted text data and the user's language preference information to a translation processing device. The translation request is sent over the network, and the translation processing device receives the request. This device inputs the received text data into a translation engine (e.g., Google Translate API) and translates it into the specified language.

[0925] Overlay of translation results

[0926] The server sends the translation results to the terminal via the network. The terminal displays the received translation results as an overlay in the user's field of view. The overlay display allows the user to check the translation results while viewing the original text.

[0927] speech synthesizer

[0928] If necessary, the device will launch a speech synthesis engine (e.g., Google Text-to-Speech API) and play back the generated translation aloud. This speech playback feature allows users to hear information aloud even in situations where visual confirmation is difficult.

[0929] Specific examples

[0930] As an example, consider the situation where a user is looking at a menu in a French restaurant.

[0931] 1. A user looks at a restaurant menu in French.

[0932] 2. The device uses its camera to capture a video frame of the menu.

[0933] 3. The device uses OCR to extract the text "Poulet rôti".

[0934] 4. The device uses an eye-tracking sensor to determine the user's point of gaze and focuses on "Poulet rôti."

[0935] 5. The terminal sends the text data "Poulet rôti" to the translation processing device.

[0936] 6. The server receives the request and uses an AI translation engine to translate "Poulet rôti" into English "Roast chicken."

[0937] 7. The server sends the translation results to the device.

[0938] 8. The device overlays the translation result "Roast chicken" on the user's field of view.

[0939] 9. If necessary, the device plays "Roast chicken" aloud, providing both visual and auditory confirmation of the information to the user.

[0940] Example prompt sentence:

[0941] "The user looks at the menu," "The device activates the camera," "The OCR engine extracts the text," "The eye-tracking sensor identifies the gaze point," "The translation processor performs the translation," "An overlay is displayed," and "A speech synthesis device plays back the audio."

[0942] The present invention is a system that allows users to easily understand text written in different languages ​​in real time, and enables smooth communication and information access by integrating visual and audio information.

[0943] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0944] Step 1:

[0945] The user wears the smart glasses and looks around at what comes into view. The user moves to capture the text they see. The input is the user's line of sight. The output is the image of the view that comes into the camera's capture range.

[0946] Specific behavior:

[0947] The object the user wants to see is placed in the field of view of the smart glasses.

[0948] The device's built-in camera activates and captures the user's field of view.

[0949] Step 2:

[0950] The device captures the field of view with a high-resolution camera. The input is the field of view captured by the camera. The output is a video frame stored in a buffer.

[0951] Specific behavior:

[0952] The device continuously captures video at a specific frame rate (e.g., 30 frames per second).

[0953] The device stores the captured video frames in a buffer.

[0954] Step 3:

[0955] The device inputs video frames into an optical character recognition (OCR) engine to extract character data. The input is the video frames stored in a buffer. The output is the character data in digital form.

[0956] Specific behavior:

[0957] The device launches an OCR engine (e.g., Tesseract OCR) and provides the captured video frames as input data.

[0958] The OCR engine analyzes the text data from the image data, extracts the digital character data, and returns it to the terminal.

[0959] Step 4:

[0960] The device uses an eye-tracking sensor to track the user's gaze. The input is the user's gaze. The output is information about the specific text region being gazed at.

[0961] Specific behavior:

[0962] The device's built-in eye tracking sensor monitors the user's eye movements in real time.

[0963] The device analyzes the gaze data and identifies the text area the user is gazing at.

[0964] Step 5:

[0965] The terminal transmits the extracted text data and the user's language preference information to the translation processing device. The input is the digital text data and the language preference information. The output is the generation and transmission of a translation request.

[0966] Specific behavior:

[0967] The terminal sends data to the translation processing device in the form of an HTTP request (e.g., REST API).

[0968] The terminal generates a translation request including digital text data and the user's language setting information, and transmits the request over the network.

[0969] Step 6:

[0970] The server receives a translation request and translates the text data using a translation engine. The input is the data in the translation request. The output is the translated text.

[0971] Specific behavior:

[0972] The server parses the request data and passes it to a translation engine (e.g., Google Translate API).

[0973] The translation engine translates the original text into the specified language, generates the translation result, and returns it to the server.

[0974] Step 7:

[0975] The server generates translation results and sends them to the terminal. The input is the translated text data. The output is the translation results sent to the terminal.

[0976] Specific behavior:

[0977] The server packages the translation results in JSON format and sends them to the device.

[0978] The server uses data compression techniques (e.g., GZIP) to minimize the data transmitted.

[0979] Step 8:

[0980] The translation results received by the terminal are overlaid on the user's field of view. The input is the translation results received from the server. The output is the translation results displayed in the user's field of view.

[0981] Specific behavior:

[0982] After the device receives the translation result, it will be displayed by properly arranging it in the original text area.

[0983] The terminal dynamically renders the translated text displayed in the user's field of view.

[0984] Step 9:

[0985] The device activates a speech synthesis engine as needed and plays back the translation result aloud. The input is the text data of the translation result. The output is the audio that is played back to the user.

[0986] Specific behavior:

[0987] The device generates voice data using a speech synthesis engine (e.g., Google Text-to-Speech API).

[0988] The device plays the generated audio data from the speaker.

[0989] This series of processes enables users to understand text in different languages ​​in real time, and enables smooth communication and information access through the integration of visual and audio information.

[0990] (Application example 1)

[0991] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0992] Passengers in self-driving vehicles face challenges when traveling to different cultural regions, such as Japan, where rapid and accurate understanding of traffic signs and information boards in different languages ​​is a major challenge. In particular, not being able to understand place names and directions while driving or in a moving vehicle can lead to confusion and misunderstanding. Current technology is limited in its ability to translate text in real time and provide visual and auditory translations, creating a need for a system that can address this issue.

[0993] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0994] In this invention, the server includes a camera for monitoring the user's field of view, a means for capturing objects visually perceived by the user, an optical character recognition (OCR) engine for extracting text information from the captured video frames, an eye-tracking sensor for tracking the user's gaze and identifying the text area most gazed at, a means for sending the extracted text information to a translation service and making a translation request, a means for overlaying the translation result received from the translation service on the user's field of view, a speech synthesis engine for playing back the translation result as voice, and a means for recognizing, translating, and displaying signs in different languages ​​in real time while the user is in a moving vehicle. This enables passengers to quickly understand signs and information boards in different languages ​​both visually and audibly.

[0995] A "camera" is a device used to capture an image of a user's field of view.

[0996] An "Optical Character Recognition (OCR) engine" is software for extracting textual information from captured video frames.

[0997] An "eye tracking sensor" is a device that detects the user's gaze and identifies the area where the gaze is most focused.

[0998] A "translation service" is an online service or server that converts extracted text information into a specified language.

[0999] A "translation request" is a data transmission requesting a translation that includes the user's language setting and extracted text information.

[1000] "Overlay display" is a technology that displays translated text information superimposed on the user's field of vision.

[1001] A "speech synthesis engine" is software that converts text information into speech and provides it to the user audibly.

[1002] "Moving vehicle" means a vehicle, including automated public transport and other means of transportation.

[1003] "Signs" refer to guide signs and warning signs installed on roads and in public places.

[1004] This invention is a system using smart glasses worn by users that allows users to recognize signs and information boards in different languages ​​in real time and provides translation results both visually and audibly, making it easier for users to understand information in cross-cultural environments.

[1005] Hardware and software used

[1006] Smart glasses: Equipped with a camera, eye tracking sensor, and a see-through display.

[1007] Server: An online system for performing translation services.

[1008] Software: OCR engines (e.g., Tesseract OCR), translation APIs (e.g., Google Translate API), and speech synthesis engines.

[1009] Processing flow

[1010] The system performs the following steps:

[1011] Image Capture

[1012] While a user wears smart glasses and moves around in an autonomous vehicle, a camera captures signs and guidance displays that come into view. The camera monitors the user's field of view in real time.

[1013] OCR processing

[1014] The server inputs the captured video frames into an OCR engine to extract the text information. The optical character recognition (OCR) engine converts the text information in the captured frames into digital data with high accuracy.

[1015] Eye tracking

[1016] The server uses an eye-tracking sensor to detect the user's gaze and identify the text area where the gaze is most focused, allowing important information to be prioritized even when multiple text areas are present.

[1017] Text Translation

[1018] The server sends the extracted text to a translation service, requesting a translation. The translation service also includes the user's language preference data and translates the text into the specified language, such as Japanese to English.

[1019] Display translation results and play audio

[1020] The user sees the translation results overlaid on the smart glasses' transparent display, and the translation results are also played back as audio by a speech synthesis engine, allowing the user to confirm the information both visually and audibly.

[1021] Specific examples

[1022] For example, consider a case where a user finds a Japanese sign indicating "next station" in a self-driving vehicle.

[1023] 1. The user puts on the smart glasses and brings the sign into view.

[1024] 2. The camera captures video of the sign.

[1025] 3. The server uses an OCR engine to extract the text "Next Station."

[1026] 4. The server uses an eye-tracking sensor to analyze the user's gaze and confirm that their gaze is focused on the "next station."

[1027] 5. The extracted text data is sent to a translation service, and a request is made to translate it into the user's preferred language (e.g., English).

[1028] 6. The translation service translates "next station" as "Next Station" and sends it back to the server.

[1029] 7. The server overlays the translation result, "Next Station," on the transparent display of the smart glasses, and simultaneously plays "Next Station" using the speech synthesis engine.

[1030] This allows the user to immediately understand visually and audibly that "next station" is "Next Station" in English.

[1031] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1032] Step 1:

[1033] The user wears smart glasses and uses a camera to capture signs and guide displays in their field of vision. The device's camera captures the image in the user's field of vision in real time and acquires the image frames. The input is the image frames captured by the camera, and the captured image data is obtained as the output.

[1034] Step 2:

[1035] The server inputs the captured video frame into an OCR engine to extract the text information. The server acquires the digital data of the video frame and uses an OCR engine (e.g., Tesseract OCR) to analyze the text information in the image. The input is the video data, and the output is the recognized text data.

[1036] Step 3:

[1037] The server uses an eye-tracking sensor to detect the user's gaze and identify the text area where the gaze is most focused. The server analyzes the gaze data obtained from the eye-tracking sensor and determines the location where the user is most focused. The input is the gaze data and recognized character data, and the output is the character area corresponding to the gaze point.

[1038] Step 4:

[1039] The server sends the extracted text information to a translation service and makes a translation request. Based on the user's language settings, the server sends a request to a translation API (e.g., Google Translate API) to translate the recognized text information into the target language. The input is the text data corresponding to the gaze point and language setting data, and the output is the completion of sending a translation request.

[1040] Step 5:

[1041] The translation service receives the request and returns the translation result into the specified language to the server. The server receives the translated data returned from the translation API. The input is the translation request, and the translated text data is returned to the server as the output.

[1042] Step 6:

[1043] The server sends the translation results to the smart glasses, which overlay the translation results on the user's field of view. The server then sends instructions to overlay the translated text data on the smart glasses' transparent display. The input is the translated text data, and the output is an overlay display on the user's field of view.

[1044] Step 7:

[1045] The server uses a speech synthesis engine to play back the translation results as audio. The server inputs the translated text data into the speech synthesis engine to generate audio data. The generated audio data is played back through the smart glasses' speakers. The input is the translated text data, and the output is audio that is generated and played back so that the user can hear it.

[1046] This series of processes allows users to understand signs and guidance displays in different languages ​​in real time.

[1047] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1048] The present invention relates to a smart glasses system worn by a user, which includes a camera, an optical character recognition (OCR) engine, an eye tracking sensor, a translation server, a translation result overlay display function, a speech synthesis engine, and an emotion engine.

[1049] Program processing explanation

[1050] Text Capture

[1051] The device constantly monitors the user's field of vision, capturing images of signs, menus, books, and other items the user is looking at through the smart glasses. For example, if the user comes across a menu written in a foreign language, the device will capture a video frame of that menu.

[1052] OCR (Optical Character Recognition)

[1053] The video frame captured by the device is input into the OCR engine to extract text data. The text "Poulet rôti" in the menu the user is viewing is extracted using OCR.

[1054] Eye tracking

[1055] The device uses an eye-tracking sensor to identify the area where the user's gaze is focused, allowing it to prioritize the text the user is most focused on, even when there are multiple text areas.

[1056] Submit a translation request

[1057] The terminal sends the extracted text data to a translation server. The translation request includes the original text data as well as the user's language preference, for example, a translation request from French to English.

[1058] Translation Processing

[1059] The server processes the received text data using an AI translation engine. The original text is converted into high-quality translated text based on the target language set by the user. The server then processes the text and sends the translation results back to the device.

[1060] Sending translation results

[1061] The translation results generated by the server are sent to the terminal via the network. Data compression and caching techniques may be used to ensure user visibility.

[1062] Displaying translation results

[1063] The device receives the translation result and displays it as an overlay in the user's field of view. The translation result is displayed visually on top of the original text, allowing the user to instantly understand the meaning. For example, "Poulet rôti" can be displayed as "Roast chicken."

[1064] Audio Support

[1065] The device can optionally use a speech synthesis engine to play back the translation results aloud, allowing users to hear the information even when they cannot see.

[1066] emotion recognition

[1067] The device uses the emotion engine to recognize emotions from the user's facial expressions, voice tone, heart rate, etc. For example, if the user is feeling stressed or anxious, this information will be detected by the emotion engine.

[1068] Adjusting the display method

[1069] The device will adjust how the translation results are displayed based on the perceived emotion: for example, if the user is feeling anxious, the translation results will be displayed in a larger font size and played aloud.

[1070] Specific examples

[1071] As an example, consider the situation where a user is looking at a menu in a French restaurant.

[1072] 1. A user looks at a restaurant menu in French.

[1073] 2. The device uses its camera to capture a video frame of the menu.

[1074] 3. The device uses OCR to extract the text "Poulet rôti".

[1075] 4. The device uses an eye-tracking sensor to determine the user's point of gaze and focuses on "Poulet rôti."

[1076] 5. The device sends the text data "Poulet rôti" to the translation server.

[1077] 6. The server receives the request and uses an AI translation engine to translate "Poulet rôti" into English "Roast chicken."

[1078] 7. The server sends the translation result "Roast chicken" to the terminal.

[1079] 8. The device overlays the translation result "Roast chicken" on the user's field of view.

[1080] 9. The device uses an emotion engine to recognize the user's emotions, and if they indicate anxiety, it displays the translation results in a larger font size and plays them aloud.

[1081] In this way, the present invention provides a system that helps users understand text in different languages ​​in real time, thereby removing language barriers and enabling smooth communication and information access.

[1082] The processing flow will be explained below.

[1083] Step 1:

[1084] A user puts on the smart glasses and looks at text that comes into their field of view. For example, the user sees a menu written in French.

[1085] Step 2:

[1086] The device activates the built-in camera and captures the user's field of view, capturing a frame of the menu.

[1087] Step 3:

[1088] The device inputs the captured video frame into the OCR engine to extract the text data. The text "Poulet rôti" is extracted by OCR.

[1089] Step 4:

[1090] The device uses an eye-tracking sensor to track the user's gaze and identify the text area that receives the most attention. It detects that the gaze is focused on the word "Poulet rôti."

[1091] Step 5:

[1092] The device sends the extracted text data "Poulet rôti" to the translation server, along with the user's language setting (from French to English).

[1093] Step 6:

[1094] The server receives the translation request and inputs the text data "Poulet rôti" into the AI ​​translation engine, which translates the text as "Roast chicken."

[1095] Step 7:

[1096] The server sends the generated translation result "Roast chicken" to the terminal.

[1097] Step 8:

[1098] The device receives the translation result "Roast chicken" and displays it as an overlay in the user's field of view, overlaying "Poulet rôti" with "Roast chicken."

[1099] Step 9:

[1100] The device uses the emotion engine to recognize emotions from the user's facial expressions, voice tone, heart rate, etc. For example, if the user is feeling anxious, the emotion engine will detect this.

[1101] Step 10:

[1102] The device adjusts how translation results are displayed based on the emotions recognized by the emotion engine: if the user is feeling anxious, the translation results will be displayed in a larger font size and played aloud.

[1103] Step 11:

[1104] (Optional) The device uses a speech synthesis engine to play the translation result "Roast chicken" aloud, allowing the user to confirm the translation both visually and audibly.

[1105] For example, consider a scenario where a user is ordering a menu at a French restaurant. This process helps users understand text in different languages ​​in real time, providing them with peace of mind in situations where they may feel uneasy.

[1106] Example 2

[1107] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1108] Conventional translation systems require a lot of effort for users to visually understand information, making it difficult to understand text in different languages ​​in real time. Furthermore, because the display does not adjust according to the user's emotional state, they are unable to provide effective support when users are feeling stressed or anxious.

[1109] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a camera for monitoring the user's field of view, a means for capturing an object visually perceived by the user, an optical character recognition engine for extracting character data from the captured video frame, an eye-tracking device for tracking the user's gaze and identifying the most gazed character region, a means for transmitting the extracted character data to a translation device and making a translation request, a means for overlaying the translation result received from the translation device on the user's field of view, and an emotion recognition engine for recognizing the user's emotions and adjusting the display method based on the user's emotional state. This makes it easier to understand texts in different languages ​​in real time and enables effective support according to the user's emotional state.

[1110] "User" refers to a person who uses the system to obtain visual information and needs translation.

[1111] "Field of view image" refers to the image of the scenery or object seen by the user, captured through the camera in the smart glasses.

[1112] "Capture device" refers to a device such as a camera for capturing an image of a user's field of view.

[1113] "Optical character recognition engine" refers to software or hardware for extracting character data from captured video frames.

[1114] "Eye tracking device" refers to a sensor that monitors the movement of a user's eyes and identifies the area of ​​text that is most frequently looked at.

[1115] "Translation device" refers to a server or service for translating extracted character data into a specified target language.

[1116] "Overlay display" refers to a method of displaying the translation results superimposed on the user's field of vision, i.e., displaying them as additional information on the original field of vision image.

[1117] An "emotion recognition engine" is software or hardware for recognizing a user's emotional state, and refers to a device that analyzes emotions from facial expressions, vocal tone, heart rate, etc.

[1118] "Adjusting the display method" refers to the process of changing the display format of the translation result depending on the user's emotional state.

[1119] The present invention relates to a smart glasses system worn by a user, specifically a system that aims to translate and display text in different languages ​​in real time. The main hardware components of this system include a camera, an eye-tracking sensor, and a speech synthesis engine, and the software components include an optical character recognition (OCR) engine, a translation server, and an emotion recognition engine.

[1120] The device constantly monitors the user's field of vision using a camera, capturing signs, menus, books, and other items that come into the user's sight. For example, when a user looks at a restaurant menu, the device captures the menu frame by frame. The camera used here could be a versatile wearable camera.

[1121] The video frames captured by the device are input into an OCR engine to extract text data. Specifically, OCR engines such as Google Cloud Vision and Tesseract can be used. For example, the string "Poulet rôti" written in French can be extracted.

[1122] Next, the device uses an eye-tracking sensor to identify the user's gaze point. A specific example of an eye-tracking device is the Tobii Eyetracker. This allows the device to prioritize the text the user is looking at, even when there are multiple text areas. For example, it can detect when the user is looking at "Poulet rôti."

[1123] The device sends the extracted text data to a translation server. This translation request includes the original text data and the user's language setting (e.g., French to English). The server receives the translation request and processes it using an AI translation engine. For example, you can use the Google Translate API or Microsoft Translator API. The server translates "Poulet rôti" to "Roast chicken."

[1124] The translation results generated by the server are sent to the terminal via the network, and data compression and caching techniques may be used to maintain user visibility.

[1125] The device receives the translation result and displays it as an overlay in the user's field of view. The translation is displayed over the original text, so the user can visually recognize the translated content. For example, "Roast chicken" is displayed in the user's field of view.

[1126] Furthermore, the device can use a speech synthesis engine such as Amazon Polly or Google Text-to-Speech to play back the translation results as audio, if necessary, supporting not only visual translation but also auditory comprehension.

[1127] The device uses an emotion engine to recognize the user's emotional state. For example, it uses technologies such as Affectiva and IBM Watson Emotional Analysis to determine the user's emotional state based on their facial expressions, heart rate, and voice tone. If the user is feeling stressed or anxious, this information will be detected by the emotion engine.

[1128] Finally, the device will adjust the way translation results are displayed based on the recognized emotion. For example, if the user is feeling anxious, the device will display the translation results in a larger font size and play a voice message, making the user feel more comfortable receiving the translation results.

[1129] Specific examples

[1130] Take the specific example of a user looking at a menu at a restaurant in France.

[1131] 1. A user looks at a restaurant menu in French.

[1132] 2. The device uses its camera to capture a video frame of the menu.

[1133] 3. The device uses OCR to extract the text "Poulet rôti".

[1134] 4. The device uses an eye-tracking sensor to determine the user's point of gaze and focuses on "Poulet rôti."

[1135] 5. The device sends the text data "Poulet rôti" to the translation server.

[1136] 6. The server receives the request and uses an AI translation engine to translate "Poulet rôti" into English "Roast chicken."

[1137] 7. The server sends the translation result "Roast chicken" to the terminal.

[1138] 8. The device overlays the translation result "Roast chicken" on the user's field of view.

[1139] 9. The device uses an emotion engine to recognize the user's emotions, and if the user indicates anxiety, the translation result will be displayed in a larger font size and played aloud.

[1140] Example prompts for generative AI models

[1141] "Video Capture": The smart glasses' camera captures the video of the user's field of vision. The captured video is analyzed by the OCR engine to extract text data. The eye-tracking sensor identifies the user's point of gaze and sends the key text to the translation server. The server generates the translation results, which the device overlays in the user's field of vision. If necessary, the speech synthesis engine plays the translation results aloud. Additionally, the emotion engine recognizes the user's emotional state and adjusts the display accordingly.

[1142] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1143] Step 1: Capture a visual image

[1144] The device constantly monitors the user's field of view. As input, the camera captures live images of the scenery and objects the user is looking at. This captured image is sent to the next step, where it is processed for text extraction. Specifically, the camera mounted on the smart glasses captures images of what is in front of the user's eyes in real time.

[1145] Step 2: OCR processing

[1146] The device inputs the captured video frames into an OCR engine. The input is the captured video from step 1. The OCR engine (e.g., Google Cloud Vision or Tesseract) extracts text data from this video. Specifically, the OCR engine analyzes the image and converts the characters into digital text. The output of this step is the extracted text data (e.g., "Poulet rôti").

[1147] Step 3: Eye tracking

[1148] The device uses an eye-tracking sensor to identify the user's gaze focus point. The input requires data capturing the user's eye movements in real time. The eye-tracking device (e.g., Tobii Eyetracker) analyzes this data and identifies the text area the user is most focused on. Specifically, the eye-tracking sensor tracks the position and movement of the user's pupils, detecting the gaze direction and focus point. The output of this step is text data with the user's gaze point identified.

[1149] Step 4: Submit a translation request

[1150] The device sends the extracted text data to the translation server. The input includes the text data extracted in step 2 and the user's language preference data (e.g., French to English translation). The request is sent to the translation server, and the text data is ready for translation. Specifically, the device sends the text and language preference over the network to the translation server. The output of this step is the translation request data sent to the translation server.

[1151] Step 5: Translation process

[1152] The server processes the received text data with an AI translation engine. The input includes the text data sent in step 4 and the user's language preference data. The AI ​​translation engine (e.g., Google Translate API or Microsoft Translator API) converts the original text into the target language. Specifically, the AI ​​analyzes the text and generates the optimal translation result. The output of this step is the translated text data (e.g., "Roast chicken").

[1153] Step 6: Send the translation

[1154] The server sends the translation results it has generated to the device. The input includes the translated text data generated in step 5. The server sends this data to the device over the network. Specifically, the server converts the translation results into an appropriate format and uses data compression and caching techniques to transmit them efficiently. The output of this step is the translated text data received by the device.

[1155] Step 7: Overlaying the translation results

[1156] The device overlays the translation result in the user's field of view. The input includes the translated text data received in step 6. Specifically, the device overlays the translated text on the original field of view, allowing the user to visually recognize the translation result. The output of this step is the translated text (e.g., "Roast chicken") overlaid in the user's field of view.

[1157] Step 8: Playback by Text-to-Speech Engine

[1158] The device uses a speech synthesis engine, if necessary, to play back the translation result. The input includes the translation text data received in step 6. The speech synthesis engine (e.g., Amazon Polly or Google Text-to-Speech) converts the text data into speech data and plays it back. Specifically, this process generates speech data from the text and plays it back through a speaker. The output of this step is a speech playback of the translation result.

[1159] Step 9: Emotion Recognition

[1160] The device uses an emotion engine to recognize the user's emotional state. Inputs include data such as the user's facial expressions, heart rate, and voice tone. The emotion recognition engine (e.g., Affectiva or IBM Watson Emotional Analysis) analyzes this data to determine the user's emotional state. Specifically, the process involves analyzing data acquired by sensors to detect emotions such as stress or anxiety. The output of this step is the recognized user's emotional state.

[1161] Step 10: Adjust how the translation results are displayed

[1162] The device adjusts the display of the translation result based on the recognized emotion. The input includes the user's emotional state recognized in step 9 and the translated text data obtained in steps 6 or 7. Specific operations include displaying the translation result in a larger font size and adjusting the volume of the audio playback. The output of this step is a display of the translation result optimized for the user's emotional state.

[1163] (Application example 2)

[1164] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1165] Language barriers still exist in many situations today, posing a major problem when it comes to communication at tourist destinations and brick-and-mortar stores. Travelers and those who are not fluent in foreign languages ​​often have difficulty understanding information on local signs, menus, and guide boards. Furthermore, there is a lack of appropriate and quick ways to obtain information in emergencies and stressful situations.

[1166] The specification process by the specification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes an imaging device for monitoring the user's field of view, a means for capturing an object visually perceived by the user, an optical character recognition (OCR) engine for extracting character data from the captured video frame, an eye-tracking sensor for tracking the user's gaze and identifying the most gazed-at character area, a means for transmitting the extracted character data to a translation server and making a translation request, a means for overlaying the translation result received from the translation server on the user's field of view, a speech synthesis engine for playing the translation result aloud as needed, and an emotion engine for recognizing the user's emotional state and adjusting the display method. This makes it possible to understand information in different languages ​​in real time and provide the information in an optimal format depending on the user's emotional state.

[1167] The "user's visual field image" refers to the image range within the visual field that the user can perceive with normal vision.

[1168] An "imaging device" is an electronic device for capturing images, such as a camera.

[1169] A "capturing means" is a device or technology that records visually perceived objects in digital form.

[1170] An "optical character recognition (OCR) engine" is software for extracting text information from images and videos.

[1171] An "eye tracking sensor" is a sensor for tracking the direction and focus of a user's gaze.

[1172] "Character data" refers to the text information extracted by the OCR engine.

[1173] A "translation server" is a server that translates input character data into another specified language.

[1174] The "means for making a translation request" is a communication means for transmitting the captured character data and a translation request to the server.

[1175] "Means for overlay display" refers to a technique for displaying translation results superimposed within the user's field of vision.

[1176] A "speech synthesis engine" is software that converts text data into speech output.

[1177] An "emotion engine" is software that recognizes a user's emotional state from their facial expressions and biometric information.

[1178] "Tourist attractions and brick-and-mortar stores" refer to places that users actually visit and stores where sales activities are carried out.

[1179] The "function for adjusting the display method" is a technology for changing the display format of presented information depending on the emotional state of the user.

[1180] The present invention provides a system for removing language barriers at tourist spots and brick-and-mortar stores. The system operates based on smart glasses worn by users and includes the following main software and hardware components:

[1181] Specifically, the following hardware and software are used:

[1182] Smart glasses: imaging device, eye tracking sensor, display

[1183] OCR engine: Tesseract OCR

[1184] Translation server: Google Cloud Translation API

[1185] Speech synthesis engine: Amazon Polly

[1186] Emotion engine: Affectiva SDK

[1187] Program processing explanation

[1188] Camera Capture

[1189] When a user looks at tourist information, signs, menus, etc. through smart glasses, the camera built into the glasses captures images within their field of view, enhancing the user's visual experience and providing real-time information.

[1190] OCR processing

[1191] The captured video is analyzed by the Tesseract OCR engine to extract text data from the image, converting the visual information into digital text.

[1192] Eye tracking

[1193] An eye-tracking sensor monitors the user's gaze and identifies the text area the user is most focused on, allowing the user to prioritize the information that is most important to them, even when multiple pieces of text are present.

[1194] Translation Request

[1195] The extracted text data is sent to a translation server, where it is translated into the specified target language in real time using the Google Cloud Translation API. The request parameters include the source language, target language, and the extracted text data.

[1196] Displaying translation results

[1197] The translation results received from the translation server are overlaid on the smart glasses display, allowing users to simultaneously view the original text and the translated text, facilitating quick understanding.

[1198] Audio playback

[1199] If necessary, the translation result is synthesized into speech using Amazon Polly and played back to the user, allowing them to confirm the information audibly as well as visually.

[1200] emotion recognition

[1201] Using the Affectiva SDK, the app analyzes the user's facial expressions and biometrics to recognize their current emotional state. Based on this, the way translation results are displayed will be automatically adjusted. For example, if the user is stressed, the translation will be displayed in a larger font or played aloud.

[1202] Specific examples

[1203] For tourist information:

[1204] "Loads the description of a Universal Studios attraction, translates it from Japanese to English, and displays it on the smart glasses."

[1205] For restaurant menus:

[1206] "Italian restaurant menus can be translated from Italian to Japanese and displayed on smart glasses, with voice guidance available if necessary."

[1207] This allows users to smoothly understand information and enjoy a comfortable experience even in different language environments.

[1208] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1209] Step 1:

[1210] The user wears the smart glasses and can see objects within their field of view at tourist spots or brick-and-mortar stores.

[1211] Input: User's field of view (2D image)

[1212] Output: Visual field video data (image frames)

[1213] How it works: The imaging device built into the smart glasses captures the user's field of view in real time with a camera and saves it as image data.

[1214] Step 2:

[1215] The video frames captured by the device are input into an OCR engine, which extracts character data from the images.

[1216] Input: Video frame (image data)

[1217] Output: Character data (text)

[1218] How it works: The Tesseract OCR engine processes the captured video data, optically recognizing characters in the video and extracting them as text data.

[1219] Step 3:

[1220] The device uses an eye-tracking sensor to identify where the user's gaze is focused.

[1221] Input: Text data, user gaze information

[1222] Output: Character data (text) that the user is gazing at

[1223] Movement: The eye-tracking sensor monitors the user's gaze in real time and identifies areas of text that the user is particularly focused on. The text data in the identified areas is then filtered.

[1224] Step 4:

[1225] The terminal transmits the extracted and identified character data to a translation server and makes a translation request.

[1226] Input: Character data (text) that the user is gazing at, translation request (original text and language setting)

[1227] Output: Translated character data (text)

[1228] How it works: The device sends a translation request to the Google Cloud Translation API and gets the text result translated into the specified target language.

[1229] Step 5:

[1230] The translation results acquired by the server are sent to the terminal and displayed as an overlay in the user's field of view.

[1231] Input: Translated character data (text)

[1232] Output: Translation results displayed in the user's field of view (overlay text)

[1233] How it works: The translation results are received from the translation server and displayed on the smart glasses display, overlaid on the original text.

[1234] Step 6:

[1235] If necessary, the device will play back the translation results aloud using a speech synthesis engine.

[1236] Input: Translated character data (text)

[1237] Output: Audio output (audio of translation results)

[1238] How it works: Amazon Polly is used to convert the translated text into speech and play it through the smart glasses' speakers.

[1239] Step 7:

[1240] The device uses an emotion engine to recognize the user's emotional state and adjust how the translation results are displayed.

[1241] Input: User's facial expression data, biometric information, translation result (text)

[1242] Output: Adjusted display (font size, color, etc.), audio playback

[1243] How it works: The Affectiva SDK analyzes the user's facial expressions and heart rate to identify emotional states such as stress or anxiety. Based on the identified emotional state, the system adjusts the way translation results are displayed and the audio playback. For example, if the user is feeling stressed, the system will display the translation results in a larger font and play the audio to emphasize it.

[1244] This is the specific processing flow of the system, which allows users to smoothly obtain information even in a multilingual environment, providing a more comfortable experience.

[1245] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1246] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1247] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1248] [Fourth embodiment]

[1249] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1250] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1251] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1252] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1253] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1254] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1255] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1256] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1257] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1258] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1259] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1260] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1261] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1262] The present invention relates to a smart glasses system worn by a user, which includes a camera, an optical character recognition (OCR) engine, an eye tracking sensor, a translation server, a translation result overlay display function, and a speech synthesis engine.

[1263] Program processing explanation

[1264] Text Capture

[1265] The device constantly monitors the user's field of vision, capturing the scenery, signs, menus, and other things the user sees through the smart glasses. For example, if the user spots a sign written in a foreign language, the camera will capture that object.

[1266] OCR (Optical Character Recognition)

[1267] The image frames captured by the device are fed into an OCR engine to extract text data, identifying text segments written by the user in different languages ​​and extracting them as digital character data.

[1268] Eye tracking

[1269] The device uses an eye-tracking sensor to identify the area where the user is focusing their gaze, allowing it to prioritize the text the user is most focused on, even in situations where there are multiple text areas.

[1270] Submit a translation request

[1271] The device sends the extracted text data to a translation server. The translation request includes the original text data as well as the user's language preference. For example, a request is made to translate a Japanese sign into English.

[1272] Translation Processing

[1273] The server processes the received text data using a translation engine. The original text is converted into high-quality translated text based on the target language set by the user. The server then processes the text and returns the translation results to the device.

[1274] Sending translation results

[1275] The translation results generated by the server are sent to the terminal via the network. Data compression and caching techniques may be used to ensure user visibility.

[1276] Displaying translation results

[1277] The device receives the translation result and displays it as an overlay in the user's field of vision. The translation result is displayed visually on top of the original text, allowing the user to instantly understand the meaning. For example, the sign "Exit" that the user is looking at is displayed as the translation result "Exit."

[1278] Audio support (optional)

[1279] The device can optionally use a speech synthesis engine to play back the translation results aloud, allowing users to hear the information even when they cannot see.

[1280] Specific examples

[1281] As an example, consider the situation where a user is looking at a menu in a French restaurant.

[1282] 1. A user looks at a restaurant menu in French.

[1283] 2. The device uses its camera to capture a video frame of the menu.

[1284] 3. The device uses OCR to extract the text "Poulet rôti".

[1285] 4. The device uses an eye-tracking sensor to determine the user's point of gaze and focuses on "Poulet rôti."

[1286] 5. The device sends the text data "Poulet rôti" to the translation server.

[1287] 6. The server receives the request and uses an AI translation engine to translate "Poulet rôti" into English "Roast chicken."

[1288] 7. The server sends the translation results to the device.

[1289] 8. The device overlays the translation result "Roast chicken" on the user's field of view.

[1290] 9. If necessary, the device plays "Roast chicken" aloud, providing both visual and auditory confirmation of the information to the user.

[1291] In this way, the present invention provides a system that helps users understand text in different languages ​​in real time, thereby removing language barriers and enabling smooth communication and information access.

[1292] The processing flow will be explained below.

[1293] Step 1:

[1294] A user puts on the smart glasses and looks at text that comes into their field of vision. For example, the user sees a sign written in French.

[1295] Step 2:

[1296] The device activates its built-in camera and captures a video of the user's field of view, capturing a video frame of the sign the user is looking at.

[1297] Step 3:

[1298] The device inputs the captured video frame into the OCR engine, which extracts text information from the video. The text "Sortie" on the sign the user is looking at is extracted.

[1299] Step 4:

[1300] The device uses an eye-tracking sensor to track the user's gaze and identify the text area that receives the most attention. It detects that the gaze is focused on the "Sortie" section.

[1301] Step 5:

[1302] The device sends the extracted text data "Sortie" to the translation server, along with the user's language setting (e.g., French to English).

[1303] Step 6:

[1304] The server receives the translation request and passes the text data "Sortie" to the AI ​​translation engine, which translates "Sortie" as "Exit."

[1305] Step 7:

[1306] The server sends the generated translation result "Exit" to the terminal.

[1307] Step 8:

[1308] The translation result "Exit" received by the device is overlaid on the user's field of view, with "Exit" displayed over the original "Sortie" part.

[1309] Step 9:

[1310] (Optional) The device uses a speech synthesis engine to play the translation result "Exit" aloud, allowing the user to confirm the translation result both visually and audibly.

[1311] In this way, users can understand texts in different languages ​​in real time.

[1312] Example 1

[1313] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1314] In conventional smart glasses systems, the technology to translate and display visual information in real time is not fully developed. As a result, users often have difficulty understanding visual information when confronted with displays in a different language in a foreign country. Furthermore, existing systems do not identify gaze points using eye tracking or integrate visual information translation and voice support, making it difficult to achieve real-time information access and smooth communication.

[1315] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1316] In this invention, the server includes an image capture device for monitoring a user's field of view, a means for capturing an object visually perceived by the user, an optical character recognition (OCR) engine for extracting character data from the captured image frame, an eye-tracking sensor for tracking the user's gaze and identifying the most gazed character area, a means for transmitting the extracted character data to a translation processing device and making a translation request, and a means for overlaying the translation result received from the translation processing device on the user's field of view. This makes it easier for users to understand text written in different languages ​​in real time, and the integration of visual and audio information enables smooth communication and information access.

[1317] An "image capture device" is a device that monitors the user's field of view and captures the object that the user visually perceives.

[1318] An "Optical Character Recognition (OCR) engine" is an engine for extracting character data from captured video frames.

[1319] The "eye tracking sensor" is a sensor that tracks the user's gaze and identifies the text area that is most closely watched.

[1320] The "translation processing device" is a device that transmits extracted character data to a translation server and makes a translation request.

[1321] The "means for overlaying and displaying in the user's field of view" refers to a means for displaying the translation result received from the translation server in an overlaid manner in the user's field of view.

[1322] "User language setting information" is setting information related to the language used by the user and the language to be translated.

[1323] A "voice synthesizer" is a device that converts character data into voice data and plays it back aloud.

[1324] The present invention relates to a system that translates text in a different language that a user visually perceives in real time and overlays it on the user's field of vision. The system includes an image capture device, an optical character recognition (OCR) engine, an eye-tracking sensor, a translation processor, and a speech synthesizer.

[1325] Image acquisition device

[1326] The device constantly monitors the user's field of view. A high-resolution camera captures the scenery and text the user sees through the smart glasses in real time. This camera captures video frames and stores them in a buffer.

[1327] Optical Character Recognition (OCR) engine

[1328] The video frames captured by the device are input into an OCR engine (e.g., Tesseract OCR) to extract text data. The OCR engine analyzes the text data from the image data and extracts it as digital text data.

[1329] Eye tracking sensor

[1330] The device uses an eye-tracking sensor to track the user's gaze, which identifies the specific text area the user is looking at and prioritizes the most-looked-at text even when multiple pieces of text are present.

[1331] Translation Processing Device

[1332] The device sends the extracted text data and the user's language preference information to a translation processing device. The translation request is sent over the network, and the translation processing device receives the request. This device inputs the received text data into a translation engine (e.g., Google Translate API) and translates it into the specified language.

[1333] Overlay of translation results

[1334] The server sends the translation results to the terminal via the network. The terminal displays the received translation results as an overlay in the user's field of view. The overlay display allows the user to check the translation results while viewing the original text.

[1335] speech synthesizer

[1336] If necessary, the device will launch a speech synthesis engine (e.g., Google Text-to-Speech API) and play back the generated translation aloud. This speech playback feature allows users to hear information aloud even in situations where visual confirmation is difficult.

[1337] Specific examples

[1338] As an example, consider the situation where a user is looking at a menu in a French restaurant.

[1339] 1. A user looks at a restaurant menu in French.

[1340] 2. The device uses its camera to capture a video frame of the menu.

[1341] 3. The device uses OCR to extract the text "Poulet rôti".

[1342] 4. The device uses an eye-tracking sensor to determine the user's point of gaze and focuses on "Poulet rôti."

[1343] 5. The terminal sends the text data "Poulet rôti" to the translation processing device.

[1344] 6. The server receives the request and uses an AI translation engine to translate "Poulet rôti" into English "Roast chicken."

[1345] 7. The server sends the translation results to the device.

[1346] 8. The device overlays the translation result "Roast chicken" on the user's field of view.

[1347] 9. If necessary, the device plays "Roast chicken" aloud, providing both visual and auditory confirmation of the information to the user.

[1348] Example prompt sentence:

[1349] "The user looks at the menu," "The device activates the camera," "The OCR engine extracts the text," "The eye-tracking sensor identifies the gaze point," "The translation processor performs the translation," "An overlay is displayed," and "A speech synthesis device plays back the audio."

[1350] The present invention is a system that allows users to easily understand text written in different languages ​​in real time, and enables smooth communication and information access by integrating visual and audio information.

[1351] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1352] Step 1:

[1353] The user wears the smart glasses and looks around at what comes into view. The user moves to capture the text they see. The input is the user's line of sight. The output is the image of the view that comes into the camera's capture range.

[1354] Specific behavior:

[1355] The object the user wants to see is placed in the field of view of the smart glasses.

[1356] The device's built-in camera activates and captures the user's field of view.

[1357] Step 2:

[1358] The device captures the field of view with a high-resolution camera. The input is the field of view captured by the camera. The output is a video frame stored in a buffer.

[1359] Specific behavior:

[1360] The device continuously captures video at a specific frame rate (e.g., 30 frames per second).

[1361] The device stores the captured video frames in a buffer.

[1362] Step 3:

[1363] The device inputs video frames into an optical character recognition (OCR) engine to extract character data. The input is the video frames stored in a buffer. The output is the character data in digital form.

[1364] Specific behavior:

[1365] The device launches an OCR engine (e.g., Tesseract OCR) and provides the captured video frames as input data.

[1366] The OCR engine analyzes the text data from the image data, extracts the digital character data, and returns it to the terminal.

[1367] Step 4:

[1368] The device uses an eye-tracking sensor to track the user's gaze. The input is the user's gaze. The output is information about the specific text region being gazed at.

[1369] Specific behavior:

[1370] The device's built-in eye tracking sensor monitors the user's eye movements in real time.

[1371] The device analyzes the gaze data and identifies the text area the user is gazing at.

[1372] Step 5:

[1373] The terminal transmits the extracted text data and the user's language preference information to the translation processing device. The input is the digital text data and the language preference information. The output is the generation and transmission of a translation request.

[1374] Specific behavior:

[1375] The terminal sends data to the translation processing device in the form of an HTTP request (e.g., REST API).

[1376] The terminal generates a translation request including digital text data and the user's language setting information, and transmits the request over the network.

[1377] Step 6:

[1378] The server receives a translation request and translates the text data using a translation engine. The input is the data in the translation request. The output is the translated text.

[1379] Specific behavior:

[1380] The server parses the request data and passes it to a translation engine (e.g., Google Translate API).

[1381] The translation engine translates the original text into the specified language, generates the translation result, and returns it to the server.

[1382] Step 7:

[1383] The server generates translation results and sends them to the terminal. The input is the translated text data. The output is the translation results sent to the terminal.

[1384] Specific behavior:

[1385] The server packages the translation results in JSON format and sends them to the device.

[1386] The server uses data compression techniques (e.g., GZIP) to minimize the data transmitted.

[1387] Step 8:

[1388] The translation results received by the terminal are overlaid on the user's field of view. The input is the translation results received from the server. The output is the translation results displayed in the user's field of view.

[1389] Specific behavior:

[1390] After the device receives the translation result, it will be displayed by properly arranging it in the original text area.

[1391] The terminal dynamically renders the translated text displayed in the user's field of view.

[1392] Step 9:

[1393] The device activates a speech synthesis engine as needed and plays back the translation result aloud. The input is the text data of the translation result. The output is the audio that is played back to the user.

[1394] Specific behavior:

[1395] The device generates voice data using a speech synthesis engine (e.g., Google Text-to-Speech API).

[1396] The device plays the generated audio data from the speaker.

[1397] This series of processes enables users to understand text in different languages ​​in real time, and enables smooth communication and information access through the integration of visual and audio information.

[1398] (Application example 1)

[1399] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1400] Passengers in self-driving vehicles face challenges when traveling to different cultural regions, such as Japan, where rapid and accurate understanding of traffic signs and information boards in different languages ​​is a major challenge. In particular, not being able to understand place names and directions while driving or in a moving vehicle can lead to confusion and misunderstanding. Current technology is limited in its ability to translate text in real time and provide visual and auditory translations, creating a need for a system that can address this issue.

[1401] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1402] In this invention, the server includes a camera for monitoring the user's field of view, a means for capturing objects visually perceived by the user, an optical character recognition (OCR) engine for extracting text information from the captured video frames, an eye-tracking sensor for tracking the user's gaze and identifying the text area most gazed at, a means for sending the extracted text information to a translation service and making a translation request, a means for overlaying the translation result received from the translation service on the user's field of view, a speech synthesis engine for playing back the translation result as voice, and a means for recognizing, translating, and displaying signs in different languages ​​in real time while the user is in a moving vehicle. This enables passengers to quickly understand signs and information boards in different languages ​​both visually and audibly.

[1403] A "camera" is a device used to capture an image of a user's field of view.

[1404] An "Optical Character Recognition (OCR) engine" is software for extracting textual information from captured video frames.

[1405] An "eye tracking sensor" is a device that detects the user's gaze and identifies the area where the gaze is most focused.

[1406] A "translation service" is an online service or server that converts extracted text information into a specified language.

[1407] A "translation request" is a data transmission requesting a translation that includes the user's language setting and extracted text information.

[1408] "Overlay display" is a technology that displays translated text information superimposed on the user's field of vision.

[1409] A "speech synthesis engine" is software that converts text information into speech and provides it to the user audibly.

[1410] "Moving vehicle" means a vehicle, including automated public transport and other means of transportation.

[1411] "Signs" refer to guide signs and warning signs installed on roads and in public places.

[1412] This invention is a system using smart glasses worn by users that allows users to recognize signs and information boards in different languages ​​in real time and provides translation results both visually and audibly, making it easier for users to understand information in cross-cultural environments.

[1413] Hardware and software used

[1414] Smart glasses: Equipped with a camera, eye tracking sensor, and a see-through display.

[1415] Server: An online system for performing translation services.

[1416] Software: OCR engines (e.g., Tesseract OCR), translation APIs (e.g., Google Translate API), and speech synthesis engines.

[1417] Processing flow

[1418] The system performs the following steps:

[1419] Image Capture

[1420] While a user wears smart glasses and moves around in an autonomous vehicle, a camera captures signs and guidance displays that come into view. The camera monitors the user's field of view in real time.

[1421] OCR processing

[1422] The server inputs the captured video frames into an OCR engine to extract the text information. The optical character recognition (OCR) engine converts the text information in the captured frames into digital data with high accuracy.

[1423] Eye tracking

[1424] The server uses an eye-tracking sensor to detect the user's gaze and identify the text area where the gaze is most focused, allowing important information to be prioritized even when multiple text areas are present.

[1425] Text Translation

[1426] The server sends the extracted text to a translation service, requesting a translation. The translation service also includes the user's language preference data and translates the text into the specified language, such as Japanese to English.

[1427] Display translation results and play audio

[1428] The user sees the translation results overlaid on the smart glasses' transparent display, and the translation results are also played back as audio by a speech synthesis engine, allowing the user to confirm the information both visually and audibly.

[1429] Specific examples

[1430] For example, consider a case where a user finds a Japanese sign indicating "next station" in a self-driving vehicle.

[1431] 1. The user puts on the smart glasses and brings the sign into view.

[1432] 2. The camera captures video of the sign.

[1433] 3. The server uses an OCR engine to extract the text "Next Station."

[1434] 4. The server uses an eye-tracking sensor to analyze the user's gaze and confirm that their gaze is focused on the "next station."

[1435] 5. The extracted text data is sent to a translation service, and a request is made to translate it into the user's preferred language (e.g., English).

[1436] 6. The translation service translates "next station" as "Next Station" and sends it back to the server.

[1437] 7. The server overlays the translation result, "Next Station," on the transparent display of the smart glasses, and simultaneously plays "Next Station" using the speech synthesis engine.

[1438] This allows the user to immediately understand visually and audibly that "next station" is "Next Station" in English.

[1439] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1440] Step 1:

[1441] The user wears smart glasses and uses a camera to capture signs and guide displays in their field of vision. The device's camera captures the image in the user's field of vision in real time and acquires the image frames. The input is the image frames captured by the camera, and the captured image data is obtained as the output.

[1442] Step 2:

[1443] The server inputs the captured video frame into an OCR engine to extract the text information. The server acquires the digital data of the video frame and uses an OCR engine (e.g., Tesseract OCR) to analyze the text information in the image. The input is the video data, and the output is the recognized text data.

[1444] Step 3:

[1445] The server uses an eye-tracking sensor to detect the user's gaze and identify the text area where the gaze is most focused. The server analyzes the gaze data obtained from the eye-tracking sensor and determines the location where the user is most focused. The input is the gaze data and recognized character data, and the output is the character area corresponding to the gaze point.

[1446] Step 4:

[1447] The server sends the extracted text information to a translation service and makes a translation request. Based on the user's language settings, the server sends a request to a translation API (e.g., Google Translate API) to translate the recognized text information into the target language. The input is the text data corresponding to the gaze point and language setting data, and the output is the completion of sending a translation request.

[1448] Step 5:

[1449] The translation service receives the request and returns the translation result into the specified language to the server. The server receives the translated data returned from the translation API. The input is the translation request, and the translated text data is returned to the server as the output.

[1450] Step 6:

[1451] The server sends the translation results to the smart glasses, which overlay the translation results on the user's field of view. The server then sends instructions to overlay the translated text data on the smart glasses' transparent display. The input is the translated text data, and the output is an overlay display on the user's field of view.

[1452] Step 7:

[1453] The server uses a speech synthesis engine to play back the translation results as audio. The server inputs the translated text data into the speech synthesis engine to generate audio data. The generated audio data is played back through the smart glasses' speakers. The input is the translated text data, and the output is audio that is generated and played back so that the user can hear it.

[1454] This series of processes allows users to understand signs and guidance displays in different languages ​​in real time.

[1455] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1456] The present invention relates to a smart glasses system worn by a user, which includes a camera, an optical character recognition (OCR) engine, an eye tracking sensor, a translation server, a translation result overlay display function, a speech synthesis engine, and an emotion engine.

[1457] Program processing explanation

[1458] Text Capture

[1459] The device constantly monitors the user's field of vision, capturing images of signs, menus, books, and other items the user is looking at through the smart glasses. For example, if the user comes across a menu written in a foreign language, the device will capture a video frame of that menu.

[1460] OCR (Optical Character Recognition)

[1461] The video frame captured by the device is input into the OCR engine to extract text data. The text "Poulet rôti" in the menu the user is viewing is extracted using OCR.

[1462] Eye tracking

[1463] The device uses an eye-tracking sensor to identify the area where the user's gaze is focused, allowing it to prioritize the text the user is most focused on, even when there are multiple text areas.

[1464] Submit a translation request

[1465] The terminal sends the extracted text data to a translation server. The translation request includes the original text data as well as the user's language preference, for example, a translation request from French to English.

[1466] Translation Processing

[1467] The server processes the received text data using an AI translation engine. The original text is converted into high-quality translated text based on the target language set by the user. The server then processes the text and sends the translation results back to the device.

[1468] Sending translation results

[1469] The translation results generated by the server are sent to the terminal via the network. Data compression and caching techniques may be used to ensure user visibility.

[1470] Displaying translation results

[1471] The device receives the translation result and displays it as an overlay in the user's field of view. The translation result is displayed visually on top of the original text, allowing the user to instantly understand the meaning. For example, "Poulet rôti" can be displayed as "Roast chicken."

[1472] Audio Support

[1473] The device can optionally use a speech synthesis engine to play back the translation results aloud, allowing users to hear the information even when they cannot see.

[1474] emotion recognition

[1475] The device uses the emotion engine to recognize emotions from the user's facial expressions, voice tone, heart rate, etc. For example, if the user is feeling stressed or anxious, this information will be detected by the emotion engine.

[1476] Adjusting the display method

[1477] The device will adjust how the translation results are displayed based on the perceived emotion: for example, if the user is feeling anxious, the translation results will be displayed in a larger font size and played aloud.

[1478] Specific examples

[1479] As an example, consider the situation where a user is looking at a menu in a French restaurant.

[1480] 1. A user looks at a restaurant menu in French.

[1481] 2. The device uses its camera to capture a video frame of the menu.

[1482] 3. The device uses OCR to extract the text "Poulet rôti".

[1483] 4. The device uses an eye-tracking sensor to determine the user's point of gaze and focuses on "Poulet rôti."

[1484] 5. The device sends the text data "Poulet rôti" to the translation server.

[1485] 6. The server receives the request and uses an AI translation engine to translate "Poulet rôti" into English "Roast chicken."

[1486] 7. The server sends the translation result "Roast chicken" to the terminal.

[1487] 8. The device overlays the translation result "Roast chicken" on the user's field of view.

[1488] 9. The device uses an emotion engine to recognize the user's emotions, and if they indicate anxiety, it displays the translation results in a larger font size and plays them aloud.

[1489] In this way, the present invention provides a system that helps users understand text in different languages ​​in real time, thereby removing language barriers and enabling smooth communication and information access.

[1490] The processing flow will be explained below.

[1491] Step 1:

[1492] A user puts on the smart glasses and looks at text that comes into their field of view. For example, the user sees a menu written in French.

[1493] Step 2:

[1494] The device activates the built-in camera and captures the user's field of view, capturing a frame of the menu.

[1495] Step 3:

[1496] The device inputs the captured video frame into the OCR engine to extract the text data. The text "Poulet rôti" is extracted by OCR.

[1497] Step 4:

[1498] The device uses an eye-tracking sensor to track the user's gaze and identify the text area that receives the most attention. It detects that the gaze is focused on the word "Poulet rôti."

[1499] Step 5:

[1500] The device sends the extracted text data "Poulet rôti" to the translation server, along with the user's language setting (from French to English).

[1501] Step 6:

[1502] The server receives the translation request and inputs the text data "Poulet rôti" into the AI ​​translation engine, which translates the text as "Roast chicken."

[1503] Step 7:

[1504] The server sends the generated translation result "Roast chicken" to the terminal.

[1505] Step 8:

[1506] The device receives the translation result "Roast chicken" and displays it as an overlay in the user's field of view, overlaying "Poulet rôti" with "Roast chicken."

[1507] Step 9:

[1508] The device uses the emotion engine to recognize emotions from the user's facial expressions, voice tone, heart rate, etc. For example, if the user is feeling anxious, the emotion engine will detect this.

[1509] Step 10:

[1510] The device adjusts how translation results are displayed based on the emotions recognized by the emotion engine: if the user is feeling anxious, the translation results will be displayed in a larger font size and played aloud.

[1511] Step 11:

[1512] (Optional) The device uses a speech synthesis engine to play the translation result "Roast chicken" aloud, allowing the user to confirm the translation both visually and audibly.

[1513] For example, consider a scenario where a user is ordering a menu at a French restaurant. This process helps users understand text in different languages ​​in real time, providing them with peace of mind in situations where they may feel uneasy.

[1514] Example 2

[1515] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1516] Conventional translation systems require a lot of effort for users to visually understand information, making it difficult to understand text in different languages ​​in real time. Furthermore, because the display does not adjust according to the user's emotional state, they are unable to provide effective support when users are feeling stressed or anxious.

[1517] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a camera for monitoring the user's field of view, a means for capturing an object visually perceived by the user, an optical character recognition engine for extracting character data from the captured video frame, an eye-tracking device for tracking the user's gaze and identifying the most gazed character region, a means for transmitting the extracted character data to a translation device and making a translation request, a means for overlaying the translation result received from the translation device on the user's field of view, and an emotion recognition engine for recognizing the user's emotions and adjusting the display method based on the user's emotional state. This makes it easier to understand texts in different languages ​​in real time and enables effective support according to the user's emotional state.

[1518] "User" refers to a person who uses the system to obtain visual information and needs translation.

[1519] "Field of view image" refers to the image of the scenery or object seen by the user, captured through the camera in the smart glasses.

[1520] "Capture device" refers to a device such as a camera for capturing an image of a user's field of view.

[1521] "Optical character recognition engine" refers to software or hardware for extracting character data from captured video frames.

[1522] "Eye tracking device" refers to a sensor that monitors the movement of a user's eyes and identifies the area of ​​text that is most frequently looked at.

[1523] "Translation device" refers to a server or service for translating extracted character data into a specified target language.

[1524] "Overlay display" refers to a method of displaying the translation results superimposed on the user's field of vision, i.e., displaying them as additional information on the original field of vision image.

[1525] An "emotion recognition engine" is software or hardware for recognizing a user's emotional state, and refers to a device that analyzes emotions from facial expressions, vocal tone, heart rate, etc.

[1526] "Adjusting the display method" refers to the process of changing the display format of the translation result depending on the user's emotional state.

[1527] The present invention relates to a smart glasses system worn by a user, specifically a system that aims to translate and display text in different languages ​​in real time. The main hardware components of this system include a camera, an eye-tracking sensor, and a speech synthesis engine, and the software components include an optical character recognition (OCR) engine, a translation server, and an emotion recognition engine.

[1528] The device constantly monitors the user's field of vision using a camera, capturing signs, menus, books, and other items that come into the user's sight. For example, when a user looks at a restaurant menu, the device captures the menu frame by frame. The camera used here could be a versatile wearable camera.

[1529] The video frames captured by the device are input into an OCR engine to extract text data. Specifically, OCR engines such as Google Cloud Vision and Tesseract can be used. For example, the string "Poulet rôti" written in French can be extracted.

[1530] Next, the device uses an eye-tracking sensor to identify the user's gaze point. A specific example of an eye-tracking device is the Tobii Eyetracker. This allows the device to prioritize the text the user is looking at, even when there are multiple text areas. For example, it can detect when the user is looking at "Poulet rôti."

[1531] The device sends the extracted text data to a translation server. This translation request includes the original text data and the user's language setting (e.g., French to English). The server receives the translation request and processes it using an AI translation engine. For example, you can use the Google Translate API or Microsoft Translator API. The server translates "Poulet rôti" to "Roast chicken."

[1532] The translation results generated by the server are sent to the terminal via the network, and data compression and caching techniques may be used to maintain user visibility.

[1533] The device receives the translation result and displays it as an overlay in the user's field of view. The translation is displayed over the original text, so the user can visually recognize the translated content. For example, "Roast chicken" is displayed in the user's field of view.

[1534] Furthermore, the device can use a speech synthesis engine such as Amazon Polly or Google Text-to-Speech to play back the translation results as audio, if necessary, supporting not only visual translation but also auditory comprehension.

[1535] The device uses an emotion engine to recognize the user's emotional state. For example, it uses technologies such as Affectiva and IBM Watson Emotional Analysis to determine the user's emotional state based on their facial expressions, heart rate, and voice tone. If the user is feeling stressed or anxious, this information will be detected by the emotion engine.

[1536] Finally, the device will adjust the way translation results are displayed based on the recognized emotion. For example, if the user is feeling anxious, the device will display the translation results in a larger font size and play a voice message, making the user feel more comfortable receiving the translation results.

[1537] Specific examples

[1538] Take the specific example of a user looking at a menu at a restaurant in France.

[1539] 1. A user looks at a restaurant menu in French.

[1540] 2. The device uses its camera to capture a video frame of the menu.

[1541] 3. The device uses OCR to extract the text "Poulet rôti".

[1542] 4. The device uses an eye-tracking sensor to determine the user's point of gaze and focuses on "Poulet rôti."

[1543] 5. The device sends the text data "Poulet rôti" to the translation server.

[1544] 6. The server receives the request and uses an AI translation engine to translate "Poulet rôti" into English "Roast chicken."

[1545] 7. The server sends the translation result "Roast chicken" to the terminal.

[1546] 8. The device overlays the translation result "Roast chicken" on the user's field of view.

[1547] 9. The device uses an emotion engine to recognize the user's emotions, and if the user indicates anxiety, the translation result will be displayed in a larger font size and played aloud.

[1548] Example prompts for generative AI models

[1549] "Video Capture": The smart glasses' camera captures the video of the user's field of vision. The captured video is analyzed by the OCR engine to extract text data. The eye-tracking sensor identifies the user's point of gaze and sends the key text to the translation server. The server generates the translation results, which the device overlays in the user's field of vision. If necessary, the speech synthesis engine plays the translation results aloud. Additionally, the emotion engine recognizes the user's emotional state and adjusts the display accordingly.

[1550] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1551] Step 1: Capture a visual image

[1552] The device constantly monitors the user's field of view. As input, the camera captures live images of the scenery and objects the user is looking at. This captured image is sent to the next step, where it is processed for text extraction. Specifically, the camera mounted on the smart glasses captures images of what is in front of the user's eyes in real time.

[1553] Step 2: OCR processing

[1554] The device inputs the captured video frames into an OCR engine. The input is the captured video from step 1. The OCR engine (e.g., Google Cloud Vision or Tesseract) extracts text data from this video. Specifically, the OCR engine analyzes the image and converts the characters into digital text. The output of this step is the extracted text data (e.g., "Poulet rôti").

[1555] Step 3: Eye tracking

[1556] The device uses an eye-tracking sensor to identify the user's gaze focus point. The input requires data capturing the user's eye movements in real time. The eye-tracking device (e.g., Tobii Eyetracker) analyzes this data and identifies the text area the user is most focused on. Specifically, the eye-tracking sensor tracks the position and movement of the user's pupils, detecting the gaze direction and focus point. The output of this step is text data with the user's gaze point identified.

[1557] Step 4: Submit a translation request

[1558] The device sends the extracted text data to the translation server. The input includes the text data extracted in step 2 and the user's language preference data (e.g., French to English translation). The request is sent to the translation server, and the text data is ready for translation. Specifically, the device sends the text and language preference over the network to the translation server. The output of this step is the translation request data sent to the translation server.

[1559] Step 5: Translation process

[1560] The server processes the received text data with an AI translation engine. The input includes the text data sent in step 4 and the user's language preference data. The AI ​​translation engine (e.g., Google Translate API or Microsoft Translator API) converts the original text into the target language. Specifically, the AI ​​analyzes the text and generates the optimal translation result. The output of this step is the translated text data (e.g., "Roast chicken").

[1561] Step 6: Send the translation

[1562] The server sends the translation results it has generated to the device. The input includes the translated text data generated in step 5. The server sends this data to the device over the network. Specifically, the server converts the translation results into an appropriate format and uses data compression and caching techniques to transmit them efficiently. The output of this step is the translated text data received by the device.

[1563] Step 7: Overlaying the translation results

[1564] The device overlays the translation result in the user's field of view. The input includes the translated text data received in step 6. Specifically, the device overlays the translated text on the original field of view, allowing the user to visually recognize the translation result. The output of this step is the translated text (e.g., "Roast chicken") overlaid in the user's field of view.

[1565] Step 8: Playback by Text-to-Speech Engine

[1566] The device uses a speech synthesis engine, if necessary, to play back the translation result. The input includes the translation text data received in step 6. The speech synthesis engine (e.g., Amazon Polly or Google Text-to-Speech) converts the text data into speech data and plays it back. Specifically, this process generates speech data from the text and plays it back through a speaker. The output of this step is a speech playback of the translation result.

[1567] Step 9: Emotion Recognition

[1568] The device uses an emotion engine to recognize the user's emotional state. Inputs include data such as the user's facial expressions, heart rate, and voice tone. The emotion recognition engine (e.g., Affectiva or IBM Watson Emotional Analysis) analyzes this data to determine the user's emotional state. Specifically, the process involves analyzing data acquired by sensors to detect emotions such as stress or anxiety. The output of this step is the recognized user's emotional state.

[1569] Step 10: Adjust how the translation results are displayed

[1570] The device adjusts the display of the translation result based on the recognized emotion. The input includes the user's emotional state recognized in step 9 and the translated text data obtained in steps 6 or 7. Specific operations include displaying the translation result in a larger font size and adjusting the volume of the audio playback. The output of this step is a display of the translation result optimized for the user's emotional state.

[1571] (Application example 2)

[1572] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1573] Language barriers still exist in many situations today, posing a major problem when it comes to communication at tourist destinations and brick-and-mortar stores. Travelers and those who are not fluent in foreign languages ​​often have difficulty understanding information on local signs, menus, and guide boards. Furthermore, there is a lack of appropriate and quick ways to obtain information in emergencies and stressful situations.

[1574] The specification process by the specification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes an imaging device for monitoring the user's field of view, a means for capturing an object visually perceived by the user, an optical character recognition (OCR) engine for extracting character data from the captured video frame, an eye-tracking sensor for tracking the user's gaze and identifying the most gazed-at character area, a means for transmitting the extracted character data to a translation server and making a translation request, a means for overlaying the translation result received from the translation server on the user's field of view, a speech synthesis engine for playing the translation result aloud as needed, and an emotion engine for recognizing the user's emotional state and adjusting the display method. This makes it possible to understand information in different languages ​​in real time and provide the information in an optimal format depending on the user's emotional state.

[1575] The "user's visual field image" refers to the image range within the visual field that the user can perceive with normal vision.

[1576] An "imaging device" is an electronic device for capturing images, such as a camera.

[1577] A "capturing means" is a device or technology that records visually perceived objects in digital form.

[1578] An "optical character recognition (OCR) engine" is software for extracting text information from images and videos.

[1579] An "eye tracking sensor" is a sensor for tracking the direction and focus of a user's gaze.

[1580] "Character data" refers to the text information extracted by the OCR engine.

[1581] A "translation server" is a server that translates input character data into another specified language.

[1582] The "means for making a translation request" is a communication means for transmitting the captured character data and a translation request to the server.

[1583] "Means for overlay display" refers to a technique for displaying translation results superimposed within the user's field of vision.

[1584] A "speech synthesis engine" is software that converts text data into speech output.

[1585] An "emotion engine" is software that recognizes a user's emotional state from their facial expressions and biometric information.

[1586] "Tourist attractions and brick-and-mortar stores" refer to places that users actually visit and stores where sales activities are carried out.

[1587] The "function for adjusting the display method" is a technology for changing the display format of presented information depending on the emotional state of the user.

[1588] The present invention provides a system for removing language barriers at tourist spots and brick-and-mortar stores. The system operates based on smart glasses worn by users and includes the following main software and hardware components:

[1589] Specifically, the following hardware and software are used:

[1590] Smart glasses: imaging device, eye tracking sensor, display

[1591] OCR engine: Tesseract OCR

[1592] Translation server: Google Cloud Translation API

[1593] Speech synthesis engine: Amazon Polly

[1594] Emotion engine: Affectiva SDK

[1595] Program processing explanation

[1596] Camera Capture

[1597] When a user looks at tourist information, signs, menus, etc. through smart glasses, the camera built into the glasses captures images within their field of view, enhancing the user's visual experience and providing real-time information.

[1598] OCR processing

[1599] The captured video is analyzed by the Tesseract OCR engine to extract text data from the image, converting the visual information into digital text.

[1600] Eye tracking

[1601] An eye-tracking sensor monitors the user's gaze and identifies the text area the user is most focused on, allowing the user to prioritize the information that is most important to them, even when multiple pieces of text are present.

[1602] Translation Request

[1603] The extracted text data is sent to a translation server, where it is translated into the specified target language in real time using the Google Cloud Translation API. The request parameters include the source language, target language, and the extracted text data.

[1604] Displaying translation results

[1605] The translation results received from the translation server are overlaid on the smart glasses display, allowing users to simultaneously view the original text and the translated text, facilitating quick understanding.

[1606] Audio playback

[1607] If necessary, the translation result is synthesized into speech using Amazon Polly and played back to the user, allowing them to confirm the information audibly as well as visually.

[1608] emotion recognition

[1609] Using the Affectiva SDK, the app analyzes the user's facial expressions and biometrics to recognize their current emotional state. Based on this, the way translation results are displayed will be automatically adjusted. For example, if the user is stressed, the translation will be displayed in a larger font or played aloud.

[1610] Specific examples

[1611] For tourist information:

[1612] "Loads the description of a Universal Studios attraction, translates it from Japanese to English, and displays it on the smart glasses."

[1613] For restaurant menus:

[1614] "Italian restaurant menus can be translated from Italian to Japanese and displayed on smart glasses, with voice guidance available if necessary."

[1615] This allows users to smoothly understand information and enjoy a comfortable experience even in different language environments.

[1616] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1617] Step 1:

[1618] The user wears the smart glasses and can see objects within their field of view at tourist spots or brick-and-mortar stores.

[1619] Input: User's field of view (2D image)

[1620] Output: Visual field video data (image frames)

[1621] How it works: The imaging device built into the smart glasses captures the user's field of view in real time with a camera and saves it as image data.

[1622] Step 2:

[1623] The video frames captured by the device are input into an OCR engine, which extracts character data from the images.

[1624] Input: Video frame (image data)

[1625] Output: Character data (text)

[1626] How it works: The Tesseract OCR engine processes the captured video data, optically recognizing characters in the video and extracting them as text data.

[1627] Step 3:

[1628] The device uses an eye-tracking sensor to identify where the user's gaze is focused.

[1629] Input: Text data, user gaze information

[1630] Output: Character data (text) that the user is gazing at

[1631] Movement: The eye-tracking sensor monitors the user's gaze in real time and identifies areas of text that the user is particularly focused on. The text data in the identified areas is then filtered.

[1632] Step 4:

[1633] The terminal transmits the extracted and identified character data to a translation server and makes a translation request.

[1634] Input: Character data (text) that the user is gazing at, translation request (original text and language setting)

[1635] Output: Translated character data (text)

[1636] How it works: The device sends a translation request to the Google Cloud Translation API and gets the text result translated into the specified target language.

[1637] Step 5:

[1638] The translation results acquired by the server are sent to the terminal and displayed as an overlay in the user's field of view.

[1639] Input: Translated character data (text)

[1640] Output: Translation results displayed in the user's field of view (overlay text)

[1641] How it works: The translation results are received from the translation server and displayed on the smart glasses display, overlaid on the original text.

[1642] Step 6:

[1643] If necessary, the device will play back the translation results aloud using a speech synthesis engine.

[1644] Input: Translated character data (text)

[1645] Output: Audio output (audio of translation results)

[1646] How it works: Amazon Polly is used to convert the translated text into speech and play it through the smart glasses' speakers.

[1647] Step 7:

[1648] The device uses an emotion engine to recognize the user's emotional state and adjust how the translation results are displayed.

[1649] Input: User's facial expression data, biometric information, translation result (text)

[1650] Output: Adjusted display (font size, color, etc.), audio playback

[1651] How it works: The Affectiva SDK analyzes the user's facial expressions and heart rate to identify emotional states such as stress or anxiety. Based on the identified emotional state, the system adjusts the way translation results are displayed and the audio playback. For example, if the user is feeling stressed, the system will display the translation in a larger font and play the audio to emphasize it.

[1652] This is the specific processing flow of the system. This process allows users to smoothly obtain information even in a multilingual environment, providing a more comfortable experience.

[1653] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1654] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1655] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1656] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1657] FIG. 9 illustrates an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and behaviors arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1658] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1659] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1660] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1661] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1662] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1663] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1664] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1665] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1666] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1667] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1668] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1669] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1670] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1671] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1672] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1673] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1674] The following is further disclosed regarding the above embodiment.

[1675] (Claim 1)

[1676] a means for capturing an object visually perceived by the user, the means comprising a camera for monitoring an image of the user's field of view;

[1677] It features an Optical Character Recognition (OCR) engine for extracting text data from captured video frames.

[1678] It has an eye-tracking sensor that tracks the user's gaze and identifies the text area that receives the most attention;

[1679] a means for transmitting the extracted text data to a translation server and making a translation request;

[1680] means for overlaying and displaying the translation results received from the translation server in the user's field of view;

[1681] A system including:

[1682] (Claim 2)

[1683] 2. The system of claim 1, wherein the translation request includes a user's language setting data.

[1684] (Claim 3)

[1685] 2. The system according to claim 1, further comprising a speech synthesis engine for not only displaying the translation result overlaid in the user's field of view as text but also reproducing it as voice.

[1686] "Example 1"

[1687] (Claim 1)

[1688] means for capturing an object visually perceived by the user, the image capturing device being provided for monitoring the user's field of view;

[1689] It has an optical character recognition (OCR) engine to extract character data from captured video frames,

[1690] Equipped with an eye-tracking sensor that tracks the user's gaze and identifies the text area that is most frequently looked at,

[1691] means for transmitting the extracted character data to a translation processing device and making a translation request;

[1692] means for overlaying and displaying the translation result received from the translation processing device in the user's field of view;

[1693] A system including:

[1694] (Claim 2)

[1695] 2. The system according to claim 1, wherein the translation request includes language setting information of the user.

[1696] (Claim 3)

[1697] 2. The system according to claim 1, further comprising a speech synthesizer for not only displaying the translation result overlaid in the user's field of vision as text information but also reproducing it as voice.

[1698] "Application Example 1"

[1699] (Claim 1)

[1700] A means for capturing an object visually perceived by the user, the means comprising an imaging device for monitoring the user's field of view;

[1701] It has an optical character recognition (OCR) engine to extract text information from captured video frames.

[1702] Equipped with an eye-tracking sensor that tracks the user's gaze and identifies the text area that is most frequently looked at,

[1703] a means for transmitting the extracted text information to a translation service and making a translation request;

[1704] means for overlaying the translation results received from the translation service onto the user's field of view;

[1705] It is equipped with a speech synthesis engine that plays back the translation results aloud,

[1706] A means for recognizing, translating and displaying signs in different languages ​​in real time while the user is in a moving vehicle;

[1707] A system including:

[1708] (Claim 2)

[1709] 2. The system of claim 1, wherein the translation request includes a user's language setting data.

[1710] (Claim 3)

[1711] 2. The system according to claim 1, wherein the translation result overlaid on the user's field of view is not only displayed as text but also reproduced by voice.

[1712] "Example 2: Combining Emotion Engines"

[1713] (Claim 1)

[1714] A means for capturing an object visually perceived by the user, the means comprising an imaging device for monitoring the user's field of view;

[1715] an optical character recognition engine for extracting character data from the captured video frames;

[1716] A gaze tracking device is provided to track the gaze of a user and identify the character area most frequently gazed upon,

[1717] means for transmitting the extracted character data to a translation device and making a translation request;

[1718] means for overlaying and displaying the translation result received from the translation device in the user's field of view;

[1719] an emotion recognition engine for recognizing an emotion of a user and adjusting a display method based on the emotion state;

[1720] A system including:

[1721] (Claim 2)

[1722] 2. The system according to claim 1, wherein the translation request includes language data set by the user.

[1723] (Claim 3)

[1724] 2. The system according to claim 1, further comprising a speech synthesis engine for not only displaying the translation result overlaid in the user's field of view as text but also reproducing it as voice.

[1725] "Application example 2 when combining emotion engines"

[1726] (Claim 1)

[1727] a means for capturing an object visually perceived by the user, the means comprising an imaging device for monitoring an image of the user's field of view;

[1728] It has an optical character recognition (OCR) engine for extracting text data from captured video frames,

[1729] Equipped with an eye-tracking sensor that tracks the user's gaze and identifies the text area that is most frequently looked at,

[1730] a means for transmitting the extracted character data to a translation server and making a translation request;

[1731] means for overlaying and displaying the translation results received from the translation server in the user's field of view;

[1732] It is equipped with a speech synthesis engine that plays back the translation results aloud as needed.

[1733] Emotion engine to recognize the user's emotional state and adjust the display method;

[1734] A system including:

[1735] (Claim 2)

[1736] 2. The system of claim 1, wherein the translation request includes the user's language setting data.

[1737] (Claim 3)

[1738] The system according to claim 1, which has a function for translating text information recognized at tourist spots and brick-and-mortar stores into the user's target language and providing it visually and audibly. [Explanation of symbols]

[1739] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a means for capturing an object visually perceived by the user, the means comprising a camera for monitoring an image of the user's field of view; It features an Optical Character Recognition (OCR) engine for extracting text data from captured video frames. It has an eye-tracking sensor that tracks the user's gaze and identifies the text area that receives the most attention; a means for transmitting the extracted text data to a translation server and making a translation request; means for overlaying and displaying the translation results received from the translation server in the user's field of view; A system including:

2. 2. The system of claim 1, wherein the translation request includes a user's language preference data.

3. 2. The system according to claim 1, further comprising a speech synthesis engine for not only displaying the translation result overlaid on the user's field of view as text but also reproducing it aloud.

Images