System

Abstract

A system is provided.SOLUTION: A system comprising: means for detecting character strings within a field of view; means for extracting specific information from the detected character strings; means for translating the extracted information; means for displaying the translated information on a display device; and means for communicating with a server via a network for the extraction and translation of the information.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Understanding the local language while traveling or on business abroad is a major challenge for many people. In particular, using a smartphone to understand the text on signs and billboards requires stopping and operating the device, which can disrupt the smooth conduct of travel or business. Furthermore, there is a need for a means to quickly and accurately understand important information. [Means for solving the problem]

[0005] The present invention provides a system that detects character strings within a field of view, extracts specific information from them, and displays the translated information on a display device. This system includes a means for capturing character strings within the field of view in real time using a camera, a means for extracting necessary information from the captured character string, a means for translating the extracted information, and a means for displaying the translation results. Furthermore, multimodal artificial intelligence is used to filter information, eliminating unnecessary information. This allows users to obtain and understand necessary information in real time without operating a smartphone.

[0006] "Means for detecting character strings within the field of view" refers to a function that uses an input device such as a camera to obtain character data present within the user's field of view in real time.

[0007] "Means for extracting specific information from detected character strings" refers to the function of using optical character recognition (OCR) technology or multimodal artificial intelligence to identify and extract only the necessary information from the acquired character data.

[0008] "Means for translating extracted information" refers to a function for converting character data sent to a translation server via a network into another language.

[0009] "Means for displaying translated information on a display device" refers to the function of displaying translated text data on the display of smart glasses or a mobile device in accordance with the user's visual perception.

[0010] "Means for communicating with a server via a network for extracting and translating said information" refers to the communication function for a local device to send and receive data to and from a remote server on a network to perform information extraction and translation tasks.

[0011] "Multimodal AI" refers to AI that has the ability to integrate and analyze multiple information sources, such as visual, audio, and text, and filter out unnecessary information.

[0012] The term "display device" refers to a device such as a display that is mounted on a device carried by a user and that visually presents information. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0014] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0017] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0018] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0019] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0021] [First embodiment]

[0022] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0023] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0026] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0029] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0033] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0034] This invention relates to a system that detects characters in the field of view, extracts specific information from them, and displays the translated information on a display device. This system is realized by combining a camera, OCR technology, multimodal artificial intelligence, a translation server, a display device, etc.

[0035] This system works by having the user wear smart glasses. The smart glasses are equipped with a camera that captures images of the user's field of view in real time. This image data is then processed by a device inside the smart glasses.

[0036] The device uses OCR technology to detect text from the captured video, allowing for real-time capture of text information from signs and other objects. The detected text is then analyzed using multimodal artificial intelligence to filter out unnecessary information. This filtering process allows only the most important information to be extracted.

[0037] After extracting the important information, the device sends it to the server via the network. The server uses a translation API to translate the received string into the specified language. The translated information is then sent back to the device.

[0038] The device then displays the translated information on the smart glasses display, which is overlaid naturally within the user's field of vision, allowing the user to instantly check the information.

[0039] As a concrete example, consider the case where a user sees a restaurant sign while walking through a city abroad. The camera in the smart glasses captures the sign's image, and OCR technology detects text such as "Restaurant," "Open," and "Hours: 11:00-23:00." Next, multimodal AI extracts key information such as "Open" and "Hours: 11:00-23:00." This information is sent to the server, where a translation process is performed, resulting in the translation "Restaurant," "Open," and "Hours: 11:00-23:00." The translation result is then sent back to the device and displayed on the smart glasses' display.

[0040] Users can also use voice input to display only specific information. For example, if they say, "Translate only business hours," the device will interpret the command using voice recognition technology and update the filtering criteria. As a result, only "Business hours: 11:00-23:00" will be displayed on the screen.

[0041] In this way, the system of the present invention provides an efficient and convenient means for users to understand visual information in real time. Since the system can translate and display information within the user's field of vision simply by wearing the smart glasses, without the need to operate a smartphone, it is extremely useful for overseas travel and business situations.

[0042] The processing flow will be explained below.

[0043] Step 1:

[0044] The device uses the camera mounted on the smart glasses to capture images of the user's field of view in real time. The camera module starts operating and continuously captures frames of the field of view at a constant frame rate.

[0045] Step 2:

[0046] The video data captured by the device is analyzed using OCR technology to detect character strings. The OCR library recognizes characters within the video frames and extracts them as text data.

[0047] Step 3:

[0048] The device uses multimodal artificial intelligence to extract only the important information from the strings detected by OCR technology, specifically identifying key information such as "Open" and "Closed" related to travel and business, and filtering out other unnecessary information.

[0049] Step 4:

[0050] The device sends the extracted important text information to the server via the network for translation. The device sends a translation request to the server using a network protocol (e.g., HTTP, HTTPS).

[0051] Step 5:

[0052] The server calls a translation API to translate the received text data into a specified language. The server's translation API (e.g., Google Cloud Translation API) translates the input text into the specified language in real time.

[0053] Step 6:

[0054] The server generates the translation result and sends it back to the terminal via the network. The translated text data is then sent to the terminal as a response.

[0055] Step 7:

[0056] The translation results received by the device are displayed in real time on the smart glasses display, and the display's rendering engine overlays the translation results at specific locations within the field of view.

[0057] Step 8:

[0058] The user issues a voice command, such as "translate only a specific sign," and the smart glasses' microphone collects the voice data, which is then converted into text using voice recognition technology.

[0059] Step 9:

[0060] The device analyzes the instructions received through voice recognition and updates the filtering conditions, adjusting the filter algorithm based on the voice command to display only specific information within the field of view.

[0061] This process allows users to visually understand important local information in real time, eliminating the need to operate a smartphone and allowing them to intuitively and quickly obtain information through smart glasses, making travel and business more convenient.

[0062] Example 1

[0063] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0064] In modern society, there is a demand for quick and accurate understanding of visual information, especially when traveling or conducting business in areas where different languages ​​are spoken. However, conventional technologies require the use of devices such as smartphones, which can reduce user convenience. Furthermore, necessary information is often not readily available. An efficient and convenient solution to these problems is needed.

[0065] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0066] In this invention, the server includes means for capturing text within the user's field of view in real time with a camera in a visual device worn by the user, means for detecting text from the captured video data using optical character recognition technology, means for using multimodal artificial intelligence to extract important information from the detected text, means for transmitting the extracted important information to a remote computer device via a network for translation, and means for receiving the translated information again and displaying it on a display device within the user's field of view, thereby enabling the user to understand visual information in real time and acquire information quickly and accurately even in different language environments.

[0067] A "visual device" is a device worn by a user to capture and display information within the user's field of vision, such as a glasses-type device.

[0068] Optical character recognition technology (OCR technology) is a technology that recognizes characters from an image and extracts them as character data.

[0069] "Multimodal AI" is an AI technology that analyzes multiple types of data (e.g., text, images, audio, etc.) and extracts meaningful information from it.

[0070] A "network" is a communications infrastructure that enables the transmission and reception of data, including the Internet and local area networks (LANs).

[0071] A "remote computing device" is a computer system connected via a network that primarily processes and stores data.

[0072] "Translation" is the process of converting text written in one language into a different language.

[0073] A "display" is a screen mounted on a visual device, and is a means by which a user visually confirms information.

[0074] "Real-time" means that data is processed and displayed with little to no consistent delay.

[0075] "Important information" is information that is determined to be useful to the user from among the information acquired by the visual device.

[0076] "Speech recognition technology" is a technology that analyzes speech and understands it as text or commands.

[0077] This invention relates to a system that uses a user-worn visual device to detect text within the user's field of view, extract specific information from it, and display the translated information. This system is realized by combining a camera, optical character recognition technology (OCR technology), multimodal artificial intelligence, a translation server, and a display device.

[0078] This system works by having the user wear a pair of glasses-type visual devices. The devices are equipped with a camera that captures images of the user's field of vision in real time. This image data is then processed by a terminal inside the device.

[0079] The device detects text from the captured video using optical character recognition (OCR) technology. The OCR technology can utilize the API of a general OCR technology provider. This allows text information from signs and other objects to be acquired in real time.

[0080] The device then analyzes the detected text using multimodal AI, such as advanced natural language processing models, to filter out unnecessary information, extracting only the important information.

[0081] After extracting the important information, the device sends it to the server via the network. The server uses a translation API to translate the received string into the specified language. The translated information is then sent back to the device.

[0082] The terminal then displays the received translation information on the display of the visual device, which is overlaid in a natural way within the user's field of vision, allowing the user to instantly confirm the information.

[0083] As a concrete example, consider a case where a user sees a restaurant sign while walking through a city abroad. The camera in the visual device captures the sign's image, and OCR technology detects text such as "Restaurant," "Open," and "Hours: 11:00-23:00." Next, multimodal artificial intelligence extracts key information such as "Open" and "Hours: 11:00-23:00." This information is sent to the server, where a translation process is performed, resulting in the translation results of "Restaurant," "Open," and "Hours: 11:00-23:00." The translation results are then sent back to the device and displayed on the visual device's display.

[0084] Users can also use voice input to display only specific information. For example, if they say, "Translate only business hours," the device will interpret the command using voice recognition technology and update the filtering criteria. As a result, only "Business hours: 11:00-23:00" will be displayed on the screen.

[0085] Here are some examples of prompts to input to the generative AI model:

[0086] "Please translate the opening hours displayed on the restaurant sign."

[0087] "Please translate and display the text of signs viewed through a visual device in real time."

[0088] In this way, the system of the present invention provides an efficient and convenient means for users to understand visual information in real time. By simply wearing the visual device, information within the user's field of vision can be translated and displayed, making it particularly useful in multilingual environments.

[0089] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0090] Processing flow

[0091] Step 1: Capture what's in your field of view

[0092] When a user wears a vision device, the camera built into the vision device captures images within the user's field of view in real time. The input is the image data acquired through the camera of the vision device, and the output is the image data. This data is sent to the terminal inside the vision device. Specifically, the camera sensor continuously captures image frames and stores them in memory.

[0093] Step 2: Character string detection (OCR processing)

[0094] The device processes the video data acquired in step 1 using optical character recognition (OCR) technology. The input is the captured video data, and the output is all the text data contained in the video. Specifically, the OCR software analyzes the video frames and returns the recognized text in a list format. For example, if a video of a sign is input, the output will be "Restaurant," "Open," and "Hours: 11:00-23:00."

[0095] Step 3: Information filtering (extracting important information)

[0096] The device uses multimodal AI to analyze the string data detected in step 2 and extract only the important information. The input is the string data obtained through OCR processing, and the output is data containing only the important information. Specifically, the AI ​​analyzes the string data, eliminates unimportant information such as "Restaurant," and extracts only important information such as "Open" and "Hours: 11:00-23:00."

[0097] Step 4: Submit a translation request

[0098] The terminal sends the important information obtained in step 3 to the server via the network. The input is the important information data, and the output is the request sent to the server. Specifically, the important information is converted into a standard format such as JSON format and sent to the server using the HTTP protocol.

[0099] Step 5: Translation process

[0100] The server translates the important information received in step 4 into the specified language using the translation API. The input is the important information data, and the output is the translated information data. Specifically, the server sends a request to the translation API and processes the received translation results. For example, "Open" and "Hours: 11:00-23:00" are translated into "Open" and "Business hours: 11:00-23:00".

[0101] Step 6: Receive and display the translation results

[0102] The terminal receives the information data translated in step 5 via the network and displays it on the display of the visual device. The input is the translated information data, and the output is the information displayed on the display of the visual device. The specific operation is to receive data from the network, transfer it to the display unit, and overlay it on the display.

[0103] These processing steps allow a user to quickly and accurately translate and display information within their field of vision simply by wearing the visual device.

[0104] (Application example 1)

[0105] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0106] To address the issue of users being unable to immediately understand the information they need due to language barriers in brick-and-mortar stores abroad, it is necessary to provide a means for users to smoothly access information regardless of language. There is also a need for a method to quickly and accurately filter and display specific information based on the user's voice commands.

[0107] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0108] In this invention, the server includes means for detecting character strings within the field of view, means for extracting specific information from the detected character strings, means for translating the extracted information, means for filtering the specific information based on a user's voice input, means for displaying the translated information on a display device, and means for communicating with the server via a network for extracting and translating the information, thereby enabling a user to translate information on product labels and guide signs in a physical store in real time and quickly filter and display specific information based on a voice command.

[0109] The "means for detecting character strings within the field of view" is a technology that uses a video capture device such as a camera to capture text within the user's field of view in real time.

[0110] "Means for extracting specific information from the detected character string" refers to a method of selecting and extracting only important information from the detected character string using optical character recognition (OCR).

[0111] The "means for translating the extracted information" refers to a technique that uses a translation server or a translation API to convert the extracted information into a different language.

[0112] The "means for filtering specific information based on voice input by the user" is a technology that recognizes the user's voice instructions and selects only the necessary information in accordance with those instructions.

[0113] The "means for displaying translated information on a display device" refers to technology that displays the translated text on smart glasses or a display so that the user can see it.

[0114] The "means for communicating with a server via a network for extracting and translating the information" refers to a method for transmitting and receiving data to and from a remote server via a network such as the Internet to extract and translate information.

[0115] The present invention provides a system for enabling users to translate and understand visual information in real time. This system is particularly useful for enabling users to quickly understand product labels and signage in brick-and-mortar stores abroad. Specific embodiments for implementing the present invention are described below.

[0116] Hardware and Software Configuration

[0117] The system includes the following major components:

[0118] 1. Smart glasses: Devices with built-in cameras and displays that are used to display information in the user's field of vision. Typical examples include Google Glass (registered trademark) and Vuzix.

[0119] 2. Camera module: Uses the camera built into the smart glasses to capture images within the user's field of view in real time.

[0120] 3. OCR module: Optical character recognition (OCR) technology to extract text from captured video data. Tesseract OCR is used as an example.

[0121] 4. Speech Recognition Module: Technology for recognizing and analyzing voice commands from the user. Google Speech API is used.

[0122] 5. Multimodal artificial intelligence: AI technology for filtering necessary information and removing unnecessary information from text extracted by OCR.

[0123] 6. Translation server: The data is sent to the server via the network and translated into the specified language using the translation API.

[0124] 7. Display module: A module for displaying translated information on the smart glasses display.

[0125] Program processing and data flow

[0126] 1. Capture camera footage:

[0127] The camera module in the smart glasses captures real-time information about signs and product labels in physical stores within the user's field of view, and this video data is processed within the device.

[0128] 2. Character Recognition (OCR):

[0129] The OCR module detects and extracts text from the captured video data. The OCR technology used is Tesseract OCR.

[0130] 3. Recognition of voice commands:

[0131] The user can give voice instructions through the smart glasses' microphone to display only specific information, and voice recognition is performed using the Google Speech API.

[0132] 4. Information filtering:

[0133] Multimodal AI filters unnecessary information from the extracted text and extracts important information, taking into account filtering conditions specified by voice commands.

[0134] 5. Translation:

[0135] The filtered information is sent to a translation server via the network and translated into the specified language using a translation API.

[0136] 6. Displaying Information:

[0137] The translated information is sent back to the device via the network and displayed on the smart glasses display via the display module, where it is naturally overlaid on the user's field of view.

[0138] Example and prompt for generative AI model

[0139] Consider a situation where a user is searching for a product in a supermarket in a foreign country. If the user gives a voice command such as "translate only the ingredient information," the system extracts ingredient information from the product label, translates it into the specified language, and displays it on the smart glasses display.

[0140] Example prompt for a generative AI model:

[0141] "From the text detected from the camera footage, extract only the parts that contain the keyword 'ingredients', translate them into Japanese, and display them on the screen."

[0142] In this way, the present invention provides an effective means for users to quickly and accurately understand the information they need in a physical store without experiencing a language barrier.

[0143] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0144] Step 1:

[0145] Camera footage capture

[0146] The device (smartglasses camera module) captures real-time images of store signs and product labels within the user's field of view.

[0147] Input: Video within the field of view of a physical store

[0148] Output: Captured video data

[0149] Step 2:

[0150] Character Recognition (OCR)

[0151] The device uses an OCR module (Tesseract OCR) to detect and extract text from the captured video data.

[0152] Input: Captured video data

[0153] Output: The extracted string

[0154] Step 3:

[0155] Recognizing voice commands

[0156] The user gives voice instructions through the microphone in the smart glasses, and the device analyzes the voice instructions using a voice recognition module (Google Speech API).

[0157] Input: User's voice commands

[0158] Output: Parsed voice command

[0159] Step 4:

[0160] Information Filtering

[0161] The device uses multimodal artificial intelligence to filter out unnecessary information from the character strings extracted by OCR and extract specific information in response to the user's voice instructions.

[0162] Input: Extracted string, parsed voice command

[0163] Output: Filtered text containing important information

[0164] Step 5:

[0165] translation

[0166] The filtered information is sent over the network to a translation server, which translates it into the specified language. The server uses a translation API to translate the filtered text.

[0167] Input: filtered text

[0168] Output: Translated information

[0169] Step 6:

[0170] Displaying Information

[0171] The translated information is then sent back to the device and displayed on the smart glasses display via the display module, allowing the user to see the translated information as an overlay in their field of vision.

[0172] Input: Translated information

[0173] Output: Translation information displayed on the smart glasses display

[0174] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0175] This invention relates to a system that detects character strings in the field of view, extracts specific information from them, and displays the translated information on a display device, and also to a system that combines an emotion engine that recognizes the user's emotions. This system is realized by combining a camera, OCR technology, multimodal artificial intelligence, a translation server, a display device, an emotion engine, etc.

[0176] The system operates under the assumption that a user wearing a smartglasses-type display device must recognize a string of characters in their field of vision. First, a camera mounted on the smartglasses captures images of the user's field of vision in real time. This image data is then processed by a terminal inside the smartglasses.

[0177] The device uses OCR technology to detect text from captured video. For example, it can recognize text written on signs and billboards in real time. The detected text information is then analyzed using multimodal artificial intelligence to extract only the important information. This extracted information filters out irrelevant information and provides only the necessary information.

[0178] After extracting the important information, it is sent over the network to a server for translation, which translates the received text into the specified language and sends the translation back to the device.

[0179] The translation results are displayed in real time on the smart glasses display, which is overlaid on the user's field of view in a natural way, allowing them to visually confirm the translated information.

[0180] Furthermore, this system is equipped with an emotion engine that recognizes the user's emotions. The emotion engine recognizes the user's emotional state by analyzing their voice and facial expressions. For example, it can determine the emotional state using voice input. Based on the user's emotions recognized by the emotion engine, the content and amount of information displayed is adjusted. For example, if the user is feeling stressed, the system reduces the amount of information displayed on the screen or provides information in a simpler format to reduce the user's burden.

[0181] As a specific example, consider a situation where a user is reading a sign at an airport in a foreign country. When the user wears smart glasses and looks at an airport sign, the camera captures the image and uses OCR technology to detect strings such as "Departures," "Arrivals," and "Restrooms." Multimodal AI extracts the information about "Departures" and "Restrooms," sends it over the network to a server for translation, and displays the translated information, including "Departing flights," "Arriving flights," and "Restrooms," on the smart glasses' display.

[0182] At the same time, the emotion engine detects tension in the user's facial expressions and voice, and if it determines that the user is feeling stressed, the system adjusts the display to show only important information, reducing the user's burden.If the user is in a hurry, the system will display only "Departing Flights" information in large letters on the screen, helping the user to quickly reach their destination.

[0183] As described above, the system of the present invention not only translates and displays information within the user's field of vision, but also adjusts the way information is presented depending on the user's emotional state, thereby providing a more comfortable and efficient user experience.

[0184] The processing flow will be explained below.

[0185] Step 1:

[0186] The device uses the camera mounted on the smart glasses to capture images of the user's field of view in real time. The camera module starts operating and continuously captures frames of the field of view at a constant frame rate.

[0187] Step 2:

[0188] The video data captured by the device is analyzed using OCR technology to detect character strings. The OCR library recognizes characters within the video frames and extracts them as text data.

[0189] Step 3:

[0190] The device uses multimodal artificial intelligence to extract only the important information from the strings detected by OCR technology, specifically identifying key information such as "Open" and "Closed" related to travel and business, and filtering out other unnecessary information.

[0191] Step 4:

[0192] The device sends the extracted important text information to the server via the network for translation. The device sends a translation request to the server using a network protocol (e.g., HTTP, HTTPS).

[0193] Step 5:

[0194] The server calls a translation API to translate the received text data into the specified language. The server's translation API (e.g., Google Cloud Translation API) translates the input text into the specified language in real time.

[0195] Step 6:

[0196] The server generates the translation result and sends it back to the terminal via the network. The translated text data is then sent to the terminal as a response.

[0197] Step 7:

[0198] The translation results received by the device are displayed in real time on the smart glasses display, and the display's rendering engine overlays the translation results at specific locations within the field of view.

[0199] Step 8:

[0200] The device runs an emotion engine to recognize the user's emotions. The emotion engine captures and analyzes the user's voice and facial expressions using a camera and microphone to recognize the user's emotional state.

[0201] Step 9:

[0202] The device adjusts the content and amount of information displayed based on the user's emotions as recognized by its emotion engine. For example, if the user is feeling stressed, the system reduces the amount of information displayed and simply displays only the important information.

[0203] As a specific example, a case will be described where a user sees a restaurant sign while walking through a city abroad.

[0204] 1. Step 1: The camera captures real-time footage of the restaurant sign.

[0205] 2. Step 2: Use OCR technology to detect strings such as "Restaurant", "Open", and "Hours: 11:00-23:00".

[0206] 3. Step 3: Multimodal AI extracts important information such as "Open" and "Hours: 11:00-23:00".

[0207] 4. Step 4: Send the extracted text information to the server and request translation.

[0208] 5. In step 5, the server calls the translation API and translates the text into the specified language. It generates translation results such as "Restaurant," "Open," and "Business hours: 11:00-23:00."

[0209] 6. Step 6: The translation result is sent back to the device.

[0210] 7. Step 7: The device displays the translation results on the smart glasses display.

[0211] 8. In step 8, the emotion engine analyzes whether the user is feeling stressed while looking at the sign. Based on their voice and facial expressions, it recognizes that the user is relaxed.

[0212] 9. Step 9: If the system recognizes that the user is relaxed, it displays a normal amount of information.

[0213] This process allows users to receive accurate information in real time, and also provides information that is optimally tailored to the user's emotional state.

[0214] Example 2

[0215] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0216] The present invention aims to enable users in foreign language speaking countries to quickly understand text in their field of view. However, existing systems have had problems such as the inclusion of irrelevant text in the process of translating and displaying information in their field of view, or the display of too much information, which can be a burden to the user. Furthermore, since there was no system with the function to adjust the displayed information according to the user's emotional state, the user experience was uniform, which could lead to stressful situations.

[0217] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0218] In this invention, the server includes means for detecting character strings in the field of view, means for extracting specific information from the detected character strings, means for translating the extracted information, means for recognizing the user's emotions and adjusting the content and amount of information to be displayed, and means for communicating with the server via a network for extracting and translating the information. This allows the user to quickly and stress-free understand foreign language character strings in the field of view by adjusting the displayed information according to the user's emotional state while eliminating irrelevant information.

[0219] "Strings within the field of view" refers to character information that exists within the range that the user can visually confirm.

[0220] "Means for detection" refers to devices or technologies that use cameras or sensors to recognize character strings from visual information.

[0221] "Means of extraction" refers to the techniques and methods for selecting the necessary information from the detected string of characters.

[0222] "Translation means" refers to software or algorithms used to convert extracted text into another language.

[0223] "Display device" means an electronic device used to display information to a user's field of vision, such as smart glasses or a display screen.

[0224] "Means for recognizing emotions" refers to software or devices that analyze the user's voice and facial expressions to determine the emotional state the user is feeling.

[0225] "Means for adjusting the content and amount of information" refers to a method for dynamically changing the type and amount of information displayed depending on the user's emotional state.

[0226] A "server" refers to a general-purpose computing device that communicates with terminals via a network and processes information and provides data.

[0227] "Network" refers to the infrastructure for data communication, including the Internet and local area networks.

[0228] "Multimodal AI" refers to an AI technology that integrates and analyzes information from multiple data sources and input formats (e.g., text, images, audio, etc.) to derive optimal results.

[0229] "Filtering" refers to the process of eliminating unnecessary or irrelevant information from a large amount of data and extracting only the necessary information.

[0230] "User" refers to a person who uses the system to obtain information within their field of view and check the translation results.

[0231] This invention provides a system that detects characters in a user's field of vision, extracts specific information from them, and displays the translated information. It also provides a system that recognizes the user's emotions and adjusts the displayed information accordingly. This system is realized by combining various elements, including a camera, optical character recognition technology (OCR), multimodal artificial intelligence, a translation server, a display device, and an emotion engine.

[0232] Hardware and Software Use

[0233] 1. Smart Glasses: The display device at the heart of this invention is a smart glass-type device worn by the user. Examples include Google Glass and Microsoft® HoloLens®. Smart glasses are equipped with a high-resolution camera that can capture images of the user's field of view in real time.

[0234] 2. On-device OCR technology: Video data is processed by on-device OCR technology (e.g., Google Cloud Vision API or Tesseract OCR) to detect text. The OCR technology analyzes text from the captured video and extracts it as digital data.

[0235] 3. On-device multimodal AI: The detected strings are analyzed by on-device multimodal AI (e.g., OpenAI® GPT, BERT) to extract only the important information. At this stage, irrelevant data is filtered out and only the necessary information is retained.

[0236] 4. Network communication: The extracted information is sent to the server via a network (Wi-Fi or mobile data communication). A secure communication protocol such as HTTPS is used.

[0237] 5. Translation server: The received information is translated into the specified language by a translation API (e.g., Google Translate API or DeepL API) running on the server. The translation result is then resent from the server to the device.

[0238] 6. Smart glasses display: The translation results are overlaid on the smart glasses display, allowing the user to visually confirm them in a way that blends naturally into their field of vision.

[0239] 7. Emotion engine: The device is equipped with an emotion engine (e.g., Microsoft Azure (registered trademark) Emotion API or Affectiva) that analyzes the user's voice and facial expressions to recognize their emotional state.

[0240] 8. Adjustment of displayed information: The content and amount of displayed information is adjusted based on the user's emotional state. For example, if the user is feeling stressed, only important information will be displayed in a larger size and presented in a simpler format.

[0241] Specific examples

[0242] Example: Use at a foreign airport

[0243] A user puts on smart glasses at a foreign airport and sees signs such as "Departures" and "Restrooms."

[0244] The camera inside the device captures the image and detects the characters using OCR technology.

[0245] The device uses multimodal artificial intelligence to extract information on "Departures" and "Restrooms" and retains only the important information.

[0246] The terminal sends this information to the server.

[0247] The server translates the received information and sends the translation results back to the terminal.

[0248] The device displays the translation results on the smart glasses display.

[0249] The device's emotion engine analyzes the user's facial expressions and voice, and if it determines that the user is feeling stressed, it will display "Departures" information in large letters, allowing the user to quickly move to their destination.

[0250] Example prompts for generative AI models

[0251] "Please outline a system for use in airports. The system uses a camera, OCR technology, multimodal artificial intelligence, a translation server, a display device, and an emotion engine to detect text in the user's field of view, translate it, and present it to the user. It also adjusts the way the information is presented based on the user's emotional state."

[0252] As described above, the system of the present invention not only translates and displays information within the user's field of vision, but also adjusts the way information is presented depending on the user's emotional state, thereby providing a more comfortable and efficient user experience.

[0253] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0254] Step 1:

[0255] The user puts on the smart glasses.

[0256] Specific operation: The user turns on the smart glasses and puts them on. The smart glasses start up and prepare the camera and display functions.

[0257] Step 2:

[0258] The device captures video with its camera.

[0259] Input: Real-time video of what is in the user's field of view.

[0260] Specific operation: The camera built into the smart glasses captures images within the user's field of view in real time and stores them in the internal memory as video data.

[0261] Output: Captured video data.

[0262] Step 3:

[0263] The device uses OCR technology to detect text from the video.

[0264] Input: Captured video data.

[0265] Specific operation: Using OCR technology (e.g., Google Cloud Vision API or Tesseract OCR), character strings are analyzed from video data and extracted as text data.

[0266] Output: Detected string information.

[0267] Step 4:

[0268] The device uses multimodal artificial intelligence to extract important information.

[0269] Input: Detected string information.

[0270] Specific operation: Using multimodal artificial intelligence (e.g., OpenAI's GPT, BERT), it analyzes input string information, filters out irrelevant data, and extracts important information.

[0271] Output: Filtered important string information.

[0272] Step 5:

[0273] The terminal transmits the extracted information to the server.

[0274] Input: Important string information.

[0275] What it does: Encrypts the extracted information and sends it securely to a server using a network protocol (e.g., HTTPS).

[0276] Output: The string data sent to the server.

[0277] Step 6:

[0278] The server translates the information it receives.

[0279] Input: String data sent from the terminal.

[0280] Specific operation: Using a translation API on the server (e.g., Google Translate API or DeepL API), the received string data is translated into the specified language.

[0281] Output: The translated string data.

[0282] Step 7:

[0283] The server sends the translation results to the terminal.

[0284] Input: Translated string data.

[0285] Specific operation: The translation results are encrypted and sent to the terminal via the network.

[0286] Output: The translation result sent to your device.

[0287] Step 8:

[0288] The device displays the translation results on the smart glasses display.

[0289] Input: Translation results sent from the server.

[0290] Specific operation: The translated information is overlaid on the smart glasses display, which displays the information in a way that blends naturally into the user's field of vision.

[0291] Output: The translation result displayed in the user's field of view.

[0292] Step 9:

[0293] The terminal uses an emotion engine to recognize the user's emotion.

[0294] Input: User's facial expressions and voice data.

[0295] Specific operation: Using an emotion engine (e.g., Microsoft Azure Emotion API or Affectiva), the system analyzes the user's voice and facial expressions in real time to recognize their emotional state.

[0296] Output: The perceived emotional state of the user.

[0297] Step 10:

[0298] The terminal adjusts the displayed information based on the user's emotions.

[0299] Input: The perceived emotional state of the user and the currently displayed information.

[0300] Specific operation: If it determines that the user is feeling stressed, it adjusts the content and amount of information displayed. For example, it adjusts the display according to the user's condition, such as displaying only important information in a larger size.

[0301] Output: Adjusted display information.

[0302] The above processing steps allow the user to quickly and stresslessly understand foreign language character strings within their field of vision.

[0303] (Application example 2)

[0304] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0305] Conventional systems that detect and translate information within the field of view can overwhelm users, causing stress. In autonomous vehicles, drivers and passengers need appropriate information, but unnecessary information occupies their field of view, which can lead to confusion. Furthermore, in today's increasingly international world, there is a need for systems that can accurately provide information in different languages ​​in real time.

[0306] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0307] In this invention, the server includes means for detecting character strings in the field of view, means for extracting specific information, means for translating the information, and means for recognizing the user's emotions and adjusting the content and amount of information displayed. This allows the user to receive information in the field of view accurately and efficiently, and in autonomous vehicles in particular, it is possible to provide only the necessary information at the appropriate time, improving the user experience.

[0308] "In-view text" refers to all written text within the user's field of view.

[0309] "Means for extracting specific information" refers to a device or technology for extracting necessary and important information from the numerous character strings present within the field of view.

[0310] "Translation means" refers to a technique or device that converts text from one language to another.

[0311] "Means for displaying on a display device" refers to a device or method for providing processed information to a user in a visually perceptible form.

[0312] "Means for communicating with a server via a network" refers to technology or devices for sending and receiving data to and from a remote server using the Internet or other digital communication means.

[0313] "Means for recognizing the user's emotions and adjusting the content and amount of information displayed" refers to technology or devices for detecting the user's emotional state and determining the appropriate type and amount of information based on that.

[0314] "Multimodal artificial intelligence" refers to artificial intelligence technology that integrates and processes multiple modalities (input formats), such as audio, video, and text.

[0315] "Devices carried by the user" refers to devices that the user can carry with them at all times, such as smart glasses or smartphones.

[0316] This invention provides a system where a user uses a mobile device, such as smart glasses, to detect text in their field of view in real time, extract and translate specific information, and then adjust the content and amount of information displayed based on the user's emotional state.

[0317] System Configuration

[0318] Hardware

[0319] Smart glasses: These devices incorporate a camera, a display, and a communication module. The camera captures images within the user's field of view, and the display displays translated information.

[0320] Server: Equipped with a high-performance processor and large storage capacity, it runs OCR technology, translation functions, multimodal artificial intelligence, and emotion engines.

[0321] software

[0322] OpenCV: An image processing library for processing video captured by the smart glasses camera and detecting text within the field of view.

[0323] Pytesseract: A library that uses OCR technology to extract text from captured images.

[0324] Google Trans: A service for translating extracted strings into a specified language.

[0325] Emotiv SDK: A software engine for recognizing a user's emotional state in real time and processing that information.

[0326] System Operation

[0327] Smart glasses use a camera to capture images within the user's field of view in real time. The captured image data is then processed by a computer inside the smart glasses. Specifically, OpenCV is used to detect character strings from the image data, and Pytesseract is used to convert the character strings into text data. Multimodal artificial intelligence is then used to extract only the necessary information.

[0328] The extracted information is sent to a server via the network and translated using Google Trans. The translation results are then sent back to the smart glasses and displayed on the display. The display is overlaid in a way that fits naturally into the user's field of vision, allowing the translated information to be visually confirmed.

[0329] Furthermore, the Emotiv SDK analyzes the user's emotional state in real time, adjusting the content and amount of information displayed based on this. For example, if the user is nervous, only the most important information will be displayed in a larger size, reducing the user's stress.

[0330] Specific examples

[0331] As a concrete example, consider a situation where a driver or passenger in an autonomous vehicle is wearing smart glasses. When the driver sees a traffic sign such as "No U-Turn" or "Road Closed," the camera captures it and translates it into "No U-Turn" or "Road Closed" in real time. Furthermore, if the emotion engine detects that the driver is stressed, only the important information is displayed in a larger size, allowing the driver to drive safely without being distracted.

[0332] Prompt Sentence Examples

[0333] "Use the camera on the smart glasses to detect traffic signs within its field of view, translate them into Japanese, and display them. Then, write a program to customize the information displayed depending on the user's emotional state."

[0334] This invention enables users to receive information within their field of vision accurately and efficiently, and in self-driving vehicles in particular, it is possible to provide only the necessary information at the appropriate time, greatly improving safety and comfort.

[0335] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0336] Step 1:

[0337] The camera in the smart glasses captures the image in the user's field of view in real time. This camera image is the input data, and the captured image is sent to the device.

[0338] Step 2:

[0339] The device uses OpenCV to detect text from captured video. Specifically, it processes video data (input) with an image analysis algorithm and extracts text written on signs, billboards, etc. (output).

[0340] Step 3:

[0341] The device converts the extracted strings into text data using Pytesseract. In this step, the strings in the image (input) are converted into text data (output) using OCR technology.

[0342] Step 4:

[0343] The device uses multimodal artificial intelligence to extract only the important information from text data. Specifically, it analyzes the text data (input) with a natural language processing algorithm, filters out unnecessary information, and extracts the necessary information (output).

[0344] Step 5:

[0345] The device sends important information to the server, which translates it into the specified language. The translation engine processes the text information (input) and generates the translation result (output).

[0346] Step 6:

[0347] The server sends the translated information to the terminal. The translation result (input) is sent in a defined data format, and the terminal receives it (output).

[0348] Step 7:

[0349] The device uses the Emotiv SDK to analyze the user's emotional state. The device processes the user's voice and facial expression data (input) with an emotion analysis algorithm to determine the user's emotional state (output).

[0350] Step 8:

[0351] The device adjusts the content and format of the information displayed based on the user's emotional state. By combining the emotional state data (input) and the translation result data (input), the device displays information (output) optimized for the user.

[0352] Step 9:

[0353] The display device presents the optimized information to the user's field of view. This final step uses the data received from the device (input) to overlay information in a visually understandable format (output).

[0354] This processing flow allows users to obtain information within their field of vision efficiently and without stress. Particularly in self-driving vehicles, it makes it possible to provide only the necessary information at the appropriate time.

[0355] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0356] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0357] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0358] [Second embodiment]

[0359] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0360] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0361] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0362] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0363] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0364] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0365] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0366] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0367] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0368] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0369] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0370] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0371] This invention relates to a system that detects characters in the field of view, extracts specific information from them, and displays the translated information on a display device. This system is realized by combining a camera, OCR technology, multimodal artificial intelligence, a translation server, a display device, etc.

[0372] This system works by having the user wear smart glasses. The smart glasses are equipped with a camera that captures images of the user's field of view in real time. This image data is then processed by a device inside the smart glasses.

[0373] The device uses OCR technology to detect text from the captured video, allowing for real-time capture of text information from signs and other objects. The detected text is then analyzed using multimodal artificial intelligence to filter out unnecessary information. This filtering process allows only the most important information to be extracted.

[0374] After extracting the important information, the device sends it to the server via the network. The server uses a translation API to translate the received string into the specified language. The translated information is then sent back to the device.

[0375] The device then displays the translated information on the smart glasses display, which is overlaid naturally within the user's field of vision, allowing the user to instantly check the information.

[0376] As a concrete example, consider the case where a user sees a restaurant sign while walking through a city abroad. The camera in the smart glasses captures the sign's image, and OCR technology detects text such as "Restaurant," "Open," and "Hours: 11:00-23:00." Next, multimodal AI extracts key information such as "Open" and "Hours: 11:00-23:00." This information is sent to the server, where a translation process is performed, resulting in the translation "Restaurant," "Open," and "Hours: 11:00-23:00." The translation result is then sent back to the device and displayed on the smart glasses' display.

[0377] Users can also use voice input to display only specific information. For example, if they say, "Translate only business hours," the device will interpret the command using voice recognition technology and update the filtering criteria. As a result, only "Business hours: 11:00-23:00" will be displayed on the screen.

[0378] In this way, the system of the present invention provides an efficient and convenient means for users to understand visual information in real time. Since the system can translate and display information within the user's field of vision simply by wearing the smart glasses, without the need to operate a smartphone, it is extremely useful for overseas travel and business situations.

[0379] The processing flow will be explained below.

[0380] Step 1:

[0381] The device uses the camera mounted on the smart glasses to capture images of the user's field of view in real time. The camera module starts operating and continuously captures frames of the field of view at a constant frame rate.

[0382] Step 2:

[0383] The video data captured by the device is analyzed using OCR technology to detect character strings. The OCR library recognizes characters within the video frames and extracts them as text data.

[0384] Step 3:

[0385] The device uses multimodal artificial intelligence to extract only the important information from the strings detected by OCR technology, specifically identifying key information such as "Open" and "Closed" related to travel and business, and filtering out other unnecessary information.

[0386] Step 4:

[0387] The device sends the extracted important text information to the server via the network for translation. The device sends a translation request to the server using a network protocol (e.g., HTTP, HTTPS).

[0388] Step 5:

[0389] The server calls a translation API to translate the received text data into the specified language. The server's translation API (e.g., Google Cloud Translation API) translates the input text into the specified language in real time.

[0390] Step 6:

[0391] The server generates the translation result and sends it back to the terminal via the network. The translated text data is then sent to the terminal as a response.

[0392] Step 7:

[0393] The translation results received by the device are displayed in real time on the smart glasses display, and the display's rendering engine overlays the translation results at specific locations within the field of view.

[0394] Step 8:

[0395] The user issues a voice command, such as "translate only a specific sign," and the smart glasses' microphone collects the voice data, which is then converted into text using voice recognition technology.

[0396] Step 9:

[0397] The device analyzes the instructions received through voice recognition and updates the filtering conditions, adjusting the filter algorithm based on the voice command to display only specific information within the field of view.

[0398] This process allows users to visually understand important local information in real time, eliminating the need to operate a smartphone and allowing them to intuitively and quickly obtain information through smart glasses, making travel and business more convenient.

[0399] Example 1

[0400] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0401] In modern society, there is a demand for quick and accurate understanding of visual information, especially when traveling or conducting business in areas where different languages ​​are spoken. However, conventional technologies require the use of devices such as smartphones, which can reduce user convenience. Furthermore, necessary information is often not readily available. An efficient and convenient solution to these problems is needed.

[0402] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0403] In this invention, the server includes means for capturing text within the user's field of view in real time with a camera in a visual device worn by the user, means for detecting text from the captured video data using optical character recognition technology, means for using multimodal artificial intelligence to extract important information from the detected text, means for transmitting the extracted important information to a remote computer device via a network for translation, and means for receiving the translated information again and displaying it on a display device within the user's field of view, thereby enabling the user to understand visual information in real time and acquire information quickly and accurately even in different language environments.

[0404] A "visual device" is a device worn by a user to capture and display information within the user's field of vision, such as a glasses-type device.

[0405] Optical character recognition technology (OCR technology) is a technology that recognizes characters from an image and extracts them as character data.

[0406] "Multimodal AI" is an AI technology that analyzes multiple types of data (e.g., text, images, audio, etc.) and extracts meaningful information from it.

[0407] A "network" is a communications infrastructure that enables the transmission and reception of data, including the Internet and local area networks (LANs).

[0408] A "remote computing device" is a computer system connected via a network that primarily processes and stores data.

[0409] "Translation" is the process of converting text written in one language into a different language.

[0410] A "display" is a screen mounted on a visual device, and is a means by which a user visually confirms information.

[0411] "Real-time" means that data is processed and displayed with little to no consistent delay.

[0412] "Important information" is information that is determined to be useful to the user from among the information acquired by the visual device.

[0413] "Speech recognition technology" is a technology that analyzes speech and understands it as text or commands.

[0414] This invention relates to a system that uses a user-worn visual device to detect text within the user's field of view, extract specific information from it, and display the translated information. This system is realized by combining a camera, optical character recognition technology (OCR technology), multimodal artificial intelligence, a translation server, and a display device.

[0415] This system works by having the user wear a pair of glasses-type visual devices. The devices are equipped with a camera that captures images of the user's field of vision in real time. This image data is then processed by a terminal inside the device.

[0416] The device detects text from the captured video using optical character recognition (OCR) technology. The OCR technology can utilize the API of a general OCR technology provider. This allows text information from signs and other objects to be acquired in real time.

[0417] The device then analyzes the detected text using multimodal AI, such as advanced natural language processing models, to filter out unnecessary information, extracting only the important information.

[0418] After extracting the important information, the device sends it to the server via the network. The server uses a translation API to translate the received string into the specified language. The translated information is then sent back to the device.

[0419] The terminal then displays the received translation information on the display of the visual device, which is overlaid in a natural way within the user's field of vision, allowing the user to instantly confirm the information.

[0420] As a concrete example, consider a case where a user sees a restaurant sign while walking through a city abroad. The camera in the visual device captures the sign's image, and OCR technology detects text such as "Restaurant," "Open," and "Hours: 11:00-23:00." Next, multimodal artificial intelligence extracts key information such as "Open" and "Hours: 11:00-23:00." This information is sent to the server, where a translation process is performed, resulting in the translation results of "Restaurant," "Open," and "Hours: 11:00-23:00." The translation results are then sent back to the device and displayed on the visual device's display.

[0421] Users can also use voice input to display only specific information. For example, if they say, "Translate only business hours," the device will interpret the command using voice recognition technology and update the filtering criteria. As a result, only "Business hours: 11:00-23:00" will be displayed on the screen.

[0422] Here are some examples of prompts to input to the generative AI model:

[0423] "Please translate the opening hours displayed on the restaurant sign."

[0424] "Please translate and display the text of signs viewed through a visual device in real time."

[0425] In this way, the system of the present invention provides an efficient and convenient means for users to understand visual information in real time. By simply wearing the visual device, information within the user's field of vision can be translated and displayed, making it particularly useful in multilingual environments.

[0426] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0427] Processing flow

[0428] Step 1: Capture what's in your field of view

[0429] When a user wears a vision device, the camera built into the vision device captures images within the user's field of view in real time. The input is the image data acquired through the camera of the vision device, and the output is the image data. This data is sent to the terminal inside the vision device. Specifically, the camera sensor continuously captures image frames and stores them in memory.

[0430] Step 2: Character string detection (OCR processing)

[0431] The device processes the video data acquired in step 1 using optical character recognition (OCR) technology. The input is the captured video data, and the output is all the text data contained in the video. Specifically, the OCR software analyzes the video frames and returns the recognized text in a list format. For example, if a video of a sign is input, the output will be "Restaurant," "Open," and "Hours: 11:00-23:00."

[0432] Step 3: Information filtering (extracting important information)

[0433] The device uses multimodal AI to analyze the string data detected in step 2 and extract only the important information. The input is the string data obtained through OCR processing, and the output is data containing only the important information. Specifically, the AI ​​analyzes the string data, eliminates unimportant information such as "Restaurant," and extracts only important information such as "Open" and "Hours: 11:00-23:00."

[0434] Step 4: Submit a translation request

[0435] The terminal sends the important information obtained in step 3 to the server via the network. The input is the important information data, and the output is the request sent to the server. Specifically, the important information is converted into a standard format such as JSON format and sent to the server using the HTTP protocol.

[0436] Step 5: Translation process

[0437] The server translates the important information received in step 4 into the specified language using the translation API. The input is the important information data, and the output is the translated information data. Specifically, the server sends a request to the translation API and processes the received translation results. For example, "Open" and "Hours: 11:00-23:00" are translated into "Open" and "Business hours: 11:00-23:00".

[0438] Step 6: Receive and display the translation results

[0439] The terminal receives the information data translated in step 5 via the network and displays it on the display of the visual device. The input is the translated information data, and the output is the information displayed on the display of the visual device. The specific operation is to receive data from the network, transfer it to the display unit, and overlay it on the display.

[0440] These processing steps allow a user to quickly and accurately translate and display information within their field of vision simply by wearing the visual device.

[0441] (Application example 1)

[0442] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0443] To address the issue of users being unable to immediately understand the information they need due to language barriers in brick-and-mortar stores abroad, it is necessary to provide a means for users to smoothly access information regardless of language. There is also a need for a method to quickly and accurately filter and display specific information based on the user's voice commands.

[0444] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0445] In this invention, the server includes means for detecting character strings within the field of view, means for extracting specific information from the detected character strings, means for translating the extracted information, means for filtering the specific information based on a user's voice input, means for displaying the translated information on a display device, and means for communicating with the server via a network for extracting and translating the information, thereby enabling a user to translate information on product labels and guide signs in a physical store in real time and quickly filter and display specific information based on a voice command.

[0446] The "means for detecting character strings within the field of view" is a technology that uses a video capture device such as a camera to capture text within the user's field of view in real time.

[0447] "Means for extracting specific information from the detected character string" refers to a method of selecting and extracting only important information from the detected character string using optical character recognition (OCR).

[0448] The "means for translating the extracted information" refers to a technique that uses a translation server or a translation API to convert the extracted information into a different language.

[0449] The "means for filtering specific information based on voice input by the user" is a technology that recognizes the user's voice instructions and selects only the necessary information in accordance with those instructions.

[0450] The "means for displaying translated information on a display device" refers to technology that displays the translated text on smart glasses or a display so that the user can see it.

[0451] The "means for communicating with a server via a network for extracting and translating the information" refers to a method for transmitting and receiving data to and from a remote server via a network such as the Internet to extract and translate information.

[0452] The present invention provides a system for enabling users to translate and understand visual information in real time. This system is particularly useful for enabling users to quickly understand product labels and signage in brick-and-mortar stores abroad. Specific embodiments for implementing the present invention are described below.

[0453] Hardware and Software Configuration

[0454] The system includes the following major components:

[0455] 1. Smart glasses: Devices with built-in cameras and displays that are used to display information in the user's field of vision. Typical examples include Google Glass and Vuzix.

[0456] 2. Camera module: Uses the camera built into the smart glasses to capture images within the user's field of view in real time.

[0457] 3. OCR module: Optical character recognition (OCR) technology to extract text from captured video data. Tesseract OCR is used as an example.

[0458] 4. Speech Recognition Module: Technology for recognizing and analyzing voice commands from the user. Google Speech API is used.

[0459] 5. Multimodal artificial intelligence: AI technology for filtering necessary information and removing unnecessary information from text extracted by OCR.

[0460] 6. Translation server: The data is sent to the server via the network and translated into the specified language using the translation API.

[0461] 7. Display module: A module for displaying translated information on the smart glasses display.

[0462] Program processing and data flow

[0463] 1. Capture camera footage:

[0464] The camera module in the smart glasses captures real-time information about signs and product labels in physical stores within the user's field of view, and this video data is processed within the device.

[0465] 2. Character Recognition (OCR):

[0466] The OCR module detects and extracts text from the captured video data. The OCR technology used is Tesseract OCR.

[0467] 3. Recognition of voice commands:

[0468] The user can give voice instructions through the smart glasses' microphone to display only specific information, and voice recognition is performed using the Google Speech API.

[0469] 4. Information filtering:

[0470] Multimodal AI filters unnecessary information from the extracted text and extracts important information, taking into account filtering conditions specified by voice commands.

[0471] 5. Translation:

[0472] The filtered information is sent to a translation server via the network and translated into the specified language using a translation API.

[0473] 6. Displaying Information:

[0474] The translated information is sent back to the device via the network and displayed on the smart glasses display via the display module, where it is naturally overlaid on the user's field of view.

[0475] Example and prompt for generative AI model

[0476] Consider a situation where a user is searching for a product in a supermarket in a foreign country. If the user gives a voice command such as "translate only the ingredient information," the system extracts ingredient information from the product label, translates it into the specified language, and displays it on the smart glasses display.

[0477] Example prompt for a generative AI model:

[0478] "From the text detected from the camera footage, extract only the parts that contain the keyword 'ingredients', translate them into Japanese, and display them on the screen."

[0479] In this way, the present invention provides an effective means for users to quickly and accurately understand the information they need in a physical store without experiencing a language barrier.

[0480] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0481] Step 1:

[0482] Camera footage capture

[0483] The device (smartglasses camera module) captures real-time images of store signs and product labels within the user's field of view.

[0484] Input: Video within the field of view of a physical store

[0485] Output: Captured video data

[0486] Step 2:

[0487] Character Recognition (OCR)

[0488] The device uses an OCR module (Tesseract OCR) to detect and extract text from the captured video data.

[0489] Input: Captured video data

[0490] Output: The extracted string

[0491] Step 3:

[0492] Recognizing voice commands

[0493] The user gives voice instructions through the microphone in the smart glasses, and the device analyzes the voice instructions using a voice recognition module (Google Speech API).

[0494] Input: User's voice commands

[0495] Output: Parsed voice command

[0496] Step 4:

[0497] Information Filtering

[0498] The device uses multimodal artificial intelligence to filter out unnecessary information from the character strings extracted by OCR and extract specific information in response to the user's voice instructions.

[0499] Input: Extracted string, parsed voice command

[0500] Output: Filtered text containing important information

[0501] Step 5:

[0502] translation

[0503] The filtered information is sent over the network to a translation server, which translates it into the specified language. The server uses a translation API to translate the filtered text.

[0504] Input: filtered text

[0505] Output: Translated information

[0506] Step 6:

[0507] Displaying Information

[0508] The translated information is then sent back to the device and displayed on the smart glasses display via the display module, allowing the user to see the translated information as an overlay in their field of vision.

[0509] Input: Translated information

[0510] Output: Translation information displayed on the smart glasses display

[0511] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0512] This invention relates to a system that detects character strings in the field of view, extracts specific information from them, and displays the translated information on a display device, and also to a system that combines an emotion engine that recognizes the user's emotions. This system is realized by combining a camera, OCR technology, multimodal artificial intelligence, a translation server, a display device, an emotion engine, etc.

[0513] The system operates under the assumption that a user wearing a smartglasses-type display device must recognize a string of characters in their field of vision. First, a camera mounted on the smartglasses captures images of the user's field of vision in real time. This image data is then processed by a terminal inside the smartglasses.

[0514] The device uses OCR technology to detect text from captured video. For example, it can recognize text written on signs and billboards in real time. The detected text information is then analyzed using multimodal artificial intelligence to extract only the important information. This extracted information filters out irrelevant information and provides only the necessary information.

[0515] After extracting the important information, it is sent over the network to a server for translation, which translates the received text into the specified language and sends the translation back to the device.

[0516] The translation results are displayed in real time on the smart glasses display, which is overlaid on the user's field of view in a natural way, allowing them to visually confirm the translated information.

[0517] Furthermore, this system is equipped with an emotion engine that recognizes the user's emotions. The emotion engine recognizes the user's emotional state by analyzing their voice and facial expressions. For example, it can determine the emotional state using voice input. Based on the user's emotions recognized by the emotion engine, the content and amount of information displayed is adjusted. For example, if the user is feeling stressed, the system reduces the amount of information displayed on the screen or provides information in a simpler format to reduce the user's burden.

[0518] As a specific example, consider a situation where a user is reading a sign at an airport in a foreign country. When the user wears smart glasses and looks at an airport sign, the camera captures the image and uses OCR technology to detect strings such as "Departures," "Arrivals," and "Restrooms." Multimodal AI extracts the information about "Departures" and "Restrooms," sends it over the network to a server for translation, and displays the translated information, including "Departing flights," "Arriving flights," and "Restrooms," on the smart glasses' display.

[0519] At the same time, the emotion engine detects tension in the user's facial expressions and voice, and if it determines that the user is feeling stressed, the system adjusts the display to show only important information, reducing the user's burden.If the user is in a hurry, the system will display only "Departing Flights" information in large letters on the screen, helping the user to quickly reach their destination.

[0520] As described above, the system of the present invention not only translates and displays information within the user's field of vision, but also adjusts the way information is presented depending on the user's emotional state, thereby providing a more comfortable and efficient user experience.

[0521] The processing flow will be explained below.

[0522] Step 1:

[0523] The device uses the camera mounted on the smart glasses to capture images of the user's field of view in real time. The camera module starts operating and continuously captures frames of the field of view at a constant frame rate.

[0524] Step 2:

[0525] The video data captured by the device is analyzed using OCR technology to detect character strings. The OCR library recognizes characters within the video frames and extracts them as text data.

[0526] Step 3:

[0527] The device uses multimodal artificial intelligence to extract only the important information from the strings detected by OCR technology, specifically identifying key information such as "Open" and "Closed" related to travel and business, and filtering out other unnecessary information.

[0528] Step 4:

[0529] The device sends the extracted important text information to the server via the network for translation. The device sends a translation request to the server using a network protocol (e.g., HTTP, HTTPS).

[0530] Step 5:

[0531] The server calls a translation API to translate the received text data into the specified language. The server's translation API (e.g., Google Cloud Translation API) translates the input text into the specified language in real time.

[0532] Step 6:

[0533] The server generates the translation result and sends it back to the terminal via the network. The translated text data is then sent to the terminal as a response.

[0534] Step 7:

[0535] The translation results received by the device are displayed in real time on the smart glasses display, and the display's rendering engine overlays the translation results at specific locations within the field of view.

[0536] Step 8:

[0537] The device runs an emotion engine to recognize the user's emotions. The emotion engine captures and analyzes the user's voice and facial expressions using a camera and microphone to recognize the user's emotional state.

[0538] Step 9:

[0539] The device adjusts the content and amount of information displayed based on the user's emotions as recognized by its emotion engine. For example, if the user is feeling stressed, the system reduces the amount of information displayed and simply displays only the important information.

[0540] As a specific example, a case will be described where a user sees a restaurant sign while walking through a city abroad.

[0541] 1. Step 1: The camera captures real-time footage of the restaurant sign.

[0542] 2. Step 2: Use OCR technology to detect strings such as "Restaurant", "Open", and "Hours: 11:00-23:00".

[0543] 3. Step 3: Multimodal AI extracts important information such as "Open" and "Hours: 11:00-23:00".

[0544] 4. Step 4: Send the extracted text information to the server and request translation.

[0545] 5. In step 5, the server calls the translation API and translates the text into the specified language. It generates translation results such as "Restaurant," "Open," and "Business hours: 11:00-23:00."

[0546] 6. Step 6: The translation result is sent back to the device.

[0547] 7. Step 7: The device displays the translation results on the smart glasses display.

[0548] 8. In step 8, the emotion engine analyzes whether the user is feeling stressed while looking at the sign. Based on their voice and facial expressions, it recognizes that the user is relaxed.

[0549] 9. Step 9: If the system recognizes that the user is relaxed, it displays a normal amount of information.

[0550] This process allows users to receive accurate information in real time, and also provides information that is optimally tailored to the user's emotional state.

[0551] Example 2

[0552] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0553] The present invention aims to enable users in foreign language speaking countries to quickly understand text in their field of view. However, existing systems have had problems such as the inclusion of irrelevant text in the process of translating and displaying information in their field of view, or the display of too much information, which can be a burden to the user. Furthermore, since there was no system with the function to adjust the displayed information according to the user's emotional state, the user experience was uniform, which could lead to stressful situations.

[0554] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0555] In this invention, the server includes means for detecting character strings in the field of view, means for extracting specific information from the detected character strings, means for translating the extracted information, means for recognizing the user's emotions and adjusting the content and amount of information to be displayed, and means for communicating with the server via a network for extracting and translating the information. This allows the user to quickly and stress-free understand foreign language character strings in the field of view by adjusting the displayed information according to the user's emotional state while eliminating irrelevant information.

[0556] "Strings within the field of view" refers to character information that exists within the range that the user can visually confirm.

[0557] "Means for detection" refers to devices or technologies that use cameras or sensors to recognize character strings from visual information.

[0558] "Means of extraction" refers to the techniques and methods for selecting the necessary information from the detected string of characters.

[0559] "Translation means" refers to software or algorithms used to convert extracted text into another language.

[0560] "Display device" means an electronic device used to display information to a user's field of vision, such as smart glasses or a display screen.

[0561] "Means for recognizing emotions" refers to software or devices that analyze the user's voice and facial expressions to determine the emotional state the user is feeling.

[0562] "Means for adjusting the content and amount of information" refers to a method for dynamically changing the type and amount of information displayed depending on the user's emotional state.

[0563] A "server" refers to a general-purpose computing device that communicates with terminals via a network and processes information and provides data.

[0564] "Network" refers to the infrastructure for data communication, including the Internet and local area networks.

[0565] "Multimodal AI" refers to an AI technology that integrates and analyzes information from multiple data sources and input formats (e.g., text, images, audio, etc.) to derive optimal results.

[0566] "Filtering" refers to the process of eliminating unnecessary or irrelevant information from a large amount of data and extracting only the necessary information.

[0567] "User" refers to a person who uses the system to obtain information within their field of view and check the translation results.

[0568] This invention provides a system that detects characters in a user's field of vision, extracts specific information from them, and displays the translated information. It also provides a system that recognizes the user's emotions and adjusts the displayed information accordingly. This system is realized by combining various elements, including a camera, optical character recognition technology (OCR), multimodal artificial intelligence, a translation server, a display device, and an emotion engine.

[0569] Hardware and Software Use

[0570] 1. Smart Glasses: The display device at the heart of this invention is a smart glass-type device worn by the user. Examples include Google Glass and Microsoft HoloLens. Smart glasses are equipped with a high-resolution camera that can capture images of the user's field of view in real time.

[0571] 2. On-device OCR technology: Video data is processed by on-device OCR technology (e.g., Google Cloud Vision API or Tesseract OCR) to detect text. The OCR technology analyzes text from the captured video and extracts it as digital data.

[0572] 3. On-device multimodal AI: The detected strings are analyzed by on-device multimodal AI (e.g., OpenAI's GPT, BERT) to extract only the important information. At this stage, irrelevant data is filtered out and only the necessary information is retained.

[0573] 4. Network communication: The extracted information is sent to the server via a network (Wi-Fi or mobile data communication). A secure communication protocol such as HTTPS is used.

[0574] 5. Translation server: The received information is translated into the specified language by a translation API (e.g., Google Translate API or DeepL API) running on the server. The translation result is then resent from the server to the device.

[0575] 6. Smart glasses display: The translation results are overlaid on the smart glasses display, allowing the user to visually confirm them in a way that blends naturally into their field of vision.

[0576] 7. Emotion engine: The device is equipped with an emotion engine (e.g., Microsoft Azure Emotion API or Affectiva) that analyzes the user's voice and facial expressions to recognize their emotional state.

[0577] 8. Adjustment of displayed information: The content and amount of displayed information is adjusted based on the user's emotional state. For example, if the user is feeling stressed, only important information will be displayed in a larger size and presented in a simpler format.

[0578] Specific examples

[0579] Example: Use at a foreign airport

[0580] A user puts on smart glasses at a foreign airport and sees signs such as "Departures" and "Restrooms."

[0581] The camera inside the device captures the image and detects the characters using OCR technology.

[0582] The device uses multimodal artificial intelligence to extract information on "Departures" and "Restrooms" and retains only the important information.

[0583] The terminal sends this information to the server.

[0584] The server translates the received information and sends the translation results back to the terminal.

[0585] The device displays the translation results on the smart glasses display.

[0586] The device's emotion engine analyzes the user's facial expressions and voice, and if it determines that the user is feeling stressed, it will display "Departures" information in large letters, allowing the user to quickly move to their destination.

[0587] Example prompts for generative AI models

[0588] "Please outline a system for use in airports. The system uses a camera, OCR technology, multimodal artificial intelligence, a translation server, a display device, and an emotion engine to detect text in the user's field of view, translate it, and present it to the user. It also adjusts the way the information is presented based on the user's emotional state."

[0589] As described above, the system of the present invention not only translates and displays information within the user's field of vision, but also adjusts the way information is presented depending on the user's emotional state, thereby providing a more comfortable and efficient user experience.

[0590] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0591] Step 1:

[0592] The user puts on the smart glasses.

[0593] Specific operation: The user turns on the smart glasses and puts them on. The smart glasses start up and prepare the camera and display functions.

[0594] Step 2:

[0595] The device captures video with its camera.

[0596] Input: Real-time video of what is in the user's field of view.

[0597] Specific operation: The camera built into the smart glasses captures images within the user's field of view in real time and stores them in the internal memory as video data.

[0598] Output: Captured video data.

[0599] Step 3:

[0600] The device uses OCR technology to detect text from the video.

[0601] Input: Captured video data.

[0602] Specific operation: Using OCR technology (e.g., Google Cloud Vision API or Tesseract OCR), character strings are analyzed from video data and extracted as text data.

[0603] Output: Detected string information.

[0604] Step 4:

[0605] The device uses multimodal artificial intelligence to extract important information.

[0606] Input: Detected string information.

[0607] Specific operation: Using multimodal artificial intelligence (e.g., OpenAI's GPT, BERT), it analyzes input string information, filters out irrelevant data, and extracts important information.

[0608] Output: Filtered important string information.

[0609] Step 5:

[0610] The terminal transmits the extracted information to the server.

[0611] Input: Important string information.

[0612] What it does: Encrypts the extracted information and sends it securely to a server using a network protocol (e.g., HTTPS).

[0613] Output: The string data sent to the server.

[0614] Step 6:

[0615] The server translates the information it receives.

[0616] Input: String data sent from the terminal.

[0617] Specific operation: Using a translation API on the server (e.g., Google Translate API or DeepL API), the received string data is translated into the specified language.

[0618] Output: The translated string data.

[0619] Step 7:

[0620] The server sends the translation results to the terminal.

[0621] Input: Translated string data.

[0622] Specific operation: The translation results are encrypted and sent to the terminal via the network.

[0623] Output: The translation result sent to your device.

[0624] Step 8:

[0625] The device displays the translation results on the smart glasses display.

[0626] Input: Translation results sent from the server.

[0627] Specific operation: The translated information is overlaid on the smart glasses display, which displays the information in a way that blends naturally into the user's field of vision.

[0628] Output: The translation result displayed in the user's field of view.

[0629] Step 9:

[0630] The terminal uses an emotion engine to recognize the user's emotion.

[0631] Input: User's facial expressions and voice data.

[0632] Specific operation: Using an emotion engine (e.g., Microsoft Azure Emotion API or Affectiva), the system analyzes the user's voice and facial expressions in real time to recognize their emotional state.

[0633] Output: The perceived emotional state of the user.

[0634] Step 10:

[0635] The terminal adjusts the displayed information based on the user's emotions.

[0636] Input: The perceived emotional state of the user and the currently displayed information.

[0637] Specific operation: If it determines that the user is feeling stressed, it adjusts the content and amount of information displayed. For example, it adjusts the display according to the user's condition, such as displaying only important information in a larger size.

[0638] Output: Adjusted display information.

[0639] The above processing steps allow the user to quickly and stresslessly understand foreign language character strings within their field of vision.

[0640] (Application example 2)

[0641] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0642] Conventional systems that detect and translate information within the field of view can overwhelm users, causing stress. In autonomous vehicles, drivers and passengers need appropriate information, but unnecessary information occupies their field of view, which can lead to confusion. Furthermore, in today's increasingly international world, there is a need for systems that can accurately provide information in different languages ​​in real time.

[0643] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0644] In this invention, the server includes means for detecting character strings in the field of view, means for extracting specific information, means for translating the information, and means for recognizing the user's emotions and adjusting the content and amount of information displayed. This allows the user to receive information in the field of view accurately and efficiently, and in autonomous vehicles in particular, it is possible to provide only the necessary information at the appropriate time, improving the user experience.

[0645] "In-view text" refers to all written text within the user's field of view.

[0646] "Means for extracting specific information" refers to a device or technology for extracting necessary and important information from the numerous character strings present within the field of view.

[0647] "Translation means" refers to a technique or device that converts text from one language to another.

[0648] "Means for displaying on a display device" refers to a device or method for providing processed information to a user in a visually perceptible form.

[0649] "Means for communicating with a server via a network" refers to technology or devices for sending and receiving data to and from a remote server using the Internet or other digital communication means.

[0650] "Means for recognizing the user's emotions and adjusting the content and amount of information displayed" refers to technology or devices for detecting the user's emotional state and determining the appropriate type and amount of information based on that.

[0651] "Multimodal artificial intelligence" refers to artificial intelligence technology that integrates and processes multiple modalities (input formats), such as audio, video, and text.

[0652] "Devices carried by the user" refers to devices that the user can carry with them at all times, such as smart glasses or smartphones.

[0653] This invention provides a system where a user uses a mobile device, such as smart glasses, to detect text in their field of view in real time, extract and translate specific information, and then adjust the content and amount of information displayed based on the user's emotional state.

[0654] System Configuration

[0655] Hardware

[0656] Smart glasses: These devices incorporate a camera, a display, and a communication module. The camera captures images within the user's field of view, and the display displays translated information.

[0657] Server: Equipped with a high-performance processor and large storage capacity, it runs OCR technology, translation functions, multimodal artificial intelligence, and emotion engines.

[0658] software

[0659] OpenCV: An image processing library for processing video captured by the smart glasses camera and detecting text within the field of view.

[0660] Pytesseract: A library that uses OCR technology to extract text from captured images.

[0661] Google Trans: A service for translating extracted strings into a specified language.

[0662] Emotiv SDK: A software engine for recognizing a user's emotional state in real time and processing that information.

[0663] System Operation

[0664] Smart glasses use a camera to capture images within the user's field of view in real time. The captured image data is then processed by a computer inside the smart glasses. Specifically, OpenCV is used to detect character strings from the image data, and Pytesseract is used to convert the character strings into text data. Multimodal artificial intelligence is then used to extract only the necessary information.

[0665] The extracted information is sent to a server via the network and translated using Google Trans. The translation results are then sent back to the smart glasses and displayed on the display. The display is overlaid in a way that fits naturally into the user's field of vision, allowing the translated information to be visually confirmed.

[0666] Furthermore, the Emotiv SDK analyzes the user's emotional state in real time, adjusting the content and amount of information displayed based on this. For example, if the user is nervous, only the most important information will be displayed in a larger size, reducing the user's stress.

[0667] Specific examples

[0668] As a concrete example, consider a situation where a driver or passenger in an autonomous vehicle is wearing smart glasses. When the driver sees a traffic sign such as "No U-Turn" or "Road Closed," the camera captures it and translates it into "No U-Turn" or "Road Closed" in real time. Furthermore, if the emotion engine detects that the driver is stressed, only the important information is displayed in a larger size, allowing the driver to drive safely without being distracted.

[0669] Prompt Sentence Examples

[0670] "Use the camera on the smart glasses to detect traffic signs within its field of view, translate them into Japanese, and display them. Then, write a program to customize the information displayed depending on the user's emotional state."

[0671] This invention enables users to receive information within their field of vision accurately and efficiently, and in self-driving vehicles in particular, it is possible to provide only the necessary information at the appropriate time, greatly improving safety and comfort.

[0672] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0673] Step 1:

[0674] The camera in the smart glasses captures the image in the user's field of view in real time. This camera image is the input data, and the captured image is sent to the device.

[0675] Step 2:

[0676] The device uses OpenCV to detect text from captured video. Specifically, it processes video data (input) with an image analysis algorithm and extracts text written on signs, billboards, etc. (output).

[0677] Step 3:

[0678] The device converts the extracted strings into text data using Pytesseract. In this step, the strings in the image (input) are converted into text data (output) using OCR technology.

[0679] Step 4:

[0680] The device uses multimodal artificial intelligence to extract only the important information from text data. Specifically, it analyzes the text data (input) with a natural language processing algorithm, filters out unnecessary information, and extracts the necessary information (output).

[0681] Step 5:

[0682] The device sends important information to the server, which translates it into the specified language. The translation engine processes the text information (input) and generates the translation result (output).

[0683] Step 6:

[0684] The server sends the translated information to the terminal. The translation result (input) is sent in a defined data format, and the terminal receives it (output).

[0685] Step 7:

[0686] The device uses the Emotiv SDK to analyze the user's emotional state. The device processes the user's voice and facial expression data (input) with an emotion analysis algorithm to determine the user's emotional state (output).

[0687] Step 8:

[0688] The device adjusts the content and format of the information displayed based on the user's emotional state. By combining the emotional state data (input) and the translation result data (input), the device displays information (output) optimized for the user.

[0689] Step 9:

[0690] The display device presents the optimized information to the user's field of view. This final step uses the data received from the device (input) to overlay information in a visually understandable format (output).

[0691] This processing flow allows users to obtain information within their field of vision efficiently and without stress. Particularly in self-driving vehicles, it makes it possible to provide only the necessary information at the appropriate time.

[0692] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0693] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0694] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0695] [Third embodiment]

[0696] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0697] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0698] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0699] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0700] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0701] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0702] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0703] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0704] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0705] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0706] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0707] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0708] This invention relates to a system that detects characters in the field of view, extracts specific information from them, and displays the translated information on a display device. This system is realized by combining a camera, OCR technology, multimodal artificial intelligence, a translation server, a display device, etc.

[0709] This system works by having the user wear smart glasses. The smart glasses are equipped with a camera that captures images of the user's field of view in real time. This image data is then processed by a device inside the smart glasses.

[0710] The device uses OCR technology to detect text from the captured video, allowing for real-time capture of text information from signs and other objects. The detected text is then analyzed using multimodal artificial intelligence to filter out unnecessary information. This filtering process allows only the most important information to be extracted.

[0711] After extracting the important information, the device sends it to the server via the network. The server uses a translation API to translate the received string into the specified language. The translated information is then sent back to the device.

[0712] The device then displays the translated information on the smart glasses display, which is overlaid naturally within the user's field of vision, allowing the user to instantly check the information.

[0713] As a concrete example, consider the case where a user sees a restaurant sign while walking through a city abroad. The camera in the smart glasses captures the sign's image, and OCR technology detects text such as "Restaurant," "Open," and "Hours: 11:00-23:00." Next, multimodal AI extracts key information such as "Open" and "Hours: 11:00-23:00." This information is sent to the server, where a translation process is performed, resulting in the translation "Restaurant," "Open," and "Hours: 11:00-23:00." The translation result is then sent back to the device and displayed on the smart glasses' display.

[0714] Users can also use voice input to display only specific information. For example, if they say, "Translate only business hours," the device will interpret the command using voice recognition technology and update the filtering criteria. As a result, only "Business hours: 11:00-23:00" will be displayed on the screen.

[0715] In this way, the system of the present invention provides an efficient and convenient means for users to understand visual information in real time. Since the system can translate and display information within the user's field of vision simply by wearing the smart glasses, without the need to operate a smartphone, it is extremely useful for overseas travel and business situations.

[0716] The processing flow will be explained below.

[0717] Step 1:

[0718] The device uses the camera mounted on the smart glasses to capture images of the user's field of view in real time. The camera module starts operating and continuously captures frames of the field of view at a constant frame rate.

[0719] Step 2:

[0720] The video data captured by the device is analyzed using OCR technology to detect character strings. The OCR library recognizes characters within the video frames and extracts them as text data.

[0721] Step 3:

[0722] The device uses multimodal artificial intelligence to extract only the important information from the strings detected by OCR technology, specifically identifying key information such as "Open" and "Closed" related to travel and business, and filtering out other unnecessary information.

[0723] Step 4:

[0724] The device sends the extracted important text information to the server via the network for translation. The device sends a translation request to the server using a network protocol (e.g., HTTP, HTTPS).

[0725] Step 5:

[0726] The server calls a translation API to translate the received text data into the specified language. The server's translation API (e.g., Google Cloud Translation API) translates the input text into the specified language in real time.

[0727] Step 6:

[0728] The server generates the translation result and sends it back to the terminal via the network. The translated text data is then sent to the terminal as a response.

[0729] Step 7:

[0730] The translation results received by the device are displayed in real time on the smart glasses display, and the display's rendering engine overlays the translation results at specific locations within the field of view.

[0731] Step 8:

[0732] The user issues a voice command, such as "translate only a specific sign," and the smart glasses' microphone collects the voice data, which is then converted into text using voice recognition technology.

[0733] Step 9:

[0734] The device analyzes the instructions received through voice recognition and updates the filtering conditions, adjusting the filter algorithm based on the voice command to display only specific information within the field of view.

[0735] This process allows users to visually understand important local information in real time, eliminating the need to operate a smartphone and allowing them to intuitively and quickly obtain information through smart glasses, making travel and business more convenient.

[0736] Example 1

[0737] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0738] In modern society, there is a demand for quick and accurate understanding of visual information, especially when traveling or conducting business in areas where different languages ​​are spoken. However, conventional technologies require the use of devices such as smartphones, which can reduce user convenience. Furthermore, necessary information is often not readily available. An efficient and convenient solution to these problems is needed.

[0739] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0740] In this invention, the server includes means for capturing text within the user's field of view in real time with a camera in a visual device worn by the user, means for detecting text from the captured video data using optical character recognition technology, means for using multimodal artificial intelligence to extract important information from the detected text, means for transmitting the extracted important information to a remote computer device via a network for translation, and means for receiving the translated information again and displaying it on a display device within the user's field of view, thereby enabling the user to understand visual information in real time and acquire information quickly and accurately even in different language environments.

[0741] A "visual device" is a device worn by a user to capture and display information within the user's field of vision, such as a glasses-type device.

[0742] Optical character recognition technology (OCR technology) is a technology that recognizes characters from an image and extracts them as character data.

[0743] "Multimodal AI" is an AI technology that analyzes multiple types of data (e.g., text, images, audio, etc.) and extracts meaningful information from it.

[0744] A "network" is a communications infrastructure that enables the transmission and reception of data, including the Internet and local area networks (LANs).

[0745] A "remote computing device" is a computer system connected via a network that primarily processes and stores data.

[0746] "Translation" is the process of converting text written in one language into a different language.

[0747] A "display" is a screen mounted on a visual device, and is a means by which a user visually confirms information.

[0748] "Real-time" means that data is processed and displayed with little to no consistent delay.

[0749] "Important information" is information that is determined to be useful to the user from among the information acquired by the visual device.

[0750] "Speech recognition technology" is a technology that analyzes speech and understands it as text or commands.

[0751] This invention relates to a system that uses a user-worn visual device to detect text within the user's field of view, extract specific information from it, and display the translated information. This system is realized by combining a camera, optical character recognition technology (OCR technology), multimodal artificial intelligence, a translation server, and a display device.

[0752] This system works by having the user wear a pair of glasses-type visual devices. The devices are equipped with a camera that captures images of the user's field of vision in real time. This image data is then processed by a terminal inside the device.

[0753] The device detects text from the captured video using optical character recognition (OCR) technology. The OCR technology can utilize the API of a general OCR technology provider. This allows text information from signs and other objects to be acquired in real time.

[0754] The device then analyzes the detected text using multimodal AI, such as advanced natural language processing models, to filter out unnecessary information, extracting only the important information.

[0755] After extracting the important information, the device sends it to the server via the network. The server uses a translation API to translate the received string into the specified language. The translated information is then sent back to the device.

[0756] The terminal then displays the received translation information on the display of the visual device, which is overlaid in a natural way within the user's field of vision, allowing the user to instantly confirm the information.

[0757] As a concrete example, consider a case where a user sees a restaurant sign while walking through a city abroad. The camera in the visual device captures the sign's image, and OCR technology detects text such as "Restaurant," "Open," and "Hours: 11:00-23:00." Next, multimodal artificial intelligence extracts key information such as "Open" and "Hours: 11:00-23:00." This information is sent to the server, where a translation process is performed, resulting in the translation results of "Restaurant," "Open," and "Hours: 11:00-23:00." The translation results are then sent back to the device and displayed on the visual device's display.

[0758] Users can also use voice input to display only specific information. For example, if they say, "Translate only business hours," the device will interpret the command using voice recognition technology and update the filtering criteria. As a result, only "Business hours: 11:00-23:00" will be displayed on the screen.

[0759] Here are some examples of prompts to input to the generative AI model:

[0760] "Please translate the opening hours displayed on the restaurant sign."

[0761] "Please translate and display the text of signs viewed through a visual device in real time."

[0762] In this way, the system of the present invention provides an efficient and convenient means for users to understand visual information in real time. By simply wearing the visual device, information within the user's field of vision can be translated and displayed, making it particularly useful in multilingual environments.

[0763] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0764] Processing flow

[0765] Step 1: Capture what's in your field of view

[0766] When a user wears a vision device, the camera built into the vision device captures images within the user's field of view in real time. The input is the image data acquired through the camera of the vision device, and the output is the image data. This data is sent to the terminal inside the vision device. Specifically, the camera sensor continuously captures image frames and stores them in memory.

[0767] Step 2: Character string detection (OCR processing)

[0768] The device processes the video data acquired in step 1 using optical character recognition (OCR) technology. The input is the captured video data, and the output is all the text data contained in the video. Specifically, the OCR software analyzes the video frames and returns the recognized text in a list format. For example, if a video of a sign is input, the output will be "Restaurant," "Open," and "Hours: 11:00-23:00."

[0769] Step 3: Information filtering (extracting important information)

[0770] The device uses multimodal AI to analyze the string data detected in step 2 and extract only the important information. The input is the string data obtained through OCR processing, and the output is data containing only the important information. Specifically, the AI ​​analyzes the string data, eliminates unimportant information such as "Restaurant," and extracts only important information such as "Open" and "Hours: 11:00-23:00."

[0771] Step 4: Submit a translation request

[0772] The terminal sends the important information obtained in step 3 to the server via the network. The input is the important information data, and the output is the request sent to the server. Specifically, the important information is converted into a standard format such as JSON format and sent to the server using the HTTP protocol.

[0773] Step 5: Translation process

[0774] The server translates the important information received in step 4 into the specified language using the translation API. The input is the important information data, and the output is the translated information data. Specifically, the server sends a request to the translation API and processes the received translation results. For example, "Open" and "Hours: 11:00-23:00" are translated into "Open" and "Business hours: 11:00-23:00".

[0775] Step 6: Receive and display the translation results

[0776] The terminal receives the information data translated in step 5 via the network and displays it on the display of the visual device. The input is the translated information data, and the output is the information displayed on the display of the visual device. The specific operation is to receive data from the network, transfer it to the display unit, and overlay it on the display.

[0777] These processing steps allow a user to quickly and accurately translate and display information within their field of vision simply by wearing the visual device.

[0778] (Application example 1)

[0779] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0780] To address the issue of users being unable to immediately understand the information they need due to language barriers in brick-and-mortar stores abroad, it is necessary to provide a means for users to smoothly access information regardless of language. There is also a need for a method to quickly and accurately filter and display specific information based on the user's voice commands.

[0781] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0782] In this invention, the server includes means for detecting character strings within the field of view, means for extracting specific information from the detected character strings, means for translating the extracted information, means for filtering the specific information based on a user's voice input, means for displaying the translated information on a display device, and means for communicating with the server via a network for extracting and translating the information, thereby enabling a user to translate information on product labels and guide signs in a physical store in real time and quickly filter and display specific information based on a voice command.

[0783] The "means for detecting character strings within the field of view" is a technology that uses a video capture device such as a camera to capture text within the user's field of view in real time.

[0784] "Means for extracting specific information from the detected character string" refers to a method of selecting and extracting only important information from the detected character string using optical character recognition (OCR).

[0785] The "means for translating the extracted information" refers to a technique that uses a translation server or a translation API to convert the extracted information into a different language.

[0786] The "means for filtering specific information based on voice input by the user" is a technology that recognizes the user's voice instructions and selects only the necessary information in accordance with those instructions.

[0787] The "means for displaying translated information on a display device" refers to technology that displays the translated text on smart glasses or a display so that the user can see it.

[0788] The "means for communicating with a server via a network for extracting and translating the information" refers to a method for transmitting and receiving data to and from a remote server via a network such as the Internet to extract and translate information.

[0789] The present invention provides a system for enabling users to translate and understand visual information in real time. This system is particularly useful for enabling users to quickly understand product labels and signage in brick-and-mortar stores abroad. Specific embodiments for implementing the present invention are described below.

[0790] Hardware and Software Configuration

[0791] The system includes the following major components:

[0792] 1. Smart glasses: Devices with built-in cameras and displays that are used to display information in the user's field of vision. Typical examples include Google Glass and Vuzix.

[0793] 2. Camera module: Uses the camera built into the smart glasses to capture images within the user's field of view in real time.

[0794] 3. OCR module: Optical character recognition (OCR) technology to extract text from captured video data. Tesseract OCR is used as an example.

[0795] 4. Speech Recognition Module: Technology for recognizing and analyzing voice commands from the user. Google Speech API is used.

[0796] 5. Multimodal artificial intelligence: AI technology for filtering necessary information and removing unnecessary information from text extracted by OCR.

[0797] 6. Translation server: The data is sent to the server via the network and translated into the specified language using the translation API.

[0798] 7. Display module: A module for displaying translated information on the smart glasses display.

[0799] Program processing and data flow

[0800] 1. Capture camera footage:

[0801] The camera module in the smart glasses captures real-time information about signs and product labels in physical stores within the user's field of view, and this video data is processed within the device.

[0802] 2. Character Recognition (OCR):

[0803] The OCR module detects and extracts text from the captured video data. The OCR technology used is Tesseract OCR.

[0804] 3. Recognition of voice commands:

[0805] The user can give voice instructions through the smart glasses' microphone to display only specific information, and voice recognition is performed using the Google Speech API.

[0806] 4. Information filtering:

[0807] Multimodal AI filters unnecessary information from the extracted text and extracts important information, taking into account filtering conditions specified by voice commands.

[0808] 5. Translation:

[0809] The filtered information is sent to a translation server via the network and translated into the specified language using a translation API.

[0810] 6. Displaying Information:

[0811] The translated information is sent back to the device via the network and displayed on the smart glasses display via the display module, where it is naturally overlaid on the user's field of view.

[0812] Example and prompt for generative AI model

[0813] Consider a situation where a user is searching for a product in a supermarket in a foreign country. If the user gives a voice command such as "translate only the ingredient information," the system extracts ingredient information from the product label, translates it into the specified language, and displays it on the smart glasses display.

[0814] Example prompt for a generative AI model:

[0815] "From the text detected from the camera footage, extract only the parts that contain the keyword 'ingredients', translate them into Japanese, and display them on the screen."

[0816] In this way, the present invention provides an effective means for users to quickly and accurately understand the information they need in a physical store without experiencing a language barrier.

[0817] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0818] Step 1:

[0819] Camera footage capture

[0820] The device (smartglasses camera module) captures real-time images of store signs and product labels within the user's field of view.

[0821] Input: Video within the field of view of a physical store

[0822] Output: Captured video data

[0823] Step 2:

[0824] Character Recognition (OCR)

[0825] The device uses an OCR module (Tesseract OCR) to detect and extract text from the captured video data.

[0826] Input: Captured video data

[0827] Output: The extracted string

[0828] Step 3:

[0829] Recognizing voice commands

[0830] The user gives voice instructions through the microphone in the smart glasses, and the device analyzes the voice instructions using a voice recognition module (Google Speech API).

[0831] Input: User's voice commands

[0832] Output: Parsed voice command

[0833] Step 4:

[0834] Information Filtering

[0835] The device uses multimodal artificial intelligence to filter out unnecessary information from the character strings extracted by OCR and extract specific information in response to the user's voice instructions.

[0836] Input: Extracted string, parsed voice command

[0837] Output: Filtered text containing important information

[0838] Step 5:

[0839] translation

[0840] The filtered information is sent over the network to a translation server, which translates it into the specified language. The server uses a translation API to translate the filtered text.

[0841] Input: filtered text

[0842] Output: Translated information

[0843] Step 6:

[0844] Displaying Information

[0845] The translated information is then sent back to the device and displayed on the smart glasses display via the display module, allowing the user to see the translated information as an overlay in their field of vision.

[0846] Input: Translated information

[0847] Output: Translation information displayed on the smart glasses display

[0848] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0849] This invention relates to a system that detects character strings in the field of view, extracts specific information from them, and displays the translated information on a display device, and also to a system that combines an emotion engine that recognizes the user's emotions. This system is realized by combining a camera, OCR technology, multimodal artificial intelligence, a translation server, a display device, an emotion engine, etc.

[0850] The system operates under the assumption that a user wearing a smartglasses-type display device must recognize a string of characters in their field of vision. First, a camera mounted on the smartglasses captures images of the user's field of vision in real time. This image data is then processed by a terminal inside the smartglasses.

[0851] The device uses OCR technology to detect text from captured video. For example, it can recognize text written on signs and billboards in real time. The detected text information is then analyzed using multimodal artificial intelligence to extract only the important information. This extracted information filters out irrelevant information and provides only the necessary information.

[0852] After extracting the important information, it is sent over the network to a server for translation, which translates the received text into the specified language and sends the translation back to the device.

[0853] The translation results are displayed in real time on the smart glasses display, which is overlaid on the user's field of view in a natural way, allowing them to visually confirm the translated information.

[0854] Furthermore, this system is equipped with an emotion engine that recognizes the user's emotions. The emotion engine recognizes the user's emotional state by analyzing their voice and facial expressions. For example, it can determine the emotional state using voice input. Based on the user's emotions recognized by the emotion engine, the content and amount of information displayed is adjusted. For example, if the user is feeling stressed, the system reduces the amount of information displayed on the screen or provides information in a simpler format to reduce the user's burden.

[0855] As a specific example, consider a situation where a user is reading a sign at an airport in a foreign country. When the user wears smart glasses and looks at an airport sign, the camera captures the image and uses OCR technology to detect strings such as "Departures," "Arrivals," and "Restrooms." Multimodal AI extracts the information about "Departures" and "Restrooms," sends it over the network to a server for translation, and displays the translated information, including "Departing flights," "Arriving flights," and "Restrooms," on the smart glasses' display.

[0856] At the same time, the emotion engine detects tension in the user's facial expressions and voice, and if it determines that the user is feeling stressed, the system adjusts the display to show only important information, reducing the user's burden.If the user is in a hurry, the system will display only "Departing Flights" information in large letters on the screen, helping the user to quickly reach their destination.

[0857] As described above, the system of the present invention not only translates and displays information within the user's field of vision, but also adjusts the way information is presented depending on the user's emotional state, thereby providing a more comfortable and efficient user experience.

[0858] The processing flow will be explained below.

[0859] Step 1:

[0860] The device uses the camera mounted on the smart glasses to capture images of the user's field of view in real time. The camera module starts operating and continuously captures frames of the field of view at a constant frame rate.

[0861] Step 2:

[0862] The video data captured by the device is analyzed using OCR technology to detect character strings. The OCR library recognizes characters within the video frames and extracts them as text data.

[0863] Step 3:

[0864] The device uses multimodal artificial intelligence to extract only the important information from the strings detected by OCR technology, specifically identifying key information such as "Open" and "Closed" related to travel and business, and filtering out other unnecessary information.

[0865] Step 4:

[0866] The device sends the extracted important text information to the server via the network for translation. The device sends a translation request to the server using a network protocol (e.g., HTTP, HTTPS).

[0867] Step 5:

[0868] The server calls a translation API to translate the received text data into the specified language. The server's translation API (e.g., Google Cloud Translation API) translates the input text into the specified language in real time.

[0869] Step 6:

[0870] The server generates the translation result and sends it back to the terminal via the network. The translated text data is then sent to the terminal as a response.

[0871] Step 7:

[0872] The translation results received by the device are displayed in real time on the smart glasses display, and the display's rendering engine overlays the translation results at specific locations within the field of view.

[0873] Step 8:

[0874] The device runs an emotion engine to recognize the user's emotions. The emotion engine captures and analyzes the user's voice and facial expressions using a camera and microphone to recognize the user's emotional state.

[0875] Step 9:

[0876] The device adjusts the content and amount of information displayed based on the user's emotions as recognized by its emotion engine. For example, if the user is feeling stressed, the system reduces the amount of information displayed and simply displays only the important information.

[0877] As a specific example, a case will be described where a user sees a restaurant sign while walking through a city abroad.

[0878] 1. Step 1: The camera captures real-time footage of the restaurant sign.

[0879] 2. Step 2: Use OCR technology to detect strings such as "Restaurant", "Open", and "Hours: 11:00-23:00".

[0880] 3. Step 3: Multimodal AI extracts important information such as "Open" and "Hours: 11:00-23:00".

[0881] 4. Step 4: Send the extracted text information to the server and request translation.

[0882] 5. In step 5, the server calls the translation API and translates the text into the specified language. It generates translation results such as "Restaurant," "Open," and "Business hours: 11:00-23:00."

[0883] 6. Step 6: The translation result is sent back to the device.

[0884] 7. Step 7: The device displays the translation results on the smart glasses display.

[0885] 8. In step 8, the emotion engine analyzes whether the user is feeling stressed while looking at the sign. Based on their voice and facial expressions, it recognizes that the user is relaxed.

[0886] 9. Step 9: If the system recognizes that the user is relaxed, it displays a normal amount of information.

[0887] This process allows users to receive accurate information in real time, and also provides information that is optimally tailored to the user's emotional state.

[0888] Example 2

[0889] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0890] The present invention aims to enable users in foreign language speaking countries to quickly understand text in their field of view. However, existing systems have had problems such as the inclusion of irrelevant text in the process of translating and displaying information in their field of view, or the display of too much information, which can be a burden to the user. Furthermore, since there was no system with the function to adjust the displayed information according to the user's emotional state, the user experience was uniform, which could lead to stressful situations.

[0891] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0892] In this invention, the server includes means for detecting character strings in the field of view, means for extracting specific information from the detected character strings, means for translating the extracted information, means for recognizing the user's emotions and adjusting the content and amount of information to be displayed, and means for communicating with the server via a network for extracting and translating the information. This allows the user to quickly and stress-free understand foreign language character strings in the field of view by adjusting the displayed information according to the user's emotional state while eliminating irrelevant information.

[0893] "Strings within the field of view" refers to character information that exists within the range that the user can visually confirm.

[0894] "Means for detection" refers to devices or technologies that use cameras or sensors to recognize character strings from visual information.

[0895] "Means of extraction" refers to the techniques and methods for selecting the necessary information from the detected string of characters.

[0896] "Translation means" refers to software or algorithms used to convert extracted text into another language.

[0897] "Display device" means an electronic device used to display information to a user's field of vision, such as smart glasses or a display screen.

[0898] "Means for recognizing emotions" refers to software or devices that analyze the user's voice and facial expressions to determine the emotional state the user is feeling.

[0899] "Means for adjusting the content and amount of information" refers to a method for dynamically changing the type and amount of information displayed depending on the user's emotional state.

[0900] A "server" refers to a general-purpose computing device that communicates with terminals via a network and processes information and provides data.

[0901] "Network" refers to the infrastructure for data communication, including the Internet and local area networks.

[0902] "Multimodal AI" refers to an AI technology that integrates and analyzes information from multiple data sources and input formats (e.g., text, images, audio, etc.) to derive optimal results.

[0903] "Filtering" refers to the process of eliminating unnecessary or irrelevant information from a large amount of data and extracting only the necessary information.

[0904] "User" refers to a person who uses the system to obtain information within their field of view and check the translation results.

[0905] This invention provides a system that detects characters in a user's field of vision, extracts specific information from them, and displays the translated information. It also provides a system that recognizes the user's emotions and adjusts the displayed information accordingly. This system is realized by combining various elements, including a camera, optical character recognition technology (OCR), multimodal artificial intelligence, a translation server, a display device, and an emotion engine.

[0906] Hardware and Software Use

[0907] 1. Smart Glasses: The display device at the heart of this invention is a smart glass-type device worn by the user. Examples include Google Glass and Microsoft HoloLens. Smart glasses are equipped with a high-resolution camera that can capture images of the user's field of view in real time.

[0908] 2. On-device OCR technology: Video data is processed by on-device OCR technology (e.g., Google Cloud Vision API or Tesseract OCR) to detect text. The OCR technology analyzes text from the captured video and extracts it as digital data.

[0909] 3. On-device multimodal AI: The detected strings are analyzed by on-device multimodal AI (e.g., OpenAI's GPT, BERT) to extract only the important information. At this stage, irrelevant data is filtered out and only the necessary information is retained.

[0910] 4. Network communication: The extracted information is sent to the server via a network (Wi-Fi or mobile data communication). A secure communication protocol such as HTTPS is used.

[0911] 5. Translation server: The received information is translated into the specified language by a translation API (e.g., Google Translate API or DeepL API) running on the server. The translation result is then resent from the server to the device.

[0912] 6. Smart glasses display: The translation results are overlaid on the smart glasses display, allowing the user to visually confirm them in a way that blends naturally into their field of vision.

[0913] 7. Emotion engine: The device is equipped with an emotion engine (e.g., Microsoft Azure Emotion API or Affectiva) that analyzes the user's voice and facial expressions to recognize their emotional state.

[0914] 8. Adjustment of displayed information: The content and amount of displayed information is adjusted based on the user's emotional state. For example, if the user is feeling stressed, only important information will be displayed in a larger size and presented in a simpler format.

[0915] Specific examples

[0916] Example: Use at a foreign airport

[0917] A user puts on smart glasses at a foreign airport and sees signs such as "Departures" and "Restrooms."

[0918] The camera inside the device captures the image and detects the characters using OCR technology.

[0919] The device uses multimodal artificial intelligence to extract information on "Departures" and "Restrooms" and retains only the important information.

[0920] The terminal sends this information to the server.

[0921] The server translates the received information and sends the translation results back to the terminal.

[0922] The device displays the translation results on the smart glasses display.

[0923] The device's emotion engine analyzes the user's facial expressions and voice, and if it determines that the user is feeling stressed, it will display "Departures" information in large letters, allowing the user to quickly move to their destination.

[0924] Example prompts for generative AI models

[0925] "Please outline a system for use in airports. The system uses a camera, OCR technology, multimodal artificial intelligence, a translation server, a display device, and an emotion engine to detect text in the user's field of view, translate it, and present it to the user. It also adjusts the way the information is presented based on the user's emotional state."

[0926] As described above, the system of the present invention not only translates and displays information within the user's field of vision, but also adjusts the way information is presented depending on the user's emotional state, thereby providing a more comfortable and efficient user experience.

[0927] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0928] Step 1:

[0929] The user puts on the smart glasses.

[0930] Specific operation: The user turns on the smart glasses and puts them on. The smart glasses start up and prepare the camera and display functions.

[0931] Step 2:

[0932] The device captures video with its camera.

[0933] Input: Real-time video of what is in the user's field of view.

[0934] Specific operation: The camera built into the smart glasses captures images within the user's field of view in real time and stores them in the internal memory as video data.

[0935] Output: Captured video data.

[0936] Step 3:

[0937] The device uses OCR technology to detect text from the video.

[0938] Input: Captured video data.

[0939] Specific operation: Using OCR technology (e.g., Google Cloud Vision API or Tesseract OCR), character strings are analyzed from video data and extracted as text data.

[0940] Output: Detected string information.

[0941] Step 4:

[0942] The device uses multimodal artificial intelligence to extract important information.

[0943] Input: Detected string information.

[0944] Specific operation: Using multimodal artificial intelligence (e.g., OpenAI's GPT, BERT), it analyzes input string information, filters out irrelevant data, and extracts important information.

[0945] Output: Filtered important string information.

[0946] Step 5:

[0947] The terminal transmits the extracted information to the server.

[0948] Input: Important string information.

[0949] What it does: Encrypts the extracted information and sends it securely to a server using a network protocol (e.g., HTTPS).

[0950] Output: The string data sent to the server.

[0951] Step 6:

[0952] The server translates the information it receives.

[0953] Input: String data sent from the terminal.

[0954] Specific operation: Using a translation API on the server (e.g., Google Translate API or DeepL API), the received string data is translated into the specified language.

[0955] Output: The translated string data.

[0956] Step 7:

[0957] The server sends the translation results to the terminal.

[0958] Input: Translated string data.

[0959] Specific operation: The translation results are encrypted and sent to the terminal via the network.

[0960] Output: The translation result sent to your device.

[0961] Step 8:

[0962] The device displays the translation results on the smart glasses display.

[0963] Input: Translation results sent from the server.

[0964] Specific operation: The translated information is overlaid on the smart glasses display, which displays the information in a way that blends naturally into the user's field of vision.

[0965] Output: The translation result displayed in the user's field of view.

[0966] Step 9:

[0967] The terminal uses an emotion engine to recognize the user's emotion.

[0968] Input: User's facial expressions and voice data.

[0969] Specific operation: Using an emotion engine (e.g., Microsoft Azure Emotion API or Affectiva), the system analyzes the user's voice and facial expressions in real time to recognize their emotional state.

[0970] Output: The perceived emotional state of the user.

[0971] Step 10:

[0972] The terminal adjusts the displayed information based on the user's emotions.

[0973] Input: The perceived emotional state of the user and the currently displayed information.

[0974] Specific operation: If it determines that the user is feeling stressed, it adjusts the content and amount of information displayed. For example, it adjusts the display according to the user's condition, such as displaying only important information in a larger size.

[0975] Output: Adjusted display information.

[0976] The above processing steps allow the user to quickly and stresslessly understand foreign language character strings within their field of vision.

[0977] (Application example 2)

[0978] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0979] Conventional systems that detect and translate information within the field of view can overwhelm users, causing stress. In autonomous vehicles, drivers and passengers need appropriate information, but unnecessary information occupies their field of view, which can lead to confusion. Furthermore, in today's increasingly international world, there is a need for systems that can accurately provide information in different languages ​​in real time.

[0980] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0981] In this invention, the server includes means for detecting character strings in the field of view, means for extracting specific information, means for translating the information, and means for recognizing the user's emotions and adjusting the content and amount of information displayed. This allows the user to receive information in the field of view accurately and efficiently, and in autonomous vehicles in particular, it is possible to provide only the necessary information at the appropriate time, improving the user experience.

[0982] "In-view text" refers to all written text within the user's field of view.

[0983] "Means for extracting specific information" refers to a device or technology for extracting necessary and important information from the numerous character strings present within the field of view.

[0984] "Translation means" refers to a technique or device that converts text from one language to another.

[0985] "Means for displaying on a display device" refers to a device or method for providing processed information to a user in a visually perceptible form.

[0986] "Means for communicating with a server via a network" refers to technology or devices for sending and receiving data to and from a remote server using the Internet or other digital communication means.

[0987] "Means for recognizing the user's emotions and adjusting the content and amount of information displayed" refers to technology or devices for detecting the user's emotional state and determining the appropriate type and amount of information based on that.

[0988] "Multimodal artificial intelligence" refers to artificial intelligence technology that integrates and processes multiple modalities (input formats), such as audio, video, and text.

[0989] "Devices carried by the user" refers to devices that the user can carry with them at all times, such as smart glasses or smartphones.

[0990] This invention provides a system where a user uses a mobile device, such as smart glasses, to detect text in their field of view in real time, extract and translate specific information, and then adjust the content and amount of information displayed based on the user's emotional state.

[0991] System Configuration

[0992] Hardware

[0993] Smart glasses: These devices incorporate a camera, a display, and a communication module. The camera captures images within the user's field of view, and the display displays translated information.

[0994] Server: Equipped with a high-performance processor and large storage capacity, it runs OCR technology, translation functions, multimodal artificial intelligence, and emotion engines.

[0995] software

[0996] OpenCV: An image processing library for processing video captured by the smart glasses camera and detecting text within the field of view.

[0997] Pytesseract: A library that uses OCR technology to extract text from captured images.

[0998] Google Trans: A service for translating extracted strings into a specified language.

[0999] Emotiv SDK: A software engine for recognizing a user's emotional state in real time and processing that information.

[1000] System Operation

[1001] Smart glasses use a camera to capture images within the user's field of view in real time. The captured image data is then processed by a computer inside the smart glasses. Specifically, OpenCV is used to detect character strings from the image data, and Pytesseract is used to convert the character strings into text data. Multimodal artificial intelligence is then used to extract only the necessary information.

[1002] The extracted information is sent to a server via the network and translated using Google Trans. The translation results are then sent back to the smart glasses and displayed on the display. The display is overlaid in a way that fits naturally into the user's field of vision, allowing the translated information to be visually confirmed.

[1003] Furthermore, the Emotiv SDK analyzes the user's emotional state in real time, adjusting the content and amount of information displayed based on this. For example, if the user is nervous, only the most important information will be displayed in a larger size, reducing the user's stress.

[1004] Specific examples

[1005] As a concrete example, consider a situation where a driver or passenger in an autonomous vehicle is wearing smart glasses. When the driver sees a traffic sign such as "No U-Turn" or "Road Closed," the camera captures it and translates it into "No U-Turn" or "Road Closed" in real time. Furthermore, if the emotion engine detects that the driver is stressed, only the important information is displayed in a larger size, allowing the driver to drive safely without being distracted.

[1006] Prompt Sentence Examples

[1007] "Use the camera on the smart glasses to detect traffic signs within its field of view, translate them into Japanese, and display them. Then, write a program to customize the information displayed depending on the user's emotional state."

[1008] This invention enables users to receive information within their field of vision accurately and efficiently, and in self-driving vehicles in particular, it is possible to provide only the necessary information at the appropriate time, greatly improving safety and comfort.

[1009] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1010] Step 1:

[1011] The camera in the smart glasses captures the image in the user's field of view in real time. This camera image is the input data, and the captured image is sent to the device.

[1012] Step 2:

[1013] The device uses OpenCV to detect text from captured video. Specifically, it processes video data (input) with an image analysis algorithm and extracts text written on signs, billboards, etc. (output).

[1014] Step 3:

[1015] The device converts the extracted strings into text data using Pytesseract. In this step, the strings in the image (input) are converted into text data (output) using OCR technology.

[1016] Step 4:

[1017] The device uses multimodal artificial intelligence to extract only the important information from text data. Specifically, it analyzes the text data (input) with a natural language processing algorithm, filters out unnecessary information, and extracts the necessary information (output).

[1018] Step 5:

[1019] The device sends important information to the server, which translates it into the specified language. The translation engine processes the text information (input) and generates the translation result (output).

[1020] Step 6:

[1021] The server sends the translated information to the terminal. The translation result (input) is sent in a defined data format, and the terminal receives it (output).

[1022] Step 7:

[1023] The device uses the Emotiv SDK to analyze the user's emotional state. The device processes the user's voice and facial expression data (input) with an emotion analysis algorithm to determine the user's emotional state (output).

[1024] Step 8:

[1025] The device adjusts the content and format of the information displayed based on the user's emotional state. By combining the emotional state data (input) and the translation result data (input), the device displays information (output) optimized for the user.

[1026] Step 9:

[1027] The display device presents the optimized information to the user's field of view. This final step uses the data received from the device (input) to overlay information in a visually understandable format (output).

[1028] This processing flow allows users to obtain information within their field of vision efficiently and without stress. Particularly in self-driving vehicles, it makes it possible to provide only the necessary information at the appropriate time.

[1029] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1030] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1031] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1032] [Fourth embodiment]

[1033] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1034] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1035] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1036] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1037] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1038] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1039] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1040] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1041] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1042] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1043] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1044] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1045] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1046] This invention relates to a system that detects characters in the field of view, extracts specific information from them, and displays the translated information on a display device. This system is realized by combining a camera, OCR technology, multimodal artificial intelligence, a translation server, a display device, etc.

[1047] This system works by having the user wear smart glasses. The smart glasses are equipped with a camera that captures images of the user's field of view in real time. This image data is then processed by a device inside the smart glasses.

[1048] The device uses OCR technology to detect text from the captured video, allowing for real-time capture of text information from signs and other objects. The detected text is then analyzed using multimodal artificial intelligence to filter out unnecessary information. This filtering process allows only the most important information to be extracted.

[1049] After extracting the important information, the device sends it to the server via the network. The server uses a translation API to translate the received string into the specified language. The translated information is then sent back to the device.

[1050] The device then displays the translated information on the smart glasses display, which is overlaid naturally within the user's field of vision, allowing the user to instantly check the information.

[1051] As a concrete example, consider the case where a user sees a restaurant sign while walking through a city abroad. The camera in the smart glasses captures the sign's image, and OCR technology detects text such as "Restaurant," "Open," and "Hours: 11:00-23:00." Next, multimodal AI extracts key information such as "Open" and "Hours: 11:00-23:00." This information is sent to the server, where a translation process is performed, resulting in the translation "Restaurant," "Open," and "Hours: 11:00-23:00." The translation result is then sent back to the device and displayed on the smart glasses' display.

[1052] Users can also use voice input to display only specific information. For example, if they say, "Translate only business hours," the device will interpret the command using voice recognition technology and update the filtering criteria. As a result, only "Business hours: 11:00-23:00" will be displayed on the screen.

[1053] In this way, the system of the present invention provides an efficient and convenient means for users to understand visual information in real time. Since the system can translate and display information within the user's field of vision simply by wearing the smart glasses, without the need to operate a smartphone, it is extremely useful for overseas travel and business situations.

[1054] The processing flow will be explained below.

[1055] Step 1:

[1056] The device uses the camera mounted on the smart glasses to capture images of the user's field of view in real time. The camera module starts operating and continuously captures frames of the field of view at a constant frame rate.

[1057] Step 2:

[1058] The video data captured by the device is analyzed using OCR technology to detect character strings. The OCR library recognizes characters within the video frames and extracts them as text data.

[1059] Step 3:

[1060] The device uses multimodal artificial intelligence to extract only the important information from the strings detected by OCR technology, specifically identifying key information such as "Open" and "Closed" related to travel and business, and filtering out other unnecessary information.

[1061] Step 4:

[1062] The device sends the extracted important text information to the server via the network for translation. The device sends a translation request to the server using a network protocol (e.g., HTTP, HTTPS).

[1063] Step 5:

[1064] The server calls a translation API to translate the received text data into the specified language. The server's translation API (e.g., Google Cloud Translation API) translates the input text into the specified language in real time.

[1065] Step 6:

[1066] The server generates the translation result and sends it back to the terminal via the network. The translated text data is then sent to the terminal as a response.

[1067] Step 7:

[1068] The translation results received by the device are displayed in real time on the smart glasses display, and the display's rendering engine overlays the translation results at specific locations within the field of view.

[1069] Step 8:

[1070] The user issues a voice command, such as "translate only a specific sign," and the smart glasses' microphone collects the voice data, which is then converted into text using voice recognition technology.

[1071] Step 9:

[1072] The device analyzes the instructions received through voice recognition and updates the filtering conditions, adjusting the filter algorithm based on the voice command to display only specific information within the field of view.

[1073] This process allows users to visually understand important local information in real time, eliminating the need to operate a smartphone and allowing them to intuitively and quickly obtain information through smart glasses, making travel and business more convenient.

[1074] Example 1

[1075] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1076] In modern society, there is a demand for quick and accurate understanding of visual information, especially when traveling or conducting business in areas where different languages ​​are spoken. However, conventional technologies require the use of devices such as smartphones, which can reduce user convenience. Furthermore, necessary information is often not readily available. An efficient and convenient solution to these problems is needed.

[1077] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1078] In this invention, the server includes means for capturing text within the user's field of view in real time with a camera in a visual device worn by the user, means for detecting text from the captured video data using optical character recognition technology, means for using multimodal artificial intelligence to extract important information from the detected text, means for transmitting the extracted important information to a remote computer device via a network for translation, and means for receiving the translated information again and displaying it on a display device within the user's field of view, thereby enabling the user to understand visual information in real time and acquire information quickly and accurately even in different language environments.

[1079] A "visual device" is a device worn by a user to capture and display information within the user's field of vision, such as a glasses-type device.

[1080] Optical character recognition technology (OCR technology) is a technology that recognizes characters from an image and extracts them as character data.

[1081] "Multimodal AI" is an AI technology that analyzes multiple types of data (e.g., text, images, audio, etc.) and extracts meaningful information from it.

[1082] A "network" is a communications infrastructure that enables the transmission and reception of data, including the Internet and local area networks (LANs).

[1083] A "remote computing device" is a computer system connected via a network that primarily processes and stores data.

[1084] "Translation" is the process of converting text written in one language into a different language.

[1085] A "display" is a screen mounted on a visual device, and is a means by which a user visually confirms information.

[1086] "Real-time" means that data is processed and displayed with little to no consistent delay.

[1087] "Important information" is information that is determined to be useful to the user from among the information acquired by the visual device.

[1088] "Speech recognition technology" is a technology that analyzes speech and understands it as text or commands.

[1089] This invention relates to a system that uses a user-worn visual device to detect text within the user's field of view, extract specific information from it, and display the translated information. This system is realized by combining a camera, optical character recognition technology (OCR technology), multimodal artificial intelligence, a translation server, and a display device.

[1090] This system works by having the user wear a pair of glasses-type visual devices. The devices are equipped with a camera that captures images of the user's field of vision in real time. This image data is then processed by a terminal inside the device.

[1091] The device detects text from the captured video using optical character recognition (OCR) technology. The OCR technology can utilize the API of a general OCR technology provider. This allows text information from signs and other objects to be acquired in real time.

[1092] The device then analyzes the detected text using multimodal AI, such as advanced natural language processing models, to filter out unnecessary information, extracting only the important information.

[1093] After extracting the important information, the device sends it to the server via the network. The server uses a translation API to translate the received string into the specified language. The translated information is then sent back to the device.

[1094] The terminal then displays the received translation information on the display of the visual device, which is overlaid in a natural way within the user's field of vision, allowing the user to instantly confirm the information.

[1095] As a concrete example, consider a case where a user sees a restaurant sign while walking through a city abroad. The camera in the visual device captures the sign's image, and OCR technology detects text such as "Restaurant," "Open," and "Hours: 11:00-23:00." Next, multimodal artificial intelligence extracts key information such as "Open" and "Hours: 11:00-23:00." This information is sent to the server, where a translation process is performed, resulting in the translation results of "Restaurant," "Open," and "Hours: 11:00-23:00." The translation results are then sent back to the device and displayed on the visual device's display.

[1096] Users can also use voice input to display only specific information. For example, if they say, "Translate only business hours," the device will interpret the command using voice recognition technology and update the filtering criteria. As a result, only "Business hours: 11:00-23:00" will be displayed on the screen.

[1097] Here are some examples of prompts to input to the generative AI model:

[1098] "Please translate the opening hours displayed on the restaurant sign."

[1099] "Please translate and display the text of signs viewed through a visual device in real time."

[1100] In this way, the system of the present invention provides an efficient and convenient means for users to understand visual information in real time. By simply wearing the visual device, information within the user's field of vision can be translated and displayed, making it particularly useful in multilingual environments.

[1101] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1102] Processing flow

[1103] Step 1: Capture what's in your field of view

[1104] When a user wears a vision device, the camera built into the vision device captures images within the user's field of view in real time. The input is the image data acquired through the camera of the vision device, and the output is the image data. This data is sent to the terminal inside the vision device. Specifically, the camera sensor continuously captures image frames and stores them in memory.

[1105] Step 2: Character string detection (OCR processing)

[1106] The device processes the video data acquired in step 1 using optical character recognition (OCR) technology. The input is the captured video data, and the output is all the text data contained in the video. Specifically, the OCR software analyzes the video frames and returns the recognized text in a list format. For example, if a video of a sign is input, the output will be "Restaurant," "Open," and "Hours: 11:00-23:00."

[1107] Step 3: Information filtering (extracting important information)

[1108] The device uses multimodal AI to analyze the string data detected in step 2 and extract only the important information. The input is the string data obtained through OCR processing, and the output is data containing only the important information. Specifically, the AI ​​analyzes the string data, eliminates unimportant information such as "Restaurant," and extracts only important information such as "Open" and "Hours: 11:00-23:00."

[1109] Step 4: Submit a translation request

[1110] The terminal sends the important information obtained in step 3 to the server via the network. The input is the important information data, and the output is the request sent to the server. Specifically, the important information is converted into a standard format such as JSON format and sent to the server using the HTTP protocol.

[1111] Step 5: Translation process

[1112] The server translates the important information received in step 4 into the specified language using the translation API. The input is the important information data, and the output is the translated information data. Specifically, the server sends a request to the translation API and processes the received translation results. For example, "Open" and "Hours: 11:00-23:00" are translated into "Open" and "Business hours: 11:00-23:00".

[1113] Step 6: Receive and display the translation results

[1114] The terminal receives the information data translated in step 5 via the network and displays it on the display of the visual device. The input is the translated information data, and the output is the information displayed on the display of the visual device. The specific operation is to receive data from the network, transfer it to the display unit, and overlay it on the display.

[1115] These processing steps allow a user to quickly and accurately translate and display information within their field of vision simply by wearing the visual device.

[1116] (Application example 1)

[1117] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1118] To address the issue of users being unable to immediately understand the information they need due to language barriers in brick-and-mortar stores abroad, it is necessary to provide a means for users to smoothly access information regardless of language. There is also a need for a method to quickly and accurately filter and display specific information based on the user's voice commands.

[1119] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1120] In this invention, the server includes means for detecting character strings within the field of view, means for extracting specific information from the detected character strings, means for translating the extracted information, means for filtering the specific information based on a user's voice input, means for displaying the translated information on a display device, and means for communicating with the server via a network for extracting and translating the information, thereby enabling a user to translate information on product labels and guide signs in a physical store in real time and quickly filter and display specific information based on a voice command.

[1121] The "means for detecting character strings within the field of view" is a technology that uses a video capture device such as a camera to capture text within the user's field of view in real time.

[1122] "Means for extracting specific information from the detected character string" refers to a method of selecting and extracting only important information from the detected character string using optical character recognition (OCR).

[1123] The "means for translating the extracted information" refers to a technique that uses a translation server or a translation API to convert the extracted information into a different language.

[1124] The "means for filtering specific information based on voice input by the user" is a technology that recognizes the user's voice instructions and selects only the necessary information in accordance with those instructions.

[1125] The "means for displaying translated information on a display device" refers to technology that displays the translated text on smart glasses or a display so that the user can see it.

[1126] The "means for communicating with a server via a network for extracting and translating the information" refers to a method for transmitting and receiving data to and from a remote server via a network such as the Internet to extract and translate information.

[1127] The present invention provides a system for enabling users to translate and understand visual information in real time. This system is particularly useful for enabling users to quickly understand product labels and signage in brick-and-mortar stores abroad. Specific embodiments for implementing the present invention are described below.

[1128] Hardware and Software Configuration

[1129] The system includes the following major components:

[1130] 1. Smart glasses: Devices with built-in cameras and displays that are used to display information in the user's field of vision. Typical examples include Google Glass and Vuzix.

[1131] 2. Camera module: Uses the camera built into the smart glasses to capture images within the user's field of view in real time.

[1132] 3. OCR module: Optical character recognition (OCR) technology to extract text from captured video data. Tesseract OCR is used as an example.

[1133] 4. Speech Recognition Module: Technology for recognizing and analyzing voice commands from the user. Google Speech API is used.

[1134] 5. Multimodal artificial intelligence: AI technology for filtering necessary information and removing unnecessary information from text extracted by OCR.

[1135] 6. Translation server: The data is sent to the server via the network and translated into the specified language using the translation API.

[1136] 7. Display module: A module for displaying translated information on the smart glasses display.

[1137] Program processing and data flow

[1138] 1. Capture camera footage:

[1139] The camera module in the smart glasses captures real-time information about signs and product labels in physical stores within the user's field of view, and this video data is processed within the device.

[1140] 2. Character Recognition (OCR):

[1141] The OCR module detects and extracts text from the captured video data. The OCR technology used is Tesseract OCR.

[1142] 3. Recognition of voice commands:

[1143] The user can give voice instructions through the smart glasses' microphone to display only specific information, and voice recognition is performed using the Google Speech API.

[1144] 4. Information filtering:

[1145] Multimodal AI filters unnecessary information from the extracted text and extracts important information, taking into account filtering conditions specified by voice commands.

[1146] 5. Translation:

[1147] The filtered information is sent to a translation server via the network and translated into the specified language using a translation API.

[1148] 6. Displaying Information:

[1149] The translated information is sent back to the device via the network and displayed on the smart glasses display via the display module, where it is naturally overlaid on the user's field of view.

[1150] Example and prompt for generative AI model

[1151] Consider a situation where a user is searching for a product in a supermarket in a foreign country. If the user gives a voice command such as "translate only the ingredient information," the system extracts ingredient information from the product label, translates it into the specified language, and displays it on the smart glasses display.

[1152] Example prompt for a generative AI model:

[1153] "From the text detected from the camera footage, extract only the parts that contain the keyword 'ingredients', translate them into Japanese, and display them on the screen."

[1154] In this way, the present invention provides an effective means for users to quickly and accurately understand the information they need in a physical store without experiencing a language barrier.

[1155] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1156] Step 1:

[1157] Camera footage capture

[1158] The device (smartglasses camera module) captures real-time images of store signs and product labels within the user's field of view.

[1159] Input: Video within the field of view of a physical store

[1160] Output: Captured video data

[1161] Step 2:

[1162] Character Recognition (OCR)

[1163] The device uses an OCR module (Tesseract OCR) to detect and extract text from the captured video data.

[1164] Input: Captured video data

[1165] Output: The extracted string

[1166] Step 3:

[1167] Recognizing voice commands

[1168] The user gives voice instructions through the microphone in the smart glasses, and the device analyzes the voice instructions using a voice recognition module (Google Speech API).

[1169] Input: User's voice commands

[1170] Output: Parsed voice command

[1171] Step 4:

[1172] Information Filtering

[1173] The device uses multimodal artificial intelligence to filter out unnecessary information from the character strings extracted by OCR and extract specific information in response to the user's voice instructions.

[1174] Input: Extracted string, parsed voice command

[1175] Output: Filtered text containing important information

[1176] Step 5:

[1177] translation

[1178] The filtered information is sent over the network to a translation server, which translates it into the specified language. The server uses a translation API to translate the filtered text.

[1179] Input: filtered text

[1180] Output: Translated information

[1181] Step 6:

[1182] Displaying Information

[1183] The translated information is then sent back to the device and displayed on the smart glasses display via the display module, allowing the user to see the translated information as an overlay in their field of vision.

[1184] Input: Translated information

[1185] Output: Translation information displayed on the smart glasses display

[1186] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1187] This invention relates to a system that detects character strings in the field of view, extracts specific information from them, and displays the translated information on a display device, and also to a system that combines an emotion engine that recognizes the user's emotions. This system is realized by combining a camera, OCR technology, multimodal artificial intelligence, a translation server, a display device, an emotion engine, etc.

[1188] The system operates under the assumption that a user wearing a smartglasses-type display device must recognize a string of characters in their field of vision. First, a camera mounted on the smartglasses captures images of the user's field of vision in real time. This image data is then processed by a terminal inside the smartglasses.

[1189] The device uses OCR technology to detect text from captured video. For example, it can recognize text written on signs and billboards in real time. The detected text information is then analyzed using multimodal artificial intelligence to extract only the important information. This extracted information filters out irrelevant information and provides only the necessary information.

[1190] After extracting the important information, it is sent over the network to a server for translation, which translates the received text into the specified language and sends the translation back to the device.

[1191] The translation results are displayed in real time on the smart glasses display, which is overlaid on the user's field of view in a natural way, allowing them to visually confirm the translated information.

[1192] Furthermore, this system is equipped with an emotion engine that recognizes the user's emotions. The emotion engine recognizes the user's emotional state by analyzing their voice and facial expressions. For example, it can determine the emotional state using voice input. Based on the user's emotions recognized by the emotion engine, the content and amount of information displayed is adjusted. For example, if the user is feeling stressed, the system reduces the amount of information displayed on the screen or provides information in a simpler format to reduce the user's burden.

[1193] As a specific example, consider a situation where a user is reading a sign at an airport in a foreign country. When the user wears smart glasses and looks at an airport sign, the camera captures the image and uses OCR technology to detect strings such as "Departures," "Arrivals," and "Restrooms." Multimodal AI extracts the information about "Departures" and "Restrooms," sends it over the network to a server for translation, and displays the translated information, including "Departing flights," "Arriving flights," and "Restrooms," on the smart glasses' display.

[1194] At the same time, the emotion engine detects tension in the user's facial expressions and voice, and if it determines that the user is feeling stressed, the system adjusts the display to show only important information, reducing the user's burden.If the user is in a hurry, the system will display only "Departing Flights" information in large letters on the screen, helping the user to quickly reach their destination.

[1195] As described above, the system of the present invention not only translates and displays information within the user's field of vision, but also adjusts the way information is presented depending on the user's emotional state, thereby providing a more comfortable and efficient user experience.

[1196] The processing flow will be explained below.

[1197] Step 1:

[1198] The device uses the camera mounted on the smart glasses to capture images of the user's field of view in real time. The camera module starts operating and continuously captures frames of the field of view at a constant frame rate.

[1199] Step 2:

[1200] The video data captured by the device is analyzed using OCR technology to detect character strings. The OCR library recognizes characters within the video frames and extracts them as text data.

[1201] Step 3:

[1202] The device uses multimodal artificial intelligence to extract only the important information from the strings detected by OCR technology, specifically identifying key information such as "Open" and "Closed" related to travel and business, and filtering out other unnecessary information.

[1203] Step 4:

[1204] The device sends the extracted important text information to the server via the network for translation. The device sends a translation request to the server using a network protocol (e.g., HTTP, HTTPS).

[1205] Step 5:

[1206] The server calls a translation API to translate the received text data into the specified language. The server's translation API (e.g., Google Cloud Translation API) translates the input text into the specified language in real time.

[1207] Step 6:

[1208] The server generates the translation result and sends it back to the terminal via the network. The translated text data is then sent to the terminal as a response.

[1209] Step 7:

[1210] The translation results received by the device are displayed in real time on the smart glasses display, and the display's rendering engine overlays the translation results at specific locations within the field of view.

[1211] Step 8:

[1212] The device runs an emotion engine to recognize the user's emotions. The emotion engine captures and analyzes the user's voice and facial expressions using a camera and microphone to recognize the user's emotional state.

[1213] Step 9:

[1214] The device adjusts the content and amount of information displayed based on the user's emotions as recognized by its emotion engine. For example, if the user is feeling stressed, the system reduces the amount of information displayed and simply displays only the important information.

[1215] As a specific example, a case will be described where a user sees a restaurant sign while walking through a city abroad.

[1216] 1. Step 1: The camera captures real-time footage of the restaurant sign.

[1217] 2. Step 2: Use OCR technology to detect strings such as "Restaurant", "Open", and "Hours: 11:00-23:00".

[1218] 3. Step 3: Multimodal AI extracts important information such as "Open" and "Hours: 11:00-23:00".

[1219] 4. Step 4: Send the extracted text information to the server and request translation.

[1220] 5. In step 5, the server calls the translation API and translates the text into the specified language. It generates translation results such as "Restaurant," "Open," and "Business hours: 11:00-23:00."

[1221] 6. Step 6: The translation result is sent back to the device.

[1222] 7. Step 7: The device displays the translation results on the smart glasses display.

[1223] 8. In step 8, the emotion engine analyzes whether the user is feeling stressed while looking at the sign. Based on their voice and facial expressions, it recognizes that the user is relaxed.

[1224] 9. Step 9: If the system recognizes that the user is relaxed, it displays a normal amount of information.

[1225] This process allows users to receive accurate information in real time, and also provides information that is optimally tailored to the user's emotional state.

[1226] Example 2

[1227] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1228] The present invention aims to enable users in foreign language speaking countries to quickly understand text in their field of view. However, existing systems have had problems such as the inclusion of irrelevant text in the process of translating and displaying information in their field of view, or the display of too much information, which can be a burden to the user. Furthermore, since there was no system with the function to adjust the displayed information according to the user's emotional state, the user experience was uniform, which could lead to stressful situations.

[1229] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1230] In this invention, the server includes means for detecting character strings in the field of view, means for extracting specific information from the detected character strings, means for translating the extracted information, means for recognizing the user's emotions and adjusting the content and amount of information to be displayed, and means for communicating with the server via a network for extracting and translating the information. This allows the user to quickly and stress-free understand foreign language character strings in the field of view by adjusting the displayed information according to the user's emotional state while eliminating irrelevant information.

[1231] "Strings within the field of view" refers to character information that exists within the range that the user can visually confirm.

[1232] "Means for detection" refers to devices or technologies that use cameras or sensors to recognize character strings from visual information.

[1233] "Means of extraction" refers to the techniques and methods for selecting the necessary information from the detected string of characters.

[1234] "Translation means" refers to software or algorithms used to convert extracted text into another language.

[1235] "Display device" means an electronic device used to display information to a user's field of vision, such as smart glasses or a display screen.

[1236] "Means for recognizing emotions" refers to software or devices that analyze the user's voice and facial expressions to determine the emotional state the user is feeling.

[1237] "Means for adjusting the content and amount of information" refers to a method for dynamically changing the type and amount of information displayed depending on the user's emotional state.

[1238] A "server" refers to a general-purpose computing device that communicates with terminals via a network and processes information and provides data.

[1239] "Network" refers to the infrastructure for data communication, including the Internet and local area networks.

[1240] "Multimodal AI" refers to an AI technology that integrates and analyzes information from multiple data sources and input formats (e.g., text, images, audio, etc.) to derive optimal results.

[1241] "Filtering" refers to the process of eliminating unnecessary or irrelevant information from a large amount of data and extracting only the necessary information.

[1242] "User" refers to a person who uses the system to obtain information within their field of view and check the translation results.

[1243] This invention provides a system that detects characters in a user's field of vision, extracts specific information from them, and displays the translated information. It also provides a system that recognizes the user's emotions and adjusts the displayed information accordingly. This system is realized by combining various elements, including a camera, optical character recognition technology (OCR), multimodal artificial intelligence, a translation server, a display device, and an emotion engine.

[1244] Hardware and Software Use

[1245] 1. Smart Glasses: The display device at the heart of this invention is a smart glass-type device worn by the user. Examples include Google Glass and Microsoft HoloLens. Smart glasses are equipped with a high-resolution camera that can capture images of the user's field of view in real time.

[1246] 2. On-device OCR technology: Video data is processed by on-device OCR technology (e.g., Google Cloud Vision API or Tesseract OCR) to detect text. The OCR technology analyzes text from the captured video and extracts it as digital data.

[1247] 3. On-device multimodal AI: The detected strings are analyzed by on-device multimodal AI (e.g., OpenAI's GPT, BERT) to extract only the important information. At this stage, irrelevant data is filtered out and only the necessary information is retained.

[1248] 4. Network communication: The extracted information is sent to the server via a network (Wi-Fi or mobile data communication). A secure communication protocol such as HTTPS is used.

[1249] 5. Translation server: The received information is translated into the specified language by a translation API (e.g., Google Translate API or DeepL API) running on the server. The translation result is then resent from the server to the device.

[1250] 6. Smart glasses display: The translation results are overlaid on the smart glasses display, allowing the user to visually confirm them in a way that blends naturally into their field of vision.

[1251] 7. Emotion engine: The device is equipped with an emotion engine (e.g., Microsoft Azure Emotion API or Affectiva) that analyzes the user's voice and facial expressions to recognize their emotional state.

[1252] 8. Adjustment of displayed information: The content and amount of displayed information is adjusted based on the user's emotional state. For example, if the user is feeling stressed, only important information will be displayed in a larger size and presented in a simpler format.

[1253] Specific examples

[1254] Example: Use at a foreign airport

[1255] A user puts on smart glasses at a foreign airport and sees signs such as "Departures" and "Restrooms."

[1256] The camera inside the device captures the image and detects the characters using OCR technology.

[1257] The device uses multimodal artificial intelligence to extract information on "Departures" and "Restrooms" and retains only the important information.

[1258] The terminal sends this information to the server.

[1259] The server translates the received information and sends the translation results back to the terminal.

[1260] The device displays the translation results on the smart glasses display.

[1261] The device's emotion engine analyzes the user's facial expressions and voice, and if it determines that the user is feeling stressed, it will display "Departures" information in large letters, allowing the user to quickly move to their destination.

[1262] Example prompts for generative AI models

[1263] "Please outline a system for use in airports. The system uses a camera, OCR technology, multimodal artificial intelligence, a translation server, a display device, and an emotion engine to detect text in the user's field of view, translate it, and present it to the user. It also adjusts the way the information is presented based on the user's emotional state."

[1264] As described above, the system of the present invention not only translates and displays information within the user's field of vision, but also adjusts the way information is presented depending on the user's emotional state, thereby providing a more comfortable and efficient user experience.

[1265] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1266] Step 1:

[1267] The user puts on the smart glasses.

[1268] Specific operation: The user turns on the smart glasses and puts them on. The smart glasses start up and prepare the camera and display functions.

[1269] Step 2:

[1270] The device captures video with its camera.

[1271] Input: Real-time video of what is in the user's field of view.

[1272] Specific operation: The camera built into the smart glasses captures images within the user's field of view in real time and stores them in the internal memory as video data.

[1273] Output: Captured video data.

[1274] Step 3:

[1275] The device uses OCR technology to detect text from the video.

[1276] Input: Captured video data.

[1277] Specific operation: Using OCR technology (e.g., Google Cloud Vision API or Tesseract OCR), character strings are analyzed from video data and extracted as text data.

[1278] Output: Detected string information.

[1279] Step 4:

[1280] The device uses multimodal artificial intelligence to extract important information.

[1281] Input: Detected string information.

[1282] Specific operation: Using multimodal artificial intelligence (e.g., OpenAI's GPT, BERT), it analyzes input string information, filters out irrelevant data, and extracts important information.

[1283] Output: Filtered important string information.

[1284] Step 5:

[1285] The terminal transmits the extracted information to the server.

[1286] Input: Important string information.

[1287] What it does: Encrypts the extracted information and sends it securely to a server using a network protocol (e.g., HTTPS).

[1288] Output: The string data sent to the server.

[1289] Step 6:

[1290] The server translates the information it receives.

[1291] Input: String data sent from the terminal.

[1292] Specific operation: Using a translation API on the server (e.g., Google Translate API or DeepL API), the received string data is translated into the specified language.

[1293] Output: The translated string data.

[1294] Step 7:

[1295] The server sends the translation results to the terminal.

[1296] Input: Translated string data.

[1297] Specific operation: The translation results are encrypted and sent to the terminal via the network.

[1298] Output: The translation result sent to your device.

[1299] Step 8:

[1300] The device displays the translation results on the smart glasses display.

[1301] Input: Translation results sent from the server.

[1302] Specific operation: The translated information is overlaid on the smart glasses display, which displays the information in a way that blends naturally into the user's field of vision.

[1303] Output: The translation result displayed in the user's field of view.

[1304] Step 9:

[1305] The terminal uses an emotion engine to recognize the user's emotion.

[1306] Input: User's facial expressions and voice data.

[1307] Specific operation: Using an emotion engine (e.g., Microsoft Azure Emotion API or Affectiva), the system analyzes the user's voice and facial expressions in real time to recognize their emotional state.

[1308] Output: The perceived emotional state of the user.

[1309] Step 10:

[1310] The terminal adjusts the displayed information based on the user's emotions.

[1311] Input: The perceived emotional state of the user and the currently displayed information.

[1312] Specific operation: If it determines that the user is feeling stressed, it adjusts the content and amount of information displayed. For example, it adjusts the display according to the user's condition, such as displaying only important information in a larger size.

[1313] Output: Adjusted display information.

[1314] The above processing steps allow the user to quickly and stresslessly understand foreign language character strings within their field of vision.

[1315] (Application example 2)

[1316] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1317] Conventional systems that detect and translate information within the field of view can overwhelm users, causing stress. In autonomous vehicles, drivers and passengers need appropriate information, but unnecessary information occupies their field of view, which can lead to confusion. Furthermore, in today's increasingly international world, there is a need for systems that can accurately provide information in different languages ​​in real time.

[1318] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1319] In this invention, the server includes means for detecting character strings in the field of view, means for extracting specific information, means for translating the information, and means for recognizing the user's emotions and adjusting the content and amount of information displayed. This allows the user to receive information in the field of view accurately and efficiently, and in autonomous vehicles in particular, it is possible to provide only the necessary information at the appropriate time, improving the user experience.

[1320] "In-view text" refers to all written text within the user's field of view.

[1321] "Means for extracting specific information" refers to a device or technology for extracting necessary and important information from the numerous character strings present within the field of view.

[1322] "Translation means" refers to a technique or device that converts text from one language to another.

[1323] "Means for displaying on a display device" refers to a device or method for providing processed information to a user in a visually perceptible form.

[1324] "Means for communicating with a server via a network" refers to technology or devices for sending and receiving data to and from a remote server using the Internet or other digital communication means.

[1325] "Means for recognizing the user's emotions and adjusting the content and amount of information displayed" refers to technology or devices for detecting the user's emotional state and determining the appropriate type and amount of information based on that.

[1326] "Multimodal artificial intelligence" refers to artificial intelligence technology that integrates and processes multiple modalities (input formats), such as audio, video, and text.

[1327] "Devices carried by the user" refers to devices that the user can carry with them at all times, such as smart glasses or smartphones.

[1328] This invention provides a system where a user uses a mobile device, such as smart glasses, to detect text in their field of view in real time, extract and translate specific information, and then adjust the content and amount of information displayed based on the user's emotional state.

[1329] System Configuration

[1330] Hardware

[1331] Smart glasses: These devices incorporate a camera, a display, and a communication module. The camera captures images within the user's field of view, and the display displays translated information.

[1332] Server: Equipped with a high-performance processor and large storage capacity, it runs OCR technology, translation functions, multimodal artificial intelligence, and emotion engines.

[1333] software

[1334] OpenCV: An image processing library for processing video captured by the smart glasses camera and detecting text within the field of view.

[1335] Pytesseract: A library that uses OCR technology to extract text from captured images.

[1336] Google Trans: A service for translating extracted strings into a specified language.

[1337] Emotiv SDK: A software engine for recognizing a user's emotional state in real time and processing that information.

[1338] System Operation

[1339] Smart glasses use a camera to capture images within the user's field of view in real time. The captured image data is then processed by a computer inside the smart glasses. Specifically, OpenCV is used to detect character strings from the image data, and Pytesseract is used to convert the character strings into text data. Multimodal artificial intelligence is then used to extract only the necessary information.

[1340] The extracted information is sent to a server via the network and translated using Google Trans. The translation results are then sent back to the smart glasses and displayed on the display. The display is overlaid in a way that fits naturally into the user's field of vision, allowing the translated information to be visually confirmed.

[1341] Furthermore, the Emotiv SDK analyzes the user's emotional state in real time, adjusting the content and amount of information displayed based on this. For example, if the user is nervous, only the most important information will be displayed in a larger size, reducing the user's stress.

[1342] Specific examples

[1343] As a concrete example, consider a situation where a driver or passenger in an autonomous vehicle is wearing smart glasses. When the driver sees a traffic sign such as "No U-Turn" or "Road Closed," the camera captures it and translates it into "No U-Turn" or "Road Closed" in real time. Furthermore, if the emotion engine detects that the driver is stressed, only the important information is displayed in a larger size, allowing the driver to drive safely without being distracted.

[1344] Prompt Sentence Examples

[1345] "Use the camera on the smart glasses to detect traffic signs within its field of view, translate them into Japanese, and display them. Then, write a program to customize the information displayed depending on the user's emotional state."

[1346] This invention enables users to receive information within their field of vision accurately and efficiently, and in self-driving vehicles in particular, it is possible to provide only the necessary information at the appropriate time, greatly improving safety and comfort.

[1347] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1348] Step 1:

[1349] The camera in the smart glasses captures the image in the user's field of view in real time. This camera image is the input data, and the captured image is sent to the device.

[1350] Step 2:

[1351] The device uses OpenCV to detect text from captured video. Specifically, it processes video data (input) with an image analysis algorithm and extracts text written on signs, billboards, etc. (output).

[1352] Step 3:

[1353] The device converts the extracted strings into text data using Pytesseract. In this step, the strings in the image (input) are converted into text data (output) using OCR technology.

[1354] Step 4:

[1355] The device uses multimodal artificial intelligence to extract only the important information from text data. Specifically, it analyzes the text data (input) with a natural language processing algorithm, filters out unnecessary information, and extracts the necessary information (output).

[1356] Step 5:

[1357] The device sends important information to the server, which translates it into the specified language. The translation engine processes the text information (input) and generates the translation result (output).

[1358] Step 6:

[1359] The server sends the translated information to the terminal. The translation result (input) is sent in a defined data format, and the terminal receives it (output).

[1360] Step 7:

[1361] The device uses the Emotiv SDK to analyze the user's emotional state. The device processes the user's voice and facial expression data (input) with an emotion analysis algorithm to determine the user's emotional state (output).

[1362] Step 8:

[1363] The device adjusts the content and format of the information displayed based on the user's emotional state. By combining the emotional state data (input) and the translation result data (input), the device displays information (output) optimized for the user.

[1364] Step 9:

[1365] The display device presents the optimized information to the user's field of view. This final step uses the data received from the device (input) to overlay information in a visually understandable format (output).

[1366] This processing flow allows users to obtain information within their field of vision efficiently and without stress. Particularly in self-driving vehicles, it makes it possible to provide only the necessary information at the appropriate time.

[1367] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1368] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1369] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1370] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1371] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1372] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1373] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1374] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1375] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1376] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1377] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1378] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1379] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1380] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1381] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1382] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1383] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1384] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1385] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1386] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1387] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1388] The following is further disclosed regarding the above embodiment.

[1389] (Claim 1)

[1390] means for detecting a string of characters in a field of view;

[1391] means for extracting specific information from the detected string;

[1392] means for translating the extracted information;

[1393] means for displaying the translated information on a display device;

[1394] The system includes means for communicating with a server over a network for extracting and translating said information.

[1395] (Claim 2)

[1396] 10. The system of claim 1, wherein the information extraction means uses multimodal artificial intelligence to filter out unnecessary information.

[1397] (Claim 3)

[1398] 2. The system according to claim 1, wherein the display device is mounted on a device carried by a user.

[1399] "Example 1"

[1400] (Claim 1)

[1401] a means for capturing in real time a character string within a field of view using a camera of a visual device worn by a user;

[1402] means for detecting a character string from the captured video data using optical character recognition technology;

[1403] a means for using multimodal artificial intelligence to extract important information from the detected strings of characters;

[1404] means for transmitting the extracted important information to a remote computer device via a network and requesting translation;

[1405] The system includes means for receiving the translated information again and displaying it on a display device that is displayed within the user's field of view.

[1406] (Claim 2)

[1407] The system according to claim 1, characterized in that it utilizes speech recognition technology to filter information based on conditions when extracting important information and requesting translation.

[1408] (Claim 3)

[1409] 2. The system according to claim 1, wherein the visual device is a glasses-type device carried by a user, and information is displayed on a display of the visual device.

[1410] "Application Example 1"

[1411] (Claim 1)

[1412] means for detecting a string of characters in a field of view;

[1413] means for extracting specific information from the detected string;

[1414] means for translating the extracted information;

[1415] means for filtering certain information based on voice input by a user;

[1416] means for displaying the translated information on a display device;

[1417] The system includes means for communicating with a server over a network for extracting and translating said information.

[1418] (Claim 2)

[1419] 10. The system of claim 1, wherein the information extraction means uses multimodal artificial intelligence to filter out unnecessary information.

[1420] (Claim 3)

[1421] 2. The system according to claim 1, wherein the display device is mounted on a device carried by a user.

[1422] "Example 2: Combining Emotion Engines"

[1423] (Claim 1)

[1424] means for detecting a string of characters in a field of view;

[1425] means for extracting specific information from the detected string;

[1426] means for translating the extracted information;

[1427] means for displaying the translated information on a display device;

[1428] A means for recognizing a user's emotions and adjusting the content and amount of information to be displayed;

[1429] The system includes means for communicating with a server over a network for extracting and translating said information.

[1430] (Claim 2)

[1431] 10. The system of claim 1, wherein the information extraction means uses multimodal artificial intelligence to filter out unnecessary information.

[1432] (Claim 3)

[1433] 2. The system according to claim 1, wherein the display device is mounted on a device carried by a user.

[1434] "Application example 2 when combining emotion engines"

[1435] (Claim 1)

[1436] means for detecting a string of characters in a field of view;

[1437] means for extracting specific information from the detected string;

[1438] means for translating the extracted information;

[1439] means for displaying the translated information on a display device;

[1440] means for communicating with a server over a network for extracting and translating said information;

[1441] A means for recognizing the user's emotions and adjusting the content and amount of information displayed;

[1442] A system including:

[1443] (Claim 2)

[1444] 10. The system of claim 1, wherein the information extraction means uses multimodal artificial intelligence to filter out unnecessary information.

[1445] (Claim 3)

[1446] 2. The system according to claim 1, wherein the display device is mounted on a device carried by a user. [Explanation of symbols]

[1447] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for detecting a string of characters in a field of view; means for extracting specific information from the detected string; means for translating the extracted information; means for displaying the translated information on a display device; The system includes means for communicating with a server over a network for extracting and translating said information.

2. 10. The system of claim 1, wherein the information extraction means uses multimodal artificial intelligence to filter out unwanted information.

3. 2. The system according to claim 1, wherein the display device is mounted on a device carried by a user.

Images