system

Abstract

Provide a system. 【Solution means】Means for capturing visual information using a camera of a wearable device, means for collecting audio information using a microphone, means for transmitting the captured visual information and the collected audio information, means for analyzing the transmitted visual information and extracting text, means for extracting important keywords from the extracted text using natural language processing technology and summarizing the text, means for analyzing the transmitted audio information and converting the audio into text, means for translating the text converted from the audio, means for transmitting the analyzed and translated results to the wearable device and displaying them, A system including the above.

Description

Technical Field

[0004]

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In modern educational and learning settings, in order to efficiently and effectively grasp information and advance learning, the ability to process a vast amount of materials and foreign language conversations in real time is required. However, with conventional methods, it is difficult to quickly extract important information or perform real-time translation, and thus the learning efficiency cannot be significantly improved. For this reason, there is a need for new technologies that collect visual and audio information using wearable devices and provide real-time analysis and feedback.

Means for Solving the Problems

[0005] <00This invention provides means for capturing visual information using a camera on a wearable device, means for collecting audio information using a microphone, and means for transmitting the captured visual information and collected audio information to a server. On the server side, means are provided for analyzing the received visual information, extracting text, extracting important keywords, and generating summaries, as well as for analyzing audio information, converting it to text, and translating it. Furthermore, by providing means for transmitting the analyzed and translated results to the wearable device and displaying them in the user's field of view, a system is constructed that enables real-time processing and feedback of visual and audio information. This allows the user to achieve efficient information gathering and learning.

[0006] A "wearable device" is a computer device that a user wears on their body and uses to collect and display information.

[0007] A "camera" is a device used to capture visual information.

[0008] A "microphone" is an input device used to collect voice information.

[0009] "Visual information" refers to image data captured by a camera.

[0010] "Audio information" refers to audio data collected by a microphone.

[0011] "Analysis" refers to the process of processing received data and extracting necessary information.

[0012] A "server" is a centralized processing system used for processing and analyzing data.

[0013] "Text" refers to character data extracted or converted from captured visual or audio information.

[0014] "Natural language processing technology" refers to the technology for a computer to understand and process human language.

[0015] "Summary" refers to information that extracts and concisely summarizes the important parts of a text.

[0016] "Speech recognition technology" refers to the technology for analyzing speech information and converting it into text.

[0017] "Translation" refers to the process of converting text expressed in one language into another language.

[0018] "Feedback" refers to the act of returning and displaying the processed information to the user.

[0019] "Real-time" refers to the act of performing information processing and feedback immediately without delay.

Brief Description of Drawings

[0020] [Figure 1] It is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] It is a conceptual diagram showing an example of the main functions of a data processing device and a smart device according to the first embodiment. [Figure 3] It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8]It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined.

Mode for Carrying Out the Invention

[0021] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0022] First, the terms used in the following description will be described.

[0023] In the following embodiments, a labeled processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of a plurality of arithmetic units. Further, the processor may be a single type of arithmetic unit or a combination of a plurality of types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0024] In the following embodiments, signed RAM (Random Access Memory) is a memory that temporarily stores information and is used as work memory by the processor.

[0025] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0026] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0027] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0028] [First Embodiment]

[0029] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0030] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0031] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0032] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0033] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0034] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0035] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0036] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0037] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0038] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0039] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0040] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0041] This invention relates to a system that uses a wearable device to collect visual and audio information, analyzes it, and provides real-time feedback to the user. This system is designed to significantly improve the efficiency of education and learning, and in particular, it extracts important keywords, generates summaries, and converts and translates audio information into text.

[0042] System Overview

[0043] This system consists of three main components: a wearable device (hereinafter referred to as "terminal"), a server that performs data analysis (hereinafter referred to as "server"), and users who utilize the service.

[0044] 1. Data collection and transmission

[0045] Device: The user wears a glasses-type wearable device. The device has a built-in camera and microphone. The camera captures visual information (e.g., pages of a book or document). The microphone collects audio information (e.g., a conversation in a foreign language).

[0046] Terminal: Encodes collected image and audio data into a digital format and transmits it to the server via the internet.

[0047] 2. Data Analysis

[0048] Server: Decodes the received visual information (image data) and extracts text from the image using OCR technology. Based on this text, natural language processing technology is applied to extract important keywords and generate a summary.

[0049] Server: Decodes the received audio information and converts the audio to text using speech recognition technology. This text is then sent to a translation API for real-time translation into the target language.

[0050] 3. Provide feedback

[0051] Server: Packages analysis results, such as summaries and translations, and sends them to the terminal.

[0052] Terminal: Decodes and renders received information in an appropriate layout to display it in the user's field of view.

[0053] User: Review the displayed information and continue learning or conversing.

[0054] Specific example

[0055] Example 1: Textbook summary display

[0056] User: Puts on glasses and begins reading textbook.

[0057] Device: Capture a page from the textbook using the camera and send the image to the server.

[0058] Server: Extracts text from images, uses natural language processing techniques to extract important keywords, and generates a summary.

[0059] Server: Sends the summary results to the terminal.

[0060] Terminal: Displays a summary within the user's field of view.

[0061] User: Review the summary and continue learning.

[0062] Example 2: Real-time translation of foreign language conversations

[0063] User: Puts on glasses and starts a conversation with a foreigner in English.

[0064] Terminal: The microphone collects the conversation and sends the audio data to the server.

[0065] Server: Converts speech to text and translates it into the target language in real time.

[0066] Server: Sends the translation result to the terminal.

[0067] Terminal: Displays the translation results within the user's field of view.

[0068] User: Check the displayed translation and continue the conversation.

[0069] This series of processes allows users to efficiently acquire information and progress in their learning.

[0070] The following describes the processing flow.

[0071] Step 1:

[0072] User: Put on the wearable device glasses and get ready.

[0073] Step 2:

[0074] Device: The device powers on, and the camera and microphone enter standby mode.

[0075] Step 3:

[0076] User: Puts on glasses and begins reading visual information (e.g., textbooks or documents).

[0077] Step 4:

[0078] Device: The camera detects the user's gaze and captures visual information.

[0079] Step 5:

[0080] Device: Simultaneously, the microphone collects ambient sound information.

[0081] Step 6:

[0082] Terminal: Encodes visual information (image data) into binary format and converts audio information into an appropriate format (e.g., PCM or FLAC format).

[0083] Step 7:

[0084] Terminal: Sends encoded and converted data to the server via the internet.

[0085] Step 8:

[0086] Server: Decodes visual information data and extracts text from images using OCR technology.

[0087] Step 9:

[0088] Server: Based on the extracted text data, it uses natural language processing techniques to extract important keywords and generate a summary.

[0089] Step 10:

[0090] Server: Decodes the audio information and converts the audio to text using speech recognition technology.

[0091] Step 11:

[0092] Server: Sends the converted text to the translation API for translation into the target language.

[0093] Step 12:

[0094] Server: Formats the analysis results, including the generated summary and translation, and sends them to the terminal.

[0095] Step 13:

[0096] Terminal: Decodes the received data and prepares it for display in the field of view with the appropriate layout.

[0097] Step 14:

[0098] User: Check the information displayed in their field of view on the device (summary, keywords, translation results, etc.) and continue learning or conversing.

[0099] Step 15:

[0100] Terminal: Continues to collect ongoing visual and auditory information and repeats the process from step 6.

[0101] Based on the steps described above, a system is created that supports efficient learning by analyzing visual and auditory information in real time and providing feedback to the user.

[0102] (Example 1)

[0103] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0104] Conventional wearable device systems have the problem of not being able to collect and analyze visual and auditory information in real time, thus failing to provide users with rapid feedback. Furthermore, the accuracy and speed of extracting important information and translation results are insufficient, highlighting the need for systems that support effective learning and communication.

[0105] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0106] In this invention, the server includes means for analyzing visual information and extracting textual information, means for extracting important keywords from the extracted textual information using natural language processing techniques and summarizing the textual information, and means for translating the textual information. This makes it possible to rapidly analyze visual and audio information and provide accurate feedback to the user in real time.

[0107] A "wearable device" is an electronic device that a user wears and uses, and that has the function of collecting visual and auditory information.

[0108] "Visual information" refers to image data and video data acquired through cameras and image acquisition devices.

[0109] "Audio information" refers to sound data collected through audio acquisition devices or microphones.

[0110] "Transmission" refers to transferring collected data to a server using communication methods such as the internet.

[0111] "Analysis" is the process of converting data into a digital format and processing it to make its contents easier to understand.

[0112] "Textual information" refers to text data extracted from visual information.

[0113] "Natural language processing technology" is a technology that enables computers to understand, analyze, and generate human language.

[0114] "Keywords" are words or phrases that are considered particularly important within a text.

[0115] A "summary" is a short, concise version of a longer text, focusing on the most important points.

[0116] Translation is the process of converting text written in one language into another language.

[0117] "Decoding" is the process of returning encoded data to its original format.

[0118] "Real-time" refers to temporal immediacy, meaning that processing and results are provided almost simultaneously.

[0119] "Rendering" is the process of visually displaying data.

[0120] A HUD (Head-Up Display) is a display technology that projects information directly into the user's field of vision.

[0121] This invention is a system that collects and analyzes visual and auditory information using a wearable device and provides real-time feedback to the user. This system enables users to efficiently acquire information and improve their learning and communication.

[0122] Hardware and software configuration

[0123] hardware

[0124] Wearable devices (devices): These are glasses-type devices with a built-in camera and microphone. The camera collects visual information, and the microphone collects audio information.

[0125] Data analysis server (server): A high-performance computer used to analyze received data.

[0126] software

[0127] OCR software: Software such as Tesseract OCR is used to extract text from visual information.

[0128] Natural Language Processing (NLP): This involves using tools such as NLTK (Natural Language Toolkit) to analyze text, extract important keywords, and generate summaries.

[0129] Speech recognition software: This includes tools such as Google's® Speech-to-Text API, which converts speech information into text.

[0130] Translation APIs: APIs such as the Google Translate API are used to translate text into other languages.

[0131] The process of data collection and transmission

[0132] When a user puts on a glasses-type wearable device, the device's camera captures visual information and its microphone collects audio information. This data is encoded in a digital format and sent to a server via the internet.

[0133] Data analysis process

[0134] The server decodes the received visual information and extracts text from the image using OCR technology. Then, it extracts important keywords using natural language processing technology and summarizes the text. The server also decodes the received audio information and converts it to text using speech recognition technology. The converted text is then translated into the target language in real time using a translation API.

[0135] Provide feedback

[0136] The server packages the analysis and translation results and sends them back to the terminal via the internet. The terminal decodes and renders the received information in a layout appropriate for the user's view. The user can then review the displayed information and continue learning or conversing.

[0137] Specific example

[0138] Examples of textbook summaries

[0139] 1. The user puts on their glasses and begins reading the textbook.

[0140] 2. The device captures a page from the textbook using its camera and sends the image to the server.

[0141] 3. The server extracts text from the image, uses natural language processing techniques to extract important keywords, and generates a summary.

[0142] 4. The server sends the summary results to the terminal.

[0143] 5. The device displays a summary within the user's field of view.

[0144] 6. The user reviews the summary and continues learning.

[0145] Examples of real-time translation of foreign language conversations

[0146] 1. The user puts on glasses and starts a conversation with a foreigner in English.

[0147] 2. The device's microphone collects the conversation and sends the audio data to the server.

[0148] 3. The server converts the speech to text and translates it into the target language in real time.

[0149] 4. The server sends the translation result to the terminal.

[0150] 5. The device displays the translation result within the user's field of view.

[0151] 6. The user confirms the displayed translation and continues the conversation.

[0152] Example of a prompt

[0153] Example prompt for generating a textbook summary:

[0154] "Extract the text from the following image and summarize the key points."

[0155] Image: [Image of a textbook page]

[0156] Example prompt for real-time translation of foreign language conversations:

[0157] "Please convert the following audio data to text and translate it into the target language."

[0158] Audio data: [Audio files of foreign language conversations]

[0159] Thus, the system of the present invention enables real-time analysis and feedback of visual and auditory information, efficiently supporting user learning and communication.

[0160] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0161] Step 1:

[0162] The user wears a wearable device.

[0163] Specific action: Place the glasses-type device on your ears and adjust it to fit properly.

[0164] Input: None.

[0165] Output: User wearing the device.

[0166] Step 2:

[0167] The device captures visual information.

[0168] Specific operation: The built-in camera acquires visual information (e.g., textbook pages) in image format.

[0169] Input: Textbook pages or visual objects.

[0170] Output: Acquired image data.

[0171] Step 3:

[0172] The device collects voice information.

[0173] Specific operation: The built-in microphone records ambient sounds (e.g., conversations in a foreign language or lecture content).

[0174] Input: Ambient sound.

[0175] Output: Collected audio data.

[0176] Step 4:

[0177] The device encodes visual and audio information into a digital format.

[0178] Specific operation: Convert the acquired images and collected audio into a digital format and compress the data.

[0179] Input: Image data, audio data.

[0180] Output: Visual and audio information converted to digital format.

[0181] Step 5:

[0182] The device sends the encoded data to the server via the internet.

[0183] Specific operation: Send data using a secure communication protocol (HTTPS).

[0184] Input: Visual and audio information converted into digital format.

[0185] Output: Data sent to the server.

[0186] Step 6:

[0187] The server decodes the received visual information and extracts text using OCR technology.

[0188] Specific operation: Input image data into OCR software and convert it to text format.

[0189] Input: Visual information (image data) sent to the server.

[0190] Output: Extracted text data.

[0191] Step 7:

[0192] The server generates a summary of the extracted text using natural language processing techniques.

[0193] Specific operation: Apply an NLP algorithm to extract key keywords and generate a summary.

[0194] Input: Extracted text data.

[0195] Output: Summary text.

[0196] Step 8:

[0197] The server decodes the received audio information and uses speech recognition technology to convert the audio into text.

[0198] Specific operation: Input audio data into speech recognition software and convert it to text format.

[0199] Input: Audio information (audio data) sent to the server.

[0200] Output: Text data converted by speech recognition.

[0201] Step 9:

[0202] The server translates the converted text in real time using a translation API.

[0203] Specific operation: Input text data into the translation API and retrieve the translation result.

[0204] Input: Text data converted by speech recognition.

[0205] Output: Translated text data.

[0206] Step 10:

[0207] The server packages the analysis and translation results and sends them to the terminal.

[0208] Specific operation: Package the summary and translation results and send them using a secure communication protocol (HTTPS).

[0209] Input: Summary text, translated text data.

[0210] Output: Data sent to the terminal.

[0211] Step 11:

[0212] The device decodes and renders the received information in a layout appropriate to the user's field of view.

[0213] Specific operation: Information is overlaid onto the user's field of view using display technology.

[0214] Input: Summary text and translated text data sent to the server.

[0215] Output: Summary text displayed in the user's field of view, translation result.

[0216] Step 12:

[0217] The user reviews the displayed information and continues learning or conversing.

[0218] Specific actions: Users use summaries and translation results to understand materials and engage in conversations in a foreign language.

[0219] Input: Information displayed within the user's field of view.

[0220] Output: Continued learning or conversation.

[0221] (Application Example 1)

[0222] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0223] Conventional factory monitoring systems have struggled to monitor production lines and equipment in real time, making them ineffective for anomaly detection and improving production efficiency. Furthermore, even when visual and auditory information was collected, no system existed that could quickly analyze it and provide appropriate feedback. Therefore, there is a need for early detection of anomalies and real-time response in factory production lines.

[0224] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0225] In this invention, the server includes means for analyzing transmitted visual information and extracting text, means for extracting important keywords from the extracted text using natural language processing technology and summarizing the text, and means for monitoring production lines and equipment in real time using a mobile device that patrols the factory. This enables real-time monitoring of production lines and equipment, allowing for early detection of abnormalities and providing appropriate feedback to improve production efficiency.

[0226] A "wearable device" is a portable electronic device that a user can wear.

[0227] A "camera" is an optical device used to capture visual information.

[0228] "Visual information" refers to image and video data acquired using a camera.

[0229] A "microphone" is an acoustic device used to collect voice information.

[0230] "Audio information" refers to sound data collected through a microphone.

[0231] "Transmission means" refers to a mechanism for encoding captured visual information and collected audio information into a digital format and transmitting it to a remote location.

[0232] "Means for extracting text" refers to technologies for identifying and extracting textual information from visual information.

[0233] "Natural language processing technology" refers to the technology that enables computers to understand, interpret, and generate human language.

[0234] "Methods for extracting important keywords" refer to techniques for selecting particularly meaningful words and phrases from text.

[0235] "Methods for summarizing text" are techniques for making the content concise while preserving the overall meaning of the text.

[0236] "Methods for converting speech to text" refer to technologies that convert collected speech data into textual information.

[0237] "Translation methods" refer to the techniques used to convert text written in one language into another language.

[0238] "Means of providing feedback" refers to a mechanism for communicating the analyzed and translated results to the user.

[0239] "Mobile devices that patrol within a factory" refers to devices such as robots that move autonomously or remotely within a factory to collect data.

[0240] "Methods for monitoring production lines and equipment in real time" refer to technologies for continuously observing the production process and the state of equipment, and for immediately detecting abnormalities.

[0241] "Methods for generating reports" refer to the techniques used to analyze collected data, organize the results, and create a report.

[0242] This invention relates to a system aimed at efficient monitoring and anomaly detection of production lines and equipment within a factory. This system uses wearable devices to collect visual and auditory information, analyzes it, and provides real-time feedback to the user. Specific embodiments are described below.

[0243] System Overview

[0244] This system consists of three main components: a wearable device (hereinafter referred to as "terminal"), a server that performs data analysis (hereinafter referred to as "server"), and users who utilize the service. This enables anomaly detection and improved production efficiency in factory environments.

[0245] 1. Data collection and transmission

[0246] Terminal: A mobile device that patrols the factory, such as a factory robot, equipped with a camera and microphone, collects visual information (production line and equipment status) and audio information (worker instructions and ambient sounds). A standard industrial camera can be used as the camera, and a high-sensitivity microphone is used for audio collection.

[0247] Terminal: Encodes collected image and audio data into a digital format and transmits it to a server via the internet. For example, it uses Wi-Fi or a wired connection.

[0248] 2. Data Analysis

[0249] Server: Decodes visual information and extracts text from images using OCR technology. Based on this text, natural language processing techniques are applied to extract important keywords and generate a summary. Specific software examples include Python, OpenCV, and Tesseract.

[0250] Server: Decodes the received audio information and converts it to text using speech recognition technology. It then sends it to a translation API for real-time translation into the target language. Examples of use cases include the Python library SpeechRecognition and GoogleTrans.

[0251] 3. Provide feedback

[0252] Server: Packages analysis results, such as summaries and translations, and sends them to the terminal. During this process, it uses web frameworks such as Flask or Django to format the data.

[0253] Terminal: Decodes and renders received information in an appropriate layout for display on monitors within the factory and on workers' smartphones / tablets.

[0254] User: Review the displayed information to help identify anomalies and improve production efficiency. Audio and visual notifications are used as specific feedback methods.

[0255] Specific example

[0256] Example 1: Anomaly detection in factory robots

[0257] 1. Terminal: A robot moving around the factory takes pictures of the production line with its camera.

[0258] 2. Server: Uses OCR to read product labels and equipment status from the received video. Generates an alert if an anomaly is detected.

[0259] 3. User: Check alerts on your mobile device and take immediate action.

[0260] Example 2: Analysis of work instructions

[0261] 1. Terminal: Records the worker's conversation using a microphone and sends the audio data to the server.

[0262] 2. Server: Converts speech to text and extracts important keywords and instructions.

[0263] 3. User: Check the progress of work and instructions on a mobile device to proceed with work efficiently.

[0264] Example of a prompt

[0265] Please explain how factory robots detect anomalies on the production line in real time.

[0266] Please provide specific examples of how data collection and analysis can be performed using a robot's camera and microphone.

[0267] Please explain in detail how to improve work efficiency in a factory using visual and audio analysis technologies.

[0268] This system will enable highly efficient and effective real-time monitoring of production lines and equipment within the factory.

[0269] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0270] Step 1:

[0271] The terminal patrols the factory, capturing visual information with its camera and collecting audio information with its microphone. Inputs include video of the factory's production lines and equipment, as well as conversations among workers and ambient sounds. Outputs include digitally encoded video and audio data.

[0272] Step 2:

[0273] The terminal transmits the collected visual and audio information to the server via the internet. The input is the data encoded in step 1, and the output is the transmission of data to the server.

[0274] Step 3:

[0275] The server decodes the received visual information and extracts text from the image using OCR technology. Specifically, text information from product labels and display panels in the video is used as input, and the output is obtained by converting it into text data.

[0276] Step 4:

[0277] The server applies natural language processing techniques to the extracted text, extracting important keywords and summarizing the text. The input is the text data obtained in step 3, and the output generates important keywords and a summarized text.

[0278] Step 5:

[0279] The server decodes the received voice information and uses voice recognition technology to convert the voice into text. Specifically, the conversation of the operator is used as input, and the output is obtained by converting it into character information.

[0280] Step 6:

[0281] The server sends the text converted from the voice to the translation API and translates it into the target language in real time. The input is the text data obtained in Step 5, and the output is the translated text generated.

[0282] Step 7:

[0283] The server packages the analyzed and translated results and sends them to the terminal. The input is the data obtained in Step 4 and Step 6, and the output is a package for feedback that is generated and sent to the terminal.

[0284] Step 8:

[0285] The terminal decodes and renders the received feedback information in an appropriate layout and visually displays it to the user. Specifically, the data is formatted and displayed, and visual notifications and alerts are displayed on the user's device.

[0286] Step 9:

[0287] The user checks the information displayed on the terminal and takes actions to immediately respond to the abnormalities in the factory. The input is the information displayed in Step 8, and the output is the improvement of the user's actions and the response to abnormalities.

[0288] Furthermore, an emotion engine for estimating the user's emotion may be combined. That is, the specific processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform specific processing using the user's emotion.

[0289] This invention relates to a system that uses a wearable device to collect visual and auditory information, analyzes it, and provides real-time feedback to the user. Furthermore, this system incorporates an emotion recognition engine that can analyze the user's emotions and adjust the displayed information and learning content accordingly. This not only improves learning efficiency but also provides flexible support tailored to the user's emotional state.

[0290] System Overview

[0291] This system consists of three main components: a wearable device (hereinafter referred to as "terminal"), a server that performs data analysis (hereinafter referred to as "server"), and users who utilize the service.

[0292] 1. Data collection and transmission

[0293] Device: The user wears a glasses-type wearable device. The device has a built-in camera and microphone. The camera captures visual information (e.g., pages of a book or document). The microphone collects audio information (e.g., a conversation in a foreign language).

[0294] Terminal: Encodes collected image and audio data into a digital format and transmits it to the server via the internet.

[0295] 2. Data Analysis

[0296] Server: Decodes the received visual information (image data) and extracts text from the image using OCR technology. Based on this text, natural language processing technology is applied to extract important keywords and generate a summary.

[0297] Server: Decodes the received audio information and converts the audio to text using speech recognition technology. This text is then sent to a translation API for real-time translation into the target language.

[0298] Server: Analyze the user's emotions using an emotion recognition engine and incorporate the analysis results into the feedback.

[0299] 3. Feedback provision

[0300] Server: Package the analysis results such as the summary and translation results, and the emotion analysis results, and send them to the terminal.

[0301] Terminal: Decode and render the received information in an appropriate layout to display it within the user's field of view.

[0302] User: Check the displayed information and continue learning or conversation.

[0303] Specific example

[0304] Example 1: Summary display of textbooks and emotion feedback

[0305] User: Put on glasses and start reading a textbook.

[0306] Terminal: Capture the pages of the textbook with a camera and send the image to the server. Collect the facial expressions and voice tones with a microphone.

[0307] Server: Extract text from the image, extract important keywords, and generate a summary using natural language processing technology. Also, analyze the user's emotions with an emotion recognition engine.

[0308] Server: Send the summary result and the emotion analysis result to the terminal.

[0309] Terminal: Display the summary and emotion feedback within the user's field of view.

[0310] User: Check the summary and consider the next learning step based on the emotion feedback.

[0311] Example 2: Real-time translation and emotion recognition of foreign language conversations

[0312] User: Puts on glasses and starts a conversation with a foreigner in English.

[0313] Terminal: The microphone collects conversation data and sends it to the server along with the user's tone of voice and facial expressions.

[0314] Server: Converts speech to text and translates it into the target language in real time. It also analyzes the user's emotions using an emotion recognition engine and includes emotion data in the conversation feedback.

[0315] Server: Sends translation results and sentiment analysis results to the terminal.

[0316] Device: Displays translation results and sentiment feedback within the field of view.

[0317] User: Review the displayed translation and sentiment feedback, then continue the conversation.

[0318] In this way, this system analyzes the user's visual and auditory information in real time, and in addition, recognizes emotions, thereby providing feedback tailored to the user's emotional state, and achieving more efficient and personalized learning support.

[0319] The following describes the processing flow.

[0320] Step 1:

[0321] User: Put on the wearable device glasses and get ready.

[0322] Step 2:

[0323] Device: The device powers on, and the camera and microphone enter standby mode.

[0324] Step 3:

[0325] User: Puts on glasses and begins reading visual information (e.g., textbooks or documents).

[0326] Step 4:

[0327] Device: The camera captures visual information and generates image data.

[0328] Step 5:

[0329] Device: Simultaneously, the microphone collects ambient audio information and saves it as audio data.

[0330] Step 6:

[0331] Terminal: Encodes visual information (image data) into binary format and converts audio information into an appropriate format (e.g., PCM or FLAC format).

[0332] Step 7:

[0333] Terminal: Sends encoded and converted data to the server via the internet.

[0334] Step 8:

[0335] Server: Decodes visual information data and extracts text from images using OCR technology.

[0336] Step 9:

[0337] Server: Based on the extracted text data, it uses natural language processing techniques to extract important keywords and generate a summary.

[0338] Step 10:

[0339] Server: Decodes the audio information and converts the audio to text using speech recognition technology.

[0340] Step 11:

[0341] Server: Sends the converted text to the translation API, which translates it into the target language in real time.

[0342] Step 12:

[0343] Server: Analyzes the user's emotional state using an emotion recognition engine. This includes the process of identifying emotions from voice tone and facial expressions.

[0344] Step 13:

[0345] Server: Packages the summary, translation results, and sentiment analysis results and sends them to the terminal.

[0346] Step 14:

[0347] Terminal: Decodes the received data and renders it in an appropriate layout for display in the user's field of view.

[0348] Step 15:

[0349] User: Check the information displayed in their field of view on the device (summary, keywords, translation results, sentiment feedback, etc.) and continue learning or conversing.

[0350] Step 16:

[0351] Terminal: Continues to collect ongoing visual and auditory information and repeats the process from step 6.

[0352] Specific example

[0353] Example 1: Textbook summary display and emotional feedback

[0354] Step 1:

[0355] User: Puts on glasses and begins reading textbook.

[0356] Step 2:

[0357] Device: Captures textbook pages with the camera and sends the images to the server. Collects facial expressions and voice tone with the microphone.

[0358] Step 3:

[0359] Server: Extracts text from images, uses natural language processing techniques to extract important keywords, and generates summaries. It also analyzes user emotions using an emotion recognition engine.

[0360] Step 4:

[0361] Server: Sends summary results and sentiment analysis results to the terminal.

[0362] Step 5:

[0363] Device: Displays summaries and sentiment feedback within the user's field of view.

[0364] Step 6:

[0365] User: Review the summary and consider the next learning steps based on the sentiment feedback.

[0366] Example 2: Real-time translation and sentiment recognition of foreign language conversations

[0367] Step 1:

[0368] User: Puts on glasses and starts a conversation with a foreigner in English.

[0369] Step 2:

[0370] Terminal: The microphone collects conversation data and sends it to the server along with the user's tone of voice and facial expressions.

[0371] Step 3:

[0372] Server: Converts speech to text and translates it into the target language in real time. It also analyzes the user's emotions using an emotion recognition engine and includes emotion data in the conversation feedback.

[0373] Step 4:

[0374] Server: Sends translation results and sentiment analysis results to the terminal.

[0375] Step 5:

[0376] Device: Displays translation results and sentiment feedback within the field of view.

[0377] Step 6:

[0378] User: Review the displayed translation and sentiment feedback, then continue the conversation.

[0379] Based on the steps described above, this system analyzes the user's visual and auditory information in real time, and in addition, recognizes emotions, thereby providing more personalized learning support.

[0380] (Example 2)

[0381] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0382] Conventional learning support systems and conversation support systems function independently, making it difficult to integrate and analyze visual and auditory information to provide real-time feedback. Furthermore, because they only provide simple information without considering the user's emotional state, the user's learning efficiency and communication experience were not optimized. The present invention aims to solve these problems and provide a system that analyzes visual and auditory information in real time, recognizes the user's emotions, and provides appropriate feedback.

[0383] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0384] In this invention, the server includes means for decoding transmitted visual information and extracting text using optical character recognition technology, means for extracting important keywords from the extracted text using natural language processing technology and summarizing the text, and means for decoding transmitted audio information and converting the audio to text using speech recognition technology. This enables integrated analysis of visual and audio information and real-time feedback that responds to the user's emotional state.

[0385] A "wearable device" is a portable electronic device that a user wears to collect visual and auditory information.

[0386] A "camera" is a device used to capture visual information.

[0387] A "microphone" is an acoustic conversion device used to collect sound information.

[0388] "Encoding into a digital format" is the process of converting an analog signal into a digital signal.

[0389] "Transmission" refers to sending collected data to another device or server.

[0390] "Decoding" is the process of converting transmitted digital data back into its original format.

[0391] "Optical character recognition technology" is a technology that automatically recognizes characters from images and converts them into text.

[0392] "Natural language processing technology" is a technology that processes and analyzes text written in natural language.

[0393] "Speech recognition technology" is a technology that analyzes speech data and converts it into text.

[0394] "Translation" refers to the process of converting text written in one language or spoken content into another language.

[0395] An "emotion recognition engine" is a general term for algorithms and technologies used to analyze and identify a user's emotions.

[0396] "Feedback" refers to providing users with information and data, such as analysis results or translation results.

[0397] This invention relates to a system that allows users to collect visual and auditory information using a wearable device, analyze it, and provide real-time feedback to the user. Furthermore, this system incorporates an emotion recognition engine that can analyze the user's emotions and adjust the displayed information and learning content accordingly. This improves learning efficiency and provides flexible support tailored to the user's emotional state. Specific embodiments of this system are described below.

[0398] Hardware configuration

[0399] Device: A wearable device worn by the user (e.g., smart glasses). This device has a built-in camera and microphone to collect visual and audio information.

[0400] Server: A remote server that performs data analysis. This server possesses advanced computing capabilities and processes and analyzes the received data.

[0401] Software Configuration

[0402] Camera: A software module for capturing visual information. For example, it can be used to photograph text information on books or signs.

[0403] Microphone: A software module for collecting audio information. For example, it can be used to record conversations or ambient sounds.

[0404] Optical Character Recognition (OCR): A technology for extracting characters from images. For example, using the "Google Cloud Vision API".

[0405] Natural Language Processing (NLP): A technique that analyzes extracted text and identifies and summarizes important keywords. For example, it uses libraries such as the "NLTK library."

[0406] Speech-to-Text (ASR) technology: A technology that converts speech into text. For example, using the "Google Speech-to-Text API".

[0407] Translation API: A technology for translating text into other languages. For example, using the "Google Translate API".

[0408] Emotion recognition engine: An algorithm for analyzing a user's emotions. For example, it uses the "Microsoft® Azure® Emotion API".

[0409] Specific example

[0410] Example 1: Textbook summary display and emotional feedback

[0411] User: Puts on glasses and begins reading textbook.

[0412] Device: Captures textbook pages with the camera and sends the images to the server. Collects facial expressions and voice tone with the microphone.

[0413] Server: Extracts text from images, uses natural language processing techniques to extract important keywords, and generates summaries. It also analyzes user emotions using an emotion recognition engine.

[0414] Server: Sends summary results and sentiment analysis results to the terminal.

[0415] Device: Displays summaries and sentiment feedback within the user's field of view.

[0416] User: Review the summary and consider the next learning steps based on the sentiment feedback.

[0417] Example 2: Real-time translation and sentiment recognition of foreign language conversations

[0418] User: Puts on glasses and starts a conversation with a foreigner in English.

[0419] Terminal: The microphone collects conversation data and sends it to the server along with the user's tone of voice and facial expressions.

[0420] Server: Converts speech to text and translates it into the target language in real time. It also analyzes the user's emotions using an emotion recognition engine and includes emotion data in the conversation feedback.

[0421] Server: Sends translation results and sentiment analysis results to the terminal.

[0422] Device: Displays translation results and sentiment feedback within the field of view.

[0423] User: Review the displayed translation and sentiment feedback, then continue the conversation.

[0424] Example of a prompt

[0425] For textbooks: "When a user reads the textbook, key keywords should be summarized and displayed in real time."

[0426] For conversations: "When a user speaks a foreign language, perform real-time emotion recognition and provide feedback along with the translation results."

[0427] In this way, the present invention can analyze visual and auditory information in real time and further recognize emotions to provide personalized feedback to the user.

[0428] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0429] Step 1:

[0430] Data collection

[0431] Terminal: The user wears a wearable device and captures visual information using a camera. Simultaneously, audio information is collected using a microphone.

[0432] Input: The visual object the user is looking at (e.g., a page in a textbook) and the sound they are hearing (e.g., surrounding conversation).

[0433] Output: Captured visual information (image data) and collected audio information (audio data).

[0434] Specific actions: Take a picture of a textbook page with the camera and encode the image in JPEG format. Record a conversation with the microphone and encode the audio in WAV format.

[0435] Step 2:

[0436] Data transmission

[0437] Terminal: Encodes collected visual and audio information into a digital format and transmits it to the server via the internet.

[0438] Input: Encoded image and audio data.

[0439] Output: Data sent to the server.

[0440] Specific operation: Compresses JPEG image data and WAV audio data, and sends them to a server via the internet.

[0441] Step 3:

[0442] Analysis of visual information

[0443] Server: Decodes the received image data and extracts text from the image using OCR technology.

[0444] Input: The encoded image data that was sent.

[0445] Output: Extracted text data.

[0446] Specific operation: Decode JPEG image data and extract text using OCR technology (e.g., Google Cloud Vision API). For example, extract the text "differential and integral calculus" from a page in a math textbook.

[0447] Step 4:

[0448] Natural Language Processing

[0449] Server: Extracts important keywords from the extracted text and applies natural language processing techniques to summarize the text.

[0450] Input: Text data extracted using OCR technology.

[0451] Output: Summary and keyword extraction results.

[0452] Specific operation: Natural language processing techniques (e.g., NLTK library) are used on the extracted text data to extract important keywords and generate a summary. For example, it might summarize as "Explanation of the basic principles of differentiation."

[0453] Step 5:

[0454] Voice analysis

[0455] Server: Decodes the received audio data and converts the audio to text using speech recognition technology.

[0456] Input: The encoded audio data that has been sent.

[0457] Output: Text data converted from speech.

[0458] Specific operation: Decodes WAV audio data and converts it to text using speech recognition technology (e.g., Google Speech-to-Text API). For example, converts "What is your name?" to text.

[0459] Step 6:

[0460] translation

[0461] Server: Sends the text converted from speech to a translation API, which translates it into the target language in real time.

[0462] Input: Data converted to text using speech recognition technology.

[0463] Output: Translation result.

[0464] Specific operation: Text data is sent to a translation API (e.g., Google Translate API) and translated in real time. For example, "What is your name?" is translated to "What is your name?".

[0465] Step 7:

[0466] Emotion analysis

[0467] Server: Analyzes user emotions using an emotion recognition engine and compiles the results.

[0468] Input: User's facial expression data and voice data.

[0469] Output: Emotion analysis results.

[0470] Specific operation: The collected facial expression data and voice tone are analyzed using an emotion recognition engine (e.g., Microsoft Azure Emotion API) to identify emotions such as "joy" or "sadness." For example, the analysis result might be "the user is satisfied."

[0471] Step 8:

[0472] Feedback generation and transmission

[0473] Server: Packages the summary results, translation results, and sentiment analysis results and sends them to the terminal.

[0474] Input: Summarization results, translation results, and sentiment analysis results.

[0475] Output: Feedback data sent to the terminal.

[0476] Specific operation: Integrate the results of summarization, translation, and sentiment analysis, format them appropriately, and send them to the terminal. For example, send the textbook summary, conversation translation results, and user sentiment analysis results as a single data package to the terminal in JPEG and text formats.

[0477] Step 9:

[0478] Displaying feedback

[0479] Terminal: Decodes the received feedback data and displays it appropriately within the user's field of view.

[0480] Input: Feedback data sent from the server.

[0481] Output: Visually displayed feedback information.

[0482] Specific operation: Adjust the layout and font size to display the decoded feedback data on the smart glasses' display.

[0483] Step 10:

[0484] Feedback review and action

[0485] User: Review the displayed feedback information and decide on the next action.

[0486] Input: Feedback information displayed on the screen.

[0487] Output: The user's next action (e.g., continue learning, continue conversation).

[0488] Specific action: Decide whether to continue learning by reading the textbook summary or to continue the conversation in the foreign language based on the translation result.

[0489] (Application Example 2)

[0490] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0491] Conventional wearable device-based information collection and analysis systems can collect and analyze visual and auditory information, but they cannot provide feedback that takes into account the user's emotional state. As a result, the user experience is uniform, and individualized support is difficult. Furthermore, improving customer service in physical stores has been challenging, as it has been difficult to provide appropriate support tailored to the individual needs and emotions of each customer.

[0492] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0493] In this invention, the server includes means for analyzing visual information and extracting text, means for analyzing audio information and converting audio to text, and means for analyzing customer emotions and providing real-time emotional feedback. This makes it possible to provide feedback tailored to the customer's emotional state, in addition to text summarization and translation results, based on the collected visual and audio information.

[0494] A "wearable device" is an electronic device that a user can wear and that is equipped with a wide variety of sensors and communication functions.

[0495] "Visual information" refers to images and video data captured using a camera.

[0496] "Audio information" refers to audio data collected using a microphone.

[0497] "Means of transmission" refers to a system that includes communication means for transferring collected data to a server or other device.

[0498] "Means of analysis" include technologies that analyze collected data and perform text extraction and speech recognition.

[0499] "Natural language processing technology" refers to all technologies that use computers to understand and generate human language.

[0500] "Key keywords" refer to words or phrases that are deemed particularly important within text data analyzed using natural language processing technology.

[0501] "Methods of summarization" refer to techniques for extracting key points from analyzed text data and compiling them into a concise format.

[0502] "Translation methods" refer to technologies that convert analyzed text data into other languages.

[0503] "Emotional feedback" refers to the process of analyzing a user's emotional state and providing specific responses or messages based on the results.

[0504] A "server" refers to a remote device that processes collected data and performs tasks such as analysis, text extraction, translation, and sentiment analysis.

[0505] "Decoding" refers to the process of converting digital data into analog or other formats.

[0506] "Rendering" refers to the process of processing digital data to display it visually.

[0507] A "user" refers to a person who uses a wearable device to receive feedback from the system.

[0508] This invention is a system that uses a wearable device to collect visual and auditory information, analyzes it in real time, and provides feedback to the user. Furthermore, by using emotion recognition technology to provide feedback tailored to the user's emotional state, it enables more effective learning and service support.

[0509] System Overview

[0510] This system consists of the following main components:

[0511] Wearable devices (devices): Glasses-type or head-mounted display-type devices equipped with a camera and microphone.

[0512] Server: A remote server that performs data analysis and generates feedback.

[0513] User: An individual who uses a wearable device.

[0514] 1. Data collection and transmission

[0515] Terminal: The user wears a glasses-type wearable device that captures visual information with its built-in camera and collects audio information with its microphone. This allows for the acquisition of visual and audio data in environments such as physical stores and classrooms.

[0516] Terminal: Collected visual information (e.g., images of product packaging) and audio information (e.g., conversations between customers and store clerks) are encoded in digital format and transmitted to a server via the internet.

[0517] 2. Data Analysis

[0518] Server: Converts received visual information into text using OCR technology, extracts important keywords using natural language processing technology, and summarizes the text. For example, it can generate detailed product information or sales information.

[0519] Server: Converts received audio information into text using speech recognition technology and translates it into other languages ​​using a translation API as needed. It also processes the analysis results using a generative AI model.

[0520] Server: Uses an emotion recognition engine to analyze the user's emotions from their voice and visual information and incorporates this into the feedback. For example, it can determine whether the customer is relaxed or stressed.

[0521] 3. Provide feedback

[0522] Server: Compiles the analysis results (summary, translation results, sentiment analysis results, etc.) and sends them to the terminal in an appropriate layout.

[0523] Terminal: Decodes and renders information to display analysis results within the user's field of view. For example, product summary information or emotion-based guidance messages may be displayed within the user's field of view.

[0524] User: Based on the feedback provided, decide on the next course of action. For example, look at other products, ask additional questions, etc.

[0525] Specific example

[0526] Example 1: Providing product information and emotional feedback

[0527] User: Walks around the store wearing smart glasses.

[0528] Device: Captures product packaging with a camera and collects conversation content with a microphone.

[0529] Server: Analyzes received visual information using OCR and summarizes product information. Analyzes customer emotions from audio information and provides information smoothly if the customer is relaxed.

[0530] Terminal: Displays product information and emotion-based messages within the user's field of view.

[0531] User: Choose a product based on the information provided, or ask the store clerk additional questions.

[0532] Example of a prompt

[0533] Image data: "Captured product image"

[0534] Audio data: "Audio of customer conversation"

[0535] Emotional data: "Analyzed emotional state (e.g., relaxed, tense)"

[0536] This system provides more personalized and effective service support by offering feedback based on the customer's emotional state. The hardware used includes smart glasses (e.g., general wearable devices) and servers (e.g., cloud servers), while the software utilizes OCR technology, speech recognition technology, natural language processing technology, and an emotion recognition engine.

[0537] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0538] Step 1:

[0539] Data collection

[0540] Subject: terminal

[0541] Specific operation: The user wears smart glasses and moves around the store. The camera captures visual information (e.g., images of products), and the microphone collects audio information (e.g., conversations between the user and store staff).

[0542] Input: Products viewed by the user, or audio uttered by the user.

[0543] Output: Digital image data and audio data.

[0544] Step 2:

[0545] Data transmission

[0546] Subject: terminal

[0547] Specific operation: The collected visual and audio information is sent to the server via the internet.

[0548] Input: Digital image and audio data.

[0549] Output: Data sent to the server.

[0550] Step 3:

[0551] Analysis of visual information

[0552] Subject: Server

[0553] Specific operation: The transmitted visual information is analyzed using OCR technology to extract text from the image. Natural language processing technology is used to identify important keywords in the extracted text and summarize the text.

[0554] Input: Image data sent to the server.

[0555] Output: Extracted text and its summary.

[0556] Step 4:

[0557] Analysis of audio information

[0558] Subject: Server

[0559] Specific operation: The transmitted audio information is converted into text using speech recognition technology. Then, the converted text is analyzed using natural language processing technology and translated into other languages ​​using a translation API as needed.

[0560] Input: Audio data sent to the server.

[0561] Output: Text converted from speech and its translation.

[0562] Step 5:

[0563] Emotion analysis

[0564] Subject: Server

[0565] Specific operation: Using an emotion recognition engine, the system analyzes the user's emotions from their voice and visual information. It identifies the user's emotional state, such as whether they are relaxed or stressed.

[0566] Input: Audio data and visual data.

[0567] Output: Analyzed emotional state data.

[0568] Step 6:

[0569] Feedback generation

[0570] Subject: Server

[0571] Specific operation: Generates feedback based on analysis and translation results, sentiment analysis results, and sends it to the device in real time.

[0572] Input: Summarized text, translation results, sentiment analysis results.

[0573] Output: Feedback data to be displayed in the user's field of view.

[0574] Step 7:

[0575] Information display

[0576] Subject: terminal

[0577] Specific operation: Decodes and renders the received feedback data and displays it in a layout appropriate to the user's field of view.

[0578] Input: Feedback data.

[0579] Output: Information displayed in the user's field of vision (e.g., product information, emotion-based guidance messages).

[0580] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0581] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0582] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0583] [Second Embodiment]

[0584] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0585] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0586] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0587] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0588] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0589] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0590] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0591] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0592] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0593] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0594] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0595] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0596] This invention relates to a system that uses a wearable device to collect visual and audio information, analyzes it, and provides real-time feedback to the user. This system is designed to significantly improve the efficiency of education and learning, and in particular, it extracts important keywords, generates summaries, and converts and translates audio information into text.

[0597] System Overview

[0598] This system consists of three main components: a wearable device (hereinafter referred to as "terminal"), a server that performs data analysis (hereinafter referred to as "server"), and users who utilize the service.

[0599] 1. Data collection and transmission

[0600] Device: The user wears a glasses-type wearable device. The device has a built-in camera and microphone. The camera captures visual information (e.g., pages of a book or document). The microphone collects audio information (e.g., a conversation in a foreign language).

[0601] Terminal: Encodes collected image and audio data into a digital format and transmits it to the server via the internet.

[0602] 2. Data Analysis

[0603] Server: Decodes the received visual information (image data) and extracts text from the image using OCR technology. Based on this text, natural language processing technology is applied to extract important keywords and generate a summary.

[0604] Server: Decodes the received audio information and converts the audio to text using speech recognition technology. This text is then sent to a translation API for real-time translation into the target language.

[0605] 3. Provide feedback

[0606] Server: Packages analysis results, such as summaries and translations, and sends them to the terminal.

[0607] Terminal: Decodes and renders received information in an appropriate layout to display it in the user's field of view.

[0608] User: Review the displayed information and continue learning or conversing.

[0609] Specific example

[0610] Example 1: Textbook summary display

[0611] User: Puts on glasses and begins reading textbook.

[0612] Device: Capture a page from the textbook using the camera and send the image to the server.

[0613] Server: Extracts text from images, uses natural language processing techniques to extract important keywords, and generates a summary.

[0614] Server: Sends the summary results to the terminal.

[0615] Terminal: Displays a summary within the user's field of view.

[0616] User: Review the summary and continue learning.

[0617] Example 2: Real-time translation of foreign language conversations

[0618] User: Puts on glasses and starts a conversation with a foreigner in English.

[0619] Terminal: The microphone collects the conversation and sends the audio data to the server.

[0620] Server: Converts speech to text and translates it into the target language in real time.

[0621] Server: Sends the translation result to the terminal.

[0622] Terminal: Displays the translation results within the user's field of view.

[0623] User: Check the displayed translation and continue the conversation.

[0624] This series of processes allows users to efficiently acquire information and progress in their learning.

[0625] The following describes the processing flow.

[0626] Step 1:

[0627] User: Put on the wearable device glasses and get ready.

[0628] Step 2:

[0629] Device: The device powers on, and the camera and microphone enter standby mode.

[0630] Step 3:

[0631] User: Puts on glasses and begins reading visual information (e.g., textbooks or documents).

[0632] Step 4:

[0633] Device: The camera detects the user's gaze and captures visual information.

[0634] Step 5:

[0635] Device: Simultaneously, the microphone collects ambient sound information.

[0636] Step 6:

[0637] Terminal: Encodes visual information (image data) into binary format and converts audio information into an appropriate format (e.g., PCM or FLAC format).

[0638] Step 7:

[0639] Terminal: Sends encoded and converted data to the server via the internet.

[0640] Step 8:

[0641] Server: Decodes visual information data and extracts text from images using OCR technology.

[0642] Step 9:

[0643] Server: Based on the extracted text data, it uses natural language processing techniques to extract important keywords and generate a summary.

[0644] Step 10:

[0645] Server: Decodes the audio information and converts the audio to text using speech recognition technology.

[0646] Step 11:

[0647] Server: Sends the converted text to the translation API for translation into the target language.

[0648] Step 12:

[0649] Server: Formats the analysis results, including the generated summary and translation, and sends them to the terminal.

[0650] Step 13:

[0651] Terminal: Decodes the received data and prepares it for display in the field of view with the appropriate layout.

[0652] Step 14:

[0653] User: Check the information displayed in their field of view on the device (summary, keywords, translation results, etc.) and continue learning or conversing.

[0654] Step 15:

[0655] Terminal: Continues to collect ongoing visual and auditory information and repeats the process from step 6.

[0656] Based on the steps described above, a system is created that supports efficient learning by analyzing visual and auditory information in real time and providing feedback to the user.

[0657] (Example 1)

[0658] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0659] Conventional wearable device systems have the problem of not being able to collect and analyze visual and auditory information in real time, thus failing to provide users with rapid feedback. Furthermore, the accuracy and speed of extracting important information and translation results are insufficient, highlighting the need for systems that support effective learning and communication.

[0660] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0661] In this invention, the server includes means for analyzing visual information and extracting textual information, means for extracting important keywords from the extracted textual information using natural language processing techniques and summarizing the textual information, and means for translating the textual information. This makes it possible to rapidly analyze visual and audio information and provide accurate feedback to the user in real time.

[0662] A "wearable device" is an electronic device that a user wears and uses, and that has the function of collecting visual and auditory information.

[0663] "Visual information" refers to image data and video data acquired through cameras and image acquisition devices.

[0664] "Audio information" refers to sound data collected through audio acquisition devices or microphones.

[0665] "Transmission" refers to transferring collected data to a server using communication methods such as the internet.

[0666] "Analysis" is the process of converting data into a digital format and processing it to make its contents easier to understand.

[0667] "Textual information" refers to text data extracted from visual information.

[0668] "Natural language processing technology" is a technology that enables computers to understand, analyze, and generate human language.

[0669] "Keywords" are words or phrases that are considered particularly important within a text.

[0670] A "summary" is a short, concise version of a longer text, focusing on the most important points.

[0671] Translation is the process of converting text written in one language into another language.

[0672] "Decoding" is the process of returning encoded data to its original format.

[0673] "Real-time" refers to temporal immediacy, meaning that processing and results are provided almost simultaneously.

[0674] "Rendering" is the process of visually displaying data.

[0675] A HUD (Head-Up Display) is a display technology that projects information directly into the user's field of vision.

[0676] This invention is a system that collects and analyzes visual and auditory information using a wearable device and provides real-time feedback to the user. This system enables users to efficiently acquire information and improve their learning and communication.

[0677] Hardware and software configuration

[0678] hardware

[0679] Wearable devices (devices): These are glasses-type devices with a built-in camera and microphone. The camera collects visual information, and the microphone collects audio information.

[0680] Data analysis server (server): A high-performance computer used to analyze received data.

[0681] software

[0682] OCR software: Software such as Tesseract OCR is used to extract text from visual information.

[0683] Natural Language Processing (NLP): This involves using tools such as NLTK (Natural Language Toolkit) to analyze text, extract important keywords, and generate summaries.

[0684] Speech recognition software: This includes tools such as the Google Speech-to-Text API, which converts speech information into text.

[0685] Translation APIs: APIs such as the Google Translate API are used to translate text into other languages.

[0686] The process of data collection and transmission

[0687] When a user puts on a glasses-type wearable device, the device's camera captures visual information and its microphone collects audio information. This data is encoded in a digital format and sent to a server via the internet.

[0688] Data analysis process

[0689] The server decodes the received visual information and extracts text from the image using OCR technology. Then, it extracts important keywords using natural language processing technology and summarizes the text. The server also decodes the received audio information and converts it to text using speech recognition technology. The converted text is then translated into the target language in real time using a translation API.

[0690] Provide feedback

[0691] The server packages the analysis and translation results and sends them back to the terminal via the internet. The terminal decodes and renders the received information in a layout appropriate for the user's view. The user can then review the displayed information and continue learning or conversing.

[0692] Specific example

[0693] Examples of textbook summaries

[0694] 1. The user puts on their glasses and begins reading the textbook.

[0695] 2. The device captures a page from the textbook using its camera and sends the image to the server.

[0696] 3. The server extracts text from the image, uses natural language processing techniques to extract important keywords, and generates a summary.

[0697] 4. The server sends the summary results to the terminal.

[0698] 5. The device displays a summary within the user's field of view.

[0699] 6. The user reviews the summary and continues learning.

[0700] Examples of real-time translation of foreign language conversations

[0701] 1. The user puts on glasses and starts a conversation with a foreigner in English.

[0702] 2. The device's microphone collects the conversation and sends the audio data to the server.

[0703] 3. The server converts the speech to text and translates it into the target language in real time.

[0704] 4. The server sends the translation result to the terminal.

[0705] 5. The device displays the translation result within the user's field of view.

[0706] 6. The user confirms the displayed translation and continues the conversation.

[0707] Example of a prompt

[0708] Example prompt for generating a textbook summary:

[0709] "Extract the text from the following image and summarize the key points."

[0710] Image: [Image of a textbook page]

[0711] Example prompt for real-time translation of foreign language conversations:

[0712] "Please convert the following audio data to text and translate it into the target language."

[0713] Audio data: [Audio files of foreign language conversations]

[0714] Thus, the system of the present invention enables real-time analysis and feedback of visual and auditory information, efficiently supporting user learning and communication.

[0715] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0716] Step 1:

[0717] The user wears a wearable device.

[0718] Specific action: Place the glasses-type device on your ears and adjust it to fit properly.

[0719] Input: None.

[0720] Output: User wearing the device.

[0721] Step 2:

[0722] The device captures visual information.

[0723] Specific operation: The built-in camera acquires visual information (e.g., textbook pages) in image format.

[0724] Input: Textbook pages or visual objects.

[0725] Output: Acquired image data.

[0726] Step 3:

[0727] The device collects voice information.

[0728] Specific operation: The built-in microphone records ambient sounds (e.g., conversations in a foreign language or lecture content).

[0729] Input: Ambient sound.

[0730] Output: Collected audio data.

[0731] Step 4:

[0732] The device encodes visual and audio information into a digital format.

[0733] Specific operation: Convert the acquired images and collected audio into a digital format and compress the data.

[0734] Input: Image data, audio data.

[0735] Output: Visual and audio information converted to digital format.

[0736] Step 5:

[0737] The device sends the encoded data to the server via the internet.

[0738] Specific operation: Send data using a secure communication protocol (HTTPS).

[0739] Input: Visual and audio information converted into digital format.

[0740] Output: Data sent to the server.

[0741] Step 6:

[0742] The server decodes the received visual information and extracts text using OCR technology.

[0743] Specific operation: Input image data into OCR software and convert it to text format.

[0744] Input: Visual information (image data) sent to the server.

[0745] Output: Extracted text data.

[0746] Step 7:

[0747] The server generates a summary of the extracted text using natural language processing techniques.

[0748] Specific operation: Apply an NLP algorithm to extract key keywords and generate a summary.

[0749] Input: Extracted text data.

[0750] Output: Summary text.

[0751] Step 8:

[0752] The server decodes the received audio information and uses speech recognition technology to convert the audio into text.

[0753] Specific operation: Input audio data into speech recognition software and convert it to text format.

[0754] Input: Audio information (audio data) sent to the server.

[0755] Output: Text data converted by speech recognition.

[0756] Step 9:

[0757] The server translates the converted text in real time using a translation API.

[0758] Specific operation: Input text data into the translation API and retrieve the translation result.

[0759] Input: Text data converted by speech recognition.

[0760] Output: Translated text data.

[0761] Step 10:

[0762] The server packages the analysis and translation results and sends them to the terminal.

[0763] Specific operation: Package the summary and translation results and send them using a secure communication protocol (HTTPS).

[0764] Input: Summary text, translated text data.

[0765] Output: Data sent to the terminal.

[0766] Step 11:

[0767] The device decodes and renders the received information in a layout appropriate to the user's field of view.

[0768] Specific operation: Information is overlaid onto the user's field of view using display technology.

[0769] Input: Summary text and translated text data sent to the server.

[0770] Output: Summary text displayed in the user's field of view, translation result.

[0771] Step 12:

[0772] The user reviews the displayed information and continues learning or conversing.

[0773] Specific actions: Users use summaries and translation results to understand materials and engage in conversations in a foreign language.

[0774] Input: Information displayed within the user's field of view.

[0775] Output: Continued learning or conversation.

[0776] (Application Example 1)

[0777] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0778] Conventional factory monitoring systems have struggled to monitor production lines and equipment in real time, making them ineffective for anomaly detection and improving production efficiency. Furthermore, even when visual and auditory information was collected, no system existed that could quickly analyze it and provide appropriate feedback. Therefore, there is a need for early detection of anomalies and real-time response in factory production lines.

[0779] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0780] In this invention, the server includes means for analyzing transmitted visual information and extracting text, means for extracting important keywords from the extracted text using natural language processing technology and summarizing the text, and means for monitoring production lines and equipment in real time using a mobile device that patrols the factory. This enables real-time monitoring of production lines and equipment, allowing for early detection of abnormalities and providing appropriate feedback to improve production efficiency.

[0781] A "wearable device" is a portable electronic device that a user can wear.

[0782] A "camera" is an optical device used to capture visual information.

[0783] "Visual information" refers to image and video data acquired using a camera.

[0784] A "microphone" is an acoustic device used to collect voice information.

[0785] "Audio information" refers to sound data collected through a microphone.

[0786] "Transmission means" refers to a mechanism for encoding captured visual information and collected audio information into a digital format and transmitting it to a remote location.

[0787] "Means for extracting text" refers to technologies for identifying and extracting textual information from visual information.

[0788] "Natural language processing technology" refers to the technology that enables computers to understand, interpret, and generate human language.

[0789] "Methods for extracting important keywords" refer to techniques for selecting particularly meaningful words and phrases from text.

[0790] "Methods for summarizing text" are techniques for making the content concise while preserving the overall meaning of the text.

[0791] "Methods for converting speech to text" refer to technologies that convert collected speech data into textual information.

[0792] "Translation methods" refer to the techniques used to convert text written in one language into another language.

[0793] "Means of providing feedback" refers to a mechanism for communicating the analyzed and translated results to the user.

[0794] "Mobile devices that patrol within a factory" refers to devices such as robots that move autonomously or remotely within a factory to collect data.

[0795] "Methods for monitoring production lines and equipment in real time" refer to technologies for continuously observing the production process and the state of equipment, and for immediately detecting abnormalities.

[0796] "Methods for generating reports" refer to the techniques used to analyze collected data, organize the results, and create a report.

[0797] This invention relates to a system aimed at efficient monitoring and anomaly detection of production lines and equipment within a factory. This system uses wearable devices to collect visual and auditory information, analyzes it, and provides real-time feedback to the user. Specific embodiments are described below.

[0798] System Overview

[0799] This system consists of three main components: a wearable device (hereinafter referred to as "terminal"), a server that performs data analysis (hereinafter referred to as "server"), and users who utilize the service. This enables anomaly detection and improved production efficiency in factory environments.

[0800] 1. Data collection and transmission

[0801] Terminal: A mobile device that patrols the factory, such as a factory robot, equipped with a camera and microphone, collects visual information (production line and equipment status) and audio information (worker instructions and ambient sounds). A standard industrial camera can be used as the camera, and a high-sensitivity microphone is used for audio collection.

[0802] Terminal: Encodes collected image and audio data into a digital format and transmits it to a server via the internet. For example, it uses Wi-Fi or a wired connection.

[0803] 2. Data Analysis

[0804] Server: Decodes visual information and extracts text from images using OCR technology. Based on this text, natural language processing techniques are applied to extract important keywords and generate a summary. Specific software examples include Python, OpenCV, and Tesseract.

[0805] Server: Decodes the received audio information and converts it to text using speech recognition technology. It then sends it to a translation API for real-time translation into the target language. Examples of use cases include the Python library SpeechRecognition and GoogleTrans.

[0806] 3. Provide feedback

[0807] Server: Packages analysis results, such as summaries and translations, and sends them to the terminal. During this process, it uses web frameworks such as Flask or Django to format the data.

[0808] Terminal: Decodes and renders received information in an appropriate layout for display on monitors within the factory and on workers' smartphones / tablets.

[0809] User: Review the displayed information to help identify anomalies and improve production efficiency. Audio and visual notifications are used as specific feedback methods.

[0810] Specific example

[0811] Example 1: Anomaly detection in factory robots

[0812] 1. Terminal: A robot moving around the factory takes pictures of the production line with its camera.

[0813] 2. Server: Uses OCR to read product labels and equipment status from the received video. Generates an alert if an anomaly is detected.

[0814] 3. User: Check alerts on your mobile device and take immediate action.

[0815] Example 2: Analysis of work instructions

[0816] 1. Terminal: Records the worker's conversation using a microphone and sends the audio data to the server.

[0817] 2. Server: Converts speech to text and extracts important keywords and instructions.

[0818] 3. User: Check the progress of work and instructions on a mobile device to proceed with work efficiently.

[0819] Example of a prompt

[0820] Please explain how factory robots detect anomalies on the production line in real time.

[0821] Please provide specific examples of how data collection and analysis can be performed using a robot's camera and microphone.

[0822] Please explain in detail how to improve work efficiency in a factory using visual and audio analysis technologies.

[0823] This system will enable highly efficient and effective real-time monitoring of production lines and equipment within the factory.

[0824] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0825] Step 1:

[0826] The terminal patrols the factory, capturing visual information with its camera and collecting audio information with its microphone. Inputs include video of the factory's production lines and equipment, as well as conversations among workers and ambient sounds. Outputs include digitally encoded video and audio data.

[0827] Step 2:

[0828] The terminal transmits the collected visual and audio information to the server via the internet. The input is the data encoded in step 1, and the output is the transmission of data to the server.

[0829] Step 3:

[0830] The server decodes the received visual information and extracts text from the image using OCR technology. Specifically, text information from product labels and display panels in the video is used as input, and the output is obtained by converting it into text data.

[0831] Step 4:

[0832] The server applies natural language processing techniques to the extracted text, extracting important keywords and summarizing the text. The input is the text data obtained in step 3, and the output generates important keywords and a summarized text.

[0833] Step 5:

[0834] The server decodes the received audio information and uses speech recognition technology to convert the audio into text. Specifically, the workers' conversations are used as input, and the output is obtained by converting them into text information.

[0835] Step 6:

[0836] The server sends the text converted from the speech to a translation API, which translates it into the target language in real time. The input is the text data obtained in step 5, and the output is the translated text.

[0837] Step 7:

[0838] The server packages the analyzed and translated results and sends them to the terminal. The input is the data obtained in steps 4 and 6, and the output is a feedback package that is generated and sent to the terminal.

[0839] Step 8:

[0840] The device decodes and renders the received feedback information in an appropriate layout and displays it visually to the user. Specifically, the data is formatted and displayed, and visual notifications and alerts are shown on the user's device.

[0841] Step 9:

[0842] The user checks the information displayed on the terminal and takes immediate action to respond to any abnormalities in the factory. The input is the information displayed in step 8, and the output is improvements to the user's actions or responses to the abnormalities.

[0843] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0844] This invention relates to a system that uses a wearable device to collect visual and auditory information, analyzes it, and provides real-time feedback to the user. Furthermore, this system incorporates an emotion recognition engine that can analyze the user's emotions and adjust the displayed information and learning content accordingly. This not only improves learning efficiency but also provides flexible support tailored to the user's emotional state.

[0845] System Overview

[0846] This system consists of three main components: a wearable device (hereinafter referred to as "terminal"), a server that performs data analysis (hereinafter referred to as "server"), and users who utilize the service.

[0847] 1. Data collection and transmission

[0848] Device: The user wears a glasses-type wearable device. The device has a built-in camera and microphone. The camera captures visual information (e.g., pages of a book or document). The microphone collects audio information (e.g., a conversation in a foreign language).

[0849] Terminal: Encodes collected image and audio data into a digital format and transmits it to the server via the internet.

[0850] 2. Data Analysis

[0851] Server: Decodes the received visual information (image data) and extracts text from the image using OCR technology. Based on this text, natural language processing technology is applied to extract important keywords and generate a summary.

[0852] Server: Decodes the received audio information and converts the audio to text using speech recognition technology. This text is then sent to a translation API for real-time translation into the target language.

[0853] Server: Uses an emotion recognition engine to analyze the user's emotions and incorporates the analysis results into the feedback.

[0854] 3. Provide feedback

[0855] Server: Packages analysis results such as summaries, translations, and sentiment analysis results, and sends them to the terminal.

[0856] Terminal: Decodes and renders received information in an appropriate layout to display it in the user's field of view.

[0857] User: Review the displayed information and continue learning or conversing.

[0858] Specific example

[0859] Example 1: Textbook summary display and emotional feedback

[0860] User: Puts on glasses and begins reading textbook.

[0861] Device: Captures textbook pages with the camera and sends the images to the server. Collects facial expressions and voice tone with the microphone.

[0862] Server: Extracts text from images, uses natural language processing techniques to extract important keywords, and generates summaries. It also analyzes user emotions using an emotion recognition engine.

[0863] Server: Sends summary results and sentiment analysis results to the terminal.

[0864] Device: Displays summaries and sentiment feedback within the user's field of view.

[0865] User: Review the summary and consider the next learning steps based on the sentiment feedback.

[0866] Example 2: Real-time translation and sentiment recognition of foreign language conversations

[0867] User: Puts on glasses and starts a conversation with a foreigner in English.

[0868] Terminal: The microphone collects conversation data and sends it to the server along with the user's tone of voice and facial expressions.

[0869] Server: Converts speech to text and translates it into the target language in real time. It also analyzes the user's emotions using an emotion recognition engine and includes emotion data in the conversation feedback.

[0870] Server: Sends translation results and sentiment analysis results to the terminal.

[0871] Device: Displays translation results and sentiment feedback within the field of view.

[0872] User: Review the displayed translation and sentiment feedback, then continue the conversation.

[0873] In this way, this system analyzes the user's visual and auditory information in real time, and in addition, recognizes emotions, thereby providing feedback tailored to the user's emotional state, and achieving more efficient and personalized learning support.

[0874] The following describes the processing flow.

[0875] Step 1:

[0876] User: Put on the wearable device glasses and get ready.

[0877] Step 2:

[0878] Device: The device powers on, and the camera and microphone enter standby mode.

[0879] Step 3:

[0880] User: Puts on glasses and begins reading visual information (e.g., textbooks or documents).

[0881] Step 4:

[0882] Device: The camera captures visual information and generates image data.

[0883] Step 5:

[0884] Device: Simultaneously, the microphone collects ambient audio information and saves it as audio data.

[0885] Step 6:

[0886] Terminal: Encodes visual information (image data) into binary format and converts audio information into an appropriate format (e.g., PCM or FLAC format).

[0887] Step 7:

[0888] Terminal: Sends encoded and converted data to the server via the internet.

[0889] Step 8:

[0890] Server: Decodes visual information data and extracts text from images using OCR technology.

[0891] Step 9:

[0892] Server: Based on the extracted text data, it uses natural language processing techniques to extract important keywords and generate a summary.

[0893] Step 10:

[0894] Server: Decodes the audio information and converts the audio to text using speech recognition technology.

[0895] Step 11:

[0896] Server: Sends the converted text to the translation API, which translates it into the target language in real time.

[0897] Step 12:

[0898] Server: Analyzes the user's emotional state using an emotion recognition engine. This includes the process of identifying emotions from voice tone and facial expressions.

[0899] Step 13:

[0900] Server: Packages the summary, translation results, and sentiment analysis results and sends them to the terminal.

[0901] Step 14:

[0902] Terminal: Decodes the received data and renders it in an appropriate layout for display in the user's field of view.

[0903] Step 15:

[0904] User: Check the information displayed in their field of view on the device (summary, keywords, translation results, sentiment feedback, etc.) and continue learning or conversing.

[0905] Step 16:

[0906] Terminal: Continues to collect ongoing visual and auditory information and repeats the process from step 6.

[0907] Specific example

[0908] Example 1: Textbook summary display and emotional feedback

[0909] Step 1:

[0910] User: Puts on glasses and begins reading textbook.

[0911] Step 2:

[0912] Device: Captures textbook pages with the camera and sends the images to the server. Collects facial expressions and voice tone with the microphone.

[0913] Step 3:

[0914] Server: Extracts text from images, uses natural language processing techniques to extract important keywords, and generates summaries. It also analyzes user emotions using an emotion recognition engine.

[0915] Step 4:

[0916] Server: Sends summary results and sentiment analysis results to the terminal.

[0917] Step 5:

[0918] Device: Displays summaries and sentiment feedback within the user's field of view.

[0919] Step 6:

[0920] User: Review the summary and consider the next learning steps based on the sentiment feedback.

[0921] Example 2: Real-time translation and sentiment recognition of foreign language conversations

[0922] Step 1:

[0923] User: Puts on glasses and starts a conversation with a foreigner in English.

[0924] Step 2:

[0925] Terminal: The microphone collects conversation data and sends it to the server along with the user's tone of voice and facial expressions.

[0926] Step 3:

[0927] Server: Converts speech to text and translates it into the target language in real time. It also analyzes the user's emotions using an emotion recognition engine and includes emotion data in the conversation feedback.

[0928] Step 4:

[0929] Server: Sends translation results and sentiment analysis results to the terminal.

[0930] Step 5:

[0931] Device: Displays translation results and sentiment feedback within the field of view.

[0932] Step 6:

[0933] User: Review the displayed translation and sentiment feedback, then continue the conversation.

[0934] Based on the steps described above, this system analyzes the user's visual and auditory information in real time, and in addition, recognizes emotions, thereby providing more personalized learning support.

[0935] (Example 2)

[0936] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0937] Conventional learning support systems and conversation support systems function independently, making it difficult to integrate and analyze visual and auditory information to provide real-time feedback. Furthermore, because they only provide simple information without considering the user's emotional state, the user's learning efficiency and communication experience were not optimized. The present invention aims to solve these problems and provide a system that analyzes visual and auditory information in real time, recognizes the user's emotions, and provides appropriate feedback.

[0938] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0939] In this invention, the server includes means for decoding transmitted visual information and extracting text using optical character recognition technology, means for extracting important keywords from the extracted text using natural language processing technology and summarizing the text, and means for decoding transmitted audio information and converting the audio to text using speech recognition technology. This enables integrated analysis of visual and audio information and real-time feedback that responds to the user's emotional state.

[0940] A "wearable device" is a portable electronic device that a user wears to collect visual and auditory information.

[0941] A "camera" is a device used to capture visual information.

[0942] A "microphone" is an acoustic conversion device used to collect sound information.

[0943] "Encoding into a digital format" is the process of converting an analog signal into a digital signal.

[0944] "Transmission" refers to sending collected data to another device or server.

[0945] "Decoding" is the process of converting transmitted digital data back into its original format.

[0946] "Optical character recognition technology" is a technology that automatically recognizes characters from images and converts them into text.

[0947] "Natural language processing technology" is a technology that processes and analyzes text written in natural language.

[0948] "Speech recognition technology" is a technology that analyzes speech data and converts it into text.

[0949] "Translation" refers to the process of converting text written in one language or spoken content into another language.

[0950] An "emotion recognition engine" is a general term for algorithms and technologies used to analyze and identify a user's emotions.

[0951] "Feedback" refers to providing users with information and data, such as analysis results or translation results.

[0952] This invention relates to a system that allows users to collect visual and auditory information using a wearable device, analyze it, and provide real-time feedback to the user. Furthermore, this system incorporates an emotion recognition engine that can analyze the user's emotions and adjust the displayed information and learning content accordingly. This improves learning efficiency and provides flexible support tailored to the user's emotional state. Specific embodiments of this system are described below.

[0953] Hardware configuration

[0954] Device: A wearable device worn by the user (e.g., smart glasses). This device has a built-in camera and microphone to collect visual and audio information.

[0955] Server: A remote server that performs data analysis. This server possesses advanced computing capabilities and processes and analyzes the received data.

[0956] Software Configuration

[0957] Camera: A software module for capturing visual information. For example, it can be used to photograph text information on books or signs.

[0958] Microphone: A software module for collecting audio information. For example, it can be used to record conversations or ambient sounds.

[0959] Optical Character Recognition (OCR): A technology for extracting characters from images. For example, using the "Google Cloud Vision API".

[0960] Natural Language Processing (NLP): A technique that analyzes extracted text and identifies and summarizes important keywords. For example, it uses libraries such as the "NLTK library."

[0961] Speech-to-Text (ASR) technology: A technology that converts speech into text. For example, using the "Google Speech-to-Text API".

[0962] Translation API: A technology for translating text into other languages. For example, using the "Google Translate API".

[0963] Emotion recognition engine: An algorithm for analyzing a user's emotions. For example, it uses the "Microsoft Azure Emotion API".

[0964] Specific example

[0965] Example 1: Textbook summary display and emotional feedback

[0966] User: Puts on glasses and begins reading textbook.

[0967] Device: Captures textbook pages with the camera and sends the images to the server. Collects facial expressions and voice tone with the microphone.

[0968] Server: Extracts text from images, uses natural language processing techniques to extract important keywords, and generates summaries. It also analyzes user emotions using an emotion recognition engine.

[0969] Server: Sends summary results and sentiment analysis results to the terminal.

[0970] Device: Displays summaries and sentiment feedback within the user's field of view.

[0971] User: Review the summary and consider the next learning steps based on the sentiment feedback.

[0972] Example 2: Real-time translation and sentiment recognition of foreign language conversations

[0973] User: Puts on glasses and starts a conversation with a foreigner in English.

[0974] Terminal: The microphone collects conversation data and sends it to the server along with the user's tone of voice and facial expressions.

[0975] Server: Converts speech to text and translates it into the target language in real time. It also analyzes the user's emotions using an emotion recognition engine and includes emotion data in the conversation feedback.

[0976] Server: Sends translation results and sentiment analysis results to the terminal.

[0977] Device: Displays translation results and sentiment feedback within the field of view.

[0978] User: Review the displayed translation and sentiment feedback, then continue the conversation.

[0979] Example of a prompt

[0980] For textbooks: "When a user reads the textbook, key keywords should be summarized and displayed in real time."

[0981] For conversations: "When a user speaks a foreign language, perform real-time emotion recognition and provide feedback along with the translation results."

[0982] In this way, the present invention can analyze visual and auditory information in real time and further recognize emotions to provide personalized feedback to the user.

[0983] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0984] Step 1:

[0985] Data collection

[0986] Terminal: The user wears a wearable device and captures visual information using a camera. Simultaneously, audio information is collected using a microphone.

[0987] Input: The visual object the user is looking at (e.g., a page in a textbook) and the sound they are hearing (e.g., surrounding conversation).

[0988] Output: Captured visual information (image data) and collected audio information (audio data).

[0989] Specific actions: Take a picture of a textbook page with the camera and encode the image in JPEG format. Record a conversation with the microphone and encode the audio in WAV format.

[0990] Step 2:

[0991] Data transmission

[0992] Terminal: Encodes collected visual and audio information into a digital format and transmits it to the server via the internet.

[0993] Input: Encoded image and audio data.

[0994] Output: Data sent to the server.

[0995] Specific operation: Compresses JPEG image data and WAV audio data, and sends them to a server via the internet.

[0996] Step 3:

[0997] Analysis of visual information

[0998] Server: Decodes the received image data and extracts text from the image using OCR technology.

[0999] Input: The encoded image data that was sent.

[1000] Output: Extracted text data.

[1001] Specific operation: Decode JPEG image data and extract text using OCR technology (e.g., Google Cloud Vision API). For example, extract the text "differential and integral calculus" from a page in a math textbook.

[1002] Step 4:

[1003] Natural Language Processing

[1004] Server: Extracts important keywords from the extracted text and applies natural language processing techniques to summarize the text.

[1005] Input: Text data extracted using OCR technology.

[1006] Output: Summary and keyword extraction results.

[1007] Specific operation: Natural language processing techniques (e.g., NLTK library) are used on the extracted text data to extract important keywords and generate a summary. For example, it might summarize as "Explanation of the basic principles of differentiation."

[1008] Step 5:

[1009] Voice analysis

[1010] Server: Decodes the received audio data and converts the audio to text using speech recognition technology.

[1011] Input: The encoded audio data that has been sent.

[1012] Output: Text data converted from speech.

[1013] Specific operation: Decodes WAV audio data and converts it to text using speech recognition technology (e.g., Google Speech-to-Text API). For example, converts "What is your name?" to text.

[1014] Step 6:

[1015] translation

[1016] Server: Sends the text converted from speech to a translation API, which translates it into the target language in real time.

[1017] Input: Data converted to text using speech recognition technology.

[1018] Output: Translation result.

[1019] Specific operation: Text data is sent to a translation API (e.g., Google Translate API) and translated in real time. For example, "What is your name?" is translated to "What is your name?".

[1020] Step 7:

[1021] Emotion analysis

[1022] Server: Analyzes user emotions using an emotion recognition engine and compiles the results.

[1023] Input: User's facial expression data and voice data.

[1024] Output: Emotion analysis results.

[1025] Specific operation: The collected facial expression data and voice tone are analyzed using an emotion recognition engine (e.g., Microsoft Azure Emotion API) to identify emotions such as "joy" or "sadness." For example, the analysis result might be "the user is satisfied."

[1026] Step 8:

[1027] Feedback generation and transmission

[1028] Server: Packages the summary results, translation results, and sentiment analysis results and sends them to the terminal.

[1029] Input: Summarization results, translation results, and sentiment analysis results.

[1030] Output: Feedback data sent to the terminal.

[1031] Specific operation: Integrate the results of summarization, translation, and sentiment analysis, format them appropriately, and send them to the terminal. For example, send the textbook summary, conversation translation results, and user sentiment analysis results as a single data package to the terminal in JPEG and text formats.

[1032] Step 9:

[1033] Displaying feedback

[1034] Terminal: Decodes the received feedback data and displays it appropriately within the user's field of view.

[1035] Input: Feedback data sent from the server.

[1036] Output: Visually displayed feedback information.

[1037] Specific operation: Adjust the layout and font size to display the decoded feedback data on the smart glasses' display.

[1038] Step 10:

[1039] Feedback review and action

[1040] User: Review the displayed feedback information and decide on the next action.

[1041] Input: Feedback information displayed on the screen.

[1042] Output: The user's next action (e.g., continue learning, continue conversation).

[1043] Specific action: Decide whether to continue learning by reading the textbook summary or to continue the conversation in the foreign language based on the translation result.

[1044] (Application Example 2)

[1045] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[1046] Conventional wearable device-based information collection and analysis systems can collect and analyze visual and auditory information, but they cannot provide feedback that takes into account the user's emotional state. As a result, the user experience is uniform, and individualized support is difficult. Furthermore, improving customer service in physical stores has been challenging, as it has been difficult to provide appropriate support tailored to the individual needs and emotions of each customer.

[1047] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[1048] In this invention, the server includes means for analyzing visual information and extracting text, means for analyzing audio information and converting audio to text, and means for analyzing customer emotions and providing real-time emotional feedback. This makes it possible to provide feedback tailored to the customer's emotional state, in addition to text summarization and translation results, based on the collected visual and audio information.

[1049] A "wearable device" is an electronic device that a user can wear and that is equipped with a wide variety of sensors and communication functions.

[1050] "Visual information" refers to images and video data captured using a camera.

[1051] "Audio information" refers to audio data collected using a microphone.

[1052] "Means of transmission" refers to a system that includes communication means for transferring collected data to a server or other device.

[1053] "Means of analysis" include technologies that analyze collected data and perform text extraction and speech recognition.

[1054] "Natural language processing technology" refers to all technologies that use computers to understand and generate human language.

[1055] "Key keywords" refer to words or phrases that are deemed particularly important within text data analyzed using natural language processing technology.

[1056] "Methods of summarization" refer to techniques for extracting key points from analyzed text data and compiling them into a concise format.

[1057] "Translation methods" refer to technologies that convert analyzed text data into other languages.

[1058] "Emotional feedback" refers to the process of analyzing a user's emotional state and providing specific responses or messages based on the results.

[1059] A "server" refers to a remote device that processes collected data and performs tasks such as analysis, text extraction, translation, and sentiment analysis.

[1060] "Decoding" refers to the process of converting digital data into analog or other formats.

[1061] "Rendering" refers to the process of processing digital data to display it visually.

[1062] A "user" refers to a person who uses a wearable device to receive feedback from the system.

[1063] This invention is a system that uses a wearable device to collect visual and auditory information, analyzes it in real time, and provides feedback to the user. Furthermore, by using emotion recognition technology to provide feedback tailored to the user's emotional state, it enables more effective learning and service support.

[1064] System Overview

[1065] This system consists of the following main components:

[1066] Wearable devices (devices): Glasses-type or head-mounted display-type devices equipped with a camera and microphone.

[1067] Server: A remote server that performs data analysis and generates feedback.

[1068] User: An individual who uses a wearable device.

[1069] 1. Data collection and transmission

[1070] Terminal: The user wears a glasses-type wearable device that captures visual information with its built-in camera and collects audio information with its microphone. This allows for the acquisition of visual and audio data in environments such as physical stores and classrooms.

[1071] Terminal: Collected visual information (e.g., images of product packaging) and audio information (e.g., conversations between customers and store clerks) are encoded in digital format and transmitted to a server via the internet.

[1072] 2. Data Analysis

[1073] Server: Converts received visual information into text using OCR technology, extracts important keywords using natural language processing technology, and summarizes the text. For example, it can generate detailed product information or sales information.

[1074] Server: Converts received audio information into text using speech recognition technology and translates it into other languages ​​using a translation API as needed. It also processes the analysis results using a generative AI model.

[1075] Server: Uses an emotion recognition engine to analyze the user's emotions from their voice and visual information and incorporates this into the feedback. For example, it can determine whether the customer is relaxed or stressed.

[1076] 3. Provide feedback

[1077] Server: Compiles the analysis results (summary, translation results, sentiment analysis results, etc.) and sends them to the terminal in an appropriate layout.

[1078] Terminal: Decodes and renders information to display analysis results within the user's field of view. For example, product summary information or emotion-based guidance messages may be displayed within the user's field of view.

[1079] User: Based on the feedback provided, decide on the next course of action. For example, look at other products, ask additional questions, etc.

[1080] Specific example

[1081] Example 1: Providing product information and emotional feedback

[1082] User: Walks around the store wearing smart glasses.

[1083] Device: Captures product packaging with a camera and collects conversation content with a microphone.

[1084] Server: Analyzes received visual information using OCR and summarizes product information. Analyzes customer emotions from audio information and provides information smoothly if the customer is relaxed.

[1085] Terminal: Displays product information and emotion-based messages within the user's field of view.

[1086] User: Choose a product based on the information provided, or ask the store clerk additional questions.

[1087] Example of a prompt

[1088] Image data: "Captured product image"

[1089] Audio data: "Audio of customer conversation"

[1090] Emotional data: "Analyzed emotional state (e.g., relaxed, tense)"

[1091] This system provides more personalized and effective service support by offering feedback based on the customer's emotional state. The hardware used includes smart glasses (e.g., general wearable devices) and servers (e.g., cloud servers), while the software utilizes OCR technology, speech recognition technology, natural language processing technology, and an emotion recognition engine.

[1092] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1093] Step 1:

[1094] Data collection

[1095] Subject: terminal

[1096] Specific operation: The user wears smart glasses and moves around the store. The camera captures visual information (e.g., images of products), and the microphone collects audio information (e.g., conversations between the user and store staff).

[1097] Input: Products viewed by the user, or audio uttered by the user.

[1098] Output: Digital image data and audio data.

[1099] Step 2:

[1100] Data transmission

[1101] Subject: terminal

[1102] Specific operation: The collected visual and audio information is sent to the server via the internet.

[1103] Input: Digital image and audio data.

[1104] Output: Data sent to the server.

[1105] Step 3:

[1106] Analysis of visual information

[1107] Subject: Server

[1108] Specific operation: The transmitted visual information is analyzed using OCR technology to extract text from the image. Natural language processing technology is used to identify important keywords in the extracted text and summarize the text.

[1109] Input: Image data sent to the server.

[1110] Output: Extracted text and its summary.

[1111] Step 4:

[1112] Analysis of audio information

[1113] Subject: Server

[1114] Specific operation: The transmitted audio information is converted into text using speech recognition technology. Then, the converted text is analyzed using natural language processing technology and translated into other languages ​​using a translation API as needed.

[1115] Input: Audio data sent to the server.

[1116] Output: Text converted from speech and its translation.

[1117] Step 5:

[1118] Emotion analysis

[1119] Subject: Server

[1120] Specific operation: Using an emotion recognition engine, the system analyzes the user's emotions from their voice and visual information. It identifies the user's emotional state, such as whether they are relaxed or stressed.

[1121] Input: Audio data and visual data.

[1122] Output: Analyzed emotional state data.

[1123] Step 6:

[1124] Feedback generation

[1125] Subject: Server

[1126] Specific operation: Generates feedback based on analysis and translation results, sentiment analysis results, and sends it to the device in real time.

[1127] Input: Summarized text, translation results, sentiment analysis results.

[1128] Output: Feedback data to be displayed in the user's field of view.

[1129] Step 7:

[1130] Information display

[1131] Subject: terminal

[1132] Specific operation: Decodes and renders the received feedback data and displays it in a layout appropriate to the user's field of view.

[1133] Input: Feedback data.

[1134] Output: Information displayed in the user's field of vision (e.g., product information, emotion-based guidance messages).

[1135] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1136] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1137] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[1138] [Third Embodiment]

[1139] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[1140] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1141] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1142] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[1143] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1144] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1145] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1146] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1147] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1148] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1149] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1150] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[1151] This invention relates to a system that uses a wearable device to collect visual and audio information, analyzes it, and provides real-time feedback to the user. This system is designed to significantly improve the efficiency of education and learning, and in particular, it extracts important keywords, generates summaries, and converts and translates audio information into text.

[1152] System Overview

[1153] This system consists of three main components: a wearable device (hereinafter referred to as "terminal"), a server that performs data analysis (hereinafter referred to as "server"), and users who utilize the service.

[1154] 1. Data collection and transmission

[1155] Device: The user wears a glasses-type wearable device. The device has a built-in camera and microphone. The camera captures visual information (e.g., pages of a book or document). The microphone collects audio information (e.g., a conversation in a foreign language).

[1156] Terminal: Encodes collected image and audio data into a digital format and transmits it to the server via the internet.

[1157] 2. Data Analysis

[1158] Server: Decodes the received visual information (image data) and extracts text from the image using OCR technology. Based on this text, natural language processing technology is applied to extract important keywords and generate a summary.

[1159] Server: Decodes the received audio information and converts the audio to text using speech recognition technology. This text is then sent to a translation API for real-time translation into the target language.

[1160] 3. Provide feedback

[1161] Server: Packages analysis results, such as summaries and translations, and sends them to the terminal.

[1162] Terminal: Decodes and renders received information in an appropriate layout to display it in the user's field of view.

[1163] User: Review the displayed information and continue learning or conversing.

[1164] Specific example

[1165] Example 1: Textbook summary display

[1166] User: Puts on glasses and begins reading textbook.

[1167] Device: Capture a page from the textbook using the camera and send the image to the server.

[1168] Server: Extracts text from images, uses natural language processing techniques to extract important keywords, and generates a summary.

[1169] Server: Sends the summary results to the terminal.

[1170] Terminal: Displays a summary within the user's field of view.

[1171] User: Review the summary and continue learning.

[1172] Example 2: Real-time translation of foreign language conversations

[1173] User: Puts on glasses and starts a conversation with a foreigner in English.

[1174] Terminal: The microphone collects the conversation and sends the audio data to the server.

[1175] Server: Converts speech to text and translates it into the target language in real time.

[1176] Server: Sends the translation result to the terminal.

[1177] Terminal: Displays the translation results within the user's field of view.

[1178] User: Check the displayed translation and continue the conversation.

[1179] This series of processes allows users to efficiently acquire information and progress in their learning.

[1180] The following describes the processing flow.

[1181] Step 1:

[1182] User: Put on the wearable device glasses and get ready.

[1183] Step 2:

[1184] Device: The device powers on, and the camera and microphone enter standby mode.

[1185] Step 3:

[1186] User: Puts on glasses and begins reading visual information (e.g., textbooks or documents).

[1187] Step 4:

[1188] Device: The camera detects the user's gaze and captures visual information.

[1189] Step 5:

[1190] Device: Simultaneously, the microphone collects ambient sound information.

[1191] Step 6:

[1192] Terminal: Encodes visual information (image data) into binary format and converts audio information into an appropriate format (e.g., PCM or FLAC format).

[1193] Step 7:

[1194] Terminal: Sends encoded and converted data to the server via the internet.

[1195] Step 8:

[1196] Server: Decodes visual information data and extracts text from images using OCR technology.

[1197] Step 9:

[1198] Server: Based on the extracted text data, it uses natural language processing techniques to extract important keywords and generate a summary.

[1199] Step 10:

[1200] Server: Decodes the audio information and converts the audio to text using speech recognition technology.

[1201] Step 11:

[1202] Server: Sends the converted text to the translation API for translation into the target language.

[1203] Step 12:

[1204] Server: Formats the analysis results, including the generated summary and translation, and sends them to the terminal.

[1205] Step 13:

[1206] Terminal: Decodes the received data and prepares it for display in the field of view with the appropriate layout.

[1207] Step 14:

[1208] User: Check the information displayed in their field of view on the device (summary, keywords, translation results, etc.) and continue learning or conversing.

[1209] Step 15:

[1210] Terminal: Continues to collect ongoing visual and auditory information and repeats the process from step 6.

[1211] Based on the steps described above, a system is created that supports efficient learning by analyzing visual and auditory information in real time and providing feedback to the user.

[1212] (Example 1)

[1213] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1214] Conventional wearable device systems have the problem of not being able to collect and analyze visual and auditory information in real time, thus failing to provide users with rapid feedback. Furthermore, the accuracy and speed of extracting important information and translation results are insufficient, highlighting the need for systems that support effective learning and communication.

[1215] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1216] In this invention, the server includes means for analyzing visual information and extracting textual information, means for extracting important keywords from the extracted textual information using natural language processing techniques and summarizing the textual information, and means for translating the textual information. This makes it possible to rapidly analyze visual and audio information and provide accurate feedback to the user in real time.

[1217] A "wearable device" is an electronic device that a user wears and uses, and that has the function of collecting visual and auditory information.

[1218] "Visual information" refers to image data and video data acquired through cameras and image acquisition devices.

[1219] "Audio information" refers to sound data collected through audio acquisition devices or microphones.

[1220] "Transmission" refers to transferring collected data to a server using communication methods such as the internet.

[1221] "Analysis" is the process of converting data into a digital format and processing it to make its contents easier to understand.

[1222] "Textual information" refers to text data extracted from visual information.

[1223] "Natural language processing technology" is a technology that enables computers to understand, analyze, and generate human language.

[1224] "Keywords" are words or phrases that are considered particularly important within a text.

[1225] A "summary" is a short, concise version of a longer text, focusing on the most important points.

[1226] Translation is the process of converting text written in one language into another language.

[1227] "Decoding" is the process of returning encoded data to its original format.

[1228] "Real-time" refers to temporal immediacy, meaning that processing and results are provided almost simultaneously.

[1229] "Rendering" is the process of visually displaying data.

[1230] A HUD (Head-Up Display) is a display technology that projects information directly into the user's field of vision.

[1231] This invention is a system that collects and analyzes visual and auditory information using a wearable device and provides real-time feedback to the user. This system enables users to efficiently acquire information and improve their learning and communication.

[1232] Hardware and software configuration

[1233] hardware

[1234] Wearable devices (devices): These are glasses-type devices with a built-in camera and microphone. The camera collects visual information, and the microphone collects audio information.

[1235] Data analysis server (server): A high-performance computer used to analyze received data.

[1236] software

[1237] OCR software: Software such as Tesseract OCR is used to extract text from visual information.

[1238] Natural Language Processing (NLP): This involves using tools such as NLTK (Natural Language Toolkit) to analyze text, extract important keywords, and generate summaries.

[1239] Speech recognition software: This includes tools such as the Google Speech-to-Text API, which converts speech information into text.

[1240] Translation APIs: APIs such as the Google Translate API are used to translate text into other languages.

[1241] The process of data collection and transmission

[1242] When a user puts on a glasses-type wearable device, the device's camera captures visual information and its microphone collects audio information. This data is encoded in a digital format and sent to a server via the internet.

[1243] Data analysis process

[1244] The server decodes the received visual information and extracts text from the image using OCR technology. Then, it extracts important keywords using natural language processing technology and summarizes the text. The server also decodes the received audio information and converts it to text using speech recognition technology. The converted text is then translated into the target language in real time using a translation API.

[1245] Provide feedback

[1246] The server packages the analysis and translation results and sends them back to the terminal via the internet. The terminal decodes and renders the received information in a layout appropriate for the user's view. The user can then review the displayed information and continue learning or conversing.

[1247] Specific example

[1248] Examples of textbook summaries

[1249] 1. The user puts on their glasses and begins reading the textbook.

[1250] 2. The device captures a page from the textbook using its camera and sends the image to the server.

[1251] 3. The server extracts text from the image, uses natural language processing techniques to extract important keywords, and generates a summary.

[1252] 4. The server sends the summary results to the terminal.

[1253] 5. The device displays a summary within the user's field of view.

[1254] 6. The user reviews the summary and continues learning.

[1255] Examples of real-time translation of foreign language conversations

[1256] 1. The user puts on glasses and starts a conversation with a foreigner in English.

[1257] 2. The device's microphone collects the conversation and sends the audio data to the server.

[1258] 3. The server converts the speech to text and translates it into the target language in real time.

[1259] 4. The server sends the translation result to the terminal.

[1260] 5. The device displays the translation result within the user's field of view.

[1261] 6. The user confirms the displayed translation and continues the conversation.

[1262] Example of a prompt

[1263] Example prompt for generating a textbook summary:

[1264] "Extract the text from the following image and summarize the key points."

[1265] Image: [Image of a textbook page]

[1266] Example prompt for real-time translation of foreign language conversations:

[1267] "Please convert the following audio data to text and translate it into the target language."

[1268] Audio data: [Audio files of foreign language conversations]

[1269] Thus, the system of the present invention enables real-time analysis and feedback of visual and auditory information, efficiently supporting user learning and communication.

[1270] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1271] Step 1:

[1272] The user wears a wearable device.

[1273] Specific action: Place the glasses-type device on your ears and adjust it to fit properly.

[1274] Input: None.

[1275] Output: User wearing the device.

[1276] Step 2:

[1277] The device captures visual information.

[1278] Specific operation: The built-in camera acquires visual information (e.g., textbook pages) in image format.

[1279] Input: Textbook pages or visual objects.

[1280] Output: Acquired image data.

[1281] Step 3:

[1282] The device collects voice information.

[1283] Specific operation: The built-in microphone records ambient sounds (e.g., conversations in a foreign language or lecture content).

[1284] Input: Ambient sound.

[1285] Output: Collected audio data.

[1286] Step 4:

[1287] The device encodes visual and audio information into a digital format.

[1288] Specific operation: Convert the acquired images and collected audio into a digital format and compress the data.

[1289] Input: Image data, audio data.

[1290] Output: Visual and audio information converted to digital format.

[1291] Step 5:

[1292] The device sends the encoded data to the server via the internet.

[1293] Specific operation: Send data using a secure communication protocol (HTTPS).

[1294] Input: Visual and audio information converted into digital format.

[1295] Output: Data sent to the server.

[1296] Step 6:

[1297] The server decodes the received visual information and extracts text using OCR technology.

[1298] Specific operation: Input image data into OCR software and convert it to text format.

[1299] Input: Visual information (image data) sent to the server.

[1300] Output: Extracted text data.

[1301] Step 7:

[1302] The server generates a summary of the extracted text using natural language processing techniques.

[1303] Specific operation: Apply an NLP algorithm to extract key keywords and generate a summary.

[1304] Input: Extracted text data.

[1305] Output: Summary text.

[1306] Step 8:

[1307] The server decodes the received audio information and uses speech recognition technology to convert the audio into text.

[1308] Specific operation: Input audio data into speech recognition software and convert it to text format.

[1309] Input: Audio information (audio data) sent to the server.

[1310] Output: Text data converted by speech recognition.

[1311] Step 9:

[1312] The server translates the converted text in real time using a translation API.

[1313] Specific operation: Input text data into the translation API and retrieve the translation result.

[1314] Input: Text data converted by speech recognition.

[1315] Output: Translated text data.

[1316] Step 10:

[1317] The server packages the analysis and translation results and sends them to the terminal.

[1318] Specific operation: Package the summary and translation results and send them using a secure communication protocol (HTTPS).

[1319] Input: Summary text, translated text data.

[1320] Output: Data sent to the terminal.

[1321] Step 11:

[1322] The device decodes and renders the received information in a layout appropriate to the user's field of view.

[1323] Specific operation: Information is overlaid onto the user's field of view using display technology.

[1324] Input: Summary text and translated text data sent to the server.

[1325] Output: Summary text displayed in the user's field of view, translation result.

[1326] Step 12:

[1327] The user reviews the displayed information and continues learning or conversing.

[1328] Specific actions: Users use summaries and translation results to understand materials and engage in conversations in a foreign language.

[1329] Input: Information displayed within the user's field of view.

[1330] Output: Continued learning or conversation.

[1331] (Application Example 1)

[1332] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1333] Conventional factory monitoring systems have struggled to monitor production lines and equipment in real time, making them ineffective for anomaly detection and improving production efficiency. Furthermore, even when visual and auditory information was collected, no system existed that could quickly analyze it and provide appropriate feedback. Therefore, there is a need for early detection of anomalies and real-time response in factory production lines.

[1334] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1335] In this invention, the server includes means for analyzing transmitted visual information and extracting text, means for extracting important keywords from the extracted text using natural language processing technology and summarizing the text, and means for monitoring production lines and equipment in real time using a mobile device that patrols the factory. This enables real-time monitoring of production lines and equipment, allowing for early detection of abnormalities and providing appropriate feedback to improve production efficiency.

[1336] A "wearable device" is a portable electronic device that a user can wear.

[1337] A "camera" is an optical device used to capture visual information.

[1338] "Visual information" refers to image and video data acquired using a camera.

[1339] A "microphone" is an acoustic device used to collect voice information.

[1340] "Audio information" refers to sound data collected through a microphone.

[1341] "Transmission means" refers to a mechanism for encoding captured visual information and collected audio information into a digital format and transmitting it to a remote location.

[1342] "Means for extracting text" refers to technologies for identifying and extracting textual information from visual information.

[1343] "Natural language processing technology" refers to the technology that enables computers to understand, interpret, and generate human language.

[1344] "Methods for extracting important keywords" refer to techniques for selecting particularly meaningful words and phrases from text.

[1345] "Methods for summarizing text" are techniques for making the content concise while preserving the overall meaning of the text.

[1346] "Methods for converting speech to text" refer to technologies that convert collected speech data into textual information.

[1347] "Translation methods" refer to the techniques used to convert text written in one language into another language.

[1348] "Means of providing feedback" refers to a mechanism for communicating the analyzed and translated results to the user.

[1349] "Mobile devices that patrol within a factory" refers to devices such as robots that move autonomously or remotely within a factory to collect data.

[1350] "Methods for monitoring production lines and equipment in real time" refer to technologies for continuously observing the production process and the state of equipment, and for immediately detecting abnormalities.

[1351] "Methods for generating reports" refer to the techniques used to analyze collected data, organize the results, and create a report.

[1352] This invention relates to a system aimed at efficient monitoring and anomaly detection of production lines and equipment within a factory. This system uses wearable devices to collect visual and auditory information, analyzes it, and provides real-time feedback to the user. Specific embodiments are described below.

[1353] System Overview

[1354] This system consists of three main components: a wearable device (hereinafter referred to as "terminal"), a server that performs data analysis (hereinafter referred to as "server"), and users who utilize the service. This enables anomaly detection and improved production efficiency in factory environments.

[1355] 1. Data collection and transmission

[1356] Terminal: A mobile device that patrols the factory, such as a factory robot, equipped with a camera and microphone, collects visual information (production line and equipment status) and audio information (worker instructions and ambient sounds). A standard industrial camera can be used as the camera, and a high-sensitivity microphone is used for audio collection.

[1357] Terminal: Encodes collected image and audio data into a digital format and transmits it to a server via the internet. For example, it uses Wi-Fi or a wired connection.

[1358] 2. Data Analysis

[1359] Server: Decodes visual information and extracts text from images using OCR technology. Based on this text, natural language processing techniques are applied to extract important keywords and generate a summary. Specific software examples include Python, OpenCV, and Tesseract.

[1360] Server: Decodes the received audio information and converts it to text using speech recognition technology. It then sends it to a translation API for real-time translation into the target language. Examples of use cases include the Python library SpeechRecognition and GoogleTrans.

[1361] 3. Provide feedback

[1362] Server: Packages analysis results, such as summaries and translations, and sends them to the terminal. During this process, it uses web frameworks such as Flask or Django to format the data.

[1363] Terminal: Decodes and renders received information in an appropriate layout for display on monitors within the factory and on workers' smartphones / tablets.

[1364] User: Review the displayed information to help identify anomalies and improve production efficiency. Audio and visual notifications are used as specific feedback methods.

[1365] Specific example

[1366] Example 1: Anomaly detection in factory robots

[1367] 1. Terminal: A robot moving around the factory takes pictures of the production line with its camera.

[1368] 2. Server: Uses OCR to read product labels and equipment status from the received video. Generates an alert if an anomaly is detected.

[1369] 3. User: Check alerts on your mobile device and take immediate action.

[1370] Example 2: Analysis of work instructions

[1371] 1. Terminal: Records the worker's conversation using a microphone and sends the audio data to the server.

[1372] 2. Server: Converts speech to text and extracts important keywords and instructions.

[1373] 3. User: Check the progress of work and instructions on a mobile device to proceed with work efficiently.

[1374] Example of a prompt

[1375] Please explain how factory robots detect anomalies on the production line in real time.

[1376] Please provide specific examples of how data collection and analysis can be performed using a robot's camera and microphone.

[1377] Please explain in detail how to improve work efficiency in a factory using visual and audio analysis technologies.

[1378] This system will enable highly efficient and effective real-time monitoring of production lines and equipment within the factory.

[1379] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1380] Step 1:

[1381] The terminal patrols the factory, capturing visual information with its camera and collecting audio information with its microphone. Inputs include video of the factory's production lines and equipment, as well as conversations among workers and ambient sounds. Outputs include digitally encoded video and audio data.

[1382] Step 2:

[1383] The terminal transmits the collected visual and audio information to the server via the internet. The input is the data encoded in step 1, and the output is the transmission of data to the server.

[1384] Step 3:

[1385] The server decodes the received visual information and extracts text from the image using OCR technology. Specifically, text information from product labels and display panels in the video is used as input, and the output is obtained by converting it into text data.

[1386] Step 4:

[1387] The server applies natural language processing techniques to the extracted text, extracting important keywords and summarizing the text. The input is the text data obtained in step 3, and the output generates important keywords and a summarized text.

[1388] Step 5:

[1389] The server decodes the received audio information and uses speech recognition technology to convert the audio into text. Specifically, the workers' conversations are used as input, and the output is obtained by converting them into text information.

[1390] Step 6:

[1391] The server sends the text converted from the speech to a translation API, which translates it into the target language in real time. The input is the text data obtained in step 5, and the output is the translated text.

[1392] Step 7:

[1393] The server packages the analyzed and translated results and sends them to the terminal. The input is the data obtained in steps 4 and 6, and the output is a feedback package that is generated and sent to the terminal.

[1394] Step 8:

[1395] The device decodes and renders the received feedback information in an appropriate layout and displays it visually to the user. Specifically, the data is formatted and displayed, and visual notifications and alerts are shown on the user's device.

[1396] Step 9:

[1397] The user checks the information displayed on the terminal and takes immediate action to respond to any abnormalities in the factory. The input is the information displayed in step 8, and the output is improvements to the user's actions or responses to the abnormalities.

[1398] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1399] This invention relates to a system that uses a wearable device to collect visual and auditory information, analyzes it, and provides real-time feedback to the user. Furthermore, this system incorporates an emotion recognition engine that can analyze the user's emotions and adjust the displayed information and learning content accordingly. This not only improves learning efficiency but also provides flexible support tailored to the user's emotional state.

[1400] System Overview

[1401] This system consists of three main components: a wearable device (hereinafter referred to as "terminal"), a server that performs data analysis (hereinafter referred to as "server"), and users who utilize the service.

[1402] 1. Data collection and transmission

[1403] Device: The user wears a glasses-type wearable device. The device has a built-in camera and microphone. The camera captures visual information (e.g., pages of a book or document). The microphone collects audio information (e.g., a conversation in a foreign language).

[1404] Terminal: Encodes collected image and audio data into a digital format and transmits it to the server via the internet.

[1405] 2. Data Analysis

[1406] Server: Decodes the received visual information (image data) and extracts text from the image using OCR technology. Based on this text, natural language processing technology is applied to extract important keywords and generate a summary.

[1407] Server: Decodes the received audio information and converts the audio to text using speech recognition technology. This text is then sent to a translation API for real-time translation into the target language.

[1408] Server: Uses an emotion recognition engine to analyze the user's emotions and incorporates the analysis results into the feedback.

[1409] 3. Provide feedback

[1410] Server: Packages analysis results such as summaries, translations, and sentiment analysis results, and sends them to the terminal.

[1411] Terminal: Decodes and renders received information in an appropriate layout to display it in the user's field of view.

[1412] User: Review the displayed information and continue learning or conversing.

[1413] Specific example

[1414] Example 1: Textbook summary display and emotional feedback

[1415] User: Puts on glasses and begins reading textbook.

[1416] Device: Captures textbook pages with the camera and sends the images to the server. Collects facial expressions and voice tone with the microphone.

[1417] Server: Extracts text from images, uses natural language processing techniques to extract important keywords, and generates summaries. It also analyzes user emotions using an emotion recognition engine.

[1418] Server: Sends summary results and sentiment analysis results to the terminal.

[1419] Device: Displays summaries and sentiment feedback within the user's field of view.

[1420] User: Review the summary and consider the next learning steps based on the sentiment feedback.

[1421] Example 2: Real-time translation and sentiment recognition of foreign language conversations

[1422] User: Puts on glasses and starts a conversation with a foreigner in English.

[1423] Terminal: The microphone collects conversation data and sends it to the server along with the user's tone of voice and facial expressions.

[1424] Server: Converts speech to text and translates it into the target language in real time. It also analyzes the user's emotions using an emotion recognition engine and includes emotion data in the conversation feedback.

[1425] Server: Sends translation results and sentiment analysis results to the terminal.

[1426] Device: Displays translation results and sentiment feedback within the field of view.

[1427] User: Review the displayed translation and sentiment feedback, then continue the conversation.

[1428] In this way, this system analyzes the user's visual and auditory information in real time, and in addition, recognizes emotions, thereby providing feedback tailored to the user's emotional state, and achieving more efficient and personalized learning support.

[1429] The following describes the processing flow.

[1430] Step 1:

[1431] User: Put on the wearable device glasses and get ready.

[1432] Step 2:

[1433] Device: The device powers on, and the camera and microphone enter standby mode.

[1434] Step 3:

[1435] User: Puts on glasses and begins reading visual information (e.g., textbooks or documents).

[1436] Step 4:

[1437] Device: The camera captures visual information and generates image data.

[1438] Step 5:

[1439] Device: Simultaneously, the microphone collects ambient audio information and saves it as audio data.

[1440] Step 6:

[1441] Terminal: Encodes visual information (image data) into binary format and converts audio information into an appropriate format (e.g., PCM or FLAC format).

[1442] Step 7:

[1443] Terminal: Sends encoded and converted data to the server via the internet.

[1444] Step 8:

[1445] Server: Decodes visual information data and extracts text from images using OCR technology.

[1446] Step 9:

[1447] Server: Based on the extracted text data, it uses natural language processing techniques to extract important keywords and generate a summary.

[1448] Step 10:

[1449] Server: Decodes the audio information and converts the audio to text using speech recognition technology.

[1450] Step 11:

[1451] Server: Sends the converted text to the translation API, which translates it into the target language in real time.

[1452] Step 12:

[1453] Server: Analyzes the user's emotional state using an emotion recognition engine. This includes the process of identifying emotions from voice tone and facial expressions.

[1454] Step 13:

[1455] Server: Packages the summary, translation results, and sentiment analysis results and sends them to the terminal.

[1456] Step 14:

[1457] Terminal: Decodes the received data and renders it in an appropriate layout for display in the user's field of view.

[1458] Step 15:

[1459] User: Check the information displayed in their field of view on the device (summary, keywords, translation results, sentiment feedback, etc.) and continue learning or conversing.

[1460] Step 16:

[1461] Terminal: Continues to collect ongoing visual and auditory information and repeats the process from step 6.

[1462] Specific example

[1463] Example 1: Textbook summary display and emotional feedback

[1464] Step 1:

[1465] User: Puts on glasses and begins reading textbook.

[1466] Step 2:

[1467] Device: Captures textbook pages with the camera and sends the images to the server. Collects facial expressions and voice tone with the microphone.

[1468] Step 3:

[1469] Server: Extracts text from images, uses natural language processing techniques to extract important keywords, and generates summaries. It also analyzes user emotions using an emotion recognition engine.

[1470] Step 4:

[1471] Server: Sends summary results and sentiment analysis results to the terminal.

[1472] Step 5:

[1473] Device: Displays summaries and sentiment feedback within the user's field of view.

[1474] Step 6:

[1475] User: Review the summary and consider the next learning steps based on the sentiment feedback.

[1476] Example 2: Real-time translation and sentiment recognition of foreign language conversations

[1477] Step 1:

[1478] User: Puts on glasses and starts a conversation with a foreigner in English.

[1479] Step 2:

[1480] Terminal: The microphone collects conversation data and sends it to the server along with the user's tone of voice and facial expressions.

[1481] Step 3:

[1482] Server: Converts speech to text and translates it into the target language in real time. It also analyzes the user's emotions using an emotion recognition engine and includes emotion data in the conversation feedback.

[1483] Step 4:

[1484] Server: Sends translation results and sentiment analysis results to the terminal.

[1485] Step 5:

[1486] Device: Displays translation results and sentiment feedback within the field of view.

[1487] Step 6:

[1488] User: Review the displayed translation and sentiment feedback, then continue the conversation.

[1489] Based on the steps described above, this system analyzes the user's visual and auditory information in real time, and in addition, recognizes emotions, thereby providing more personalized learning support.

[1490] (Example 2)

[1491] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1492] Conventional learning support systems and conversation support systems function independently, making it difficult to integrate and analyze visual and auditory information to provide real-time feedback. Furthermore, because they only provide simple information without considering the user's emotional state, the user's learning efficiency and communication experience were not optimized. The present invention aims to solve these problems and provide a system that analyzes visual and auditory information in real time, recognizes the user's emotions, and provides appropriate feedback.

[1493] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1494] In this invention, the server includes means for decoding transmitted visual information and extracting text using optical character recognition technology, means for extracting important keywords from the extracted text using natural language processing technology and summarizing the text, and means for decoding transmitted audio information and converting the audio to text using speech recognition technology. This enables integrated analysis of visual and audio information and real-time feedback that responds to the user's emotional state.

[1495] A "wearable device" is a portable electronic device that a user wears to collect visual and auditory information.

[1496] A "camera" is a device used to capture visual information.

[1497] A "microphone" is an acoustic conversion device used to collect sound information.

[1498] "Encoding into a digital format" is the process of converting an analog signal into a digital signal.

[1499] "Transmission" refers to sending collected data to another device or server.

[1500] "Decoding" is the process of converting transmitted digital data back into its original format.

[1501] "Optical character recognition technology" is a technology that automatically recognizes characters from images and converts them into text.

[1502] "Natural language processing technology" is a technology that processes and analyzes text written in natural language.

[1503] "Speech recognition technology" is a technology that analyzes speech data and converts it into text.

[1504] "Translation" refers to the process of converting text written in one language or spoken content into another language.

[1505] An "emotion recognition engine" is a general term for algorithms and technologies used to analyze and identify a user's emotions.

[1506] "Feedback" refers to providing users with information and data, such as analysis results or translation results.

[1507] This invention relates to a system that allows users to collect visual and auditory information using a wearable device, analyze it, and provide real-time feedback to the user. Furthermore, this system incorporates an emotion recognition engine that can analyze the user's emotions and adjust the displayed information and learning content accordingly. This improves learning efficiency and provides flexible support tailored to the user's emotional state. Specific embodiments of this system are described below.

[1508] Hardware configuration

[1509] Device: A wearable device worn by the user (e.g., smart glasses). This device has a built-in camera and microphone to collect visual and audio information.

[1510] Server: A remote server that performs data analysis. This server possesses advanced computing capabilities and processes and analyzes the received data.

[1511] Software Configuration

[1512] Camera: A software module for capturing visual information. For example, it can be used to photograph text information on books or signs.

[1513] Microphone: A software module for collecting audio information. For example, it can be used to record conversations or ambient sounds.

[1514] Optical Character Recognition (OCR): A technology for extracting characters from images. For example, using the "Google Cloud Vision API".

[1515] Natural Language Processing (NLP): A technique that analyzes extracted text and identifies and summarizes important keywords. For example, it uses libraries such as the "NLTK library."

[1516] Speech-to-Text (ASR) technology: A technology that converts speech into text. For example, using the "Google Speech-to-Text API".

[1517] Translation API: A technology for translating text into other languages. For example, using the "Google Translate API".

[1518] Emotion recognition engine: An algorithm for analyzing a user's emotions. For example, it uses the "Microsoft Azure Emotion API".

[1519] Specific example

[1520] Example 1: Textbook summary display and emotional feedback

[1521] User: Puts on glasses and begins reading textbook.

[1522] Device: Captures textbook pages with the camera and sends the images to the server. Collects facial expressions and voice tone with the microphone.

[1523] Server: Extracts text from images, uses natural language processing techniques to extract important keywords, and generates summaries. It also analyzes user emotions using an emotion recognition engine.

[1524] Server: Sends summary results and sentiment analysis results to the terminal.

[1525] Device: Displays summaries and sentiment feedback within the user's field of view.

[1526] User: Review the summary and consider the next learning steps based on the sentiment feedback.

[1527] Example 2: Real-time translation and sentiment recognition of foreign language conversations

[1528] User: Puts on glasses and starts a conversation with a foreigner in English.

[1529] Terminal: The microphone collects conversation data and sends it to the server along with the user's tone of voice and facial expressions.

[1530] Server: Converts speech to text and translates it into the target language in real time. It also analyzes the user's emotions using an emotion recognition engine and includes emotion data in the conversation feedback.

[1531] Server: Sends translation results and sentiment analysis results to the terminal.

[1532] Device: Displays translation results and sentiment feedback within the field of view.

[1533] User: Review the displayed translation and sentiment feedback, then continue the conversation.

[1534] Example of a prompt

[1535] For textbooks: "When a user reads the textbook, key keywords should be summarized and displayed in real time."

[1536] For conversations: "When a user speaks a foreign language, perform real-time emotion recognition and provide feedback along with the translation results."

[1537] In this way, the present invention can analyze visual and auditory information in real time and further recognize emotions to provide personalized feedback to the user.

[1538] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1539] Step 1:

[1540] Data collection

[1541] Terminal: The user wears a wearable device and captures visual information using a camera. Simultaneously, audio information is collected using a microphone.

[1542] Input: The visual object the user is looking at (e.g., a page in a textbook) and the sound they are hearing (e.g., surrounding conversation).

[1543] Output: Captured visual information (image data) and collected audio information (audio data).

[1544] Specific actions: Take a picture of a textbook page with the camera and encode the image in JPEG format. Record a conversation with the microphone and encode the audio in WAV format.

[1545] Step 2:

[1546] Data transmission

[1547] Terminal: Encodes collected visual and audio information into a digital format and transmits it to the server via the internet.

[1548] Input: Encoded image and audio data.

[1549] Output: Data sent to the server.

[1550] Specific operation: Compresses JPEG image data and WAV audio data, and sends them to a server via the internet.

[1551] Step 3:

[1552] Analysis of visual information

[1553] Server: Decodes the received image data and extracts text from the image using OCR technology.

[1554] Input: The encoded image data that was sent.

[1555] Output: Extracted text data.

[1556] Specific operation: Decode JPEG image data and extract text using OCR technology (e.g., Google Cloud Vision API). For example, extract the text "differential and integral calculus" from a page in a math textbook.

[1557] Step 4:

[1558] Natural Language Processing

[1559] Server: Extracts important keywords from the extracted text and applies natural language processing techniques to summarize the text.

[1560] Input: Text data extracted using OCR technology.

[1561] Output: Summary and keyword extraction results.

[1562] Specific operation: Natural language processing techniques (e.g., NLTK library) are used on the extracted text data to extract important keywords and generate a summary. For example, it might summarize as "Explanation of the basic principles of differentiation."

[1563] Step 5:

[1564] Voice analysis

[1565] Server: Decodes the received audio data and converts the audio to text using speech recognition technology.

[1566] Input: The encoded audio data that has been sent.

[1567] Output: Text data converted from speech.

[1568] Specific operation: Decodes WAV audio data and converts it to text using speech recognition technology (e.g., Google Speech-to-Text API). For example, converts "What is your name?" to text.

[1569] Step 6:

[1570] translation

[1571] Server: Sends the text converted from speech to a translation API, which translates it into the target language in real time.

[1572] Input: Data converted to text using speech recognition technology.

[1573] Output: Translation result.

[1574] Specific operation: Text data is sent to a translation API (e.g., Google Translate API) and translated in real time. For example, "What is your name?" is translated to "What is your name?".

[1575] Step 7:

[1576] Emotion analysis

[1577] Server: Analyzes user emotions using an emotion recognition engine and compiles the results.

[1578] Input: User's facial expression data and voice data.

[1579] Output: Emotion analysis results.

[1580] Specific operation: The collected facial expression data and voice tone are analyzed using an emotion recognition engine (e.g., Microsoft Azure Emotion API) to identify emotions such as "joy" or "sadness." For example, the analysis result might be "the user is satisfied."

[1581] Step 8:

[1582] Feedback generation and transmission

[1583] Server: Packages the summary results, translation results, and sentiment analysis results and sends them to the terminal.

[1584] Input: Summarization results, translation results, and sentiment analysis results.

[1585] Output: Feedback data sent to the terminal.

[1586] Specific operation: Integrate the results of summarization, translation, and sentiment analysis, format them appropriately, and send them to the terminal. For example, send the textbook summary, conversation translation results, and user sentiment analysis results as a single data package to the terminal in JPEG and text formats.

[1587] Step 9:

[1588] Displaying feedback

[1589] Terminal: Decodes the received feedback data and displays it appropriately within the user's field of view.

[1590] Input: Feedback data sent from the server.

[1591] Output: Visually displayed feedback information.

[1592] Specific operation: Adjust the layout and font size to display the decoded feedback data on the smart glasses' display.

[1593] Step 10:

[1594] Feedback review and action

[1595] User: Review the displayed feedback information and decide on the next action.

[1596] Input: Feedback information displayed on the screen.

[1597] Output: The user's next action (e.g., continue learning, continue conversation).

[1598] Specific action: Decide whether to continue learning by reading the textbook summary or to continue the conversation in the foreign language based on the translation result.

[1599] (Application Example 2)

[1600] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1601] Conventional wearable device-based information collection and analysis systems can collect and analyze visual and auditory information, but they cannot provide feedback that takes into account the user's emotional state. As a result, the user experience is uniform, and individualized support is difficult. Furthermore, improving customer service in physical stores has been challenging, as it has been difficult to provide appropriate support tailored to the individual needs and emotions of each customer.

[1602] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[1603] In this invention, the server includes means for analyzing visual information and extracting text, means for analyzing audio information and converting audio to text, and means for analyzing customer emotions and providing real-time emotional feedback. This makes it possible to provide feedback tailored to the customer's emotional state, in addition to text summarization and translation results, based on the collected visual and audio information.

[1604] A "wearable device" is an electronic device that a user can wear and that is equipped with a wide variety of sensors and communication functions.

[1605] "Visual information" refers to images and video data captured using a camera.

[1606] "Audio information" refers to audio data collected using a microphone.

[1607] "Means of transmission" refers to a system that includes communication means for transferring collected data to a server or other device.

[1608] "Means of analysis" include technologies that analyze collected data and perform text extraction and speech recognition.

[1609] "Natural language processing technology" refers to all technologies that use computers to understand and generate human language.

[1610] "Key keywords" refer to words or phrases that are deemed particularly important within text data analyzed using natural language processing technology.

[1611] "Methods of summarization" refer to techniques for extracting key points from analyzed text data and compiling them into a concise format.

[1612] "Translation methods" refer to technologies that convert analyzed text data into other languages.

[1613] "Emotional feedback" refers to the process of analyzing a user's emotional state and providing specific responses or messages based on the results.

[1614] A "server" refers to a remote device that processes collected data and performs tasks such as analysis, text extraction, translation, and sentiment analysis.

[1615] "Decoding" refers to the process of converting digital data into analog or other formats.

[1616] "Rendering" refers to the process of processing digital data to display it visually.

[1617] A "user" refers to a person who uses a wearable device to receive feedback from the system.

[1618] This invention is a system that uses a wearable device to collect visual and auditory information, analyzes it in real time, and provides feedback to the user. Furthermore, by using emotion recognition technology to provide feedback tailored to the user's emotional state, it enables more effective learning and service support.

[1619] System Overview

[1620] This system consists of the following main components:

[1621] Wearable devices (devices): Glasses-type or head-mounted display-type devices equipped with a camera and microphone.

[1622] Server: A remote server that performs data analysis and generates feedback.

[1623] User: An individual who uses a wearable device.

[1624] 1. Data collection and transmission

[1625] Terminal: The user wears a glasses-type wearable device that captures visual information with its built-in camera and collects audio information with its microphone. This allows for the acquisition of visual and audio data in environments such as physical stores and classrooms.

[1626] Terminal: Collected visual information (e.g., images of product packaging) and audio information (e.g., conversations between customers and store clerks) are encoded in digital format and transmitted to a server via the internet.

[1627] 2. Data Analysis

[1628] Server: Converts received visual information into text using OCR technology, extracts important keywords using natural language processing technology, and summarizes the text. For example, it can generate detailed product information or sales information.

[1629] Server: Converts received audio information into text using speech recognition technology and translates it into other languages ​​using a translation API as needed. It also processes the analysis results using a generative AI model.

[1630] Server: Uses an emotion recognition engine to analyze the user's emotions from their voice and visual information and incorporates this into the feedback. For example, it can determine whether the customer is relaxed or stressed.

[1631] 3. Provide feedback

[1632] Server: Compiles the analysis results (summary, translation results, sentiment analysis results, etc.) and sends them to the terminal in an appropriate layout.

[1633] Terminal: Decodes and renders information to display analysis results within the user's field of view. For example, product summary information or emotion-based guidance messages may be displayed within the user's field of view.

[1634] User: Based on the feedback provided, decide on the next course of action. For example, look at other products, ask additional questions, etc.

[1635] Specific example

[1636] Example 1: Providing product information and emotional feedback

[1637] User: Walks around the store wearing smart glasses.

[1638] Device: Captures product packaging with a camera and collects conversation content with a microphone.

[1639] Server: Analyzes received visual information using OCR and summarizes product information. Analyzes customer emotions from audio information and provides information smoothly if the customer is relaxed.

[1640] Terminal: Displays product information and emotion-based messages within the user's field of view.

[1641] User: Choose a product based on the information provided, or ask the store clerk additional questions.

[1642] Example of a prompt

[1643] Image data: "Captured product image"

[1644] Audio data: "Audio of customer conversation"

[1645] Emotional data: "Analyzed emotional state (e.g., relaxed, tense)"

[1646] This system provides more personalized and effective service support by offering feedback based on the customer's emotional state. The hardware used includes smart glasses (e.g., general wearable devices) and servers (e.g., cloud servers), while the software utilizes OCR technology, speech recognition technology, natural language processing technology, and an emotion recognition engine.

[1647] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1648] Step 1:

[1649] Data collection

[1650] Subject: terminal

[1651] Specific operation: The user wears smart glasses and moves around the store. The camera captures visual information (e.g., images of products), and the microphone collects audio information (e.g., conversations between the user and store staff).

[1652] Input: Products viewed by the user, or audio uttered by the user.

[1653] Output: Digital image data and audio data.

[1654] Step 2:

[1655] Data transmission

[1656] Subject: terminal

[1657] Specific operation: The collected visual and audio information is sent to the server via the internet.

[1658] Input: Digital image and audio data.

[1659] Output: Data sent to the server.

[1660] Step 3:

[1661] Analysis of visual information

[1662] Subject: Server

[1663] Specific operation: The transmitted visual information is analyzed using OCR technology to extract text from the image. Natural language processing technology is used to identify important keywords in the extracted text and summarize the text.

[1664] Input: Image data sent to the server.

[1665] Output: Extracted text and its summary.

[1666] Step 4:

[1667] Analysis of audio information

[1668] Subject: Server

[1669] Specific operation: The transmitted audio information is converted into text using speech recognition technology. Then, the converted text is analyzed using natural language processing technology and translated into other languages ​​using a translation API as needed.

[1670] Input: Audio data sent to the server.

[1671] Output: Text converted from speech and its translation.

[1672] Step 5:

[1673] Emotion analysis

[1674] Subject: Server

[1675] Specific operation: Using an emotion recognition engine, the system analyzes the user's emotions from their voice and visual information. It identifies the user's emotional state, such as whether they are relaxed or stressed.

[1676] Input: Audio data and visual data.

[1677] Output: Analyzed emotional state data.

[1678] Step 6:

[1679] Feedback generation

[1680] Subject: Server

[1681] Specific operation: Generates feedback based on analysis and translation results, sentiment analysis results, and sends it to the device in real time.

[1682] Input: Summarized text, translation results, sentiment analysis results.

[1683] Output: Feedback data to be displayed in the user's field of view.

[1684] Step 7:

[1685] Information display

[1686] Subject: terminal

[1687] Specific operation: Decodes and renders the received feedback data and displays it in a layout appropriate to the user's field of view.

[1688] Input: Feedback data.

[1689] Output: Information displayed in the user's field of vision (e.g., product information, emotion-based guidance messages).

[1690] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1691] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1692] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[1693] [Fourth Embodiment]

[1694] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[1695] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1696] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1697] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[1698] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1699] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1700] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1701] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[1702] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1703] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1704] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1705] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1706] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1707] This invention relates to a system that uses a wearable device to collect visual and audio information, analyzes it, and provides real-time feedback to the user. This system is designed to significantly improve the efficiency of education and learning, and in particular, it extracts important keywords, generates summaries, and converts and translates audio information into text.

[1708] System Overview

[1709] This system consists of three main components: a wearable device (hereinafter referred to as "terminal"), a server that performs data analysis (hereinafter referred to as "server"), and users who utilize the service.

[1710] 1. Data collection and transmission

[1711] Device: The user wears a glasses-type wearable device. The device has a built-in camera and microphone. The camera captures visual information (e.g., pages of a book or document). The microphone collects audio information (e.g., a conversation in a foreign language).

[1712] Terminal: Encodes collected image and audio data into a digital format and transmits it to the server via the internet.

[1713] 2. Data Analysis

[1714] Server: Decodes the received visual information (image data) and extracts text from the image using OCR technology. Based on this text, natural language processing technology is applied to extract important keywords and generate a summary.

[1715] Server: Decodes the received audio information and converts the audio to text using speech recognition technology. This text is then sent to a translation API for real-time translation into the target language.

[1716] 3. Provide feedback

[1717] Server: Packages analysis results, such as summaries and translations, and sends them to the terminal.

[1718] Terminal: Decodes and renders received information in an appropriate layout to display it in the user's field of view.

[1719] User: Review the displayed information and continue learning or conversing.

[1720] Specific example

[1721] Example 1: Textbook summary display

[1722] User: Puts on glasses and begins reading textbook.

[1723] Device: Capture a page from the textbook using the camera and send the image to the server.

[1724] Server: Extracts text from images, uses natural language processing techniques to extract important keywords, and generates a summary.

[1725] Server: Sends the summary results to the terminal.

[1726] Terminal: Displays a summary within the user's field of view.

[1727] User: Review the summary and continue learning.

[1728] Example 2: Real-time translation of foreign language conversations

[1729] User: Puts on glasses and starts a conversation with a foreigner in English.

[1730] Terminal: The microphone collects the conversation and sends the audio data to the server.

[1731] Server: Converts speech to text and translates it into the target language in real time.

[1732] Server: Sends the translation result to the terminal.

[1733] Terminal: Displays the translation results within the user's field of view.

[1734] User: Check the displayed translation and continue the conversation.

[1735] This series of processes allows users to efficiently acquire information and progress in their learning.

[1736] The following describes the processing flow.

[1737] Step 1:

[1738] User: Put on the wearable device glasses and get ready.

[1739] Step 2:

[1740] Device: The device powers on, and the camera and microphone enter standby mode.

[1741] Step 3:

[1742] User: Puts on glasses and begins reading visual information (e.g., textbooks or documents).

[1743] Step 4:

[1744] Device: The camera detects the user's gaze and captures visual information.

[1745] Step 5:

[1746] Device: Simultaneously, the microphone collects ambient sound information.

[1747] Step 6:

[1748] Terminal: Encodes visual information (image data) into binary format and converts audio information into an appropriate format (e.g., PCM or FLAC format).

[1749] Step 7:

[1750] Terminal: Sends encoded and converted data to the server via the internet.

[1751] Step 8:

[1752] Server: Decodes visual information data and extracts text from images using OCR technology.

[1753] Step 9:

[1754] Server: Based on the extracted text data, it uses natural language processing techniques to extract important keywords and generate a summary.

[1755] Step 10:

[1756] Server: Decodes the audio information and converts the audio to text using speech recognition technology.

[1757] Step 11:

[1758] Server: Sends the converted text to the translation API for translation into the target language.

[1759] Step 12:

[1760] Server: Formats the analysis results, including the generated summary and translation, and sends them to the terminal.

[1761] Step 13:

[1762] Terminal: Decodes the received data and prepares it for display in the field of view with the appropriate layout.

[1763] Step 14:

[1764] User: Check the information displayed in their field of view on the device (summary, keywords, translation results, etc.) and continue learning or conversing.

[1765] Step 15:

[1766] Terminal: Continues to collect ongoing visual and auditory information and repeats the process from step 6.

[1767] Based on the steps described above, a system is created that supports efficient learning by analyzing visual and auditory information in real time and providing feedback to the user.

[1768] (Example 1)

[1769] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1770] Conventional wearable device systems have the problem of not being able to collect and analyze visual and auditory information in real time, thus failing to provide users with rapid feedback. Furthermore, the accuracy and speed of extracting important information and translation results are insufficient, highlighting the need for systems that support effective learning and communication.

[1771] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1772] In this invention, the server includes means for analyzing visual information and extracting textual information, means for extracting important keywords from the extracted textual information using natural language processing techniques and summarizing the textual information, and means for translating the textual information. This makes it possible to rapidly analyze visual and audio information and provide accurate feedback to the user in real time.

[1773] A "wearable device" is an electronic device that a user wears and uses, and that has the function of collecting visual and auditory information.

[1774] "Visual information" refers to image data and video data acquired through cameras and image acquisition devices.

[1775] "Audio information" refers to sound data collected through audio acquisition devices or microphones.

[1776] "Transmission" refers to transferring collected data to a server using communication methods such as the internet.

[1777] "Analysis" is the process of converting data into a digital format and processing it to make its contents easier to understand.

[1778] "Textual information" refers to text data extracted from visual information.

[1779] "Natural language processing technology" is a technology that enables computers to understand, analyze, and generate human language.

[1780] "Keywords" are words or phrases that are considered particularly important within a text.

[1781] A "summary" is a short, concise version of a longer text, focusing on the most important points.

[1782] Translation is the process of converting text written in one language into another language.

[1783] "Decoding" is the process of returning encoded data to its original format.

[1784] "Real-time" refers to temporal immediacy, meaning that processing and results are provided almost simultaneously.

[1785] "Rendering" is the process of visually displaying data.

[1786] A HUD (Head-Up Display) is a display technology that projects information directly into the user's field of vision.

[1787] This invention is a system that collects and analyzes visual and auditory information using a wearable device and provides real-time feedback to the user. This system enables users to efficiently acquire information and improve their learning and communication.

[1788] Hardware and software configuration

[1789] hardware

[1790] Wearable devices (devices): These are glasses-type devices with a built-in camera and microphone. The camera collects visual information, and the microphone collects audio information.

[1791] Data analysis server (server): A high-performance computer used to analyze received data.

[1792] software

[1793] OCR software: Software such as Tesseract OCR is used to extract text from visual information.

[1794] Natural Language Processing (NLP): This involves using tools such as NLTK (Natural Language Toolkit) to analyze text, extract important keywords, and generate summaries.

[1795] Speech recognition software: This includes tools such as the Google Speech-to-Text API, which converts speech information into text.

[1796] Translation APIs: APIs such as the Google Translate API are used to translate text into other languages.

[1797] The process of data collection and transmission

[1798] When a user puts on a glasses-type wearable device, the device's camera captures visual information and its microphone collects audio information. This data is encoded in a digital format and sent to a server via the internet.

[1799] Data analysis process

[1800] The server decodes the received visual information and extracts text from the image using OCR technology. Then, it extracts important keywords using natural language processing technology and summarizes the text. The server also decodes the received audio information and converts it to text using speech recognition technology. The converted text is then translated into the target language in real time using a translation API.

[1801] Provide feedback

[1802] The server packages the analysis and translation results and sends them back to the terminal via the internet. The terminal decodes and renders the received information in a layout appropriate for the user's view. The user can then review the displayed information and continue learning or conversing.

[1803] Specific example

[1804] Examples of textbook summaries

[1805] 1. The user puts on their glasses and begins reading the textbook.

[1806] 2. The device captures a page from the textbook using its camera and sends the image to the server.

[1807] 3. The server extracts text from the image, uses natural language processing techniques to extract important keywords, and generates a summary.

[1808] 4. The server sends the summary results to the terminal.

[1809] 5. The device displays a summary within the user's field of view.

[1810] 6. The user reviews the summary and continues learning.

[1811] Examples of real-time translation of foreign language conversations

[1812] 1. The user puts on glasses and starts a conversation with a foreigner in English.

[1813] 2. The device's microphone collects the conversation and sends the audio data to the server.

[1814] 3. The server converts the speech to text and translates it into the target language in real time.

[1815] 4. The server sends the translation result to the terminal.

[1816] 5. The device displays the translation result within the user's field of view.

[1817] 6. The user confirms the displayed translation and continues the conversation.

[1818] Example of a prompt

[1819] Example prompt for generating a textbook summary:

[1820] "Extract the text from the following image and summarize the key points."

[1821] Image: [Image of a textbook page]

[1822] Example prompt for real-time translation of foreign language conversations:

[1823] "Please convert the following audio data to text and translate it into the target language."

[1824] Audio data: [Audio files of foreign language conversations]

[1825] Thus, the system of the present invention enables real-time analysis and feedback of visual and auditory information, efficiently supporting user learning and communication.

[1826] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1827] Step 1:

[1828] The user wears a wearable device.

[1829] Specific action: Place the glasses-type device on your ears and adjust it to fit properly.

[1830] Input: None.

[1831] Output: User wearing the device.

[1832] Step 2:

[1833] The device captures visual information.

[1834] Specific operation: The built-in camera acquires visual information (e.g., textbook pages) in image format.

[1835] Input: Textbook pages or visual objects.

[1836] Output: Acquired image data.

[1837] Step 3:

[1838] The device collects voice information.

[1839] Specific operation: The built-in microphone records ambient sounds (e.g., conversations in a foreign language or lecture content).

[1840] Input: Ambient sound.

[1841] Output: Collected audio data.

[1842] Step 4:

[1843] The device encodes visual and audio information into a digital format.

[1844] Specific operation: Convert the acquired images and collected audio into a digital format and compress the data.

[1845] Input: Image data, audio data.

[1846] Output: Visual and audio information converted to digital format.

[1847] Step 5:

[1848] The device sends the encoded data to the server via the internet.

[1849] Specific operation: Send data using a secure communication protocol (HTTPS).

[1850] Input: Visual and audio information converted into digital format.

[1851] Output: Data sent to the server.

[1852] Step 6:

[1853] The server decodes the received visual information and extracts text using OCR technology.

[1854] Specific operation: Input image data into OCR software and convert it to text format.

[1855] Input: Visual information (image data) sent to the server.

[1856] Output: Extracted text data.

[1857] Step 7:

[1858] The server generates a summary of the extracted text using natural language processing techniques.

[1859] Specific operation: Apply an NLP algorithm to extract key keywords and generate a summary.

[1860] Input: Extracted text data.

[1861] Output: Summary text.

[1862] Step 8:

[1863] The server decodes the received audio information and uses speech recognition technology to convert the audio into text.

[1864] Specific operation: Input audio data into speech recognition software and convert it to text format.

[1865] Input: Audio information (audio data) sent to the server.

[1866] Output: Text data converted by speech recognition.

[1867] Step 9:

[1868] The server translates the converted text in real time using a translation API.

[1869] Specific operation: Input text data into the translation API and retrieve the translation result.

[1870] Input: Text data converted by speech recognition.

[1871] Output: Translated text data.

[1872] Step 10:

[1873] The server packages the analysis and translation results and sends them to the terminal.

[1874] Specific operation: Package the summary and translation results and send them using a secure communication protocol (HTTPS).

[1875] Input: Summary text, translated text data.

[1876] Output: Data sent to the terminal.

[1877] Step 11:

[1878] The device decodes and renders the received information in a layout appropriate to the user's field of view.

[1879] Specific operation: Information is overlaid onto the user's field of view using display technology.

[1880] Input: Summary text and translated text data sent to the server.

[1881] Output: Summary text displayed in the user's field of view, translation result.

[1882] Step 12:

[1883] The user reviews the displayed information and continues learning or conversing.

[1884] Specific actions: Users use summaries and translation results to understand materials and engage in conversations in a foreign language.

[1885] Input: Information displayed within the user's field of view.

[1886] Output: Continued learning or conversation.

[1887] (Application Example 1)

[1888] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1889] Conventional factory monitoring systems have struggled to monitor production lines and equipment in real time, making them ineffective for anomaly detection and improving production efficiency. Furthermore, even when visual and auditory information was collected, no system existed that could quickly analyze it and provide appropriate feedback. Therefore, there is a need for early detection of anomalies and real-time response in factory production lines.

[1890] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1891] In this invention, the server includes means for analyzing transmitted visual information and extracting text, means for extracting important keywords from the extracted text using natural language processing technology and summarizing the text, and means for monitoring production lines and equipment in real time using a mobile device that patrols the factory. This enables real-time monitoring of production lines and equipment, allowing for early detection of abnormalities and providing appropriate feedback to improve production efficiency.

[1892] A "wearable device" is a portable electronic device that a user can wear.

[1893] A "camera" is an optical device used to capture visual information.

[1894] "Visual information" refers to image and video data acquired using a camera.

[1895] A "microphone" is an acoustic device used to collect voice information.

[1896] "Audio information" refers to sound data collected through a microphone.

[1897] "Transmission means" refers to a mechanism for encoding captured visual information and collected audio information into a digital format and transmitting it to a remote location.

[1898] "Means for extracting text" refers to technologies for identifying and extracting textual information from visual information.

[1899] "Natural language processing technology" refers to the technology that enables computers to understand, interpret, and generate human language.

[1900] "Methods for extracting important keywords" refer to techniques for selecting particularly meaningful words and phrases from text.

[1901] "Methods for summarizing text" are techniques for making the content concise while preserving the overall meaning of the text.

[1902] "Methods for converting speech to text" refer to technologies that convert collected speech data into textual information.

[1903] "Translation methods" refer to the techniques used to convert text written in one language into another language.

[1904] "Means of providing feedback" refers to a mechanism for communicating the analyzed and translated results to the user.

[1905] "Mobile devices that patrol within a factory" refers to devices such as robots that move autonomously or remotely within a factory to collect data.

[1906] "Methods for monitoring production lines and equipment in real time" refer to technologies for continuously observing the production process and the state of equipment, and for immediately detecting abnormalities.

[1907] "Methods for generating reports" refer to the techniques used to analyze collected data, organize the results, and create a report.

[1908] This invention relates to a system aimed at efficient monitoring and anomaly detection of production lines and equipment within a factory. This system uses wearable devices to collect visual and auditory information, analyzes it, and provides real-time feedback to the user. Specific embodiments are described below.

[1909] System Overview

[1910] This system consists of three main components: a wearable device (hereinafter referred to as "terminal"), a server that performs data analysis (hereinafter referred to as "server"), and users who utilize the service. This enables anomaly detection and improved production efficiency in factory environments.

[1911] 1. Data collection and transmission

[1912] Terminal: A mobile device that patrols the factory, such as a factory robot, equipped with a camera and microphone, collects visual information (production line and equipment status) and audio information (worker instructions and ambient sounds). A standard industrial camera can be used as the camera, and a high-sensitivity microphone is used for audio collection.

[1913] Terminal: Encodes collected image and audio data into a digital format and transmits it to a server via the internet. For example, it uses Wi-Fi or a wired connection.

[1914] 2. Data Analysis

[1915] Server: Decodes visual information and extracts text from images using OCR technology. Based on this text, natural language processing techniques are applied to extract important keywords and generate a summary. Specific software examples include Python, OpenCV, and Tesseract.

[1916] Server: Decodes the received audio information and converts it to text using speech recognition technology. It then sends it to a translation API for real-time translation into the target language. Examples of use cases include the Python library SpeechRecognition and GoogleTrans.

[1917] 3. Provide feedback

[1918] Server: Packages analysis results, such as summaries and translations, and sends them to the terminal. During this process, it uses web frameworks such as Flask or Django to format the data.

[1919] Terminal: Decodes and renders received information in an appropriate layout for display on monitors within the factory and on workers' smartphones / tablets.

[1920] User: Review the displayed information to help identify anomalies and improve production efficiency. Audio and visual notifications are used as specific feedback methods.

[1921] Specific example

[1922] Example 1: Anomaly detection in factory robots

[1923] 1. Terminal: A robot moving around the factory takes pictures of the production line with its camera.

[1924] 2. Server: Uses OCR to read product labels and equipment status from the received video. Generates an alert if an anomaly is detected.

[1925] 3. User: Check alerts on your mobile device and take immediate action.

[1926] Example 2: Analysis of work instructions

[1927] 1. Terminal: Records the worker's conversation using a microphone and sends the audio data to the server.

[1928] 2. Server: Converts speech to text and extracts important keywords and instructions.

[1929] 3. User: Check the progress of work and instructions on a mobile device to proceed with work efficiently.

[1930] Example of a prompt

[1931] Please explain how factory robots detect anomalies on the production line in real time.

[1932] Please provide specific examples of how data collection and analysis can be performed using a robot's camera and microphone.

[1933] Please explain in detail how to improve work efficiency in a factory using visual and audio analysis technologies.

[1934] This system will enable highly efficient and effective real-time monitoring of production lines and equipment within the factory.

[1935] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1936] Step 1:

[1937] The terminal patrols the factory, capturing visual information with its camera and collecting audio information with its microphone. Inputs include video of the factory's production lines and equipment, as well as conversations among workers and ambient sounds. Outputs include digitally encoded video and audio data.

[1938] Step 2:

[1939] The terminal transmits the collected visual and audio information to the server via the internet. The input is the data encoded in step 1, and the output is the transmission of data to the server.

[1940] Step 3:

[1941] The server decodes the received visual information and extracts text from the image using OCR technology. Specifically, text information from product labels and display panels in the video is used as input, and the output is obtained by converting it into text data.

[1942] Step 4:

[1943] The server applies natural language processing techniques to the extracted text, extracting important keywords and summarizing the text. The input is the text data obtained in step 3, and the output generates important keywords and a summarized text.

[1944] Step 5:

[1945] The server decodes the received audio information and uses speech recognition technology to convert the audio into text. Specifically, the workers' conversations are used as input, and the output is obtained by converting them into text information.

[1946] Step 6:

[1947] The server sends the text converted from the speech to a translation API, which translates it into the target language in real time. The input is the text data obtained in step 5, and the output is the translated text.

[1948] Step 7:

[1949] The server packages the analyzed and translated results and sends them to the terminal. The input is the data obtained in steps 4 and 6, and the output is a feedback package that is generated and sent to the terminal.

[1950] Step 8:

[1951] The device decodes and renders the received feedback information in an appropriate layout and displays it visually to the user. Specifically, the data is formatted and displayed, and visual notifications and alerts are shown on the user's device.

[1952] Step 9:

[1953] The user checks the information displayed on the terminal and takes immediate action to respond to any abnormalities in the factory. The input is the information displayed in step 8, and the output is improvements to the user's actions or responses to the abnormalities.

[1954] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1955] This invention relates to a system that uses a wearable device to collect visual and auditory information, analyzes it, and provides real-time feedback to the user. Furthermore, this system incorporates an emotion recognition engine that can analyze the user's emotions and adjust the displayed information and learning content accordingly. This not only improves learning efficiency but also provides flexible support tailored to the user's emotional state.

[1956] System Overview

[1957] This system consists of three main components: a wearable device (hereinafter referred to as "terminal"), a server that performs data analysis (hereinafter referred to as "server"), and users who utilize the service.

[1958] 1. Data collection and transmission

[1959] Device: The user wears a glasses-type wearable device. The device has a built-in camera and microphone. The camera captures visual information (e.g., pages of a book or document). The microphone collects audio information (e.g., a conversation in a foreign language).

[1960] Terminal: Encodes collected image and audio data into a digital format and transmits it to the server via the internet.

[1961] 2. Data Analysis

[1962] Server: Decodes the received visual information (image data) and extracts text from the image using OCR technology. Based on this text, natural language processing technology is applied to extract important keywords and generate a summary.

[1963] Server: Decodes the received audio information and converts the audio to text using speech recognition technology. This text is then sent to a translation API for real-time translation into the target language.

[1964] Server: Uses an emotion recognition engine to analyze the user's emotions and incorporates the analysis results into the feedback.

[1965] 3. Provide feedback

[1966] Server: Packages analysis results such as summaries, translations, and sentiment analysis results, and sends them to the terminal.

[1967] Terminal: Decodes and renders received information in an appropriate layout to display it in the user's field of view.

[1968] User: Review the displayed information and continue learning or conversing.

[1969] Specific example

[1970] Example 1: Textbook summary display and emotional feedback

[1971] User: Puts on glasses and begins reading textbook.

[1972] Device: Captures textbook pages with the camera and sends the images to the server. Collects facial expressions and voice tone with the microphone.

[1973] Server: Extracts text from images, uses natural language processing techniques to extract important keywords, and generates summaries. It also analyzes user emotions using an emotion recognition engine.

[1974] Server: Sends summary results and sentiment analysis results to the terminal.

[1975] Device: Displays summaries and sentiment feedback within the user's field of view.

[1976] User: Review the summary and consider the next learning steps based on the sentiment feedback.

[1977] Example 2: Real-time translation and sentiment recognition of foreign language conversations

[1978] User: Puts on glasses and starts a conversation with a foreigner in English.

[1979] Terminal: The microphone collects conversation data and sends it to the server along with the user's tone of voice and facial expressions.

[1980] Server: Converts speech to text and translates it into the target language in real time. It also analyzes the user's emotions using an emotion recognition engine and includes emotion data in the conversation feedback.

[1981] Server: Sends translation results and sentiment analysis results to the terminal.

[1982] Device: Displays translation results and sentiment feedback within the field of view.

[1983] User: Review the displayed translation and sentiment feedback, then continue the conversation.

[1984] In this way, this system analyzes the user's visual and auditory information in real time, and in addition, recognizes emotions, thereby providing feedback tailored to the user's emotional state, and achieving more efficient and personalized learning support.

[1985] The following describes the processing flow.

[1986] Step 1:

[1987] User: Put on the wearable device glasses and get ready.

[1988] Step 2:

[1989] Device: The device powers on, and the camera and microphone enter standby mode.

[1990] Step 3:

[1991] User: Puts on glasses and begins reading visual information (e.g., textbooks or documents).

[1992] Step 4:

[1993] Device: The camera captures visual information and generates image data.

[1994] Step 5:

[1995] Device: Simultaneously, the microphone collects ambient audio information and saves it as audio data.

[1996] Step 6:

[1997] Terminal: Encodes visual information (image data) into binary format and converts audio information into an appropriate format (e.g., PCM or FLAC format).

[1998] Step 7:

[1999] Terminal: Sends encoded and converted data to the server via the internet.

[2000] Step 8:

[2001] Server: Decodes visual information data and extracts text from images using OCR technology.

[2002] Step 9:

[2003] Server: Based on the extracted text data, it uses natural language processing techniques to extract important keywords and generate a summary.

[2004] Step 10:

[2005] Server: Decodes the audio information and converts the audio to text using speech recognition technology.

[2006] Step 11:

[2007] Server: Sends the converted text to the translation API, which translates it into the target language in real time.

[2008] Step 12:

[2009] Server: Analyzes the user's emotional state using an emotion recognition engine. This includes the process of identifying emotions from voice tone and facial expressions.

[2010] Step 13:

[2011] Server: Packages the summary, translation results, and sentiment analysis results and sends them to the terminal.

[2012] Step 14:

[2013] Terminal: Decodes the received data and renders it in an appropriate layout for display in the user's field of view.

[2014] Step 15:

[2015] User: Check the information displayed in their field of view on the device (summary, keywords, translation results, sentiment feedback, etc.) and continue learning or conversing.

[2016] Step 16:

[2017] Terminal: Continues to collect ongoing visual and auditory information and repeats the process from step 6.

[2018] Specific example

[2019] Example 1: Textbook summary display and emotional feedback

[2020] Step 1:

[2021] User: Puts on glasses and begins reading textbook.

[2022] Step 2:

[2023] Device: Captures textbook pages with the camera and sends the images to the server. Collects facial expressions and voice tone with the microphone.

[2024] Step 3:

[2025] Server: Extracts text from images, uses natural language processing techniques to extract important keywords, and generates summaries. It also analyzes user emotions using an emotion recognition engine.

[2026] Step 4:

[2027] Server: Sends summary results and sentiment analysis results to the terminal.

[2028] Step 5:

[2029] Device: Displays summaries and sentiment feedback within the user's field of view.

[2030] Step 6:

[2031] User: Review the summary and consider the next learning steps based on the sentiment feedback.

[2032] Example 2: Real-time translation and sentiment recognition of foreign language conversations

[2033] Step 1:

[2034] User: Puts on glasses and starts a conversation with a foreigner in English.

[2035] Step 2:

[2036] Terminal: The microphone collects conversation data and sends it to the server along with the user's tone of voice and facial expressions.

[2037] Step 3:

[2038] Server: Converts speech to text and translates it into the target language in real time. It also analyzes the user's emotions using an emotion recognition engine and includes emotion data in the conversation feedback.

[2039] Step 4:

[2040] Server: Sends translation results and sentiment analysis results to the terminal.

[2041] Step 5:

[2042] Device: Displays translation results and sentiment feedback within the field of view.

[2043] Step 6:

[2044] User: Review the displayed translation and sentiment feedback, then continue the conversation.

[2045] Based on the steps described above, this system analyzes the user's visual and auditory information in real time, and in addition, recognizes emotions, thereby providing more personalized learning support.

[2046] (Example 2)

[2047] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[2048] Conventional learning support systems and conversation support systems function independently, making it difficult to integrate and analyze visual and auditory information to provide real-time feedback. Furthermore, because they only provide simple information without considering the user's emotional state, the user's learning efficiency and communication experience were not optimized. The present invention aims to solve these problems and provide a system that analyzes visual and auditory information in real time, recognizes the user's emotions, and provides appropriate feedback.

[2049] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[2050] In this invention, the server includes means for decoding transmitted visual information and extracting text using optical character recognition technology, means for extracting important keywords from the extracted text using natural language processing technology and summarizing the text, and means for decoding transmitted audio information and converting the audio to text using speech recognition technology. This enables integrated analysis of visual and audio information and real-time feedback that responds to the user's emotional state.

[2051] A "wearable device" is a portable electronic device that a user wears to collect visual and auditory information.

[2052] A "camera" is a device used to capture visual information.

[2053] A "microphone" is an acoustic conversion device used to collect sound information.

[2054] "Encoding into a digital format" is the process of converting an analog signal into a digital signal.

[2055] "Transmission" refers to sending collected data to another device or server.

[2056] "Decoding" is the process of converting transmitted digital data back into its original format.

[2057] "Optical character recognition technology" is a technology that automatically recognizes characters from images and converts them into text.

[2058] "Natural language processing technology" is a technology that processes and analyzes text written in natural language.

[2059] "Speech recognition technology" is a technology that analyzes speech data and converts it into text.

[2060] "Translation" refers to the process of converting text written in one language or spoken content into another language.

[2061] An "emotion recognition engine" is a general term for algorithms and technologies used to analyze and identify a user's emotions.

[2062] "Feedback" refers to providing users with information and data, such as analysis results or translation results.

[2063] This invention relates to a system that allows users to collect visual and auditory information using a wearable device, analyze it, and provide real-time feedback to the user. Furthermore, this system incorporates an emotion recognition engine that can analyze the user's emotions and adjust the displayed information and learning content accordingly. This improves learning efficiency and provides flexible support tailored to the user's emotional state. Specific embodiments of this system are described below.

[2064] Hardware configuration

[2065] Device: A wearable device worn by the user (e.g., smart glasses). This device has a built-in camera and microphone to collect visual and audio information.

[2066] Server: A remote server that performs data analysis. This server possesses advanced computing capabilities and processes and analyzes the received data.

[2067] Software Configuration

[2068] Camera: A software module for capturing visual information. For example, it can be used to photograph text information on books or signs.

[2069] Microphone: A software module for collecting audio information. For example, it can be used to record conversations or ambient sounds.

[2070] Optical Character Recognition (OCR): A technology for extracting characters from images. For example, using the "Google Cloud Vision API".

[2071] Natural Language Processing (NLP): A technique that analyzes extracted text and identifies and summarizes important keywords. For example, it uses libraries such as the "NLTK library."

[2072] Speech-to-Text (ASR) technology: A technology that converts speech into text. For example, using the "Google Speech-to-Text API".

[2073] Translation API: A technology for translating text into other languages. For example, using the "Google Translate API".

[2074] Emotion recognition engine: An algorithm for analyzing a user's emotions. For example, it uses the "Microsoft Azure Emotion API".

[2075] Specific example

[2076] Example 1: Textbook summary display and emotional feedback

[2077] User: Puts on glasses and begins reading textbook.

[2078] Device: Captures textbook pages with the camera and sends the images to the server. Collects facial expressions and voice tone with the microphone.

[2079] Server: Extracts text from images, uses natural language processing techniques to extract important keywords, and generates summaries. It also analyzes user emotions using an emotion recognition engine.

[2080] Server: Sends summary results and sentiment analysis results to the terminal.

[2081] Device: Displays summaries and sentiment feedback within the user's field of view.

[2082] User: Review the summary and consider the next learning steps based on the sentiment feedback.

[2083] Example 2: Real-time translation and sentiment recognition of foreign language conversations

[2084] User: Puts on glasses and starts a conversation with a foreigner in English.

[2085] Terminal: The microphone collects conversation data and sends it to the server along with the user's tone of voice and facial expressions.

[2086] Server: Converts speech to text and translates it into the target language in real time. It also analyzes the user's emotions using an emotion recognition engine and includes emotion data in the conversation feedback.

[2087] Server: Sends translation results and sentiment analysis results to the terminal.

[2088] Device: Displays translation results and sentiment feedback within the field of view.

[2089] User: Review the displayed translation and sentiment feedback, then continue the conversation.

[2090] Example of a prompt

[2091] For textbooks: "When a user reads the textbook, key keywords should be summarized and displayed in real time."

[2092] For conversations: "When a user speaks a foreign language, perform real-time emotion recognition and provide feedback along with the translation results."

[2093] In this way, the present invention can analyze visual and auditory information in real time and further recognize emotions to provide personalized feedback to the user.

[2094] The flow of the specific processing in Example 2 will be explained using Figure 13.

[2095] Step 1:

[2096] Data collection

[2097] Terminal: The user wears a wearable device and captures visual information using a camera. Simultaneously, audio information is collected using a microphone.

[2098] Input: The visual object the user is looking at (e.g., a page in a textbook) and the sound they are hearing (e.g., surrounding conversation).

[2099] Output: Captured visual information (image data) and collected audio information (audio data).

[2100] Specific actions: Take a picture of a textbook page with the camera and encode the image in JPEG format. Record a conversation with the microphone and encode the audio in WAV format.

[2101] Step 2:

[2102] Data transmission

[2103] Terminal: Encodes collected visual and audio information into a digital format and transmits it to the server via the internet.

[2104] Input: Encoded image and audio data.

[2105] Output: Data sent to the server.

[2106] Specific operation: Compresses JPEG image data and WAV audio data, and sends them to a server via the internet.

[2107] Step 3:

[2108] Analysis of visual information

[2109] Server: Decodes the received image data and extracts text from the image using OCR technology.

[2110] Input: The encoded image data that was sent.

[2111] Output: Extracted text data.

[2112] Specific operation: Decode JPEG image data and extract text using OCR technology (e.g., Google Cloud Vision API). For example, extract the text "differential and integral calculus" from a page in a math textbook.

[2113] Step 4:

[2114] Natural Language Processing

[2115] Server: Extracts important keywords from the extracted text and applies natural language processing techniques to summarize the text.

[2116] Input: Text data extracted using OCR technology.

[2117] Output: Summary and keyword extraction results.

[2118] Specific operation: Natural language processing techniques (e.g., NLTK library) are used on the extracted text data to extract important keywords and generate a summary. For example, it might summarize as "Explanation of the basic principles of differentiation."

[2119] Step 5:

[2120] Voice analysis

[2121] Server: Decodes the received audio data and converts the audio to text using speech recognition technology.

[2122] Input: The encoded audio data that has been sent.

[2123] Output: Text data converted from speech.

[2124] Specific operation: Decodes WAV audio data and converts it to text using speech recognition technology (e.g., Google Speech-to-Text API). For example, converts "What is your name?" to text.

[2125] Step 6:

[2126] translation

[2127] Server: Sends the text converted from speech to a translation API, which translates it into the target language in real time.

[2128] Input: Data converted to text using speech recognition technology.

[2129] Output: Translation result.

[2130] Specific operation: Text data is sent to a translation API (e.g., Google Translate API) and translated in real time. For example, "What is your name?" is translated to "What is your name?".

[2131] Step 7:

[2132] Emotion analysis

[2133] Server: Analyzes user emotions using an emotion recognition engine and compiles the results.

[2134] Input: User's facial expression data and voice data.

[2135] Output: Emotion analysis results.

[2136] Specific operation: The collected facial expression data and voice tone are analyzed using an emotion recognition engine (e.g., Microsoft Azure Emotion API) to identify emotions such as "joy" or "sadness." For example, the analysis result might be "the user is satisfied."

[2137] Step 8:

[2138] Feedback generation and transmission

[2139] Server: Packages the summary results, translation results, and sentiment analysis results and sends them to the terminal.

[2140] Input: Summarization results, translation results, and sentiment analysis results.

[2141] Output: Feedback data sent to the terminal.

[2142] Specific operation: Integrate the results of summarization, translation, and sentiment analysis, format them appropriately, and send them to the terminal. For example, send the textbook summary, conversation translation results, and user sentiment analysis results as a single data package to the terminal in JPEG and text formats.

[2143] Step 9:

[2144] Displaying feedback

[2145] Terminal: Decodes the received feedback data and displays it appropriately within the user's field of view.

[2146] Input: Feedback data sent from the server.

[2147] Output: Visually displayed feedback information.

[2148] Specific operation: Adjust the layout and font size to display the decoded feedback data on the smart glasses' display.

[2149] Step 10:

[2150] Feedback review and action

[2151] User: Review the displayed feedback information and decide on the next action.

[2152] Input: Feedback information displayed on the screen.

[2153] Output: The user's next action (e.g., continue learning, continue conversation).

[2154] Specific action: Decide whether to continue learning by reading the textbook summary or to continue the conversation in the foreign language based on the translation result.

[2155] (Application Example 2)

[2156] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[2157] Conventional wearable device-based information collection and analysis systems can collect and analyze visual and auditory information, but they cannot provide feedback that takes into account the user's emotional state. As a result, the user experience is uniform, and individualized support is difficult. Furthermore, improving customer service in physical stores has been challenging, as it has been difficult to provide appropriate support tailored to the individual needs and emotions of each customer.

[2158] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[2159] In this invention, the server includes means for analyzing visual information and extracting text, means for analyzing audio information and converting audio to text, and means for analyzing customer emotions and providing real-time emotional feedback. This makes it possible to provide feedback tailored to the customer's emotional state, in addition to text summarization and translation results, based on the collected visual and audio information.

[2160] A "wearable device" is an electronic device that a user can wear and that is equipped with a wide variety of sensors and communication functions.

[2161] "Visual information" refers to images and video data captured using a camera.

[2162] "Audio information" refers to audio data collected using a microphone.

[2163] "Means of transmission" refers to a system that includes communication means for transferring collected data to a server or other device.

[2164] "Means of analysis" include technologies that analyze collected data and perform text extraction and speech recognition.

[2165] "Natural language processing technology" refers to all technologies that use computers to understand and generate human language.

[2166] "Key keywords" refer to words or phrases that are deemed particularly important within text data analyzed using natural language processing technology.

[2167] "Methods of summarization" refer to techniques for extracting key points from analyzed text data and compiling them into a concise format.

[2168] "Translation methods" refer to technologies that convert analyzed text data into other languages.

[2169] "Emotional feedback" refers to the process of analyzing a user's emotional state and providing specific responses or messages based on the results.

[2170] A "server" refers to a remote device that processes collected data and performs tasks such as analysis, text extraction, translation, and sentiment analysis.

[2171] "Decoding" refers to the process of converting digital data into analog or other formats.

[2172] "Rendering" refers to the process of processing digital data to display it visually.

[2173] A "user" refers to a person who uses a wearable device to receive feedback from the system.

[2174] This invention is a system that uses a wearable device to collect visual and auditory information, analyzes it in real time, and provides feedback to the user. Furthermore, by using emotion recognition technology to provide feedback tailored to the user's emotional state, it enables more effective learning and service support.

[2175] System Overview

[2176] This system consists of the following main components:

[2177] Wearable devices (devices): Glasses-type or head-mounted display-type devices equipped with a camera and microphone.

[2178] Server: A remote server that performs data analysis and generates feedback.

[2179] User: An individual who uses a wearable device.

[2180] 1. Data collection and transmission

[2181] Terminal: The user wears a glasses-type wearable device that captures visual information with its built-in camera and collects audio information with its microphone. This allows for the acquisition of visual and audio data in environments such as physical stores and classrooms.

[2182] Terminal: Collected visual information (e.g., images of product packaging) and audio information (e.g., conversations between customers and store clerks) are encoded in digital format and transmitted to a server via the internet.

[2183] 2. Data Analysis

[2184] Server: Converts received visual information into text using OCR technology, extracts important keywords using natural language processing technology, and summarizes the text. For example, it can generate detailed product information or sales information.

[2185] Server: Converts received audio information into text using speech recognition technology and translates it into other languages ​​using a translation API as needed. It also processes the analysis results using a generative AI model.

[2186] Server: Uses an emotion recognition engine to analyze the user's emotions from their voice and visual information and incorporates this into the feedback. For example, it can determine whether the customer is relaxed or stressed.

[2187] 3. Provide feedback

[2188] Server: Compiles the analysis results (summary, translation results, sentiment analysis results, etc.) and sends them to the terminal in an appropriate layout.

[2189] Terminal: Decodes and renders information to display analysis results within the user's field of view. For example, product summary information or emotion-based guidance messages may be displayed within the user's field of view.

[2190] User: Based on the feedback provided, decide on the next course of action. For example, look at other products, ask additional questions, etc.

[2191] Specific example

[2192] Example 1: Providing product information and emotional feedback

[2193] User: Walks around the store wearing smart glasses.

[2194] Device: Captures product packaging with a camera and collects conversation content with a microphone.

[2195] Server: Analyzes received visual information using OCR and summarizes product information. Analyzes customer emotions from audio information and provides information smoothly if the customer is relaxed.

[2196] Terminal: Displays product information and emotion-based messages within the user's field of view.

[2197] User: Choose a product based on the information provided, or ask the store clerk additional questions.

[2198] Example of a prompt

[2199] Image data: "Captured product image"

[2200] Audio data: "Audio of customer conversation"

[2201] Emotional data: "Analyzed emotional state (e.g., relaxed, tense)"

[2202] This system provides more personalized and effective service support by offering feedback based on the customer's emotional state. The hardware used includes smart glasses (e.g., general wearable devices) and servers (e.g., cloud servers), while the software utilizes OCR technology, speech recognition technology, natural language processing technology, and an emotion recognition engine.

[2203] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[2204] Step 1:

[2205] Data collection

[2206] Subject: terminal

[2207] Specific operation: The user wears smart glasses and moves around the store. The camera captures visual information (e.g., images of products), and the microphone collects audio information (e.g., conversations between the user and store staff).

[2208] Input: Products viewed by the user, or audio uttered by the user.

[2209] Output: Digital image data and audio data.

[2210] Step 2:

[2211] Data transmission

[2212] Subject: terminal

[2213] Specific operation: The collected visual and audio information is sent to the server via the internet.

[2214] Input: Digital image and audio data.

[2215] Output: Data sent to the server.

[2216] Step 3:

[2217] Analysis of visual information

[2218] Subject: Server

[2219] Specific operation: The transmitted visual information is analyzed using OCR technology to extract text from the image. Natural language processing technology is used to identify important keywords in the extracted text and summarize the text.

[2220] Input: Image data sent to the server.

[2221] Output: Extracted text and its summary.

[2222] Step 4:

[2223] Analysis of audio information

[2224] Subject: Server

[2225] Specific operation: The transmitted audio information is converted into text using speech recognition technology. Then, the converted text is analyzed using natural language processing technology and translated into other languages ​​using a translation API as needed.

[2226] Input: Audio data sent to the server.

[2227] Output: Text converted from speech and its translation.

[2228] Step 5:

[2229] Emotion analysis

[2230] Subject: Server

[2231] Specific operation: Using an emotion recognition engine, the system analyzes the user's emotions from their voice and visual information. It identifies the user's emotional state, such as whether they are relaxed or stressed.

[2232] Input: Audio data and visual data.

[2233] Output: Analyzed emotional state data.

[2234] Step 6:

[2235] Feedback generation

[2236] Subject: Server

[2237] Specific operation: Generates feedback based on analysis and translation results, sentiment analysis results, and sends it to the device in real time.

[2238] Input: Summarized text, translation results, sentiment analysis results.

[2239] Output: Feedback data to be displayed in the user's field of view.

[2240] Step 7:

[2241] Information display

[2242] Subject: terminal

[2243] Specific operation: Decodes and renders the received feedback data and displays it in a layout appropriate to the user's field of view.

[2244] Input: Feedback data.

[2245] Output: Information displayed in the user's field of vision (e.g., product information, emotion-based guidance messages).

[2246] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[2247] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2248] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[2249] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2250] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[2251] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[2252] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[2253] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[2254] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[2255] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[2256] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[2257] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[2258] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[2259] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2260] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[2261] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[2262] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[2263] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[2264] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[2265] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[2266] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[2267] The following is further disclosed regarding the embodiments described above.

[2268] (Claim 1)

[2269] A means of capturing visual information using the camera of a wearable device,

[2270] A means of collecting audio information using a microphone,

[2271] A means for transmitting captured visual information and collected audio information,

[2272] A means for analyzing transmitted visual information and extracting text,

[2273] A method for extracting important keywords from extracted text using natural language processing techniques and summarizing the text,

[2274] A means for analyzing transmitted audio information and converting the audio into text,

[2275] A means of translating text converted from speech,

[2276] Means for transmitting and displaying the analyzed and translated results on a wearable device,

[2277] A system that includes this.

[2278] (Claim 2)

[2279] The system according to claim 1, further comprising means for identifying a region from captured visual information and analyzing specific text within that region.

[2280] (Claim 3)

[2281] The system according to claim 1, further comprising means for applying speech recognition technology based on the collection of voice information and displaying the analysis results in real time.

[2282] "Example 1"

[2283] (Claim 1)

[2284] A means for acquiring visual information using an image acquisition device of a wearable device,

[2285] A means for collecting voice information using a voice collection device,

[2286] A means for transmitting acquired visual information and collected audio information,

[2287] A means for analyzing transmitted visual information and extracting textual information,

[2288] A means of extracting important keywords from extracted text information using natural language processing technology and summarizing the text information,

[2289] A means for analyzing transmitted audio information and converting it into text information,

[2290] A means of translating the converted text information,

[2291] Means for transmitting and displaying the analyzed and translated results on a wearable device,

[2292] A means of applying speech recognition technology to analyze collected speech information in real time,

[2293] A system that includes this.

[2294] (Claim 2)

[2295] The system according to claim 1, further comprising means for identifying an area from acquired visual information and analyzing specific character information within that area.

[2296] (Claim 3)

[2297] The system according to claim 1, further comprising means for rendering the analysis and translation results in an appropriate layout within the field of view.

[2298] "Application Example 1"

[2299] (Claim 1)

[2300] A means of capturing visual information using the camera of a wearable device,

[2301] A means of collecting audio information using a microphone,

[2302] A means for transmitting captured visual information and collected audio information,

[2303] A means for analyzing transmitted visual information and extracting text,

[2304] A method for extracting important keywords from extracted text using natural language processing techniques and summarizing the text,

[2305] A means for analyzing transmitted audio information and converting the audio into text,

[2306] A means of translating text converted from speech,

[2307] Means for transmitting and displaying the analyzed and translated results on a wearable device,

[2308] A means of monitoring production lines and equipment in real time using mobile devices that patrol the factory,

[2309] A method for analyzing factory conditions in real time based on the collection of visual and audio information by a robot equipped with a camera and microphone,

[2310] A means of generating reports on production efficiency and anomaly detection based on the analyzed information, and providing feedback,

[2311] A system that includes this.

[2312] (Claim 2)

[2313] The system according to claim 1, further comprising means for identifying a region from captured visual information and analyzing specific text within that region.

[2314] (Claim 3)

[2315] The system according to claim 1, further comprising means for applying speech recognition technology based on the collection of voice information and displaying the analysis results in real time.

[2316] "Example 2 of combining an emotion engine"

[2317] (Claim 1)

[2318] A means of capturing visual information using the camera of a wearable device,

[2319] A means of collecting audio information using a microphone,

[2320] A means for encoding and transmitting collected visual and audio information in digital format,

[2321] A means for decoding transmitted visual information and extracting text using optical character recognition technology,

[2322] A method for extracting important keywords from extracted text using natural language processing techniques and summarizing the text,

[2323] A means for decoding transmitted audio information and converting the audio into text using speech recognition technology,

[2324] A means of translating audio converted to text and translating it into other languages ​​in real time,

[2325] A method that uses an emotion recognition engine to analyze user emotions and incorporate emotional data into feedback,

[2326] Means for transmitting and displaying the analyzed and translated results, as well as the sentiment analysis results, to a wearable device,

[2327] A system that includes this.

[2328] (Claim 2)

[2329] The system according to claim 1, further comprising means for identifying a region from captured visual information and analyzing specific text within that region.

[2330] (Claim 3)

[2331] The system according to claim 1, further comprising means for applying speech recognition technology based on the collection of voice information and displaying the analysis results in real time.

[2332] "Application example 2 when combining with an emotional engine"

[2333] (Claim 1)

[2334] A means of capturing visual information using the camera of a wearable device,

[2335] A means of collecting audio information using a microphone,

[2336] A means for transmitting captured visual information and collected audio information,

[2337] A means for analyzing transmitted visual information and extracting text,

[2338] A method for extracting important keywords from extracted text using natural language processing techniques and summarizing the text,

[2339] A means for analyzing transmitted audio information and converting the audio into text,

[2340] A means of translating text converted from speech,

[2341] A means of analyzing customer emotions and providing real-time emotional feedback,

[2342] Means for transmitting and displaying the analyzed and translated results on a wearable device,

[2343] A system that includes this.

[2344] (Claim 2)

[2345] The system according to claim 1, further comprising means for identifying a region from captured visual information and analyzing specific text within that region.

[2346] (Claim 3)

[2347] The system according to claim 1, further comprising means for applying speech recognition technology based on the collection of voice information and displaying the analysis results in real time. [Explanation of Symbols]

[2348] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of capturing visual information using the camera of a wearable device, A means of collecting audio information using a microphone, A means for transmitting captured visual information and collected audio information, A means for analyzing transmitted visual information and extracting text, A method for extracting important keywords from extracted text using natural language processing techniques and summarizing the text, A means for analyzing transmitted audio information and converting the audio into text, A means of translating text converted from speech, Means for transmitting and displaying the analyzed and translated results on a wearable device, A system that includes this.

2. The system according to claim 1, further comprising means for identifying a region from captured visual information and analyzing specific text within that region.

3. The system according to claim 1, further comprising means for applying speech recognition technology based on the collection of voice information and displaying the analysis results in real time.

Images