system

Abstract

We provide the system. [Solution] An image capture means that captures an image of an object that the user is focusing on, A voice acquisition means for acquiring user voice commands, A visual information analysis means that analyzes captured image data and extracts necessary metadata, A voice analysis method that analyzes acquired voice data to understand the content of the user's questions, Information generation means that generates answers to user questions based on data obtained from the visual information analysis means and the voice analysis means, A display means for visually displaying the information generated by the information generation means, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] There is a lot of information in the street, and it is particularly important for urban residents to obtain detailed information from advertisements and billboards that they are concerned about. However, it is necessary to take multiple steps using a smartphone. These steps require a certain search ability and active actions, which pose a high hurdle especially for busy working people and those who lack confidence in their search ability. In addition, image search requires an action of taking a photo and is vulnerable to complex searches. Thus, a simple and efficient information collection means is required to smoothly access necessary information from visual information.

Means for Solving the Problems

[0005] This invention is a system that uses smart glasses and earphones to automatically perform information retrieval by combining visual information and voice instructions, and provides the results to the user. The system includes image capture means, voice acquisition means, visual information analysis means, voice analysis means, information generation means, and display means. The server receives image data and voice data via the image capture means, which captures an image of the object the user is looking at, and the voice acquisition means, which acquires voice instructions. The received image data is analyzed by the visual information analysis means to extract metadata of the advertisement or sign that the user is looking at. The voice instructions are also analyzed by the voice analysis means to understand the content of the user's question. Based on these analysis results, the information generation means searches for information and generates an answer to the user's question. The generated information is displayed on the smart glasses via the display means and provided to the user. This makes it possible to collect information simply and efficiently without requiring search skills or proactivity.

[0006] "Image capture means" refers to an element that includes a device and functions for acquiring an image of an object that the user is focusing on.

[0007] "Voice acquisition means" refers to an element that includes a device and functions for acquiring a user's voice commands and processing them as digital data.

[0008] "Visual information analysis means" refers to an element that includes a device and functions for analyzing acquired image data and extracting specific information or metadata from it.

[0009] "Voice analysis means" refers to an element that includes a device and functions for analyzing acquired voice data and understanding the content and intent of the user's questions.

[0010] "Information generation means" refers to an element that includes a device and functions for generating answers to user questions using data obtained from visual information analysis means and audio analysis means.

[0011] "Display means" refers to an element that includes a device and functions for visually displaying generated information to the user. [Brief explanation of the drawing]

[0012] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14]It is a sequence diagram showing the processing flow of a data processing system in Application Example 2 when a sentiment engine is combined.

Embodiments for Carrying Out the Invention

[0013] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0014] First, the terms used in the following description will be explained.

[0015] In the following embodiments, a numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0016] In the following embodiments, a numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0017] In the following embodiments, a numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0018] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0019] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0020] [First Embodiment]

[0021] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0022] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0023] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0024] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0025] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0026] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0027] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0028] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0029] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0030] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0031] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0032] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0033] This invention is a system that uses smart glasses and earphones to combine visual information and voice instructions, automatically retrieves information, and provides the results to the user.

[0034] System Configuration

[0035] 1. Devices (smart glasses & earphones):

[0036] Smart glasses incorporate image capture and display mechanisms.

[0037] The earphones have a built-in mechanism for acquiring sound.

[0038] 2. Server:

[0039] The server is equipped with visual information analysis means, audio analysis means, and information generation means, which work together in coordination.

[0040] Program processing

[0041] When the device is activated, the smart glasses' camera captures an image of the object the user is looking at. This image is sent to the server by the device. Simultaneously, the earphones receive the user's voice commands, and this voice data is also sent to the server.

[0042] The server first analyzes the transmitted image data using visual information analysis tools. Specifically, it uses image recognition algorithms to extract text and specific objects from the image and generate metadata.

[0043] Next, the audio data is analyzed by a speech analysis tool. Using a speech recognition algorithm, the audio data is converted into text, and the user's question and intent are understood based on this text data. This combines the aforementioned metadata and the question content to generate a search query.

[0044] The information generation system uses this search query to search for information in databases and on the web. The information obtained as a search result is formatted into an appropriate format and then sent from the server to the terminal.

[0045] The device (smart glasses) provides the user with received information visually through a display mechanism. The user can confirm the information displayed on the smart glasses' screen.

[0046] Specific example

[0047] Let's consider an example where a user is looking at a poster for a new movie in the street and asks a question into their earphones: "Tell me the showtimes for this movie." The smart glasses' camera captures an image of the poster and sends it to the server. The earphones receive the voice command and also send it to the server.

[0048] On the server, the poster image is analyzed by a visual information analysis device, and the movie title and other metadata are extracted. Simultaneously, the voice instructions are analyzed by a voice analysis device, and the user's question is understood. Based on this data, an information generation device searches for the movie's screening schedule and generates the results.

[0049] Ultimately, information is transmitted from the server to the smart glasses, allowing the user to view the showtimes displayed on the smart glasses' screen. In this way, a system is provided that allows the user to efficiently obtain the necessary information.

[0050] The following describes the processing flow.

[0051] Step 1:

[0052] The server verifies the connection between the smart glasses and earphones and performs the necessary initial setup. This setup involves verifying the UUIDs (Universal Unique Identifiers) of the smart glasses and earphones, and checking the battery status and network connectivity of each device. After confirming everything is working correctly, a notification is sent to the device indicating that the initial setup is complete.

[0053] Step 2:

[0054] The device (smart glasses & earphones) starts up and begins communicating with the server. Upon receiving notification from the server that the initial setup is complete, an alert is displayed on the smart glasses' screen to inform the user that it is ready for use.

[0055] Step 3:

[0056] The user focuses on a movie poster they see in the street. This fixed gaze triggers the smart glasses' camera.

[0057] Step 4:

[0058] The device (smart glasses) activates its camera and captures an image of the poster the user is looking at. The camera automatically focuses and takes a clear image. The image data is stored in temporary memory.

[0059] Step 5:

[0060] The device (smart glasses) compresses and encodes the image data it captures and prepares it for transmission to the server. Once ready, the image data is sent to the server.

[0061] Step 6:

[0062] The user asks into their earphone, "Tell me the showtimes for this movie." The voice command is then acquired.

[0063] Step 7:

[0064] The device (earphones) records the user's voice data and saves it to temporary memory. The recorded data is compressed and encoded, and then prepared to be sent to the server. Once ready, the voice data is sent to the server.

[0065] Step 8:

[0066] The server analyzes the received image data using visual information analysis tools. Image recognition algorithms are used to identify text (movie titles and information) and objects within the image. Necessary metadata (title, location, date and time, etc.) is extracted.

[0067] Step 9:

[0068] The server analyzes the received audio data using an audio analysis tool. An audio recognition algorithm is used to convert the audio data into text. The user's question is analyzed from the converted text data, and relevant queries are generated.

[0069] Step 10:

[0070] The server activates an information generation system based on data obtained from visual information analysis and audio analysis, and generates an answer to the user's question (showtimes). The generated answer data is then converted into a display format.

[0071] Step 11:

[0072] The server sends the formatted response data to the smart glasses.

[0073] Step 12:

[0074] The device (smart glasses) decodes the received response data and prepares it for display on the screen. The screen then displays the showtime schedule information to the user.

[0075] Step 13:

[0076] The user checks the showtimes displayed on their smart glasses. They can continue to obtain information by asking further questions or focusing on other advertisements or signs as needed.

[0077] (Example 1)

[0078] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0079] Conventional information acquisition systems have struggled to implement efficient search functions that combine images and audio. Furthermore, there was a lack of systems that could instantly acquire information when a user gives voice commands about an object they are visually focusing on. Additionally, there was a lack of means to integrate visual and audio information to generate search queries and provide users with appropriate information.

[0080] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0081] In this invention, the server includes visual information analysis means, audio analysis means, search query generation means, and information generation means. This enables the efficient generation of search queries by integrating visual and audio information, allowing the user to instantly obtain necessary information about the object they are focusing on.

[0082] An "image capture means" is a device used to capture an image of an object that the user is focusing on.

[0083] A "voice acquisition means" is a device used to acquire voice commands from a user.

[0084] "Visual information analysis means" refers to an algorithm or system for extracting text or specific objects from captured image data and generating metadata.

[0085] "Voice analysis means" refers to an algorithm or system that converts acquired voice data into text and understands the content and intent of the user's questions.

[0086] "Search query generation means" refers to an algorithm or system for generating queries for information retrieval based on the results of visual information analysis means and audio analysis means.

[0087] "Information generation means" refers to an algorithm or system for searching databases or the web for information based on a search query and formatting it into an appropriate format.

[0088] A "display means" is a device used to visually provide the user with information obtained as search results.

[0089] This invention is a system that combines images of objects the user is focusing on with audio instructions to perform information retrieval and display the results. Specifically, it uses smart glasses and earphones to acquire visual and audio information, which is then analyzed and searched on a server to provide the user with the necessary information. The details of each hardware and software component of this system, as well as examples, are described below.

[0090] System Configuration

[0091] Devices (smart glasses & earphones):

[0092] Smart glasses incorporate both an image capture device (camera) and a display device (display). Commercially available smart glasses are a concrete example of this.

[0093] Earphones have a built-in voice acquisition mechanism (microphone). A concrete example is commercially available Bluetooth earphones.

[0094] server:

[0095] The server is equipped with visual information analysis means, speech analysis means, search query generation means, and information generation means. These means are implemented by various algorithms and software.

[0096] System operation

[0097] To use the system, the user puts on smart glasses and earphones and activates the terminal. The smart glasses' camera captures an image of the object the user is looking at, and the earphone's microphone captures the user's voice commands. This data is sent from the terminal to the server, where the following processing takes place.

[0098] Visual Information Analysis: The server analyzes the received image data using visual information analysis tools (e.g., image recognition algorithms) and extracts text and specific objects from the image. This generates metadata.

[0099] Speech information analysis: Next, the speech data is converted into text using speech analysis tools (e.g., speech recognition algorithms) to understand the user's questions and intentions.

[0100] Search query generation: Search queries are generated based on metadata obtained from visual information and audio analysis results.

[0101] Information generation: Using a search query, the information generation system searches for information in databases and on the web. The obtained information is formatted in an appropriate format and sent to the terminal.

[0102] Information display: The device displays information received from the server on the smart glasses' display, which the user can then review.

[0103] Specific example

[0104] For example, consider a scenario where a user is looking at a poster for a new movie while walking down the street. The user speaks into their earphones, "Tell me the showtimes for this movie." The smart glasses' camera captures an image of the poster, and the earphones receive the voice command. This data is sent to a server and processed using the following steps.

[0105] The image data is analyzed on the server, and the movie title and other metadata are extracted.

[0106] Voice instructions are converted into text, and the user's questions are understood.

[0107] A search query is generated based on the extracted metadata and the user's question.

[0108] The information generation system searches for movie screening schedules and formats the results.

[0109] The formatted information is sent to the device and displayed on the smart glasses' screen.

[0110] Example of a prompt

[0111] "Could you tell me the screening schedule for the movie posters I'm keeping an eye on?"

[0112] "Please search for and display reviews for this product."

[0113] In this way, users can efficiently obtain the necessary information and utilize it beneficially in their daily lives and work.

[0114] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0115] Step 1: Start the device

[0116] The user puts on the smart glasses and earphones and activates the system. This puts the smart glasses' camera (image capture means) and the earphones' microphone (voice acquisition means) into standby mode. The input is the user's operation, and the output is the camera and microphone entering standby mode.

[0117] Step 2: Image capture and audio acquisition

[0118] The device uses a camera built into the smart glasses to capture an image of the object the user is looking at. Simultaneously, the microphone in the earphones captures the user's voice commands. The input consists of visual information of the object and the user's voice commands, and the output consists of captured image data and audio data.

[0119] Step 3: Data transmission

[0120] The terminal sends the captured image data and acquired audio data to the server. This process uses data communication. The input is image data and audio data, and the output is this data transferred to the server.

[0121] Step 4: Analyze visual information on the server

[0122] The server analyzes the received image data using visual information analysis tools. Specifically, it uses image recognition algorithms (e.g., image analysis libraries) to extract text and specific objects (e.g., movie titles) from the images. The input is image data, and the output is the extracted metadata.

[0123] Step 5: Analyze the audio information on the server

[0124] The server analyzes the audio data using speech analysis tools. It converts the audio data into text using a speech recognition algorithm (e.g., a speech analysis library) to understand the user's questions and intentions. The input is audio data, and the output is text data.

[0125] Step 6: Generate search queries

[0126] The server generates search queries based on metadata obtained from visual information analysis and audio analysis results. For example, it combines extracted movie titles with the user's question, "showtimes," to form a search query. The input is metadata and text data, and the output is a generated search query.

[0127] Step 7: Information retrieval and formatting

[0128] The server uses information generation tools to search databases and web resources based on the generated search query. It then formats the search results into an appropriate format. For example, movie screening schedules might be presented in a table. The input is a search query, and the output is formatted information.

[0129] Step 8: Submitting Information

[0130] The server sends formatted information to the terminal. The input is formatted information, and the output is information transferred to the terminal.

[0131] Step 9: Displaying information on the device

[0132] The device displays information received from the server on the smart glasses' screen. The user can visually confirm the information displayed on the smart glasses' screen. The input is information sent from the server, and the output is a visual display.

[0133] (Application Example 1)

[0134] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0135] In autonomous vehicles, there is a lack of safe and efficient methods for passengers to obtain real-time information about their destination and surrounding facilities. Existing navigation systems provide visual information, but they lack sufficient real-time integration with the external environment, and there is a need for an effective system to improve passenger convenience.

[0136] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0137] In this invention, the server includes image capture means, voice acquisition means, visual information analysis means, voice analysis means, information generation means, display means, search means, and facility information acquisition means. This makes it possible to capture images of facilities that passengers are looking at using smart glasses and to obtain detailed information and reviews in real time through voice commands.

[0138] An "image capture means" is a device for acquiring an image of an object that the user is focusing on.

[0139] A "voice acquisition means" is a device for acquiring voice commands from a user.

[0140] A "visual information analysis device" is a device that analyzes acquired image data and extracts information about the object.

[0141] A "voice analysis device" is a device that analyzes acquired voice data to understand the user's instructions.

[0142] An "information generation means" is a device for generating necessary information based on analyzed visual and auditory information.

[0143] A "display means" is a device for visually presenting generated information to the user.

[0144] A "search device" is a device that generates appropriate search queries based on acquired visual and audio information and retrieves information from the internet or a database.

[0145] A "facility information acquisition means" is a device that analyzes information obtained through a search means and acquires detailed information about the facility being monitored.

[0146] This invention is a system that uses smart glasses and earphones to combine visual information and voice instructions, automatically retrieves information, and provides the results to the user. This system is particularly useful in autonomous vehicles to allow passengers to obtain real-time information about their destination and surrounding facilities. The system configuration and processing details are described below.

[0147] System Configuration

[0148] Hardware:

[0149] Smart glasses: Equipped with image capture means and display means.

[0150] Earphones: Equipped with a means of acquiring sound.

[0151] The server is equipped with visual information analysis means, speech analysis means, information generation means, search means, and facility information acquisition means.

[0152] software:

[0153] OpenCV: Used for image capture and preprocessing.

[0154] SpeechRecognition: This is a library for performing speech recognition.

[0155] requests: This is a library for performing API communication for image analysis and information retrieval.

[0156] Reproduction steps

[0157] 1. Smart glasses capture images:

[0158] The user observes the external environment through smart glasses. The smart glasses' camera captures an image of the object the user is looking at. This image is temporarily stored within the smart glasses.

[0159] 2. The earphones receive voice commands:

[0160] The user gives voice commands into the earphones. The microphone built into the earphones captures the voice data. This voice data is transmitted to the server via the smart glasses.

[0161] 3. The server analyzes the visual information:

[0162] Image data sent to the server is analyzed by a visual information analysis system. Image processing libraries such as OpenCV are used for this analysis. As a result of the analysis, metadata of the object being viewed is generated.

[0163] 4. The server analyzes the audio data:

[0164] The audio data is analyzed by a speech analysis tool. The SpeechRecognition library is used for this analysis. The audio data is converted into text data, and the user's instructions are understood.

[0165] 5. The server generates the information:

[0166] Based on visual and auditory information, appropriate search queries are generated. The search method retrieves the necessary information via the internet or databases. Specifically, it retrieves detailed information, reviews, and business hours related to the target object.

[0167] 6. Smart glasses display information:

[0168] The generated information is transmitted to the smart glasses' display and provided to the user visually. The user can view the information in real time through the smart glasses.

[0169] Specific example

[0170] For example, a passenger might look at a restaurant through smart glasses and ask a question into their earphone: "What's the reputation of this restaurant?" The smart glasses' camera captures an image of the restaurant and sends it to the server. Simultaneously, the earphone receives a voice command and also sends it to the server. The server performs image analysis and voice recognition to retrieve the necessary data. As a result, restaurant reviews and ratings are displayed on the smart glasses, allowing the user to check the information in real time.

[0171] Example of a prompt

[0172] A user looks at a specific establishment (e.g., a restaurant) through smart glasses and asks, "What's the reputation of this restaurant?" into their earphones. The camera in the smart glasses captures an image of the restaurant, and this image is sent to a server. The server performs image analysis (e.g., extracting the restaurant's name) and speech recognition (e.g., transcribing the question into text), and a search query is generated based on this information. Based on the search query, restaurant reviews and ratings are retrieved and displayed on the smart glasses. Finally, the smart glasses display shows the restaurant's review score and some rating comments.

[0173] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0174] Step 1:

[0175] The smart glasses are activated and capture an image of the object the user is looking at.

[0176] Input: Visual information of the object the smart glasses are focusing on.

[0177] Data processing: Image capture means (camera in smart glasses) acquires image data.

[0178] Output: Captured image data

[0179] Step 2:

[0180] The earphones receive the user's voice commands.

[0181] Input: User voice commands

[0182] Data processing: The voice acquisition device (microphone in the earphones) acquires voice data and transmits it to the smart glasses.

[0183] Output: Acquired audio data

[0184] Step 3:

[0185] The smart glasses send the captured image data and acquired audio data to the server.

[0186] Input: Image data, audio data

[0187] Data processing: Data is transmitted from the smart glasses to the server via communication.

[0188] Output: Image data and audio data sent to the server

[0189] Step 4:

[0190] The server analyzes the visual information and generates metadata about the object being focused on.

[0191] Input: Sent image data

[0192] Data processing: Visual information analysis tools (e.g., OpenCV library) perform image recognition and generate metadata for the object.

[0193] Output: Generated metadata (e.g., name and characteristics of the object)

[0194] Step 5:

[0195] The server analyzes the voice data and converts the user's voice instructions into text.

[0196] Input: Sent audio data

[0197] Data processing: A speech analysis tool (e.g., SpeechRecognition library) performs speech recognition and converts the audio data into text.

[0198] Output: Analyzed text data (e.g., user's question)

[0199] Step 6:

[0200] The server generates search queries based on visual and audio information and searches for information on the internet or in databases.

[0201] Input: Generated metadata, parsed text data

[0202] Data processing: The search tool generates appropriate search queries and searches for information in external databases or on the internet.

[0203] Output: Search results (e.g., detailed information and reviews of the target facility)

[0204] Step 7:

[0205] The server feeds the search results back to the smart glasses and displays the information.

[0206] Input: Search Results

[0207] Data processing: The information generation device formats the search results into an appropriate format and transmits them to the smart glasses.

[0208] Output: Detailed information displayed on smart glasses (e.g., facility reviews and ratings)

[0209] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0210] This invention is a system that uses smart glasses and earphones to automatically retrieve information by combining visual information and voice commands, and provides the results to the user. Furthermore, by incorporating an emotion engine, it has the function of recognizing the user's emotional state and adjusting the content and format of the information displayed based on that state.

[0211] System Configuration

[0212] 1. Devices (smart glasses & earphones):

[0213] Smart glasses incorporate image capture capabilities, an emotion engine, and display capabilities.

[0214] The earphones have a built-in mechanism for acquiring sound.

[0215] 2. Server:

[0216] The server is equipped with visual information analysis means, audio analysis means, and information generation means, which work together in coordination.

[0217] Program processing

[0218] When the device is activated, the smart glasses' camera captures an image of the object the user is looking at. This image is sent to the server by the device. Simultaneously, the earphones receive the user's voice commands, and the voice data is also sent to the server.

[0219] The server analyzes the received image data using visual information analysis tools. Specifically, it uses image recognition algorithms to identify text and specific objects within the image and extracts the necessary metadata.

[0220] Next, the audio data is analyzed by an audio analysis tool. A speech recognition algorithm is used to convert the audio data into text, understanding the user's question and intent. This combines the aforementioned metadata and the question to generate a search query.

[0221] The information generation system uses this search query to search for information in databases and on the web. The information obtained as a search result is converted into a format and then sent from the server to the terminal.

[0222] The device (smart glasses) also incorporates an emotion engine that analyzes the user's emotional state from their facial expressions and voice tone. This emotion engine has the function of adjusting the content and format of the generated information based on the analyzed emotional state. For example, if the system recognizes that the user is prone to anxiety, it will simplify the displayed content and apply an interface designed to reduce stress.

[0223] Information is displayed on the smart glasses' screen via a display device. The user can review the information displayed on the smart glasses and, if necessary, ask further questions or gather additional information.

[0224] Specific example

[0225] Let's consider an example where a user is looking at a poster for a new movie in the street and asks a question into their earphones: "Tell me the showtimes for this movie." The smart glasses' camera captures an image of the poster and sends it to the server. The earphones receive the voice command and send it to the server.

[0226] On the server, the poster image is analyzed by a visual information analysis device, and the movie title and other metadata are extracted. Simultaneously, the voice instructions are analyzed by a voice analysis device, and the user's question is understood. Based on this data, an information generation device searches for the movie screening schedule and generates an answer.

[0227] The device's (smart glasses') emotion engine recognizes signs of anxiety when the user is asked a question. Based on this recognition, the displayed information is adjusted to be concise and easy to understand. Additional information to help the user relax (e.g., the location of a relaxing cafe) may also be provided.

[0228] Ultimately, the information is displayed on the smart glasses' screen, allowing the user to efficiently obtain the necessary information.

[0229] The following describes the processing flow.

[0230] Step 1:

[0231] The server verifies the connection between the smart glasses and earphones and performs the necessary initial setup. This setup involves verifying the UUIDs (Universal Unique Identifiers) of the smart glasses and earphones, and checking the battery status and network connectivity of each device. After confirming everything is working correctly, a notification is sent to the device indicating that the initial setup is complete.

[0232] Step 2:

[0233] The device (smart glasses & earphones) starts up and begins communicating with the server. Upon receiving notification from the server that the initial setup is complete, an alert is displayed on the smart glasses' screen to inform the user that it is ready for use.

[0234] Step 3:

[0235] The user focuses on a movie poster they see in the street. This fixed gaze triggers the smart glasses' camera.

[0236] Step 4:

[0237] The device (smart glasses) activates its camera and captures an image of the poster the user is looking at. The camera automatically focuses and takes a clear image. The image data is stored in temporary memory.

[0238] Step 5:

[0239] The device (smart glasses) compresses and encodes the image data it captures and prepares it for transmission to the server. Once ready, the image data is sent to the server.

[0240] Step 6:

[0241] The user asks into their earphone, "Tell me the showtimes for this movie." The voice command is then acquired.

[0242] Step 7:

[0243] The device (earphones) records the user's voice data and saves it to temporary memory. The recorded data is compressed and encoded, and then prepared to be sent to the server. Once ready, the voice data is sent to the server.

[0244] Step 8:

[0245] The server analyzes the received image data using visual information analysis tools. Image recognition algorithms are used to identify text (movie titles and information) and objects within the image. Necessary metadata (title, location, date and time, etc.) is extracted.

[0246] Step 9:

[0247] The server analyzes the received audio data using an audio analysis tool. An audio recognition algorithm is used to convert the audio data into text. The user's question is analyzed from the converted text data, and relevant queries are generated.

[0248] Step 10:

[0249] The server activates an information generation system based on data obtained from visual information analysis and audio analysis, and generates an answer to the user's question (showtimes). The generated answer data is then converted into a display format.

[0250] Step 11:

[0251] The server sends the formatted response data to the smart glasses.

[0252] Step 12:

[0253] The terminal (smart glasses) decodes the received response data and prepares it for display on the display device.

[0254] Step 13:

[0255] The emotion engine built into the device (smart glasses) analyzes the user's facial expressions and voice tone to recognize their emotional state. For example, if the user is feeling anxious, the emotion engine will detect this.

[0256] Step 14:

[0257] Based on the emotional state recognized by the emotion engine, the information generation means adjusts the display content and format of the information it generates. For example, when the user is anxious, the information generation means simplifies the display content and provides additional relaxing content to reduce stress (e.g., information about nearby cafes).

[0258] Step 15:

[0259] The device (smart glasses) displays the adjusted information on its screen. The user is visually provided with information such as movie schedules and places to relax.

[0260] Step 16:

[0261] The user checks the showtimes and relaxation information displayed on the smart glasses. Further information can be obtained by asking additional questions or selecting a new object to focus on, as needed.

[0262] This processing flow provides users with a system that allows them to obtain the necessary information efficiently and with reduced stress.

[0263] (Example 2)

[0264] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0265] Conventional information retrieval systems lack the ability to simultaneously process the user's visual focus and voice commands, making it difficult for users to efficiently obtain information. Furthermore, they fail to provide information in a format optimal for the user because they display information uniformly without considering the user's emotional state.

[0266] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0267] In this invention, the server includes image capture means, audio acquisition means, visual information analysis means, audio analysis means, information generation means, display means, and emotion analysis means. This enables information retrieval and display based on the user's visual focus and audio instructions. Furthermore, it can provide information in an appropriate format according to the user's emotional state.

[0268] "Image capture means" refers to a device or function for capturing an image of an object that the user is focusing on.

[0269] "Voice acquisition means" refers to a device or function for acquiring voice commands from a user.

[0270] "Visual information analysis means" refers to a device or function for analyzing captured image data and identifying text and objects within the image.

[0271] "Speech analysis means" refers to a device or function for converting acquired speech data into text and analyzing it.

[0272] "Information generation means" refers to a device or function that generates search queries based on analyzed data, obtains necessary information, and formats it.

[0273] "Display means" refers to a device or function for displaying generated information to the user.

[0274] "Emotional analysis means" refers to a device or function that analyzes the user's emotional state from their facial expressions and voice tone, and adjusts the content and format of the information displayed based on the results.

[0275] This invention is a system including image capture means, sound acquisition means, visual information analysis means, sound analysis means, information generation means, display means, and emotion analysis means. This system automatically acquires information about an object that the user is visually fixated on and searches for and provides appropriate information based on voice instructions. It also has a function to analyze the user's emotional state and adjust the content and format of the information displayed accordingly.

[0276] Hardware and software configuration

[0277] Devices (smart glasses & earphones):

[0278] Smart glasses: They incorporate image capture means (camera), emotion analysis means, and display means.

[0279] Earphones: They have a built-in means of acquiring sound.

[0280] server:

[0281] Visual information analysis means: Using an image recognition algorithm, it analyzes the captured image data.

[0282] Voice analysis means: Using a voice recognition algorithm, it converts the acquired voice into text and analyzes it.

[0283] Information generation means: Based on the analyzed data, it searches for and formats the necessary information.

[0284] Program processing

[0285] The process of implementing the invention:

[0286] When the user uses the system, first wear smart glasses and wear earphones. When the system starts, the camera of the smart glasses captures an image of the object the user is looking at. This image data is sent from the terminal to the server, and at the same time, the earphones acquire the user's voice instruction and also send it to the server.

[0287] On the server, the received image data is analyzed by the visual information analysis means to identify specific text or objects from the visual information and extract the necessary metadata. At the same time, the sent voice data is analyzed by the voice analysis means to convert the voice instruction into text and understand the content of the user's question. Based on this data, the information generation means creates a search query, acquires and formats the necessary information from the database or the web, and sends it to the terminal.

[0288] The emotion analysis means incorporated in the smart glasses analyzes the emotional state from the user's expression and voice tone. Based on the analysis result, it adjusts the display content and form of the information and provides the information in a form suitable for the user.

[0289] Specific examples

[0290] Example 1:

[0291] A user spots a poster for a new movie in the street and asks into their earphones, "Tell me the screening schedule for this movie." The smart glasses' camera captures an image of the poster and sends it to the server. Simultaneously, the earphones capture the voice command and send it to the server.

[0292] Specific actions:

[0293] The server's visual information analysis tool extracts metadata such as "movie title" and "director" from the movie poster.

[0294] The voice analysis system converts the user's voice instructions into text and understands the content of the question.

[0295] The information generation system searches for and formats movie screening schedules.

[0296] The emotion analysis tool recognizes signs of anxiety and stress in the user, adjusts the information display to a concise and easy-to-understand format, and may even provide location information for cafes where users can relax.

[0297] Information is displayed on the smart glasses' screen.

[0298] Example of a prompt:

[0299] "Please tell me about the latest Marvel movie. I'd like to know the screening schedule."

[0300] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0301] Step 1:

[0302] Starting up and preparing the device

[0303] The device (smart glasses & earphones) is activated. This prepares the smart glasses' camera and display, and the earphones' audio acquisition capabilities.

[0304] Specific operation: When the smart glasses are powered on, the capture function and display of the built-in camera are ready. At the same time, the earphones are also paired, and the voice input is ready to receive.

[0305] Input: Power on the smart glasses, pair the earphones.

[0306] Output: The camera and display are ready, the voice acquisition means is ready.

[0307] Step 2:

[0308] User instruction input

[0309] The user wears the smart glasses and stares at the object in front of him / her, and voices an instruction to obtain information towards the earphones.

[0310] Specific operation: The user stares at a poster of a new movie in the street and voices "Tell me the screening schedule of this movie".

[0311] Input: User's line of sight, voice instruction.

[0312] Output: Capture of visual information, acquisition of voice information.

[0313] Step 3:

[0314] Data acquisition and transmission to the server

[0315] The terminal (smart glasses & earphones) captures an image of the object the user is looking at with the camera and acquires the voice instruction with the earphones. These data are sent to the server.

[0316] Specific operation: The camera of the smart glasses captures an image of the movie poster and sends the data to the server. At the same time, the earphones capture the user's voice instruction "Tell me the screening schedule of this movie" and send it to the server.

[0317] Input: Captured image data, acquired audio data.

[0318] Output: Sending image data and audio data to the server.

[0319] Step 4:

[0320] Data analysis on the server

[0321] The server analyzes the received image data using visual information analysis tools to identify text and objects within the image and extract metadata. It also analyzes audio data using audio analysis tools, converts the audio data into text, and understands the user's questions and intentions.

[0322] Specific operation: The server's visual information analysis means extracts metadata such as "movie title," "director," and "lead actor" from the movie poster. The voice analysis means converts the user's voice command into text such as "I want to know the screening schedule for this movie" and understands the content of the question.

[0323] Input: Image data, audio data.

[0324] Output: Extracted metadata, parsed text data.

[0325] Step 5:

[0326] Generating search queries and retrieving information

[0327] The server's information generation mechanism generates search queries based on extracted metadata and voice commands. These queries are used to search for the necessary information in databases and on the web, and the retrieved information is then formatted.

[0328] Specific operation: The server generates a search query containing the "movie title" and "showtimes," and searches for showtimes from the movie's official website and other databases. The retrieved information is then formatted in a format suitable for the user.

[0329] Input: Metadata, parsed text data.

[0330] Output: Generated search query, formatted information.

[0331] Step 6:

[0332] Sentiment analysis and information format adjustment

[0333] The device's (smart glasses') emotion analysis mechanism analyzes the user's emotional state from their facial expressions and voice tone. Based on the analysis results, it adjusts the content and format of the information provided.

[0334] Specific operation: The emotion analysis system recognizes signs of anxiety and stress from the user's facial expressions and voice tone, and adjusts the displayed content to a concise and simple format. In addition, it provides location information for cafes where users can relax.

[0335] Input: Formatted information, user's facial expressions, and voice tone.

[0336] Output: Adjusted information format, additional relaxation information.

[0337] Step 7:

[0338] Information display and user feedback

[0339] The adjusted information is displayed on the smart glasses' screen via the device's display mechanism. The user can review the displayed information and, if necessary, ask further questions or gather additional information.

[0340] Specific operation: The smart glasses display concisely shows the "movie title," "showtimes," "map of the nearest theater," and "information on relaxing cafes." The user can then check the location of the nearest theater and go to see a movie.

[0341] Input: Adjusted information format, additional relaxation information.

[0342] Output: Information displayed on the smart glasses' screen.

[0343] (Application Example 2)

[0344] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0345] In traditional in-store shopping experiences, it has been difficult for customers to quickly and accurately obtain product information. Furthermore, there was no system that provided information while considering the customer's emotional state, posing a challenge to improving customer satisfaction. Additionally, the lack of quick information access in physical stores sometimes caused customer stress. To address these challenges, a system is needed that accurately analyzes customer visual information and voice commands to provide product information in real time.

[0346] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0347] In this invention, the server includes visual information analysis means, voice analysis means, information generation means, and emotion analysis means. This enables image capture based on the customer's gaze, accurate analysis of voice instructions, real-time product information provision, and information provision tailored to the customer's emotional state.

[0348] An "image capture means" is a device for acquiring an image of an object that the user is focusing on.

[0349] A "voice acquisition means" is a device for acquiring voice commands from a user.

[0350] A "visual information analysis device" is a device for analyzing acquired image data and extracting necessary metadata.

[0351] A "voice analysis device" is a device that analyzes acquired voice data to understand the user's intent and the content of their questions.

[0352] An "information generation means" is a device that generates search queries based on visual and auditory information, and searches for and generates necessary information.

[0353] A "display means" is a device for providing generated information to the user visually.

[0354] An "emotion analysis device" is a device that analyzes the user's emotional state from acquired image and audio data and adjusts the content and format of the information displayed based on the results.

[0355] A "physical store" is a store that has a physical presence and sells products directly to customers.

[0356] "Smart glasses" are glasses-type devices worn by the user that have the function of capturing and displaying images.

[0357] A "search query" is a statement of inquiry generated for the purpose of retrieving information.

[0358] This invention is a system that uses smart glasses and earphones to automatically search for information by combining visual information and voice instructions, and provides the results to the user. Furthermore, by incorporating emotion analysis means, it has the function of recognizing the user's emotional state and adjusting the content and format of the information displayed based on that state.

[0359] System Configuration

[0360] 1. Device (smart glasses & earphones)

[0361] Smart glasses incorporate image capture and display mechanisms.

[0362] The earphones have a built-in mechanism for acquiring sound.

[0363] 2. Server

[0364] The server is equipped with visual information analysis means, audio analysis means, information generation means, and emotion analysis means, all of which work together in coordination.

[0365] Explanation of the program's processing

[0366] When the device is activated, the smart glasses' camera captures an image of the object the user is looking at. This image is sent to the server by the device. Simultaneously, the earphones receive the user's voice commands, and the voice data is also sent to the server.

[0367] The server analyzes and generates data as follows:

[0368] Hardware and software details:

[0369] Smart glasses (image capture and display): Smart glasses have the function of capturing images of objects that the user is looking at and displaying information about them.

[0370] Earphones (voice acquisition): Earphones have the function of acquiring the user's voice commands.

[0371] Server (visual information analysis, audio analysis, information generation, sentiment analysis): The server analyzes and generates data using the following software.

[0372] Visual information analysis method: OpenCV

[0373] Voice analysis method: Google Cloud Speech-to-Text API

[0374] Information generation method: MySQL, Elasticsearch

[0375] Emotion analysis method: Microsoft Azure's Emotion API

[0376] Specifically, the server analyzes the received image data using visual information analysis tools. Specifically, it uses OpenCV to identify text and specific objects within the image and extracts the necessary metadata. Next, the audio data is analyzed using speech analysis tools. The Google Cloud Speech-to-Text API is used to convert the audio data into text, understanding the user's question and intent. This combines the aforementioned metadata with the audio instructions to generate a search query. The information generation tool then uses this search query to retrieve data from MySQL or Elasticsearch and search for and generate the necessary information.

[0377] Specific example

[0378] This scenario describes a user browsing shirts in a physical store and asking a question into their earphones, "Do you have this shirt in my size?" The smart glasses' camera captures an image of the shirt and sends it to the server. The earphones receive the user's voice command and send it to the server.

[0379] On the server, images of shirts are analyzed by a visual information analysis tool (OpenCV) to extract barcodes and product information. Simultaneously, voice commands are analyzed by a speech analysis tool (Google Cloud Speech-to-Text API) to understand the user's questions. Based on this data, information generation tools (MySQL, Elasticsearch) search for and generate shirt size information. An emotion analysis tool (Microsoft Azure's Emotion API) analyzes the user's emotional state from their facial expressions and voice tone, and adjusts the displayed content accordingly.

[0380] The adjusted information is displayed on the smart glasses' screen, providing real-time information such as, "This shirt is available in sizes S, M, and L."

[0381] Example of a prompt

[0382] "Please provide the application prompt for capturing a barcode and displaying product information in real time when a customer is viewing a product using smart glasses."

[0383] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0384] Step 1:

[0385] The user puts on smart glasses and activates the device using earphones.

[0386] Input: Terminal startup command

[0387] Output: Smart glasses and earphones are ready.

[0388] Operation: When the device is powered on, the smart glasses and earphones connect and become ready for use.

[0389] Step 2:

[0390] The smart glasses' camera captures images based on the user's gaze.

[0391] Input: Objects that come into the user's line of sight

[0392] Output: Captured image data

[0393] Operation: The smart glasses' camera acquires an image of the object based on the gaze and sends that data to the device.

[0394] Step 3:

[0395] The earphones receive the user's voice commands.

[0396] Input: User's voice instructions

[0397] Output: Acquired audio data

[0398] Operation: The earphones pick up the user's voice and transmit that audio data to the device.

[0399] Step 4:

[0400] The device sends the captured image and audio data to the server.

[0401] Input: Acquisition of image data and audio data

[0402] Output: Data sent to the server

[0403] Operation: The device transfers image data and audio data acquired by the device to the server.

[0404] Step 5:

[0405] The server analyzes the image data using a visual information analysis tool (OpenCV).

[0406] Input: Image data sent to the server

[0407] Output: Analyzed metadata (e.g., barcode information)

[0408] Operation: The server uses OpenCV to analyze image data and extract the necessary metadata.

[0409] Step 6:

[0410] The server analyzes the audio data using a speech analysis tool (Google Cloud Speech-to-Text API).

[0411] Input: Audio data sent to the server

[0412] Output: Transcripted audio data

[0413] Operation: The server uses the Google Cloud Speech-to-Text API to convert the audio data into text and understand the user's question.

[0414] Step 7:

[0415] The server generates search queries based on visual and audio information and retrieves the information using information generation tools (MySQL, Elasticsearch).

[0416] Input: Analyzed metadata and transcribed audio data

[0417] Output: Search query and generated information

[0418] Operation: The server generates search queries and uses MySQL or Elasticsearch to search for and generate the necessary information.

[0419] Step 8:

[0420] The server analyzes the user's emotional state using an emotion analysis tool (Microsoft Azure's Emotion API).

[0421] Input: Image data and audio data

[0422] Output: Adjustments based on the user's emotional state

[0423] Operation: The server uses Microsoft Azure's Emotion API to analyze the user's emotional state and adjust the displayed content and format accordingly.

[0424] Step 9:

[0425] The server sends the adjusted information to the terminal.

[0426] Input: Adjusted information

[0427] Output: Adjustment information sent to the terminal

[0428] Operation: The server transfers the adjusted information to the terminal.

[0429] Step 10:

[0430] The device displays information on the smart glasses' screen.

[0431] Input: Receiving adjusted information from the terminal

[0432] Output: Information displayed on smart glasses

[0433] Operation: The device displays the information it receives on the smart glasses' screen, providing it to the user visually.

[0434] Example prompt: "Please provide a prompt for an application that captures a barcode and displays product information in real time when a customer is viewing a product using smart glasses."

[0435] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0436] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0437] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0438] [Second Embodiment]

[0439] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0440] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0441] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0442] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0443] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0444] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0445] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0446] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0447] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0448] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0449] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0450] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0451] This invention is a system that uses smart glasses and earphones to combine visual information and voice instructions, automatically retrieves information, and provides the results to the user.

[0452] System Configuration

[0453] 1. Devices (smart glasses & earphones):

[0454] Smart glasses incorporate image capture and display mechanisms.

[0455] The earphones have a built-in mechanism for acquiring sound.

[0456] 2. Server:

[0457] The server is equipped with visual information analysis means, audio analysis means, and information generation means, which work together in coordination.

[0458] Program processing

[0459] When the device is activated, the smart glasses' camera captures an image of the object the user is looking at. This image is sent to the server by the device. Simultaneously, the earphones receive the user's voice commands, and this voice data is also sent to the server.

[0460] The server first analyzes the transmitted image data using visual information analysis tools. Specifically, it uses image recognition algorithms to extract text and specific objects from the image and generate metadata.

[0461] Next, the audio data is analyzed by an audio analysis tool. Using a speech recognition algorithm, the audio data is converted into text, and the user's question and intent are understood based on this text data. This combines the aforementioned metadata and the question content to generate a search query.

[0462] The information generation system uses this search query to search for information in databases and on the web. The information obtained as a search result is formatted into an appropriate format and then sent from the server to the terminal.

[0463] The device (smart glasses) provides the user with received information visually through a display mechanism. The user can confirm the information displayed on the smart glasses' screen.

[0464] Specific example

[0465] Let's consider an example where a user is looking at a poster for a new movie in the street and asks a question into their earphones: "Tell me the showtimes for this movie." The smart glasses' camera captures an image of the poster and sends it to the server. The earphones receive the voice command and also send it to the server.

[0466] On the server, the poster image is analyzed by a visual information analysis device, and the movie title and other metadata are extracted. Simultaneously, the voice instructions are analyzed by a voice analysis device, and the user's question is understood. Based on this data, an information generation device searches for the movie's screening schedule and generates the results.

[0467] Ultimately, information is transmitted from the server to the smart glasses, allowing the user to view the showtimes displayed on the smart glasses' screen. In this way, a system is provided that allows the user to efficiently obtain the necessary information.

[0468] The following describes the processing flow.

[0469] Step 1:

[0470] The server verifies the connection between the smart glasses and earphones and performs the necessary initial setup. This setup involves verifying the UUIDs (Universal Unique Identifiers) of the smart glasses and earphones, and checking the battery status and network connectivity of each device. After confirming everything is working correctly, a notification is sent to the device indicating that the initial setup is complete.

[0471] Step 2:

[0472] The device (smart glasses & earphones) starts up and begins communicating with the server. Upon receiving notification from the server that the initial setup is complete, an alert is displayed on the smart glasses' screen to inform the user that it is ready for use.

[0473] Step 3:

[0474] The user focuses on a movie poster they see in the street. This fixed gaze triggers the smart glasses' camera.

[0475] Step 4:

[0476] The device (smart glasses) activates its camera and captures an image of the poster the user is looking at. The camera automatically focuses and takes a clear image. The image data is stored in temporary memory.

[0477] Step 5:

[0478] The device (smart glasses) compresses and encodes the image data it captures and prepares it for transmission to the server. Once ready, the image data is sent to the server.

[0479] Step 6:

[0480] The user asks into their earphone, "Tell me the showtimes for this movie." The voice command is then acquired.

[0481] Step 7:

[0482] The device (earphones) records the user's voice data and saves it to temporary memory. The recorded data is compressed and encoded, and then prepared to be sent to the server. Once ready, the voice data is sent to the server.

[0483] Step 8:

[0484] The server analyzes the received image data using visual information analysis tools. Image recognition algorithms are used to identify text (movie titles and information) and objects within the image. Necessary metadata (title, location, date and time, etc.) is extracted.

[0485] Step 9:

[0486] The server analyzes the received audio data using an audio analysis tool. An audio recognition algorithm is used to convert the audio data into text. The user's question is analyzed from the converted text data, and relevant queries are generated.

[0487] Step 10:

[0488] The server activates an information generation system based on data obtained from visual information analysis and audio analysis, and generates an answer to the user's question (showtimes). The generated answer data is then converted into a display format.

[0489] Step 11:

[0490] The server sends the formatted response data to the smart glasses.

[0491] Step 12:

[0492] The device (smart glasses) decodes the received response data and prepares it for display on the screen. The screen then displays the showtime schedule information to the user.

[0493] Step 13:

[0494] The user checks the showtimes displayed on their smart glasses. They can continue to obtain information by asking further questions or focusing on other advertisements or signs as needed.

[0495] (Example 1)

[0496] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0497] Conventional information acquisition systems have struggled to implement efficient search functions that combine images and audio. Furthermore, there was a lack of systems that could instantly acquire information when a user gives voice commands about an object they are visually focusing on. Additionally, there was a lack of means to integrate visual and audio information to generate search queries and provide users with appropriate information.

[0498] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0499] In this invention, the server includes visual information analysis means, audio analysis means, search query generation means, and information generation means. This enables the efficient generation of search queries by integrating visual and audio information, allowing the user to instantly obtain necessary information about the object they are focusing on.

[0500] An "image capture means" is a device used to capture an image of an object that the user is focusing on.

[0501] A "voice acquisition means" is a device used to acquire voice commands from a user.

[0502] "Visual information analysis means" refers to an algorithm or system for extracting text or specific objects from captured image data and generating metadata.

[0503] "Voice analysis means" refers to an algorithm or system that converts acquired voice data into text and understands the content and intent of the user's questions.

[0504] "Search query generation means" refers to an algorithm or system for generating queries for information retrieval based on the results of visual information analysis means and audio analysis means.

[0505] "Information generation means" refers to an algorithm or system for searching databases or the web for information based on a search query and formatting it into an appropriate format.

[0506] A "display means" is a device used to visually provide the user with information obtained as search results.

[0507] This invention is a system that combines images of objects the user is focusing on with audio instructions to perform information retrieval and display the results. Specifically, it uses smart glasses and earphones to acquire visual and audio information, which is then analyzed and searched on a server to provide the user with the necessary information. The details of each hardware and software component of this system, as well as examples, are described below.

[0508] System Configuration

[0509] Devices (smart glasses & earphones):

[0510] Smart glasses incorporate both an image capture device (camera) and a display device (display). Commercially available smart glasses are a concrete example of this.

[0511] Earphones have a built-in voice acquisition mechanism (microphone). A concrete example is commercially available Bluetooth earphones.

[0512] server:

[0513] The server is equipped with visual information analysis means, speech analysis means, search query generation means, and information generation means. These means are implemented by various algorithms and software.

[0514] System operation

[0515] To use the system, the user puts on smart glasses and earphones and activates the terminal. The smart glasses' camera captures an image of the object the user is looking at, and the earphone's microphone captures the user's voice commands. This data is sent from the terminal to the server, where the following processing takes place.

[0516] Visual Information Analysis: The server analyzes the received image data using visual information analysis tools (e.g., image recognition algorithms) and extracts text and specific objects from the image. This generates metadata.

[0517] Speech information analysis: Next, the speech data is converted into text using speech analysis tools (e.g., speech recognition algorithms) to understand the user's questions and intentions.

[0518] Search query generation: Search queries are generated based on metadata obtained from visual information and audio analysis results.

[0519] Information generation: Using a search query, the information generation system searches for information in databases and on the web. The obtained information is formatted in an appropriate format and sent to the terminal.

[0520] Information display: The device displays information received from the server on the smart glasses' display, which the user can then review.

[0521] Specific example

[0522] For example, consider a scenario where a user is looking at a poster for a new movie while walking down the street. The user speaks into their earphones, "Tell me the showtimes for this movie." The smart glasses' camera captures an image of the poster, and the earphones receive the voice command. This data is sent to a server and processed using the following steps.

[0523] The image data is analyzed on the server, and the movie title and other metadata are extracted.

[0524] Voice instructions are converted into text, and the user's questions are understood.

[0525] A search query is generated based on the extracted metadata and the user's question.

[0526] The information generation system searches for movie screening schedules and formats the results.

[0527] The formatted information is sent to the device and displayed on the smart glasses' screen.

[0528] Example of a prompt

[0529] "Could you tell me the screening schedule for the movie posters I'm keeping an eye on?"

[0530] "Please search for and display reviews for this product."

[0531] In this way, users can efficiently obtain the necessary information and utilize it beneficially in their daily lives and work.

[0532] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0533] Step 1: Start the device

[0534] The user puts on the smart glasses and earphones and activates the system. This puts the smart glasses' camera (image capture means) and the earphones' microphone (voice acquisition means) into standby mode. The input is the user's operation, and the output is the camera and microphone entering standby mode.

[0535] Step 2: Image capture and audio acquisition

[0536] The device uses a camera built into the smart glasses to capture an image of the object the user is looking at. Simultaneously, the microphone in the earphones captures the user's voice commands. The input consists of visual information of the object and the user's voice commands, and the output consists of captured image data and audio data.

[0537] Step 3: Data transmission

[0538] The terminal sends the captured image data and acquired audio data to the server. This process uses data communication. The input is image data and audio data, and the output is this data transferred to the server.

[0539] Step 4: Analyze visual information on the server

[0540] The server analyzes the received image data using visual information analysis tools. Specifically, it uses image recognition algorithms (e.g., image analysis libraries) to extract text and specific objects (e.g., movie titles) from the images. The input is image data, and the output is the extracted metadata.

[0541] Step 5: Analyze the audio information on the server

[0542] The server analyzes the audio data using speech analysis tools. It converts the audio data into text using a speech recognition algorithm (e.g., a speech analysis library) to understand the user's questions and intentions. The input is audio data, and the output is text data.

[0543] Step 6: Generate search queries

[0544] The server generates search queries based on metadata obtained from visual information analysis and audio analysis results. For example, it combines extracted movie titles with the user's question, "showtimes," to form a search query. The input is metadata and text data, and the output is a generated search query.

[0545] Step 7: Information retrieval and formatting

[0546] The server uses information generation tools to search databases and web resources based on the generated search query. It then formats the search results into an appropriate format. For example, movie screening schedules might be presented in a table. The input is a search query, and the output is formatted information.

[0547] Step 8: Submitting Information

[0548] The server sends formatted information to the terminal. The input is formatted information, and the output is information transferred to the terminal.

[0549] Step 9: Displaying information on the device

[0550] The device displays information received from the server on the smart glasses' screen. The user can visually confirm the information displayed on the smart glasses' screen. The input is information sent from the server, and the output is a visual display.

[0551] (Application Example 1)

[0552] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0553] In autonomous vehicles, there is a lack of safe and efficient methods for passengers to obtain real-time information about their destination and surrounding facilities. Existing navigation systems provide visual information, but they lack sufficient real-time integration with the external environment, and there is a need for an effective system to improve passenger convenience.

[0554] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0555] In this invention, the server includes image capture means, voice acquisition means, visual information analysis means, voice analysis means, information generation means, display means, search means, and facility information acquisition means. This makes it possible to capture images of facilities that passengers are looking at using smart glasses and to obtain detailed information and reviews in real time through voice commands.

[0556] An "image capture means" is a device for acquiring an image of an object that the user is focusing on.

[0557] A "voice acquisition means" is a device for acquiring voice commands from a user.

[0558] A "visual information analysis device" is a device that analyzes acquired image data and extracts information about the object.

[0559] A "voice analysis device" is a device that analyzes acquired voice data to understand the user's instructions.

[0560] An "information generation means" is a device for generating necessary information based on analyzed visual and auditory information.

[0561] A "display means" is a device for visually presenting generated information to the user.

[0562] A "search device" is a device that generates appropriate search queries based on acquired visual and audio information and retrieves information from the internet or a database.

[0563] A "facility information acquisition means" is a device that analyzes information obtained through a search means and acquires detailed information about the facility being monitored.

[0564] This invention is a system that uses smart glasses and earphones to combine visual information and voice instructions, automatically retrieves information, and provides the results to the user. This system is particularly useful in autonomous vehicles to allow passengers to obtain real-time information about their destination and surrounding facilities. The system configuration and processing details are described below.

[0565] System Configuration

[0566] Hardware:

[0567] Smart glasses: Equipped with image capture means and display means.

[0568] Earphones: Equipped with a means of acquiring sound.

[0569] The server is equipped with visual information analysis means, speech analysis means, information generation means, search means, and facility information acquisition means.

[0570] software:

[0571] OpenCV: Used for image capture and preprocessing.

[0572] SpeechRecognition: This is a library for performing speech recognition.

[0573] requests: This is a library for performing API communication for image analysis and information retrieval.

[0574] Reproduction steps

[0575] 1. Smart glasses capture images:

[0576] The user observes the external environment through smart glasses. The smart glasses' camera captures an image of the object the user is looking at. This image is temporarily stored within the smart glasses.

[0577] 2. The earphones receive voice commands:

[0578] The user gives voice commands into the earphones. The microphone built into the earphones captures the voice data. This voice data is transmitted to the server via the smart glasses.

[0579] 3. The server analyzes the visual information:

[0580] Image data sent to the server is analyzed by a visual information analysis system. Image processing libraries such as OpenCV are used for this analysis. As a result of the analysis, metadata of the object being viewed is generated.

[0581] 4. The server analyzes the audio data:

[0582] The audio data is analyzed by a speech analysis tool. The SpeechRecognition library is used for this analysis. The audio data is converted into text data, and the user's instructions are understood.

[0583] 5. The server generates the information:

[0584] Based on visual and auditory information, appropriate search queries are generated. The search method retrieves the necessary information via the internet or databases. Specifically, it obtains detailed information, reviews, and business hours related to the target object.

[0585] 6. Smart glasses display information:

[0586] The generated information is transmitted to the smart glasses' display and provided to the user visually. The user can view the information in real time through the smart glasses.

[0587] Specific example

[0588] For example, a passenger might look at a restaurant through smart glasses and ask a question into their earphone: "What's the reputation of this restaurant?" The smart glasses' camera captures an image of the restaurant and sends it to the server. Simultaneously, the earphone receives a voice command and also sends it to the server. The server performs image analysis and voice recognition to retrieve the necessary data. As a result, restaurant reviews and ratings are displayed on the smart glasses, allowing the user to check the information in real time.

[0589] Example of a prompt

[0590] A user looks at a specific establishment (e.g., a restaurant) through smart glasses and asks, "What's the reputation of this restaurant?" into their earphones. The camera in the smart glasses captures an image of the restaurant, and this image is sent to a server. The server performs image analysis (e.g., extracting the restaurant's name) and speech recognition (e.g., transcribing the question into text), and a search query is generated based on this information. Based on the search query, restaurant reviews and ratings are retrieved and displayed on the smart glasses. Finally, the smart glasses display shows the restaurant's review score and some rating comments.

[0591] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0592] Step 1:

[0593] The smart glasses are activated and capture an image of the object the user is looking at.

[0594] Input: Visual information of the object the smart glasses are focusing on.

[0595] Data processing: Image capture means (camera in smart glasses) acquires image data.

[0596] Output: Captured image data

[0597] Step 2:

[0598] The earphones receive the user's voice commands.

[0599] Input: User voice commands

[0600] Data processing: The voice acquisition device (microphone in the earphones) acquires voice data and transmits it to the smart glasses.

[0601] Output: Acquired audio data

[0602] Step 3:

[0603] The smart glasses send the captured image data and acquired audio data to the server.

[0604] Input: Image data, audio data

[0605] Data processing: Data is transmitted from the smart glasses to the server via communication.

[0606] Output: Image data and audio data sent to the server

[0607] Step 4:

[0608] The server analyzes the visual information and generates metadata about the object being focused on.

[0609] Input: Sent image data

[0610] Data processing: Visual information analysis tools (e.g., OpenCV library) perform image recognition and generate metadata for the object.

[0611] Output: Generated metadata (e.g., name and characteristics of the object)

[0612] Step 5:

[0613] The server analyzes the voice data and converts the user's voice instructions into text.

[0614] Input: Sent audio data

[0615] Data processing: A speech analysis tool (e.g., SpeechRecognition library) performs speech recognition and converts the audio data into text.

[0616] Output: Analyzed text data (e.g., user's question)

[0617] Step 6:

[0618] The server generates search queries based on visual and audio information and searches for information on the internet or in databases.

[0619] Input: Generated metadata, parsed text data

[0620] Data processing: The search tool generates appropriate search queries and searches for information in external databases or on the internet.

[0621] Output: Search results (e.g., detailed information and reviews of the target facility)

[0622] Step 7:

[0623] The server feeds the search results back to the smart glasses and displays the information.

[0624] Input: Search Results

[0625] Data processing: The information generation device formats the search results into an appropriate format and transmits them to the smart glasses.

[0626] Output: Detailed information displayed on smart glasses (e.g., facility reviews and ratings)

[0627] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0628] This invention is a system that uses smart glasses and earphones to automatically retrieve information by combining visual information and voice commands, and provides the results to the user. Furthermore, by incorporating an emotion engine, it has the function of recognizing the user's emotional state and adjusting the content and format of the information displayed based on that state.

[0629] System Configuration

[0630] 1. Devices (smart glasses & earphones):

[0631] Smart glasses incorporate image capture capabilities, an emotion engine, and display capabilities.

[0632] The earphones have a built-in mechanism for acquiring sound.

[0633] 2. Server:

[0634] The server is equipped with visual information analysis means, audio analysis means, and information generation means, which work together in coordination.

[0635] Program processing

[0636] When the device is activated, the smart glasses' camera captures an image of the object the user is looking at. This image is sent to the server by the device. Simultaneously, the earphones receive the user's voice commands, and the voice data is also sent to the server.

[0637] The server analyzes the received image data using visual information analysis tools. Specifically, it uses image recognition algorithms to identify text and specific objects within the image and extracts the necessary metadata.

[0638] Next, the audio data is analyzed by an audio analysis tool. A speech recognition algorithm is used to convert the audio data into text, understanding the user's question and intent. This combines the aforementioned metadata and the question to generate a search query.

[0639] The information generation system uses this search query to search for information in databases and on the web. The information obtained as a search result is converted into a format and then sent from the server to the terminal.

[0640] The device (smart glasses) also incorporates an emotion engine that analyzes the user's emotional state from their facial expressions and voice tone. This emotion engine has the function of adjusting the content and format of the generated information based on the analyzed emotional state. For example, if the system recognizes that the user is prone to anxiety, it will simplify the displayed content and apply an interface designed to reduce stress.

[0641] Information is displayed on the smart glasses' screen via a display device. The user can review the information displayed on the smart glasses and, if necessary, ask further questions or gather additional information.

[0642] Specific example

[0643] Let's consider an example where a user is looking at a poster for a new movie in the street and asks a question into their earphones: "Tell me the showtimes for this movie." The smart glasses' camera captures an image of the poster and sends it to the server. The earphones receive the voice command and send it to the server.

[0644] On the server, the poster image is analyzed by a visual information analysis device, and the movie title and other metadata are extracted. Simultaneously, the voice instructions are analyzed by a voice analysis device, and the user's question is understood. Based on this data, an information generation device searches for the movie screening schedule and generates an answer.

[0645] The device's (smart glasses') emotion engine recognizes signs of anxiety when the user is asked a question. Based on this recognition, the displayed information is adjusted to be concise and easy to understand. Additional information to help the user relax (e.g., the location of a relaxing cafe) may also be provided.

[0646] Ultimately, the information is displayed on the smart glasses' screen, allowing the user to efficiently obtain the necessary information.

[0647] The following describes the processing flow.

[0648] Step 1:

[0649] The server verifies the connection between the smart glasses and earphones and performs the necessary initial setup. This setup involves verifying the UUIDs (Universal Unique Identifiers) of the smart glasses and earphones, and checking the battery status and network connectivity of each device. After confirming everything is working correctly, a notification is sent to the device indicating that the initial setup is complete.

[0650] Step 2:

[0651] The device (smart glasses & earphones) starts up and begins communicating with the server. Upon receiving notification from the server that the initial setup is complete, an alert is displayed on the smart glasses' screen to inform the user that it is ready for use.

[0652] Step 3:

[0653] The user focuses on a movie poster they see in the street. This fixed gaze triggers the smart glasses' camera.

[0654] Step 4:

[0655] The device (smart glasses) activates its camera and captures an image of the poster the user is looking at. The camera automatically focuses and takes a clear image. The image data is stored in temporary memory.

[0656] Step 5:

[0657] The device (smart glasses) compresses and encodes the image data it captures and prepares it for transmission to the server. Once ready, the image data is sent to the server.

[0658] Step 6:

[0659] The user asks into their earphone, "Tell me the showtimes for this movie." The voice command is then acquired.

[0660] Step 7:

[0661] The device (earphones) records the user's voice data and saves it to temporary memory. The recorded data is compressed and encoded, and then prepared to be sent to the server. Once ready, the voice data is sent to the server.

[0662] Step 8:

[0663] The server analyzes the received image data using visual information analysis tools. Image recognition algorithms are used to identify text (movie titles and information) and objects within the image. Necessary metadata (title, location, date and time, etc.) is extracted.

[0664] Step 9:

[0665] The server analyzes the received audio data using an audio analysis tool. An audio recognition algorithm is used to convert the audio data into text. The user's question is analyzed from the converted text data, and relevant queries are generated.

[0666] Step 10:

[0667] The server activates an information generation system based on data obtained from visual information analysis and audio analysis, and generates an answer to the user's question (showtimes). The generated answer data is then converted into a display format.

[0668] Step 11:

[0669] The server sends the formatted response data to the smart glasses.

[0670] Step 12:

[0671] The terminal (smart glasses) decodes the received response data and prepares it for display on the display device.

[0672] Step 13:

[0673] The emotion engine built into the device (smart glasses) analyzes the user's facial expressions and voice tone to recognize their emotional state. For example, if the user is feeling anxious, the emotion engine will detect this.

[0674] Step 14:

[0675] Based on the emotional state recognized by the emotion engine, the information generation means adjusts the display content and format of the information it generates. For example, when the user is anxious, the information generation means simplifies the display content and provides additional relaxing content to reduce stress (e.g., information about nearby cafes).

[0676] Step 15:

[0677] The device (smart glasses) displays the adjusted information on its screen. The user is visually provided with information such as movie schedules and places to relax.

[0678] Step 16:

[0679] The user checks the showtimes and relaxation information displayed on the smart glasses. Further information can be obtained by asking additional questions or selecting a new object to focus on, as needed.

[0680] This processing flow provides users with a system that allows them to obtain the necessary information efficiently and with reduced stress.

[0681] (Example 2)

[0682] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0683] Conventional information retrieval systems lack the ability to simultaneously process the user's visual focus and voice commands, making it difficult for users to efficiently obtain information. Furthermore, they fail to provide information in a format optimal for the user because they display information uniformly without considering the user's emotional state.

[0684] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0685] In this invention, the server includes image capture means, audio acquisition means, visual information analysis means, audio analysis means, information generation means, display means, and emotion analysis means. This enables information retrieval and display based on the user's visual focus and audio instructions. Furthermore, it can provide information in an appropriate format according to the user's emotional state.

[0686] "Image capture means" refers to a device or function for capturing an image of an object that the user is focusing on.

[0687] "Voice acquisition means" refers to a device or function for acquiring voice commands from a user.

[0688] "Visual information analysis means" refers to a device or function for analyzing captured image data and identifying text and objects within the image.

[0689] "Speech analysis means" refers to a device or function for converting acquired speech data into text and analyzing it.

[0690] "Information generation means" refers to a device or function that generates search queries based on analyzed data, obtains necessary information, and formats it.

[0691] "Display means" refers to a device or function for displaying generated information to the user.

[0692] "Emotional analysis means" refers to a device or function that analyzes the user's emotional state from their facial expressions and voice tone, and adjusts the content and format of the information displayed based on the results.

[0693] This invention is a system including image capture means, sound acquisition means, visual information analysis means, sound analysis means, information generation means, display means, and emotion analysis means. This system automatically acquires information about an object that the user is visually fixated on and searches for and provides appropriate information based on voice instructions. It also has a function to analyze the user's emotional state and adjust the content and format of the information displayed accordingly.

[0694] Hardware and software configuration

[0695] Devices (smart glasses & earphones):

[0696] Smart glasses: They incorporate image capture means (camera), emotion analysis means, and display means.

[0697] Earphones: They have a built-in means of acquiring sound.

[0698] server:

[0699] Visual information analysis method: An image recognition algorithm is used to analyze the captured image data.

[0700] Speech analysis method: A speech recognition algorithm is used to convert the acquired speech into text and analyze it.

[0701] Information generation means: Based on the analyzed data, the necessary information is searched and formatted.

[0702] Program processing

[0703] The process by which an invention is implemented:

[0704] When a user uses the system, they first put on smart glasses and then earphones. Once the system starts up, the smart glasses' camera captures an image of the object the user is looking at. This image data is sent from the terminal to the server, and the earphones also pick up the user's voice commands, which are also sent to the server.

[0705] On the server, received image data is analyzed by a visual information analysis device to identify specific text and objects from the visual information and extract the necessary metadata. Simultaneously, transmitted audio data is analyzed by an audio analysis device to convert audio instructions into text and understand the user's question. Based on this data, an information generation device creates a search query, retrieves and formats the necessary information from the database or the web, and sends it to the terminal.

[0706] The emotion analysis system integrated into the smart glasses analyzes the user's emotional state from their facial expressions and voice tone. Based on the analysis results, it adjusts the content and format of the information displayed, providing information in a format suitable for the user.

[0707] Specific example

[0708] Example 1:

[0709] A user spots a poster for a new movie in the street and asks into their earphones, "Tell me the screening schedule for this movie." The smart glasses' camera captures an image of the poster and sends it to the server. Simultaneously, the earphones capture the voice command and send it to the server.

[0710] Specific actions:

[0711] The server's visual information analysis tool extracts metadata such as "movie title" and "director" from the movie poster.

[0712] The voice analysis system converts the user's voice instructions into text and understands the content of the question.

[0713] The information generation system searches for and formats movie screening schedules.

[0714] The emotion analysis tool recognizes signs of anxiety and stress in the user, adjusts the information display to a concise and easy-to-understand format, and may even provide location information for cafes where users can relax.

[0715] Information is displayed on the smart glasses' screen.

[0716] Example of a prompt:

[0717] "Please tell me about the latest Marvel movie. I'd like to know the screening schedule."

[0718] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0719] Step 1:

[0720] Starting up and preparing the device

[0721] The device (smart glasses & earphones) is activated. This prepares the smart glasses' camera and display, and the earphones' audio acquisition capabilities.

[0722] Specific operation: When the smart glasses are powered on, the built-in camera's capture function and the display become ready. At the same time, the earphones are paired and ready to receive voice input.

[0723] Input: Turn on the smart glasses and pair the earphones.

[0724] Output: Camera and display ready, audio acquisition method ready.

[0725] Step 2:

[0726] User instruction input

[0727] The user wears smart glasses and, while looking at an object in front of them, gives voice instructions into the earphones indicating that they want to obtain information.

[0728] Specific action: The user looks at a poster for a new movie while walking down the street and says aloud, "Tell me the screening schedule for this movie."

[0729] Input: User's gaze, voice commands.

[0730] Output: Capture of visual information, acquisition of audio information.

[0731] Step 3:

[0732] Data acquisition and transmission to the server

[0733] The device (smart glasses & earphones) captures images of objects the user is looking at using its camera and receives voice commands through the earphones. This data is then sent to the server.

[0734] Specific operation: The smart glasses' camera captures an image of a movie poster and sends the data to the server. Simultaneously, the earphones capture the user's voice command, "Tell me the showtimes for this movie," and send it to the server.

[0735] Input: Captured image data, acquired audio data.

[0736] Output: Sending image data and audio data to the server.

[0737] Step 4:

[0738] Data analysis on the server

[0739] The server analyzes the received image data using visual information analysis tools to identify text and objects within the image and extract metadata. It also analyzes audio data using audio analysis tools, converts the audio data into text, and understands the user's questions and intentions.

[0740] Specific operation: The server's visual information analysis means extracts metadata such as "movie title," "director," and "lead actor" from the movie poster. The voice analysis means converts the user's voice command into text such as "I want to know the screening schedule for this movie" and understands the content of the question.

[0741] Input: Image data, audio data.

[0742] Output: Extracted metadata, parsed text data.

[0743] Step 5:

[0744] Generating search queries and retrieving information

[0745] The server's information generation mechanism generates search queries based on extracted metadata and voice commands. These queries are used to search for the necessary information in databases and on the web, and the retrieved information is then formatted.

[0746] Specific operation: The server generates a search query containing the "movie title" and "showtimes," and searches for showtimes from the movie's official website and other databases. The retrieved information is then formatted in a format suitable for the user.

[0747] Input: Metadata, parsed text data.

[0748] Output: Generated search query, formatted information.

[0749] Step 6:

[0750] Sentiment analysis and information format adjustment

[0751] The device's (smart glasses') emotion analysis mechanism analyzes the user's emotional state from their facial expressions and voice tone. Based on the analysis results, it adjusts the content and format of the information provided.

[0752] Specific operation: The emotion analysis system recognizes signs of anxiety and stress from the user's facial expressions and voice tone, and adjusts the displayed content to a concise and simple format. In addition, it provides location information for cafes where users can relax.

[0753] Input: Formatted information, user's facial expressions, and voice tone.

[0754] Output: Adjusted information format, additional relaxation information.

[0755] Step 7:

[0756] Information display and user feedback

[0757] The adjusted information is displayed on the smart glasses' screen via the device's display mechanism. The user can review the displayed information and, if necessary, ask further questions or gather additional information.

[0758] Specific operation: The smart glasses display concisely shows the "movie title," "showtimes," "map of the nearest theater," and "information on relaxing cafes." The user can then check the location of the nearest theater and go to see a movie.

[0759] Input: Adjusted information format, additional relaxation information.

[0760] Output: Information displayed on the smart glasses' screen.

[0761] (Application Example 2)

[0762] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0763] In traditional in-store shopping experiences, it has been difficult for customers to quickly and accurately obtain product information. Furthermore, there was no system that provided information while considering the customer's emotional state, posing a challenge to improving customer satisfaction. Additionally, the lack of quick information access in physical stores sometimes caused customer stress. To address these challenges, a system is needed that accurately analyzes customer visual information and voice commands to provide product information in real time.

[0764] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0765] In this invention, the server includes visual information analysis means, voice analysis means, information generation means, and emotion analysis means. This enables image capture based on the customer's gaze, accurate analysis of voice instructions, real-time product information provision, and information provision tailored to the customer's emotional state.

[0766] An "image capture means" is a device for acquiring an image of an object that the user is focusing on.

[0767] A "voice acquisition means" is a device for acquiring voice commands from a user.

[0768] A "visual information analysis device" is a device for analyzing acquired image data and extracting necessary metadata.

[0769] A "voice analysis device" is a device that analyzes acquired voice data to understand the user's intent and the content of their questions.

[0770] An "information generation means" is a device that generates search queries based on visual and auditory information, and searches for and generates necessary information.

[0771] A "display means" is a device for providing generated information to the user visually.

[0772] An "emotion analysis device" is a device that analyzes the user's emotional state from acquired image and audio data and adjusts the content and format of the information displayed based on the results.

[0773] A "physical store" is a store that has a physical presence and sells products directly to customers.

[0774] "Smart glasses" are glasses-type devices worn by the user that have the function of capturing and displaying images.

[0775] A "search query" is a statement of inquiry generated for the purpose of retrieving information.

[0776] This invention is a system that uses smart glasses and earphones to automatically search for information by combining visual information and voice instructions, and provides the results to the user. Furthermore, by incorporating emotion analysis means, it has the function of recognizing the user's emotional state and adjusting the content and format of the information displayed based on that state.

[0777] System Configuration

[0778] 1. Device (smart glasses & earphones)

[0779] Smart glasses incorporate image capture and display mechanisms.

[0780] The earphones have a built-in mechanism for acquiring sound.

[0781] 2. Server

[0782] The server is equipped with visual information analysis means, audio analysis means, information generation means, and emotion analysis means, all of which work together in coordination.

[0783] Explanation of the program's processing

[0784] When the device is activated, the smart glasses' camera captures an image of the object the user is looking at. This image is sent to the server by the device. Simultaneously, the earphones receive the user's voice commands, and the voice data is also sent to the server.

[0785] The server analyzes and generates data as follows:

[0786] Hardware and software details:

[0787] Smart glasses (image capture and display): Smart glasses have the function of capturing images of objects that the user is looking at and displaying information about them.

[0788] Earphones (voice acquisition): Earphones have the function of acquiring the user's voice commands.

[0789] Server (visual information analysis, audio analysis, information generation, sentiment analysis): The server analyzes and generates data using the following software.

[0790] Visual information analysis method: OpenCV

[0791] Voice analysis method: Google Cloud Speech-to-Text API

[0792] Information generation method: MySQL, Elasticsearch

[0793] Emotion analysis method: Microsoft Azure's Emotion API

[0794] Specifically, the server analyzes the received image data using visual information analysis tools. Specifically, it uses OpenCV to identify text and specific objects within the image and extracts the necessary metadata. Next, the audio data is analyzed using speech analysis tools. The Google Cloud Speech-to-Text API is used to convert the audio data into text, understanding the user's question and intent. This combines the aforementioned metadata with the audio instructions to generate a search query. The information generation tool then uses this search query to retrieve data from MySQL or Elasticsearch and search for and generate the necessary information.

[0795] Specific example

[0796] This scenario describes a user browsing shirts in a physical store and asking a question into their earphones, "Do you have this shirt in my size?" The smart glasses' camera captures an image of the shirt and sends it to the server. The earphones receive the user's voice command and send it to the server.

[0797] On the server, images of shirts are analyzed by a visual information analysis tool (OpenCV) to extract barcodes and product information. Simultaneously, voice commands are analyzed by a speech analysis tool (Google Cloud Speech-to-Text API) to understand the user's questions. Based on this data, information generation tools (MySQL, Elasticsearch) search for and generate shirt size information. An emotion analysis tool (Microsoft Azure's Emotion API) analyzes the user's emotional state from their facial expressions and voice tone, and adjusts the displayed content accordingly.

[0798] The adjusted information is displayed on the smart glasses' screen, providing real-time information such as, "This shirt is available in sizes S, M, and L."

[0799] Example of a prompt

[0800] "Please provide the application prompt for capturing a barcode and displaying product information in real time when a customer is viewing a product using smart glasses."

[0801] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0802] Step 1:

[0803] The user puts on smart glasses and activates the device using earphones.

[0804] Input: Terminal startup command

[0805] Output: Smart glasses and earphones are ready.

[0806] Operation: When the device is powered on, the smart glasses and earphones connect and become ready for use.

[0807] Step 2:

[0808] The smart glasses' camera captures images based on the user's gaze.

[0809] Input: Objects that come into the user's line of sight

[0810] Output: Captured image data

[0811] Operation: The smart glasses' camera acquires an image of the object based on the gaze and sends that data to the device.

[0812] Step 3:

[0813] The earphones receive the user's voice commands.

[0814] Input: User's voice instructions

[0815] Output: Acquired audio data

[0816] Operation: The earphones pick up the user's voice and transmit that audio data to the device.

[0817] Step 4:

[0818] The device sends the captured image and audio data to the server.

[0819] Input: Acquisition of image data and audio data

[0820] Output: Data sent to the server

[0821] Operation: The device transfers image data and audio data acquired by the device to the server.

[0822] Step 5:

[0823] The server analyzes the image data using a visual information analysis tool (OpenCV).

[0824] Input: Image data sent to the server

[0825] Output: Analyzed metadata (e.g., barcode information)

[0826] Operation: The server uses OpenCV to analyze image data and extract the necessary metadata.

[0827] Step 6:

[0828] The server analyzes the audio data using a speech analysis tool (Google Cloud Speech-to-Text API).

[0829] Input: Audio data sent to the server

[0830] Output: Transcripted audio data

[0831] Operation: The server uses the Google Cloud Speech-to-Text API to convert the audio data into text and understand the user's question.

[0832] Step 7:

[0833] The server generates search queries based on visual and audio information and retrieves the information using information generation tools (MySQL, Elasticsearch).

[0834] Input: Analyzed metadata and transcribed audio data

[0835] Output: Search query and generated information

[0836] Operation: The server generates search queries and uses MySQL or Elasticsearch to search for and generate the necessary information.

[0837] Step 8:

[0838] The server analyzes the user's emotional state using an emotion analysis tool (Microsoft Azure's Emotion API).

[0839] Input: Image data and audio data

[0840] Output: Adjustments based on the user's emotional state

[0841] Operation: The server uses Microsoft Azure's Emotion API to analyze the user's emotional state and adjust the displayed content and format accordingly.

[0842] Step 9:

[0843] The server sends the adjusted information to the terminal.

[0844] Input: Adjusted information

[0845] Output: Adjustment information sent to the terminal

[0846] Operation: The server transfers the adjusted information to the terminal.

[0847] Step 10:

[0848] The device displays information on the smart glasses' screen.

[0849] Input: Receiving adjusted information from the terminal

[0850] Output: Information displayed on smart glasses

[0851] Operation: The device displays the information it receives on the smart glasses' screen, providing it to the user visually.

[0852] Example prompt: "Please provide a prompt for an application that captures a barcode and displays product information in real time when a customer is viewing a product using smart glasses."

[0853] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0854] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0855] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0856] [Third Embodiment]

[0857] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0858] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0859] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0860] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0861] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0862] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0863] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0864] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0865] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0866] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0867] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0868] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0869] This invention is a system that uses smart glasses and earphones to combine visual information and voice instructions, automatically retrieves information, and provides the results to the user.

[0870] System Configuration

[0871] 1. Devices (smart glasses & earphones):

[0872] Smart glasses incorporate image capture and display mechanisms.

[0873] The earphones have a built-in mechanism for acquiring sound.

[0874] 2. Server:

[0875] The server is equipped with visual information analysis means, audio analysis means, and information generation means, which work together in coordination.

[0876] Program processing

[0877] When the device is activated, the smart glasses' camera captures an image of the object the user is looking at. This image is sent to the server by the device. Simultaneously, the earphones receive the user's voice commands, and this voice data is also sent to the server.

[0878] The server first analyzes the transmitted image data using visual information analysis tools. Specifically, it uses image recognition algorithms to extract text and specific objects from the image and generate metadata.

[0879] Next, the audio data is analyzed by an audio analysis tool. Using a speech recognition algorithm, the audio data is converted into text, and the user's question and intent are understood based on this text data. This combines the aforementioned metadata and the question content to generate a search query.

[0880] The information generation system uses this search query to search for information in databases and on the web. The information obtained as a search result is formatted into an appropriate format and then sent from the server to the terminal.

[0881] The device (smart glasses) provides the user with received information visually through a display mechanism. The user can confirm the information displayed on the smart glasses' screen.

[0882] Specific example

[0883] Let's consider an example where a user is looking at a poster for a new movie in the street and asks a question into their earphones: "Tell me the showtimes for this movie." The smart glasses' camera captures an image of the poster and sends it to the server. The earphones receive the voice command and also send it to the server.

[0884] On the server, the poster image is analyzed by a visual information analysis device, and the movie title and other metadata are extracted. Simultaneously, the voice instructions are analyzed by a voice analysis device, and the user's question is understood. Based on this data, an information generation device searches for the movie's screening schedule and generates the results.

[0885] Ultimately, information is transmitted from the server to the smart glasses, allowing the user to view the showtimes displayed on the smart glasses' screen. In this way, a system is provided that allows the user to efficiently obtain the necessary information.

[0886] The following describes the processing flow.

[0887] Step 1:

[0888] The server verifies the connection between the smart glasses and earphones and performs the necessary initial setup. This setup involves verifying the UUIDs (Universal Unique Identifiers) of the smart glasses and earphones, and checking the battery status and network connectivity of each device. After confirming everything is working correctly, a notification is sent to the device indicating that the initial setup is complete.

[0889] Step 2:

[0890] The device (smart glasses & earphones) starts up and begins communicating with the server. Upon receiving notification from the server that the initial setup is complete, an alert is displayed on the smart glasses' screen to inform the user that it is ready for use.

[0891] Step 3:

[0892] The user focuses on a movie poster they see in the street. This fixed gaze triggers the smart glasses' camera.

[0893] Step 4:

[0894] The device (smart glasses) activates its camera and captures an image of the poster the user is looking at. The camera automatically focuses and takes a clear image. The image data is stored in temporary memory.

[0895] Step 5:

[0896] The device (smart glasses) compresses and encodes the image data it captures and prepares it for transmission to the server. Once ready, the image data is sent to the server.

[0897] Step 6:

[0898] The user asks into their earphone, "Tell me the showtimes for this movie." The voice command is then acquired.

[0899] Step 7:

[0900] The device (earphones) records the user's voice data and saves it to temporary memory. The recorded data is compressed and encoded, and then prepared to be sent to the server. Once ready, the voice data is sent to the server.

[0901] Step 8:

[0902] The server analyzes the received image data using visual information analysis tools. Image recognition algorithms are used to identify text (movie titles and information) and objects within the image. Necessary metadata (title, location, date and time, etc.) is extracted.

[0903] Step 9:

[0904] The server analyzes the received audio data using an audio analysis tool. An audio recognition algorithm is used to convert the audio data into text. The user's question is analyzed from the converted text data, and relevant queries are generated.

[0905] Step 10:

[0906] The server activates an information generation system based on data obtained from visual information analysis and audio analysis, and generates an answer to the user's question (showtimes). The generated answer data is then converted into a display format.

[0907] Step 11:

[0908] The server sends the formatted response data to the smart glasses.

[0909] Step 12:

[0910] The device (smart glasses) decodes the received response data and prepares it for display on the screen. The screen then displays the showtime schedule information to the user.

[0911] Step 13:

[0912] The user checks the showtimes displayed on their smart glasses. They can continue to obtain information by asking further questions or focusing on other advertisements or signs as needed.

[0913] (Example 1)

[0914] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0915] Conventional information acquisition systems have struggled to implement efficient search functions that combine images and audio. Furthermore, there was a lack of systems that could instantly acquire information when a user gives voice commands about an object they are visually focusing on. Additionally, there was a lack of means to integrate visual and audio information to generate search queries and provide users with appropriate information.

[0916] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0917] In this invention, the server includes visual information analysis means, audio analysis means, search query generation means, and information generation means. This enables the efficient generation of search queries by integrating visual and audio information, allowing the user to instantly obtain necessary information about the object they are focusing on.

[0918] An "image capture means" is a device used to capture an image of an object that the user is focusing on.

[0919] A "voice acquisition means" is a device used to acquire voice commands from a user.

[0920] "Visual information analysis means" refers to an algorithm or system for extracting text or specific objects from captured image data and generating metadata.

[0921] "Voice analysis means" refers to an algorithm or system that converts acquired voice data into text and understands the content and intent of the user's questions.

[0922] "Search query generation means" refers to an algorithm or system for generating queries for information retrieval based on the results of visual information analysis means and audio analysis means.

[0923] "Information generation means" refers to an algorithm or system for searching databases or the web for information based on a search query and formatting it into an appropriate format.

[0924] A "display means" is a device used to visually provide the user with information obtained as search results.

[0925] This invention is a system that combines images of objects the user is focusing on with audio instructions to perform information retrieval and display the results. Specifically, it uses smart glasses and earphones to acquire visual and audio information, which is then analyzed and searched on a server to provide the user with the necessary information. The details of each hardware and software component of this system, as well as examples, are described below.

[0926] System Configuration

[0927] Devices (smart glasses & earphones):

[0928] Smart glasses incorporate both an image capture device (camera) and a display device (display). Commercially available smart glasses are a concrete example of this.

[0929] Earphones have a built-in voice acquisition mechanism (microphone). A concrete example is commercially available Bluetooth earphones.

[0930] server:

[0931] The server is equipped with visual information analysis means, speech analysis means, search query generation means, and information generation means. These means are implemented by various algorithms and software.

[0932] System operation

[0933] To use the system, the user puts on smart glasses and earphones and activates the terminal. The smart glasses' camera captures an image of the object the user is looking at, and the earphone's microphone captures the user's voice commands. This data is sent from the terminal to the server, where the following processing takes place.

[0934] Visual Information Analysis: The server analyzes the received image data using visual information analysis tools (e.g., image recognition algorithms) and extracts text and specific objects from the image. This generates metadata.

[0935] Speech information analysis: Next, the speech data is converted into text using speech analysis tools (e.g., speech recognition algorithms) to understand the user's questions and intentions.

[0936] Search query generation: Search queries are generated based on metadata obtained from visual information and audio analysis results.

[0937] Information generation: Using a search query, the information generation system searches for information in databases and on the web. The obtained information is formatted in an appropriate format and sent to the terminal.

[0938] Information display: The device displays information received from the server on the smart glasses' display, which the user can then review.

[0939] Specific example

[0940] For example, consider a scenario where a user is looking at a poster for a new movie while walking down the street. The user speaks into their earphones, "Tell me the showtimes for this movie." The smart glasses' camera captures an image of the poster, and the earphones receive the voice command. This data is sent to a server and processed using the following steps.

[0941] The image data is analyzed on the server, and the movie title and other metadata are extracted.

[0942] Voice instructions are converted into text, and the user's questions are understood.

[0943] A search query is generated based on the extracted metadata and the user's question.

[0944] The information generation system searches for movie screening schedules and formats the results.

[0945] The formatted information is sent to the device and displayed on the smart glasses' screen.

[0946] Example of a prompt

[0947] "Could you tell me the screening schedule for the movie posters I'm keeping an eye on?"

[0948] "Please search for and display reviews for this product."

[0949] In this way, users can efficiently obtain the necessary information and utilize it beneficially in their daily lives and work.

[0950] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0951] Step 1: Start the device

[0952] The user puts on the smart glasses and earphones and activates the system. This puts the smart glasses' camera (image capture means) and the earphones' microphone (voice acquisition means) into standby mode. The input is the user's operation, and the output is the camera and microphone entering standby mode.

[0953] Step 2: Image capture and audio acquisition

[0954] The device uses a camera built into the smart glasses to capture an image of the object the user is looking at. Simultaneously, the microphone in the earphones captures the user's voice commands. The input consists of visual information of the object and the user's voice commands, and the output consists of captured image data and audio data.

[0955] Step 3: Data transmission

[0956] The terminal sends the captured image data and acquired audio data to the server. This process uses data communication. The input is image data and audio data, and the output is this data transferred to the server.

[0957] Step 4: Analyze visual information on the server

[0958] The server analyzes the received image data using visual information analysis tools. Specifically, it uses image recognition algorithms (e.g., image analysis libraries) to extract text and specific objects (e.g., movie titles) from the images. The input is image data, and the output is the extracted metadata.

[0959] Step 5: Analyze the audio information on the server

[0960] The server analyzes the audio data using speech analysis tools. It converts the audio data into text using a speech recognition algorithm (e.g., a speech analysis library) to understand the user's questions and intentions. The input is audio data, and the output is text data.

[0961] Step 6: Generate search queries

[0962] The server generates search queries based on metadata obtained from visual information analysis and audio analysis results. For example, it combines extracted movie titles with the user's question, "showtimes," to form a search query. The input is metadata and text data, and the output is a generated search query.

[0963] Step 7: Information retrieval and formatting

[0964] The server uses information generation tools to search databases and web resources based on the generated search query. It then formats the search results into an appropriate format. For example, movie screening schedules might be presented in a table. The input is a search query, and the output is formatted information.

[0965] Step 8: Submitting Information

[0966] The server sends formatted information to the terminal. The input is formatted information, and the output is information transferred to the terminal.

[0967] Step 9: Displaying information on the device

[0968] The device displays information received from the server on the smart glasses' screen. The user can visually confirm the information displayed on the smart glasses' screen. The input is information sent from the server, and the output is a visual display.

[0969] (Application Example 1)

[0970] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0971] In autonomous vehicles, there is a lack of safe and efficient methods for passengers to obtain real-time information about their destination and surrounding facilities. Existing navigation systems provide visual information, but they lack sufficient real-time integration with the external environment, and there is a need for an effective system to improve passenger convenience.

[0972] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0973] In this invention, the server includes image capture means, voice acquisition means, visual information analysis means, voice analysis means, information generation means, display means, search means, and facility information acquisition means. This makes it possible to capture images of facilities that passengers are looking at using smart glasses and to obtain detailed information and reviews in real time through voice commands.

[0974] An "image capture means" is a device for acquiring an image of an object that the user is focusing on.

[0975] A "voice acquisition means" is a device for acquiring voice commands from a user.

[0976] A "visual information analysis device" is a device that analyzes acquired image data and extracts information about the object.

[0977] A "voice analysis device" is a device that analyzes acquired voice data to understand the user's instructions.

[0978] An "information generation means" is a device for generating necessary information based on analyzed visual and auditory information.

[0979] A "display means" is a device for visually presenting generated information to the user.

[0980] A "search device" is a device that generates appropriate search queries based on acquired visual and audio information and retrieves information from the internet or a database.

[0981] A "facility information acquisition means" is a device that analyzes information obtained through a search means and acquires detailed information about the facility being monitored.

[0982] This invention is a system that uses smart glasses and earphones to combine visual information and voice instructions, automatically retrieves information, and provides the results to the user. This system is particularly useful in autonomous vehicles to allow passengers to obtain real-time information about their destination and surrounding facilities. The system configuration and processing details are described below.

[0983] System Configuration

[0984] Hardware:

[0985] Smart glasses: Equipped with image capture means and display means.

[0986] Earphones: Equipped with a means of acquiring sound.

[0987] The server is equipped with visual information analysis means, speech analysis means, information generation means, search means, and facility information acquisition means.

[0988] software:

[0989] OpenCV: Used for image capture and preprocessing.

[0990] SpeechRecognition: This is a library for performing speech recognition.

[0991] requests: This is a library for performing API communication for image analysis and information retrieval.

[0992] Reproduction steps

[0993] 1. Smart glasses capture images:

[0994] The user observes the external environment through smart glasses. The smart glasses' camera captures an image of the object the user is looking at. This image is temporarily stored within the smart glasses.

[0995] 2. The earphones receive voice commands:

[0996] The user gives voice commands into the earphones. The microphone built into the earphones captures the voice data. This voice data is transmitted to the server via the smart glasses.

[0997] 3. The server analyzes the visual information:

[0998] Image data sent to the server is analyzed by a visual information analysis system. Image processing libraries such as OpenCV are used for this analysis. As a result of the analysis, metadata of the object being viewed is generated.

[0999] 4. The server analyzes the audio data:

[1000] The audio data is analyzed by a speech analysis tool. The SpeechRecognition library is used for this analysis. The audio data is converted into text data, and the user's instructions are understood.

[1001] 5. The server generates the information:

[1002] Based on visual and auditory information, appropriate search queries are generated. The search method retrieves the necessary information via the internet or databases. Specifically, it obtains detailed information, reviews, and business hours related to the target object.

[1003] 6. Smart glasses display information:

[1004] The generated information is transmitted to the smart glasses' display and provided to the user visually. The user can view the information in real time through the smart glasses.

[1005] Specific example

[1006] For example, a passenger might look at a restaurant through smart glasses and ask a question into their earphone: "What's the reputation of this restaurant?" The smart glasses' camera captures an image of the restaurant and sends it to the server. Simultaneously, the earphone receives a voice command and also sends it to the server. The server performs image analysis and voice recognition to retrieve the necessary data. As a result, restaurant reviews and ratings are displayed on the smart glasses, allowing the user to check the information in real time.

[1007] Example of a prompt

[1008] A user looks at a specific establishment (e.g., a restaurant) through smart glasses and asks, "What's the reputation of this restaurant?" into their earphones. The camera in the smart glasses captures an image of the restaurant, and this image is sent to a server. The server performs image analysis (e.g., extracting the restaurant's name) and speech recognition (e.g., transcribing the question into text), and a search query is generated based on this information. Based on the search query, restaurant reviews and ratings are retrieved and displayed on the smart glasses. Finally, the smart glasses display shows the restaurant's review score and some rating comments.

[1009] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1010] Step 1:

[1011] The smart glasses are activated and capture an image of the object the user is looking at.

[1012] Input: Visual information of the object the smart glasses are focusing on.

[1013] Data processing: Image capture means (camera in smart glasses) acquires image data.

[1014] Output: Captured image data

[1015] Step 2:

[1016] The earphones receive the user's voice commands.

[1017] Input: User voice commands

[1018] Data processing: The voice acquisition device (microphone in the earphones) acquires voice data and transmits it to the smart glasses.

[1019] Output: Acquired audio data

[1020] Step 3:

[1021] The smart glasses send the captured image data and acquired audio data to the server.

[1022] Input: Image data, audio data

[1023] Data processing: Data is transmitted from the smart glasses to the server via communication.

[1024] Output: Image data and audio data sent to the server

[1025] Step 4:

[1026] The server analyzes the visual information and generates metadata about the object being focused on.

[1027] Input: Sent image data

[1028] Data processing: Visual information analysis tools (e.g., OpenCV library) perform image recognition and generate metadata for the object.

[1029] Output: Generated metadata (e.g., name and characteristics of the object)

[1030] Step 5:

[1031] The server analyzes the voice data and converts the user's voice instructions into text.

[1032] Input: Sent audio data

[1033] Data processing: A speech analysis tool (e.g., SpeechRecognition library) performs speech recognition and converts the audio data into text.

[1034] Output: Analyzed text data (e.g., user's question)

[1035] Step 6:

[1036] The server generates search queries based on visual and audio information and searches for information on the internet or in databases.

[1037] Input: Generated metadata, parsed text data

[1038] Data processing: The search tool generates appropriate search queries and searches for information in external databases or on the internet.

[1039] Output: Search results (e.g., detailed information and reviews of the target facility)

[1040] Step 7:

[1041] The server feeds the search results back to the smart glasses and displays the information.

[1042] Input: Search Results

[1043] Data processing: The information generation device formats the search results into an appropriate format and transmits them to the smart glasses.

[1044] Output: Detailed information displayed on smart glasses (e.g., facility reviews and ratings)

[1045] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1046] This invention is a system that uses smart glasses and earphones to automatically retrieve information by combining visual information and voice commands, and provides the results to the user. Furthermore, by incorporating an emotion engine, it has the function of recognizing the user's emotional state and adjusting the content and format of the information displayed based on that state.

[1047] System Configuration

[1048] 1. Devices (smart glasses & earphones):

[1049] Smart glasses incorporate image capture capabilities, an emotion engine, and display capabilities.

[1050] The earphones have a built-in mechanism for acquiring sound.

[1051] 2. Server:

[1052] The server is equipped with visual information analysis means, audio analysis means, and information generation means, which work together in coordination.

[1053] Program processing

[1054] When the device is activated, the smart glasses' camera captures an image of the object the user is looking at. This image is sent to the server by the device. Simultaneously, the earphones receive the user's voice commands, and the voice data is also sent to the server.

[1055] The server analyzes the received image data using visual information analysis tools. Specifically, it uses image recognition algorithms to identify text and specific objects within the image and extracts the necessary metadata.

[1056] Next, the audio data is analyzed by an audio analysis tool. A speech recognition algorithm is used to convert the audio data into text, understanding the user's question and intent. This combines the aforementioned metadata and the question to generate a search query.

[1057] The information generation system uses this search query to search for information in databases and on the web. The information obtained as a search result is converted into a format and then sent from the server to the terminal.

[1058] The device (smart glasses) also incorporates an emotion engine that analyzes the user's emotional state from their facial expressions and voice tone. This emotion engine has the function of adjusting the content and format of the generated information based on the analyzed emotional state. For example, if the system recognizes that the user is prone to anxiety, it will simplify the displayed content and apply an interface designed to reduce stress.

[1059] Information is displayed on the smart glasses' screen via a display device. The user can review the information displayed on the smart glasses and, if necessary, ask further questions or gather additional information.

[1060] Specific example

[1061] Let's consider an example where a user is looking at a poster for a new movie in the street and asks a question into their earphones: "Tell me the showtimes for this movie." The smart glasses' camera captures an image of the poster and sends it to the server. The earphones receive the voice command and send it to the server.

[1062] On the server, the poster image is analyzed by a visual information analysis device, and the movie title and other metadata are extracted. Simultaneously, the voice instructions are analyzed by a voice analysis device, and the user's question is understood. Based on this data, an information generation device searches for the movie screening schedule and generates an answer.

[1063] The device's (smart glasses') emotion engine recognizes signs of anxiety when the user is asked a question. Based on this recognition, the displayed information is adjusted to be concise and easy to understand. Additional information to help the user relax (e.g., the location of a relaxing cafe) may also be provided.

[1064] Ultimately, the information is displayed on the smart glasses' screen, allowing the user to efficiently obtain the necessary information.

[1065] The following describes the processing flow.

[1066] Step 1:

[1067] The server verifies the connection between the smart glasses and earphones and performs the necessary initial setup. This setup involves verifying the UUIDs (Universal Unique Identifiers) of the smart glasses and earphones, and checking the battery status and network connectivity of each device. After confirming everything is working correctly, a notification is sent to the device indicating that the initial setup is complete.

[1068] Step 2:

[1069] The device (smart glasses & earphones) starts up and begins communicating with the server. Upon receiving notification from the server that the initial setup is complete, an alert is displayed on the smart glasses' screen to inform the user that it is ready for use.

[1070] Step 3:

[1071] The user focuses on a movie poster they see in the street. This fixed gaze triggers the smart glasses' camera.

[1072] Step 4:

[1073] The device (smart glasses) activates its camera and captures an image of the poster the user is looking at. The camera automatically focuses and takes a clear image. The image data is stored in temporary memory.

[1074] Step 5:

[1075] The device (smart glasses) compresses and encodes the image data it captures and prepares it for transmission to the server. Once ready, the image data is sent to the server.

[1076] Step 6:

[1077] The user asks into their earphone, "Tell me the showtimes for this movie." The voice command is then acquired.

[1078] Step 7:

[1079] The device (earphones) records the user's voice data and saves it to temporary memory. The recorded data is compressed and encoded, and then prepared to be sent to the server. Once ready, the voice data is sent to the server.

[1080] Step 8:

[1081] The server analyzes the received image data using visual information analysis tools. Image recognition algorithms are used to identify text (movie titles and information) and objects within the image. Necessary metadata (title, location, date and time, etc.) is extracted.

[1082] Step 9:

[1083] The server analyzes the received audio data using an audio analysis tool. An audio recognition algorithm is used to convert the audio data into text. The user's question is analyzed from the converted text data, and relevant queries are generated.

[1084] Step 10:

[1085] The server activates an information generation system based on data obtained from visual information analysis and audio analysis, and generates an answer to the user's question (showtimes). The generated answer data is then converted into a display format.

[1086] Step 11:

[1087] The server sends the formatted response data to the smart glasses.

[1088] Step 12:

[1089] The terminal (smart glasses) decodes the received response data and prepares it for display on the display device.

[1090] Step 13:

[1091] The emotion engine built into the device (smart glasses) analyzes the user's facial expressions and voice tone to recognize their emotional state. For example, if the user is feeling anxious, the emotion engine will detect this.

[1092] Step 14:

[1093] Based on the emotional state recognized by the emotion engine, the information generation means adjusts the display content and format of the information it generates. For example, when the user is anxious, the information generation means simplifies the display content and provides additional relaxing content to reduce stress (e.g., information about nearby cafes).

[1094] Step 15:

[1095] The device (smart glasses) displays the adjusted information on its screen. The user is visually provided with information such as movie schedules and places to relax.

[1096] Step 16:

[1097] The user checks the showtimes and relaxation information displayed on the smart glasses. Further information can be obtained by asking additional questions or selecting a new object to focus on, as needed.

[1098] This processing flow provides users with a system that allows them to obtain the necessary information efficiently and with reduced stress.

[1099] (Example 2)

[1100] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1101] Conventional information retrieval systems lack the ability to simultaneously process the user's visual focus and voice commands, making it difficult for users to efficiently obtain information. Furthermore, they fail to provide information in a format optimal for the user because they display information uniformly without considering the user's emotional state.

[1102] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1103] In this invention, the server includes image capture means, audio acquisition means, visual information analysis means, audio analysis means, information generation means, display means, and emotion analysis means. This enables information retrieval and display based on the user's visual focus and audio instructions. Furthermore, it can provide information in an appropriate format according to the user's emotional state.

[1104] "Image capture means" refers to a device or function for capturing an image of an object that the user is focusing on.

[1105] "Voice acquisition means" refers to a device or function for acquiring voice commands from a user.

[1106] "Visual information analysis means" refers to a device or function for analyzing captured image data and identifying text and objects within the image.

[1107] "Speech analysis means" refers to a device or function for converting acquired speech data into text and analyzing it.

[1108] "Information generation means" refers to a device or function that generates search queries based on analyzed data, obtains necessary information, and formats it.

[1109] "Display means" refers to a device or function for displaying generated information to the user.

[1110] "Emotional analysis means" refers to a device or function that analyzes the user's emotional state from their facial expressions and voice tone, and adjusts the content and format of the information displayed based on the results.

[1111] This invention is a system including image capture means, sound acquisition means, visual information analysis means, sound analysis means, information generation means, display means, and emotion analysis means. This system automatically acquires information about an object that the user is visually fixated on and searches for and provides appropriate information based on voice instructions. It also has a function to analyze the user's emotional state and adjust the content and format of the information displayed accordingly.

[1112] Hardware and software configuration

[1113] Devices (smart glasses & earphones):

[1114] Smart glasses: They incorporate image capture means (camera), emotion analysis means, and display means.

[1115] Earphones: They have a built-in means of acquiring sound.

[1116] server:

[1117] Visual information analysis method: An image recognition algorithm is used to analyze the captured image data.

[1118] Speech analysis method: A speech recognition algorithm is used to convert the acquired speech into text and analyze it.

[1119] Information generation means: Based on the analyzed data, the necessary information is searched and formatted.

[1120] Program processing

[1121] The process by which an invention is implemented:

[1122] When a user uses the system, they first put on smart glasses and then earphones. Once the system starts up, the smart glasses' camera captures an image of the object the user is looking at. This image data is sent from the terminal to the server, and the earphones also pick up the user's voice commands, which are also sent to the server.

[1123] On the server, received image data is analyzed by a visual information analysis device to identify specific text and objects from the visual information and extract the necessary metadata. Simultaneously, transmitted audio data is analyzed by an audio analysis device to convert audio instructions into text and understand the user's question. Based on this data, an information generation device creates a search query, retrieves and formats the necessary information from the database or the web, and sends it to the terminal.

[1124] The emotion analysis system integrated into the smart glasses analyzes the user's emotional state from their facial expressions and voice tone. Based on the analysis results, it adjusts the content and format of the information displayed, providing information in a format suitable for the user.

[1125] Specific example

[1126] Example 1:

[1127] A user spots a poster for a new movie in the street and asks into their earphones, "Tell me the screening schedule for this movie." The smart glasses' camera captures an image of the poster and sends it to the server. Simultaneously, the earphones capture the voice command and send it to the server.

[1128] Specific actions:

[1129] The server's visual information analysis tool extracts metadata such as "movie title" and "director" from the movie poster.

[1130] The voice analysis system converts the user's voice instructions into text and understands the content of the question.

[1131] The information generation system searches for and formats movie screening schedules.

[1132] The emotion analysis tool recognizes signs of anxiety and stress in the user, adjusts the information display to a concise and easy-to-understand format, and may even provide location information for cafes where users can relax.

[1133] Information is displayed on the smart glasses' screen.

[1134] Example of a prompt:

[1135] "Please tell me about the latest Marvel movie. I'd like to know the screening schedule."

[1136] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1137] Step 1:

[1138] Starting up and preparing the device

[1139] The device (smart glasses & earphones) is activated. This prepares the smart glasses' camera and display, and the earphones' audio acquisition capabilities.

[1140] Specific operation: When the smart glasses are powered on, the built-in camera's capture function and the display become ready. At the same time, the earphones are paired and ready to receive voice input.

[1141] Input: Turn on the smart glasses and pair the earphones.

[1142] Output: Camera and display ready, audio acquisition method ready.

[1143] Step 2:

[1144] User instruction input

[1145] The user wears smart glasses and, while looking at an object in front of them, gives voice instructions into the earphones indicating that they want to obtain information.

[1146] Specific action: The user looks at a poster for a new movie while walking down the street and says aloud, "Tell me the screening schedule for this movie."

[1147] Input: User's gaze, voice commands.

[1148] Output: Capture of visual information, acquisition of audio information.

[1149] Step 3:

[1150] Data acquisition and transmission to the server

[1151] The device (smart glasses & earphones) captures images of objects the user is looking at using its camera and receives voice commands through the earphones. This data is then sent to the server.

[1152] Specific operation: The smart glasses' camera captures an image of a movie poster and sends the data to the server. Simultaneously, the earphones capture the user's voice command, "Tell me the showtimes for this movie," and send it to the server.

[1153] Input: Captured image data, acquired audio data.

[1154] Output: Sending image data and audio data to the server.

[1155] Step 4:

[1156] Data analysis on the server

[1157] The server analyzes the received image data using visual information analysis tools to identify text and objects within the image and extract metadata. It also analyzes audio data using audio analysis tools, converts the audio data into text, and understands the user's questions and intentions.

[1158] Specific operation: The server's visual information analysis means extracts metadata such as "movie title," "director," and "lead actor" from the movie poster. The voice analysis means converts the user's voice command into text such as "I want to know the screening schedule for this movie" and understands the content of the question.

[1159] Input: Image data, audio data.

[1160] Output: Extracted metadata, parsed text data.

[1161] Step 5:

[1162] Generating search queries and retrieving information

[1163] The server's information generation mechanism generates search queries based on extracted metadata and voice commands. These queries are used to search for the necessary information in databases and on the web, and the retrieved information is then formatted.

[1164] Specific operation: The server generates a search query containing the "movie title" and "showtimes," and searches for showtimes from the movie's official website and other databases. The retrieved information is then formatted in a format suitable for the user.

[1165] Input: Metadata, parsed text data.

[1166] Output: Generated search query, formatted information.

[1167] Step 6:

[1168] Sentiment analysis and information format adjustment

[1169] The device's (smart glasses') emotion analysis mechanism analyzes the user's emotional state from their facial expressions and voice tone. Based on the analysis results, it adjusts the content and format of the information provided.

[1170] Specific operation: The emotion analysis system recognizes signs of anxiety and stress from the user's facial expressions and voice tone, and adjusts the displayed content to a concise and simple format. In addition, it provides location information for cafes where users can relax.

[1171] Input: Formatted information, user's facial expressions, and voice tone.

[1172] Output: Adjusted information format, additional relaxation information.

[1173] Step 7:

[1174] Information display and user feedback

[1175] The adjusted information is displayed on the smart glasses' screen via the device's display mechanism. The user can review the displayed information and, if necessary, ask further questions or gather additional information.

[1176] Specific operation: The smart glasses display concisely shows the "movie title," "showtimes," "map of the nearest theater," and "information on relaxing cafes." The user can then check the location of the nearest theater and go to see a movie.

[1177] Input: Adjusted information format, additional relaxation information.

[1178] Output: Information displayed on the smart glasses' screen.

[1179] (Application Example 2)

[1180] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1181] In traditional in-store shopping experiences, it has been difficult for customers to quickly and accurately obtain product information. Furthermore, there was no system that provided information while considering the customer's emotional state, posing a challenge to improving customer satisfaction. Additionally, the lack of quick information access in physical stores sometimes caused customer stress. To address these challenges, a system is needed that accurately analyzes customer visual information and voice commands to provide product information in real time.

[1182] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[1183] In this invention, the server includes visual information analysis means, voice analysis means, information generation means, and emotion analysis means. This enables image capture based on the customer's gaze, accurate analysis of voice instructions, real-time product information provision, and information provision tailored to the customer's emotional state.

[1184] An "image capture means" is a device for acquiring an image of an object that the user is focusing on.

[1185] A "voice acquisition means" is a device for acquiring voice commands from a user.

[1186] A "visual information analysis device" is a device for analyzing acquired image data and extracting necessary metadata.

[1187] A "voice analysis device" is a device that analyzes acquired voice data to understand the user's intent and the content of their questions.

[1188] An "information generation means" is a device that generates search queries based on visual and auditory information, and searches for and generates necessary information.

[1189] A "display means" is a device for providing generated information to the user visually.

[1190] An "emotion analysis device" is a device that analyzes the user's emotional state from acquired image and audio data and adjusts the content and format of the information displayed based on the results.

[1191] A "physical store" is a store that has a physical presence and sells products directly to customers.

[1192] "Smart glasses" are glasses-type devices worn by the user that have the function of capturing and displaying images.

[1193] A "search query" is a statement of inquiry generated for the purpose of retrieving information.

[1194] This invention is a system that uses smart glasses and earphones to automatically search for information by combining visual information and voice instructions, and provides the results to the user. Furthermore, by incorporating emotion analysis means, it has the function of recognizing the user's emotional state and adjusting the content and format of the information displayed based on that state.

[1195] System Configuration

[1196] 1. Device (smart glasses & earphones)

[1197] Smart glasses incorporate image capture and display mechanisms.

[1198] The earphones have a built-in mechanism for acquiring sound.

[1199] 2. Server

[1200] The server is equipped with visual information analysis means, audio analysis means, information generation means, and emotion analysis means, all of which work together in coordination.

[1201] Explanation of the program's processing

[1202] When the device is activated, the smart glasses' camera captures an image of the object the user is looking at. This image is sent to the server by the device. Simultaneously, the earphones receive the user's voice commands, and the voice data is also sent to the server.

[1203] The server analyzes and generates data as follows:

[1204] Hardware and software details:

[1205] Smart glasses (image capture and display): Smart glasses have the function of capturing images of objects that the user is looking at and displaying information about them.

[1206] Earphones (voice acquisition): Earphones have the function of acquiring the user's voice commands.

[1207] Server (visual information analysis, audio analysis, information generation, sentiment analysis): The server analyzes and generates data using the following software.

[1208] Visual information analysis method: OpenCV

[1209] Voice analysis method: Google Cloud Speech-to-Text API

[1210] Information generation method: MySQL, Elasticsearch

[1211] Emotion analysis method: Microsoft Azure's Emotion API

[1212] Specifically, the server analyzes the received image data using visual information analysis tools. Specifically, it uses OpenCV to identify text and specific objects within the image and extracts the necessary metadata. Next, the audio data is analyzed using speech analysis tools. The Google Cloud Speech-to-Text API is used to convert the audio data into text, understanding the user's question and intent. This combines the aforementioned metadata with the audio instructions to generate a search query. The information generation tool then uses this search query to retrieve data from MySQL or Elasticsearch and search for and generate the necessary information.

[1213] Specific example

[1214] This scenario describes a user browsing shirts in a physical store and asking a question into their earphones, "Do you have this shirt in my size?" The smart glasses' camera captures an image of the shirt and sends it to the server. The earphones receive the user's voice command and send it to the server.

[1215] On the server, images of shirts are analyzed by a visual information analysis tool (OpenCV) to extract barcodes and product information. Simultaneously, voice commands are analyzed by a speech analysis tool (Google Cloud Speech-to-Text API) to understand the user's questions. Based on this data, information generation tools (MySQL, Elasticsearch) search for and generate shirt size information. An emotion analysis tool (Microsoft Azure's Emotion API) analyzes the user's emotional state from their facial expressions and voice tone, and adjusts the displayed content accordingly.

[1216] The adjusted information is displayed on the smart glasses' screen, providing real-time information such as, "This shirt is available in sizes S, M, and L."

[1217] Example of a prompt

[1218] "Please provide the application prompt for capturing a barcode and displaying product information in real time when a customer is viewing a product using smart glasses."

[1219] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1220] Step 1:

[1221] The user puts on smart glasses and activates the device using earphones.

[1222] Input: Terminal startup command

[1223] Output: Smart glasses and earphones are ready.

[1224] Operation: When the device is powered on, the smart glasses and earphones connect and become ready for use.

[1225] Step 2:

[1226] The smart glasses' camera captures images based on the user's gaze.

[1227] Input: Objects that come into the user's line of sight

[1228] Output: Captured image data

[1229] Operation: The smart glasses' camera acquires an image of the object based on the gaze and sends that data to the device.

[1230] Step 3:

[1231] The earphones receive the user's voice commands.

[1232] Input: User's voice instructions

[1233] Output: Acquired audio data

[1234] Operation: The earphones pick up the user's voice and transmit that audio data to the device.

[1235] Step 4:

[1236] The device sends the captured image and audio data to the server.

[1237] Input: Acquisition of image data and audio data

[1238] Output: Data sent to the server

[1239] Operation: The device transfers image data and audio data acquired by the device to the server.

[1240] Step 5:

[1241] The server analyzes the image data using a visual information analysis tool (OpenCV).

[1242] Input: Image data sent to the server

[1243] Output: Analyzed metadata (e.g., barcode information)

[1244] Operation: The server uses OpenCV to analyze image data and extract the necessary metadata.

[1245] Step 6:

[1246] The server analyzes the audio data using a speech analysis tool (Google Cloud Speech-to-Text API).

[1247] Input: Audio data sent to the server

[1248] Output: Transcripted audio data

[1249] Operation: The server uses the Google Cloud Speech-to-Text API to convert the audio data into text and understand the user's question.

[1250] Step 7:

[1251] The server generates search queries based on visual and audio information and retrieves the information using information generation tools (MySQL, Elasticsearch).

[1252] Input: Analyzed metadata and transcribed audio data

[1253] Output: Search query and generated information

[1254] Operation: The server generates search queries and uses MySQL or Elasticsearch to search for and generate the necessary information.

[1255] Step 8:

[1256] The server analyzes the user's emotional state using an emotion analysis tool (Microsoft Azure's Emotion API).

[1257] Input: Image data and audio data

[1258] Output: Adjustments based on the user's emotional state

[1259] Operation: The server uses Microsoft Azure's Emotion API to analyze the user's emotional state and adjust the displayed content and format accordingly.

[1260] Step 9:

[1261] The server sends the adjusted information to the terminal.

[1262] Input: Adjusted information

[1263] Output: Adjustment information sent to the terminal

[1264] Operation: The server transfers the adjusted information to the terminal.

[1265] Step 10:

[1266] The device displays information on the smart glasses' screen.

[1267] Input: Receiving adjusted information from the terminal

[1268] Output: Information displayed on smart glasses

[1269] Operation: The device displays the information it receives on the smart glasses' screen, providing it to the user visually.

[1270] Example prompt: "Please provide a prompt for an application that captures a barcode and displays product information in real time when a customer is viewing a product using smart glasses."

[1271] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1272] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1273] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[1274] [Fourth Embodiment]

[1275] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[1276] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1277] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1278] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[1279] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1280] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1281] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1282] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[1283] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1284] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1285] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1286] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1287] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1288] This invention is a system that uses smart glasses and earphones to combine visual information and voice instructions, automatically retrieves information, and provides the results to the user.

[1289] System Configuration

[1290] 1. Devices (smart glasses & earphones):

[1291] Smart glasses incorporate image capture and display mechanisms.

[1292] The earphones have a built-in mechanism for acquiring sound.

[1293] 2. Server:

[1294] The server is equipped with visual information analysis means, audio analysis means, and information generation means, which work together in coordination.

[1295] Program processing

[1296] When the device is activated, the smart glasses' camera captures an image of the object the user is looking at. This image is sent to the server by the device. Simultaneously, the earphones receive the user's voice commands, and this voice data is also sent to the server.

[1297] The server first analyzes the transmitted image data using visual information analysis tools. Specifically, it uses image recognition algorithms to extract text and specific objects from the image and generate metadata.

[1298] Next, the audio data is analyzed by an audio analysis tool. Using a speech recognition algorithm, the audio data is converted into text, and the user's question and intent are understood based on this text data. This combines the aforementioned metadata and the question content to generate a search query.

[1299] The information generation system uses this search query to search for information in databases and on the web. The information obtained as a search result is formatted into an appropriate format and then sent from the server to the terminal.

[1300] The device (smart glasses) provides the user with received information visually through a display mechanism. The user can confirm the information displayed on the smart glasses' screen.

[1301] Specific example

[1302] Let's consider an example where a user is looking at a poster for a new movie in the street and asks a question into their earphones: "Tell me the showtimes for this movie." The smart glasses' camera captures an image of the poster and sends it to the server. The earphones receive the voice command and also send it to the server.

[1303] On the server, the poster image is analyzed by a visual information analysis device, and the movie title and other metadata are extracted. Simultaneously, the voice instructions are analyzed by a voice analysis device, and the user's question is understood. Based on this data, an information generation device searches for the movie's screening schedule and generates the results.

[1304] Ultimately, information is transmitted from the server to the smart glasses, allowing the user to view the showtimes displayed on the smart glasses' screen. In this way, a system is provided that allows the user to efficiently obtain the necessary information.

[1305] The following describes the processing flow.

[1306] Step 1:

[1307] The server verifies the connection between the smart glasses and earphones and performs the necessary initial setup. This setup involves verifying the UUIDs (Universal Unique Identifiers) of the smart glasses and earphones, and checking the battery status and network connectivity of each device. After confirming everything is working correctly, a notification is sent to the device indicating that the initial setup is complete.

[1308] Step 2:

[1309] The device (smart glasses & earphones) starts up and begins communicating with the server. Upon receiving notification from the server that the initial setup is complete, an alert is displayed on the smart glasses' screen to inform the user that it is ready for use.

[1310] Step 3:

[1311] The user focuses on a movie poster they see in the street. This fixed gaze triggers the smart glasses' camera.

[1312] Step 4:

[1313] The device (smart glasses) activates its camera and captures an image of the poster the user is looking at. The camera automatically focuses and takes a clear image. The image data is stored in temporary memory.

[1314] Step 5:

[1315] The device (smart glasses) compresses and encodes the image data it captures and prepares it for transmission to the server. Once ready, the image data is sent to the server.

[1316] Step 6:

[1317] The user asks into their earphone, "Tell me the showtimes for this movie." The voice command is then acquired.

[1318] Step 7:

[1319] The device (earphones) records the user's voice data and saves it to temporary memory. The recorded data is compressed and encoded, and then prepared to be sent to the server. Once ready, the voice data is sent to the server.

[1320] Step 8:

[1321] The server analyzes the received image data using visual information analysis tools. Image recognition algorithms are used to identify text (movie titles and information) and objects within the image. Necessary metadata (title, location, date and time, etc.) is extracted.

[1322] Step 9:

[1323] The server analyzes the received audio data using an audio analysis tool. An audio recognition algorithm is used to convert the audio data into text. The user's question is analyzed from the converted text data, and relevant queries are generated.

[1324] Step 10:

[1325] The server activates an information generation system based on data obtained from visual information analysis and audio analysis, and generates an answer to the user's question (showtimes). The generated answer data is then converted into a display format.

[1326] Step 11:

[1327] The server sends the formatted response data to the smart glasses.

[1328] Step 12:

[1329] The device (smart glasses) decodes the received response data and prepares it for display on the screen. The screen then displays the showtime schedule information to the user.

[1330] Step 13:

[1331] The user checks the showtimes displayed on their smart glasses. They can continue to obtain information by asking further questions or focusing on other advertisements or signs as needed.

[1332] (Example 1)

[1333] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1334] Conventional information acquisition systems have struggled to implement efficient search functions that combine images and audio. Furthermore, there was a lack of systems that could instantly acquire information when a user gives voice commands about an object they are visually focusing on. Additionally, there was a lack of means to integrate visual and audio information to generate search queries and provide users with appropriate information.

[1335] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1336] In this invention, the server includes visual information analysis means, audio analysis means, search query generation means, and information generation means. This enables the efficient generation of search queries by integrating visual and audio information, allowing the user to instantly obtain necessary information about the object they are focusing on.

[1337] An "image capture means" is a device used to capture an image of an object that the user is focusing on.

[1338] A "voice acquisition means" is a device used to acquire voice commands from a user.

[1339] "Visual information analysis means" refers to an algorithm or system for extracting text or specific objects from captured image data and generating metadata.

[1340] "Voice analysis means" refers to an algorithm or system that converts acquired voice data into text and understands the content and intent of the user's questions.

[1341] "Search query generation means" refers to an algorithm or system for generating queries for information retrieval based on the results of visual information analysis means and audio analysis means.

[1342] "Information generation means" refers to an algorithm or system for searching databases or the web for information based on a search query and formatting it into an appropriate format.

[1343] A "display means" is a device used to visually provide the user with information obtained as search results.

[1344] This invention is a system that combines images of objects the user is focusing on with audio instructions to perform information retrieval and display the results. Specifically, it uses smart glasses and earphones to acquire visual and audio information, which is then analyzed and searched on a server to provide the user with the necessary information. The details of each hardware and software component of this system, as well as examples, are described below.

[1345] System Configuration

[1346] Devices (smart glasses & earphones):

[1347] Smart glasses incorporate both an image capture device (camera) and a display device (display). Commercially available smart glasses are a concrete example of this.

[1348] Earphones have a built-in voice acquisition mechanism (microphone). A concrete example is commercially available Bluetooth earphones.

[1349] server:

[1350] The server is equipped with visual information analysis means, speech analysis means, search query generation means, and information generation means. These means are implemented by various algorithms and software.

[1351] System operation

[1352] To use the system, the user puts on smart glasses and earphones and activates the terminal. The smart glasses' camera captures an image of the object the user is looking at, and the earphone's microphone captures the user's voice commands. This data is sent from the terminal to the server, where the following processing takes place.

[1353] Visual Information Analysis: The server analyzes the received image data using visual information analysis tools (e.g., image recognition algorithms) and extracts text and specific objects from the image. This generates metadata.

[1354] Speech information analysis: Next, the speech data is converted into text using speech analysis tools (e.g., speech recognition algorithms) to understand the user's questions and intentions.

[1355] Search query generation: Search queries are generated based on metadata obtained from visual information and audio analysis results.

[1356] Information generation: Using a search query, the information generation system searches for information in databases and on the web. The obtained information is formatted in an appropriate format and sent to the terminal.

[1357] Information display: The device displays information received from the server on the smart glasses' display, which the user can then review.

[1358] Specific example

[1359] For example, consider a scenario where a user is looking at a poster for a new movie while walking down the street. The user speaks into their earphones, "Tell me the showtimes for this movie." The smart glasses' camera captures an image of the poster, and the earphones receive the voice command. This data is sent to a server and processed using the following steps.

[1360] The image data is analyzed on the server, and the movie title and other metadata are extracted.

[1361] Voice instructions are converted into text, and the user's questions are understood.

[1362] A search query is generated based on the extracted metadata and the user's question.

[1363] The information generation system searches for movie screening schedules and formats the results.

[1364] The formatted information is sent to the device and displayed on the smart glasses' screen.

[1365] Example of a prompt

[1366] "Could you tell me the screening schedule for the movie posters I'm keeping an eye on?"

[1367] "Please search for and display reviews for this product."

[1368] In this way, users can efficiently obtain the necessary information and utilize it beneficially in their daily lives and work.

[1369] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1370] Step 1: Start the device

[1371] The user puts on the smart glasses and earphones and activates the system. This puts the smart glasses' camera (image capture means) and the earphones' microphone (voice acquisition means) into standby mode. The input is the user's operation, and the output is the camera and microphone entering standby mode.

[1372] Step 2: Image capture and audio acquisition

[1373] The device uses a camera built into the smart glasses to capture an image of the object the user is looking at. Simultaneously, the microphone in the earphones captures the user's voice commands. The input consists of visual information of the object and the user's voice commands, and the output consists of captured image data and audio data.

[1374] Step 3: Data transmission

[1375] The terminal sends the captured image data and acquired audio data to the server. This process uses data communication. The input is image data and audio data, and the output is this data transferred to the server.

[1376] Step 4: Analyze visual information on the server

[1377] The server analyzes the received image data using visual information analysis tools. Specifically, it uses image recognition algorithms (e.g., image analysis libraries) to extract text and specific objects (e.g., movie titles) from the images. The input is image data, and the output is the extracted metadata.

[1378] Step 5: Analyze the audio information on the server

[1379] The server analyzes the audio data using speech analysis tools. It converts the audio data into text using a speech recognition algorithm (e.g., a speech analysis library) to understand the user's questions and intentions. The input is audio data, and the output is text data.

[1380] Step 6: Generate search queries

[1381] The server generates search queries based on metadata obtained from visual information analysis and audio analysis results. For example, it combines extracted movie titles with the user's question, "showtimes," to form a search query. The input is metadata and text data, and the output is a generated search query.

[1382] Step 7: Information retrieval and formatting

[1383] The server uses information generation tools to search databases and web resources based on the generated search query. It then formats the search results into an appropriate format. For example, movie screening schedules might be presented in a table. The input is a search query, and the output is formatted information.

[1384] Step 8: Submitting Information

[1385] The server sends formatted information to the terminal. The input is formatted information, and the output is information transferred to the terminal.

[1386] Step 9: Displaying information on the device

[1387] The device displays information received from the server on the smart glasses' screen. The user can visually confirm the information displayed on the smart glasses' screen. The input is information sent from the server, and the output is a visual display.

[1388] (Application Example 1)

[1389] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1390] In autonomous vehicles, there is a lack of safe and efficient methods for passengers to obtain real-time information about their destination and surrounding facilities. Existing navigation systems provide visual information, but they lack sufficient real-time integration with the external environment, and there is a need for an effective system to improve passenger convenience.

[1391] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1392] In this invention, the server includes image capture means, voice acquisition means, visual information analysis means, voice analysis means, information generation means, display means, search means, and facility information acquisition means. This makes it possible to capture images of facilities that passengers are looking at using smart glasses and to obtain detailed information and reviews in real time through voice commands.

[1393] An "image capture means" is a device for acquiring an image of an object that the user is focusing on.

[1394] A "voice acquisition means" is a device for acquiring voice commands from a user.

[1395] A "visual information analysis device" is a device that analyzes acquired image data and extracts information about the object.

[1396] A "voice analysis device" is a device that analyzes acquired voice data to understand the user's instructions.

[1397] An "information generation means" is a device for generating necessary information based on analyzed visual and auditory information.

[1398] A "display means" is a device for visually presenting generated information to the user.

[1399] A "search device" is a device that generates appropriate search queries based on acquired visual and audio information and retrieves information from the internet or a database.

[1400] A "facility information acquisition means" is a device that analyzes information obtained through a search means and acquires detailed information about the facility being monitored.

[1401] This invention is a system that uses smart glasses and earphones to combine visual information and voice instructions, automatically retrieves information, and provides the results to the user. This system is particularly useful in autonomous vehicles to allow passengers to obtain real-time information about their destination and surrounding facilities. The system configuration and processing details are described below.

[1402] System Configuration

[1403] Hardware:

[1404] Smart glasses: Equipped with image capture means and display means.

[1405] Earphones: Equipped with a means of acquiring sound.

[1406] The server is equipped with visual information analysis means, speech analysis means, information generation means, search means, and facility information acquisition means.

[1407] software:

[1408] OpenCV: Used for image capture and preprocessing.

[1409] SpeechRecognition: This is a library for performing speech recognition.

[1410] requests: This is a library for performing API communication for image analysis and information retrieval.

[1411] Reproduction steps

[1412] 1. Smart glasses capture images:

[1413] The user observes the external environment through smart glasses. The smart glasses' camera captures an image of the object the user is looking at. This image is temporarily stored within the smart glasses.

[1414] 2. The earphones receive voice commands:

[1415] The user gives voice commands into the earphones. The microphone built into the earphones captures the voice data. This voice data is transmitted to the server via the smart glasses.

[1416] 3. The server analyzes the visual information:

[1417] Image data sent to the server is analyzed by a visual information analysis system. Image processing libraries such as OpenCV are used for this analysis. As a result of the analysis, metadata of the object being viewed is generated.

[1418] 4. The server analyzes the audio data:

[1419] The audio data is analyzed by a speech analysis tool. The SpeechRecognition library is used for this analysis. The audio data is converted into text data, and the user's instructions are understood.

[1420] 5. The server generates the information:

[1421] Based on visual and auditory information, appropriate search queries are generated. The search method retrieves the necessary information via the internet or databases. Specifically, it obtains detailed information, reviews, and business hours related to the target object.

[1422] 6. Smart glasses display information:

[1423] The generated information is transmitted to the smart glasses' display and provided to the user visually. The user can view the information in real time through the smart glasses.

[1424] Specific example

[1425] For example, a passenger might look at a restaurant through smart glasses and ask a question into their earphone: "What's the reputation of this restaurant?" The smart glasses' camera captures an image of the restaurant and sends it to the server. Simultaneously, the earphone receives a voice command and also sends it to the server. The server performs image analysis and voice recognition to retrieve the necessary data. As a result, restaurant reviews and ratings are displayed on the smart glasses, allowing the user to check the information in real time.

[1426] Example of a prompt

[1427] A user looks at a specific establishment (e.g., a restaurant) through smart glasses and asks, "What's the reputation of this restaurant?" into their earphones. The camera in the smart glasses captures an image of the restaurant, and this image is sent to a server. The server performs image analysis (e.g., extracting the restaurant's name) and speech recognition (e.g., transcribing the question into text), and a search query is generated based on this information. Based on the search query, restaurant reviews and ratings are retrieved and displayed on the smart glasses. Finally, the smart glasses display shows the restaurant's review score and some rating comments.

[1428] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1429] Step 1:

[1430] The smart glasses are activated and capture an image of the object the user is looking at.

[1431] Input: Visual information of the object the smart glasses are focusing on.

[1432] Data processing: Image capture means (camera in smart glasses) acquires image data.

[1433] Output: Captured image data

[1434] Step 2:

[1435] The earphones receive the user's voice commands.

[1436] Input: User voice commands

[1437] Data processing: The voice acquisition device (microphone in the earphones) acquires voice data and transmits it to the smart glasses.

[1438] Output: Acquired audio data

[1439] Step 3:

[1440] The smart glasses send the captured image data and acquired audio data to the server.

[1441] Input: Image data, audio data

[1442] Data processing: Data is transmitted from the smart glasses to the server via communication.

[1443] Output: Image data and audio data sent to the server

[1444] Step 4:

[1445] The server analyzes the visual information and generates metadata about the object being focused on.

[1446] Input: Sent image data

[1447] Data processing: Visual information analysis tools (e.g., OpenCV library) perform image recognition and generate metadata for the object.

[1448] Output: Generated metadata (e.g., name and characteristics of the object)

[1449] Step 5:

[1450] The server analyzes the voice data and converts the user's voice instructions into text.

[1451] Input: Sent audio data

[1452] Data processing: A speech analysis tool (e.g., SpeechRecognition library) performs speech recognition and converts the audio data into text.

[1453] Output: Analyzed text data (e.g., user's question)

[1454] Step 6:

[1455] The server generates search queries based on visual and audio information and searches for information on the internet or in databases.

[1456] Input: Generated metadata, parsed text data

[1457] Data processing: The search tool generates appropriate search queries and searches for information in external databases or on the internet.

[1458] Output: Search results (e.g., detailed information and reviews of the target facility)

[1459] Step 7:

[1460] The server feeds the search results back to the smart glasses and displays the information.

[1461] Input: Search Results

[1462] Data processing: The information generation device formats the search results into an appropriate format and transmits them to the smart glasses.

[1463] Output: Detailed information displayed on smart glasses (e.g., facility reviews and ratings)

[1464] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1465] This invention is a system that uses smart glasses and earphones to automatically retrieve information by combining visual information and voice commands, and provides the results to the user. Furthermore, by incorporating an emotion engine, it has the function of recognizing the user's emotional state and adjusting the content and format of the information displayed based on that state.

[1466] System Configuration

[1467] 1. Devices (smart glasses & earphones):

[1468] Smart glasses incorporate image capture capabilities, an emotion engine, and display capabilities.

[1469] The earphones have a built-in mechanism for acquiring sound.

[1470] 2. Server:

[1471] The server is equipped with visual information analysis means, audio analysis means, and information generation means, which work together in coordination.

[1472] Program processing

[1473] When the device is activated, the smart glasses' camera captures an image of the object the user is looking at. This image is sent to the server by the device. Simultaneously, the earphones receive the user's voice commands, and the voice data is also sent to the server.

[1474] The server analyzes the received image data using visual information analysis tools. Specifically, it uses image recognition algorithms to identify text and specific objects within the image and extracts the necessary metadata.

[1475] Next, the audio data is analyzed by an audio analysis tool. A speech recognition algorithm is used to convert the audio data into text, understanding the user's question and intent. This combines the aforementioned metadata and the question to generate a search query.

[1476] The information generation system uses this search query to search for information in databases and on the web. The information obtained as a search result is converted into a format and then sent from the server to the terminal.

[1477] The device (smart glasses) also incorporates an emotion engine that analyzes the user's emotional state from their facial expressions and voice tone. This emotion engine has the function of adjusting the content and format of the generated information based on the analyzed emotional state. For example, if the system recognizes that the user is prone to anxiety, it will simplify the displayed content and apply an interface designed to reduce stress.

[1478] Information is displayed on the smart glasses' screen via a display device. The user can review the information displayed on the smart glasses and, if necessary, ask further questions or gather additional information.

[1479] Specific example

[1480] Let's consider an example where a user is looking at a poster for a new movie in the street and asks a question into their earphones: "Tell me the showtimes for this movie." The smart glasses' camera captures an image of the poster and sends it to the server. The earphones receive the voice command and send it to the server.

[1481] On the server, the poster image is analyzed by a visual information analysis device, and the movie title and other metadata are extracted. Simultaneously, the voice instructions are analyzed by a voice analysis device, and the user's question is understood. Based on this data, an information generation device searches for the movie screening schedule and generates an answer.

[1482] The device's (smart glasses') emotion engine recognizes signs of anxiety when the user is asked a question. Based on this recognition, the displayed information is adjusted to be concise and easy to understand. Additional information to help the user relax (e.g., the location of a relaxing cafe) may also be provided.

[1483] Ultimately, the information is displayed on the smart glasses' screen, allowing the user to efficiently obtain the necessary information.

[1484] The following describes the processing flow.

[1485] Step 1:

[1486] The server verifies the connection between the smart glasses and earphones and performs the necessary initial setup. This setup involves verifying the UUIDs (Universal Unique Identifiers) of the smart glasses and earphones, and checking the battery status and network connectivity of each device. After confirming everything is working correctly, a notification is sent to the device indicating that the initial setup is complete.

[1487] Step 2:

[1488] The device (smart glasses & earphones) starts up and begins communicating with the server. Upon receiving notification from the server that the initial setup is complete, an alert is displayed on the smart glasses' screen to inform the user that it is ready for use.

[1489] Step 3:

[1490] The user focuses on a movie poster they see in the street. This fixed gaze triggers the smart glasses' camera.

[1491] Step 4:

[1492] The device (smart glasses) activates its camera and captures an image of the poster the user is looking at. The camera automatically focuses and takes a clear image. The image data is stored in temporary memory.

[1493] Step 5:

[1494] The device (smart glasses) compresses and encodes the image data it captures and prepares it for transmission to the server. Once ready, the image data is sent to the server.

[1495] Step 6:

[1496] The user asks into their earphone, "Tell me the showtimes for this movie." The voice command is then acquired.

[1497] Step 7:

[1498] The device (earphones) records the user's voice data and saves it to temporary memory. The recorded data is compressed and encoded, and then prepared to be sent to the server. Once ready, the voice data is sent to the server.

[1499] Step 8:

[1500] The server analyzes the received image data using visual information analysis tools. Image recognition algorithms are used to identify text (movie titles and information) and objects within the image. Necessary metadata (title, location, date and time, etc.) is extracted.

[1501] Step 9:

[1502] The server analyzes the received audio data using an audio analysis tool. An audio recognition algorithm is used to convert the audio data into text. The user's question is analyzed from the converted text data, and relevant queries are generated.

[1503] Step 10:

[1504] The server activates an information generation system based on data obtained from visual information analysis and audio analysis, and generates an answer to the user's question (showtimes). The generated answer data is then converted into a display format.

[1505] Step 11:

[1506] The server sends the formatted response data to the smart glasses.

[1507] Step 12:

[1508] The terminal (smart glasses) decodes the received response data and prepares it for display on the display device.

[1509] Step 13:

[1510] The emotion engine built into the device (smart glasses) analyzes the user's facial expressions and voice tone to recognize their emotional state. For example, if the user is feeling anxious, the emotion engine will detect this.

[1511] Step 14:

[1512] Based on the emotional state recognized by the emotion engine, the information generation means adjusts the display content and format of the information it generates. For example, when the user is anxious, the information generation means simplifies the display content and provides additional relaxing content to reduce stress (e.g., information about nearby cafes).

[1513] Step 15:

[1514] The device (smart glasses) displays the adjusted information on its screen. The user is visually provided with information such as movie schedules and places to relax.

[1515] Step 16:

[1516] The user checks the showtimes and relaxation information displayed on the smart glasses. Further information can be obtained by asking additional questions or selecting a new object to focus on, as needed.

[1517] This processing flow provides users with a system that allows them to obtain the necessary information efficiently and with reduced stress.

[1518] (Example 2)

[1519] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1520] Conventional information retrieval systems lack the ability to simultaneously process the user's visual focus and voice commands, making it difficult for users to efficiently obtain information. Furthermore, they fail to provide information in a format optimal for the user because they display information uniformly without considering the user's emotional state.

[1521] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1522] In this invention, the server includes image capture means, audio acquisition means, visual information analysis means, audio analysis means, information generation means, display means, and emotion analysis means. This enables information retrieval and display based on the user's visual focus and audio instructions. Furthermore, it can provide information in an appropriate format according to the user's emotional state.

[1523] "Image capture means" refers to a device or function for capturing an image of an object that the user is focusing on.

[1524] "Voice acquisition means" refers to a device or function for acquiring voice commands from a user.

[1525] "Visual information analysis means" refers to a device or function for analyzing captured image data and identifying text and objects within the image.

[1526] "Speech analysis means" refers to a device or function for converting acquired speech data into text and analyzing it.

[1527] "Information generation means" refers to a device or function that generates search queries based on analyzed data, obtains necessary information, and formats it.

[1528] "Display means" refers to a device or function for displaying generated information to the user.

[1529] "Emotional analysis means" refers to a device or function that analyzes the user's emotional state from their facial expressions and voice tone, and adjusts the content and format of the information displayed based on the results.

[1530] This invention is a system including image capture means, sound acquisition means, visual information analysis means, sound analysis means, information generation means, display means, and emotion analysis means. This system automatically acquires information about an object that the user is visually fixated on and searches for and provides appropriate information based on voice instructions. It also has a function to analyze the user's emotional state and adjust the content and format of the information displayed accordingly.

[1531] Hardware and software configuration

[1532] Devices (smart glasses & earphones):

[1533] Smart glasses: They incorporate image capture means (camera), emotion analysis means, and display means.

[1534] Earphones: They have a built-in means of acquiring sound.

[1535] server:

[1536] Visual information analysis method: An image recognition algorithm is used to analyze the captured image data.

[1537] Speech analysis method: A speech recognition algorithm is used to convert the acquired speech into text and analyze it.

[1538] Information generation means: Based on the analyzed data, the necessary information is searched and formatted.

[1539] Program processing

[1540] The process by which an invention is implemented:

[1541] When a user uses the system, they first put on smart glasses and then earphones. Once the system starts up, the smart glasses' camera captures an image of the object the user is looking at. This image data is sent from the terminal to the server, and the earphones also pick up the user's voice commands, which are also sent to the server.

[1542] On the server, received image data is analyzed by a visual information analysis device to identify specific text and objects from the visual information and extract the necessary metadata. Simultaneously, transmitted audio data is analyzed by an audio analysis device to convert audio instructions into text and understand the user's question. Based on this data, an information generation device creates a search query, retrieves and formats the necessary information from the database or the web, and sends it to the terminal.

[1543] The emotion analysis system integrated into the smart glasses analyzes the user's emotional state from their facial expressions and voice tone. Based on the analysis results, it adjusts the content and format of the information displayed, providing information in a format suitable for the user.

[1544] Specific example

[1545] Example 1:

[1546] A user spots a poster for a new movie in the street and asks into their earphones, "Tell me the screening schedule for this movie." The smart glasses' camera captures an image of the poster and sends it to the server. Simultaneously, the earphones capture the voice command and send it to the server.

[1547] Specific actions:

[1548] The server's visual information analysis tool extracts metadata such as "movie title" and "director" from the movie poster.

[1549] The voice analysis system converts the user's voice instructions into text and understands the content of the question.

[1550] The information generation system searches for and formats movie screening schedules.

[1551] The emotion analysis tool recognizes signs of anxiety and stress in the user, adjusts the information display to a concise and easy-to-understand format, and may even provide location information for cafes where users can relax.

[1552] Information is displayed on the smart glasses' screen.

[1553] Example of a prompt:

[1554] "Please tell me about the latest Marvel movie. I'd like to know the screening schedule."

[1555] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1556] Step 1:

[1557] Starting up and preparing the device

[1558] The device (smart glasses & earphones) is activated. This prepares the smart glasses' camera and display, and the earphones' audio acquisition capabilities.

[1559] Specific operation: When the smart glasses are powered on, the built-in camera's capture function and the display become ready. At the same time, the earphones are paired and ready to receive voice input.

[1560] Input: Turn on the smart glasses and pair the earphones.

[1561] Output: Camera and display ready, audio acquisition method ready.

[1562] Step 2:

[1563] User instruction input

[1564] The user wears smart glasses and, while looking at an object in front of them, gives voice instructions into the earphones indicating that they want to obtain information.

[1565] Specific action: The user looks at a poster for a new movie while walking down the street and says aloud, "Tell me the screening schedule for this movie."

[1566] Input: User's gaze, voice commands.

[1567] Output: Capture of visual information, acquisition of audio information.

[1568] Step 3:

[1569] Data acquisition and transmission to the server

[1570] The device (smart glasses & earphones) captures images of objects the user is looking at using its camera and receives voice commands through the earphones. This data is then sent to the server.

[1571] Specific operation: The smart glasses' camera captures an image of a movie poster and sends the data to the server. Simultaneously, the earphones capture the user's voice command, "Tell me the showtimes for this movie," and send it to the server.

[1572] Input: Captured image data, acquired audio data.

[1573] Output: Sending image data and audio data to the server.

[1574] Step 4:

[1575] Data analysis on the server

[1576] The server analyzes the received image data using visual information analysis tools to identify text and objects within the image and extract metadata. It also analyzes audio data using audio analysis tools, converts the audio data into text, and understands the user's questions and intentions.

[1577] Specific operation: The server's visual information analysis means extracts metadata such as "movie title," "director," and "lead actor" from the movie poster. The voice analysis means converts the user's voice command into text such as "I want to know the screening schedule for this movie" and understands the content of the question.

[1578] Input: Image data, audio data.

[1579] Output: Extracted metadata, parsed text data.

[1580] Step 5:

[1581] Generating search queries and retrieving information

[1582] The server's information generation mechanism generates search queries based on extracted metadata and voice commands. These queries are used to search for the necessary information in databases and on the web, and the retrieved information is then formatted.

[1583] Specific operation: The server generates a search query containing the "movie title" and "showtimes," and searches for showtimes from the movie's official website and other databases. The retrieved information is then formatted in a format suitable for the user.

[1584] Input: Metadata, parsed text data.

[1585] Output: Generated search query, formatted information.

[1586] Step 6:

[1587] Sentiment analysis and information format adjustment

[1588] The device's (smart glasses') emotion analysis mechanism analyzes the user's emotional state from their facial expressions and voice tone. Based on the analysis results, it adjusts the content and format of the information provided.

[1589] Specific operation: The emotion analysis system recognizes signs of anxiety and stress from the user's facial expressions and voice tone, and adjusts the displayed content to a concise and simple format. In addition, it provides location information for cafes where users can relax.

[1590] Input: Formatted information, user's facial expressions, and voice tone.

[1591] Output: Adjusted information format, additional relaxation information.

[1592] Step 7:

[1593] Information display and user feedback

[1594] The adjusted information is displayed on the smart glasses' screen via the device's display mechanism. The user can review the displayed information and, if necessary, ask further questions or gather additional information.

[1595] Specific operation: The smart glasses display concisely shows the "movie title," "showtimes," "map of the nearest theater," and "information on relaxing cafes." The user can then check the location of the nearest theater and go to see a movie.

[1596] Input: Adjusted information format, additional relaxation information.

[1597] Output: Information displayed on the smart glasses' screen.

[1598] (Application Example 2)

[1599] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1600] In traditional in-store shopping experiences, it has been difficult for customers to quickly and accurately obtain product information. Furthermore, there was no system that provided information while considering the customer's emotional state, posing a challenge to improving customer satisfaction. Additionally, the lack of quick information access in physical stores sometimes caused customer stress. To address these challenges, a system is needed that accurately analyzes customer visual information and voice commands to provide product information in real time.

[1601] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[1602] In this invention, the server includes visual information analysis means, voice analysis means, information generation means, and emotion analysis means. This enables image capture based on the customer's gaze, accurate analysis of voice instructions, real-time product information provision, and information provision tailored to the customer's emotional state.

[1603] An "image capture means" is a device for acquiring an image of an object that the user is focusing on.

[1604] A "voice acquisition means" is a device for acquiring voice commands from a user.

[1605] A "visual information analysis device" is a device for analyzing acquired image data and extracting necessary metadata.

[1606] A "voice analysis device" is a device that analyzes acquired voice data to understand the user's intent and the content of their questions.

[1607] An "information generation means" is a device that generates search queries based on visual and auditory information, and searches for and generates necessary information.

[1608] A "display means" is a device for providing generated information to the user visually.

[1609] An "emotion analysis device" is a device that analyzes the user's emotional state from acquired image and audio data and adjusts the content and format of the information displayed based on the results.

[1610] A "physical store" is a store that has a physical presence and sells products directly to customers.

[1611] "Smart glasses" are glasses-type devices worn by the user that have the function of capturing and displaying images.

[1612] A "search query" is a statement of inquiry generated for the purpose of retrieving information.

[1613] This invention is a system that uses smart glasses and earphones to automatically search for information by combining visual information and voice instructions, and provides the results to the user. Furthermore, by incorporating emotion analysis means, it has the function of recognizing the user's emotional state and adjusting the content and format of the information displayed based on that state.

[1614] System Configuration

[1615] 1. Device (smart glasses & earphones)

[1616] Smart glasses incorporate image capture and display mechanisms.

[1617] The earphones have a built-in mechanism for acquiring sound.

[1618] 2. Server

[1619] The server is equipped with visual information analysis means, audio analysis means, information generation means, and emotion analysis means, all of which work together in coordination.

[1620] Explanation of the program's processing

[1621] When the device is activated, the smart glasses' camera captures an image of the object the user is looking at. This image is sent to the server by the device. Simultaneously, the earphones receive the user's voice commands, and the voice data is also sent to the server.

[1622] The server analyzes and generates data as follows:

[1623] Hardware and software details:

[1624] Smart glasses (image capture and display): Smart glasses have the function of capturing images of objects that the user is looking at and displaying information about them.

[1625] Earphones (voice acquisition): Earphones have the function of acquiring the user's voice commands.

[1626] Server (visual information analysis, audio analysis, information generation, sentiment analysis): The server analyzes and generates data using the following software.

[1627] Visual information analysis method: OpenCV

[1628] Voice analysis method: Google Cloud Speech-to-Text API

[1629] Information generation method: MySQL, Elasticsearch

[1630] Emotion analysis method: Microsoft Azure's Emotion API

[1631] Specifically, the server analyzes the received image data using visual information analysis tools. Specifically, it uses OpenCV to identify text and specific objects within the image and extracts the necessary metadata. Next, the audio data is analyzed using speech analysis tools. The Google Cloud Speech-to-Text API is used to convert the audio data into text, understanding the user's question and intent. This combines the aforementioned metadata with the audio instructions to generate a search query. The information generation tool then uses this search query to retrieve data from MySQL or Elasticsearch and search for and generate the necessary information.

[1632] Specific example

[1633] This scenario describes a user browsing shirts in a physical store and asking a question into their earphones, "Do you have this shirt in my size?" The smart glasses' camera captures an image of the shirt and sends it to the server. The earphones receive the user's voice command and send it to the server.

[1634] On the server, images of shirts are analyzed by a visual information analysis tool (OpenCV) to extract barcodes and product information. Simultaneously, voice commands are analyzed by a speech analysis tool (Google Cloud Speech-to-Text API) to understand the user's questions. Based on this data, information generation tools (MySQL, Elasticsearch) search for and generate shirt size information. An emotion analysis tool (Microsoft Azure's Emotion API) analyzes the user's emotional state from their facial expressions and voice tone, and adjusts the displayed content accordingly.

[1635] The adjusted information is displayed on the smart glasses' screen, providing real-time information such as, "This shirt is available in sizes S, M, and L."

[1636] Example of a prompt

[1637] "Please provide the application prompt for capturing a barcode and displaying product information in real time when a customer is viewing a product using smart glasses."

[1638] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1639] Step 1:

[1640] The user puts on smart glasses and activates the device using earphones.

[1641] Input: Terminal startup command

[1642] Output: Smart glasses and earphones are ready.

[1643] Operation: When the device is powered on, the smart glasses and earphones connect and become ready for use.

[1644] Step 2:

[1645] The smart glasses' camera captures images based on the user's gaze.

[1646] Input: Objects that come into the user's line of sight

[1647] Output: Captured image data

[1648] Operation: The smart glasses' camera acquires an image of the object based on the gaze and sends that data to the device.

[1649] Step 3:

[1650] The earphones receive the user's voice commands.

[1651] Input: User's voice instructions

[1652] Output: Acquired audio data

[1653] Operation: The earphones pick up the user's voice and transmit that audio data to the device.

[1654] Step 4:

[1655] The device sends the captured image and audio data to the server.

[1656] Input: Acquisition of image data and audio data

[1657] Output: Data sent to the server

[1658] Operation: The device transfers image data and audio data acquired by the device to the server.

[1659] Step 5:

[1660] The server analyzes the image data using a visual information analysis tool (OpenCV).

[1661] Input: Image data sent to the server

[1662] Output: Analyzed metadata (e.g., barcode information)

[1663] Operation: The server uses OpenCV to analyze image data and extract the necessary metadata.

[1664] Step 6:

[1665] The server analyzes the audio data using a speech analysis tool (Google Cloud Speech-to-Text API).

[1666] Input: Audio data sent to the server

[1667] Output: Transcripted audio data

[1668] Operation: The server uses the Google Cloud Speech-to-Text API to convert the audio data into text and understand the user's question.

[1669] Step 7:

[1670] The server generates search queries based on visual and audio information and retrieves the information using information generation tools (MySQL, Elasticsearch).

[1671] Input: Analyzed metadata and transcribed audio data

[1672] Output: Search query and generated information

[1673] Operation: The server generates search queries and uses MySQL or Elasticsearch to search for and generate the necessary information.

[1674] Step 8:

[1675] The server analyzes the user's emotional state using an emotion analysis tool (Microsoft Azure's Emotion API).

[1676] Input: Image data and audio data

[1677] Output: Adjustments based on the user's emotional state

[1678] Operation: The server uses Microsoft Azure's Emotion API to analyze the user's emotional state and adjust the displayed content and format accordingly.

[1679] Step 9:

[1680] The server sends the adjusted information to the terminal.

[1681] Input: Adjusted information

[1682] Output: Adjustment information sent to the terminal

[1683] Operation: The server transfers the adjusted information to the terminal.

[1684] Step 10:

[1685] The device displays information on the smart glasses' screen.

[1686] Input: Receiving adjusted information from the terminal

[1687] Output: Information displayed on smart glasses

[1688] Operation: The device displays the information it receives on the smart glasses' screen, providing it to the user visually.

[1689] Example prompt: "Please provide a prompt for an application that captures a barcode and displays product information in real time when a customer is viewing a product using smart glasses."

[1690] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1691] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1692] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[1693] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1694] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[1695] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[1696] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[1697] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[1698] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[1699] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[1700] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[1701] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[1702] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[1703] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1704] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[1705] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[1706] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[1707] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[1708] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[1709] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[1710] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[1711] The following is further disclosed regarding the embodiments described above.

[1712] (Claim 1)

[1713] Image capture means,

[1714] A means of acquiring sound,

[1715] Visual information analysis means,

[1716] A voice analysis means,

[1717] Information generation means and

[1718] Display means and

[1719] A system that includes this.

[1720] (Claim 2)

[1721] The system according to claim 1, characterized in that the image capture means captures an image of an object that the user is looking at.

[1722] (Claim 3)

[1723] The system according to claim 1, characterized in that the voice acquisition means acquires voice instructions from the user.

[1724] (Claim 4)

[1725] The system according to claim 1, characterized in that the visual information analysis means analyzes captured image data and extracts necessary metadata.

[1726] (Claim 5)

[1727] The system according to claim 1, characterized in that the voice analysis means analyzes the acquired voice data and understands the content of the user's question.

[1728] (Claim 6)

[1729] The system according to claim 1, characterized in that the information generation means generates an answer to a user's question based on data obtained from the visual information analysis means and the voice analysis means.

[1730] (Claim 7)

[1731] The system according to claim 1, characterized in that the display means visually displays the information generated by the information generation means.

[1732] "Example 1"

[1733] (Claim 1)

[1734] Image capture means,

[1735] A means of acquiring sound,

[1736] Visual information analysis means,

[1737] A voice analysis means,

[1738] A means for generating search queries,

[1739] Information generation means and

[1740] Display means and

[1741] A system that includes this.

[1742] (Claim 2)

[1743] The system according to claim 1, characterized in that the image capture means captures an image of an object that the user is looking at, and the voice acquisition means acquires the user's voice instructions.

[1744] (Claim 3)

[1745] The system according to claim 1, characterized in that the visual information analysis means extracts text and specific objects from captured image data and generates metadata.

[1746] "Application Example 1"

[1747] (Claim 1)

[1748] Image capture means,

[1749] A means of acquiring sound,

[1750] Visual information analysis means,

[1751] A voice analysis means,

[1752] Information generation means and

[1753] Display means and

[1754] Search methods,

[1755] Means of obtaining facility information,

[1756] A system that includes this.

[1757] (Claim 2)

[1758] The system according to claim 1, characterized in that the image capture means captures an image of an object that the user is looking at.

[1759] (Claim 3)

[1760] The system according to claim 1, characterized in that the voice acquisition means acquires voice instructions from the user.

[1761] (Claim 4)

[1762] The system according to claim 1, characterized in that the search means generates a search query based on information obtained by the image capture means and the audio acquisition means, and searches for information via the internet or a database.

[1763] (Claim 5)

[1764] The system according to claim 4, characterized in that the facility information acquisition means analyzes the information acquired by the search means and provides the user with detailed information about the facility being monitored.

[1765] "Example 2 of combining an emotion engine"

[1766] (Claim 1)

[1767] Image capture means,

[1768] A means of acquiring sound,

[1769] Visual information analysis means,

[1770] A voice analysis means,

[1771] Information generation means and

[1772] Display means and

[1773] Emotion analysis methods,

[1774] A system that includes this.

[1775] (Claim 2)

[1776] The system according to claim 1, characterized in that the image capture means captures an image of an object that the user is looking at, and the visual information analysis means analyzes the image.

[1777] (Claim 3)

[1778] The system according to claim 1, characterized in that the voice acquisition means acquires voice instructions from the user, and the voice analysis means converts the voice into text and analyzes it.

[1779] (Claim 4)

[1780] The system according to claim 1, characterized in that the emotion analysis means analyzes the user's emotional state and adjusts the display content and format based on the results.

[1781] "Application example 2 when combining with an emotional engine"

[1782] (Claim 1)

[1783] Image capture means,

[1784] A means of acquiring sound,

[1785] Visual information analysis means,

[1786] A voice analysis means,

[1787] Information generation means and

[1788] Display means and

[1789] Emotion analysis methods,

[1790] A system that includes this.

[1791] (Claim 2)

[1792] The system according to claim 1, characterized in that the image capture means captures an image of an object that the user is looking at.

[1793] (Claim 3)

[1794] The system according to claim 1, characterized in that the voice acquisition means acquires voice instructions from the user.

[1795] (Claim 4)

[1796] The system according to claim 1, characterized in that the emotion analysis means analyzes the user's emotional state from acquired image and audio data and adjusts the display content and format based on the analysis results.

[1797] (Claim 5)

[1798] The system according to claim 1, characterized in that the display means displays the adjusted information on the display of smart glasses and provides product information in a physical store. [Explanation of Symbols]

[1799] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

[Claim 1] An image capture means for capturing an image of an object that the user is focusing on, A voice acquisition means for acquiring user voice commands, A visual information analysis means that analyzes captured image data and extracts necessary metadata, A voice analysis method that analyzes acquired voice data to understand the content of the user's questions, Information generation means that generates answers to user questions based on data obtained from the visual information analysis means and the voice analysis means, A display means for visually displaying the information generated by the information generation means, A system that includes this.

Images