System

Abstract

To provide a system.SOLUTION: A system comprises: a feeling engine that recognizes a user's feeling; response generation means that inputs a specific prompt sentence to a generative AI model and generates a response to a request desired by the user in accordance with the feeling; and reproduction means that reproduces information related to the response through an earphone worn by the user, and displays the information on a contact lens having a camera and a display function and worn by the user. The response generation means, when the user is excited upon seeing a landscape, identifies the landscape being seen by the user by analyzing data from the camera that captures the landscape being seen by the user, and generates information related to the identified landscape as information related to the response.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional information acquisition methods often rely solely on either sight or hearing, limiting the user's freedom in acquiring information. Furthermore, information acquisition is primarily done manually, making it difficult to automatically acquire information according to the user's intentions. [Means for solving the problem]

[0005] The system of the present disclosure includes an emotion engine that recognizes a user's emotions, an answer generation means that generates an answer to a request desired by the user in accordance with the emotion by inputting a specific prompt sentence into a generative AI model, and a playback means that plays information related to the answer on earphones worn by the user and also displays it on contact lenses that have a camera and display function and are worn by the user, and when the user is excited by a view, the answer generation means identifies the view the user is looking at by analyzing data from the camera that captured the view the user is looking at, and generates information related to the identified view as information related to the answer. [Brief explanation of the drawings]

[0006] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11]FIG. 2 is a sequence diagram showing a flow of processing in the data processing system according to the first embodiment of the first form example. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Embodiment 1. [Figure 13] FIG. 10 is a sequence diagram showing a processing flow of a data processing system in a second embodiment of the second form example. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Embodiment Example 2. [Figure 15] FIG. 10 is a sequence diagram showing the flow of processing in a data processing system according to a third embodiment of the third embodiment. [Figure 16] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Embodiment 3. [Figure 17] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the first embodiment of the first form example when an emotion engine is combined. [Figure 18] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Form Example 1 when an emotion engine is combined. [Figure 19] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the second embodiment of the second form example when an emotion engine is combined. [Figure 20] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Form Example 2 when an emotion engine is combined. [Figure 21] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the third embodiment of the third form example when an emotion engine is combined. [Figure 22] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Form Example 3 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0007] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0008] First, the terms used in the following description will be explained.

[0009] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, the processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (TENSOR PROCESSING UNIT (registered trademark)).

[0010] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0011] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0012] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0013] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0014] [First embodiment]

[0015] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0016] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0017] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0018] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0019] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0020] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0021] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0022] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0023] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0024] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0025] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0026] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0027] "Example 1"

[0028] One embodiment of the present invention is a system that links earphones and contact lenses. This system allows a user to obtain answers to their desired requests through their ears or eyes. Specifically, when a user makes a voice inquiry, the system generates a response and provides the information through the earphones or visually through the contact lenses. "Embodiment Example 2"

[0029] In the future, contact lenses may be equipped with a camera, allowing the system to recognize the scenery or object the user is looking at and provide relevant information. For example, if the user is looking at a building, the system can provide information such as the building's name and history.

[0030] "Example 3"

[0031] In the future, contact lenses could also be equipped with retinal projection and display capabilities, allowing information to be displayed directly in the user's field of vision. For example, if a user is walking down a street, a map of the street and the route to their destination could be displayed.

[0032] The processing flow of each embodiment will be described below.

[0033] "Example 1"

[0034] Step 1: The user speaks a query.

[0035] Step 2: The system analyzes the user's query and generates an answer to it.

[0036] Step 3: The generated answer is presented to the user either audibly through earphones or visually through contact lenses.

[0037] "Example 2"

[0038] Step 1: The camera mounted on the contact lens captures the scenery or object the user is looking at.

[0039] Step 2: The system analyzes the captured image and generates relevant information.

[0040] Step 3: The generated information is presented to the user either audibly through earphones or visually through contact lenses.

[0041] "Example 3"

[0042] Step 1: The user speaks their destination.

[0043] Step 2: The system calculates the route to the input destination.

[0044] Step 3: The calculated route is displayed directly in the user's field of vision using the contact lens' built-in retinal projection and display capabilities.

[0045] Example 1

[0046] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0047] With conventional information provision systems, it is difficult for users to receive a quick and efficient response when making a voice inquiry. Furthermore, the limited visual means of providing information reduces user convenience. Furthermore, processing voice data and generating responses using generative AI models are complex, and a system that can manage these processes in an integrated manner is needed.

[0048] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0049] In this invention, the server includes a means for a user to make a voice inquiry, a means for a terminal to transmit voice data to the server, a means for the server to convert the voice data into text data, a means for the server to generate an answer using a generative AI model, a means for the server to convert the answer into voice data or image data, a means for the server to transmit the converted data to the terminal, and a means for the terminal to provide information to the user through earphones or contact lenses, thereby enabling the user to quickly and efficiently obtain information by voice or visually.

[0050] The server of the present disclosure includes an emotion engine that recognizes a user's emotions, an answer generation means that generates an answer to a request desired by the user in accordance with the emotion by inputting a specific prompt sentence into a generative AI model, and a playback means that plays information related to the answer on earphones worn by the user and displays it on contact lenses that have a camera and display function and are worn by the user. If the user is excited by a scene, the answer generation means may identify the scene the user is viewing by analyzing data from the camera that captures the scene the user is viewing, and generate information related to the identified scene as information related to the answer. The server may also include means for generating information related to the identified scene, including the history and characteristics of the identified scene. The server may also include means for calculating a route from the user's current location to the user's destination using map information providing means, and displaying the calculated route information on the contact lenses.

[0051] "Means for users to make voice inquiries" refers to a device or interface that allows users to input voice, such as a microphone built into earphones.

[0052] "Means for a terminal to transmit voice data to a server" refers to a device or software that has the function of transmitting voice data to a server via a communication network such as the Internet.

[0053] "Means by which the server converts voice data into text data" refers to software or algorithms that use voice recognition technology to convert voice data into text data.

[0054] "Means by which the server generates an answer using a generative AI model" refers to software or algorithms that use a generative AI model to generate appropriate answers based on text data.

[0055] "Means by which the server converts the response into voice data or image data" refers to software or algorithms for converting the generated text data into voice data or image data.

[0056] "Means for transmitting data converted by the server to the terminal" refers to a device or software that has the function of transmitting converted audio data or image data to the terminal via a communication network such as the Internet.

[0057] "Means by which a terminal provides information to a user through earphones or contact lenses" refers to devices or software that have the function of providing received audio data as audio through earphones or the function of providing image data visually through contact lenses.

[0058] MODE FOR CARRYING OUT THE INVENTION

[0059] This invention is a system in which a user makes a voice inquiry and receives a response via voice or visually through earphones or contact lenses. This system is implemented using the following hardware and software.

[0060] Hardware and Software Configuration

[0061] 1. User terminal: A device equipped with a microphone for user voice input, such as a smartphone or dedicated device.

[0062] 2. Server: A high-performance computer for processing voice data, generating answers using generative AI models, and converting and transmitting data.

[0063] 3. Speech Recognition Software: Software for converting voice data into text data, such as Google® Cloud Speech-to-Text.

[0064] 4. Generative AI models: AI models for generating appropriate answers based on text data, such as OpenAI's GPT-4.

[0065] 5. Speech synthesis software: Software for converting text data into speech data, such as Amazon Polly.

[0066] 6. Contact Lenses: Contact lenses that have a display function to provide visual information.

[0067] System Operation

[0068] When a user asks a question by voice through the microphone built into the earphones, the device captures the voice data and sends it over the internet to a server. The server converts the voice data into text data using Google Cloud Speech-to-Text. The server then uses a generative AI model (GPT-4) to generate an appropriate answer based on the text data.

[0069] The generated answer is converted into either audio or image data, depending on the user's selection. In the case of audio data, the server uses Amazon Polly to convert the text data into audio data. In the case of image data, the server converts the text into a format for visual display. The converted data is then sent back to the device via the Internet.

[0070] The terminal provides the received audio data audibly through earphones or visually through contact lenses, allowing the user to quickly and efficiently obtain information audibly or visually.

[0071] Specific examples

[0072] For example, if a user speaks into the earphones, "What's the weather today?", the device captures the audio and sends it to the server. The server converts the audio to text using Google Cloud Speech-to-Text, generating the text "What's the weather today?". The server then uses GPT-4 to generate the answer "The weather is sunny today." If the user selects the earphones, the server converts the text to speech using Amazon Polly and sends the audio data to the device. The device then provides the audio "The weather is sunny today" to the user through the earphones.

[0073] Example prompt sentence:

[0074] User: "What's the weather like today?"

[0075] Server: "The weather is sunny today."

[0076] The system allows users to obtain information quickly and efficiently, either audibly or visually.

[0077] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0078] Step 1:

[0079] The user makes a voice inquiry.

[0080] The user asks a question by voice through the microphone built into the earphones. For example, "What's the weather like today?" This voice data becomes the input.

[0081] Step 2:

[0082] The terminal transmits the voice data to the server.

[0083] The terminal captures the user's voice data and transmits it to a server via the Internet, where the input is the user's voice data and the output is the voice data transmitted to the server.

[0084] Step 3:

[0085] The server converts the voice data into text data.

[0086] The server uses speech recognition software to convert the received voice data into text data, for example, using Google Cloud Speech-to-Text. In this case, the input is voice data, and the output is text data such as "What's the weather like today?"

[0087] Step 4:

[0088] The server generates an answer using a generative AI model.

[0089] The server uses a generative AI model to generate an appropriate answer based on the text data. For example, OpenAI's GPT-4 is used. In this case, the input is the text data "What's the weather like today?" and the output is the answer text "Today's weather is sunny."

[0090] Step 5:

[0091] The server converts the response into audio or image data.

[0092] The server converts the generated text data into either voice data or image data, depending on the user's selection. For voice data, the server uses speech synthesis software, such as Amazon Polly. For image data, the server converts the text into a format for visual display. In this case, the input is the text data "Today's weather is sunny," and the output is either voice data or image data.

[0093] Step 6:

[0094] The server transmits the converted data to the terminal.

[0095] The server transmits the converted audio data or image data to the terminal via the Internet. At this time, the input is the audio data or image data, and the output is the data transmitted to the terminal.

[0096] Step 7:

[0097] The device provides information to the user through earphones or contact lenses.

[0098] The data received by the device is provided either audibly through earphones or visually through contact lenses. For example, the earphones may say "Today's weather is sunny" or the contact lenses may display the text "Today's weather is sunny." In this case, the input is the data sent from the server, and the output is the information provided to the user.

[0099] (Application example 1)

[0100] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0101] In autonomous vehicles, users are required to quickly and accurately obtain destination and route information. It is also important to grasp traffic conditions and vehicle status in real time and notify users at the appropriate time. However, conventional systems lack efficient means to provide this information, resulting in reduced user convenience. Furthermore, the lack of integration between visual and auditory information provision can cause confusion when users obtain information.

[0102] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0103] In this invention, the server includes earphones and contact lenses, a means for linking them, a means for making a response to a user's desired request available to the user aurally or visually, a means for recognizing the user's voice input and calculating optimal route information using a generative AI model, a means for providing voice guidance of the calculated route information, and a means for visually displaying important information through the contact lenses. This allows the user to quickly and accurately obtain route information through voice input and visually confirm important information.

[0104] An "earphone" is a small acoustic device that delivers sound directly to the user's ears.

[0105] A "contact lens" is a thin lens that is worn on a user's eye to provide visual information.

[0106] "Means for linking" refers to the technical means by which the earphones and contact lenses communicate with each other and share information.

[0107] "Means for making available to the user audibly or visually a response to a desired request" means means for providing information generated based on the user's voice input audibly through earphones or visually through contact lenses.

[0108] "Means for recognizing a user's voice input and calculating optimal route information using a generative AI model" refers to means for analyzing a user's voice and calculating an optimal route using a generative AI model.

[0109] The "means for providing audio guidance of calculated route information" refers to a means for conveying calculated route information to the user by audio.

[0110] "Means for visually displaying important information through contact lenses" refers to means for displaying important information such as traffic conditions and warnings on contact lenses.

[0111] The system for implementing this invention links earphones and contact lenses, and when a user makes a voice inquiry, it uses a generative AI model to calculate optimal route information, provides voice guidance through the earphones, and visually displays important information through the contact lenses.

[0112] Hardware and software used

[0113] Hardware:

[0114] Smartphone or smart glasses

[0115] earphones

[0116] Contact Lens Display

[0117] software:

[0118] speech_recognition library: for performing speech recognition

[0119] pyttsx3 library: for speech synthesis

[0120] The requests library: to obtain route information

[0121] some_contact_lens_display_library: To display information on a contact lens display

[0122] System Operation

[0123] 1. Voice input recognition:

[0124] The user inputs their destination by voice through the microphone attached to their smartphone or smart glasses. The voice data is converted into text using a speech recognition library (speech_recognition).

[0125] 2. Calculating route information:

[0126] The recognized text data is input into a generative AI model to calculate the optimal route information, which is obtained through an external route planning API (using the requests library).

[0127] 3. Voice guidance:

[0128] The calculated route information is converted into voice using a speech synthesis library (pyttsx3) and guided to the user through earphones.

[0129] 4. Visual Indication:

[0130] Display important traffic information and warnings on the contact lens display using some_contact_lens_display_library.

[0131] Specific examples

[0132] When a user voice-inquires, "Tell me the route to Tokyo Station," the system calculates the optimal route and announces through the earphones, "The optimal route has been calculated. We are now departing." The system also displays route information and important traffic information on the contact lens display.

[0133] Prompt Sentence Examples

[0134] "Tell me the route to Tokyo Station."

[0135] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0136] Step 1:

[0137] The user inputs the destination by voice through a microphone attached to the smartphone or smart glasses.

[0138] Input: User's voice data

[0139] Processing: Use the speech recognition library (speech_recognition) to convert the audio data into text.

[0140] Output: Text data (e.g., "Please tell me the route to Tokyo Station.")

[0141] Step 2:

[0142] The server receives the text data and inputs it into the generative AI model.

[0143] Input: Text data

[0144] Processing: Uses a generative AI model to calculate optimal route information.

[0145] Output: Route information data (e.g., optimal route, travel time, traffic conditions, etc.)

[0146] Step 3:

[0147] The server obtains the route information data and converts it into voice using a voice synthesis library (pyttsx3).

[0148] Input: Route information data

[0149] Processing: Convert the route information into speech using a speech synthesis library.

[0150] Output: Voice data (e.g. "Optimal route calculated. Let's go.")

[0151] Step 4:

[0152] The terminal provides the user with audio data through earphones.

[0153] Input: Audio data

[0154] Processing: Play audio data through earphones.

[0155] Output: The user hears the voice prompts.

[0156] Step 5:

[0157] The server displays important traffic information and warnings on the contact lens display.

[0158] Input: Route information data and additional traffic information data

[0159] Processing: Use some_contact_lens_display_library to visually display important information.

[0160] Output: Visual information displayed on the contact lens display (e.g., route arrows, traffic warnings, etc.)

[0161] Step 6:

[0162] The user views visual information through the contact lens display.

[0163] Input: Visual information displayed on a contact lens display

[0164] Action: The user checks the visual information and takes the necessary action.

[0165] Output: User understands visual information and acts upon it

[0166] Example 2

[0167] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0168] In conventional systems, it was difficult to obtain information about the objects and scenery that the user was looking at in real time. Also, the means by which the user could obtain answers to their desired requests by ear or by sight were limited, making it cumbersome to obtain information. Furthermore, there was a lack of a means to efficiently provide information related to the objects and scenery that the user was looking at.

[0169] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for receiving video data from the contact lenses, means for analyzing the received video data and identifying the viewed object or scenery, and means for using a generative AI model for generating information related to the identified object or scenery. This makes it possible to obtain information about the object or scenery the user is viewing in real time and provide it efficiently.

[0170] "Earphones" are small acoustic devices that deliver sound directly to the ears.

[0171] A "contact lens" is a thin, transparent or translucent lens that is worn directly on the eye and is used to correct vision or provide a specific function.

[0172] "Means of cooperation" refers to the technology or method that enables multiple devices or systems to communicate with each other and operate in cooperation.

[0173] "Means for making available to the user audibly or visually a response to a request" refers to techniques or methods for providing the information desired by the user audibly or visually.

[0174] "Means for receiving video data" refers to the technology or method for receiving video information obtained from a camera or sensor.

[0175] "Means for analyzing video data and identifying the object or scene being viewed" refers to techniques or methods for processing received video information and identifying the object or scene being viewed by the user.

[0176] A "means of using a generative AI model" is a technique or method for using artificial intelligence to perform a specific task, and is used here to generate relevant information.

[0177] The "means for providing information to users" refers to the techniques and methods for transmitting the generated information to users.

[0178] MODE FOR CARRYING OUT THE INVENTION

[0179] This invention is a system that incorporates a camera function into contact lenses worn by a user, recognizes the scenery and objects the user is looking at, and provides related information. A specific embodiment of this system is described below.

[0180] Hardware and software used

[0181] Contact Lenses: Contact lenses are equipped with a camera that can capture the scenery and objects the user is looking at in real time.

[0182] Terminal: A device such as a smartphone or smart glasses is used. These terminals receive video data from the contact lenses and send it to the server.

[0183] Server: A server with high-performance computing power is used to analyze the received video data and identify the objects and scenery being viewed.

[0184] Image recognition software: We use image recognition software such as OpenCV and TENSORFLOW (registered trademark), which can analyze video data and identify objects and scenery.

[0185] Generative AI models: Use generative AI models such as GPT-4 to generate information related to identified objects and scenes.

[0186] Program processing

[0187] The server receives the video data sent from the device. The received video data is analyzed using image recognition software. As a result of the analysis, the object or scene the user is looking at is identified. The server then uses a generative AI model to generate information related to the identified object or scene. This information is provided to the user via the device.

[0188] Specific examples

[0189] For example, if a user is looking at a building, the contact lens camera captures video of the building. The video data is sent via the device to a server, which uses image recognition software to identify the building. The server then uses a generative AI model to generate information about the building (e.g., "This building is a historic landmark and was built in the 19th century.") This information is then provided to the user via the device.

[0190] Prompt Sentence Examples

[0191] "Please tell us the name and history of the building you are viewing. The video data for the building is as follows."

[0192] By inputting this prompt into a generative AI model, detailed information about the building the user is looking at can be generated.

[0193] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0194] Program processing flow

[0195] Step 1:

[0196] A user wears the contact lenses and views a scene or object.

[0197] Input: Scenery and objects within the user's field of vision.

[0198] Output: Video data captured by the contact lens camera.

[0199] How it works: When a user wears contact lenses and looks at, say, Tokyo Tower, the camera in the contact lenses captures the image in real time.

[0200] Step 2:

[0201] The terminal receives the video data from the contact lens.

[0202] Input: Video data transmitted from contact lenses.

[0203] Output: Video data stored on the device.

[0204] Specific operation: A device such as a smartphone or smart glasses receives video data of Tokyo Tower transmitted from the contact lenses.

[0205] Step 3:

[0206] The terminal transmits the video data to the server.

[0207] Input: Video data stored on the device.

[0208] Output: Video data sent to the server.

[0209] Specific operation: The device sends the received video data of Tokyo Tower to the server via Wi-Fi or mobile data communication.

[0210] Step 4:

[0211] The server analyzes the video data and identifies the objects and scenery being viewed.

[0212] Input: Video data sent to the server.

[0213] Output: Information about identified objects and scenery.

[0214] Specific operation: The server uses image recognition software such as OpenCV or TensorFlow to analyze the received video data of Tokyo Tower and identify it as Tokyo Tower.

[0215] Step 5:

[0216] The server uses the generative AI model to generate relevant information.

[0217] Input: Identified object and scene information.

[0218] Output: The relevant information generated.

[0219] Specific operation: The server uses a generative AI model such as GPT-4 to generate information about the identified Tokyo Tower (for example, "Tokyo Tower was completed in 1958 and is 333 meters tall").

[0220] Step 6:

[0221] The server transmits the generated information to the terminal.

[0222] Input: Generated relevant information.

[0223] Output: Relevant information sent to the terminal.

[0224] Specific operation: Information about Tokyo Tower generated by the server is sent to the device via Wi-Fi or mobile data communication.

[0225] Step 7:

[0226] The terminal provides the information to the user.

[0227] Input: Relevant information sent to the terminal.

[0228] Output: Information provided to the user.

[0229] Specific operation: The information about Tokyo Tower received by the device is displayed to the user on the screen of their smartphone or smart glasses, allowing the user to check detailed information about Tokyo Tower in real time.

[0230] (Application example 2)

[0231] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0232] Conventional systems have had difficulty obtaining information about the object a user is looking at in real time. Furthermore, there are limited ways for users to obtain answers to their requests by ear or sight, which hinders the user experience. Furthermore, there is a lack of ways to instantly provide detailed product information in physical stores.

[0233] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0234] In this invention, the server includes earphones and contact lenses, a means for linking them, a means for making a response to a user's desired request available to the user's ears or eyes, a means for recognizing an object the user is looking at and providing information about the object, and a means for displaying the information in real time. This allows the user to obtain information about the object they are looking at in real time and to obtain a response to a user's desired request to the user's ears or eyes. Furthermore, detailed product information can be provided instantly in a physical store, improving the user experience.

[0235] "Earphones" are small audio devices that deliver sound directly to the ears.

[0236] A "contact lens" is a thin lens that is placed directly on the surface of the eye and is used for vision correction, decoration, or to provide a specific function.

[0237] "Means of cooperation" are methods and technologies that allow multiple devices or systems to exchange information with each other and work in cooperation.

[0238] "Means for providing audible or visual responses to user requests" refers to techniques or methods that provide audio or visual responses to information or instructions requested by users.

[0239] "Means for recognizing an object that a user is looking at and providing information about that object" refers to a technology or method for identifying an object that comes into the user's field of vision and obtaining and providing data or information related to that object.

[0240] The "means for displaying the information in real time" refers to a technique or method for visually presenting acquired information to a user immediately without delay.

[0241] A system for implementing the present invention includes earphones, contact lenses, a smartphone or smart glasses, a server, and software for linking these components. Specific embodiments are described below.

[0242] Hardware and software used

[0243] Earphone: A small acoustic device that delivers sound to the user.

[0244] Contact lenses: Thin lenses with built-in camera functions.

[0245] Smartphone or smart glasses: A device that performs image recognition and displays information.

[0246] Server: A central system that manages databases and provides information.

[0247] Image recognition software: Google Cloud Vision API, etc.

[0248] Database management system: MySQL (registered trademark), etc.

[0249] Information delivery system: Custom web application.

[0250] Program processing

[0251] The server includes the following means:

[0252] 1. Image capture means: The contact lenses capture objects that come into the user's field of view.

[0253] 2. Image transmission means: Transmit the captured image to a smartphone or smart glasses.

[0254] 3. Image Recognition: Image recognition software on a smartphone or smart glasses analyzes the image and identifies the object.

[0255] 4. Database query means: Retrieving information about the identified object from a database.

[0256] 5. Information provision means: The acquired information is displayed in real time on the user's smartphone or smart glasses.

[0257] Specific examples

[0258] Let's say a user is looking at a particular shampoo in a brick-and-mortar store. The contact lens captures an image of the shampoo and sends it to their smartphone. The Google Cloud Vision API analyzes the image and identifies the shampoo. It retrieves information about the shampoo (price, ingredients, reviews, etc.) from a MySQL database and displays it on their smartphone.

[0259] Prompt Sentence Examples

[0260] Design an application that identifies the product a user is looking at and provides detailed information about that product in real time. The hardware required is a contact lens with a built-in camera and a smartphone. The software required is Google Cloud Vision API and MySQL. Please explain the specific process, including a concrete example where the user is looking at shampoo.

[0261] In this way, users can get real-time information about the object they are looking at, hear or see the answer to their desired request, and instantly get detailed product information in a physical store, improving the user experience.

[0262] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0263] Step 1:

[0264] Image Capture

[0265] A user wears contact lenses and browses products in a physical store. The contact lens's camera captures the products in the user's field of view.

[0266] Input: Products in the user's field of view

[0267] Output: Image data of the captured product

[0268] How it works: A camera built into the contact lens periodically scans the field of view and captures an image of any products it detects.

[0269] Step 2:

[0270] Image transmission

[0271] The captured image data is transmitted to a smartphone or smart glasses.

[0272] Input: Captured product image data

[0273] Output: Image data sent to a smartphone or smart glasses

[0274] How it works: Image data is sent from the contact lens to a smartphone or smart glasses via Bluetooth or Wi-Fi.

[0275] Step 3:

[0276] Image Recognition

[0277] Image recognition software (such as Google Cloud Vision API) on a smartphone or smart glasses analyzes the image and identifies the product.

[0278] Input: Image data sent to a smartphone or smart glasses

[0279] Output: Identification information of the identified product

[0280] How it works: The Google Cloud Vision API sends image data to the cloud, analyzes it to identify the product, and returns the product's identification information as the analysis result.

[0281] Step 4:

[0282] Database queries

[0283] Based on the identification information of the identified product, the server retrieves information about the product from a database (e.g., MySQL).

[0284] Input: Identification information of the identified product

[0285] Output: Product details (price, ingredients, reviews, etc.)

[0286] What happens: The server queries the MySQL database to retrieve details of products that match the identification information.

[0287] Step 5:

[0288] Information provision

[0289] Detailed information about the acquired products is displayed in real time on the user's smartphone or smart glasses.

[0290] Input: Product details (price, ingredients, reviews, etc.)

[0291] Output: Product details displayed on a smartphone or smart glasses

[0292] Specific operation: The detailed information of the acquired product is visually displayed on the display of the smartphone or smart glasses, and the user can check it.

[0293] In this way, users can get real-time information about the object they are looking at, hear or see the answer to their desired request, and instantly get detailed product information in a physical store, improving the user experience.

[0294] Example 3

[0295] Next, a description will be given of a third embodiment of the third embodiment. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0296] Conventional navigation systems require users to hold a smartphone or other device in their hand to visually check information, which reduces safety and convenience while traveling. Furthermore, detailed information is difficult to grasp using voice guidance alone. To solve these issues, a system that displays information directly in the user's field of vision is needed.

[0297] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0298] In this invention, the server includes means for inputting a request desired by the user, means for transmitting the request to the server, means for the server to obtain the user's location information, means for the server to calculate a route using the map information providing means, means for the server to transmit the route information calculated by the server to the terminal, means for the terminal to analyze the route information and convert it into display data, and means for displaying information in the user's field of view, thereby enabling the user to reach their destination efficiently and safely using the information displayed directly in their field of view.

[0299] The "means for inputting a user's desired request" refers to a device or software that provides an interface for the user to input their destination and necessary information.

[0300] The "means for transmitting the request to the server" refers to a communication device or protocol for transmitting the request input by the user to the server via a network.

[0301] The "means by which the server obtains the user's location information" refers to technology for identifying the user's current location using a GPS device or other location information acquisition device.

[0302] "Means for the server to calculate a route using map information providing means" refers to a technology that uses an API or software that provides map information to calculate the optimal route from the user's current location to the destination.

[0303] The "means for transmitting route information calculated by the server to the terminal" refers to a communication device or protocol for transmitting route information calculated by the server to the user's terminal via a network.

[0304] The "means by which the terminal analyzes route information and converts it into display data" refers to software or hardware that analyzes the received route information and converts it into a data format that can be displayed in the user's field of view.

[0305] "Means for displaying information in the user's field of vision" refers to a device for displaying information directly in the user's field of vision using a display built into a contact lens or retinal projection technology.

[0306] This invention provides a system that displays information directly in the user's field of view. This system includes a series of processes for inputting a user's desired request, calculating optimal route information based on the request, and displaying the information in the user's field of view.

[0307] First, the user inputs a prompt such as "Show me the route to a new restaurant" into the device. This prompt is used when the user wants to know the route to a destination. The device sends this prompt to the server. The server uses a GPS device to obtain the user's current location information. Next, the server uses a general map information API as a means of providing map information to calculate the optimal route from the user's current location to the destination.

[0308] The calculated route information is sent from the server to the device. The device analyzes the received route information and converts it into data for display on the contact lenses. This data is then converted into a format that is displayed directly in the user's field of vision using retinal projection technology. Finally, the display built into the user's contact lenses displays the converted route information directly in the user's field of vision. This allows the user to head to their destination while viewing the route information displayed in their field of vision.

[0309] As a concrete example, consider the case where a user wants to go to a new restaurant. The user types "Show me the route to the new restaurant" into the device. The device sends this prompt to the server. The server obtains the user's current location information and calculates the optimal route using a map information API. The calculation results are sent to the device, which analyzes the information and converts it into data to be displayed on the contact lenses. Finally, the route information is displayed in the user's field of vision, allowing the user to reach their destination without getting lost.

[0310] This system enables the user to reach the destination efficiently and safely by using the information displayed directly in the user's field of view. The flow of the identification process in the third embodiment will be described with reference to FIG.

[0311] Step 1:

[0312] The user types the prompt into the terminal.

[0313] When a user wants to know the route to a destination, the user inputs a prompt sentence into the terminal, such as "Show me the route to a new restaurant." The input prompt sentence is acquired through an input device of the terminal.

[0314] Step 2:

[0315] The terminal sends the prompt to the server.

[0316] The terminal sends the prompt text entered by the user to the server via the network. At this time, the user's current location information is also sent along with the prompt text. The input is the prompt text and location information, and the output is the data to be sent to the server.

[0317] Step 3:

[0318] The server obtains the user's location information.

[0319] The server receives the user's current location information sent from the terminal and uses it for the next process. The input is the location information from the terminal, and the output is the acquired location information.

[0320] Step 4:

[0321] The server calculates the route using the map information providing means.

[0322] The server uses a general map information API to provide map information and calculates the optimal route from the user's current location to the destination. The input is location information and destination information, and the output is the calculated route information.

[0323] Step 5:

[0324] The server sends the calculated route information to the terminal.

[0325] The server transmits the calculated route information to the user's terminal via the network. The input is the calculated route information, and the output is the data to be transmitted to the terminal.

[0326] Step 6:

[0327] The terminal analyzes the route information and converts it into display data.

[0328] The terminal analyzes the received route information and converts it into data to be displayed on the contact lens. The input is the received route information, and the output is the display data.

[0329] Step 7:

[0330] Display information in the user's field of view.

[0331] The display built into the user's contact lenses displays the converted route information directly in the user's field of vision. The input is display data, and the output is information displayed in the user's field of vision. This allows the user to head to their destination while viewing the route information displayed in their field of vision.

[0332] (Application example 3)

[0333] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0334] To safely and efficiently reach their destinations, autonomous vehicle drivers need to be provided with accurate, real-time visual navigation information. However, conventional navigation systems require drivers to take their eyes off the road, which compromises safety. Furthermore, voice assistant functionality is not fully integrated, requiring the driver to manually operate the system, which also reduces safety. To address these challenges, a system that allows drivers to obtain the necessary information without having to avert their eyes is needed.

[0335] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[0336] In this invention, the server includes earphones and contact lenses, a means for linking them, a means for making responses to user requests available to the ears or eyes, a means for displaying navigation information in the field of view of the driver of the autonomous vehicle when the driver wears the contact lenses, and a means for equipping the driver with a voice assistant function and displaying information in response to the driver's voice commands. This enables the driver to obtain navigation information in real time without averting their eyes, and to reach their destination safely and efficiently.

[0337] An "earphone" is a small acoustic device that transmits sound directly to the ear.

[0338] A "contact lens" is a thin lens that is worn directly on the surface of the eye to correct vision or display information.

[0339] "Means of cooperation" refers to the technology or method that enables multiple devices or systems to communicate with each other and operate in cooperation.

[0340] "Means for making available to the user audio or visual responses to requests" refers to techniques or methods for providing information audibly or visually in response to a user's request.

[0341] "Means for displaying navigation information in the field of vision of the driver of an autonomous vehicle by wearing contact lenses" refers to technologies and methods for visually providing route and traffic information to the destination through contact lenses worn by the driver of an autonomous vehicle.

[0342] "Means for being equipped with a voice assistant function and displaying information in response to the driver's voice commands" refers to technologies and methods for using voice recognition technology to understand the driver's voice instructions and provide the necessary information based on them.

[0343] The following describes an embodiment of the present invention. The system components include earphones, contact lenses, a server, and a smartphone. These devices work together to provide real-time navigation information to the user.

[0344] System Program

[0345] The server implements the system using the following hardware and software.

[0346] Hardware: Smartphone, retinal projection contact lenses, earphones

[0347] Software: OpenCV, pyttsx3, Navigation API

[0348] Program processing explanation

[0349] When the user inputs a destination, the server uses the navigation API to obtain route and traffic information to the destination. The obtained information is overlaid on the view using OpenCV. Furthermore, the voice assistant reads the information aloud using pyttsx3.

[0350] Specifically, when a user enters a destination into their smartphone, the server calls the navigation API to obtain route and traffic information. Next, to display the obtained information on the contact lenses, OpenCV is used to overlay the information on the field of view. At the same time, using pyttsx3, the voice assistant reads out, "The route to your destination is XX. The current traffic conditions are △△."

[0351] Specific examples

[0352] For example, if a user inputs "Tokyo Tower" as their destination, the server will use the navigation API to obtain route and traffic information to "Tokyo Tower." The obtained information is displayed in the user's field of vision through the contact lenses, and the voice assistant will read out, "The route to Tokyo Tower is ____. The current traffic conditions are ____."

[0353] Prompt Sentence Examples

[0354] Enter your destination: Tokyo Tower

[0355] Getting route information...

[0356] Getting traffic information...

[0357] Displaying information in your field of view...

[0358] Your voice assistant is reading out information...

[0359] In this way, the user can obtain real-time navigation information without having to look away, enabling them to reach their destination safely and efficiently.

[0360] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0361] Step 1:

[0362] The user inputs the destination into the smartphone.

[0363] Input: Destination (e.g. Tokyo Tower)

[0364] Output: Destination information

[0365] Specific operation: The user starts the smartphone application and inputs the destination. The input destination information is sent to the server.

[0366] Step 2:

[0367] The server calls the navigation API to obtain route and traffic information.

[0368] Input: Destination information

[0369] Output: Route information, traffic information

[0370] Specific operation: The server sends destination information to the navigation API, obtains route information to the destination and current traffic information, and stores the obtained information within the server.

[0371] Step 3:

[0372] The server processes the route and traffic information it has acquired to display it on the contact lenses.

[0373] Input: Route information, traffic information

[0374] Output: Data for display

[0375] Specific operation: The server uses OpenCV to process the acquired route and traffic information into data for visual display. The processed data is then sent to the contact lens.

[0376] Step 4:

[0377] The contact lenses overlay route and traffic information onto your field of vision.

[0378] Input: Data to display

[0379] Output: Visual navigation information

[0380] How it works: The contact lenses receive display data sent from the server and overlay it on the user's field of view in real time, allowing the user to check navigation information without looking away.

[0381] Step 5:

[0382] The server uses a voice assistant to read out route and traffic information.

[0383] Input: Route information, traffic information

[0384] Output: Voice navigation information

[0385] Specific operation: The server uses pyttsx3 to read out the acquired route and traffic information. The voice assistant tells the user, "The route to your destination is XX. The current traffic conditions are △△."

[0386] Step 6:

[0387] The user requests additional information using voice commands.

[0388] Input: Voice command

[0389] Output: Additional information

[0390] How it works: When a user issues a voice command, the server uses voice recognition technology to interpret the command and provide any additional information needed, such as displaying information in response to a request like "Tell me where the nearest gas station is."

[0391] In this way, the user can obtain real-time navigation information without having to look away, enabling them to reach their destination safely and efficiently.

[0392] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0393] "Example 1"

[0394] The invention, combined with the emotion engine, recognizes a user's emotions and provides information accordingly. Specifically, when a user feels joy, the system recognizes that emotion and provides information that is likely to bring joy to the user. For example, when a user is excited about a music festival, the system recognizes that emotion and provides information about upcoming concerts by the same artist.

[0395] "Example 2"

[0396] The invention, which combines a camera function and an emotion engine, simultaneously recognizes what the user is looking at and the emotion they are feeling, enabling it to provide information accordingly. Specifically, when a user is moved by a beautiful landscape, the system recognizes that emotion and visual information and provides them with places with similarly beautiful scenery. For example, when a user is moved by a sunset by the sea, the system recognizes that emotion and visual information and provides them with nearby famous places for beautiful sunsets.

[0397] "Example 3"

[0398] The invention, which combines retinal projection, display functionality, and an emotion engine, visually provides information according to the user's emotions. Specifically, when the user is confused, the system recognizes that emotion and displays the necessary information directly in the user's field of vision. For example, when the user is lost and confused, the system recognizes that emotion and displays the route to the destination directly in the user's field of vision.

[0399] The processing flow of each embodiment will be described below.

[0400] "Example 1"

[0401] Step 1: Recognize the user's emotions with the emotion engine.

[0402] Step 2: Based on the recognized emotions, select information that is likely to bring joy to the user.

[0403] Step 3: Present the selected information to the user audibly or visually.

[0404] "Example 2"

[0405] Step 1: Obtain the user's visual information using the camera function.

[0406] Step 2: Recognize the user's emotions with the emotion engine.

[0407] Step 3: Select relevant information based on the acquired visual information and perceived emotions.

[0408] Step 4: Present the selected information to the user audibly or visually.

[0409] "Example 3"

[0410] Step 1: Recognize the user's emotions with the emotion engine.

[0411] Step 2: Select the information you need based on the emotions you recognize.

[0412] Step 3: The selected information is displayed directly in the user's field of vision using retinal projection and display functions.

[0413] Example 1

[0414] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0415] Conventional information provision systems have had difficulty in providing prompt and appropriate responses to user requests. Furthermore, they were unable to provide information in response to the user's emotions, limiting the user experience. Furthermore, there was a lack of visual information provision methods, limiting the means by which users could obtain information.

[0416] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0417] In this invention, the server includes means for converting voice data into text data using voice recognition software, means for generating answers based on the text data using a generative AI model, and means for analyzing the user's emotions using an emotion engine and providing information according to the emotions. This enables prompt and appropriate provision of answers to user requests, and also realizes provision of information according to the user's emotions. In addition, by including means for providing information audibly through earphones and means for providing information visually through contact lenses, the means by which users can obtain information are diversified, improving the user experience.

[0418] "Earphones" are acoustic devices that deliver sound directly to the user's ears.

[0419] A "contact lens" is an optical device that is worn on a user's eye to provide visual information.

[0420] "Means for linking" refers to the technical means for enabling the earphones and contact lenses to communicate with each other and share information.

[0421] "Speech recognition software" is a program for converting voice data into text data.

[0422] A "generative AI model" is an artificial intelligence model that generates appropriate answers based on input text data.

[0423] An "emotion engine" is software that analyzes a user's emotions and provides information according to those emotions.

[0424] The "means for providing by voice" refers to a technical means for conveying the generated answer to the user as voice.

[0425] The "visual presentation means" refers to a technical means for displaying the generated answer to the user as visual information.

[0426] "Retinal projection" is a technology that projects information directly onto the user's retina through contact lenses.

[0427] The "display function" is a function that provides visual information to the user through the contact lenses.

[0428] This invention is a system that links earphones and contact lenses to enable a user to obtain answers to requests they desire through their ears or eyes. Specific embodiments of this system will be described below.

[0429] First, the user makes a voice inquiry through the earphone microphone. For example, they might ask, "What events do you recommend for next weekend?" This voice data is sent to the server.

[0430] The server uses speech recognition software (e.g., Google Cloud Speech-to-Text) to convert the user's voice data into text data. It captures the voice signal as digital data, analyzes the data, and converts it into a string of characters.

[0431] The server then uses a generative AI model (e.g., OpenAI GPT-4) to generate appropriate answers based on the text data. For example, for the text "What events do you recommend for next weekend?", the AI ​​model generates an answer such as "There's a music festival in the city next weekend."

[0432] The generated answer is analyzed using an emotion engine (e.g., IBM Watson® Tone Analyzer). The emotion engine recognizes emotions from the user's voice and provides information corresponding to the emotion. For example, if the user expresses joy, the server provides additional information that the user is likely to find joyful in response to the emotion. Specifically, if the server recognizes that the user is excited about a music festival, the server provides additional information about upcoming concerts by the same artist.

[0433] The generated answer is provided either audibly through earphones or visually through contact lenses. For visual information provision, augmented reality (AR) technology (e.g., Microsoft® HoloLens®) is used. For example, if the user is using earphones, the server may announce aloud, "There's a music festival in the city next weekend." If the user is using contact lenses, detailed information about the event is displayed visually.

[0434] As a concrete example, consider a case where a user makes a voice inquiry such as, "What events do you recommend for next weekend?" In this case, the server performs the following process.

[0435] 1. A user asks by voice, "What events do you recommend for next weekend?"

[0436] 2. The server uses voice recognition software to convert the voice data into text data.

[0437] 3. Use a generative AI model to generate appropriate answers based on the text data.

[0438] 4. Use an emotion engine to analyze the user's emotions and provide information according to their emotions.

[0439] 5. Provide generated answers audibly through earphones or visually through contact lenses.

[0440] Example prompt sentence:

[0441] "When a user asks by voice what events are recommended for the upcoming weekend, create a program that generates appropriate responses and provides information based on the user's emotions."

[0442] In this way, a system is realized that provides quick and appropriate responses to user requests.

[0443] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0444] Step 1:

[0445] The user speaks a query into the earphone microphone, for example, "What events do you recommend for next weekend?" The input is the user's voice data, and the output is an audio signal sent to the server through the earphone microphone.

[0446] Step 2:

[0447] The server uses speech recognition software to convert the user's voice data into text data. Specifically, it captures the voice signal as digital data, analyzes the data, and converts it into a string of characters. The input is voice data, and the output is text data.

[0448] Step 3:

[0449] The server uses a generative AI model to generate an appropriate answer based on the text data. For example, for the text "What events do you recommend for next weekend?", the AI ​​model generates an answer such as "There is a music festival in the city next weekend." The input is text data, and the output is the generated answer text.

[0450] Step 4:

[0451] The server uses an emotion engine to analyze the user's emotions. Specifically, it recognizes emotions from the user's voice and adjusts the system to provide information according to the emotion. For example, if the user expresses joy, the server provides additional information that the user is likely to find joyful according to the emotion. The input is the user's voice data and the generated answer text, and the output is an answer text adjusted according to the emotion.

[0452] Step 5:

[0453] The server provides the generated answer either audibly through earphones or visually through contact lenses. Augmented reality technology is used to provide visual information. For example, if the user is using earphones, the server may announce audibly, "There's a music festival in the city next weekend." If the user is using contact lenses, the server may visually display detailed information about the event. The input is the adjusted answer text, and the output is an audio notification or visual information.

[0454] (Application example 1)

[0455] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0456] Conventional information provision systems provide uniform information without considering the user's feelings, resulting in low user satisfaction. Furthermore, the means of providing information—visual and audio—are separate, resulting in an inconsistent user experience. Furthermore, there is a lack of technology to efficiently process users' voice requests and provide appropriate information.

[0457] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0458] In this invention, the server includes earphones and contact lenses, a means for linking them, a means for making a response to a user's desired request available aurally or visually, a means for recognizing the user's emotion and providing information corresponding to that emotion, a means for converting a voice request into text, a means for displaying visual information on the contact lenses, and a means for providing voice information through the earphones. This makes it possible to provide appropriate information according to the user's emotion, and the consistent provision of visual and voice information can improve the user experience.

[0459] An "earphone" is a small acoustic device that transmits sound directly to the user's ears.

[0460] A "contact lens" is a thin lens that displays visual information directly to a user's eye.

[0461] "Means for linking" refers to the technical means by which the earphones and contact lenses communicate with each other and share information.

[0462] "Means for making a response to a user's desired request available to the ears or eyes" refers to means for generating information based on the user's request and providing it through earphones or contact lenses.

[0463] "Means for recognizing a user's emotions and providing information according to those emotions" refers to technical means for analyzing a user's emotional state and providing information appropriate to those emotions.

[0464] A "means for converting a voice request to text" is a technical means for converting a user's voice input into text data.

[0465] The "means for displaying visual information on a contact lens" refers to a technical means for displaying the generated visual information to a user through the contact lens.

[0466] The "means for providing audio information through earphones" refers to the technical means for transmitting the generated audio information to the user through the earphones.

[0467] The following system configuration and processing procedure will be described as an embodiment of the present invention.

[0468] System Configuration

[0469] This system uses the following hardware and software:

[0470] Hardware: smartphones, earphones, smart contact lenses

[0471] Software: Speech recognition library (speech_recognition), emotion recognition library (emotion_recognition), content recommendation engine (content_recommender), visual information provision module (visual_display), audio output module (audio_output)

[0472] Processing flow

[0473] 1. Getting a voice request

[0474] The user makes a voice request through the smartphone's microphone, which is then converted into text using a speech recognition library.

[0475] 2. Emotion recognition

[0476] An emotion recognition library is used to analyze emotions from the user's voice, thereby identifying the user's current emotional state.

[0477] 3. Content Recommendation

[0478] A content recommendation engine is used to select appropriate content based on the user's request and emotional state. For example, if a user requests "Tell me about recent movies" and the emotion is recognized as "joy," the latest comedy movies will be recommended.

[0479] 4. Providing visual information

[0480] A visual information provision module is used to display visual information of selected content (e.g., movie posters and trailers) on the smart contact lenses.

[0481] 5. Provision of audio information

[0482] An audio output module is used to provide an overview and detailed information of the selected content via audio through earphones.

[0483] Specific examples

[0484] If a user makes a voice request such as "Tell me about recent movies," and the system recognizes the user's emotion as "joy," the system will recommend the latest comedy movies. The movie poster will be displayed on the smart contact lenses, and a synopsis of the movie will be spoken through the earphones.

[0485] Prompt Sentence Examples

[0486] User: "What are some recent movies?"

[0487] System: "It seems you're feeling joyful. I recommend the latest comedy, 'The Kingdom of Laughter.' Display the movie poster."

[0488] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0489] Step 1:

[0490] The user makes a voice request through the smartphone's microphone.

[0491] Input: User's voice request

[0492] Processing: The smartphone microphone captures the audio data.

[0493] Output: Audio data

[0494] Step 2:

[0495] The server uses a speech recognition library (speech_recognition) to convert the voice data into text.

[0496] Input: Audio data

[0497] Processing: The speech recognition library analyzes the audio data and generates corresponding text data.

[0498] Output: Text data (user request content)

[0499] Step 3:

[0500] The server uses an emotion recognition library (emotion_recognition) to analyze emotions from the user's voice.

[0501] Input: Audio data

[0502] Processing: An emotion recognition library analyzes the audio data to identify the user's emotional state.

[0503] Output: Emotion data (user's emotional state)

[0504] Step 4:

[0505] The server uses a content recommendation engine (content_recommender) to select appropriate content based on the user's request and emotional state.

[0506] Input: Text data, emotion data

[0507] Processing: The content recommendation engine analyzes the text data and emotional data to select the most appropriate content.

[0508] Output: Recommended content data

[0509] Step 5:

[0510] The server uses a visual information providing module (visual_display) to display visual information of the selected content on the smart contact lens.

[0511] Input: Recommended content data

[0512] Processing: The visual information providing module analyzes the recommended content data, generates visual information, and sends it to the smart contact lens.

[0513] Output: Visual information (what is displayed on the contact lenses)

[0514] Step 6:

[0515] The server uses the audio output module (audio_output) to provide an overview and detailed information of the selected content through the earphones.

[0516] Input: Recommended content data

[0517] Processing: The audio output module analyzes the recommended content data, generates audio information, and sends it to the earphones.

[0518] Output: Audio information (content provided through earphones)

[0519] Example 2

[0520] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0521] Conventional systems have difficulty integrating visual information with emotional responses to provide appropriate information. Furthermore, there is a lack of means to provide real-time information related to the objects and scenery the user is looking at. This has resulted in insufficient information being provided to improve the user experience.

[0522] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0523] In this invention, the server includes means for collecting video data through a camera function installed in the contact lenses, means for collecting user emotion data through an emotion engine, means for analyzing the collected data and identifying an object or scene the user is looking at, means for identifying the user's emotion, means for integrating the visual information and the emotion data and generating related information, and means for providing the generated information to the user. This makes it possible to provide information related to the object or scene the user is looking at in real time, improving the user experience.

[0524] "Earphones" are acoustic devices that deliver sound directly to the user's ears.

[0525] A "contact lens" is a thin lens that is worn on a user's eye and is a device for acquiring visual information.

[0526] "Means for linking" refers to the technical means for enabling the earphones and contact lenses to communicate with each other.

[0527] "Means for making available to the user audibly or visually a response to a user's desired request" refers to means for providing information audibly or visually based on a user's request.

[0528] The "camera function" is a photographing device for collecting video data.

[0529] An "emotion engine" is software or hardware for analyzing a user's emotions.

[0530] "Means for collecting video data" refers to means for acquiring visual information using a camera function.

[0531] The "means for collecting emotion data" is a means for acquiring the user's emotional state using an emotion engine.

[0532] "Means for analyzing collected data and identifying the objects and scenery the user is looking at" refers to means for analyzing video data and identifying the objects the user is visually recognizing.

[0533] The "means for identifying the user's emotion" is a means for analyzing emotion data to identify the user's emotional state.

[0534] "Means for integrating visual information and emotional data to generate related information" refers to means for generating related information by combining the objects or scenery the user is looking at with the emotions they are feeling at the time.

[0535] The "means for providing the generated information to the user" refers to a means for providing the generated information to the user audibly or visually.

[0536] This invention is a system that uses contact lenses and earphones worn by a user to collect and analyze visual information and emotional data of the user. A specific embodiment of this system will be described below.

[0537] First, the user wears contact lenses equipped with a camera function that captures real-time images of the scenery and objects the user is looking at, and then wears earphones to receive audio information.

[0538] The device receives the video data transmitted from the contact lenses. This video data is analyzed using image recognition software (e.g., OpenCV). The analyzed data is used to identify the objects and scenes the user is looking at.

[0539] At the same time, the device collects the user's emotional data using an emotion engine (e.g., Affectiva), which identifies the emotion the user is feeling based on the user's facial expressions and biometric data (heart rate, electrodermal activity, etc.).

[0540] The collected video data and emotion data are sent from the device to a server. The server integrates this data and associates the object or scenery the user is looking at with the emotion at that time. For example, if the user is impressed by a sunset by the sea, the server can use that information to provide nearby spots with beautiful sunsets.

[0541] The generated information is sent from the server to the terminal, which then provides the information to the user via voice or visual means. For example, if the user is excited to see Tokyo Tower, the terminal will display information about the history and sightseeing of Tokyo Tower.

[0542] As a concrete example, consider the case where a user is looking at a certain building. The server analyzes the video data sent from the camera and identifies the building as "Tokyo Tower." At the same time, if the emotion engine recognizes the user's emotion as "excitement," the server provides information about the history and sightseeing of Tokyo Tower.

[0543] Example prompts to input to a generative AI model:

[0544] "If a user is excited to see Tokyo Tower, provide them with historical and tourist information about the building."

[0545] In this way, a system is realized in which the server, terminal, and user work together to provide appropriate information to the user.

[0546] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0547] Step 1:

[0548] A user wears contact lenses to acquire visual information.

[0549] Input: User's visual information

[0550] Output: Image data from contact lenses

[0551] Specific operation: The user wears contact lenses equipped with a camera function and looks at the surrounding scenery and objects.

[0552] Step 2:

[0553] The device collects video data through its camera function.

[0554] Input: Image data from contact lenses

[0555] Output: Video data sent to the device

[0556] Specific operation: The device receives the video data captured by the contact lenses.

[0557] Step 3:

[0558] The device collects the user's emotional data through an emotion engine.

[0559] Input: User's facial expressions and biometric data

[0560] Output: Emotion data from the emotion engine

[0561] Specific operation: The device uses an emotion engine to analyze biometric data such as the user's facial expressions, heart rate, and electrodermal activity to identify the user's emotions.

[0562] Step 4:

[0563] The terminal transmits the collected data to the server.

[0564] Input: Video data, emotion data

[0565] Output: Data sent to the server

[0566] Specific operation: The device sends the collected video data and emotion data to the server.

[0567] Step 5:

[0568] The server analyzes the video data and identifies the objects and scenery the user is looking at.

[0569] Input: Video data

[0570] Output: Information on identified objects and scenery

[0571] Specific operation: The server uses image recognition software (e.g., OpenCV) to analyze the video data and identify the objects and scenery the user is looking at.

[0572] Step 6:

[0573] The server analyzes the emotion data and identifies the user's emotion.

[0574] Input: Emotion data

[0575] Output: Identified user sentiment

[0576] Specific operation: The server uses an emotion engine (e.g., Affectiva) to analyze the emotion data and identify the user's emotion.

[0577] Step 7:

[0578] The server integrates visual information and emotional data to generate related information.

[0579] Input: Information about identified objects and scenery, identified user emotions

[0580] Output: Related information generated

[0581] Specific operation: The server integrates visual information and emotional data and generates relevant information based on the object or scenery the user is looking at and their current emotions.

[0582] Step 8:

[0583] The server transmits the generated information to the terminal.

[0584] Input: Generated related information

[0585] Output: Information sent to the terminal

[0586] Specific operation: The server sends the generated information to the terminal.

[0587] Step 9:

[0588] The terminal provides the information to the user.

[0589] Input: Information sent from the server

[0590] Output: Information provided to the user

[0591] Specific operation: The terminal provides the information received from the server to the user by voice or visually. For example, if the user is excited to see Tokyo Tower, the terminal will display the history and tourist information of Tokyo Tower.

[0592] (Application example 2)

[0593] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0594] Conventional tourist guide systems have had the problem of being unable to provide real-time information about the scenery and buildings the user is viewing, and are unable to provide information based on the user's emotions. This has resulted in the user being unable to receive appropriate information at the moment of their interest, which has led to a decline in the quality of their tourist experience.

[0595] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0596] In this invention, the server includes means for acquiring image data in real time from contact lenses equipped with a camera function, means for recognizing scenery and buildings viewed by the user using an image recognition library, means for analyzing the user's emotions using an emotion recognition engine, means for acquiring information about the recognized scenery and buildings from a tourist information database, means for generating information to be provided to the user in natural language using a generative AI model, and means for suggesting other tourist spots based on the user's emotions. This makes it possible to provide information about scenery and buildings viewed by the user in real time, and further provide information based on the user's emotions.

[0597] "Earphones" are small acoustic devices that deliver sound directly to the ears.

[0598] A "contact lens" is a thin lens worn on the surface of the eye for vision correction or cosmetic purposes.

[0599] The "camera function" is a function for taking pictures and videos.

[0600] An "image recognition library" is a software library for analyzing image data and recognizing specific objects or patterns.

[0601] An "emotion recognition engine" is a software engine for analyzing emotions from a user's facial expressions and voice.

[0602] A "tourist information database" is a database that stores information about tourist spots and buildings.

[0603] A "generative AI model" is an artificial intelligence model that generates information in natural language based on input data.

[0604] "Real-time" refers to immediate processing or response without delay.

[0605] "Scenery" is a broad visual object that refers to natural and urban landscapes.

[0606] "Building" means a structure used for residence, business, or other purposes.

[0607] "User" refers to a person who uses a system or device.

[0608] "Information provision" is the act of providing necessary data and knowledge to users.

[0609] "Tourist spots" are places and facilities that tourists visit.

[0610] The following system configuration and processing procedure will be described as an embodiment of the present invention.

[0611] System Configuration

[0612] This system uses the following hardware and software:

[0613] Hardware

[0614] Earphones: Small acoustic devices that deliver sound directly to the ears.

[0615] Contact lenses: Thin lenses worn on the surface of the eye for vision correction or cosmetic purposes. They also have a camera function.

[0616] Smartphone: A central device in the system that runs various software.

[0617] software

[0618] Image recognition library (e.g., OpenCV): A software library used to analyze image data and recognize specific objects or patterns.

[0619] Emotion recognition engine (e.g., Microsoft Azure (registered trademark) Emotion API): A software engine for analyzing emotions from a user's facial expressions and voice.

[0620] Tourism information database (e.g., Google Places API): A database that stores information about tourist spots and buildings.

[0621] Generative AI model (e.g., GPT-4): An artificial intelligence model that generates information in natural language based on input data.

[0622] Processing steps

[0623] 1. Acquisition of image data

[0624] The contact lenses' built-in camera captures real-time image data of the scenery and buildings the user is looking at, and the smartphone receives and processes this data.

[0625] 2. Image Recognition

[0626] The smartphone uses an image recognition library (OpenCV) to analyze the acquired image data, thereby identifying the scenery and buildings the user is looking at.

[0627] 3. Emotion recognition

[0628] An emotion recognition engine (Microsoft Azure Emotion API) is used on the smartphone to analyze the user's emotions from their facial expressions and voice, thereby determining whether they are emotional.

[0629] 4. Information acquisition

[0630] Information about recognized landscapes and buildings is obtained from the tourist information database (Google Places API), including the history and characteristics of the place, and its reputation as a tourist attraction.

[0631] 5. Information generation

[0632] Using a generative AI model (GPT-4), information is generated in natural language based on the acquired information, providing users with information in a format that is easy for them to understand.

[0633] 6. Information provision

[0634] The smartphone provides the generated information to the user either audibly through earphones or visually using the display function of contact lenses.

[0635] 7. Emotion-based recommendations

[0636] If the user is impressed, the generative AI model will be used to suggest other tourist spots that will give a similar impression, further enriching the user's sightseeing experience.

[0637] Specific examples

[0638] If a user is moved by a sunset at the seaside, the system will recognize that emotion and visual information and suggest nearby beautiful sunset spots.

[0639] Prompt Sentence Examples

[0640] Using image data of the scenery the user is looking at as input, please use an image recognition library to recognize the scenery and provide information about that scenery. Also, analyze the user's emotions with an emotion recognition engine, and if the user is impressed, please suggest other tourist spots that will give them a similar impression.

[0641] In this way, a tourist guide system is realized that combines a contact lens equipped with a camera function and an emotion engine.

[0642] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0643] Step 1:

[0644] Acquisition of image data

[0645] Image data of the scenery and buildings that the user sees through a camera mounted on the contact lens is acquired in real time.

[0646] Input: Image data of the scenery and buildings the user is viewing.

[0647] Output: Image data acquired in real time.

[0648] How it works: The contact lens camera captures the user's field of vision and sends that data to a smartphone.

[0649] Step 2:

[0650] Image Recognition

[0651] The device (smartphone) uses an image recognition library (OpenCV) to analyze the acquired image data and identify the scenery and buildings the user is looking at.

[0652] Input: Image data acquired in real time.

[0653] Output: Information about recognized scenery and buildings.

[0654] What it does: Your smartphone inputs image data into OpenCV and runs algorithms to recognize specific objects and patterns.

[0655] Step 3:

[0656] emotion recognition

[0657] The device (smartphone) uses an emotion recognition engine (Microsoft Azure Emotion API) to analyze emotions from the user's facial expressions and voice.

[0658] Input: User's facial and voice data.

[0659] Output: Parsed user emotion information.

[0660] Specific operation: The smartphone inputs the user's facial and voice data into an emotion recognition engine and runs an algorithm to analyze emotions.

[0661] Step 4:

[0662] Information acquisition

[0663] The device (smartphone) retrieves information about recognized scenery and buildings from a tourist information database (Google Places API).

[0664] Input: Recognized scenery and building information.

[0665] Output: Detailed information about the scenery and buildings.

[0666] Specific operation: The smartphone inputs information about recognized scenery and buildings into the Google Places API and retrieves related tourist information.

[0667] Step 5:

[0668] information generation

[0669] The device (smartphone) uses a generative AI model (GPT-4) to generate information to provide to the user in natural language based on the information it has acquired.

[0670] Input: Detailed information about the scenery and buildings.

[0671] Output: Generated information in natural language.

[0672] Specific operation: Tourist information acquired by a smartphone is input into GPT-4, which generates natural language sentences to provide to the user.

[0673] Step 6:

[0674] Information provision

[0675] The device (smartphone) provides the generated information to the user either audibly through earphones or visually using the display function of contact lenses.

[0676] Input: Natural language generated information.

[0677] Output: Information provided to the user.

[0678] How it works: The smartphone sends the generated information to earphones for audio or contact lenses for visual display.

[0679] Step 7:

[0680] Emotion-based suggestions

[0681] Based on the user's emotions, the device (smartphone) suggests other tourist spots that will give a similar impression.

[0682] Input: Parsed user emotion information.

[0683] Output: Information on suggested other tourist spots.

[0684] Specific operation: Based on the emotion information, the smartphone uses a generative AI model to generate information to suggest other tourist spots and provides it to the user.

[0685] Example 3

[0686] Next, a description will be given of a third embodiment of the third embodiment. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0687] Conventional information provision systems have had difficulty in providing appropriate information in real time according to the user's emotions. In addition, there was a lack of means to provide information quickly in urgent situations, such as when the user was confused or lost. This meant that users had to spend a lot of time and effort to obtain the information they needed.

[0688] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0689] In this invention, the server includes means for collecting biometric data of a user, means for analyzing the collected biometric data to identify the user's emotion, means for generating necessary information based on the identified emotion, and means for displaying the generated information in the user's field of view, thereby making it possible to provide appropriate information according to the user's emotion in real time.

[0690] "Means for collecting user biometric data" refers to devices and sensors for acquiring biometric information such as the user's heart rate, facial expression, and voice in real time.

[0691] "Means for analyzing collected biometric data to identify the user's emotions" refers to software or algorithms for analyzing and identifying the user's emotional state based on collected biometric data.

[0692] "Means for generating necessary information based on the identified emotion" refers to a system or program for generating appropriate information according to the user's emotional state.

[0693] "Means for displaying the generated information in the user's field of vision" refers to a device or technology for directly displaying the generated information in the user's field of vision.

[0694] This invention is a system that provides information corresponding to a user's emotions in real time. The system collects the user's biometric data, analyzes the data to identify the emotion, generates necessary information based on the identified emotion, and displays the generated information in the user's field of view.

[0695] Hardware and software used

[0696] Hardware

[0697] 1. Contact lenses: Contact lenses with retinal projection and display functions are used, specifically smart contact lenses and AR contact lenses.

[0698] 2. Sensors: The contact lenses are equipped with sensors to detect heart rate and facial expressions.

[0699] 3. Microphone: The contact lens has a built-in microphone to collect the user's voice.

[0700] software

[0701] 1. Emotion engine: Software that analyzes the user's biometric data to identify emotions. Specifically, it uses "EmotionAPI" and "Affectiva SDK."

[0702] 2. Map database and route calculation algorithm: Software that generates the optimal route based on the user's current location and destination. Specifically, Google Maps API is used.

[0703] 3. Communication protocol: A protocol for securely sending and receiving data between the server and the device. Specifically, "HTTPS" is used.

[0704] Specific operation of the system

[0705] Collecting user emotion data

[0706] The user wears contact lenses equipped with retinal projection and display functions. The device measures the user's heart rate with a built-in sensor and collects audio data with a microphone. For example, when the user mutters, "Oh no, I'm lost," the device collects audio data.

[0707] Emotional Data Analysis

[0708] The device sends the collected biometric data to a server, which uses the Emotion API to analyze the voice data and identify whether the user is confused, for example, by detecting fluctuations in voice tone or heart rate.

[0709] Generate the necessary information

[0710] The server obtains the user's current location from GPS data and generates the optimal route to the destination. For example, it uses Google Maps API to calculate the shortest route from the current location to the destination.

[0711] Sending information

[0712] The server sends the generated route information to the terminal using the HTTPS protocol, and the terminal processes the received route information and prepares it for display.

[0713] Displaying Information

[0714] The device uses retinal projection and display functions to display the route to the destination in the user's field of vision. For example, instructions such as "Turn right at the next intersection" are displayed in the upper right corner of the field of vision. This allows the user to visually confirm the information while heading towards the destination.

[0715] Examples and prompts

[0716] As a concrete example, consider a case where a user is lost and confused. When the emotion engine recognizes that the user is confused, the server analyzes the information and generates a route to the destination. The generated route information is sent to the device and displayed in the user's field of vision through the contact lenses.

[0717] Example prompt sentence:

[0718] "When the user is confused, create a program that displays the route to the destination in the user's field of vision."

[0719] This system allows the user to receive appropriate information according to their emotions in real time. For example, if the user is lost and confused, the system recognizes their emotions and displays the route to their destination directly in their field of vision. This allows the user to reach their destination quickly and efficiently. The flow of the identification process in Example 3 will be described with reference to FIG. 21.

[0720] Step 1: Collecting user biometric data

[0721] The user wears contact lenses equipped with retinal projection and display functions. The device measures the user's heart rate with a built-in sensor and collects voice data with a microphone. For example, when the user mutters, "Oh no, I'm lost," the voice data is collected. The input is the user's biometric data (heart rate, voice), and the output is the collected biometric data.

[0722] Step 2: Analyze the emotion data

[0723] The device sends the collected biometric data to the server, which then uses the Emotion API to analyze the voice data and identify whether the user is confused. For example, it can detect confusion from voice tone or heart rate fluctuations. The input is the collected biometric data, and the output is the identified emotional state (confusion).

[0724] Step 3: Generate the necessary information

[0725] The server obtains the user's current location from GPS data and generates the optimal route to the destination. For example, it uses the Google Maps API to calculate the shortest route from the current location to the destination. The input is the identified emotional state (confusion) and the user's current location, and the output is the generated route information.

[0726] Step 4: Submit your information

[0727] The server sends the generated route information to the terminal using the HTTPS protocol. The terminal processes the received route information and prepares it for display. The input is the generated route information, and the output is the route information ready for display.

[0728] Step 5: Viewing information

[0729] The device uses retinal projection and display functions to display the route to the destination in the user's field of view. For example, instructions such as "Turn right at the next intersection" are displayed in the upper right corner of the field of view. This allows the user to visually confirm the information while heading towards the destination. The input is route information that is ready to be displayed, and the output is the route information displayed in the user's field of view.

[0730] (Application example 3)

[0731] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0732] In conventional self-driving vehicles, there are limited ways for drivers and passengers to visually check road information and routes to their destinations in real time, which makes it difficult to quickly provide appropriate information, especially when users feel confused or anxious.In addition, there is also the problem that the user experience is not improved because information cannot be provided in accordance with the user's emotions.

[0733] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[0734] In this invention, the server includes earphones and contact lenses, a means for linking them, a means for making a response to a user's desired request available to the ear or the eye, a means for recognizing the user's emotions, a means for displaying information in the user's field of vision according to the user's emotions, and a means for displaying road information and a route to a destination in the field of vision of a driver or passenger of an autonomous vehicle in real time. This makes it possible to quickly provide appropriate information and improve the user experience even when the user feels confused or anxious.

[0735] "Earphones" are small acoustic devices that deliver sound directly to the ears.

[0736] A "contact lens" is a thin lens that is worn on the surface of the eye for vision correction, decoration, or information display.

[0737] "Means of cooperation" refers to the technology or method that allows multiple devices or systems to communicate with each other and operate in cooperation.

[0738] "Means for making available to the user audio or visual responses to requests" refers to techniques or methods for providing audio or visual information in response to a user's request.

[0739] "Means for recognizing a user's emotions" refers to techniques or methods for analyzing a user's facial expressions, voice, behavior, etc. to determine their emotional state.

[0740] "Means for displaying information in the user's field of view according to the user's emotions" refers to techniques and methods for visually providing appropriate information based on the user's emotional state.

[0741] "Means for displaying road information and routes to destinations in real time to the driver or passengers of an autonomous vehicle" refers to technologies and methods that enable drivers and passengers in an autonomous vehicle to visually check the current road conditions and routes to destinations in real time.

[0742] A system for implementing this invention includes earphones, contact lenses, means for linking these, means for making a response to a user's desired request available to the ear or eye, means for recognizing the user's emotions, means for displaying information in the field of view according to the user's emotions, and means for displaying road information and routes to a destination in the field of view of a driver or passenger of an autonomous vehicle in real time.

[0743] 1. System Program

[0744] To recognize the user's emotions, the server captures the user's face using a camera and analyzes the user's emotions using an emotion recognition engine (EmotionRecognizer).If the user feels confused or anxious, the server obtains route information to the destination from a route planner (RoutePlanner) and displays it directly in the user's field of view through a display module (DisplayModule).

[0745] 2. Hardware and Software Used

[0746] This system uses hardware and software such as smart glasses (e.g., Google Glass (registered trademark)), a camera, an emotion recognition engine (EmotionRecognizer), a route planner (RoutePlanner), and a display module (DisplayModule). The camera captures the user's face, and the emotion recognition engine analyzes the image data to recognize emotions. The route planner calculates route information to the destination, and the display module displays that information in the user's field of vision.

[0747] 3. Specific Examples

[0748] For example, if a user feels confused or anxious while riding in an autonomous vehicle, the system will recognize that emotion and display route information to the destination in real time in the user's field of vision, allowing the user to check the route to their destination with peace of mind.

[0749] Prompt Sentence Examples

[0750] "If the user is stumped, create an application that displays the route to their destination in their field of vision."

[0751] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0752] Step 1:

[0753] The server captures the user's face using the camera mounted on the smart glasses.

[0754] Input: Video data from the smart glasses camera

[0755] Output: User's face image

[0756] Specific operation: The camera captures video in real time and sends the video data to the server.

[0757] Step 2:

[0758] The server inputs the captured facial image into an emotion recognition engine (EmotionRecognizer) and analyzes the user's emotions.

[0759] Input: User's face image

[0760] Output: User's emotional state (e.g., confusion, anxiety, joy, etc.)

[0761] Specific operation: The emotion recognition engine analyzes facial images and determines emotions based on facial expressions and features.

[0762] Step 3:

[0763] If the server determines that the user's emotional state is confused or anxious, it requests route information to the destination from a route planner.

[0764] Input: User's emotional state

[0765] Output: Route information request

[0766] Specific operation: Based on the output of the emotion recognition engine, the server instructs the route planner to obtain route information.

[0767] Step 4:

[0768] The route planner calculates the optimal route based on the current location and destination information and sends that information back to the server.

[0769] Input: current location information, destination information

[0770] Output: Route information to the destination

[0771] How it works: The route planner uses map data and algorithms to calculate the optimal route and sends the results to the server.

[0772] Step 5:

[0773] The server sends the acquired route information to a display module (DisplayModule) and displays it in the user's field of view.

[0774] Input: Route information to destination

[0775] Output: Route information displayed in the field of view

[0776] Specific operation: The server sends route information to the display module, which displays the information in the user's field of view.

[0777] Step 6:

[0778] The user checks the route information displayed in their field of vision through the smart glasses and understands the route to their destination.

[0779] Input: Route information displayed in the field of view

[0780] Output: User understanding and behavior

[0781] Specific operation: The user checks the route to the destination based on the information displayed in their field of vision and travels with peace of mind.

[0782] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0783] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0784] Another example of generative AI is Gemini (registered trademark) (Internet search engine). <url: https: gemini.google.com ?hl="ja">) are mentioned.

[0785] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0786] [Second embodiment]

[0787] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0788] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0789] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure.

[0790] The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0791] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0792] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0793] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0794] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0795] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0796] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0797] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0798] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0799] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0800] "Example 1"

[0801] One embodiment of the present invention is a system that links earphones and contact lenses. This system allows a user to obtain answers to their desired requests through their ears or eyes. Specifically, when a user makes a voice inquiry, the system generates a response and provides the information through the earphones or visually through the contact lenses. "Embodiment Example 2"

[0802] In the future, contact lenses may be equipped with a camera, allowing the system to recognize the scenery or object the user is looking at and provide relevant information. For example, if the user is looking at a building, the system can provide information such as the building's name and history.

[0803] "Example 3"

[0804] In the future, contact lenses could also be equipped with retinal projection and display capabilities, allowing information to be displayed directly in the user's field of vision. For example, if a user is walking down a street, a map of the street and the route to their destination could be displayed.

[0805] The processing flow of each embodiment will be described below.

[0806] "Example 1"

[0807] Step 1: The user speaks a query.

[0808] Step 2: The system analyzes the user's query and generates an answer to it.

[0809] Step 3: The generated answer is presented to the user either audibly through earphones or visually through contact lenses.

[0810] "Example 2"

[0811] Step 1: The camera mounted on the contact lens captures the scenery or object the user is looking at.

[0812] Step 2: The system analyzes the captured image and generates relevant information.

[0813] Step 3: The generated information is presented to the user either audibly through earphones or visually through contact lenses.

[0814] "Example 3"

[0815] Step 1: The user speaks their destination.

[0816] Step 2: The system calculates the route to the input destination.

[0817] Step 3: The calculated route is displayed directly in the user's field of vision using the contact lens' built-in retinal projection and display capabilities.

[0818] Example 1

[0819] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0820] With conventional information provision systems, it is difficult for users to receive a quick and efficient response when making a voice inquiry. Furthermore, the limited visual means of providing information reduces user convenience. Furthermore, processing voice data and generating responses using generative AI models are complex, and a system that can manage these processes in an integrated manner is needed.

[0821] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0822] In this invention, the server includes a means for a user to make a voice inquiry, a means for a terminal to transmit voice data to the server, a means for the server to convert the voice data into text data, a means for the server to generate an answer using a generative AI model, a means for the server to convert the answer into voice data or image data, a means for the server to transmit the converted data to the terminal, and a means for the terminal to provide information to the user through earphones or contact lenses, thereby enabling the user to quickly and efficiently obtain information by voice or visually.

[0823] "Means for users to make voice inquiries" refers to a device or interface that allows users to input voice, such as a microphone built into earphones.

[0824] "Means for a terminal to transmit voice data to a server" refers to a device or software that has the function of transmitting voice data to a server via a communication network such as the Internet.

[0825] "Means by which the server converts voice data into text data" refers to software or algorithms that use voice recognition technology to convert voice data into text data.

[0826] "Means by which the server generates an answer using a generative AI model" refers to software or algorithms that use a generative AI model to generate appropriate answers based on text data.

[0827] "Means by which the server converts the response into voice data or image data" refers to software or algorithms for converting the generated text data into voice data or image data.

[0828] "Means for transmitting data converted by the server to the terminal" refers to a device or software that has the function of transmitting converted audio data or image data to the terminal via a communication network such as the Internet.

[0829] "Means by which a terminal provides information to a user through earphones or contact lenses" refers to devices or software that have the function of providing received audio data as audio through earphones or the function of providing image data visually through contact lenses.

[0830] MODE FOR CARRYING OUT THE INVENTION

[0831] This invention is a system in which a user makes a voice inquiry and receives a response via voice or visually through earphones or contact lenses. This system is implemented using the following hardware and software.

[0832] Hardware and Software Configuration

[0833] 1. User terminal: A device equipped with a microphone for user voice input, such as a smartphone or dedicated device.

[0834] 2. Server: A high-performance computer for processing voice data, generating answers using generative AI models, and converting and transmitting data.

[0835] 3. Speech recognition software: Software for converting voice data into text data, such as Google Cloud Speech-to-Text.

[0836] 4. Generative AI models: AI models for generating appropriate answers based on text data, such as OpenAI's GPT-4.

[0837] 5. Speech synthesis software: Software for converting text data into speech data, such as Amazon Polly.

[0838] 6. Contact Lenses: Contact lenses that have a display function to provide visual information.

[0839] System Operation

[0840] When a user asks a question by voice through the microphone built into the earphones, the device captures the voice data and sends it over the internet to a server. The server converts the voice data into text data using Google Cloud Speech-to-Text. The server then uses a generative AI model (GPT-4) to generate an appropriate answer based on the text data.

[0841] The generated answer is converted into either audio or image data, depending on the user's selection. In the case of audio data, the server uses Amazon Polly to convert the text data into audio data. In the case of image data, the server converts the text into a format for visual display. The converted data is then sent back to the device via the Internet.

[0842] The terminal provides the received audio data audibly through earphones or visually through contact lenses, allowing the user to quickly and efficiently obtain information audibly or visually.

[0843] Specific examples

[0844] For example, if a user speaks into the earphones, "What's the weather today?", the device captures the audio and sends it to the server. The server converts the audio to text using Google Cloud Speech-to-Text, generating the text "What's the weather today?". The server then uses GPT-4 to generate the answer "The weather is sunny today." If the user selects the earphones, the server converts the text to speech using Amazon Polly and sends the audio data to the device. The device then provides the audio "The weather is sunny today" to the user through the earphones.

[0845] Example prompt sentence:

[0846] User: "What's the weather like today?"

[0847] Server: "The weather is sunny today."

[0848] The system allows users to obtain information quickly and efficiently, either audibly or visually.

[0849] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0850] Step 1:

[0851] The user makes a voice inquiry.

[0852] The user asks a question by voice through the microphone built into the earphones. For example, "What's the weather like today?" This voice data becomes the input.

[0853] Step 2:

[0854] The terminal transmits the voice data to the server.

[0855] The terminal captures the user's voice data and transmits it to a server via the Internet, where the input is the user's voice data and the output is the voice data transmitted to the server.

[0856] Step 3:

[0857] The server converts the voice data into text data.

[0858] The server uses speech recognition software to convert the received voice data into text data, for example, using Google Cloud Speech-to-Text. In this case, the input is voice data, and the output is text data such as "What's the weather like today?"

[0859] Step 4:

[0860] The server generates an answer using a generative AI model.

[0861] The server uses a generative AI model to generate an appropriate answer based on the text data. For example, OpenAI's GPT-4 is used. In this case, the input is the text data "What's the weather like today?" and the output is the answer text "Today's weather is sunny."

[0862] Step 5:

[0863] The server converts the response into audio or image data.

[0864] The server converts the generated text data into either voice data or image data, depending on the user's selection. For voice data, the server uses speech synthesis software, such as Amazon Polly. For image data, the server converts the text into a format for visual display. In this case, the input is the text data "Today's weather is sunny," and the output is either voice data or image data.

[0865] Step 6:

[0866] The server transmits the converted data to the terminal.

[0867] The server transmits the converted audio data or image data to the terminal via the Internet. At this time, the input is the audio data or image data, and the output is the data transmitted to the terminal.

[0868] Step 7:

[0869] The device provides information to the user through earphones or contact lenses.

[0870] The data received by the device is provided either audibly through earphones or visually through contact lenses. For example, the earphones may say "Today's weather is sunny" or the contact lenses may display the text "Today's weather is sunny." In this case, the input is the data sent from the server, and the output is the information provided to the user.

[0871] (Application example 1)

[0872] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0873] In autonomous vehicles, users are required to quickly and accurately obtain destination and route information. It is also important to grasp traffic conditions and vehicle status in real time and notify users at the appropriate time. However, conventional systems lack efficient means to provide this information, resulting in reduced user convenience. Furthermore, the lack of integration between visual and auditory information provision can cause confusion when users obtain information.

[0874] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0875] In this invention, the server includes earphones and contact lenses, a means for linking them, a means for making a response to a user's desired request available to the user aurally or visually, a means for recognizing the user's voice input and calculating optimal route information using a generative AI model, a means for providing voice guidance of the calculated route information, and a means for visually displaying important information through the contact lenses. This allows the user to quickly and accurately obtain route information through voice input and visually confirm important information.

[0876] An "earphone" is a small acoustic device that delivers sound directly to the user's ears.

[0877] A "contact lens" is a thin lens that is worn on a user's eye to provide visual information.

[0878] "Means for linking" refers to the technical means by which the earphones and contact lenses communicate with each other and share information.

[0879] "Means for making available to the user audibly or visually a response to a desired request" means means for providing information generated based on the user's voice input audibly through earphones or visually through contact lenses.

[0880] "Means for recognizing a user's voice input and calculating optimal route information using a generative AI model" refers to means for analyzing a user's voice and calculating an optimal route using a generative AI model.

[0881] The "means for providing audio guidance of calculated route information" refers to a means for conveying calculated route information to the user by audio.

[0882] "Means for visually displaying important information through contact lenses" refers to means for displaying important information such as traffic conditions and warnings on contact lenses.

[0883] The system for implementing this invention links earphones and contact lenses, and when a user makes a voice inquiry, it uses a generative AI model to calculate optimal route information, provides voice guidance through the earphones, and visually displays important information through the contact lenses.

[0884] Hardware and software used

[0885] Hardware:

[0886] Smartphone or smart glasses

[0887] earphones

[0888] Contact Lens Display

[0889] software:

[0890] speech_recognition library: for performing speech recognition

[0891] pyttsx3 library: for speech synthesis

[0892] The requests library: to obtain route information

[0893] some_contact_lens_display_library: To display information on a contact lens display

[0894] System Operation

[0895] 1. Voice input recognition:

[0896] The user inputs their destination by voice through the microphone attached to their smartphone or smart glasses. The voice data is converted into text using a speech recognition library (speech_recognition).

[0897] 2. Calculating route information:

[0898] The recognized text data is input into a generative AI model to calculate the optimal route information, which is obtained through an external route planning API (using the requests library).

[0899] 3. Voice guidance:

[0900] The calculated route information is converted into voice using a speech synthesis library (pyttsx3) and guided to the user through earphones.

[0901] 4. Visual Indication:

[0902] Display important traffic information and warnings on the contact lens display using some_contact_lens_display_library.

[0903] Specific examples

[0904] When a user voice-inquires, "Tell me the route to Tokyo Station," the system calculates the optimal route and announces through the earphones, "The optimal route has been calculated. We are now departing." The system also displays route information and important traffic information on the contact lens display.

[0905] Prompt Sentence Examples

[0906] "Tell me the route to Tokyo Station."

[0907] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0908] Step 1:

[0909] The user inputs the destination by voice through a microphone attached to the smartphone or smart glasses.

[0910] Input: User's voice data

[0911] Processing: Use the speech recognition library (speech_recognition) to convert the audio data into text.

[0912] Output: Text data (e.g., "Please tell me the route to Tokyo Station.")

[0913] Step 2:

[0914] The server receives the text data and inputs it into the generative AI model.

[0915] Input: Text data

[0916] Processing: Uses a generative AI model to calculate optimal route information.

[0917] Output: Route information data (e.g., optimal route, travel time, traffic conditions, etc.)

[0918] Step 3:

[0919] The server obtains the route information data and converts it into voice using a voice synthesis library (pyttsx3).

[0920] Input: Route information data

[0921] Processing: Convert the route information into speech using a speech synthesis library.

[0922] Output: Voice data (e.g. "Optimal route calculated. Let's go.")

[0923] Step 4:

[0924] The terminal provides the user with audio data through earphones.

[0925] Input: Audio data

[0926] Processing: Play audio data through earphones.

[0927] Output: The user hears the voice prompts.

[0928] Step 5:

[0929] The server displays important traffic information and warnings on the contact lens display.

[0930] Input: Route information data and additional traffic information data

[0931] Processing: Use some_contact_lens_display_library to visually display important information.

[0932] Output: Visual information displayed on the contact lens display (e.g., route arrows, traffic warnings, etc.)

[0933] Step 6:

[0934] The user views visual information through the contact lens display.

[0935] Input: Visual information displayed on a contact lens display

[0936] Action: The user checks the visual information and takes the necessary action.

[0937] Output: User understands visual information and acts upon it

[0938] Example 2

[0939] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0940] In conventional systems, it was difficult to obtain information about the objects and scenery that the user was looking at in real time. Also, the means by which the user could obtain answers to their desired requests by ear or by sight were limited, making it cumbersome to obtain information. Furthermore, there was a lack of a means to efficiently provide information related to the objects and scenery that the user was looking at.

[0941] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for receiving video data from the contact lenses, means for analyzing the received video data and identifying the viewed object or scenery, and means for using a generative AI model for generating information related to the identified object or scenery. This makes it possible to obtain information about the object or scenery the user is viewing in real time and provide it efficiently.

[0942] "Earphones" are small acoustic devices that deliver sound directly to the ears.

[0943] A "contact lens" is a thin, transparent or translucent lens that is worn directly on the eye and is used to correct vision or provide a specific function.

[0944] "Means of cooperation" refers to the technology or method that enables multiple devices or systems to communicate with each other and operate in cooperation.

[0945] "Means for making available to the user audibly or visually a response to a request" refers to techniques or methods for providing the information desired by the user audibly or visually.

[0946] "Means for receiving video data" refers to the technology or method for receiving video information obtained from a camera or sensor.

[0947] "Means for analyzing video data and identifying the object or scene being viewed" refers to techniques or methods for processing received video information and identifying the object or scene being viewed by the user.

[0948] A "means of using a generative AI model" is a technique or method for using artificial intelligence to perform a specific task, and is used here to generate relevant information.

[0949] The "means for providing information to users" refers to the techniques and methods for transmitting the generated information to users.

[0950] MODE FOR CARRYING OUT THE INVENTION

[0951] This invention is a system that incorporates a camera function into contact lenses worn by a user, recognizes the scenery and objects the user is looking at, and provides related information. A specific embodiment of this system is described below.

[0952] Hardware and software used

[0953] Contact Lenses: Contact lenses are equipped with a camera that can capture the scenery and objects the user is looking at in real time.

[0954] Terminal: A device such as a smartphone or smart glasses is used. These terminals receive video data from the contact lenses and send it to the server.

[0955] Server: A server with high-performance computing power is used to analyze the received video data and identify the objects and scenery being viewed.

[0956] Image recognition software: Uses image recognition software such as OpenCV and TensorFlow, which can analyze video data and identify objects and scenery.

[0957] Generative AI models: Use generative AI models such as GPT-4 to generate information related to identified objects and scenes.

[0958] Program processing

[0959] The server receives the video data sent from the device. The received video data is analyzed using image recognition software. As a result of the analysis, the object or scene the user is looking at is identified. The server then uses a generative AI model to generate information related to the identified object or scene. This information is provided to the user via the device.

[0960] Specific examples

[0961] For example, if a user is looking at a building, the contact lens camera captures video of the building. The video data is sent via the device to a server, which uses image recognition software to identify the building. The server then uses a generative AI model to generate information about the building (e.g., "This building is a historic landmark and was built in the 19th century.") This information is then provided to the user via the device.

[0962] Prompt Sentence Examples

[0963] "Please tell us the name and history of the building you are viewing. The video data for the building is as follows."

[0964] By inputting this prompt into a generative AI model, detailed information about the building the user is looking at can be generated.

[0965] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0966] Program processing flow

[0967] Step 1:

[0968] A user wears the contact lenses and views a scene or object.

[0969] Input: Scenery and objects within the user's field of vision.

[0970] Output: Video data captured by the contact lens camera.

[0971] How it works: When a user wears contact lenses and looks at, say, Tokyo Tower, the camera in the contact lenses captures the image in real time.

[0972] Step 2:

[0973] The terminal receives the video data from the contact lens.

[0974] Input: Video data transmitted from contact lenses.

[0975] Output: Video data stored on the device.

[0976] Specific operation: A device such as a smartphone or smart glasses receives video data of Tokyo Tower transmitted from the contact lenses.

[0977] Step 3:

[0978] The terminal transmits the video data to the server.

[0979] Input: Video data stored on the device.

[0980] Output: Video data sent to the server.

[0981] Specific operation: The device sends the received video data of Tokyo Tower to the server via Wi-Fi or mobile data communication.

[0982] Step 4:

[0983] The server analyzes the video data and identifies the objects and scenery being viewed.

[0984] Input: Video data sent to the server.

[0985] Output: Information about identified objects and scenery.

[0986] Specific operation: The server uses image recognition software such as OpenCV or TensorFlow to analyze the received video data of Tokyo Tower and identify it as Tokyo Tower.

[0987] Step 5:

[0988] The server uses the generative AI model to generate relevant information.

[0989] Input: Identified object and scene information.

[0990] Output: The relevant information generated.

[0991] Specific operation: The server uses a generative AI model such as GPT-4 to generate information about the identified Tokyo Tower (for example, "Tokyo Tower was completed in 1958 and is 333 meters tall").

[0992] Step 6:

[0993] The server transmits the generated information to the terminal.

[0994] Input: Generated relevant information.

[0995] Output: Relevant information sent to the terminal.

[0996] Specific operation: Information about Tokyo Tower generated by the server is sent to the device via Wi-Fi or mobile data communication.

[0997] Step 7:

[0998] The terminal provides the information to the user.

[0999] Input: Relevant information sent to the terminal.

[1000] Output: Information provided to the user.

[1001] Specific operation: The information about Tokyo Tower received by the device is displayed to the user on the screen of their smartphone or smart glasses, allowing the user to check detailed information about Tokyo Tower in real time.

[1002] (Application example 2)

[1003] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1004] Conventional systems have had difficulty obtaining information about the object a user is looking at in real time. Furthermore, there are limited ways for users to obtain answers to their requests by ear or sight, which hinders the user experience. Furthermore, there is a lack of ways to instantly provide detailed product information in physical stores.

[1005] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1006] In this invention, the server includes earphones and contact lenses, a means for linking them, a means for making a response to a user's desired request available to the user's ears or eyes, a means for recognizing an object the user is looking at and providing information about the object, and a means for displaying the information in real time. This allows the user to obtain information about the object they are looking at in real time and to obtain a response to a user's desired request to the user's ears or eyes. Furthermore, detailed product information can be provided instantly in a physical store, improving the user experience.

[1007] "Earphones" are small audio devices that deliver sound directly to the ears.

[1008] A "contact lens" is a thin lens that is placed directly on the surface of the eye and is used for vision correction, decoration, or to provide a specific function.

[1009] "Means of cooperation" are methods and technologies that allow multiple devices or systems to exchange information with each other and work in cooperation.

[1010] "Means for providing audible or visual responses to user requests" refers to techniques or methods that provide audio or visual responses to information or instructions requested by users.

[1011] "Means for recognizing an object that a user is looking at and providing information about that object" refers to a technology or method for identifying an object that comes into the user's field of vision and obtaining and providing data or information related to that object.

[1012] The "means for displaying the information in real time" refers to a technique or method for visually presenting acquired information to a user immediately without delay.

[1013] A system for implementing the present invention includes earphones, contact lenses, a smartphone or smart glasses, a server, and software for linking these components. Specific embodiments are described below.

[1014] Hardware and software used

[1015] Earphone: A small acoustic device that delivers sound to the user.

[1016] Contact lenses: Thin lenses with built-in camera functions.

[1017] Smartphone or smart glasses: A device that performs image recognition and displays information.

[1018] Server: A central system that manages databases and provides information.

[1019] Image recognition software: Google Cloud Vision API, etc.

[1020] Database management system: MySQL, etc.

[1021] Information delivery system: Custom web application.

[1022] Program processing

[1023] The server includes the following means:

[1024] 1. Image capture means: The contact lenses capture objects that come into the user's field of view.

[1025] 2. Image transmission means: Transmit the captured image to a smartphone or smart glasses.

[1026] 3. Image Recognition: Image recognition software on a smartphone or smart glasses analyzes the image and identifies the object.

[1027] 4. Database query means: Retrieving information about the identified object from a database.

[1028] 5. Information provision means: The acquired information is displayed in real time on the user's smartphone or smart glasses.

[1029] Specific examples

[1030] Let's say a user is looking at a particular shampoo in a brick-and-mortar store. The contact lens captures an image of the shampoo and sends it to their smartphone. The Google Cloud Vision API analyzes the image and identifies the shampoo. It retrieves information about the shampoo (price, ingredients, reviews, etc.) from a MySQL database and displays it on their smartphone.

[1031] Prompt Sentence Examples

[1032] Design an application that identifies the product a user is looking at and provides detailed information about that product in real time. The hardware required is a contact lens with a built-in camera and a smartphone. The software required is Google Cloud Vision API and MySQL. Please explain the specific process, including a concrete example where the user is looking at shampoo.

[1033] In this way, users can get real-time information about the object they are looking at, hear or see the answer to their desired request, and instantly get detailed product information in a physical store, improving the user experience.

[1034] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1035] Step 1:

[1036] Image Capture

[1037] A user wears contact lenses and browses products in a physical store. The contact lens's camera captures the products in the user's field of view.

[1038] Input: Products in the user's field of view

[1039] Output: Image data of the captured product

[1040] How it works: A camera built into the contact lens periodically scans the field of view and captures an image of any products it detects.

[1041] Step 2:

[1042] Image transmission

[1043] The captured image data is transmitted to a smartphone or smart glasses.

[1044] Input: Captured product image data

[1045] Output: Image data sent to a smartphone or smart glasses

[1046] How it works: Image data is sent from the contact lens to a smartphone or smart glasses via Bluetooth or Wi-Fi.

[1047] Step 3:

[1048] Image Recognition

[1049] Image recognition software (such as Google Cloud Vision API) on a smartphone or smart glasses analyzes the image and identifies the product.

[1050] Input: Image data sent to a smartphone or smart glasses

[1051] Output: Identification information of the identified product

[1052] How it works: The Google Cloud Vision API sends image data to the cloud, analyzes it to identify the product, and returns the product's identification information as the analysis result.

[1053] Step 4:

[1054] Database queries

[1055] Based on the identification information of the identified product, the server retrieves information about the product from a database (e.g., MySQL).

[1056] Input: Identification information of the identified product

[1057] Output: Product details (price, ingredients, reviews, etc.)

[1058] What happens: The server queries the MySQL database to retrieve details of products that match the identification information.

[1059] Step 5:

[1060] Information provision

[1061] Detailed information about the acquired products is displayed in real time on the user's smartphone or smart glasses.

[1062] Input: Product details (price, ingredients, reviews, etc.)

[1063] Output: Product details displayed on a smartphone or smart glasses

[1064] Specific operation: The detailed information of the acquired product is visually displayed on the display of the smartphone or smart glasses, and the user can check it.

[1065] In this way, users can get real-time information about the object they are looking at, hear or see the answer to their desired request, and instantly get detailed product information in a physical store, improving the user experience.

[1066] Example 3

[1067] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1068] Conventional navigation systems require users to hold a smartphone or other device in their hand to visually check information, which reduces safety and convenience while traveling. Furthermore, detailed information is difficult to grasp using voice guidance alone. To solve these issues, a system that displays information directly in the user's field of vision is needed.

[1069] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1070] In this invention, the server includes means for inputting a request desired by the user, means for transmitting the request to the server, means for the server to obtain the user's location information, means for the server to calculate a route using the map information providing means, means for the server to transmit the route information calculated by the server to the terminal, means for the terminal to analyze the route information and convert it into display data, and means for displaying information in the user's field of view, thereby enabling the user to reach their destination efficiently and safely using the information displayed directly in their field of view.

[1071] The "means for inputting a user's desired request" refers to a device or software that provides an interface for the user to input their destination and necessary information.

[1072] The "means for transmitting the request to the server" refers to a communication device or protocol for transmitting the request input by the user to the server via a network.

[1073] The "means by which the server obtains the user's location information" refers to technology for identifying the user's current location using a GPS device or other location information acquisition device.

[1074] "Means for the server to calculate a route using map information providing means" refers to a technology that uses an API or software that provides map information to calculate the optimal route from the user's current location to the destination.

[1075] The "means for transmitting route information calculated by the server to the terminal" refers to a communication device or protocol for transmitting route information calculated by the server to the user's terminal via a network.

[1076] The "means by which the terminal analyzes route information and converts it into display data" refers to software or hardware that analyzes the received route information and converts it into a data format that can be displayed in the user's field of view.

[1077] "Means for displaying information in the user's field of vision" refers to a device for displaying information directly in the user's field of vision using a display built into a contact lens or retinal projection technology.

[1078] This invention provides a system that displays information directly in the user's field of view. This system includes a series of processes for inputting a user's desired request, calculating optimal route information based on the request, and displaying the information in the user's field of view.

[1079] First, the user inputs a prompt such as "Show me the route to a new restaurant" into the device. This prompt is used when the user wants to know the route to a destination. The device sends this prompt to the server. The server uses a GPS device to obtain the user's current location information. Next, the server uses a general map information API as a means of providing map information to calculate the optimal route from the user's current location to the destination.

[1080] The calculated route information is sent from the server to the device. The device analyzes the received route information and converts it into data for display on the contact lenses. This data is then converted into a format that is displayed directly in the user's field of vision using retinal projection technology. Finally, the display built into the user's contact lenses displays the converted route information directly in the user's field of vision. This allows the user to head to their destination while viewing the route information displayed in their field of vision.

[1081] As a concrete example, consider the case where a user wants to go to a new restaurant. The user types "Show me the route to the new restaurant" into the device. The device sends this prompt to the server. The server obtains the user's current location information and calculates the optimal route using a map information API. The calculation results are sent to the device, which analyzes the information and converts it into data to be displayed on the contact lenses. Finally, the route information is displayed in the user's field of vision, allowing the user to reach their destination without getting lost.

[1082] This system enables the user to reach the destination efficiently and safely by using the information displayed directly in the user's field of view. The flow of the identification process in the third embodiment will be described with reference to FIG.

[1083] Step 1:

[1084] The user types the prompt into the terminal.

[1085] When a user wants to know the route to a destination, the user inputs a prompt sentence into the terminal, such as "Show me the route to a new restaurant." The input prompt sentence is acquired through an input device of the terminal.

[1086] Step 2:

[1087] The terminal sends the prompt to the server.

[1088] The terminal sends the prompt text entered by the user to the server via the network. At this time, the user's current location information is also sent along with the prompt text. The input is the prompt text and location information, and the output is the data to be sent to the server.

[1089] Step 3:

[1090] The server obtains the user's location information.

[1091] The server receives the user's current location information sent from the terminal and uses it for the next process. The input is the location information from the terminal, and the output is the acquired location information.

[1092] Step 4:

[1093] The server calculates the route using the map information providing means.

[1094] The server uses a general map information API to provide map information and calculates the optimal route from the user's current location to the destination. The input is location information and destination information, and the output is the calculated route information.

[1095] Step 5:

[1096] The server sends the calculated route information to the terminal.

[1097] The server transmits the calculated route information to the user's terminal via the network. The input is the calculated route information, and the output is the data to be transmitted to the terminal.

[1098] Step 6:

[1099] The terminal analyzes the route information and converts it into display data.

[1100] The terminal analyzes the received route information and converts it into data to be displayed on the contact lens. The input is the received route information, and the output is the display data.

[1101] Step 7:

[1102] Display information in the user's field of view.

[1103] The display built into the user's contact lenses displays the converted route information directly in the user's field of vision. The input is display data, and the output is information displayed in the user's field of vision. This allows the user to head to their destination while viewing the route information displayed in their field of vision.

[1104] (Application example 3)

[1105] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1106] To safely and efficiently reach their destinations, autonomous vehicle drivers need to be provided with accurate, real-time visual navigation information. However, conventional navigation systems require drivers to take their eyes off the road, which compromises safety. Furthermore, voice assistant functionality is not fully integrated, requiring the driver to manually operate the system, which also reduces safety. To address these challenges, a system that allows drivers to obtain the necessary information without having to avert their eyes is needed.

[1107] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1108] In this invention, the server includes earphones and contact lenses, a means for linking them, a means for making responses to user requests available to the ears or eyes, a means for displaying navigation information in the field of view of the driver of the autonomous vehicle when the driver wears the contact lenses, and a means for equipping the driver with a voice assistant function and displaying information in response to the driver's voice commands. This enables the driver to obtain navigation information in real time without averting their eyes, and to reach their destination safely and efficiently.

[1109] An "earphone" is a small acoustic device that transmits sound directly to the ear.

[1110] A "contact lens" is a thin lens that is worn directly on the surface of the eye to correct vision or display information.

[1111] "Means of cooperation" refers to the technology or method that enables multiple devices or systems to communicate with each other and operate in cooperation.

[1112] "Means for making available to the user audio or visual responses to requests" refers to techniques or methods for providing information audibly or visually in response to a user's request.

[1113] "Means for displaying navigation information in the field of vision of the driver of an autonomous vehicle by wearing contact lenses" refers to technologies and methods for visually providing route and traffic information to the destination through contact lenses worn by the driver of an autonomous vehicle.

[1114] "Means for being equipped with a voice assistant function and displaying information in response to the driver's voice commands" refers to technologies and methods for using voice recognition technology to understand the driver's voice instructions and provide the necessary information based on them.

[1115] The following describes an embodiment of the present invention. The system components include earphones, contact lenses, a server, and a smartphone. These devices work together to provide real-time navigation information to the user.

[1116] System Program

[1117] The server implements the system using the following hardware and software.

[1118] Hardware: Smartphone, retinal projection contact lenses, earphones

[1119] Software: OpenCV, pyttsx3, Navigation API

[1120] Program processing explanation

[1121] When the user inputs a destination, the server uses the navigation API to obtain route and traffic information to the destination. The obtained information is overlaid on the view using OpenCV. Furthermore, the voice assistant reads the information aloud using pyttsx3.

[1122] Specifically, when a user enters a destination into their smartphone, the server calls the navigation API to obtain route and traffic information. Next, to display the obtained information on the contact lenses, OpenCV is used to overlay the information on the field of view. At the same time, using pyttsx3, the voice assistant reads out, "The route to your destination is XX. The current traffic conditions are △△."

[1123] Specific examples

[1124] For example, if a user inputs "Tokyo Tower" as their destination, the server will use the navigation API to obtain route and traffic information to "Tokyo Tower." The obtained information is displayed in the user's field of vision through the contact lenses, and the voice assistant will read out, "The route to Tokyo Tower is ____. The current traffic conditions are ____."

[1125] Prompt Sentence Examples

[1126] Enter your destination: Tokyo Tower

[1127] Getting route information...

[1128] Getting traffic information...

[1129] Displaying information in your field of view...

[1130] Your voice assistant is reading out information...

[1131] In this way, the user can obtain real-time navigation information without having to look away, enabling them to reach their destination safely and efficiently.

[1132] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1133] Step 1:

[1134] The user inputs the destination into the smartphone.

[1135] Input: Destination (e.g. Tokyo Tower)

[1136] Output: Destination information

[1137] Specific operation: The user starts the smartphone application and inputs the destination. The input destination information is sent to the server.

[1138] Step 2:

[1139] The server calls the navigation API to obtain route and traffic information.

[1140] Input: Destination information

[1141] Output: Route information, traffic information

[1142] Specific operation: The server sends destination information to the navigation API, obtains route information to the destination and current traffic information, and stores the obtained information within the server.

[1143] Step 3:

[1144] The server processes the route and traffic information it has acquired to display it on the contact lenses.

[1145] Input: Route information, traffic information

[1146] Output: Data for display

[1147] Specific operation: The server uses OpenCV to process the acquired route and traffic information into data for visual display. The processed data is then sent to the contact lens.

[1148] Step 4:

[1149] The contact lenses overlay route and traffic information onto your field of vision.

[1150] Input: Data to display

[1151] Output: Visual navigation information

[1152] How it works: The contact lenses receive display data sent from the server and overlay it on the user's field of view in real time, allowing the user to check navigation information without looking away.

[1153] Step 5:

[1154] The server uses a voice assistant to read out route and traffic information.

[1155] Input: Route information, traffic information

[1156] Output: Voice navigation information

[1157] Specific operation: The server uses pyttsx3 to read out the acquired route and traffic information. The voice assistant tells the user, "The route to your destination is XX. The current traffic conditions are △△."

[1158] Step 6:

[1159] The user requests additional information using voice commands.

[1160] Input: Voice command

[1161] Output: Additional information

[1162] How it works: When a user issues a voice command, the server uses voice recognition technology to interpret the command and provide any additional information needed, such as displaying information in response to a request like "Tell me where the nearest gas station is."

[1163] In this way, the user can obtain real-time navigation information without having to look away, enabling them to reach their destination safely and efficiently.

[1164] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1165] "Example 1"

[1166] The invention, combined with the emotion engine, recognizes a user's emotions and provides information accordingly. Specifically, when a user feels joy, the system recognizes that emotion and provides information that is likely to bring joy to the user. For example, when a user is excited about a music festival, the system recognizes that emotion and provides information about upcoming concerts by the same artist.

[1167] "Example 2"

[1168] The invention, which combines a camera function and an emotion engine, simultaneously recognizes what the user is looking at and the emotion they are feeling, enabling it to provide information accordingly. Specifically, when a user is moved by a beautiful landscape, the system recognizes that emotion and visual information and provides them with places with similarly beautiful scenery. For example, when a user is moved by a sunset by the sea, the system recognizes that emotion and visual information and provides them with nearby famous places for beautiful sunsets.

[1169] "Example 3"

[1170] The invention, which combines retinal projection, display functionality, and an emotion engine, visually provides information according to the user's emotions. Specifically, when the user is confused, the system recognizes that emotion and displays the necessary information directly in the user's field of vision. For example, when the user is lost and confused, the system recognizes that emotion and displays the route to the destination directly in the user's field of vision.

[1171] The processing flow of each embodiment will be described below.

[1172] "Example 1"

[1173] Step 1: Recognize the user's emotions with the emotion engine.

[1174] Step 2: Based on the recognized emotions, select information that is likely to bring joy to the user.

[1175] Step 3: Present the selected information to the user audibly or visually.

[1176] "Example 2"

[1177] Step 1: Obtain the user's visual information using the camera function.

[1178] Step 2: Recognize the user's emotions with the emotion engine.

[1179] Step 3: Select relevant information based on the acquired visual information and perceived emotions.

[1180] Step 4: Present the selected information to the user audibly or visually.

[1181] "Example 3"

[1182] Step 1: Recognize the user's emotions with the emotion engine.

[1183] Step 2: Select the information you need based on the emotions you recognize.

[1184] Step 3: The selected information is displayed directly in the user's field of vision using retinal projection and display functions.

[1185] Example 1

[1186] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1187] Conventional information provision systems have had difficulty in providing prompt and appropriate responses to user requests. Furthermore, they were unable to provide information in response to the user's emotions, limiting the user experience. Furthermore, there was a lack of visual information provision methods, limiting the means by which users could obtain information.

[1188] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1189] In this invention, the server includes means for converting voice data into text data using voice recognition software, means for generating answers based on the text data using a generative AI model, and means for analyzing the user's emotions using an emotion engine and providing information according to the emotions. This enables prompt and appropriate provision of answers to user requests, and also realizes provision of information according to the user's emotions. In addition, by including means for providing information audibly through earphones and means for providing information visually through contact lenses, the means by which users can obtain information are diversified, improving the user experience.

[1190] "Earphones" are acoustic devices that deliver sound directly to the user's ears.

[1191] A "contact lens" is an optical device that is worn on a user's eye to provide visual information.

[1192] "Means for linking" refers to the technical means for enabling the earphones and contact lenses to communicate with each other and share information.

[1193] "Speech recognition software" is a program for converting voice data into text data.

[1194] A "generative AI model" is an artificial intelligence model that generates appropriate answers based on input text data.

[1195] An "emotion engine" is software that analyzes a user's emotions and provides information according to those emotions.

[1196] The "means for providing by voice" refers to a technical means for conveying the generated answer to the user as voice.

[1197] The "visual presentation means" refers to a technical means for displaying the generated answer to the user as visual information.

[1198] "Retinal projection" is a technology that projects information directly onto the user's retina through contact lenses.

[1199] The "display function" is a function that provides visual information to the user through the contact lenses.

[1200] This invention is a system that links earphones and contact lenses to enable a user to obtain answers to requests they desire through their ears or eyes. Specific embodiments of this system will be described below.

[1201] First, the user makes a voice inquiry through the earphone microphone. For example, they might ask, "What events do you recommend for next weekend?" This voice data is sent to the server.

[1202] The server uses speech recognition software (e.g., Google Cloud Speech-to-Text) to convert the user's voice data into text data. It captures the voice signal as digital data, analyzes the data, and converts it into a string of characters.

[1203] The server then uses a generative AI model (e.g., OpenAI GPT-4) to generate appropriate answers based on the text data. For example, for the text "What events do you recommend for next weekend?", the AI ​​model generates an answer such as "There's a music festival in the city next weekend."

[1204] The generated answers are analyzed using an emotion engine (e.g., IBM Watson Tone Analyzer). The emotion engine recognizes emotions from the user's voice and provides information corresponding to those emotions. For example, if the user expresses joy, the server provides additional information that the user is likely to find joyful. Specifically, if the server recognizes that the user is excited about a music festival, it provides additional information about upcoming concerts by the same artist.

[1205] The generated answers are provided either audibly through earphones or visually through contact lenses. Augmented reality (AR) technology (e.g., Microsoft HoloLens) is used to provide visual information. For example, if the user is using earphones, the server may announce aloud, "There's a music festival in your city next weekend." If the user is using contact lenses, detailed information about the event is displayed visually.

[1206] As a concrete example, consider a case where a user makes a voice inquiry such as, "What events do you recommend for next weekend?" In this case, the server performs the following process.

[1207] 1. A user asks by voice, "What events do you recommend for next weekend?"

[1208] 2. The server uses voice recognition software to convert the voice data into text data.

[1209] 3. Use a generative AI model to generate appropriate answers based on the text data.

[1210] 4. Use an emotion engine to analyze the user's emotions and provide information according to their emotions.

[1211] 5. Provide generated answers audibly through earphones or visually through contact lenses.

[1212] Example prompt sentence:

[1213] "When a user asks by voice what events are recommended for the upcoming weekend, create a program that generates appropriate responses and provides information based on the user's emotions."

[1214] In this way, a system is realized that provides quick and appropriate responses to user requests.

[1215] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1216] Step 1:

[1217] The user speaks a query into the earphone microphone, for example, "What events do you recommend for next weekend?" The input is the user's voice data, and the output is an audio signal sent to the server through the earphone microphone.

[1218] Step 2:

[1219] The server uses speech recognition software to convert the user's voice data into text data. Specifically, it captures the voice signal as digital data, analyzes the data, and converts it into a string of characters. The input is voice data, and the output is text data.

[1220] Step 3:

[1221] The server uses a generative AI model to generate an appropriate answer based on the text data. For example, for the text "What events do you recommend for next weekend?", the AI ​​model generates an answer such as "There is a music festival in the city next weekend." The input is text data, and the output is the generated answer text.

[1222] Step 4:

[1223] The server uses an emotion engine to analyze the user's emotions. Specifically, it recognizes emotions from the user's voice and adjusts the system to provide information according to the emotion. For example, if the user expresses joy, the server provides additional information that the user is likely to find joyful according to the emotion. The input is the user's voice data and the generated answer text, and the output is an answer text adjusted according to the emotion.

[1224] Step 5:

[1225] The server provides the generated answer either audibly through earphones or visually through contact lenses. Augmented reality technology is used to provide visual information. For example, if the user is using earphones, the server may announce audibly, "There's a music festival in the city next weekend." If the user is using contact lenses, the server may visually display detailed information about the event. The input is the adjusted answer text, and the output is an audio notification or visual information.

[1226] (Application example 1)

[1227] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1228] Conventional information provision systems provide uniform information without considering the user's feelings, resulting in low user satisfaction. Furthermore, the means of providing information—visual and audio—are separate, resulting in an inconsistent user experience. Furthermore, there is a lack of technology to efficiently process users' voice requests and provide appropriate information.

[1229] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1230] In this invention, the server includes earphones and contact lenses, a means for linking them, a means for making a response to a user's desired request available aurally or visually, a means for recognizing the user's emotion and providing information corresponding to that emotion, a means for converting a voice request into text, a means for displaying visual information on the contact lenses, and a means for providing voice information through the earphones. This makes it possible to provide appropriate information according to the user's emotion, and the consistent provision of visual and voice information can improve the user experience.

[1231] An "earphone" is a small acoustic device that transmits sound directly to the user's ears.

[1232] A "contact lens" is a thin lens that displays visual information directly to a user's eye.

[1233] "Means for linking" refers to the technical means by which the earphones and contact lenses communicate with each other and share information.

[1234] "Means for making a response to a user's desired request available to the ears or eyes" refers to means for generating information based on the user's request and providing it through earphones or contact lenses.

[1235] "Means for recognizing a user's emotions and providing information according to those emotions" refers to technical means for analyzing a user's emotional state and providing information appropriate to those emotions.

[1236] A "means for converting a voice request to text" is a technical means for converting a user's voice input into text data.

[1237] The "means for displaying visual information on a contact lens" refers to a technical means for displaying the generated visual information to a user through the contact lens.

[1238] The "means for providing audio information through earphones" refers to the technical means for transmitting the generated audio information to the user through the earphones.

[1239] The following system configuration and processing procedure will be described as an embodiment of the present invention.

[1240] System Configuration

[1241] This system uses the following hardware and software:

[1242] Hardware: smartphones, earphones, smart contact lenses

[1243] Software: Speech recognition library (speech_recognition), emotion recognition library (emotion_recognition), content recommendation engine (content_recommender), visual information provision module (visual_display), audio output module (audio_output)

[1244] Processing flow

[1245] 1. Getting a voice request

[1246] The user makes a voice request through the smartphone's microphone, which is then converted into text using a speech recognition library.

[1247] 2. Emotion recognition

[1248] An emotion recognition library is used to analyze emotions from the user's voice, thereby identifying the user's current emotional state.

[1249] 3. Content Recommendation

[1250] A content recommendation engine is used to select appropriate content based on the user's request and emotional state. For example, if a user requests "Tell me about recent movies" and the emotion is recognized as "joy," the latest comedy movies will be recommended.

[1251] 4. Providing visual information

[1252] A visual information provision module is used to display visual information of selected content (e.g., movie posters and trailers) on the smart contact lenses.

[1253] 5. Provision of audio information

[1254] An audio output module is used to provide an overview and detailed information of the selected content via audio through earphones.

[1255] Specific examples

[1256] If a user makes a voice request such as "Tell me about recent movies," and the system recognizes the user's emotion as "joy," the system will recommend the latest comedy movies. The movie poster will be displayed on the smart contact lenses, and a synopsis of the movie will be spoken through the earphones.

[1257] Prompt Sentence Examples

[1258] User: "What are some recent movies?"

[1259] System: "It seems you're feeling joyful. I recommend the latest comedy, 'The Kingdom of Laughter.' Display the movie poster."

[1260] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1261] Step 1:

[1262] The user makes a voice request through the smartphone's microphone.

[1263] Input: User's voice request

[1264] Processing: The smartphone microphone captures the audio data.

[1265] Output: Audio data

[1266] Step 2:

[1267] The server uses a speech recognition library (speech_recognition) to convert the voice data into text.

[1268] Input: Audio data

[1269] Processing: The speech recognition library analyzes the audio data and generates corresponding text data.

[1270] Output: Text data (user request content)

[1271] Step 3:

[1272] The server uses an emotion recognition library (emotion_recognition) to analyze emotions from the user's voice.

[1273] Input: Audio data

[1274] Processing: An emotion recognition library analyzes the audio data to identify the user's emotional state.

[1275] Output: Emotion data (user's emotional state)

[1276] Step 4:

[1277] The server uses a content recommendation engine (content_recommender) to select appropriate content based on the user's request and emotional state.

[1278] Input: Text data, emotion data

[1279] Processing: The content recommendation engine analyzes the text data and emotional data to select the most appropriate content.

[1280] Output: Recommended content data

[1281] Step 5:

[1282] The server uses a visual information providing module (visual_display) to display visual information of the selected content on the smart contact lens.

[1283] Input: Recommended content data

[1284] Processing: The visual information providing module analyzes the recommended content data, generates visual information, and sends it to the smart contact lens.

[1285] Output: Visual information (what is displayed on the contact lenses)

[1286] Step 6:

[1287] The server uses the audio output module (audio_output) to provide an overview and detailed information of the selected content through the earphones.

[1288] Input: Recommended content data

[1289] Processing: The audio output module analyzes the recommended content data, generates audio information, and sends it to the earphones.

[1290] Output: Audio information (content provided through earphones)

[1291] Example 2

[1292] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1293] Conventional systems have difficulty integrating visual information with emotional responses to provide appropriate information. Furthermore, there is a lack of means to provide real-time information related to the objects and scenery the user is looking at. This has resulted in insufficient information being provided to improve the user experience.

[1294] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1295] In this invention, the server includes means for collecting video data through a camera function installed in the contact lenses, means for collecting user emotion data through an emotion engine, means for analyzing the collected data and identifying an object or scene the user is looking at, means for identifying the user's emotion, means for integrating the visual information and the emotion data and generating related information, and means for providing the generated information to the user. This makes it possible to provide information related to the object or scene the user is looking at in real time, improving the user experience.

[1296] "Earphones" are acoustic devices that deliver sound directly to the user's ears.

[1297] A "contact lens" is a thin lens that is worn on a user's eye and is a device for acquiring visual information.

[1298] "Means for linking" refers to the technical means for enabling the earphones and contact lenses to communicate with each other.

[1299] "Means for making available to the user audibly or visually a response to a user's desired request" refers to means for providing information audibly or visually based on a user's request.

[1300] The "camera function" is a photographing device for collecting video data.

[1301] An "emotion engine" is software or hardware for analyzing a user's emotions.

[1302] "Means for collecting video data" refers to means for acquiring visual information using a camera function.

[1303] The "means for collecting emotion data" is a means for acquiring the user's emotional state using an emotion engine.

[1304] "Means for analyzing collected data and identifying the objects and scenery the user is looking at" refers to means for analyzing video data and identifying the objects the user is visually recognizing.

[1305] The "means for identifying the user's emotion" is a means for analyzing emotion data to identify the user's emotional state.

[1306] "Means for integrating visual information and emotional data to generate related information" refers to means for generating related information by combining the objects or scenery the user is looking at with the emotions they are feeling at the time.

[1307] The "means for providing the generated information to the user" refers to a means for providing the generated information to the user audibly or visually.

[1308] This invention is a system that uses contact lenses and earphones worn by a user to collect and analyze visual information and emotional data of the user. A specific embodiment of this system will be described below.

[1309] First, the user wears contact lenses equipped with a camera function that captures real-time images of the scenery and objects the user is looking at, and then wears earphones to receive audio information.

[1310] The device receives the video data transmitted from the contact lenses. This video data is analyzed using image recognition software (e.g., OpenCV). The analyzed data is used to identify the objects and scenes the user is looking at.

[1311] At the same time, the device collects the user's emotional data using an emotion engine (e.g., Affectiva), which identifies the emotion the user is feeling based on the user's facial expressions and biometric data (heart rate, electrodermal activity, etc.).

[1312] The collected video data and emotion data are sent from the device to a server. The server integrates this data and associates the object or scenery the user is looking at with the emotion at that time. For example, if the user is impressed by a sunset by the sea, the server can use that information to provide nearby spots with beautiful sunsets.

[1313] The generated information is sent from the server to the terminal, which then provides the information to the user via voice or visual means. For example, if the user is excited to see Tokyo Tower, the terminal will display information about the history and sightseeing of Tokyo Tower.

[1314] As a concrete example, consider the case where a user is looking at a certain building. The server analyzes the video data sent from the camera and identifies the building as "Tokyo Tower." At the same time, if the emotion engine recognizes the user's emotion as "excitement," the server provides information about the history and sightseeing of Tokyo Tower.

[1315] Example prompts to input to a generative AI model:

[1316] "If a user is excited to see Tokyo Tower, provide them with historical and tourist information about the building."

[1317] In this way, a system is realized in which the server, terminal, and user work together to provide appropriate information to the user.

[1318] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1319] Step 1:

[1320] A user wears contact lenses to acquire visual information.

[1321] Input: User's visual information

[1322] Output: Image data from contact lenses

[1323] Specific operation: The user wears contact lenses equipped with a camera function and looks at the surrounding scenery and objects.

[1324] Step 2:

[1325] The device collects video data through its camera function.

[1326] Input: Image data from contact lenses

[1327] Output: Video data sent to the device

[1328] Specific operation: The device receives the video data captured by the contact lenses.

[1329] Step 3:

[1330] The device collects the user's emotional data through an emotion engine.

[1331] Input: User's facial expressions and biometric data

[1332] Output: Emotion data from the emotion engine

[1333] Specific operation: The device uses an emotion engine to analyze biometric data such as the user's facial expressions, heart rate, and electrodermal activity to identify the user's emotions.

[1334] Step 4:

[1335] The terminal transmits the collected data to the server.

[1336] Input: Video data, emotion data

[1337] Output: Data sent to the server

[1338] Specific operation: The device sends the collected video data and emotion data to the server.

[1339] Step 5:

[1340] The server analyzes the video data and identifies the objects and scenery the user is looking at.

[1341] Input: Video data

[1342] Output: Information on identified objects and scenery

[1343] Specific operation: The server uses image recognition software (e.g., OpenCV) to analyze the video data and identify the objects and scenery the user is looking at.

[1344] Step 6:

[1345] The server analyzes the emotion data and identifies the user's emotion.

[1346] Input: Emotion data

[1347] Output: Identified user sentiment

[1348] Specific operation: The server uses an emotion engine (e.g., Affectiva) to analyze the emotion data and identify the user's emotion.

[1349] Step 7:

[1350] The server integrates visual information and emotional data to generate related information.

[1351] Input: Information about identified objects and scenery, identified user emotions

[1352] Output: Related information generated

[1353] Specific operation: The server integrates visual information and emotional data and generates relevant information based on the object or scenery the user is looking at and their current emotions.

[1354] Step 8:

[1355] The server transmits the generated information to the terminal.

[1356] Input: Generated related information

[1357] Output: Information sent to the terminal

[1358] Specific operation: The server sends the generated information to the terminal.

[1359] Step 9:

[1360] The terminal provides the information to the user.

[1361] Input: Information sent from the server

[1362] Output: Information provided to the user

[1363] Specific operation: The terminal provides the information received from the server to the user by voice or visually. For example, if the user is excited to see Tokyo Tower, the terminal will display the history and tourist information of Tokyo Tower.

[1364] (Application example 2)

[1365] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1366] Conventional tourist guide systems have had the problem of being unable to provide real-time information about the scenery and buildings the user is viewing, and are unable to provide information based on the user's emotions. This has resulted in the user being unable to receive appropriate information at the moment of their interest, which has led to a decline in the quality of their tourist experience.

[1367] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1368] In this invention, the server includes means for acquiring image data in real time from contact lenses equipped with a camera function, means for recognizing scenery and buildings viewed by the user using an image recognition library, means for analyzing the user's emotions using an emotion recognition engine, means for acquiring information about the recognized scenery and buildings from a tourist information database, means for generating information to be provided to the user in natural language using a generative AI model, and means for suggesting other tourist spots based on the user's emotions. This makes it possible to provide information about scenery and buildings viewed by the user in real time, and further provide information based on the user's emotions.

[1369] "Earphones" are small acoustic devices that deliver sound directly to the ears.

[1370] A "contact lens" is a thin lens worn on the surface of the eye for vision correction or cosmetic purposes.

[1371] The "camera function" is a function for taking pictures and videos.

[1372] An "image recognition library" is a software library for analyzing image data and recognizing specific objects or patterns.

[1373] An "emotion recognition engine" is a software engine for analyzing emotions from a user's facial expressions and voice.

[1374] A "tourist information database" is a database that stores information about tourist spots and buildings.

[1375] A "generative AI model" is an artificial intelligence model that generates information in natural language based on input data.

[1376] "Real-time" refers to immediate processing or response without delay.

[1377] "Scenery" is a broad visual object that refers to natural and urban landscapes.

[1378] "Building" means a structure used for residence, business, or other purposes.

[1379] "User" refers to a person who uses a system or device.

[1380] "Information provision" is the act of providing necessary data and knowledge to users.

[1381] "Tourist spots" are places and facilities that tourists visit.

[1382] The following system configuration and processing procedure will be described as an embodiment of the present invention.

[1383] System Configuration

[1384] This system uses the following hardware and software:

[1385] Hardware

[1386] Earphones: Small acoustic devices that deliver sound directly to the ears.

[1387] Contact lenses: Thin lenses worn on the surface of the eye for vision correction or cosmetic purposes. They also have a camera function.

[1388] Smartphone: A central device in the system that runs various software.

[1389] software

[1390] Image recognition library (e.g., OpenCV): A software library used to analyze image data and recognize specific objects or patterns.

[1391] Emotion recognition engine (e.g., Microsoft Azure Emotion API): A software engine for analyzing emotions from a user's facial expressions and voice.

[1392] Tourism information database (e.g., Google Places API): A database that stores information about tourist spots and buildings.

[1393] Generative AI model (e.g., GPT-4): An artificial intelligence model that generates information in natural language based on input data.

[1394] Processing steps

[1395] 1. Acquisition of image data

[1396] The contact lenses' built-in camera captures real-time image data of the scenery and buildings the user is looking at, and the smartphone receives and processes this data.

[1397] 2. Image Recognition

[1398] The smartphone uses an image recognition library (OpenCV) to analyze the acquired image data, thereby identifying the scenery and buildings the user is looking at.

[1399] 3. Emotion recognition

[1400] An emotion recognition engine (Microsoft Azure Emotion API) is used on the smartphone to analyze the user's emotions from their facial expressions and voice, thereby determining whether they are emotional.

[1401] 4. Information acquisition

[1402] Information about recognized landscapes and buildings is obtained from the tourist information database (Google Places API), including the history and characteristics of the place, and its reputation as a tourist attraction.

[1403] 5. Information generation

[1404] Using a generative AI model (GPT-4), information is generated in natural language based on the acquired information, providing users with information in a format that is easy for them to understand.

[1405] 6. Information provision

[1406] The smartphone provides the generated information to the user either audibly through earphones or visually using the display function of contact lenses.

[1407] 7. Emotion-based recommendations

[1408] If the user is impressed, the generative AI model will be used to suggest other tourist spots that will give a similar impression, further enriching the user's sightseeing experience.

[1409] Specific examples

[1410] If a user is moved by a sunset at the seaside, the system will recognize that emotion and visual information and suggest nearby beautiful sunset spots.

[1411] Prompt Sentence Examples

[1412] Using image data of the scenery the user is looking at as input, please use an image recognition library to recognize the scenery and provide information about that scenery. Also, analyze the user's emotions with an emotion recognition engine, and if the user is impressed, please suggest other tourist spots that will give them a similar impression.

[1413] In this way, a tourist guide system is realized that combines a contact lens equipped with a camera function and an emotion engine.

[1414] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1415] Step 1:

[1416] Acquisition of image data

[1417] Image data of the scenery and buildings that the user sees through a camera mounted on the contact lens is acquired in real time.

[1418] Input: Image data of the scenery and buildings the user is viewing.

[1419] Output: Image data acquired in real time.

[1420] How it works: The contact lens camera captures the user's field of vision and sends that data to a smartphone.

[1421] Step 2:

[1422] Image Recognition

[1423] The device (smartphone) uses an image recognition library (OpenCV) to analyze the acquired image data and identify the scenery and buildings the user is looking at.

[1424] Input: Image data acquired in real time.

[1425] Output: Information about recognized scenery and buildings.

[1426] What it does: Your smartphone inputs image data into OpenCV and runs algorithms to recognize specific objects and patterns.

[1427] Step 3:

[1428] emotion recognition

[1429] The device (smartphone) uses an emotion recognition engine (Microsoft Azure Emotion API) to analyze emotions from the user's facial expressions and voice.

[1430] Input: User's facial and voice data.

[1431] Output: Parsed user emotion information.

[1432] Specific operation: The smartphone inputs the user's facial and voice data into an emotion recognition engine and runs an algorithm to analyze emotions.

[1433] Step 4:

[1434] Information acquisition

[1435] The device (smartphone) retrieves information about recognized scenery and buildings from a tourist information database (Google Places API).

[1436] Input: Recognized scenery and building information.

[1437] Output: Detailed information about the scenery and buildings.

[1438] Specific operation: The smartphone inputs information about recognized scenery and buildings into the Google Places API and retrieves related tourist information.

[1439] Step 5:

[1440] information generation

[1441] The device (smartphone) uses a generative AI model (GPT-4) to generate information to provide to the user in natural language based on the information it has acquired.

[1442] Input: Detailed information about the scenery and buildings.

[1443] Output: Generated information in natural language.

[1444] Specific operation: Tourist information acquired by a smartphone is input into GPT-4, which generates natural language sentences to provide to the user.

[1445] Step 6:

[1446] Information provision

[1447] The device (smartphone) provides the generated information to the user either audibly through earphones or visually using the display function of contact lenses.

[1448] Input: Natural language generated information.

[1449] Output: Information provided to the user.

[1450] How it works: The smartphone sends the generated information to earphones for audio or contact lenses for visual display.

[1451] Step 7:

[1452] Emotion-based suggestions

[1453] Based on the user's emotions, the device (smartphone) suggests other tourist spots that will give a similar impression.

[1454] Input: Parsed user emotion information.

[1455] Output: Information on suggested other tourist spots.

[1456] Specific operation: Based on the emotion information, the smartphone uses a generative AI model to generate information to suggest other tourist spots and provides it to the user.

[1457] Example 3

[1458] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1459] Conventional information provision systems have had difficulty in providing appropriate information in real time according to the user's emotions. In addition, there was a lack of means to provide information quickly in urgent situations, such as when the user was confused or lost. This meant that users had to spend a lot of time and effort to obtain the information they needed.

[1460] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1461] In this invention, the server includes means for collecting biometric data of a user, means for analyzing the collected biometric data to identify the user's emotion, means for generating necessary information based on the identified emotion, and means for displaying the generated information in the user's field of view, thereby making it possible to provide appropriate information according to the user's emotion in real time.

[1462] "Means for collecting user biometric data" refers to devices and sensors for acquiring biometric information such as the user's heart rate, facial expression, and voice in real time.

[1463] "Means for analyzing collected biometric data to identify the user's emotions" refers to software or algorithms for analyzing and identifying the user's emotional state based on collected biometric data.

[1464] "Means for generating necessary information based on the identified emotion" refers to a system or program for generating appropriate information according to the user's emotional state.

[1465] "Means for displaying the generated information in the user's field of vision" refers to a device or technology for directly displaying the generated information in the user's field of vision.

[1466] This invention is a system that provides information corresponding to a user's emotions in real time. The system collects the user's biometric data, analyzes the data to identify the emotion, generates necessary information based on the identified emotion, and displays the generated information in the user's field of view.

[1467] Hardware and software used

[1468] Hardware

[1469] 1. Contact lenses: Contact lenses with retinal projection and display functions are used, specifically smart contact lenses and AR contact lenses.

[1470] 2. Sensors: The contact lenses are equipped with sensors to detect heart rate and facial expressions.

[1471] 3. Microphone: The contact lens has a built-in microphone to collect the user's voice.

[1472] software

[1473] 1. Emotion engine: Software that analyzes the user's biometric data to identify emotions. Specifically, it uses "EmotionAPI" and "Affectiva SDK."

[1474] 2. Map database and route calculation algorithm: Software that generates the optimal route based on the user's current location and destination. Specifically, Google Maps API is used.

[1475] 3. Communication protocol: A protocol for securely sending and receiving data between the server and the device. Specifically, "HTTPS" is used.

[1476] Specific operation of the system

[1477] Collecting user emotion data

[1478] The user wears contact lenses equipped with retinal projection and display functions. The device measures the user's heart rate with a built-in sensor and collects audio data with a microphone. For example, when the user mutters, "Oh no, I'm lost," the device collects audio data.

[1479] Emotional Data Analysis

[1480] The device sends the collected biometric data to a server, which uses the Emotion API to analyze the voice data and identify whether the user is confused, for example, by detecting fluctuations in voice tone or heart rate.

[1481] Generate the necessary information

[1482] The server obtains the user's current location from GPS data and generates the optimal route to the destination. For example, it uses Google Maps API to calculate the shortest route from the current location to the destination.

[1483] Sending information

[1484] The server sends the generated route information to the terminal using the HTTPS protocol, and the terminal processes the received route information and prepares it for display.

[1485] Displaying Information

[1486] The device uses retinal projection and display functions to display the route to the destination in the user's field of vision. For example, instructions such as "Turn right at the next intersection" are displayed in the upper right corner of the field of vision. This allows the user to visually confirm the information while heading towards the destination.

[1487] Examples and prompts

[1488] As a concrete example, consider a case where a user is lost and confused. When the emotion engine recognizes that the user is confused, the server analyzes the information and generates a route to the destination. The generated route information is sent to the device and displayed in the user's field of vision through the contact lenses.

[1489] Example prompt sentence:

[1490] "When the user is confused, create a program that displays the route to the destination in the user's field of vision."

[1491] This system allows the user to receive appropriate information according to their emotions in real time. For example, if the user is lost and confused, the system recognizes their emotions and displays the route to their destination directly in their field of vision. This allows the user to reach their destination quickly and efficiently. The flow of the identification process in Example 3 will be described with reference to FIG. 21.

[1492] Step 1: Collecting user biometric data

[1493] The user wears contact lenses equipped with retinal projection and display functions. The device measures the user's heart rate with a built-in sensor and collects voice data with a microphone. For example, when the user mutters, "Oh no, I'm lost," the voice data is collected. The input is the user's biometric data (heart rate, voice), and the output is the collected biometric data.

[1494] Step 2: Analyze the emotion data

[1495] The device sends the collected biometric data to the server, which then uses the Emotion API to analyze the voice data and identify whether the user is confused. For example, it can detect confusion from voice tone or heart rate fluctuations. The input is the collected biometric data, and the output is the identified emotional state (confusion).

[1496] Step 3: Generate the necessary information

[1497] The server obtains the user's current location from GPS data and generates the optimal route to the destination. For example, it uses the Google Maps API to calculate the shortest route from the current location to the destination. The input is the identified emotional state (confusion) and the user's current location, and the output is the generated route information.

[1498] Step 4: Submit your information

[1499] The server sends the generated route information to the terminal using the HTTPS protocol. The terminal processes the received route information and prepares it for display. The input is the generated route information, and the output is the route information ready for display.

[1500] Step 5: Viewing information

[1501] The device uses retinal projection and display functions to display the route to the destination in the user's field of view. For example, instructions such as "Turn right at the next intersection" are displayed in the upper right corner of the field of view. This allows the user to visually confirm the information while heading towards the destination. The input is route information that is ready to be displayed, and the output is the route information displayed in the user's field of view.

[1502] (Application example 3)

[1503] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1504] In conventional self-driving vehicles, there are limited ways for drivers and passengers to visually check road information and routes to their destinations in real time, which makes it difficult to quickly provide appropriate information, especially when users feel confused or anxious.In addition, there is also the problem that the user experience is not improved because information cannot be provided in accordance with the user's emotions.

[1505] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1506] In this invention, the server includes earphones and contact lenses, a means for linking them, a means for making a response to a user's desired request available to the ear or the eye, a means for recognizing the user's emotions, a means for displaying information in the user's field of vision according to the user's emotions, and a means for displaying road information and a route to a destination in the field of vision of a driver or passenger of an autonomous vehicle in real time. This makes it possible to quickly provide appropriate information and improve the user experience even when the user feels confused or anxious.

[1507] "Earphones" are small acoustic devices that deliver sound directly to the ears.

[1508] A "contact lens" is a thin lens that is worn on the surface of the eye for vision correction, decoration, or information display.

[1509] "Means of cooperation" refers to the technology or method that allows multiple devices or systems to communicate with each other and operate in cooperation.

[1510] "Means for making available to the user audio or visual responses to requests" refers to techniques or methods for providing audio or visual information in response to a user's request.

[1511] "Means for recognizing a user's emotions" refers to techniques or methods for analyzing a user's facial expressions, voice, behavior, etc. to determine their emotional state.

[1512] "Means for displaying information in the user's field of view according to the user's emotions" refers to techniques and methods for visually providing appropriate information based on the user's emotional state.

[1513] "Means for displaying road information and routes to destinations in real time to the driver or passengers of an autonomous vehicle" refers to technologies and methods that enable drivers and passengers in an autonomous vehicle to visually check the current road conditions and routes to destinations in real time.

[1514] A system for implementing this invention includes earphones, contact lenses, means for linking these, means for making a response to a user's desired request available to the ear or eye, means for recognizing the user's emotions, means for displaying information in the field of view according to the user's emotions, and means for displaying road information and routes to a destination in the field of view of a driver or passenger of an autonomous vehicle in real time.

[1515] 1. System Program

[1516] To recognize the user's emotions, the server captures the user's face using a camera and analyzes the user's emotions using an emotion recognition engine (EmotionRecognizer).If the user feels confused or anxious, the server obtains route information to the destination from a route planner (RoutePlanner) and displays it directly in the user's field of view through a display module (DisplayModule).

[1517] 2. Hardware and Software Used

[1518] This system uses hardware and software such as smart glasses (e.g., Google Glass), a camera, an emotion recognition engine, a route planner, and a display module. The camera captures the user's face, and the emotion recognition engine analyzes the image data to recognize emotions. The route planner calculates route information to the destination, and the display module displays that information in the user's field of view.

[1519] 3. Specific Examples

[1520] For example, if a user feels confused or anxious while riding in an autonomous vehicle, the system will recognize that emotion and display route information to the destination in real time in the user's field of vision, allowing the user to check the route to their destination with peace of mind.

[1521] Prompt Sentence Examples

[1522] "If the user is stumped, create an application that displays the route to their destination in their field of vision."

[1523] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1524] Step 1:

[1525] The server captures the user's face using the camera mounted on the smart glasses.

[1526] Input: Video data from the smart glasses camera

[1527] Output: User's face image

[1528] Specific operation: The camera captures video in real time and sends the video data to the server.

[1529] Step 2:

[1530] The server inputs the captured facial image into an emotion recognition engine (EmotionRecognizer) and analyzes the user's emotions.

[1531] Input: User's face image

[1532] Output: User's emotional state (e.g., confusion, anxiety, joy, etc.)

[1533] Specific operation: The emotion recognition engine analyzes facial images and determines emotions based on facial expressions and features.

[1534] Step 3:

[1535] If the server determines that the user's emotional state is confused or anxious, it requests route information to the destination from a route planner.

[1536] Input: User's emotional state

[1537] Output: Route information request

[1538] Specific operation: Based on the output of the emotion recognition engine, the server instructs the route planner to obtain route information.

[1539] Step 4:

[1540] The route planner calculates the optimal route based on the current location and destination information and sends that information back to the server.

[1541] Input: current location information, destination information

[1542] Output: Route information to the destination

[1543] How it works: The route planner uses map data and algorithms to calculate the optimal route and sends the results to the server.

[1544] Step 5:

[1545] The server sends the acquired route information to a display module (DisplayModule) and displays it in the user's field of view.

[1546] Input: Route information to destination

[1547] Output: Route information displayed in the field of view

[1548] Specific operation: The server sends route information to the display module, which displays the information in the user's field of view.

[1549] Step 6:

[1550] The user checks the route information displayed in their field of vision through the smart glasses and understands the route to their destination.

[1551] Input: Route information displayed in the field of view

[1552] Output: User understanding and behavior

[1553] Specific operation: The user checks the route to the destination based on the information displayed in their field of vision and travels with peace of mind.

[1554] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1555] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1556] Another example of generative AI is Gemini (internet search engine). <url: https: gemini.google.com ?hl="ja">) are mentioned.

[1557] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1558] [Third embodiment]

[1559] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1560] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1561] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1562] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1563] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1564] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1565] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1566] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1567] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1568] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1569] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1570] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[1571] "Example 1"

[1572] One embodiment of the present invention is a system that links earphones and contact lenses. This system allows a user to obtain answers to their desired requests through their ears or eyes. Specifically, when a user makes a voice inquiry, the system generates a response and provides the information through the earphones or visually through the contact lenses. "Embodiment Example 2"

[1573] In the future, contact lenses may be equipped with a camera, allowing the system to recognize the scenery or object the user is looking at and provide relevant information. For example, if the user is looking at a building, the system can provide information such as the building's name and history.

[1574] "Example 3"

[1575] In the future, contact lenses could also be equipped with retinal projection and display capabilities, allowing information to be displayed directly in the user's field of vision. For example, if a user is walking down a street, a map of the street and the route to their destination could be displayed.

[1576] The processing flow of each embodiment will be described below.

[1577] "Example 1"

[1578] Step 1: The user speaks a query.

[1579] Step 2: The system analyzes the user's query and generates an answer to it.

[1580] Step 3: The generated answer is presented to the user either audibly through earphones or visually through contact lenses.

[1581] "Example 2"

[1582] Step 1: The camera mounted on the contact lens captures the scenery or object the user is looking at.

[1583] Step 2: The system analyzes the captured image and generates relevant information.

[1584] Step 3: The generated information is presented to the user either audibly through earphones or visually through contact lenses.

[1585] "Example 3"

[1586] Step 1: The user speaks their destination.

[1587] Step 2: The system calculates the route to the input destination.

[1588] Step 3: The calculated route is displayed directly in the user's field of vision using the contact lens' built-in retinal projection and display capabilities.

[1589] Example 1

[1590] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1591] With conventional information provision systems, it is difficult for users to receive a quick and efficient response when making a voice inquiry. Furthermore, the limited visual means of providing information reduces user convenience. Furthermore, processing voice data and generating responses using generative AI models are complex, and a system that can manage these processes in an integrated manner is needed.

[1592] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1593] In this invention, the server includes a means for a user to make a voice inquiry, a means for a terminal to transmit voice data to the server, a means for the server to convert the voice data into text data, a means for the server to generate an answer using a generative AI model, a means for the server to convert the answer into voice data or image data, a means for the server to transmit the converted data to the terminal, and a means for the terminal to provide information to the user through earphones or contact lenses, thereby enabling the user to quickly and efficiently obtain information by voice or visually.

[1594] "Means for users to make voice inquiries" refers to a device or interface that allows users to input voice, such as a microphone built into earphones.

[1595] "Means for a terminal to transmit voice data to a server" refers to a device or software that has the function of transmitting voice data to a server via a communication network such as the Internet.

[1596] "Means by which the server converts voice data into text data" refers to software or algorithms that use voice recognition technology to convert voice data into text data.

[1597] "Means by which the server generates an answer using a generative AI model" refers to software or algorithms that use a generative AI model to generate appropriate answers based on text data.

[1598] "Means by which the server converts the response into voice data or image data" refers to software or algorithms for converting the generated text data into voice data or image data.

[1599] "Means for transmitting data converted by the server to the terminal" refers to a device or software that has the function of transmitting converted audio data or image data to the terminal via a communication network such as the Internet.

[1600] "Means by which a terminal provides information to a user through earphones or contact lenses" refers to devices or software that have the function of providing received audio data as audio through earphones or the function of providing image data visually through contact lenses.

[1601] MODE FOR CARRYING OUT THE INVENTION

[1602] This invention is a system in which a user makes a voice inquiry and receives a response via voice or visually through earphones or contact lenses. This system is implemented using the following hardware and software.

[1603] Hardware and Software Configuration

[1604] 1. User terminal: A device equipped with a microphone for user voice input, such as a smartphone or dedicated device.

[1605] 2. Server: A high-performance computer for processing voice data, generating answers using generative AI models, and converting and transmitting data.

[1606] 3. Speech recognition software: Software for converting voice data into text data, such as Google Cloud Speech-to-Text.

[1607] 4. Generative AI models: AI models for generating appropriate answers based on text data, such as OpenAI's GPT-4.

[1608] 5. Speech synthesis software: Software for converting text data into speech data, such as Amazon Polly.

[1609] 6. Contact Lenses: Contact lenses that have a display function to provide visual information.

[1610] System Operation

[1611] When a user asks a question by voice through the microphone built into the earphones, the device captures the voice data and sends it over the internet to a server. The server converts the voice data into text data using Google Cloud Speech-to-Text. The server then uses a generative AI model (GPT-4) to generate an appropriate answer based on the text data.

[1612] The generated answer is converted into either audio or image data, depending on the user's selection. In the case of audio data, the server uses Amazon Polly to convert the text data into audio data. In the case of image data, the server converts the text into a format for visual display. The converted data is then sent back to the device via the Internet.

[1613] The terminal provides the received audio data audibly through earphones or visually through contact lenses, allowing the user to quickly and efficiently obtain information audibly or visually.

[1614] Specific examples

[1615] For example, if a user speaks into the earphones, "What's the weather today?", the device captures the audio and sends it to the server. The server converts the audio to text using Google Cloud Speech-to-Text, generating the text "What's the weather today?". The server then uses GPT-4 to generate the answer "The weather is sunny today." If the user selects the earphones, the server converts the text to speech using Amazon Polly and sends the audio data to the device. The device then provides the audio "The weather is sunny today" to the user through the earphones.

[1616] Example prompt sentence:

[1617] User: "What's the weather like today?"

[1618] Server: "The weather is sunny today."

[1619] The system allows users to obtain information quickly and efficiently, either audibly or visually.

[1620] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1621] Step 1:

[1622] The user makes a voice inquiry.

[1623] The user asks a question by voice through the microphone built into the earphones. For example, "What's the weather like today?" This voice data becomes the input.

[1624] Step 2:

[1625] The terminal transmits the voice data to the server.

[1626] The terminal captures the user's voice data and transmits it to a server via the Internet, where the input is the user's voice data and the output is the voice data transmitted to the server.

[1627] Step 3:

[1628] The server converts the voice data into text data.

[1629] The server uses speech recognition software to convert the received voice data into text data, for example, using Google Cloud Speech-to-Text. In this case, the input is voice data, and the output is text data such as "What's the weather like today?"

[1630] Step 4:

[1631] The server generates an answer using a generative AI model.

[1632] The server uses a generative AI model to generate an appropriate answer based on the text data. For example, OpenAI's GPT-4 is used. In this case, the input is the text data "What's the weather like today?" and the output is the answer text "Today's weather is sunny."

[1633] Step 5:

[1634] The server converts the response into audio or image data.

[1635] The server converts the generated text data into either voice data or image data, depending on the user's selection. For voice data, the server uses speech synthesis software, such as Amazon Polly. For image data, the server converts the text into a format for visual display. In this case, the input is the text data "Today's weather is sunny," and the output is either voice data or image data.

[1636] Step 6:

[1637] The server transmits the converted data to the terminal.

[1638] The server transmits the converted audio data or image data to the terminal via the Internet. At this time, the input is the audio data or image data, and the output is the data transmitted to the terminal.

[1639] Step 7:

[1640] The device provides information to the user through earphones or contact lenses.

[1641] The data received by the device is provided either audibly through earphones or visually through contact lenses. For example, the earphones may say "Today's weather is sunny" or the contact lenses may display the text "Today's weather is sunny." In this case, the input is the data sent from the server, and the output is the information provided to the user.

[1642] (Application example 1)

[1643] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1644] In autonomous vehicles, users are required to quickly and accurately obtain destination and route information. It is also important to grasp traffic conditions and vehicle status in real time and notify users at the appropriate time. However, conventional systems lack efficient means to provide this information, resulting in reduced user convenience. Furthermore, the lack of integration between visual and auditory information provision can cause confusion when users obtain information.

[1645] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1646] In this invention, the server includes earphones and contact lenses, a means for linking them, a means for making a response to a user's desired request available to the user aurally or visually, a means for recognizing the user's voice input and calculating optimal route information using a generative AI model, a means for providing voice guidance of the calculated route information, and a means for visually displaying important information through the contact lenses. This allows the user to quickly and accurately obtain route information through voice input and visually confirm important information.

[1647] An "earphone" is a small acoustic device that delivers sound directly to the user's ears.

[1648] A "contact lens" is a thin lens that is worn on a user's eye to provide visual information.

[1649] "Means for linking" refers to the technical means by which the earphones and contact lenses communicate with each other and share information.

[1650] "Means for making available to the user audibly or visually a response to a desired request" means means for providing information generated based on the user's voice input audibly through earphones or visually through contact lenses.

[1651] "Means for recognizing a user's voice input and calculating optimal route information using a generative AI model" refers to means for analyzing a user's voice and calculating an optimal route using a generative AI model.

[1652] The "means for providing audio guidance of calculated route information" refers to a means for conveying calculated route information to the user by audio.

[1653] "Means for visually displaying important information through contact lenses" refers to means for displaying important information such as traffic conditions and warnings on contact lenses.

[1654] The system for implementing this invention links earphones and contact lenses, and when a user makes a voice inquiry, it uses a generative AI model to calculate optimal route information, provides voice guidance through the earphones, and visually displays important information through the contact lenses.

[1655] Hardware and software used

[1656] Hardware:

[1657] Smartphone or smart glasses

[1658] earphones

[1659] Contact Lens Display

[1660] software:

[1661] speech_recognition library: for performing speech recognition

[1662] pyttsx3 library: for speech synthesis

[1663] The requests library: to obtain route information

[1664] some_contact_lens_display_library: To display information on a contact lens display

[1665] System Operation

[1666] 1. Voice input recognition:

[1667] The user inputs their destination by voice through the microphone attached to their smartphone or smart glasses. The voice data is converted into text using a speech recognition library (speech_recognition).

[1668] 2. Calculating route information:

[1669] The recognized text data is input into a generative AI model to calculate the optimal route information, which is obtained through an external route planning API (using the requests library).

[1670] 3. Voice guidance:

[1671] The calculated route information is converted into voice using a speech synthesis library (pyttsx3) and guided to the user through earphones.

[1672] 4. Visual Indication:

[1673] Display important traffic information and warnings on the contact lens display using some_contact_lens_display_library.

[1674] Specific examples

[1675] When a user voice-inquires, "Tell me the route to Tokyo Station," the system calculates the optimal route and announces through the earphones, "The optimal route has been calculated. We are now departing." The system also displays route information and important traffic information on the contact lens display.

[1676] Prompt Sentence Examples

[1677] "Tell me the route to Tokyo Station."

[1678] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1679] Step 1:

[1680] The user inputs the destination by voice through a microphone attached to the smartphone or smart glasses.

[1681] Input: User's voice data

[1682] Processing: Use the speech recognition library (speech_recognition) to convert the audio data into text.

[1683] Output: Text data (e.g., "Please tell me the route to Tokyo Station.")

[1684] Step 2:

[1685] The server receives the text data and inputs it into the generative AI model.

[1686] Input: Text data

[1687] Processing: Uses a generative AI model to calculate optimal route information.

[1688] Output: Route information data (e.g., optimal route, travel time, traffic conditions, etc.)

[1689] Step 3:

[1690] The server obtains the route information data and converts it into voice using a voice synthesis library (pyttsx3).

[1691] Input: Route information data

[1692] Processing: Convert the route information into speech using a speech synthesis library.

[1693] Output: Voice data (e.g. "Optimal route calculated. Let's go.")

[1694] Step 4:

[1695] The terminal provides the user with audio data through earphones.

[1696] Input: Audio data

[1697] Processing: Play audio data through earphones.

[1698] Output: The user hears the voice prompts.

[1699] Step 5:

[1700] The server displays important traffic information and warnings on the contact lens display.

[1701] Input: Route information data and additional traffic information data

[1702] Processing: Use some_contact_lens_display_library to visually display important information.

[1703] Output: Visual information displayed on the contact lens display (e.g., route arrows, traffic warnings, etc.)

[1704] Step 6:

[1705] The user views visual information through the contact lens display.

[1706] Input: Visual information displayed on a contact lens display

[1707] Action: The user checks the visual information and takes the necessary action.

[1708] Output: User understands visual information and acts upon it

[1709] Example 2

[1710] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1711] In conventional systems, it was difficult to obtain information about the objects and scenery that the user was looking at in real time. Also, the means by which the user could obtain answers to their desired requests by ear or by sight were limited, making it cumbersome to obtain information. Furthermore, there was a lack of a means to efficiently provide information related to the objects and scenery that the user was looking at.

[1712] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for receiving video data from the contact lenses, means for analyzing the received video data and identifying the viewed object or scenery, and means for using a generative AI model for generating information related to the identified object or scenery. This makes it possible to obtain information about the object or scenery the user is viewing in real time and provide it efficiently.

[1713] "Earphones" are small acoustic devices that deliver sound directly to the ears.

[1714] A "contact lens" is a thin, transparent or translucent lens that is worn directly on the eye and is used to correct vision or provide a specific function.

[1715] "Means of cooperation" refers to the technology or method that enables multiple devices or systems to communicate with each other and operate in cooperation.

[1716] "Means for making available to the user audibly or visually a response to a request" refers to techniques or methods for providing the information desired by the user audibly or visually.

[1717] "Means for receiving video data" refers to the technology or method for receiving video information obtained from a camera or sensor.

[1718] "Means for analyzing video data and identifying the object or scene being viewed" refers to techniques or methods for processing received video information and identifying the object or scene being viewed by the user.

[1719] A "means of using a generative AI model" is a technique or method for using artificial intelligence to perform a specific task, and is used here to generate relevant information.

[1720] The "means for providing information to users" refers to the techniques and methods for transmitting the generated information to users.

[1721] MODE FOR CARRYING OUT THE INVENTION

[1722] This invention is a system that incorporates a camera function into contact lenses worn by a user, recognizes the scenery and objects the user is looking at, and provides related information. A specific embodiment of this system is described below.

[1723] Hardware and software used

[1724] Contact Lenses: Contact lenses are equipped with a camera that can capture the scenery and objects the user is looking at in real time.

[1725] Terminal: A device such as a smartphone or smart glasses is used. These terminals receive video data from the contact lenses and send it to the server.

[1726] Server: A server with high-performance computing power is used to analyze the received video data and identify the objects and scenery being viewed.

[1727] Image recognition software: Uses image recognition software such as OpenCV and TensorFlow, which can analyze video data and identify objects and scenery.

[1728] Generative AI models: Use generative AI models such as GPT-4 to generate information related to identified objects and scenes.

[1729] Program processing

[1730] The server receives the video data sent from the device. The received video data is analyzed using image recognition software. As a result of the analysis, the object or scene the user is looking at is identified. The server then uses a generative AI model to generate information related to the identified object or scene. This information is provided to the user via the device.

[1731] Specific examples

[1732] For example, if a user is looking at a building, the contact lens camera captures video of the building. The video data is sent via the device to a server, which uses image recognition software to identify the building. The server then uses a generative AI model to generate information about the building (e.g., "This building is a historic landmark and was built in the 19th century.") This information is then provided to the user via the device.

[1733] Prompt Sentence Examples

[1734] "Please tell us the name and history of the building you are viewing. The video data for the building is as follows."

[1735] By inputting this prompt into a generative AI model, detailed information about the building the user is looking at can be generated.

[1736] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1737] Program processing flow

[1738] Step 1:

[1739] A user wears the contact lenses and views a scene or object.

[1740] Input: Scenery and objects within the user's field of vision.

[1741] Output: Video data captured by the contact lens camera.

[1742] How it works: When a user wears contact lenses and looks at, say, Tokyo Tower, the camera in the contact lenses captures the image in real time.

[1743] Step 2:

[1744] The terminal receives the video data from the contact lens.

[1745] Input: Video data transmitted from contact lenses.

[1746] Output: Video data stored on the device.

[1747] Specific operation: A device such as a smartphone or smart glasses receives video data of Tokyo Tower transmitted from the contact lenses.

[1748] Step 3:

[1749] The terminal transmits the video data to the server.

[1750] Input: Video data stored on the device.

[1751] Output: Video data sent to the server.

[1752] Specific operation: The device sends the received video data of Tokyo Tower to the server via Wi-Fi or mobile data communication.

[1753] Step 4:

[1754] The server analyzes the video data and identifies the objects and scenery being viewed.

[1755] Input: Video data sent to the server.

[1756] Output: Information about identified objects and scenery.

[1757] Specific operation: The server uses image recognition software such as OpenCV or TensorFlow to analyze the received video data of Tokyo Tower and identify it as Tokyo Tower.

[1758] Step 5:

[1759] The server uses the generative AI model to generate relevant information.

[1760] Input: Identified object and scene information.

[1761] Output: The relevant information generated.

[1762] Specific operation: The server uses a generative AI model such as GPT-4 to generate information about the identified Tokyo Tower (for example, "Tokyo Tower was completed in 1958 and is 333 meters tall").

[1763] Step 6:

[1764] The server transmits the generated information to the terminal.

[1765] Input: Generated relevant information.

[1766] Output: Relevant information sent to the terminal.

[1767] Specific operation: Information about Tokyo Tower generated by the server is sent to the device via Wi-Fi or mobile data communication.

[1768] Step 7:

[1769] The terminal provides the information to the user.

[1770] Input: Relevant information sent to the terminal.

[1771] Output: Information provided to the user.

[1772] Specific operation: The information about Tokyo Tower received by the device is displayed to the user on the screen of their smartphone or smart glasses, allowing the user to check detailed information about Tokyo Tower in real time.

[1773] (Application example 2)

[1774] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1775] Conventional systems have had difficulty obtaining information about the object a user is looking at in real time. Furthermore, there are limited ways for users to obtain answers to their requests by ear or sight, which hinders the user experience. Furthermore, there is a lack of ways to instantly provide detailed product information in physical stores.

[1776] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1777] In this invention, the server includes earphones and contact lenses, a means for linking them, a means for making a response to a user's desired request available to the user's ears or eyes, a means for recognizing an object the user is looking at and providing information about the object, and a means for displaying the information in real time. This allows the user to obtain information about the object they are looking at in real time and to obtain a response to a user's desired request to the user's ears or eyes. Furthermore, detailed product information can be provided instantly in a physical store, improving the user experience.

[1778] "Earphones" are small audio devices that deliver sound directly to the ears.

[1779] A "contact lens" is a thin lens that is placed directly on the surface of the eye and is used for vision correction, decoration, or to provide a specific function.

[1780] "Means of cooperation" are methods and technologies that allow multiple devices or systems to exchange information with each other and work in cooperation.

[1781] "Means for providing audible or visual responses to user requests" refers to techniques or methods that provide audio or visual responses to information or instructions requested by users.

[1782] "Means for recognizing an object that a user is looking at and providing information about that object" refers to a technology or method for identifying an object that comes into the user's field of vision and obtaining and providing data or information related to that object.

[1783] The "means for displaying the information in real time" refers to a technique or method for visually presenting acquired information to a user immediately without delay.

[1784] A system for implementing the present invention includes earphones, contact lenses, a smartphone or smart glasses, a server, and software for linking these components. Specific embodiments are described below.

[1785] Hardware and software used

[1786] Earphone: A small acoustic device that delivers sound to the user.

[1787] Contact lenses: Thin lenses with built-in camera functions.

[1788] Smartphone or smart glasses: A device that performs image recognition and displays information.

[1789] Server: A central system that manages databases and provides information.

[1790] Image recognition software: Google Cloud Vision API, etc.

[1791] Database management system: MySQL, etc.

[1792] Information delivery system: Custom web application.

[1793] Program processing

[1794] The server includes the following means:

[1795] 1. Image capture means: The contact lenses capture objects that come into the user's field of view.

[1796] 2. Image transmission means: Transmit the captured image to a smartphone or smart glasses.

[1797] 3. Image Recognition: Image recognition software on a smartphone or smart glasses analyzes the image and identifies the object.

[1798] 4. Database query means: Retrieving information about the identified object from a database.

[1799] 5. Information provision means: The acquired information is displayed in real time on the user's smartphone or smart glasses.

[1800] Specific examples

[1801] Let's say a user is looking at a particular shampoo in a brick-and-mortar store. The contact lens captures an image of the shampoo and sends it to their smartphone. The Google Cloud Vision API analyzes the image and identifies the shampoo. It retrieves information about the shampoo (price, ingredients, reviews, etc.) from a MySQL database and displays it on their smartphone.

[1802] Prompt Sentence Examples

[1803] Design an application that identifies the product a user is looking at and provides detailed information about that product in real time. The hardware required is a contact lens with a built-in camera and a smartphone. The software required is Google Cloud Vision API and MySQL. Please explain the specific process, including a concrete example where the user is looking at shampoo.

[1804] In this way, users can get real-time information about the object they are looking at, hear or see the answer to their desired request, and instantly get detailed product information in a physical store, improving the user experience.

[1805] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1806] Step 1:

[1807] Image Capture

[1808] A user wears contact lenses and browses products in a physical store. The contact lens's camera captures the products in the user's field of view.

[1809] Input: Products in the user's field of view

[1810] Output: Image data of the captured product

[1811] How it works: A camera built into the contact lens periodically scans the field of view and captures an image of any products it detects.

[1812] Step 2:

[1813] Image transmission

[1814] The captured image data is transmitted to a smartphone or smart glasses.

[1815] Input: Captured product image data

[1816] Output: Image data sent to a smartphone or smart glasses

[1817] How it works: Image data is sent from the contact lens to a smartphone or smart glasses via Bluetooth or Wi-Fi.

[1818] Step 3:

[1819] Image Recognition

[1820] Image recognition software (such as Google Cloud Vision API) on a smartphone or smart glasses analyzes the image and identifies the product.

[1821] Input: Image data sent to a smartphone or smart glasses

[1822] Output: Identification information of the identified product

[1823] How it works: The Google Cloud Vision API sends image data to the cloud, analyzes it to identify the product, and returns the product's identification information as the analysis result.

[1824] Step 4:

[1825] Database queries

[1826] Based on the identification information of the identified product, the server retrieves information about the product from a database (e.g., MySQL).

[1827] Input: Identification information of the identified product

[1828] Output: Product details (price, ingredients, reviews, etc.)

[1829] What happens: The server queries the MySQL database to retrieve details of products that match the identification information.

[1830] Step 5:

[1831] Information provision

[1832] Detailed information about the acquired products is displayed in real time on the user's smartphone or smart glasses.

[1833] Input: Product details (price, ingredients, reviews, etc.)

[1834] Output: Product details displayed on a smartphone or smart glasses

[1835] Specific operation: The detailed information of the acquired product is visually displayed on the display of the smartphone or smart glasses, and the user can check it.

[1836] In this way, users can get real-time information about the object they are looking at, hear or see the answer to their desired request, and instantly get detailed product information in a physical store, improving the user experience.

[1837] Example 3

[1838] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1839] Conventional navigation systems require users to hold a smartphone or other device in their hand to visually check information, which reduces safety and convenience while traveling. Furthermore, detailed information is difficult to grasp using voice guidance alone. To solve these issues, a system that displays information directly in the user's field of vision is needed.

[1840] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1841] In this invention, the server includes means for inputting a request desired by the user, means for transmitting the request to the server, means for the server to obtain the user's location information, means for the server to calculate a route using the map information providing means, means for the server to transmit the route information calculated by the server to the terminal, means for the terminal to analyze the route information and convert it into display data, and means for displaying information in the user's field of view, thereby enabling the user to reach their destination efficiently and safely using the information displayed directly in their field of view.

[1842] The "means for inputting a user's desired request" refers to a device or software that provides an interface for the user to input their destination and necessary information.

[1843] The "means for transmitting the request to the server" refers to a communication device or protocol for transmitting the request input by the user to the server via a network.

[1844] The "means by which the server obtains the user's location information" refers to technology for identifying the user's current location using a GPS device or other location information acquisition device.

[1845] "Means for the server to calculate a route using map information providing means" refers to a technology that uses an API or software that provides map information to calculate the optimal route from the user's current location to the destination.

[1846] The "means for transmitting route information calculated by the server to the terminal" refers to a communication device or protocol for transmitting route information calculated by the server to the user's terminal via a network.

[1847] The "means by which the terminal analyzes route information and converts it into display data" refers to software or hardware that analyzes the received route information and converts it into a data format that can be displayed in the user's field of view.

[1848] "Means for displaying information in the user's field of vision" refers to a device for displaying information directly in the user's field of vision using a display built into a contact lens or retinal projection technology.

[1849] This invention provides a system that displays information directly in the user's field of view. This system includes a series of processes for inputting a user's desired request, calculating optimal route information based on the request, and displaying the information in the user's field of view.

[1850] First, the user inputs a prompt such as "Show me the route to a new restaurant" into the device. This prompt is used when the user wants to know the route to a destination. The device sends this prompt to the server. The server uses a GPS device to obtain the user's current location information. Next, the server uses a general map information API as a means of providing map information to calculate the optimal route from the user's current location to the destination.

[1851] The calculated route information is sent from the server to the device. The device analyzes the received route information and converts it into data for display on the contact lenses. This data is then converted into a format that is displayed directly in the user's field of vision using retinal projection technology. Finally, the display built into the user's contact lenses displays the converted route information directly in the user's field of vision. This allows the user to head to their destination while viewing the route information displayed in their field of vision.

[1852] As a concrete example, consider the case where a user wants to go to a new restaurant. The user types "Show me the route to the new restaurant" into the device. The device sends this prompt to the server. The server obtains the user's current location information and calculates the optimal route using a map information API. The calculation results are sent to the device, which analyzes the information and converts it into data to be displayed on the contact lenses. Finally, the route information is displayed in the user's field of vision, allowing the user to reach their destination without getting lost.

[1853] This system enables the user to reach the destination efficiently and safely by using the information displayed directly in the user's field of view. The flow of the identification process in the third embodiment will be described with reference to FIG.

[1854] Step 1:

[1855] The user types the prompt into the terminal.

[1856] When a user wants to know the route to a destination, the user inputs a prompt sentence into the terminal, such as "Show me the route to a new restaurant." The input prompt sentence is acquired through an input device of the terminal.

[1857] Step 2:

[1858] The terminal sends the prompt to the server.

[1859] The terminal sends the prompt text entered by the user to the server via the network. At this time, the user's current location information is also sent along with the prompt text. The input is the prompt text and location information, and the output is the data to be sent to the server.

[1860] Step 3:

[1861] The server obtains the user's location information.

[1862] The server receives the user's current location information sent from the terminal and uses it for the next process. The input is the location information from the terminal, and the output is the acquired location information.

[1863] Step 4:

[1864] The server calculates the route using the map information providing means.

[1865] The server uses a general map information API to provide map information and calculates the optimal route from the user's current location to the destination. The input is location information and destination information, and the output is the calculated route information.

[1866] Step 5:

[1867] The server sends the calculated route information to the terminal.

[1868] The server transmits the calculated route information to the user's terminal via the network. The input is the calculated route information, and the output is the data to be transmitted to the terminal.

[1869] Step 6:

[1870] The terminal analyzes the route information and converts it into display data.

[1871] The terminal analyzes the received route information and converts it into data to be displayed on the contact lens. The input is the received route information, and the output is the display data.

[1872] Step 7:

[1873] Display information in the user's field of view.

[1874] The display built into the user's contact lenses displays the converted route information directly in the user's field of vision. The input is display data, and the output is information displayed in the user's field of vision. This allows the user to head to their destination while viewing the route information displayed in their field of vision.

[1875] (Application example 3)

[1876] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1877] To safely and efficiently reach their destinations, autonomous vehicle drivers need to be provided with accurate, real-time visual navigation information. However, conventional navigation systems require drivers to take their eyes off the road, which compromises safety. Furthermore, voice assistant functionality is not fully integrated, requiring the driver to manually operate the system, which also reduces safety. To address these challenges, a system that allows drivers to obtain the necessary information without having to avert their eyes is needed.

[1878] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1879] In this invention, the server includes earphones and contact lenses, a means for linking them, a means for making responses to user requests available to the ears or eyes, a means for displaying navigation information in the field of view of the driver of the autonomous vehicle when the driver wears the contact lenses, and a means for equipping the driver with a voice assistant function and displaying information in response to the driver's voice commands. This enables the driver to obtain navigation information in real time without averting their eyes, and to reach their destination safely and efficiently.

[1880] An "earphone" is a small acoustic device that transmits sound directly to the ear.

[1881] A "contact lens" is a thin lens that is worn directly on the surface of the eye to correct vision or display information.

[1882] "Means of cooperation" refers to the technology or method that enables multiple devices or systems to communicate with each other and operate in cooperation.

[1883] "Means for making available to the user audio or visual responses to requests" refers to techniques or methods for providing information audibly or visually in response to a user's request.

[1884] "Means for displaying navigation information in the field of vision of the driver of an autonomous vehicle by wearing contact lenses" refers to technologies and methods for visually providing route and traffic information to the destination through contact lenses worn by the driver of an autonomous vehicle.

[1885] "Means for being equipped with a voice assistant function and displaying information in response to the driver's voice commands" refers to technologies and methods for using voice recognition technology to understand the driver's voice instructions and provide the necessary information based on them.

[1886] The following describes an embodiment of the present invention. The system components include earphones, contact lenses, a server, and a smartphone. These devices work together to provide real-time navigation information to the user.

[1887] System Program

[1888] The server implements the system using the following hardware and software.

[1889] Hardware: Smartphone, retinal projection contact lenses, earphones

[1890] Software: OpenCV, pyttsx3, Navigation API

[1891] Program processing explanation

[1892] When the user inputs a destination, the server uses the navigation API to obtain route and traffic information to the destination. The obtained information is overlaid on the view using OpenCV. Furthermore, the voice assistant reads the information aloud using pyttsx3.

[1893] Specifically, when a user enters a destination into their smartphone, the server calls the navigation API to obtain route and traffic information. Next, to display the obtained information on the contact lenses, OpenCV is used to overlay the information on the field of view. At the same time, using pyttsx3, the voice assistant reads out, "The route to your destination is XX. The current traffic conditions are △△."

[1894] Specific examples

[1895] For example, if a user inputs "Tokyo Tower" as their destination, the server will use the navigation API to obtain route and traffic information to "Tokyo Tower." The obtained information is displayed in the user's field of vision through the contact lenses, and the voice assistant will read out, "The route to Tokyo Tower is ____. The current traffic conditions are ____."

[1896] Prompt Sentence Examples

[1897] Enter your destination: Tokyo Tower

[1898] Getting route information...

[1899] Getting traffic information...

[1900] Displaying information in your field of view...

[1901] Your voice assistant is reading out information...

[1902] In this way, the user can obtain real-time navigation information without having to look away, enabling them to reach their destination safely and efficiently.

[1903] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1904] Step 1:

[1905] The user inputs the destination into the smartphone.

[1906] Input: Destination (e.g. Tokyo Tower)

[1907] Output: Destination information

[1908] Specific operation: The user starts the smartphone application and inputs the destination. The input destination information is sent to the server.

[1909] Step 2:

[1910] The server calls the navigation API to obtain route and traffic information.

[1911] Input: Destination information

[1912] Output: Route information, traffic information

[1913] Specific operation: The server sends destination information to the navigation API, obtains route information to the destination and current traffic information, and stores the obtained information within the server.

[1914] Step 3:

[1915] The server processes the route and traffic information it has acquired to display it on the contact lenses.

[1916] Input: Route information, traffic information

[1917] Output: Data for display

[1918] Specific operation: The server uses OpenCV to process the acquired route and traffic information into data for visual display. The processed data is then sent to the contact lens.

[1919] Step 4:

[1920] The contact lenses overlay route and traffic information onto your field of vision.

[1921] Input: Data to display

[1922] Output: Visual navigation information

[1923] How it works: The contact lenses receive display data sent from the server and overlay it on the user's field of view in real time, allowing the user to check navigation information without looking away.

[1924] Step 5:

[1925] The server uses a voice assistant to read out route and traffic information.

[1926] Input: Route information, traffic information

[1927] Output: Voice navigation information

[1928] Specific operation: The server uses pyttsx3 to read out the acquired route and traffic information. The voice assistant tells the user, "The route to your destination is XX. The current traffic conditions are △△."

[1929] Step 6:

[1930] The user requests additional information using voice commands.

[1931] Input: Voice command

[1932] Output: Additional information

[1933] How it works: When a user issues a voice command, the server uses voice recognition technology to interpret the command and provide any additional information needed, such as displaying information in response to a request like "Tell me where the nearest gas station is."

[1934] In this way, the user can obtain real-time navigation information without having to look away, enabling them to reach their destination safely and efficiently.

[1935] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1936] "Example 1"

[1937] The invention, combined with the emotion engine, recognizes a user's emotions and provides information accordingly. Specifically, when a user feels joy, the system recognizes that emotion and provides information that is likely to bring joy to the user. For example, when a user is excited about a music festival, the system recognizes that emotion and provides information about upcoming concerts by the same artist.

[1938] "Example 2"

[1939] The invention, which combines a camera function and an emotion engine, simultaneously recognizes what the user is looking at and the emotion they are feeling, enabling it to provide information accordingly. Specifically, when a user is moved by a beautiful landscape, the system recognizes that emotion and visual information and provides them with places with similarly beautiful scenery. For example, when a user is moved by a sunset by the sea, the system recognizes that emotion and visual information and provides them with nearby famous places for beautiful sunsets.

[1940] "Example 3"

[1941] The invention, which combines retinal projection, display functionality, and an emotion engine, visually provides information according to the user's emotions. Specifically, when the user is confused, the system recognizes that emotion and displays the necessary information directly in the user's field of vision. For example, when the user is lost and confused, the system recognizes that emotion and displays the route to the destination directly in the user's field of vision.

[1942] The processing flow of each embodiment will be described below.

[1943] "Example 1"

[1944] Step 1: Recognize the user's emotions with the emotion engine.

[1945] Step 2: Based on the recognized emotions, select information that is likely to bring joy to the user.

[1946] Step 3: Present the selected information to the user audibly or visually.

[1947] "Example 2"

[1948] Step 1: Obtain the user's visual information using the camera function.

[1949] Step 2: Recognize the user's emotions with the emotion engine.

[1950] Step 3: Select relevant information based on the acquired visual information and perceived emotions.

[1951] Step 4: Present the selected information to the user audibly or visually.

[1952] "Example 3"

[1953] Step 1: Recognize the user's emotions with the emotion engine.

[1954] Step 2: Select the information you need based on the emotions you recognize.

[1955] Step 3: The selected information is displayed directly in the user's field of vision using retinal projection and display functions.

[1956] Example 1

[1957] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1958] Conventional information provision systems have had difficulty in providing prompt and appropriate responses to user requests. Furthermore, they were unable to provide information in response to the user's emotions, limiting the user experience. Furthermore, there was a lack of visual information provision methods, limiting the means by which users could obtain information.

[1959] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1960] In this invention, the server includes means for converting voice data into text data using voice recognition software, means for generating answers based on the text data using a generative AI model, and means for analyzing the user's emotions using an emotion engine and providing information according to the emotions. This enables prompt and appropriate provision of answers to user requests, and also realizes provision of information according to the user's emotions. In addition, by including means for providing information audibly through earphones and means for providing information visually through contact lenses, the means by which users can obtain information are diversified, improving the user experience.

[1961] "Earphones" are acoustic devices that deliver sound directly to the user's ears.

[1962] A "contact lens" is an optical device that is worn on a user's eye to provide visual information.

[1963] "Means for linking" refers to the technical means for enabling the earphones and contact lenses to communicate with each other and share information.

[1964] "Speech recognition software" is a program for converting voice data into text data.

[1965] A "generative AI model" is an artificial intelligence model that generates appropriate answers based on input text data.

[1966] An "emotion engine" is software that analyzes a user's emotions and provides information according to those emotions.

[1967] The "means for providing by voice" refers to a technical means for conveying the generated answer to the user as voice.

[1968] The "visual presentation means" refers to a technical means for displaying the generated answer to the user as visual information.

[1969] "Retinal projection" is a technology that projects information directly onto the user's retina through contact lenses.

[1970] The "display function" is a function that provides visual information to the user through the contact lenses.

[1971] This invention is a system that links earphones and contact lenses to enable a user to obtain answers to requests they desire through their ears or eyes. Specific embodiments of this system will be described below.

[1972] First, the user makes a voice inquiry through the earphone microphone. For example, they might ask, "What events do you recommend for next weekend?" This voice data is sent to the server.

[1973] The server uses speech recognition software (e.g., Google Cloud Speech-to-Text) to convert the user's voice data into text data. It captures the voice signal as digital data, analyzes the data, and converts it into a string of characters.

[1974] The server then uses a generative AI model (e.g., OpenAI GPT-4) to generate appropriate answers based on the text data. For example, for the text "What events do you recommend for next weekend?", the AI ​​model generates an answer such as "There's a music festival in the city next weekend."

[1975] The generated answers are analyzed using an emotion engine (e.g., IBM Watson Tone Analyzer). The emotion engine recognizes emotions from the user's voice and provides information corresponding to those emotions. For example, if the user expresses joy, the server provides additional information that the user is likely to find joyful. Specifically, if the server recognizes that the user is excited about a music festival, it provides additional information about upcoming concerts by the same artist.

[1976] The generated answers are provided either audibly through earphones or visually through contact lenses. Augmented reality (AR) technology (e.g., Microsoft HoloLens) is used to provide visual information. For example, if the user is using earphones, the server may announce aloud, "There's a music festival in your city next weekend." If the user is using contact lenses, detailed information about the event is displayed visually.

[1977] As a concrete example, consider a case where a user makes a voice inquiry such as, "What events do you recommend for next weekend?" In this case, the server performs the following process.

[1978] 1. A user asks by voice, "What events do you recommend for next weekend?"

[1979] 2. The server uses voice recognition software to convert the voice data into text data.

[1980] 3. Use a generative AI model to generate appropriate answers based on the text data.

[1981] 4. Use an emotion engine to analyze the user's emotions and provide information according to their emotions.

[1982] 5. Provide generated answers audibly through earphones or visually through contact lenses.

[1983] Example prompt sentence:

[1984] "When a user asks by voice what events are recommended for the upcoming weekend, create a program that generates appropriate responses and provides information based on the user's emotions."

[1985] In this way, a system is realized that provides quick and appropriate responses to user requests.

[1986] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1987] Step 1:

[1988] The user speaks a query into the earphone microphone, for example, "What events do you recommend for next weekend?" The input is the user's voice data, and the output is an audio signal sent to the server through the earphone microphone.

[1989] Step 2:

[1990] The server uses speech recognition software to convert the user's voice data into text data. Specifically, it captures the voice signal as digital data, analyzes the data, and converts it into a string of characters. The input is voice data, and the output is text data.

[1991] Step 3:

[1992] The server uses a generative AI model to generate an appropriate answer based on the text data. For example, for the text "What events do you recommend for next weekend?", the AI ​​model generates an answer such as "There is a music festival in the city next weekend." The input is text data, and the output is the generated answer text.

[1993] Step 4:

[1994] The server uses an emotion engine to analyze the user's emotions. Specifically, it recognizes emotions from the user's voice and adjusts the system to provide information according to the emotion. For example, if the user expresses joy, the server provides additional information that the user is likely to find joyful according to the emotion. The input is the user's voice data and the generated answer text, and the output is an answer text adjusted according to the emotion.

[1995] Step 5:

[1996] The server provides the generated answer either audibly through earphones or visually through contact lenses. Augmented reality technology is used to provide visual information. For example, if the user is using earphones, the server may announce audibly, "There's a music festival in the city next weekend." If the user is using contact lenses, the server may visually display detailed information about the event. The input is the adjusted answer text, and the output is an audio notification or visual information.

[1997] (Application example 1)

[1998] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1999] Conventional information provision systems provide uniform information without considering the user's feelings, resulting in low user satisfaction. Furthermore, the means of providing information—visual and audio—are separate, resulting in an inconsistent user experience. Furthermore, there is a lack of technology to efficiently process users' voice requests and provide appropriate information.

[2000] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[2001] In this invention, the server includes earphones and contact lenses, a means for linking them, a means for making a response to a user's desired request available aurally or visually, a means for recognizing the user's emotion and providing information corresponding to that emotion, a means for converting a voice request into text, a means for displaying visual information on the contact lenses, and a means for providing voice information through the earphones. This makes it possible to provide appropriate information according to the user's emotion, and the consistent provision of visual and voice information can improve the user experience.

[2002] An "earphone" is a small acoustic device that transmits sound directly to the user's ears.

[2003] A "contact lens" is a thin lens that displays visual information directly to a user's eye.

[2004] "Means for linking" refers to the technical means by which the earphones and contact lenses communicate with each other and share information.

[2005] "Means for making a response to a user's desired request available to the ears or eyes" refers to means for generating information based on the user's request and providing it through earphones or contact lenses.

[2006] "Means for recognizing a user's emotions and providing information according to those emotions" refers to technical means for analyzing a user's emotional state and providing information appropriate to those emotions.

[2007] A "means for converting a voice request to text" is a technical means for converting a user's voice input into text data.

[2008] The "means for displaying visual information on a contact lens" refers to a technical means for displaying the generated visual information to a user through the contact lens.

[2009] The "means for providing audio information through earphones" refers to the technical means for transmitting the generated audio information to the user through the earphones.

[2010] The following system configuration and processing procedure will be described as an embodiment of the present invention.

[2011] System Configuration

[2012] This system uses the following hardware and software:

[2013] Hardware: smartphones, earphones, smart contact lenses

[2014] Software: Speech recognition library (speech_recognition), emotion recognition library (emotion_recognition), content recommendation engine (content_recommender), visual information provision module (visual_display), audio output module (audio_output)

[2015] Processing flow

[2016] 1. Getting a voice request

[2017] The user makes a voice request through the smartphone's microphone, which is then converted into text using a speech recognition library.

[2018] 2. Emotion recognition

[2019] An emotion recognition library is used to analyze emotions from the user's voice, thereby identifying the user's current emotional state.

[2020] 3. Content Recommendation

[2021] A content recommendation engine is used to select appropriate content based on the user's request and emotional state. For example, if a user requests "Tell me about recent movies" and the emotion is recognized as "joy," the latest comedy movies will be recommended.

[2022] 4. Providing visual information

[2023] A visual information provision module is used to display visual information of selected content (e.g., movie posters and trailers) on the smart contact lenses.

[2024] 5. Provision of audio information

[2025] An audio output module is used to provide an overview and detailed information of the selected content via audio through earphones.

[2026] Specific examples

[2027] If a user makes a voice request such as "Tell me about recent movies," and the system recognizes the user's emotion as "joy," the system will recommend the latest comedy movies. The movie poster will be displayed on the smart contact lenses, and a synopsis of the movie will be spoken through the earphones.

[2028] Prompt Sentence Examples

[2029] User: "What are some recent movies?"

[2030] System: "It seems you're feeling joyful. I recommend the latest comedy, 'The Kingdom of Laughter.' Display the movie poster."

[2031] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[2032] Step 1:

[2033] The user makes a voice request through the smartphone's microphone.

[2034] Input: User's voice request

[2035] Processing: The smartphone microphone captures the audio data.

[2036] Output: Audio data

[2037] Step 2:

[2038] The server uses a speech recognition library (speech_recognition) to convert the voice data into text.

[2039] Input: Audio data

[2040] Processing: The speech recognition library analyzes the audio data and generates corresponding text data.

[2041] Output: Text data (user request content)

[2042] Step 3:

[2043] The server uses an emotion recognition library (emotion_recognition) to analyze emotions from the user's voice.

[2044] Input: Audio data

[2045] Processing: An e...

Claims

1. an emotion engine that recognizes the user's emotions; A response generating means for generating a response to a request desired by the user according to the emotion by inputting a specific prompt sentence into a generating AI model; a playback means for playing back information related to the answer through earphones worn by the user and displaying the information on contact lenses worn by the user, the contact lenses having a camera and a display function; Including, When the user becomes excited by looking at a scenery, the answer generation means identifies the scenery the user is looking at by analyzing data from the camera that captured the scenery the user is looking at, and generates information related to the identified scenery as information related to the answer.

2. The system of claim 1 further comprising means for generating a history and characteristics of the identified landscape as information associated with the identified landscape.

3. 2. The system according to claim 1, further comprising means for calculating a route from the user's current location to the user's destination using map information providing means, and displaying calculated route information on the contact lens.

Images