Method, system, and computer program product for a voice interface of a wearable computing device (wearable computing device voice interface)

Abstract

To provide a method, system and computer program product for wearable computing device audio interface.SOLUTION: In a method of receiving auditory signals and responding to commands of a user, an auditory signal is received by a wearable contact lens from a user during a listening mode. The wearable contact lens is positioned adjacently to an eye of the user during the reception, and a connection from the wearable contact lens to a paired computing device of the user is made through a personal network connection. The auditory signal is transferred from the wearable contact lens, through the personal network connection, to the paired computing device of the user.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present disclosure relates to wearable computing, and more particularly, to an audio interface for contact lenses.

Background Art

[0002] A wearable computing device can include one or more computing devices worn on or within a user's body. The wearable computing device can be configured to send or receive signals to or from a person or another paired computing device. The wearable computing device may need to be placed next to or adjacent to a particular part of the user in order to receive signals from the user, but should not interfere with the user's perception or movement.

Summary of the Invention

Problems to be Solved by the Invention

[0003] Provided are a method, a system, and a computer program product for an audio interface of a wearable computing device.

Means for Solving the Problems

[0004] Embodiments disclose a method, a system, and a computer program product. An auditory signal is received by a wearable contact lens. The auditory signal is received from a user. The reception is performed during a listening mode. The wearable contact lens is placed adjacent to the user's eye during reception. A connection is made from the wearable contact lens to the user's paired computing device. The connection is made via a personal network connection. The auditory signal is transferred from the wearable contact lens via the personal network connection to the user's paired computing device.

[0005] The above summary is not intended to describe every embodiment or all implementations illustrated by the present disclosure.

[0006] The drawings included in this application are incorporated herein and form a part of this specification. The drawings illustrate and describe embodiments of the present disclosure and are helpful in explaining the principles of the present disclosure. The drawings only exemplify specific embodiments and do not limit the scope of the present disclosure.

Brief Description of the Drawings

[0007]

Figure 1

Figure 2

Figure 3

Figure 4

Modes for Carrying Out the Invention

[0008] Although the present invention can be implemented in various improved and alternative forms, the details are shown and described in the drawings by way of example. However, it should be understood that the intention is not to limit the scope of the present invention to the specific embodiments described. On the contrary, the intention is to cover all improvements, equivalents, and alternatives within the spirit and scope of the present invention.

[0009] Aspects of the present disclosure relate to wearable computing. More specifically, they relate to an audio interface for contact lenses. The present disclosure is not necessarily limited to such applications, but various aspects of the present disclosure will be understood through the description of various examples using this context.

[0010] Wearable computing is becoming a common way to communicate computing devices with each other or to use computing devices. Wearable computing may be the operation or communication of a wearable computer adjacent to or within a user. A wearable computer (wearable terminal) can include a processor, a memory, and one or more sensors or inputs / outputs configured or shaped to be worn. Often, a wearable terminal is configured to communicate through a user using one or more interfaces different or distinct from other computing interfaces. For example, a desktop, laptop, or smartphone terminal (non-wearable terminal) can include a screen, keyboard, mouse, speaker, touch screen, or other related input / output communication interface (communication interface).

[0011] The communication interfaces of non-wearable terminals may be useful and efficient, but they have drawbacks. Non-wearable terminals may only be useful or convenient in limited situations and use cases. For example, to use a non-wearable terminal, a user would have to sit at a desk, interrupt what they are doing, and type on a keyboard. In another example, to use a non-wearable terminal, a user would have to hold the terminal, look at it, and spend time and effort to communicate.

[0012] However, wearable terminals have drawbacks with respect to communication interfaces. There are cases where the user wants to execute terminal communication according to the environment, such as searching for or accessing data. For example, the user may want to know information about a person depicted in a book or magazine in front of the device. A user using a non-wearable terminal can input many words and phrases to search for or investigate information about a magazine. On the other hand, in the case of a wearable terminal, the screen of the terminal may be small, or it may not be equipped with a keyboard for input.

[0013] One of the new computing platforms in the wearable computing space is the smart contact lens (smart lens). The smart lens may be a contact lens worn on or adjacent to the user's eye. The smart lens may be a contact lens that corrects vision. The smart lens may also be a lens that does not correct vision. The smart lens may incorporate data processing capabilities and be able to sense the user's environment. For example, one or more cameras placed within the smart lens may be able to take images or videos. The smart lens may be able to provide augmented reality to the user. For example, one or more displays embedded in the smart lens may be able to render text or graphics and place them as a composite view with the user's real environment. One of the drawbacks of the smart lens is that it may not be able to receive input from the user.

[0014] Voice-based computing systems can have several advantages when used in wearable applications. A voice-based system can include a microphone and be communicatively coupled to a device. For example, a smart home assistant can be placed in a user's environment to receive voice commands and respond to the user. In another example, a smartphone can include a microphone configured to receive a user's voice commands. In yet another example, a user may wear a headset or lavalier microphone. Voice-based systems can have drawbacks when used with smart lenses. For example, a smart home assistant is a stationary computing device and may only be useful in limited situations where the user is in the same room as, or standing close to, the smart home assistant. Another drawback is that voice-based systems can be obtrusive or difficult to use. For example, a headset may not be useful in situations where the user does not want to remember to wear it, such as in a gym or training area, or where using the headset may be functionally difficult.

[0015] A voice-enabled contact lens (VCCL) may have advantages over other smart lenses. The VCCL can be a wearable contact lens that the user wears. Specifically, the VCCL can be worn on or adjacent to the surface of the eye. The lens can be shaped and configured to correct, modify, or adjust the user's field of view. In some embodiments, the VCCL may not be configured to adjust the user's vision. For example, the shape, thickness, or other optical or focusing or both adjustment characteristics of the VCCL lens can be sized and shaped to permit or enable the user's environment to reach the eye without the user's field of view being corrected or altered.

[0016] The VCCL may be activated by receiving one or more sounds from the user's environment. Specifically, the VCCL may include a microphone, a voice coil, or other related transceivers. The microphone may be on the surface of the VCCL. The VCCL may also include a processor and a memory. The processor and the memory may be communicably connected to the microphone. The VCCL can receive an auditory signal from a person wearing a contact lens (e.g., the user). The microphone of the VCCL can be configured to transmit the received sound to the processor of the VCCL.

[0017] Furthermore, the VCCL may transmit an auditory signal (e.g., audio, sound, voice, sound wave) to a paired computing device. For example, the VCCL may include a network connection. The VCCL can be configured to connect to a smartphone via the network connection. After receiving the sound, the VCCL can transfer the received sound to a paired computing device for processing the received sound. For example, a user may be standing in the basement of a three-story building and want to control a network-connected speaker on the third floor. The user can say "Play music in the bedroom", and the microphone of the VCCL can receive the auditory signal including the speech. Next, the VCCL can transmit this speech to a smartphone connected to the network. The smartphone can operate based on the speech by performing processing for properly handling the request. Specifically, the smartphone can transmit a command to the network-connected speaker on the third floor via, for example, a local area network, thereby enabling the network-connected speaker to start playing music.

[0018] In some embodiments, the VCCL may be configured to perform natural language processing. For example, the VCCL can send instructions to a paired computing device to perform natural language processing. Natural language processing may include various operations via hardware, software, or a combination thereof. For example, natural language processing can operate on one or more data sources, search applications, and report analyzers. Natural language processing may be a computer module that analyzes content and other information received from a processor located in the VCCL or the paired computing device. Natural language processing may include performing various methods and techniques for analyzing text information (e.g., parsing, semantic analysis, etc.). Natural language processing can be configured to recognize and analyze any number of natural languages. In some embodiments, natural language processing can analyze a document or a section of content from a microphone in a lens worn by or in front of the user's eyes. Various software components or hardware components (not shown) or both of natural language processing may include, but are not limited to, a tokenizer, a part-of-speech (POS) tagger, a semantic relation identifier, and a syntactic relation identifier. Natural language processing may include a support vector machine (SVM) generator for processing the content of topics found in a corpus and classifying the topics.

[0019] In some embodiments, the tokenizer may be a computer module that performs lexical analysis. The tokenizer can convert a sequence of characters into a sequence of tokens. Tokens may be strings included in an utterance signal or a voice signal, or a copied electronic document (electronic document). For example, an utterance may be converted from a voice signal to an electronic document, and the electronic document may be classified and converted into characters with meanings assigned. Further, in some embodiments, the tokenizer can identify word boundaries within an electronic document and split a section of any text within the document into constituent text elements such as words, multi-word tokens, numbers, and punctuation marks. In some embodiments, the tokenizer can receive a string, identify the morphemes within the string, and classify them into tokens.

[0020] In various embodiments, the part-of-speech tagger may be a computer module that marks up words during an utterance to correspond to specific parts of speech. The part-of-speech tagger can read an electronic document or other text in natural language and assign parts of speech to each word or other token. The part-of-speech tagger can determine the part of speech corresponding to a word (or other text element) based on the definition of the word and the context of the word. The context of the word may be based on the relationship with similar and related words within a phrase, sentence, or voice signal.

[0021] In some embodiments, the context of the word may be based on one or more already analyzed electronic documents. Examples of parts of speech that can be assigned to a word include, but are not limited to, nouns, verbs, adjectives, adverbs, etc. Examples of other parts of speech that the part-of-speech tagger can assign include, but are not limited to, comparative or superlative adverbs, wh-adverbs, conjunctions, determiners, invariable words, possessives, prepositions, wh-pronouns, etc. In some embodiments, the part-of-speech tagger can tag a section of tokens with a part-of-speech category or otherwise annotate them. In some embodiments, the part-of-speech tagger can tag a section of tokens or words to be parsed by a natural language processing system.

[0022] In some embodiments, the semantic relationship identifier may be a computer module configured to identify the semantic relationships of text elements (e.g., words, phrases) recognized in electronic documents and auditory signals. In some embodiments, the semantic relationship identifier can identify the functional dependencies of entities with other semantic relationships.

[0023] In various embodiments, the syntactic relationship identifier may be a computer module configured to identify the syntactic relationships within a clause composed of tokens. The syntactic relationship identifier can identify the grammatical structure of a sentence, for example, which groups of words are related as phrases and which words are the subject or object of a verb. The syntactic relationship identifier may conform to a formal grammar.

[0024] In some embodiments, the natural language processor may be a computer module capable of analyzing the utterances of auditory signals stored in an electronic document and generating a data structure corresponding to one or more parts of the electronic document. For example, in response to receiving a request from a wearer of VCCL, the processor can perform natural language processing and output text elements parsed from the data. In some embodiments, the parsed text elements can be represented in the form of a parse tree or other graph structure. To generate the parsed text elements, natural language processing can operate computer modules including a tokenizer, a part-of-speech (POS) tagger, an SVM generator, a semantic relationship identifier, a syntactic relationship identifier, etc.

[0025] In some embodiments, the VCCL may be configured to identify the wearer's emotions. Specifically, each lens may include one or more external sensors such as a wide-angle camera or a pressure sensor. The sensors can observe the user's emotions by photographing the facial expressions. The sensor data image can be provided to one or more secondary terminals such as a virtual reality (VR) headset (not shown). The sensor data can be used via the VR headset for playing games or creating virtual characters such as avatars. The sensor may be an infrared emitter configured to emit light rays around the surface of the user's face to obtain sensor data for identifying facial expressions.

[0026] The sensor data can be used in combination with an auditory signal. For example, the user can issue a voice command, and the audible speech and facial sensor data can be transmitted to a smartphone to process and identify the context of the command. The context may indicate a priority. For example, the user may make a vigilant or stressed facial expression indicating that the command has a high priority. The sensor data can be used to identify or form the user's behavior pattern or preference. Specifically, the sensor data captured from the VCCL can be recorded together with the data and used to store, record, or otherwise capture the emotional state of the user while performing one or more tasks in the real space. For example, the user may be instructing the camera to take a photo by issuing an audible speech of "Take a photo." At the same time as the speech, the user may be smiling. The VCCL can capture the smiling face and the speech, and the surrounding image can be captured by a paired computing device. The computing device can associate the image with the emotion of "happiness."

[0027] The VCCL may be configured to respond to specific eyes, eyelids, or other gestures (e.g., a camera embedded in the VCCL that captures the movement of the user's hand or arm). Specifically, by utilizing the movement of the eyelids, the user can control the timing of executing voice commands. The VCCL can continuously track the movement patterns of the eyes, line of sight, or the movement of the eyelids or a combination thereof. Based on the movement pattern of the eyelids, the user can arrange the voice commands that have already been issued, re-prioritize the voice commands, etc. For example, gestures can generally be executed by the user. In addition to receiving the auditory signals transferred to the paired computing device, the executed gestures can be identified by the VCCL. Priorities can be assigned based on the identified gestures. The priority can be adjusted based on the gestures. For example, the VCCL can be configured to assign default priorities to all received auditory signals. When the user blinks both eyes and the VCCL identifies the blink, the priority of the received auditory signals can be adjusted, such as being lowered or raised. Before being sent to the paired computing device, the VCCL can assign or adjust the priority, such as by overwriting the priority from the default to low or high based on the identified blink gesture.

[0028] FIG. 1 is a diagram showing the representative main components of a usable computer system 100 (also referred to as a computer) according to some embodiments of the present disclosure. It should be understood that the individual components may differ in complexity, quantity, type, or configuration or combinations thereof. The specific examples shown are for illustrative purposes only and are not necessarily limited to such variations. The computer system 100 can include a processor 110, a memory 120, an input / output interface (herein referred to as I / O or I / O interface) 130, and a main bus 140. The main bus 140 can provide a communication path for other components of the computer system 100. In some embodiments, the main bus 140 can be connected to other components such as a special electronic signal processor (not shown).

[0029] The processor 110 of the computer system 100 may be composed of one or more cores 112A, 112B, 112C, 112D (collectively 112). The processor 110 may further include one or more memory buffers or caches (not shown) for temporarily storing instructions and transmitting data to the cores 112. The cores 112 can issue instructions regarding inputs transmitted from the cache or memory 120 and outputs of results to the cache or memory 120. The cores 112 may be composed of one or more circuits configured to execute one or more methods according to embodiments of the present disclosure. In some embodiments, the computer system 100 may include multiple processors 110. Also, in some embodiments, the computer system 100 may be a single processor 110 with a single core 112.

[0030] The memory 120 of the computer system 100 may include a memory controller 122. In some embodiments, the memory 120 can include a random access memory, a storage device, or a storage medium (either volatile or non-volatile) for storing data and programs. In some embodiments, the memory takes the form of a module (e.g., a dual in-line memory module). The memory controller 122 can communicate with the processor 110 to facilitate the storage and retrieval of information within the memory 120. The memory controller 122 can communicate with the I / O interface 130 to facilitate the storage and retrieval of inputs or outputs within the memory 120.

[0031] The I / O interface 130 may include an I / O bus 150, a terminal interface 152, a storage interface 154, an I / O device interface 156, and a network interface 158. The I / O interface 130 can connect the main bus 140 to the I / O bus 150. The I / O interface 130 can transmit instructions and data from the processor 110 and the memory 120 to various interfaces of the I / O bus 150. The I / O interface 130 can also transmit instructions and data from various interfaces of the I / O bus 150 to the processor 110 and the memory 120. The various interfaces may include the terminal interface 152, the storage interface 154, the I / O device interface 156, and the network interface 158. In some embodiments, the various interfaces may include a subset of the aforementioned interfaces (e.g., an industrial application embedded computer system may not include the terminal interface 152 and the storage interface 154).

[0032] The logical modules of the entire computer system 100 (including, but not limited to, the memory 120, the processor 110, and the I / O interface 130) may communicate failures and changes to one or more components to a hypervisor or an OS (not shown). The hypervisor or the OS can allocate various resources available in the computer system 100 and track the location of data in the memory 120 and the location of processes allocated to various cores 112. In embodiments where components are combined or rearranged, the aspects and functions of the logical modules can be combined or the arrangement can be changed. These variations will be apparent to those skilled in the art.

[0033] FIG. 2 is a diagram showing an example of a system 200 that performs voice-based processing using a wearable contact lens according to some embodiments of the present disclosure. The wearable contact lens may be a VCCL in some embodiments. The system 200 may be part of the real world. Specifically, the user 210 can have an appendage 212 (e.g., an arm) and a head 220. The head 220 of the user 210 can have one or more facial features such as a mouth 222, a nose 224, and two eyes 226-1 and 226-2 (collectively eyes 226). The eyes 226 can include eyelids such as eyelids 228-1 and 228-2 for each of the eyes 226-1 and 226-2.

[0034] The VCCL may include two voice-responsive contact lenses 230-1 and 230-2 (collectively referred to as lenses 230-1 and 230-2 as VCCL 230). The VCCL 230 may include two microphones 232-1 and 232-2 (collectively microphones 232). The microphones 232 may be made of a material suitable for responding based on an auditory signal. For example, the microphones 232 may be composed of relevant components such as a transducer element or capsule, and a diaphragm or voice coil. The microphones 232 may be translucent, transparent, or substantially transparent. Specifically, the components' composition may be transparent. The material of the microphones 232 can be selected from one or more of fluoropolymers, polyethylene terephthalate, polypropylene, polytetrafluoroethylene (PTFE), hexafluoropropylene (FEP), and copolymers of PTFE and FEP, etc.

[0035] VCCL230 may be communicatively connected to a paired computing device. Specifically, VCCL230 may include a wireless network transceiver for sending and receiving network communications to and from a paired computing device via a personal network connection. For example, VCCL230 may communicate with smartphone 240 via a personal network connection. In another example, VCCL230 may communicate with smartwatch 250 via a personal network connection. VCCL230 may be configured to receive one or more auditory signals and transfer the auditory signals to a paired computing device. Specifically, microphone 232 may receive an audio command from user 210 when the user speaks, vocalizes, or otherwise engages in voice communication. For example, user 210 may verbally utter "What's the weather like today?", and microphone 232 of VCCL230 may receive an auditory signal from the user including the verbal utterance. VCCL230 may connect to smartphone 240 and transfer the utterance "What's the weather like today?" to smartphone 240 via a personal network connection. Smartphone 240 may execute a network request via a wide area network (e.g., the Internet) to perform natural language processing and retrieve information based on the utterance (e.g., to retrieve weather data from an Internet source).

[0036] The utterance may be sent to a paired computing device in an audio-based format. The paired computing device may be configured to execute a response in a voice-based or audio-based system format, and VCCL230 may be ready to format an audio response for user use. For example, the user may utter "What is the score of the soccer game?", and VCCL230 may transfer the utterance to the smartwatch 250. The smartwatch 250 may receive the utterance but may not recognize the format or function of VCCL230. For example, the smartwatch 250 may be configured to understand VCCL230 as a headset communicating via a network or an audio-based smart home assistant. The smartwatch 250 may be able to respond to the utterance by obtaining sports information from a network. The smartwatch 250 may be able to transfer information in an audio-based system format such as text or an audio clip. VCCL230 may be configured to format a response to the visual interface of the lens 230. For example, a processor (not shown) of VCCL230 may be able to execute an algorithm from voice to text based on the response sent by the smartwatch 250. VCCL230 may be able to visually provide information to the user 210. For example, the lens 230 may include one or more displays configured to project an augmented reality view. A response including the score of the soccer game may be visually provided in the user's field of view. This may be convenient in situations where it is impolite or not permitted to have the computing device speak the information out loud, or where it is not permitted to look at or take out the device. For example, in a restricted location such as a library, interaction with the smartphone 240 may not be permitted.

[0037] VCCL230 may be configured to respond to the gestures of user 210. For example, one or more image sensors (not shown) of VCCL230 may be configured to capture the movement within the field of view of user 210. The image sensors of VCCL230 can capture a user who is waving arm 212 with a specific movement. VCCL230 can send the image to smartphone 240 via a personal network connection in order to perform image analysis for recognizing the movement of arm 212. When the movement of arm 212 is recognized, smartphone 240 can start an operation (for example, taking a photo with smartphone 240, recording ambient sound).

[0038] VCCL230 may include one or more sensors for detecting the movement of the user's face 220. Specifically, VCCL230 may include one or more pressure or optical sensors (not shown) capable of detecting the movement of the user's eyelids 228. For example, the user 210 can move one eyelid 228-1, and the sensor of VCCL230 can recognize a command based on the movement of one eyelid. Facial gestures can be processed by VCCL230 simultaneously with the reception of an auditory signal. For example, the user can voice the command "Send a message to my spouse that I will be late for work", and the microphone 232 can receive the command. And almost simultaneously, the user can move both eyelids 228, and the pressure sensor can identify the user's gesture of moving both eyelids. Accordingly, VCCL230 can assign a priority to the command "Send a message to my spouse that I will be late for work" before transferring the command to the smartphone 240. Next, the user 210 can decide to finish work and make a gesture by moving eyelid 228-2. VCCL230 can delay, cancel, or adjust the priority of the message in response to the gesture. Specifically, VCCL230 can send an updated priority to the smartphone 240 regarding the command "Send a message to my spouse that I will be late for work".

[0039] In some embodiments, VCCL230 may authenticate user 210. Specifically, user 210 may train or pair VCCL230 with smartphone 240. In the pairing procedure, user 210 may place VCCL230 in contact with or adjacent to user 210's eye 226, or insert VCCL230 into user 210's eye 226. When placed on eye 226, the microphone 232 of VCCL230 may begin to listen for auditory signals. User 210 may be prompted by smartphone 240 to read or speak one or more words, sentences, phrases, or emit an auditory signal. The distance 234-1 between microphone 232-1 and mouth 222 may be recorded or stored in the memory of VCCL230 or smartphone 240. Similarly, the distance 234-2 between microphone 232-2 and mouth 222 may also be recorded or stored. Based on distances 234-1 and 234-2, a signature of user 210 may be generated. Sounds not from user 210 may be ignored. For example, at a second time (e.g., minutes, hours, weeks) after the pairing procedure, a second user may wear VCCL230 or insert it into the second user's eye. The second user may attempt to communicate by generating an auditory signal (e.g., speak, sing, shout) to operate VCCL230. VCCL230 may measure the distance between the second user's eyes (based on the delay or volume received at microphones 232-1 and 232-2). VCCL230 may compare the measured distance with the signature of user 210 and identify a mismatch (e.g., the measured distance indicates that the second user's eyes are not the same as distances 234-1 and 234-2). Based on the identified mismatch, VCCL230 may not operate when used by the second user.

[0040] In some embodiments, VCCL230 may perform error correction on the auditory signal. For example, user 210 may be able to verbally utter "What time is it now?". The auditory signal may be emitted from mouth 222, and microphone 232-1 of VCCL230-1 may record the first copy of the utterance. At the same time, microphone 232-2 of VCCL230-2 may record the second copy of the utterance "What time is it now?". Each of VCCL230 may be able to send the first copy and the second copy of the utterance to smartwatch 250. Smartwatch 250 may be able to perform error correction or another related error correction algorithm on the two copies of the utterance. Based on the execution of the error correction algorithm, smartwatch 250 may be able to improve the accuracy or precision when performing speech on the text of the utterance "What time is it now?" from the user.

[0041] In some embodiments, VCCL230 may perform learning on historical data. For example, VCCL230 can be configured to identify the focal length of user 210 and an object (not shown) recognized in the field of view of user 210. VCCL230 can transfer the identified focal length and the recognized object to smartphone 240. The transfer may be performed during the processing of an oral command received by microphone 232 of VCCL230. In addition to generating a response to the oral command to smartphone 240, historical data such as the recognized object and the identified focal length can be stored in a data store such as the memory of smartphone 240. VCCL230 or smartphone 240 or both can perform historical learning and identify relationships between various parts of the previous requests of user 210. For example, the relationships may be those that identify the user's focal length and the direction of focus, the type of auditory signal, the time, the position of user 210, and the like. Using historical learning, VCCL230 or smartphone 240 or both can predict the auditory signal of user 210. For example, the user can focus on and gaze at a distant object and ask "What's the weather?" between 6:30 am and 7:17 am on weekdays. In response to the reception of the auditory signal, smartphone 240 can identify the visual information from VCCL230 and determine that the user is requesting weather information before processing (e.g., performing text conversion of the speech). Smartphone 240 can start searching for weather information from an Internet-based weather information service. While the request is being processed, smartphone 240 can start processing the auditory signal and confirm that the user 210's speech is actually a weather request. Based on the confirmation that the speech is a weather request, smartphone 240 can provide weather information to user 210 more quickly, such as requesting weather information at the time of reception before performing text conversion of the speech of user 210.

[0042] Figure 3(A) shows the outer surface 302 of an example of a lens 300 configured to receive an auditory signal, according to some embodiments of the present disclosure. The example of the lens 300 may be a VCCL such as VCCL230-1 or VCCL230-2. The outer surface 302 can be configured to be distal from or away from the wearer's eye. For example, the outer surface 302 may generally be convex. The lens 300 may include a plurality of components embedded within the lens. For example, the lens 300 may include a processing module 310, at least one external sensor 320, a display 330, and a power source 340. The lens 300 may also include an internal lens network 350. The internal lens network 350 may include one or more communication paths to facilitate the exchange of data and communicatively connect the internal components. The internal components and network 350 can be arranged so as not to obstruct the wearer's field of view. Specifically, a contact lens wearer can have a maximum line of sight, field of view, or vision of 370 and a minimum of 380. At a maximum of 370, the wearer may focus such that they can see the entire environment including multiple objects. In the case of a minimum of 380, the wearer may focus such that they can only see one or two objects within the environment.

[0043] The processing module 310 may include a computer system such as computer 100. The processing module 310 may be a system-on-chip computing system. Specifically, the processing module 310 may include a processor, memory, a wireless transceiver, and other components (not shown). The processing module 310 can be configured to transfer data to and receive data from a paired computing device. For example, the processing module 310 can utilize a wireless transceiver that connects to and transmits and receives data via the connection to a smartphone.

[0044] At least one external sensor 320 may be a single sensor such as a camera or other related imaging sensor. The external sensor 320 can be configured to capture images outwardly, such as capturing the wearer's field of view or line of sight of the lens 300. The external sensor 320 may be another type of sensor such as a pressure sensor that captures eye or eyelid movement, or the user's direction, field of view, shape, focal length, or other eye-related information. The external sensor 320 may be a plurality of sensors. For example, the sensor 320 can include a visual sensor for observing the environment and capturing images. Further, the sensor 320 may be an optical sensor configured to identify gestures such as blinking, glancing, eyelid movement, open eyes, and closed eyes. Additionally, the sensor 320 may be a focus sensor configured to measure the user's line of sight.

[0045] The display 330 may be an embedded display configured to present information to a user. For example, the display 330 may be an augmented reality display configured to synthesize information and the environment from a paired computing device (not shown) or a terminal such as the lens 300. The display 330 can be configured to extend across the visual portion of the lens 300. For example, the display 330 can be arranged over up to 370 of the wearer's vision. The power source 340 may be a battery related to lithium, polymer, or other battery technologies. The power source 340 may be a dynamic power source. For example, the power source 340 can receive a charge or take in electrical energy based on the movement of the wearer, or the movement of the eyes, eyelids, or another relevant part of the wearer. In another example, when the wearer moves around, the relative position of the lens 300 can change with respect to the gravitational force on the earth, and the power source 340 can capture the changing force applied to the lens 300 based on that movement. The power source 340 may also be a receiver configured to receive electricity from an external power source. For example, the power source 340 can periodically receive radio waves or another relevant signal from a paired computing device and convert the energy into electricity for the lens 300 in response to the periodic radio waves.

[0046] The lens 300 may also include a microphone 360. The microphone 360 can be disposed on or adjacent to the outer surface 302 of the lens 300. For example, as shown in FIG. 3(A), the microphone 360 can be disposed on or covering other parts or components of the lens 300. The microphone 360 can be attached to the outer surface of the lens 300 by an adhesive such as a transparent adhesive. The microphone 360 can be integrally coupled to the lens 300. For example, the lens 300 can be formed by molding a polymer or other transparent material on the outer surface 302. The portion of the lens 300 without the microphone outside the circle 360 may have a first thickness. The portion of the lens 300 with the microphone 360 inside the circle 360 may have a second thickness. The second thickness may be thinner than the first thickness to form a void (not shown) or a space between the outer surface 302 and the inner surface of the lens 300. The void is configured to allow movement along an inner axis (not shown), and the inner axis can extend from the outer surface 302 through the lens 300 to the inner surface. In some embodiments, the microphone 360 may be sprayed onto the lens 300. For example, the microphone 360 can be particles containing a sound-responsive material and an adhesive. The machine can spray the microphone 360 onto the lens 300 within the boundary of the circle 360. The microphone 360 can extend over most of the outer surface 302.

[0047] The microphone 360 may be composed of a transparent material that is sensitive to sound waves or auditory signals. For example, the microphone 360 may be a transparent magnetic coil material. In another example, the microphone 360 may include a plurality of magnetic particles in which transparent particles are dispersed in a coil configured to receive sound waves or auditory signals. The microphone 360 can be extended beyond the wearer's visual area. For example, the microphone 360 may have a diameter greater than the maximum visual acuity or line of sight of 370 that can be perceived or seen while the wearer is wearing the lens 300. The microphone 360 can be configured to be sensitive only to the user (the wearer of the lens 300). For example, the range or distance of sensitivity to receive auditory signals by the microphone 360 may be less than 1 foot. In another example, sounds or conversations beyond 3 feet of the lens 300 may not be detected by the microphone 360. The microphone 360 can be communicatively connected to the internal lens network 360 and configured to provide auditory signals to other components of the lens 300. For example, the microphone 360 can be configured to convert sound waves from the wearer into auditory signals. The microphone 360 can transfer auditory signals to the processing module 310 via the internal lens network 360.

[0048] FIG. 3(B) is a diagram showing the inner surface 304 of an example of the lens 300 configured to receive an auditory signal in accordance with the environment viewable by the lens 300 according to some embodiments of the present disclosure. The inner surface 304 can be configured or shaped to be disposed on or adjacent to the outer surface of the wearer's eye. For example, the inner surface 304 may generally be concave. The lens 300 can be configured to view an exemplary environment that the wearer can view. The environment may include a plurality of real-world elements such as people, places, buildings, objects, animals, plants, and the like. Initially, as shown in FIG. 3(B), the environment 390 may include a table 392, a flower 394, and a book 396.

[0049] Lens 300 can be configured to respond to the wearer's verbal requests or other auditory signals. For example, while looking at the environment 390, the user can issue a verbal command in the form of speech or utterance. The microphone 360 of the lens 300 can vibrate, sense, or otherwise receive the sound waves of the utterance. The lens 300 can receive an auditory signal from the microphone 360 and execute processing or send the processing to another device. In a first example, the lens 300 can receive a verbal command via the microphone 360 and start listening. Further, the microphone 360 can continuously listen during the activation mode. While in the activation mode, the microphone 360 can transfer the auditory signal to the processing element 310 periodically (e.g., every 100 milliseconds). The processing element 310 can be configured to identify activation passwords (e.g., "start listening", "activate contact"). The lens 310 can operate in the activation mode by default. For example, the lens 310 can operate in the activation mode when it is not operating in the active mode or listening mode for processing auditory signals, commands, questions, and responses other than starting listening. In some embodiments, the external sensor 320 can receive a gesture command to start listening. The user can speak about questions unrelated to the environment 390, and in response, the lens 300 can process the questions. For example, the wearer can ask for the time, the microphone 360 can receive the question, and send the question to the processor 310. The processor 310 can determine the response to the time and provide a visual depiction (e.g., pixels rendering Roman numerals of the time) to the display 330.

[0050] The user can make utterances about the environment related to environment 390. For example, while wearing the lens 300, the user can focus on or look at the flower 394 and verbally say, "What kind of flower is this?" The external sensor 320 can be configured to observe the environment in response to the user's line of sight between a maximum visual acuity of 370 and a minimum visual acuity of 380 of the environment 390. The external sensor 320 can be configured to observe the environment in response to the utterance. For example, the processing element 310 is configured to parse "What kind of flower is this?" and based on the content of the utterance, can determine that the user is asking about something within the field of view or something related to the user's line of sight. The processing element 310 can be configured to perform syntactic analysis simultaneously with the reception of the utterance. For example, while each word of the phrase "What kind of flower is this?" is being uttered, the microphone 360 may transfer the question to the processing element 310 word by word. When the processing element 310 receives the word "kind" in the phrase "What kind of flower is this?", the lens 300 determines that the user is requesting information about something visual and immediately reacts by instructing the external sensor 320 to be activated and start capturing information about the environment 390 and the wearer's line of sight. For example, the external sensor 320 can operate in a deactivated mode, and during the deactivated mode, the external sensor 320 may not be able to capture an image of the environment. The external sensor 320 may, for example, be made to operate in the deactivated mode by default in order to reduce or eliminate the power drawn from the power source 340. The external sensor 320 can identify from the wearer's line of sight that the flower 394 is the target of the environment 390 for which the utterance "What kind of flower is this?" is directed. The lens 300 can transfer the utterance "What kind of flower is this?" to a paired computing device in response to the environment captured by the external sensor 320. The lens 300 can receive a response in the form of auditory data such as "That flower is a chrysanthemum."The processing element 310 can perform a conversion process from auditory data such as "that flower is a chrysanthemum" from voice to text, and render the received auditory data on the display 330 for the wearer of the lens 300 as visual text "that flower is a chrysanthemum".

[0051] The lens 300 can operate based on predicting the wearer's requests. Specifically, the lens 300 can be configured to analyze the voice commands received from the microphone 360 and identify from the external sensor 320 whether the wearer's eyes are focused on a specific object (e.g., the book 396). When the lens 300 activates the external sensor 320 and the focus or line of sight is identified, the processing element 310 can capture and record the command and historical information of the eye's focus, line of sight, or direction or a combination thereof. Based on historical learning, the lens 300 can predict whether there is any correlation between the auditory signal and the direction of the eye's focus, and thereby, the lens 300 may execute shooting based on the reception of the utterance from the microphone 360, which is a pre - stage for the processing element 310 to process the verbal utterance or transfer it to a paired computing device for processing.

[0052] For example, the lens 300 can identify the user's line of sight in response to identifying that the utterance received from the microphone 360 is a shooting command. The processing element 310 can record information regarding the user's line of sight simultaneously with the shooting command. Specifically, the processing element 310 can have a buffer, cache, or other memory that can record various eyes, foci, or lines of sight or combinations thereof of the user for several seconds before and during the shooting command. The lens 300 can identify the user's second line of sight. The lens 300 can identify the user's shooting intention. Specifically, the processing element 310 of the lens 300 can compare the second line of sight with the line of sight information recorded from past shooting commands. In response to identifying the user's shooting intention, the lens 300 can instruct the external sensor 320 to perform a second image capture.

[0053] Figure 4 is a diagram illustrating an example of a method 400 for receiving an auditory signal and responding to a user's command according to some embodiments of the present disclosure. The method 400 may be performed by a contact lens such as the VCCL 230. The method 400 may also be performed by a paired computing device such as the smartphone 240. Some parts of the method 400 may be performed by the lens, and other parts may be performed by the paired computing device.

[0054] Starting from start 405, method 400 begins by monitoring command 410. The command may be provided in the form of a verbal command. For example, one or more auditory signals can be received by the transparent lens of VCCL230 during monitoring. The command may be provided in the form of a gesture. For example, one or more eye, line of sight, focus, eyelid, head movement, or arm gestures can be executed and received by VCCL. Monitoring the command may also include transferring the command to a paired computing device such as a smartwatch or smartphone. The command can be processed by VCCL. In some embodiments, the command may be passed to a paired computing device for processing. In some embodiments, only certain commands can be monitored by VCCL in step 410. For example, VCCL operates in a startup mode and can continuously listen only to a set of basic sounds or noises. Under certain circumstances, the speaking user or the user speaking the startup password can configure the form of VCCL monitoring. VCCL can be configured to respond only to words or commands or the user's mode switching request, but not otherwise. In some embodiments, the command can be received from a paired computing device. For example, a personal network connection between VCCL and a smartphone may enable data transmission and reception. The smartphone can transmit a specific command to VCCL via the personal network connection based on the determination that the user's past verbal utterances included in the auditory signal transmitted from VCCL indicate that the user is attempting to perform an action.

[0055] If the command is a shooting command (YES in step 420), the VCCL can start shooting an image in step 430. Shooting the image may include a command to activate the camera or to make it operable in an activated mode. During the activated mode, the camera can receive visual information. The camera may be part of the VCCL, such as a built-in camera configured to shoot the user's environment. The camera can be configured to operate in a deactivated mode when it is not starting to shoot an image in step 430. The shot may be a series of images, such as a video. In step 440, the captured image can be transferred to the paired computing device. For example, the image can be transferred after the connection to the paired computing device of the VCCL wearer is established. In some embodiments, other data than the image can be transferred. For example, the VCCL can record or associate other data, such as a timestamp or a specific eye movement that occurs simultaneously with the period during which the image is shot, with the captured image. The other data and the associated captured image can be transferred together to the paired computing device.

[0056] If the command is a listening command (YES in step 450), the VCCL can start receiving auditory signals such as spoken commands, audible voices, and other sounds in step 460. The sound may be received for a preset period. For example, when the VCCL enters the listening mode, a timer can be started by the processing element of the VCCL. The timer may be determined based on a single preset value (e.g., 10-second listening, 4-second listening). The sound can be received over a plurality of preset periods. For example, based on the first listening command received in step 410, the VCCL can execute the reception of the sound in step 460 for 10 seconds in the listening mode. Further, based on the second listening command received in step 410, the VCCL can execute (for example, for dictation purposes) for 2 minutes in the listening mode. The VCCL can receive the sound until the user stops speaking during the listening mode. For example, when entering the listening mode, the VCCL can continue to receive the wearer's auditory signals until the auditory signal does not exceed a preset listening threshold. The preset listening threshold may be an auditory signal exceeding a specific decibel level. The preset listening threshold may also be an auditory signal that continuously or almost continuously exceeds a specific decibel level (e.g., exceeding 25 decibels every 1800 milliseconds). After receiving the auditory signal, the VCCL can end the listening mode, enter the hold mode or the startup mode again, and resume waiting for the startup signal (e.g., monitoring the command in step 410). The VCCL can be configured to end the listening command in response to receiving another command from the wearer. For example, the wearer can verbally utter "stop listening", and in response, the VCCL can end the listening mode.

[0057] The received sound can be transferred to the computing device paired in step 470. The transfer of the received sound may include establishing or re - establishing a connection to the paired computing device. For example, the VCCL and the smartphone may have been communicatively connected via a wireless personal network connection in the past. The personal network connection enables data to be wirelessly transmitted between the two devices during the connection, and enables the two devices to operate in sleep mode. In sleep mode, either the VCCL or the paired computing device can reconnect to each other by sending a resume communication signal.

[0058] After the sound is transferred in step 470, method 400 can end the process in step 495. In some embodiments, method 400 can repeat the process after the sound is transferred in step 470 and continue to monitor the command in step 410.

[0059] The present invention may be a system, method, or computer program product or a combination thereof in any executable technical detail fusion. The computer program product may include a computer - readable storage medium (or media) having computer - readable program instructions for causing a processor to execute aspects of the present invention.

[0060] A computer-readable storage medium is a tangible device that holds and stores instructions for use by an instruction execution device. The computer-readable storage medium can be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. To enumerate a more specific example of a computer-readable storage medium, it can be a portable computer floppy disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable and rewritable random access memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, an encoded device such as a punch card or a raised structure having grooves in which machine instructions are recorded, and any suitable combination of the foregoing. A computer-readable storage medium as used herein should not be construed to be a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., an optical pulse passing through an optical fiber cable), or an electrical signal transmitted via a wire.

[0061] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to respective computing / processing devices or to an external computer or external storage device via a network (e.g., the Internet, a local area network, a wide area network, or a wireless network network or a combination thereof). The network is composed of copper wire transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers, or edge servers or a combination thereof. The network adapter card or network interface of each computing / processing device receives the computer-readable program instructions from the network and transfers the computer-readable program instructions for storage in a computer-readable storage medium within each respective computing / processing device.

[0062] Computer-readable program instructions for carrying out the operations of the present invention may be source code or object code written in any combination of one or more programming languages, including assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, configuration data for integrated circuits, or a combination of procedural programming languages such as the "C" programming language and similar programming languages and object-oriented programming languages such as Smalltalk, C++. The computer-readable program instructions may be executable entirely on the user's computer as a stand-alone software package, or partially on the user's computer. Alternatively, it may be executable partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, for example, an electronic circuit including a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) may execute the computer-readable program instructions by utilizing the state information of the computer-readable program instructions to personalize for carrying out aspects of the present invention.

[0063] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0064] These computer-readable program instructions can be provided to a computer's processor or other programmable data processing apparatus to generate a machine, such that the instructions, when executed via the processor of the computer or other programmable data processing apparatus, generate means for implementing the functions / operations specified in one or more blocks of a flowchart and / or block diagram. These computer-readable storage media can also be stored in a computer-readable storage medium connectable to a computer, a programmable data processing apparatus, or other devices that function in a particular manner or a combination thereof, such that the computer-readable program instructions stored therein constitute one of the manufactured articles that include instructions for implementing the functions / acts specified in one or more blocks of a flowchart and / or block diagram.

[0065] Like instructions for performing the functions / acts specified in one or more blocks of a flowchart and / or block diagram on a computer, other programmable apparatus, or other devices, the computer-readable program instructions can also be loaded onto a computer, other programmable apparatus, or other devices and execute a series of operational steps thereon to generate a computer-implemented process.

[0066] The flowcharts and block diagrams in the figures illustrate the structure, functionality, and operation of the system, method, and computer program product according to various embodiments of the present invention in an executable manner. In this regard, each block in the flowchart or block diagram may represent a module, segment, or portion of an instruction, which constitutes one or more executable instructions for implementing the specified logical function. In some alternative embodiments, the functions shown in the blocks may occur in a different order than shown in the figures. For example, two consecutive blocks shown may actually be accomplished as one step, and at the same time, may be executed in a substantially simultaneous, partially or wholly temporally overlapping manner, or the blocks may be executed in the reverse order depending on the functionality. It should also be noted that each block of the block diagram or flowchart diagram or both, and combinations of blocks of the block diagram or flowchart diagram or both, can be implemented by a special purpose hardware-based system that performs the specified functions or operations or a combination of special purpose hardware and computer instructions.

[0067] The descriptions of the various embodiments of the present disclosure are presented for illustrative purposes, but are not intended to be exhaustive or limited to the disclosed embodiments. It will be apparent to those skilled in the art that many modifications and variations are possible without departing from the scope and spirit of the described embodiments. The terms used herein are selected to explain the principles of the embodiments, the practical application to technologies found in the market or technological improvements, or to enable those skilled in the art to understand the embodiments disclosed herein.

[0068] The description of various embodiments of the present disclosure is presented for illustrative purposes, but is not intended to be exhaustive nor to be limited to the disclosed embodiments. It will be apparent to those skilled in the art that many modifications and variations are possible without departing from the scope and spirit of the described embodiments. The terms used herein are selected to describe the principles of the embodiments, the practical application to or technical improvement of the technology seen in the market, or to enable those skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method performed by a processor, comprising: The processor receiving an auditory signal from a user during a listening mode by means of a wearable contact lens, the wearable contact lens being disposed adjacent to the user's eye during the receiving; The processor identifying a gesture performed by the user by means of the wearable contact lens; The processor adjusting a priority order of the auditory signal based on the gesture by means of the wearable contact lens; The processor connecting to a paired computing device of the user via a personal network connection from the wearable contact lens; The processor transferring the auditory signal to the paired computing device of the user via the personal network connection from the wearable contact lens; The processor transmitting the priority order to the paired computing device of the user by means of the wearable contact lens; A method comprising the above.

2. The method according to claim 1, further comprising: The processor detecting a listening command for starting reception of the auditory signal by means of the wearable contact lens; The processor starting the listening mode by means of the wearable contact lens.

3. The listening command is a gesture performed by the user. The method according to claim 2.

4. The method according to claim 2, further comprising: The processor receiving a second command by means of the wearable contact lens; The processor ending the listening mode based on the second command by means of the wearable contact lens.

5. The listening command is received from the paired computing device via the personal network connection. The method according to claim 2.

6. The listening mode is started for a preset period, and The method is The method according to claim 2, further comprising the processor ending the listening mode based on the preset period by the wearable contact lens. The method according to claim 2. **Claim 7** The listening command is an auditory command of the user. The wearable contact lens operates in a startup mode when not operating in the listening mode. The method according to claim 2. **Claim 8** The wearable contact lens is provided with a camera for taking an image of the user's environment. The method comprises: the processor receiving a shooting command by the wearable contact lens; the processor analyzing the shooting command by the wearable contact lens; the processor starting image shooting of the user's field of view by the camera of the wearable contact lens. The method according to claim 1, further comprising the above. **Claim 9** The camera operates in a deactivated mode. The method comprises: the processor further activating the camera of the wearable contact lens by the wearable contact lens before the image shooting in response to the shooting command. The method according to claim 8. **Claim 10** The shooting command is a gesture executed by the user. The method according to claim 8. **Claim 11** The processor identifying the user's line of sight by the wearable contact lens in response to the shooting command; the processor recording the user's line of sight by the wearable contact lens simultaneously with the shooting command; the processor identifying a second line of sight of the user by the wearable contact lens; the processor determining a shooting intention based on the second line of sight and the recording of the line of sight by the wearable contact lens; the processor starting second image shooting of the user's second field of view based on the shooting intention by the camera of the wearable contact lens. The method according to claim 8, further comprising the above. **Claim 12** The second image shooting is started before analyzing a second shooting command. The method according to claim 11. **Claim 13** The auditory signal is a voice command of the user, The wearable contact lens includes a display configured to provide visual information to the user, The method is, The processor receives, by the wearable contact lens, a response to the voice command, The processor renders the response by the display of the wearable contact lens, The method according to claim 1, further comprising.

14. A memory including one or more instructions, A processor communicatively connected to the memory, The processor, in response to reading the one or more instructions, Receiving, by a wearable contact lens, an auditory signal from a user during a listening mode, wherein the wearable contact lens is disposed adjacent to the user's eye during the receiving, Identifying, by the wearable contact lens, a gesture performed by the user, Adjusting, by the wearable contact lens, a priority of the auditory signal based on the gesture, Connecting, from the wearable contact lens, to a paired computing device of the user via a personal network connection, Transferring, from the wearable contact lens, the auditory signal to the paired computing device of the user via the personal network connection, Transmitting, by the wearable contact lens, the priority to the paired computing device of the user, A system that performs.

15. The auditory signal is a voice command of the user, The wearable contact lens includes a display configured to provide visual information to the user, The processor is, Receiving, by the wearable contact lens, a response to the voice command, Rendering, by the display of the wearable contact lens, the response, The system according to claim 14, further performing.

16. The response is rendered as part of an extended reality image including the response and the user's field of view, The system according to claim 15.

17. One or more computer-readable storage media, Comprising program instructions collectively stored on the one or more computer-readable storage media, The program instructions are Receiving an auditory signal from a user during a listening mode by a wearable contact lens, wherein the wearable contact lens is disposed adjacent to the user's eye during the receiving; Identifying a gesture performed by the user by the wearable contact lens; Adjusting a priority order of the auditory signal based on the gesture by the wearable contact lens; Connecting from the wearable contact lens to a paired computing device of the user via a personal network connection; Transferring the auditory signal from the wearable contact lens to the paired computing device of the user via the personal network connection; Transmitting the priority order to the paired computing device of the user by the wearable contact lens; A computer program product that performs the above.

18. The program instructions are Detecting a listening command for starting reception of an auditory signal by the wearable contact lens; Starting the listening mode by the wearable contact lens; The computer program product according to claim 17, further performing the above.

19. The listening mode is started for a preset period, The program instructions further perform ending the listening mode by the wearable contact lens based on the preset period, The computer program product according to claim 18.

Images