A system, method, and storage medium for presenting images with a mobile device

Abstract

Systems, methods, and non-transitory computer-readable media for presenting images on a mobile device. An image is presented by monitoring one or more physical features surrounding the mobile device using at least one sensor, determining a context state of the mobile device in accordance with the monitored one or more physical features, selecting an image from a plurality of images associated with the determined context state, generating at least one overlay image from the selected image, and presenting the at least one overlay image using an optical assembly of the mobile device.

Description

[0001] This application claims priority to U.S. Application 16 / 824,250, filed March 19, 2020, entitled “Context-Based Image State Selection,” the contents of which are incorporated by reference in their entirety. TECHNICAL FIELD

[0002] The subject matter relates to mobile devices, e.g., eyewear devices, and more specifically, to visually presenting context-based images. BACKGROUND

[0003] Mobile devices, including cellular telephones and eyewear devices, such as smart glasses, headsets, and headsets, integrate image displays and cameras. The devices can capture and present images. Many mobile devices also integrate sensors capable of determining physical parameters around the mobile device. BRIEF DESCRIPTION OF DRAWINGS

[0004] The accompanying drawings are included to provide a further understanding of one or more embodiments and are incorporated in and constitute a part of this specification. The drawings are not intended to be restrictive in any way. In the drawings, like reference numerals refer to like structures throughout the various figures, wherein like reference numerals with an alphabetical suffix have different meanings as apposed to reference numerals without the alphabetical suffix. When referring to a plurality of structures of the same or similar elements, or when referring to a non-specific one of the same or similar elements, the alphabetical suffix can be omitted.

[0005] Figure 1A is a side view of an example hardware configuration of an eyewear device, including a visible light camera on a temple and a speaker on a temple.

[0006] Figure 1B and 1C is Figure 1A is a rear view of an example hardware configuration of an eyewear device, including two different types of image displays.

[0007] Figure 2 is Figure 1A is a top cross-sectional view of an eyewear device temple, showing a visible light camera, a head movement tracker, and a circuit board.

[0008] Figure 3A is a high-level functional block diagram of an example image selection and display system, including an eyewear device, a mobile device, and a server system connected through various networks;

[0009] Figure 3B is a simplified block diagram of a hardware configuration example of a server system for Figure 3A an audio visualizer system of

[0010] Figure 4 is a simplified block diagram of a mobile device hardware configuration.

[0011] Figure 5A , 5BAnd 5C are used to determine the background state of a mobile device and generate an overlapping image using an image selected based on the background state for use in... Figure 3A A flowchart illustrating example steps of an image selection and display system.

[0012] Figure 6A It is a diagram depicting a set of images corresponding to the background state.

[0013] Figure 6B , 6C 6D and 6E are perspective views of a scene observed through transparent optical components of an eye-wearing device, with a background state that is compatible with mobile devices. Figure 6B , 6C And 6D) or the physical parameters of the mobile device itself ( Figure 6E The corresponding overlapping image. Detailed Implementation

[0014] The following specific implementation includes examples of selecting images to display on a user's mobile device (e.g., an eye-wearing device) based on the background state of the mobile device's operation. For example, if the mobile device is in a car, it might select images related to driving; if it's in a store, it might select images related to shopping. This could be used to provide users of the mobile device with visual alerts / reminders related to the activity / background. For instance, driving-related images might include animated images of someone wearing a seatbelt, while shopping-related images might display signs indicating that a sale is in progress.

[0015] The detailed embodiments described below illustrate numerous specific details by way of example in order to provide a thorough understanding of the teachings. However, it will be apparent to those skilled in the art that these details can be practiced without them. In other instances, to avoid unnecessarily obscuring various aspects of this subject matter, well-known methods, processes, components, and circuits are described at a relatively high level without detailed description.

[0016] As used herein, the term "coupled" means any logical, optical, physical, or electrical connection, link, etc., that transmits an electrical signal generated or provided by one system element to another coupled element. Unless otherwise stated, coupled elements or devices are not necessarily directly connected to each other and may be separated by intermediate components, elements, or communication media that can modify, manipulate, or carry electrical signals. The term "on" means directly supported by one element, or indirectly supported by that element through another element integrated into or supported by that element. As used herein, the term "about" means ±10% of the stated amount.

[0017] The orientation of the mobile device, eyewear device, related components, and any kit incorporating a camera shown in any of the figures is for purposes of illustration and discussion only. In particular programmed operation, the device can be oriented in any other direction suitable for the particular application, such as up, down, sideways, or any other direction. Moreover, to the extent used herein, any directional terms, such as front, back, inward, outward, toward, left, right, lateral, longitudinal, up, down, high, low, top, bottom, and side, are used by way of example only, and do not limit the direction or orientation of any camera or camera component configured as described herein.

[0018] The following description will set forth a number of examples for purposes of illustration and discussion. The examples are not intended to limit the scope of the concepts, which are defined by the appended claims and their equivalents. The description of the examples is intended solely to assist in understanding the concepts.

[0019] Reference will now be made to the examples shown in the drawings and discussed below.

[0020] Figure 1A An example hardware configuration of a mobile device in the form of eyewear device 100 is described for determining context states in which the mobile device is operating and displaying images associated with those states. The mobile device can take other forms, such as a cell phone or tablet computer. Moreover, the eyewear device 100 can take other forms and can incorporate other types of frames, such as a headband, headset, or helmet. The eyewear device 100 includes at least a visible light camera 114 at a corner 110B of the frame for capturing images in a viewing area, such as a field of view. The illustrated eyewear device 100 also includes a speaker 115 and a microphone 116.

[0021] The visible light camera 114 is sensitive to the range of visible light wavelengths. As shown in the example, the visible light camera 114 has a front-facing field of view configured to capture images of a scene that is considered to be the optical assembly 180B from the perspective of the wearer. Examples of such visible light cameras 114 include high resolution complementary metal-oxide-semiconductor (CMOS) image sensors and video graphics array (VGA) cameras, such as 640p (e.g., 640 x 480 pixels, for a total of 0.3 megapixels), 720p, or 1080p. The eyewear device 100 captures image sensor data from the visible light camera 114, as well as optional other data such as geolocation data and audio data (via the microphone 116), digitizes the data using one or more processors, and stores the digitized data in memory. The term "field of view" refers to the area of observation that a user of the mobile device views with his or her eyes through the optical assembly 180 or that is presented on the mobile device display with information captured with the visible light camera 114.

[0022] The visible light camera 114 can be coupled to the image processor (element 312 in Figure 3A ) for digital processing and adding time stamps and location coordinates corresponding to the time and location coordinates of the captured scene images. The image processor 312 includes circuitry for receiving signals from the visible light camera 114 and processing the signals from the visible light camera 114 into a format suitable for storage in the memory (element 334 in Figure 3A ). The time stamps can be added by the image processor 312 or other processor that controls the operation of the visible light camera 114. The image processor 312 can additionally add location coordinates, such as location coordinates received from a global positioning system (element 331 in Figure 3A ).

[0023] The microphone 116 can be coupled to an audio processor for digital processing and adding time stamps indicating when audio was captured. The audio processor includes circuitry for receiving signals from the microphone 116 (or from the memory) and processing the signals into a format suitable for storage in the memory 334 and / or presentation by the speaker 115. The time stamps can be added by the audio processor or other processor that controls the operation of the speaker 115 and microphone 116.

[0024] As shown in Figure 1A , 1B and 1C, the eyewear device 100 includes a frame 105 with a left temple 107A connected to a right temple 107B by a bridge 106 that fits over a user's nose. The left and right temples 107A-B include respective eye apertures 175A-B that can each accommodate a respective optical assembly 180A-B. Left and right ear stems 125A-B extend from respective sides 170A-B of the frame 105, for example, through respective left and right temple tips 110A-B. Each ear stem 125A-B is connected to the frame 105 by a respective hinge 126A-B. The substrate or material that makes up the frame 105, temple tips 110, and ear stems 125A-B can include plastic, acetate, metal, or combinations thereof. The temple tips 110A-B can be integrated into or connected with the frame 105 and / or the ear stems 125A-B.

[0025] Although shown with two optical assemblies 180A-B, the eyewear device 100 can employ other arrangements, such as a single assembly or three optical assemblies, or different arrangements of optical assemblies 180A-B depending on the application or intended user of the eyewear device 100.

[0026] In one example, as shown in Figure 1BAs shown in FIG. 1, each optical assembly 180A-B includes a display matrix 171 and one or more optical layers 176A-N. The display matrix 171 can include a liquid crystal display (LCD), an organic light emitting diode display (OLED), or other such display. The one or more optical layers 176 can include lenses, optical coatings, prisms, mirrors, waveguides, light strips, and other optical components in any combination. The term lens as used herein refers to a transparent or translucent sheet of glass or plastic that has curved and flat surfaces that can cause light to converge / diverge or little or no light to converge / diverge.

[0027] The optical layers 176A-N can include prisms that are of suitable size and configuration and are equipped with a first surface for receiving light from the display matrix and a second surface for emitting light to the user's eye. The prisms of the optical layers 176A-N can extend to all or at least a portion of the respective apertures 175A-B formed in the left and right temples 107A-B so that the user sees the second surface of the prisms when looking through the corresponding left and right temples 107A-B with the eyes. The first surface of the prisms of the optical layers 176A-N is arranged to face upward along the frame 105 and the display matrix covers the prisms, ensuring that photons and light emitted from the display matrix strike the first surface. The size and shape of the prisms can ensure that light is refracted within the prisms and directed toward the user's eyes through the second surface of the prisms of the optical layers 176A-N. In this regard, the second surface of the prisms of the optical layers 176A-N can be convex to direct light toward the center of the eye. The size and shape of the prisms can be selected to magnify the image projected by the display matrix 171 and pass light through the prisms so that the image viewed from the second surface is larger in one or more dimensions than the image emitted from the display matrix 171.

[0028] In another example, as shown in FIG. 2, the image display device of the optical assembly 180A-B includes a projection image display. The illustrated projection image display includes a laser projector 150 (e.g., a three-color laser projector using a scanning mirror or galvanometer) and light strips 155A-N placed adjacent to one of the temple corners 110A-B of the eyewear device 100, where the light strips 155A-N are spaced apart over the lens width or lens depth between the front and back surfaces of the optical assembly 180A-B. Figure 1C

[0029] When photons projected from the laser projector 150 pass through the lenses of the optical assemblies 180A and 180B, the photons encounter the light strips 155A-N. When a particular photon encounters a particular light strip, the photon is either redirected to the user's eye or passed to the next light strip. By combining laser projector 150 modulation and light strip modulation, a particular photon or beam of light is controlled. In one example, the processor controls the light strips 155A-N by initiating a mechanical, acoustic, or electromagnetic signal. ​

[0030] In one example, the visible output produced on the optical assembly 180A-B of the eyewear device 100 includes an overlay image that covers at least a portion of the field of view through the optical assembly 180A-B. In one example, the optical assembly 180A-B is a see-through display that presents the overlay image as an overlay on the scene (or features within the scene) that the wearer views through the lenses of the optical assembly. In another example, the optical assembly 180A-B is a non-see-through display (e.g., an opaque display) that presents the overlay image by combining the overlay with a live image captured by the camera 114 of the eyewear device so as to be shown to the user on the display.

[0031] As noted above, the eyewear device 100 is coupled with a processor and memory, e.g., in the eyewear device 100 body or within other components of the system. The eyewear device 100 or system can then process the images captured by the eye, e.g., the coupled memory and processor in the system to process the images captured by the eye to track the eye movements. This processing of the captured images establishes a saccadic pathway to identify the movements of the user’s eye. The saccadic pathway includes a sequence or series of eye movements based on changes in the captured eye reflections. Eye movements are generally divided into fixations and saccades— when the eye pauses at a location and when the eye moves to another location, respectively. The resulting series of fixations and saccades is referred to as a saccadic pathway. Smooth pursuit describes the eye following a moving object. Fixational eye movements include microsaccadic movements: small and involuntary saccades that occur during attempts to fixate. The saccadic pathway is then utilized to determine the field of view adjustment.

[0032] An eye direction database can be established during the calibration process. Since the known fixed locations of the various points of interest are known during calibration, the saccadic pathway database can be used to establish a similarity to the calibration images previously obtained. Because the known fixed locations of the points of interest are known from the calibration images and recorded in the eye direction database, the eyewear device 100 can determine where the user’s eye is looking by comparing the currently captured eye image to the eye direction database. The calibration image that is most similar to the currently captured image can have a known fixed location of the point of interest as a good approximation of the eye direction of the currently captured image.

[0033] Figure 2 is Figure 1A Top cross-sectional view of the right temple corner 110B of the eyewear device 100 showing the right visible light camera 114, the head motion tracker 109, and the microphone 116. The left visible light camera is configured and mounted in substantially similar fashion to the right visible light camera 114, differing only in its connection and coupling on the left face 170A.

[0034] The right temple corner 110B includes a temple corner body and a temple corner cover, at Figure 2The cross-section of the right temple 125C is shown with the corner cover omitted. As shown, the eyewear device 100 includes a circuit board, which can be a flexible printed circuit board (PCB) 240, with controller circuitry for the right visible light camera 114, a microphone, low power wireless circuitry (e.g., for wireless short-range network communication via Bluetooth TM , high speed wireless circuitry (e.g., for wireless local area network communication via WiFi). The right hinge 126B connects the right corner 110B with the right temple 125C of the eyewear device 100. In some examples, components of the right visible light camera 114, the flexible PCB 140, or other electrical joints or contacts can be located on the right temple 125C or the right hinge 126B.

[0035] The head movement tracker 109 includes, for example, an inertial measurement unit (IMU). An IMU is an electronic device that combines the use of accelerometers and gyroscopes, and sometimes magnetometers, to measure and report an object's specific force, angular rate, and sometimes the magnetic field surrounding the object. The working principle of an IMU is to use one or more accelerometers to detect linear acceleration and one or more gyroscopes to detect rotational rates. A typical configuration of an IMU is three axes, each with an accelerometer, a gyroscope, and a magnetometer: a horizontal axis for left-right movement (X), a vertical axis (Y) for up-down movement, and a depth or distance axis for up-down movement (Z). The gyroscope detects the gravity vector. The magnetometer defines the rotation in the magnetic field (e.g., facing south, north, etc.), which acts like a compass that generates a heading reference. The three accelerometers detect acceleration along the above-defined horizontal, vertical, and depth axes, which can be defined relative to the ground, the eyewear device 100, or a user wearing the eyewear device 100.

[0036] The eyewear device 100 detects movement of the user of the eyewear device 100 by tracking the user head motion via the head movement tracker 109. The head movement includes a change in the head direction along the horizontal axis, the vertical axis, or a combination thereof relative to an initial head direction during presentation of an initial display image on the image display. In an example, tracking the user head motion via the head movement tracker 109 includes measuring, via the inertial measurement unit 109, an initial head direction along the horizontal axis (e.g., X-axis), the vertical axis (e.g., Y-axis), or a combination thereof (e.g., lateral or diagonal movement). Tracking the user head motion via the head movement tracker 109 also includes measuring, via the inertial measurement unit 109, a successive head direction along the horizontal axis, the vertical axis, or a combination thereof during presentation of the initial display image.

[0037] Figure 3Ais a high-level functional block diagram of an example image selection and display system 300. The image selection and display system 300 includes a mobile device, namely the eyewear device 100 in the example. The mobile device can communicate with other mobile devices 390 or server systems 398 over one or more wireless networks or wireless links. The image selection and display system 300 further includes the other mobile devices 390 and the server systems 398. The mobile devices 390 can be smartphones, tablets, laptops, access points, or other devices capable of connecting with the eyewear device 100 using, for example, low-power wireless connections 325 and high-speed wireless connections 337. The mobile devices 390 access the server systems 398 and the network 395. The network 395 can include any combination of wired and wireless connections.

[0038] The eyewear device 100 includes and supports the visible light camera 114, the speaker 115, the microphone 116, the user interface 301, the image displays of the optical assembly 180, the image display drivers 342, the image processor 312, the audio processor, the low-power circuitry 320, and the high-speed circuitry 330. Figure 3A The components for the eyewear device 100 shown in FIG. 1 are located on one or more circuit boards, such as PCBs or flexible PCBs, in the temples. Alternatively or additionally, the components shown can be located in the temples, the corners, the frame, the hinges, or the bridge of the eyewear device 100. The memory 334 includes a feature analyzer 344, a feature model 345, state selection programming 346, and content recognition programming 347 to perform the functions described herein for image selection and display. The memory 334 also includes a rendering engine 348 for rendering overlapping images on the displays 180A and 180B using the image processor 312 and the image display drivers 342.

[0039] The feature analyzer 344 executes instructions to cause the eyewear device 100 to process objects and / or aspects identified from a scene viewed through the eyewear device 100. The feature model 345 is a trained machine learning model to identify objects (such as doors, forks, plates, cars, windows, people, animals, etc.) and / or aspects (such as motions, straight lines, curves, materials). The state selection programming 346 implements instructions to cause the eyewear device 100 to determine a context state in which the eyewear device 100 is currently operating. The content recognition programming 347 implements instructions to cause the eyewear device 100 to select content (such as images) responsive to the determined context state.

[0040] As Figure 3AAs shown, the high-speed circuit 330 includes a high-speed processor 343, a memory 334, and a high-speed wireless circuit 336. In one example, an image display driver 342 is operated by the high-speed processor 343 to drive image display of the optical assembly 180. The high-speed processor 343 can employ any processor capable of implementing the high-speed communications and operations of any general computing system required by the eyewear device 100. The high-speed processor 343 includes the processing resources required to manage high-speed data transfers to a wireless local area network (WLAN) over a high-speed wireless connection 337 using the high-speed wireless circuit 336. In some examples, the high-speed processor 343 executes an operating system of the eyewear device 100, such as a LINUX operating system or other such operating system, and the operating system is stored in the memory 334 for execution. Among other tasks, the high-speed processor 343 executes the software architecture of the eyewear device 100, managing data transfers with the high-speed wireless circuit 336. In some examples, the high-speed wireless circuit 336 is configured to implement Institute of Electrical and Electronics Engineers (IEEE) 802.11 communication standards, also referred to herein as Wi-Fi. In other examples, the high-speed wireless circuit 336 implements other high-speed communication standards.

[0041] The low-power wireless circuit 324 and the high-speed wireless circuit 336 of the eyewear device 100 can include short-range transceivers (Bluetooth™) as well as wireless wide, local, or wide area network transceivers (e.g., cellular or WiFi). The mobile device 390 includes transceivers that communicate via the low-power wireless connection 325 and the high-speed wireless connection 337, and can be implemented using the details of the architecture of the eyewear device 100, as can other elements of the network 395.

[0042] The memory 334 includes a storage device capable of storing a variety of data and applications, among other things, including camera data generated by the visible light cameras 114A-B and the image processor 312, as well as images generated by the image display driver 342 for display on the optical assembly 180 image display and audio data generated by the microphone 116 and the audio processor. While the memory 334 is shown as integrated with the high-speed circuit 330, in other examples, the memory 334 can be a separate element of the eyewear device 100. In some examples, circuitry by wire can provide a connection from the image processor 312 / audio processor or the low-power processor 324 to the memory 334 through a chip including the high-speed processor 343. In other examples, the high-speed processor 343 can manage addressing of the memory 334 such that the low-power processor 324 will direct the high-speed processor 343 whenever a read or write operation involving the memory 334 is required.

[0043] The eyewear device 100 further includes a global positioning system 331, a compass 332, an inertial measurement unit 333. The GPS 331 is a satellite radio navigation system that receives geographic position and time information from GPS satellites. The compass 332 provides a direction relative to a geographic cardinal point (or point). The IMU 333 is an electronic device that measures and reports force, angular rate, and / or orientation using a combination of accelerometers, gyroscopes, and / or magnetometers.

[0044] The eyewear device 100 can connect with a host. For example, the eyewear device 100 can pair with a mobile device 390 via a high-speed wireless connection 337, or connect to a server system 398 via a network 395. In one example, the eyewear device 100 captures an image of a scene through the camera 114 and sends the image to the host for forwarding to the server system 398 for training of the feature model 364. In another example, the eyewear device 100 receives an image and / or instructions from the host.

[0045] The eyewear device 100 also includes other output and input components. The other output components include acoustic components (e.g., a speaker 115), haptic components (e.g., a vibratory motor), and other signal generators. The input components of the eyewear device 100, the mobile device 390, and the server system 398 can include alphanumeric input components (e.g., a keyboard, a touchscreen configured to receive alphanumeric input, a photo-optical keyboard, or other alphanumeric input components), point-based input components (e.g., a mouse, a trackpad, a trackball, a joystick, a motion sensor, or other pointing instruments), tactile input components (e.g., a physical button, a touchscreen that provides location and force of touches, or other tactile input components), audio input components (e.g., a microphone), and the like.

[0046] The image selection and display system 300 can select to include other peripheral device elements 319. Such peripheral device elements 319 can include biometric sensors, additional sensors, or display elements integrated with the eyewear device 100. For example, the peripheral device elements 319 can include any I / O components including output components, motion components, position components, or any other such elements described herein.

[0047] For example, the biometric component of the image selection and display system 300 includes components that detect expressions (e.g., hand expressions, facial expressions, vocal expressions, body gestures, or eye tracking), measure biosignals (e.g., blood pressure, heart rate, body temperature, perspiration, or brain waves), identify a person (e.g., voice identification, retinal identification, facial identification, fingerprint identification, or electroencephalogram-based identification), and the like. The motion component includes acceleration sensor components (e.g., accelerometer), gravitation sensor components, rotation sensor components (e.g., gyroscope), and so forth. The position component includes location sensor components (e.g., a Global Position System (GPS) receiver component), WiFi or Bluetooth™ transceiver

[0048] In one example, the image processor 312 includes a custom microprocessor integrated circuit (IC) for processing image sensor data from the visible light camera 114, and volatile memory for the microprocessor to operate from. To reduce the time taken for the image processor 312 to power up and process data, a non-volatile read-only memory (ROM) can be integrated on the IC with instructions to operate or boot the image processor 312. This ROM can be minimized to match the minimum size needed to provide the basic functionality needed to collect sensor data from the visible light camera 114, so that no additional functionality is present that would cause a delay in boot time. The ROM can be configured for direct memory access (DMA) to the volatile memory of the microprocessor of the image processor 312. DMA allows for inter-memory transfer from the ROM to the system memory of the image processor 312 without relying on the operation of the main controller of the image processor 312. Providing DMA to this boot ROM further reduces the time for the image processor 312 to power up until it can process and store sensor data from the visible light camera 114. In some examples, minimal processing of the camera signal from the visible light camera 114 is performed by the image processor 312, while additional processing can be performed by an application operating on the mobile device 390 or the server system 398.

[0049] The low-power circuit 320 includes a low-power processor 322 and a low-power wireless circuit 324. These elements of the low-power circuit 320 can be implemented as separate elements or as part of a single system-on-a-chip on a single IC. The low-power processor 324 contains logic for managing the other elements of the eyewear device 100. The low-power processor 324 is configured to receive input signals or instruction communications from the mobile device 390 over a low-power wireless connection 325. Further details regarding these instructions are described further below. The low-power wireless circuit 324 includes circuit elements for implementing a low-power wireless communication system over a short-range network. Bluetooth™ Smart, also known as Bluetooth™ Low Energy, is one standard implementation of a low-power wireless communication system that can be used to implement the low-power wireless circuit 324. In other examples, other low-power communication systems can be used.

[0050] The elements of the mobile device 390 and the network 395, the low-power wireless connection 325, and the high-speed wireless architecture 337 can be implemented using detailed information of the architecture of the mobile device 390, such as utilizing the short-range XCVR and the WWAN XCVR of the mobile device 390 described in U.S. Patent No. 8, 1 12, 114, which is incorporated by reference herein in its entirety. Figure 4

[0051] As shown in FIG. 3, the eyewear device 100 includes a high-speed circuit 330 and a low-power circuit 320. The high-speed circuit 330 includes a high-speed processor 332 and a high-speed wireless circuit 334. The low-power circuit 320 includes a low-power processor 322 and a low-power wireless circuit 324. These elements of the high-speed circuit 330 and the low-power circuit 320 can be implemented as separate elements or as part of a single system-on-a-chip on a single IC. Figure 3B As shown in FIG. 3, the eyewear device 100 includes a high-speed circuit 330 and a low-power circuit 320. The high-speed circuit 330 includes a high-speed processor 332 and a high-speed wireless circuit 334. The low-power circuit 320 includes a low-power processor 322 and a low-power wireless circuit 324. These elements of the high-speed circuit 330 and the low-power circuit 320 can be implemented as separate elements or as part of a single system-on-a-chip on a single IC.

[0052] In one example, the server system 398 receives scene images of the eyewear device 100 through the mobile device 390 and other eyewear devices over the network 395 for use by the neural network programming 365 to train the feature model 364. The server system 398 sends the trained feature model to the eyewear device 100 for use in identifying features corresponding to background states.

[0053] The feature model 346 of the eyewear device 100 can be a mirror of the feature model 364 of the server system 398. The feature model 346 of the eyewear device 100 is stored locally in read-only memory (ROM), erasable programmable read-only memory (EPROM), or flash memory of the high-speed circuit 330.

[0054] Figure 4 ​This is a high-level functional block diagram of an example mobile device 390 providing processing for the image selection and display system 300 of Figure 3. It shows a touchscreen element of the mobile device 390 with a feature analyzer 344, image retrieval programming 346, content recognition programming, and device position / orientation programming 348, as well as other applications such as chat applications. Examples of usable touchscreen mobile devices include (but are not limited to) smartphones, personal digital assistants (PDAs), tablets, laptops, or other portable devices. However, the structure and operation of touchscreen devices are provided by way of example, and the subject matter described herein is not limited thereto. Therefore, for the purpose of discussion, Figure 4 A block diagram of an exemplary mobile device 390 is provided, which has a touchscreen display for displaying content and receiving user input as (or partly as) a user interface. The mobile device 390 also includes a camera, such as a visible light camera, and a microphone 471.

[0055] like Figure 4 As shown, the mobile device 390 includes at least one digital transceiver (XCVR) 410, shown as a WWAN XCVR, for digital wireless communication via a wide-area wireless mobile communication network. The mobile device 390 also includes additional digital or analog transceivers, such as a short-range XCVR 420 for short-range network communication, for example, via NFC, VLC, DECT, ZigBee, Bluetooth™, or WiFi. Taking the short-range XCVR 420 as an example, it can take any available form of bidirectional wireless local area network (WLAN) transceiver, the type of which is compatible with one or more standard communication protocols implemented in the WLAN, such as a Wi-Fi standard according to IEEE 802.11 and WiMAX.

[0056] To generate location coordinates for locating the mobile device 390, the mobile device 390 may include a Global Positioning System (GPS) receiver 331. Alternatively, the mobile device 390 may utilize one or both of a short-range XCVR 420 and a WWAN XCVR 410 to generate location coordinates for positioning. For example, GPS-based positioning systems such as cellular networks, WiFi, or Bluetooth™ can generate very accurate location coordinates, especially when used in combination. These location coordinates can be transmitted to the eye-wearing device 100 via one or more network connections through the XCVR 420. Furthermore, the mobile device 390 may include a compass 332 and an inertial measurement unit 333 for determining orientation information.

[0057] The transceivers 410, 420 (network communication interfaces) are compliant with one or more of the various digital wireless communication standards used by modern mobile networks. Examples of WWAN transceivers 410 include, but are not limited to, transceivers configured to operate in accordance with Code Division Multiple Access (CDMA) and Third Generation Partnership Project (3GPP) network technologies, including but not limited to Third Generation Partnership Project 2 (or 3GPP2) and LTE, sometimes referred to as "4G." For example, the transceivers 410, 420 provide for two-way wireless communication of information including digitized audio signals, still and video images, web page information for display and input associated with web pages, and various types of mobile messaging communications with the mobile device 390 for user authorization policies.

[0058] The mobile device 390 also includes a microprocessor, shown as CPU 430. A processor is an electrical circuit having components constructed and arranged to perform one or more processing functions, typically various data processing functions. In the example, components that make up a programmable CPU are used, although discrete logic components can be used. For example, the microprocessor includes one or more integrated circuit (IC) chips that integrate electronic components that perform the functions of the CPU. The processor 430 can be based on any known or available microprocessor architecture, such as a Reduced Instruction Set Computing (RISC) using the ARM architecture, which is very common in mobile devices and other portable electronic devices today. Other processor circuits can be used to form the CPU 430 or processor hardware in smartphones, laptops, and tablets.

[0059] The microprocessor 430 acts as a programmable host controller for the mobile device 390 by configuring the mobile device 390 to perform various operations, for example, in accordance with instructions or programming that the processor 430 can execute. For example, such operations can include various general operations of the mobile device, as well as operations related to determining the device location when capturing images and determining the device location and orientation when generating and presenting image overlays. Although the processor can be configured by employing hardwired logic, typical processors in mobile devices are general processing circuits that are configured by executing programming.

[0060] The mobile device 390 includes a memory or storage system for storing data and programming. In the example, the memory system can include flash memory 440A and random access memory (RAM) 440B. The RAM 440B is used as short-term storage of instructions and data for processing by the processor 430, for example, as working data processing memory. The flash memory 440A generally provides longer-term storage.

[0061] Depending on the device type, mobile device 390 stores and runs a mobile operating system through which specific applications can be executed, which may include a feature analyzer 344, a feature model 345, state selection programming 346, content recognition programming 347, and a rendering engine 348. However, in some implementations, programming may be implemented in firmware or a combination of firmware and application layers. For example, instructions for capturing scene images, recognizing features, analyzing features, selecting states, recognizing content, and generating overlays may exist in the firmware (e.g., using a dedicated GPU or VPU SOC). Instructions for generating visible output to the user may exist in the application. Applications (such as state selection programming 346, content recognition programming, and other applications) may be native applications, hybrid applications, or web applications running on mobile device 390 (e.g., dynamic web pages executed by a web browser). Examples of mobile operating systems include Google Android, Apple iOS (for iPhones or iPads), Windows Mobile, Amazon Fire OS, RIM BlackBerry OS, and others.

[0062] Figure 5A , 5B 5C and 5C are flowcharts 500, 530, and 550, respectively, illustrating example operations of a mobile device (such as eye-wearing device 100) or mobile device 390, as well as other components of the image selection and display system 300. Although shown in sequence, one or more boxes in flowcharts 500, 530, and / or 550 may be reordered or parallelized depending on the implementation.

[0063] The flowchart below refers to an example where the mobile device is an eye-wearing device 100 for selecting and displaying images. It will be understood that the functions described with reference to eye-wearing device 100 can be performed by other eye-wearing devices and other mobile devices such as mobile phones and tablets. From the description herein, appropriate modifications to the implementation of the following operations using other mobile devices (including devices with see-through displays and devices with non-see-through displays such as touchscreens) can be readily understood.

[0064] refer to Figure 5A In flowchart 500, in block 502, the eye-wearing device 100 monitors physical features. Physical features are characteristics surrounding or associated with the eye-wearing device 100 that can be used to determine the current context of using the eye-wearing device. By way of non-limiting example, physical features include speed, time of day, location, surrounding images, etc. The eye-wearing device 100 monitors physical features using one or more sensors such as a camera 114, a microphone 116, a GPS 331, a compass 332, and an inertial measurement unit. The eye-wearing device 100 can store physical features in memory 334 for retrieval during processing.

[0065] In block 504, the eyewear device 100 determines its current operating context state. The eyewear device 100 processes the physical features monitored in step 502 to determine its context state. The context state of the eyewear device 100 represents the settings, situation, and / or location in which the eyewear device 100 is currently operating. By way of non-limiting example, the context states include dining, night, shopping, driving, outing, and moving.

[0066] In one example, the eyewear device 100 retrieves its surrounding images and other physical features from the memory 334 and applies object recognition to the images using the feature analyzer 344 and the feature models 345. If plates and forks are identified in the images, the eyewear device 100 determines that the current context is dining. If the sun is recognized, the eyewear device 100 determines that the current context is outing. If there can be multiple contexts, such as outing and dining, the eyewear device 100 can select the most appropriate situation, for example, by weighting the recognized objects (e.g., plates or scenery) according to whether the wearer's gaze is directed at the object or by applying the feature models 345 on the images to select the most appropriate context according to past training examples.

[0067] In another example, the eyewear device 100 receives location information (e.g., from the GPS 331) and speed information (e.g., based on GPS signals from the GPS 331 or inputs from the IMU 333). If the eyewear device 100 is located in a store and the speed is less than 3 miles per hour, the eyewear device 100 determines that the current context is shopping. Alternatively, if the eyewear device 100 is located in a lane and the speed is 65 miles per hour, the eyewear device 100 determines that the current context is driving in a car. The feature analyzer 344 can apply the feature models 345 to the images to refine the context, for example, driving a car versus being a passenger and riding a car.

[0068] In block 506, the eyewear device 100 selects an image associated with the determined context state (block 504). Each potential context state is associated with a state, an image, and an associated image stored in the memory 334. Upon determining the context state, the image processor 312 of the eyewear device 100 selects the appropriate image by retrieving the image associated with the determined context state from the memory 334.

[0069] In one example, the user / wearer of the eyewear device 100 selects a theme, such as avocados, from a plurality of available configuration themes during a configuration phase. According to this example, the theme is associated with a collection of a plurality of images, where each image corresponds to a potential context state. Figure 6AA potential theme 600 (i.e., avocados) is illustrated, which includes a collection of images representative of six potential states 602a-f associated with a cartoon avocado 604. There are six images associated with the cartoon avocado 606a-f, one for each potential state 602a-f (i.e., the night state 602b is associated with the avocado under a blanket 606b, and the out state 602e is associated with the avocado under sunglasses 606e). In another example, the user / wearer can design their own theme by selecting images corresponding to each potential background state.

[0070] In block 508, the eyewear device 100 generates an overlay image including the selected image for presentation by the eyewear device 100. In one example, the image processor 312 of the eyewear device 100 generates the overlay image by analyzing the images within the field of view captured by the camera 114 (representative of the images viewed by the user / wearer through the eyewear device 100), and positioning the selected image so that it does not obscure recognized objects in the field of view. In another example, the image processor 312 of the eyewear device 100 positions the selected image so that it is always located in the same position within the user’s field of view (e.g., located above and to the right of center). In another example, the image processor 312 of the eyewear device 100 registers the selected image to a particular object (e.g., a plate) and continually adjusts the position of the selected image in the field of view so that it is always in the same position relative to the particular object (e.g., in a 10 o’clock position adjacent to the edge of the plate), as described below with reference to Figure 5B .

[0071] In block 510, the eyewear device 100 presents the overlay image on the optical assembly 180 of the eyewear device 100. The image processor 312 of the eyewear device 100 displays the replacement overlay image on the image display of the optical assembly 180 via the image display driver 342.

[0072] Figure 6B An example scene with a field of view as seen through the optical assembly 180 of the eyewear device 100, including an overlay image, is depicted, including an overlay image. In the illustrated example, the scene is a lane of a roadway viewed from the sidewalk outside of a car. The overlay image includes a “moving” avocado 606f associated with the moving environment 602f (e.g., determined in accordance with block 504 based on speed and position relative to the roadway). As shown, the image display driver 342 moves the avocado 606f to a central position located in an unobstructed area (e.g., the lane 610) of the roadway. Figure 6B

[0073] Figure 6C ​Another example scene is depicted as seen through optical assembly 180, where the scene is a hallway viewed from within a car. The overlaid image includes a "driving" avocado 606d associated with the driving environment 602d (e.g., determined in accordance with block 504 from speed, road position, identification of cars 620a,b on the road through object recognition, and identification of objects within the car through object recognition, such as steering wheel 622, driver seat 624a, and front passenger seat 624b, etc.). As shown, driving avocado 606d is registered to a position between car seats 624a and b, as described below, with reference to Figure 5B .

[0074] Figure 6D Another example scene is depicted as seen through optical assembly 180, where the scene includes eating at a table. The overlaid image includes a "dining" avocado 606a associated with the dining environment 602a (e.g., identification of objects such as plate 630a and fork 630b through object recognition, determined in accordance with block 504). As shown, dining avocado 606a is registered to a position of plate 630a (e.g., 10 o'clock), as described below, with reference to Figure 5B .

[0075] In block 512, eyewear device 100 monitors parameters of eyewear device 100, such as battery level. By way of non-limiting example, other parameters include WiFi signal level, error, etc.

[0076] In decision block 514, a decision is made as to the monitored parameter. If the monitored parameter exceeds a threshold, processing occurs in block 516. It is noted that the basis for the decision can be a parameter that exceeds an upper limit (as shown in Figure 5A ), a parameter that reaches a limit (e.g., a critical error code), a parameter that falls below a lower limit (e.g., battery charge level, minimum signal level for effective communication, etc.), or other types of comparisons. If the monitored parameter does not exceed a threshold, processing occurs at block 502, and the selection process repeats and the replacement image is selected as the background for the change in operation of eyewear device 100.

[0077] In block 516, eyewear device 100 selects an overlay image that is responsive to the monitored parameter that exceeds a threshold (block 514). In one example, when the battery level of eyewear device 100 is below a predefined level (e.g., 10%), an overlay image indicating sleep (e.g., night content image 606b shown in Figure 6E or other images) will be selected to replace the background image (which will be Figure 6D in contrast to Figure 6E ).

[0078] In block 518, the eyewear device 100 generates an overlay image including the selected overlay image for presentation by the eyewear device 100. The image processor 312 of the eyewear device 100 generates the replacement overlay image, as described above with reference to block 508.

[0079] In block 520, the eyewear device 100 presents the overlay image on the optical assembly 180 of the eyewear device 100. The image processor 312 of the eyewear device 100, through the image display driver 342, presents the replacement overlay image, as described above with reference to block 510.

[0080] Figure 6E An example scene with a field of view is depicted that is seen through the optical assembly 180 of the eyewear device 100, including an overlay image, including a replacement overlay image. In the example shown, the scene is a dining scene, with the dining background avocado 606a replaced with a night / sleep avocado 606b, for example, to gently remind the user that it is time to charge the eyewear device 100. Figure 6D

[0081] Referring to the flowchart 530 of Figure 5B In block 532, the eyewear device 100 identifies one or more physical features (e.g., a plate, a fork, a steering wheel, a backrest, etc.). The feature analyzer 344 can determine the confidence level of the features by applying the feature models 345 to the images captured by the eyewear device.

[0082] In block 534, the eyewear device 100 determines the confidence of the physical features. The feature analyzer 344 can determine the confidence level of the features by applying the feature models 345 to the images captured by the eyewear device.

[0083] In block 536, the eyewear device 100 registers the selected image to the physical feature with the highest confidence. The image processor 312 can register the selected image to the physical feature.

[0084] In block 538, the eyewear device 100 generates an overlay image of the selected image adjacent to the registered physical feature. The image processor 312 of the eyewear device 100 generates the overlay image, as described above with reference to block 508.

[0085] Referring to the flowchart 550 of Figure 5C In block 552, the eyewear device 100 identifies its location. The eyewear device can use GPS coordinates in the GPS 331 and a database including specific locations (such as stores, restaurants, parks, historical landmarks stored in a database accessible to the processor 343, for example) to identify its location.

[0086] In block 554, the eyewear device 100 determines supplemental information for the identified location. The supplemental information can include pricing and sales information, among other information.

[0087] ​In block 556, the eyewear device 100 modifies the image to incorporate supplemental information. For example, the shopping avocado 606c can be modified to add the word "SALE" (as shown) or the current applicable percentage discount on the shopping bag. In this way, the system is able to provide the user with more context in a friendly manner. Figure 6A

[0088] Any of the methods described herein, such as the feature analyzer 344, the feature model 345, the state selection programming 346, and the programming of the rendering engine 348 for the eyewear device 100, the mobile device 390, and the server system 398, can be embodied in one or more methods, as method steps or as one or more application programs as previously described. According to some examples, an "application program" or "firmware" is a program that performs a function defined in the program, such as logic embodied in software or hardware instructions. The one or more application programs can be generated using various programming languages such as object-oriented programming languages (e.g., Objective-C, Java, or C++) or procedural programming languages (e.g., C or assembly language). In a particular example, a third-party application program (e.g., an application program developed using the ANDROID™ or IOS™ software development kit (SDK) by an entity other than the vendor of the particular platform) can be mobile software running on a mobile operating system such as IOS, ANDROID™, WINDOWS® Phone, and the like. In this example, the third-party application program can invoke the API calls provided by the operating system to facilitate the functionality described herein. The application programs can be stored in any type of computer-readable medium or storage device, such as a storage disc, or memory inside a one or more general purpose computers. Further, the methods and processes disclosed herein can also be embodied in a special purpose computer, or in application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or complex programmable logic devices (CPLDs).

[0089] ​Program aspects of the technology can be thought of as "products" or "articles of manufacture" typically in the form of executable code and / or associated data that is carried or otherwise embodied in some transitory or non-transitory medium, such as some semiconductor memory. For example, a programming code can be implemented in a tangible storage medium such as a compact diskette, a floppy disk, a hard disk, a memory stick, a Flash drive, a register block of a processor, or any other machine-readable medium including a processor of a computer, processor of an article of manufacture, or processor of another device. Programming code in whole or in part can be downloaded from a website or transmitted from a file server to a recipient computer or processor. Such machines or processors can transform a physical article of manufacture into a different state upon execution of the programming code. The programming code can transform the physical, tangible article of manufacture in whole or in part during execution simply by way of its dynamic properties of operation. Alternatively or additionally, the programming code can transform a physical article of manufacture in whole or in part by way of the processor's state of operation, including without limitation, the processor's electrical paths, which physically transform during execution. The programming code can be implemented in a high level procedural or object-oriented programming language to convey the instructions, declarations, data structures, and the like to be operated on by a processor. The programming code can be implemented in assembly or machine language, if desired. It is to be further understood that throughout this discussion a "processor" can be one or more processors, and a "computer" can be one or more computers. Thus, a processor can be a single processor or multiple processors, and a computer can be a single computer or multiple computers.

[0090] Hence, a machine readable medium can take many forms of tangible storage medium. For example, a non-volatile memory medium includes the optical, magnetic or semiconductor storage medium, alone or in combination, for example, optical magnetic or semiconductor systems, or any other information storage disks, including hard drives, floppy disks, and the like. A volatile memory medium includes dynamic memory, such as a main memory of some computer platforms, which can be implemented as a random access memory ("RAM") or the like, as discussed elsewhere herein. A transmission medium can include a wire, cable, or printed material carrying signals and can include electromagnetic energy carrying signals, such as a radio frequency ("RF") or infrared ("IR") signals. Thus, the communication media / medium of a computer readable medium can further be any medium that can communicate, propagate or transport programming code. Herein, the term "programming code" can be used interchangeably with the term "code" or "software."

[0091] The scope of protection is defined entirely by the appended claims. This scope is intended and should be interpreted as being as broad as when interpreted in accordance with this specification, and the general meaning of the language used in the claims is consistent with the following examination history and is interpreted to include all structural and functional equivalents. No claim contains any subject matter intended to include, nor should it be interpreted in this manner, any subject matter that fails to meet the requirements of Sections 101, 102, or 103 of the Patent Act. No unplanned coverage of such subject matter is claimed herein.

[0092] Except as described above, the contents described and shown are not intended to be, nor should they be construed as, causing any component, step, function, object, benefit, advantage, or equivalent to be proprietary to the public, whether or not it is stated in the claims.

[0093] It should be understood that, unless otherwise specified herein, the terms and expressions used herein have the general meanings assigned to them in relation to their respective fields of research. Relational terms such as “first” and “second” may be used only to distinguish one entity or action from another, without necessarily requiring or implying any actual relationship or order between the entities or actions. The terms “comprising,” “including,” “containing,” “included,” or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes or comprises a list of elements or steps includes not only those elements or steps but also other elements or steps not expressly listed or inherent to the process, method, article, or apparatus. Without further limitation, an element preceded by “a” does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0094] Unless otherwise stated, all measurements, numerical values, ratings, positions, grades, dimensions, etc., described in this specification (including the following claims) are approximate values, not precise values. Such quantities are intended to have a reasonable range consistent with the function they pertain to and with the conventions of the field to which they pertain. For example, unless expressly stated otherwise, parameter values, etc., may differ from the specified quantities by ±10%.

[0095] Furthermore, in the foregoing specific embodiments, various examples combine different features for the purpose of simplifying this disclosure. This method of disclosure should not be construed as reflecting an intention that the claimed examples require more features than those expressly stated in each claim. Rather, as reflected in the following claims, the subject matter to be protected does not consist of all the features of any single disclosed example. Therefore, the following claims are hereby incorporated into the specific embodiments, each claim existing independently as a separate subject matter for protection.

[0096] While the forgoing describes what is considered to be the best mode and other examples, it is not to be understood that the application disclosed herein is limited in its use or application to this exact construction and examples. It is therefore clear that various modifications can be made in the application disclosed herein without departing from the scope of the concept thereof. The following claims are intended to cover any and all modifications and equivalents.

Claims

1. A system for presenting images with a mobile device, comprising: a mobile device, comprising: an optical assembly, wherein the optical assembly has a viewing area for viewing a scene in a field of view, and is configured to present an overlay image to a user on the scene in the viewing area; at least one sensor configured to determine one or more physical features surrounding the mobile device; a processor coupled to the mobile device; a memory accessible to the processor, the memory comprising a plurality of themes, each theme having a plurality of related images for each image associated with a respective context state, a first theme of the plurality of themes having a first image associated with the first theme and associated with a first respective context state in the memory, and a second image associated with the first theme and associated with a second respective context state in the memory, the second image being different from the first image, a second theme of the plurality of themes having a third image associated with the second theme and associated with the first respective context state in the memory, the third image being different from the first image, and a fourth image associated with the second theme and associated with the second respective context state in the memory, the fourth image being different from the second image; and programming in the memory, wherein execution of the programming by the processor configures the system to perform functions, wherein the functions comprise: receiving a theme selection from a user selecting one of the plurality of themes; monitoring, using the at least one sensor, one or more physical features surrounding the mobile device; determining a context state of the mobile device based on the monitored one or more physical features; selecting an image from the plurality of related images associated with the determined context state and the theme selection of the user, the selected image comprising the first image retrieved from the plurality of related images in the memory when the theme selection is the first of the plurality of themes, and the third image retrieved from the plurality of related images in the memory when the theme selection is the second of the plurality of themes, when the context state is determined to be the first respective context state; generating at least one overlay image from the selected image, wherein the selected image is positioned within the field of view for presentation; presenting the at least one overlay image on the optical assembly; determining a different context state of the mobile device based on the monitored one or more physical features; selecting another image from the plurality of related images associated with the determined different context state, the selected another image comprising the second image retrieved from the plurality of related images in the memory when the theme selection is the first of the plurality of themes, and the fourth image retrieved from the plurality of related images in the memory when the theme selection is the second of the plurality of themes, when the context state is determined to be the second respective context state; generating at least one replacement overlay image from the selected another image, wherein the selected another image is positioned within the field of view at the same location for presentation; and presenting the at least one replacement overlay image on the optical assembly. presenting the at least one replacement overlay image on the optical assembly.

2. The system of claim 1, wherein, the mobile device is an eyewear device, comprising: a frame supporting the optical assembly; and temple arms extending from sides of the frame; wherein the optical assembly is a see-through optical assembly supported by the frame.

3. The system of claim 1, wherein, the optical assembly includes a camera for capturing images of a scene and a display for displaying captured images and the at least one overlay image.

4. The system of claim 1, wherein, the processor executing program code further configures the system to perform additional functions, wherein the additional functions include: monitoring a parameter of the mobile device; determining when the monitored parameter exceeds a threshold; selecting an overlay image from the plurality of relevant images responsive to the monitored parameter exceeding the threshold; generating at least one replacement overlay image from the selected image; and presenting the at least one replacement overlay image on the optical assembly.

5. The system of claim 1, wherein, the processor executing program code further configures the system to perform additional functions, wherein the additional functions include: identifying a physical feature in a field of view of the optical assembly; and registering the image to the physical feature; wherein the function of generating the at least one overlay image from the selected image includes positioning the image adjacent to the physical feature to which it is registered.

6. The system of claim 1, wherein, the processor executing program code configures the system to generate overlay images, wherein the functions include: modifying the selected image responsive to the one or more physical features surrounding the mobile device.

7. The system of claim 1, wherein, the processor executing program code further configures the system to perform additional functions, wherein the additional functions include: identifying a location of the mobile device using the at least one sensor; and determining supplemental information for the selected image associated with the identified location; wherein the processor executing program code configures the system to generate overlay images includes the function of modifying the selected image to include the supplemental information.

8. The system of claim 1, wherein, the one or more physical features surrounding the mobile device include at least one physical object, a location coordinate, a time of day, or a speed.

9. A method of presenting images with a mobile device, the method comprising: receiving a theme selection from a user selecting one of a plurality of themes in a memory, a first theme of the plurality of themes having a first image associated with the first theme and associated with a first corresponding background state in the memory, and a second image associated with the first theme and associated with a second corresponding background state in the memory, the second image being different from the first image; a second theme of the plurality of themes having a third image associated with the second theme and associated with the first corresponding background state in the memory, the third image being different from the first image, and a fourth image associated with the second theme and associated with the second corresponding background state in the memory, the fourth image being different from the second image; monitoring one or more physical features surrounding the mobile device using at least one sensor; determining a background state of the mobile device based on the monitoring of the one or more physical features; and presenting an image associated with the determined background state on the mobile device. selecting an image from a plurality of related images associated with the determined context state and the theme option of the user, the selected image comprising a first image retrieved from the plurality of related images in the memory when the theme option is a first one of the plurality of themes, and a third image retrieved from the plurality of related images in the memory when the theme option is a second one of the plurality of themes in determining the first respective context state; generating at least one overlaid image from the selected image, wherein the selected image is positioned within a field of view for presentation; presenting the at least one overlaid image with an optical assembly of the mobile device; determining a different context state of the mobile device in accordance with the monitored one or more physical features; selecting another image from the plurality of related images associated with the determined different context state, the selected another image comprising a second image retrieved from the plurality of related images in the memory when the theme option is the first one of the plurality of themes in determining the second respective context state, and a fourth image retrieved from the plurality of related images in the memory when the theme option is the second one of the plurality of themes in determining the second respective context state; generating at least one replacement overlaid image from the selected another image, wherein the selected another image is positioned at the same location within the field of view for presentation; and presenting the at least one replacement overlaid image on the optical assembly. the mobile device is an eyewear device comprising a frame supporting the optical assembly and temples extending laterally from the frame; wherein the optical assembly is a see-through optical assembly supported by the frame.

10. The method of claim 9, wherein, 11. The method of claim 9, further comprising: monitoring parameters of the mobile device; determining when the monitored parameters exceed a threshold; selecting an overlay image from the plurality of related images responsive to the monitored parameters exceeding the threshold; generating at least one replacement overlaid image from the selected image; and presenting the at least one replacement overlaid image on the optical assembly.

12. The method of claim 9, further comprising: identifying physical features in a field of view of the optical assembly; registering the image to the physical features; wherein the generating the at least one overlaid image from the selected image comprises positioning the image adjacent to the physical features to which it is registered. the generating the overlaid image comprises: modifying the selected image responsive to the one or more physical features surrounding the mobile device.

13. The method of claim 9, wherein, 14. The method of claim 9, further comprising: identifying a location of the mobile device using the at least one sensor; and determining supplemental information of the selected image associated with the identified location; wherein the generating the overlaid image comprises modifying the image to incorporate the supplemental information. the one or more physical features surrounding the mobile device comprises at least one physical object, location coordinates, time of day, or velocity. ​ ​ 15. The method of claim 9, wherein, ​ 16. A non-transitory computer readable medium storing program code that, when executed, can cause an electronic processor of a mobile device to perform the steps of: receiving a theme selection from a user selecting one of a plurality of themes in a memory, a first theme of the plurality of themes having a first image associated with the first theme and associated with a first corresponding background state in the memory, and a second image associated with the first theme and associated with a second corresponding background state in the memory, the second image being different than the first image; a second theme of the plurality of themes having a third image associated with the second theme and associated with the first corresponding background state in the memory, the third image being different than the first image, and a fourth image associated with the second theme and associated with the second corresponding background state in the memory, the fourth image being different than the second image; monitoring one or more physical features surrounding the mobile device using at least one sensor; determining a background state of the mobile device from the monitoring of the one or more physical features; selecting an image from a plurality of related images associated with the determined background state and the user theme selection, the selected image comprising the first image retrieved from the plurality of related images in the memory when the theme selection is the first of the plurality of themes, and the third image retrieved from the plurality of related images in the memory when the theme selection is the second of the plurality of themes, in determining the first corresponding background state; generating at least one overlay image from the selected image, wherein the selected image is positioned within a field of view for presentation; presenting the at least one overlay image with an optical assembly of the mobile device; determining a different background state of the mobile device from the monitoring of the one or more physical features; selecting another image from the plurality of related images associated with the determined different background state, the selected other image comprising the second image retrieved from the plurality of related images in the memory when the theme selection is the first of the plurality of themes, and the fourth image retrieved from the plurality of related images in the memory when the theme selection is the second of the plurality of themes, in determining the second corresponding background state; generating at least one replacement overlay image from the selected other image, wherein the selected other image is positioned in the same location within the field of view for presentation; and presenting the at least one replacement overlay image on the optical assembly. the stored program code, when executed, generates the at least one overlay image by:

17. The non-transitory computer-readable medium of claim 16, wherein, identifying a physical feature in a field of view of the optical assembly; and registering the image to the physical feature; wherein the generating the at least one overlay image from the selected image comprises positioning the image adjacent to the physical feature to which it is registered. the stored program code, when executed, generates the at least one overlay image by: identifying a physical feature in a field of view of the optical assembly; and 18. The non-transitory computer-readable medium of claim 16, wherein, registering the image to the physical feature; wherein the generating the at least one overlay image from the selected image comprises positioning the image adjacent to the physical feature to which it is registered. identifying a location of the mobile device using the at least one sensor; determining supplemental information for the selected image associated with the identified location; wherein the generating the overlaid image includes modifying the image to incorporate the supplemental information.

Images