Geospatial image surface processing and selection

Abstract

Systems, methods, and non-transitory computer readable media for displaying images on a mobile device. The method includes monitoring a location and orientation of the mobile device, requesting a previously captured image corresponding to a captured location of the previously captured image relative to a server location, receiving the previously captured image from the server, selecting an image from among the requested previously captured images in response to the location and orientation of the mobile device, and transmitting the selected image to the server. In one embodiment, the method includes generating an overlay image from a selected image, including image icons associated with the selected image, displaying the overlay image on an optical component, receiving an image selection identifying one of the image icons in the displayed overlay image, and displaying the selected image associated with the identified image icon in a viewing region of the optical component, and displaying the image.

Description

This application is a divisional application of the invention patent with the invention name “Surface processing and selection of geospatial images”, the application date is February 25, 2021, and the application number is 202180022350.6. CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. patent application Ser. No. 16 / 821,188, filed on Mar. 17, 2020, and entitled “Geospatial Imagery Surface Processing and Selection,” the contents of which are incorporated herein by reference in their entirety. Background Art

[0002] Mobile devices, including cell phones and eyewear such as smart glasses, headbands, and headsets, integrate image displays and cameras. These devices can capture and display images. Many mobile devices also incorporate sensors that can determine the physical location of the mobile device. BRIEF DESCRIPTION OF THE DRAWINGS

[0003] The accompanying drawings illustrate one or more embodiments by way of example only, and not by way of limitation.In the accompanying drawings, like reference numerals indicate the same or similar elements.

[0004] Figure 1A is a side view of an example hardware configuration for an eyewear device that includes a visible-light camera on the corner of the eyepiece and a speaker on the temple.

[0005] Figure 1B and Figure 1C yes Figure 1A Rear view of an example hardware configuration for an eyewear device in Figure 1, including two different types of image displays.

[0006] Figure 2A and Figure 2B yes Figure 1A Rear view of an example hardware configuration for a goggles in , including eye tracking hardware.

[0007] Figure 2C is a diagram describing the technology for tracking eye movements.

[0008] Figure 2D yes Figure 1A A cross-sectional view from above of the eyewear corner, showing the visible light camera, head tracker, and circuit board.

[0009] Figure 3 is a high-level functional block diagram of an example image selection and display system that includes eyewear, mobile devices, and server systems connected via various networks.

[0010] Figure 4 A simplified block diagram showing an example hardware configuration for a mobile device

[0011] Figure 5A and 5B is used to capture and distribute images for Figure 3 Flowchart of example steps used in the image selection and display system.

[0012] Figure 5C 、 5D , 5E and 5F are used to display the overlapping of images for Figure 3 Flowchart of example steps used in the image selection and display system.

[0013] Figure 6A A perspective view of a scene as viewed through the see-through optical assembly of the eyewear device, where selectable images correspond to image capture locations relative to the user's physical location.

[0014] Figure 6B is through Figure 6A A perspective view of a scene as viewed by a perspective optical component, with a selected image displayed by the perspective optical component. DETAILED DESCRIPTION

[0015] The following detailed description includes examples for viewing audio and video content captured by others on a user's mobile device (e.g., a mobile eyewear device), based on a capture location relative to the location of the user's mobile device. This enables the user of the mobile device to see nearby activities and details, mimicking the functionality of high-powered binoculars and enabling the user to see content that would otherwise be obstructed from the user's field of view from their current location (e.g., seeing a performer on a stage inside a restaurant while the user is outside). In one example, content captured by others (or previously captured by the user) is tagged with location coordinates (e.g., GPS coordinates) and stored on a server. The user's mobile device provides its current location to the server, which retrieves content corresponding to that location and sends it to the mobile device. Mobile devices overlay icons related to the scene content being viewed on a mobile device for the user to select. When the user selects an icon, the device displays the related content.

[0016] The following detailed description sets forth numerous specific details by way of example in order to provide a thorough understanding of the relevant teachings. However, it should be apparent to one skilled in the art that the present teachings can be practiced without these details. In other instances, generally accepted methods, procedures, components, and circuit descriptions are provided at a relatively high level without detailed description to avoid unnecessarily obscuring aspects of the present teachings.

[0017] As used herein, the term "coupled" refers to any logical, optical, physical, or electrical connection, link, or the like, by which an electrical signal generated or provided by one system element is transmitted to another coupled element. Unless otherwise specified, coupled elements or devices are not necessarily directly connected to each other and may be separated by intermediate components, elements, or communication media that can modify, manipulate, or carry electrical signals. The term "on..." refers to being directly supported by an element or indirectly supported by the element through another element integrated into or supported by the element. As used herein, the term "about" means ±10% of the stated amount.

[0018] The orientations of mobile devices, eyewear, related components, and any assembly incorporating a camera shown in any of the figures are examples only and are used for purposes of illustration and discussion. During operation of a particular program, the device may be oriented in any other orientation suitable for a particular application, such as up, down, sideways, or any other orientation. Furthermore, for purposes of this disclosure, any directional terms, such as front, back, inward, outward, toward, left, right, lateral, longitudinal, up, down, high, low, top, bottom, and side, are used only as examples and do not limit the direction or orientation of any camera or camera assembly configured as described herein.

[0019] The following description will partially set forth the objects, advantages, and novel features of the examples. Such other objects, advantages, and novel features will be apparent to those skilled in the art upon review of the following description and the accompanying drawings, or may be learned and grasped by making or operating the examples. The objects and advantages of the subject matter may be realized and obtained by the methods, means, and combinations particularly pointed out in the appended claims.

[0020] Reference will now be made in detail to the examples illustrated in the accompanying drawings and discussed below.

[0021] Figure 1A An example hardware configuration for a mobile device for capturing and displaying content (e.g., visual content such as images and video and audio) in the form of an eye-mounted device 100 is described. The mobile device may take other forms, such as a mobile phone or tablet. In addition, the eye-mounted device 100 may take other forms and may be combined with other types of frames, such as headgear, headphones, or helmets. The eye-mounted device 100 includes at least one visible light camera 114 located at a corner 110B of the eyeglasses for capturing images in a visible area (e.g., a field of view). The illustrated eye-mounted device 100 also includes a speaker 115 and a microphone 116.

[0022] Visible light camera 114 is sensitive to wavelengths in the visible light range. As shown in this example, visible light camera 114 has a frontal field of view from the wearer's perspective, configured to capture images of the scene being viewed through optical assembly 180B. Examples of such visible light cameras 114 include high-resolution complementary metal oxide semiconductor (CMOS) image sensors and video graphics array (VGA) cameras, such as 640p (e.g., 640×480 pixels, for a total of 0.3 megapixels), 720p, or 1080p. The eye-mounted device 100 captures image sensor data from visible light camera 114, and optionally other data such as geolocation data and audio data (via microphone 116), digitizes the data using one or more processors, and stores the digitized data in memory. The term "field of view" is intended to describe the viewing area that a mobile device user can view with their eyes through optical assembly 180, or to display information captured by visible light camera 114 on a display of the mobile device.

[0023] The visible light camera 114 may be coupled to an image processor ( Figure 3 Image processor 312 includes circuitry for receiving signals from visible light camera 114 and processing the signals from visible light camera 114 into a format suitable for storage in memory. Figure 3 334 in FIG. 334). A timestamp may be added by the image processor 312 or other processor controlling the operation of the visible light camera 114. The image processor 312 may additionally add location coordinates, such as those received from a global positioning system (GPS). Figure 3 Component 331).

[0024] The microphone 116 may be coupled to the audio processor ( Figure 3 The audio processor 313 includes circuitry that receives signals from the microphone 116 (or from memory) and processes them into a format suitable for storage in the memory 334 and / or presentation using the speaker 115. The timestamp may be added by the audio processor 313 or another processor that controls the operation of the speaker 115 and microphone 116.

[0025] like Figure 1A 、 1BAs shown in FIG1C , the eyewear device 100 includes a frame 105, wherein a left frame 107A is connected to a right frame 107B via a nose bridge 106 that conforms to the user's nose. The left and right frames 107A-B include respective apertures 175A-B that house respective optical components 180A-B. For example, left and right temples 125A-B extend from sides 170A-B of the frame 105 via respective left and right corners 110A-B, respectively. Each temple 125A-B is connected to the frame 105 via a respective hinge 126A-B. The substrate or material forming the frame 105, corners 110, and temples 125A-B may include plastic, cellulose acetate plastic, metal, or a combination thereof. The corners 110A-B may be integrated with or connected to the frame 105 and / or temples 125A-B.

[0026] Although shown as having two optical components 180A-B, the eye-mounted device 100 may also adopt other arrangements, such as a single or three optical components, or may adopt different arrangements of the optical components 180A-B depending on the application of the eye-mounted device 100 or the specific circumstances of the intended user.

[0027] In one example, as shown in FIG1B , each optical assembly 180A-B includes a display matrix 171 and one or more optical layers 176A-N. The display matrix 171 may include a liquid crystal display (LCD), an organic light emitting diode (OLED) display, or other such display. The one or more optical layers 176 may include lenses, optical coatings, prisms, reflectors, waveguides, optical strips, and any combination of other optical components. The term lens, as used herein, refers to a curved and / or flat transparent or translucent glass or plastic member used for covering, with or without light converging / diverging properties.

[0028] Optical layers 176A-N may include prisms of suitable size and configuration, with a first surface for receiving light from the display matrix and a second surface for emitting light toward the user's eyes. The prisms of optical layers 176A-N may extend into all or at least a portion of the respective openings 175A-B formed in the left and right frames 107A-B, such that the second surfaces of the prisms are visible to the user when the user looks through the respective left and right frames 107A-B. The first surfaces of the prisms of optical layers 176A-N are arranged to face upward along the frame 105, with the display matrix overlying the prisms, ensuring that photons and light emitted from the display matrix strike the first surfaces. The prisms are sized and shaped to ensure that light is refracted within the prisms and directed toward the user's eyes by the second surfaces of the prisms of optical layers 176A-N. In this regard, the second surfaces of the prisms of optical layers 176A-N may be convex to direct light toward the center of the eye. The prisms may be sized and shaped selectively to magnify the image projected by the display matrix 171 and to pass light through the prisms so that the image viewed from the second surface is larger than the image emitted from the display matrix 171 in one or more dimensions.

[0029] In another example, Figure 1C As described above, the image display device of the optical assembly 180A-B includes a projected image display. The projected image display includes a laser projector 150 (e.g., a three-color laser projector using a scanning mirror or a galvanometer) and light strips 155A-N. The laser projector 150 is placed near one of the lens corners 110A-B of the eye-mounted device 100. The light strips 155A-N are spaced apart across the width of the lens of the optical assembly 180A-B or across the depth of the lens between the front and back surfaces of the lens.

[0030] When photons projected from laser projector 150 pass through the lenses of optical assemblies 180A and 180B, they encounter light strips 155A-N. When a particular photon encounters a particular light strip, it is either redirected to the user's eye or passed to the next light strip. The specific photons or beams can be controlled by combining laser projector 150 modulation with light strip modulation. In one example, a processor controls light strips 155A-N by activating mechanical, acoustic, or electromagnetic signals.

[0031] In one example, the visible light output generated by the optical components 180A-B of the eye-mounted device 100 includes an overlay image that covers at least a portion of the field of view of the optical components 180A-B. In one example, the optical components 180A-B are see-through displays that present the overlay image as an overlay on a scene (or features within a scene) viewed by the wearer through the lenses of the optical components. In another example, the optical components 180A-B are not see-through displays (e.g., opaque displays) and display the overlay image by combining an overlay with a real-time image captured by the camera 114 of the eye-mounted device for presentation to the user on a display.

[0032] Figure 2A is a rear view of an example hardware configuration of the eye-mounted device 100, which includes an eye tracker 213 on the frame 105 for tracking the eye movements of a user of the eye-mounted device 100. The eye tracker 213 of the eye-mounted device 100 includes an infrared emitter 215 and an infrared camera 220. Visible light cameras typically include a blue light filter to block infrared light detection. In the example, the infrared camera 220 is a visible light camera, such as a low-resolution video graphics array (VGA) camera (e.g., 640×480 pixels, a total of 0.3 megapixels) with the blue light filter removed. The infrared emitter 215 and the infrared camera 220 are co-located on the frame 105, for example, both are shown connected to the upper portion of the left eye frame 107A. As described in further detail below, the frame 105 or one or more of the left and right eye corners 110A-B include a circuit board for the infrared emitter 215 and the infrared camera 220. For example, the infrared emitter 215 and the infrared camera 220 can be connected to the circuit board by soldering.

[0033] Other arrangements of the infrared emitter 215 and the infrared camera 220 may be implemented, including an arrangement where both the infrared emitter 215 and the infrared camera 220 are located on the right frame 107B, or at different locations on the frame 105, such as the infrared emitter 215 on the left frame 107A and the infrared camera 220 on the right frame 107B. In another example, the infrared emitter 215 is located on the frame 105 and the infrared camera 220 is located on one of the corners 110A-B, or vice versa.

[0034] Essentially, the infrared emitter 215 may be attached to any location on the frame 105, the left corner 110A, or the right corner 110B for emitting a pattern of infrared light 250 directed at the user's eye 252 ( Figure 2C Similarly, infrared camera 220 may be attached to substantially anywhere on frame 105, left corner 110A, or right corner 110B to capture at least one reflective change 254 in the infrared light emission pattern of the user's eye.

[0035] Infrared emitter 215 and infrared camera 220 are positioned inwardly toward the user's eyes, covering a portion or all of the eye angle, to capture infrared images of the eyes and thereby track eye movements. For example, infrared emitter 215 and infrared camera 220 may be positioned directly in front of the eyes, on top of frame 105, or in corners 110A-B at either end of frame 105. Eye movement includes changes in eye direction from the initial eye direction along the horizontal axis, the vertical axis, or a combination thereof, during the presentation of an initial display image on the image display of optical assemblies 180A-B.

[0036] The eye tracker 213 can track eye movements by measuring the point in the eye's gaze direction (the location at which the user is looking within the optical components 180A-B of the eye-mounted device 100), comparing the currently captured image with a previously captured calibration image, or detecting the position of the eyes relative to the head. For example, the eye tracker 213 non-invasively measures eye movements using video images from which the eye position is extracted. As described above, a pattern of infrared light is emitted by the infrared emitter 215, and the infrared light is reflected back from the eye, with changes in the reflected light being sensed and imaged by a camera, such as the infrared camera 220. The data forming the captured infrared image is then analyzed to extract eye movement from the changes in the reflection. This type of video-based eye tracker typically uses corneal reflection (illustrated by the first Purkinje spot) and pupil center as features to be tracked over time. In a second example, a dual Purkinje spot eye tracker uses reflections from the anterior cornea (illustrated by the first Purkinje spot) and the posterior lens (illustrated by the fourth Purkinje spot) as features to be tracked. In a third example, image features from inside the eye, such as retinal blood vessels, are tracked and appear as the user's eye moves.

[0037] Before using the eye tracker 213 to track eye position, the eye-mounted device 100 may be calibrated based on the unique anatomical features of the user's eyes. Typically, the user gazes at a point or a series of points, and the eye tracker 213 records values corresponding to each gaze point. Prior to presenting an initial display image via the image display of the optical components 180A-B, the eye-mounted device 100 calibrates the eye tracker 213 by presenting a series of calibration images to the user's eyes via the image display of the optical components 180A-B. Each calibration image has a respective target point at a respective known fixed position on the horizontal and vertical axes. In response to the user's eyes viewing the respective target points, the eye-mounted device 100 records the position of the eye's anatomical features relative to the respective known fixed points of the respective target points in an eye direction (e.g., scan path) database.

[0038] After calibration, the video-based eye tracker 213 can focus on one or both eyes of a user and record the eye movements of a user (e.g., the wearer of the eye-mounted device 100) as they gaze at the image display of the optical components 180A-B. When infrared or near-infrared non-collimated light shines on the pupil of the eye as pupil-visible infrared light through the infrared emitter 215, a corneal reflection is generated in the change in reflection of the infrared light. The vector between the center of the pupil and the corneal reflection in the captured infrared image contains the change in reflection of the infrared light and can be used to calculate the gaze point on the surface or the direction of the eye's gaze.

[0039] Two common infrared and near-infrared (also known as active light) eye tracking techniques can be used: bright pupil and dark pupil. Whether bright or dark pupil is used depends on the position of the illumination source (infrared emitter 215) relative to the infrared camera 220 and the user's eye. If the illumination from infrared emitter 215 is coaxial with the optical path, when the light reflects off the retina, the eye acts as a reflector, creating a bright pupil effect similar to red eye. If the illumination from infrared emitter 215 is off-path, the pupil appears dark because the return reflection from the retina is directed away from the infrared camera 220.

[0040] In one example, the infrared emitter 215 of the eye tracker 213 emits infrared illumination, which may be near-infrared light or other short-wavelength beams of low-energy radiation, at the user's eye. Alternatively, or in addition, the eye tracker 213 may include an emitter that emits wavelengths of light other than infrared light, and the eye tracker 213 may also include a camera sensitive to the wavelength to receive and capture images having the wavelength. For example, the eye tracker 213 may include a visible light camera, such as a red-green-blue (RGB) camera, that captures light from the eye in the visible light range.

[0041] As described above, the eye-mounted device 100 is coupled to a processor and memory, such as within the eye-mounted device 100 itself or another part of the system. The eye-mounted device 100 or the system can then process the captured eye images. For example, the memory and processor coupled to the system process the captured eye images to track eye movements. This processing of the captured images establishes a scan path to identify the user's eye movements. The scan path includes a sequence or series of eye movements based on changes in captured eye reflections. Eye movements are generally categorized into fixations and saccades—when the eye gaze remains at a certain location and when the eye moves to another location, respectively. The resulting series of fixations and saccades is called a scan path. Smooth pursuits describe the eyes following a moving object. Fixed eye movements include microsaccades: small, involuntary glances that occur during attempted fixations. The scan path is then used to determine field of view adjustments.

[0042] An eye direction database can be built during calibration. Since the known fixed positions of the respective target points are known during calibration, this scan path database can be used to establish similarities with previous calibration images. Because the known fixed positions of the target points are known from the calibration images and recorded in the eye direction database, the eyewear device 100 can determine where the user's eyes are looking by comparing the currently captured eye image with the eye direction database. The calibration image that is most similar to the currently captured image may have the known fixed positions of the target points, which can be used as a good approximation of the eye direction for the currently captured image.

[0043] Figure 2B 2 is a rear view of another example hardware configuration of the eye-mounted device 200. In this example configuration, the eye-mounted device 200 is depicted as having an eye tracker 213 on the right corner 210B for tracking the eye movements of the user of the eye-mounted device. As shown, an infrared emitter 215 and an infrared camera 220 are co-located on the right corner 210B. The eye tracker 213 or one or more components of the eye tracker 213 can alternatively or additionally be located at the left corner 210A and other locations of the eye-mounted device 200, such as the frame 105. The eye tracker 213 has an infrared emitter 215 and an infrared camera 220 and Figure 2A , but the eye tracker 213 may be modified to be sensitive to different wavelengths of light, as previously described in Figure 2A As described in.

[0044] Figure 2D yes Figure 1A A top cross-sectional view of the center corner of the eyewear device 100 is shown, showing the right visible light camera 114, the head motion tracker 109, and the microphone 116. The left visible light camera is substantially similar in configuration and mounting location to the right visible light camera 114, differing only in that it is connected and coupled to the left side surface 170A. As shown, the eyewear device 100 includes a circuit board, which may be a flexible printed circuit board (PCB) 240. A right hinge 126B connects the right corner 110B to the right temple 125B of the eyewear device 100. In some examples, the right visible light camera 114, the flexible PCB 240, or other electrical connectors or contacts may be located on the right temple 125B or the right hinge 126B.

[0045] For example, the head motion tracker 109 includes an inertial measurement unit (IMU). An IMU is an electronic device that uses a combination of accelerometers and gyroscopes, and sometimes a magnetometer, to measure and report the specific force, angular velocity, and sometimes the magnetic field surrounding the body. The IMU works by detecting linear acceleration using one or more accelerometers and detecting rotation rate using one or more gyroscopes. A typical configuration of an IMU includes an accelerometer, a gyroscope, and a magnetometer for each of three axes: the horizontal axis (X) for left-right movement, the vertical axis (Y) for top-bottom movement, and the depth axis or distance axis (Z) for up-down movement. The gyroscope detects the gravity vector. The magnetometer defines rotation in the magnetic field (such as facing south, north, etc.), acting like a compass that generates a heading reference value. The three accelerometers are used to detect acceleration along the horizontal, vertical, and depth axes defined above, which can be defined relative to the ground, the eye-mounted device 100, or the user wearing the eye-mounted device 100.

[0046] The eye-mounted device 100 detects movement of a user of the eye-mounted device 100 by tracking the user's head movement via the head motion tracker 109. Head movement includes a change in head orientation along a horizontal axis, a vertical axis, or a combination thereof relative to an initial head orientation during presentation of an initial display image on the image display. In one example, tracking the user's head movement via the head motion tracker 109 includes measuring the initial head orientation along a horizontal axis (e.g., an X-axis), a vertical axis (e.g., a Y-axis), or a combination thereof (e.g., lateral or diagonal movement) via the inertial measurement unit 109. Tracking the user's head movement via the head motion tracker 109 also includes measuring continuous head orientation along the horizontal axis, the vertical axis, or a combination thereof during presentation of the initial display image via the inertial measurement unit 109.

[0047] The right mirror corner 110B includes a mirror corner body 110B and a mirror corner cap. Figure 2D The corner cap is omitted from the cross-section shown. Disposed within the right corner 110B are various interconnected circuit boards, such as printed circuit boards or flexible printed circuit boards, which include controller circuitry for the right visible light camera 114, a microphone, low-power wireless circuitry (such as for communicating via Bluetooth), and a plurality of other circuits. TM wireless short-range network communications), high-speed wireless circuits (such as for wireless local area network communications via WiFi).

[0048] Flexible printed circuit board 240 is disposed within right corner 110B and is coupled to one or more other components within right corner 110B. Although shown as being formed on a circuit board within right corner 110B, right visible light camera 114B may also be formed on a circuit board within left corner 110A, temples 125A-B, or frame 105.

[0049] Figure 3is a high-level functional block diagram of an example image selection and display system 300. The image selection and display system 300 includes a mobile device, which in this example is the eye-mounted device 100. The mobile device can communicate with other mobile devices 390 or a server system 398 via one or more wireless networks or wireless links. The image selection and display system 300 also includes other mobile devices 390 and a server system 398. The mobile device 390 can be a smartphone, tablet, laptop, access point, or other such device capable of connecting to the eye-mounted device 100 using, for example, a low-power wireless connection 325 and a high-speed wireless connection 337. The mobile device 390 connects to the server system 398 via a network 395. The network 395 can include any combination of wired and wireless connections.

[0050] The eye-mounted device 100 includes and supports a visible light camera 114 , a speaker 115 , a microphone 116 , a user interface 301 , an image display of the optical assembly 180 , an image display driver 342 , an image processor 312 , an audio processor 313 , a low-power circuit 320 , and a high-speed circuit 330 . Figure 3 The components for the eye-mounted device 100 shown in FIG are located on one or more circuit boards, such as printed circuit boards or flexible printed circuit boards, in the temples. Alternatively, or in addition, the components shown may be located in the corners, frames, hinges, or nose bridges of the eye-mounted device 100. The memory 334 includes image capture program programming 344, image retrieval program programming 345, and device position / orientation program programming 346 to perform the functions described herein for image selection and display. The memory 334 further includes a rendering engine 348 that uses the image processor 312 and the image display driver 342 to render the overlaid images on the displays 180A and 180B.

[0051] The image capture program 344 executes instructions to cause the eye-mounted device 100 to capture an image of a scene using the visible light camera 114 and add a timestamp and location coordinates. The image retrieval program 345 executes instructions to cause the eye-mounted device 100 to request an image from the server system 398 or the memory 334 based on the location at which the image was captured, i.e., relative to the current location of the eye-mounted device 100. The device position / orientation program 346 executes instructions to cause the eye-mounted device 100 to determine the current location of the eye-mounted device 100 and determine the orientation of the eye-mounted device (e.g., determining the field of view through the optical assembly 180).

[0052] like Figure 3As shown, the high-speed circuitry 330 includes a high-speed processor 343, memory 334, and high-speed wireless circuitry 336. In this example, the image display driver 342 is operated by the high-speed processor 343 to drive the image display of the optical assembly 180. The high-speed processor 343 can be any processor capable of implementing the high-speed communications and operations required by any general-purpose computing system for the eye-mounted device 100. The high-speed processor 343 includes the processing resources required to manage high-speed data transmission to a wireless local area network (WLAN) over a high-speed wireless connection 337 using the high-speed wireless circuitry 336. In some examples, the high-speed processor 343 executes an operating system for the eye-mounted device 100, such as a Linux operating system or other such operating system, and the operating system is stored in the memory 334 for execution. In addition to any other tasks, the high-speed processor 343, which executes the software architecture of the eye-mounted device 100, manages data transmission using the high-speed wireless circuitry 336. In some examples, the high-speed wireless circuitry 336 is configured to implement the Institute of Electrical and Electronics Engineers (IEEE) 802.11 communication standard, also referred to herein as Wi-Fi. In other examples, high-speed wireless circuitry 336 implements other high-speed communication standards.

[0053] The low-power wireless circuit 324 and the high-speed wireless circuit 336 of the eyewear device 100 may include a short-range transceiver (Bluetooth TM ) and a wireless wide area, local area, or wide area network transceiver (such as a cellular network or WiFi). The mobile device 390 includes a transceiver that communicates via a low power wireless connection 325 and a high speed wireless connection 337 and can be implemented using details of the architecture of the eyewear device 100, as well as other elements of the network 395.

[0054] The memory 334 comprises a storage device capable of storing various data and applications, including, among other things, camera data generated by the visible light camera 114 and the image processor 312, images generated by the image display driver 342 for display on the image display of the optical assembly 180, and audio data generated by the microphone 116 and the audio processor 313. While the memory 334 is shown as being integrated with the high-speed circuitry 330, in other examples, the memory 334 may be a separate component of the eye-mounted device 100. In some examples, circuitry may provide a connection from the image processor 312 / audio processor 313 or the low-power processor 324 to the memory 334 via a chip including the high-speed processor 343. In other examples, the high-speed processor 343 may manage addressing of the memory 334 so that the low-power processor 324 will direct the high-speed processor 343 whenever a read or write operation involving the memory 334 is required.

[0055] The eyewear device 100 also includes a global positioning system (GPS) 331, a compass 332, and an inertial measurement unit (IMU) 333. The GPS 331 is a receiver for a satellite-based radio navigation system that receives geographic location and time information from GPS satellites. The compass 332 provides direction relative to geographic cardinal directions (or points). The IMU 333 is an electronic device that uses a combination of accelerometers, gyroscopes, and / or magnetometers to measure and report force, angular rate, and / or orientation.

[0056] The eye-mounted device 100 can be connected to a host. For example, the eye-mounted device 100 can be paired with a mobile device 390 via a high-speed wireless connection 337, or connected to a server system 398 via a network 395. In one example, the eye-mounted device 100 captures an image of a scene through the camera 114 and sends the image (along with location and timestamp information) to the host for forwarding to the server system 398. In another example, the eye-mounted device 100 receives images and / or instructions from the host.

[0057] The eye-mounted device 100 also includes other output components and input components. Other output components include acoustic components (such as speakers 115), tactile components (such as vibration motors), and other signal generators. The input components of the eye-mounted device 100, the mobile device 390, and the server system 398 may include alphanumeric input components (such as keyboards, touch screens configured to receive alphanumeric input, optical keyboards, or other alphanumeric input components), point-based input components (such as mice, touch pads, trackballs, joysticks, motion sensors, or other pointing devices), tactile input components (such as physical buttons, touch screens or other tactile input components that provide touch location and force or touch gestures), audio input components (such as microphones), and the like.

[0058] The image selection and display system 300 can optionally include additional peripheral device elements 319. Such peripheral device elements 319 can include biometric sensors, additional sensors, or display elements integrated with the eye-mounted device 100. For example, the peripheral device elements 319 can include any input / output components, including output components, motion components, position components, or any other such components described herein.

[0059] For example, the biometric component of the image selection and display system 300 includes a component for detecting expressions (such as hand expressions, facial expressions, voice expressions, body postures, or eye tracking), measuring biological signals (such as blood pressure, heart rate, body temperature, sweating, or brain waves), identifying people (such as voice recognition, retinal recognition, facial recognition, fingerprint recognition, or electroencephalogram-based recognition), etc. The motion component includes an acceleration sensor component (such as an accelerometer), a gravity sensor component, a rotation sensor component (such as a gyroscope), etc. The position component includes a position sensor component for generating position coordinates (such as a global positioning system (GPS) receiver component), a WiFi or Bluetooth receiver for generating positioning system coordinates, etc. TM transceiver, altitude sensor components (such as an altimeter or barometer that detects air pressure from which altitude can be derived), direction sensor components (such as a magnetometer), etc. Such positioning system coordinates can also be received from mobile device 390 via low power wireless circuitry 324 or high speed wireless circuitry 336 over wireless connections 325 and 337.

[0060] In one example, image processor 312 includes a microprocessor integrated circuit (IC) customized to process image sensor data from visible light camera 114, as well as volatile memory used by the microprocessor for operation. To reduce the time image processor 312 spends processing data upon power-up, a non-volatile read-only memory (ROM) containing instructions for operating or booting image processor 312 may be integrated on the IC. This ROM may be minimized to match the minimum size required to provide the basic functionality of collecting sensor data from visible light camera 114, eliminating the need for additional functionality that would delay startup boot time. The ROM may be configured with direct memory access (DMA) to the volatile memory of the image processor 312 microprocessor. DMA allows data to be transferred from the ROM to the image processor 312 memory independently of the operation of the image processor 312's main controller. Providing DMA to the boot ROM further reduces the amount of time from power-up until image processor 312 is ready to process and store sensor data from visible light camera 114. In some examples, the camera signal from visible light camera 114 is minimally processed by image processor 312 , and additional processing may be performed by an application running on mobile device 390 or server system 398 .

[0061] The low power circuit 320 includes a low power processor 322 and a low power radio circuit 324. These elements of the low power circuit 320 may be implemented as separate elements or may be implemented on a single integrated circuit as part of a system on a single chip. The low power processor 324 includes logic for managing the other elements of the eye-mounted device 100. The low power processor 324 is configured to receive input signals or instructions communicated from the mobile device 390 via the low power wireless connection 325. Additional details related to such instructions are further described below. The low power radio circuit 324 includes circuit elements that implement a low power wireless communication system over a short range network. Bluetooth TM Smart, also known as Bluetooth Low Energy TM , which is a standard implementation of a low power wireless communication system, may be used to implement low power wireless circuitry 324. In other examples, other low power communication systems may be used.

[0062] The elements of the mobile device 390 and the network 395, the low power wireless connection 325 and the high speed wireless architecture 337 may be implemented using architectural details of the mobile device 390, such as using Figure 4 The short-range transceiver and wireless wide area network transceiver of the mobile device 390 described in.

[0063] Figure 4 It is for Figure 3 , a high-level functional block diagram of an example mobile device 390 is provided for processing the image selection and display system 300. Shown are elements of a touch screen type mobile device 390 loaded with image capture program programming 344, image retrieval program programming 345, and device location / orientation program programming 346, as well as other applications, such as a chat application. Examples of touch screen type mobile devices that may be used include, but are not limited to, smartphones, personal digital assistants (PDAs), tablet computers, laptop computers, or other portable devices. However, the structure and operation of touch screen type devices are provided by way of example, and the subject technology described herein is not limited thereto. Therefore, for purposes of discussion, Figure 4 A block diagram of an exemplary mobile device 390 is provided that has a touch screen display for displaying content and receiving user input as (or part of) a user interface. The mobile device 390 also includes a camera 470, such as a visible light camera, and a microphone 471.

[0064] like Figure 4As shown, the mobile device 390 includes at least one digital transceiver (XCVR) 410, shown as a wireless wide area network transceiver, namely WWAN XCVR, for digital wireless communication via a wide area wireless mobile communication network. The mobile device 390 also includes additional digital or analog transceivers, such as a short-range transceiver 420 for short-range network communication, for example by means of NFC, VLC, DECT, ZigBee, Bluetooth TM , or WiFi. Taking the short-range transceiver 420 as an example, it can take the form of any available two-way wireless local area network (WLAN) transceiver, the type of which is compatible with one or more standard communication protocols implemented in wireless local area networks, such as one of the Wi-Fi standards according to IEEE 802.11 and WiMAX.

[0065] To generate location coordinates for locating the mobile device 390, the mobile device 390 may include a global positioning system (GPS) receiver 331. Alternatively, or in addition, the mobile device 390 may utilize one or both of the short-range transceiver 420 and the wireless wide area network transceiver 410 to generate location coordinates for positioning. For example, a positioning system based on GPS, a cellular network, WiFi, or Bluetooth TM Very accurate position coordinates can be generated, especially when used in combination. Such position coordinates can be transmitted to the eye-mounted device 100 via the transceiver 420 through one or more network connections. In addition, the mobile device 390 may include a compass 332 and an inertial measurement unit 333 for determining direction information.

[0066] Transceivers 410, 420 (network communication interfaces) conform to one or more digital wireless communication standards used by modern mobile networks. Examples of wireless wide area network transceivers 410 include, but are not limited to, transceivers configured to operate in accordance with Code Division Multiple Access (CDMA) and Third Generation Partnership Project (3GPP) network technologies, including, but not limited to, Third Generation Partnership Project 2 (or 3GPP2) and LTE, sometimes referred to as "4G." For example, transceivers 410, 420 provide two-way wireless information communication, including digitized audio signals, still images and video signals, web page information for display and input related to the web page, and various types of mobile messaging communications with mobile device 390 for user authorization policies.

[0067] The mobile device 390 also includes a microprocessor, such as the CPU 430 shown. A processor is a circuit whose components are constructed and arranged to perform one or more processing functions, typically various data processing functions. Although discrete logic components can be used, components that constitute a programmable CPU are used in the example. For example, a microprocessor includes one or more integrated circuit (IC) chips that integrate electronic components that perform CPU functions. Taking the processor 430 as an example, it can be based on any known or available microprocessor architecture, such as a reduced instruction set computing (RISC) using the ARM architecture, which is very common in today's mobile devices and other portable electronic devices. Other processor circuits can be used to form the CPU 430 or processor hardware in smartphones, laptops, and tablets.

[0068] The microprocessor 430 acts as a programmable host controller for the mobile device 390 by configuring the mobile device 390 to perform various operations, such as according to instructions or program programming executable by the processor 430. For example, such operations may include various general operations of the mobile device, as well as operations related to determining the device's position when capturing images and determining the device's position and orientation when generating and presenting image overlays. Although the processor can be configured by employing hard-wired logic, a typical processor in a mobile device is a general processing circuit that is configured by executing program programming.

[0069] Mobile device 390 includes a memory or storage device system for storing data and program programming. In this example, the memory system may include flash memory 440A and random access memory (RAM) 440B. Random access memory 440B serves as short-term storage for instructions and data processed by processor 430, such as working data processing memory. Flash memory 440A generally provides longer-term storage.

[0070] Depending on the device type, the mobile device 390 stores and runs a mobile operating system through which specific applications are executed, which may include image capture programming 344, image retrieval programming 345, device position / orientation programming 346, and a rendering engine 348. However, in some implementations, programming may be implemented in firmware or a combination of firmware and application layers. For example, instructions for capturing scene images, tracking device position and orientation information, and generating overlays may reside in firmware (such as with a dedicated GPU or VPU SOC). Instructions for generating visible output to the user may reside in an application. Applications, such as audio visualization programming 344 and other applications, may be native applications, hybrid applications, or web applications (such as dynamic web pages executed by a web browser) running on the mobile device 390. Examples of mobile operating systems include Google Android, Apple iOS (for iPhone or iPad devices), Windows Mobile, Amazon FireOS, RIM BlackBerry operating system, etc.

[0071] Figure 5A 、 5B 5C, 5D, 5E, and 5F are flowcharts 500, 505, 520, 530, 532, and 536, respectively, illustrating example operations of a mobile device (e.g., eye-mounted device 100) or mobile device 390 and other components of the image selection and display system 300. Although shown as appearing sequentially, one or more blocks in flowchart 500, flowchart 505, flowchart 520, flowchart 530, flowchart 532, and / or flowchart 536 may be reordered or parallelized depending on implementation.

[0072] The following flowchart is described with reference to an example in which the mobile device is a goggles-mounted device 100 that can capture and display images. It is understood that the functions described with reference to goggles-mounted device 100 can be performed by other goggles-mounted devices and other mobile devices, such as mobile phones and tablets. Based on the description herein, it is readily understood that appropriate modifications to implement the following content on other mobile devices (including devices with see-through displays and devices with non-see-through displays, such as touchscreens) can be readily understood.

[0073] refer to Figure 5AReferring to flowchart 500 of FIG. 5 , in block 502 , the eye-mounted device 100 captures content (e.g., images, audio, and associated metadata). The eye-mounted device 100 captures images, such as using the visible light camera 114, and stores the images in the memory 334. Furthermore, the eye-mounted device 100 captures location coordinates and timestamp information, such as using the global positioning system 331 and the image processor 312, and stores the location coordinates and timestamp information in the memory 334. The eye-mounted device 100 stores the location coordinates and timestamp information as metadata associated with each image it captures. Images, audio, and metadata are captured by many mobile devices in the outside world.

[0074] In block 504, the eyewear device 100 sends the image and associated metadata to the server 398. The eyewear device 100 (and other mobile devices) sends the image and metadata captured in block 502 to the server system 398, such as via the network 395 and optionally the mobile device 390, for aggregation and central storage.

[0075] refer to Figure 5B Referring to flowchart 505, in block 506, server 398 receives the image and metadata, and in block 508, server 398 stores the received image and metadata. In one example, server 398 receives and stores images and metadata from millions of mobile devices. In this way, server 398 creates a repository of images for subsequent retrieval by mobile devices.

[0076] In block 510, the server 398 receives the location coordinates of the eye-mounted device 100. In one example, the server 398 receives the GPS location coordinates from the eye-mounted device 100. The server 398 may receive the location coordinates via the network 395 and, optionally, the mobile device 390.

[0077] In block 512, the server 398 retrieves images from memory that correspond to capture locations associated with the location coordinates of the eye-mounted device 100. In one example, the server identifies a coordinate range surrounding the location coordinates of the eye-mounted device 100, which includes, for example, all global positioning system coordinates received from the mobile device within a half-mile of the current location of the eye-mounted device 100. The server 398 then identifies all images with location coordinate metadata that are within the coordinate range surrounding the eye-mounted device 100.

[0078] In block 514, the server 398 sends the images corresponding to the location where they were captured, in association with the location coordinates of the eye-mounted device 100, to the eye-mounted device 100. In one example, the server 398 organizes the images chronologically and based on distance from the eye-mounted device 100, such that recent images that are relatively close to the current location of the eye-mounted device 100 are sent before older images and / or metadata images that indicate relatively distant locations.

[0079] refer to Figure 5C Referring to flowchart 520 of FIG. 5 , in block 522 , the eye-mounted device 100 monitors its location. The eye-mounted device 100 may monitor its location using the global positioning system 331 . The location coordinates monitored by the eye-mounted device may be used to capture location coordinate metadata that is added to the image and sent to the server 398 and used to retrieve and display content as described below.

[0080] In block 524, the eye-mounted device 100 monitors its orientation relative to its field of view. The field of view of the eye-mounted device 100 is the view seen through the optical elements (assuming a see-through display). The eye-mounted device 100 monitors its orientation in three-dimensional space (e.g., two axes X and Y or three axes X, Y, and Z) and rotation about one or more axes (e.g., pitch, yaw, and / or roll). The eye-mounted device 100 can monitor its orientation using various sensors, such as a compass 332 for direction and an inertial measurement unit 333 for orientation. In the example of a tablet computer as the mobile device, the field of view is the image viewed on the screen, which is captured substantially simultaneously by the tablet computer's visible light camera.

[0081] In block 526, the eye-mounted device 100 sends its location coordinates (the location coordinates detected in block 522) to the server 398. The eye-mounted device 100 sends its location coordinates to the server system 398 via, for example, the network 395 and optionally the mobile device 390.

[0082] In block 528, the eye-mounted device 100 receives an image corresponding to its location from the server 398 (the server 398 determines the image, as described with reference to block 512). The eye-mounted device 100 receives the image from the server system 398 via, for example, the network 395 and optionally the mobile device 390.

[0083] In block 530, the eye-mounted device 100 selects an image corresponding to the position and orientation of the eye-mounted device 100 for display in the viewing area of the eye-mounted device 100. In one example, the eye-mounted device 100 selects an image corresponding to the position and orientation of the eye-mounted device 100 by determining the orientation of the mobile device (block 530a; Figure 5D), determines its viewing area (e.g., field of view; block 530b), calculates a range of position coordinates within the field of view for the determined orientation (block 530c), and selects images having position coordinates within the calculated range of position coordinates (block 530d). The orientation can be determined with reference to block 524, as described above. The field of view can then be determined based on an angular value associated with the optical assembly (e.g., a 30-degree cone around the direction in which the central axis of the optical assembly points). A coordinate range is then calculated for all coordinates within the field of view extending half a mile from the eye-mounted device 100 (e.g., all coordinates within a 30-degree cone with a distance of half a mile between the tip and base of the cone). The eye-mounted device 100 then identifies all images having position coordinate metadata within the calculated range of coordinates.

[0084] In block 532, the eye-mounted device 100 generates an overlay image, including an image icon associated with the selected image, for display on the mobile device. In one example, the eye-mounted device 100 generates the overlay image by monitoring the orientation of the mobile device (532a), monitoring the field of view of the mobile device (532b), calculating position coordinates within the field of view for the determined orientation (532c), and mapping the selected image to the calculated position coordinates (532d). Optionally, the eye-mounted device 100 may identify visual cues within the field of view (532e) to improve the mapping and display of content. For example, buildings, restaurants, concert venues, and landmarks may be identified (e.g., using object recognition technology). The eye-mounted device 100 then generates an image icon (the image icon may be a thumbnail of the received content, and the position of the image icon within the overlay image responsive to the calculated position coordinates and visual cues) (532f).

[0085] The eye-mounted device 100 can aggregate content corresponding to a specific location. In one example, all content within a location associated with a specific visual cue, such as a restaurant, can be represented by an icon thumbnail image corresponding to the latest content (such as an image or video) within that location. In another example, icons can be stacked on top of each other, with only the icon for the latest content fully visible. Furthermore, the eye-mounted device 100 can change icons based on factors such as distance from the visual cue or the amount of content, with icons appearing smaller the further away from the mobile device and larger for more content.

[0086] In block 534 , the eye-mounted device 100 presents the overlay image on the optical assembly 180 of the eye-mounted device 100 . Figure 6AAn example scene viewed through the optical assembly 180 of the eye-mounted device 100 with an overlay image is depicted. In the example shown, the scene is a pier extending from a beach into the water. The overlay image includes three icons 600. The first icon 600a represents content captured at a specific location on the beachfront. The second icon 600b represents content depicting a street performer captured at the midpoint of the pier. The third icon 600c represents content captured at the end of the pier. In the example shown, the first icon 600a is larger than the second and third icons 600b, 600c, because the content was captured at a point close to the current location of the eye-mounted device 100. In other examples, for example, if videos of 10 street performers were captured within the last hour and only one image was captured at the waterfront, the second icon 600b might be larger than the first icon 600a.

[0087] In block 536, the eye-mounted device 100 receives an image selection identifying one of the image icons in the displayed overlay image. The eye-mounted device may receive the image selection via the user interface 301. In one example, the optical assembly 180 and the eye tracker 213 provide input for the selection. Figure 6A A cursor 602 is depicted positioned on the optical assembly 180, indicating the current location of the user interaction. The eye tracker 213 tracks the eyes of the wearer of the eye-mounted device 100 and moves the cursor in response to eye movements (blocks 536a and 536b; Figure 5F ). To select content, the user can adjust their gaze to one of the icons, which will move the cursor 602 to the icon. In one example, the content is selected when the user gazes at the icon associated with the content for more than a predetermined time (e.g., 250 milliseconds; boxes 536c and 536d). In another example, the content is selected when the user gazes at the icon associated with the content and performs a specific action detected by the eye tracker 213 (e.g., blinking twice in rapid succession). For example, in the example of a mobile device such as a tablet computer with a touch screen display, the icons are displayed superimposed on the touch screen of the mobile device, and the user can select the icon by pressing the icon with their finger.

[0088] In block 538 , the eye-mounted device 100 presents the selected image associated with the identified image icon. The image processor 312 and the image display driver 342 present the image on the optical assembly 180 of the eye-mounted device 100 . Figure 6B Content 604 (eg, a video) is depicted on the optical assembly 180. In the example shown, the content 604 is aligned with the icon 600b ( Figure 6A ) and is presented in response to selection of that icon.

[0089] In block 540, the eye-mounted device 100 monitors for an image termination selection. The eye-mounted device may receive the image termination selection via the user interface 301. In one example, the eye tracker 213 provides an image termination selection in response to a specific action detected by the eye tracker 213 (e.g., blinking three times in rapid succession) or gazing in a specific direction for a prolonged period of time (e.g., looking up and to the right for more than 250 milliseconds). In another example, the user interface 301 may be a physical button or a touchpad on the eye-mounted device 100, and the user may select image termination by pressing the button or sliding a finger (e.g., downward) on the touchpad.

[0090] In block 542, the eye-mounted device 100 processes a decision based on whether an image termination selection is received. If an image termination selection is received, the process proceeds to block 544, where the image display is canceled and the optical components resume depicting the image. Figure 6A The icon shown in FIG. 5 and the steps in blocks 522 to 542 are repeated. If the image termination selection is not received, the process proceeds to block 546.

[0091] In block 546, the eye-mounted device 100 processes a decision based on whether there is additional content corresponding to the location of the selected image associated with the identified image icon. If there is no additional content, the optical assembly resumes depicting the image as shown in FIG. Figure 6A The icon shown is selected and the steps in blocks 522 - 542 are repeated. If additional content exists (eg, one or more additional images corresponding to the selected image position associated with the identified image icon), processing is performed in block 548.

[0092] In block 548, the eye-mounted device 100 receives an image-forward selection. Additional images may be stored so that they are presented in reverse chronological order (i.e., from newest to oldest). The image-forward selection cycles through the ordered images. In one example, the eye tracker 213 provides an image-forward selection in response to a specific action detected by the eye tracker 213 (e.g., blinking twice in rapid succession) or gazing in a specific direction for a prolonged period of time (e.g., down and left for more than 250 milliseconds). In another example, the user interface 301 may be a physical button or touchpad on the eye-mounted device 100, and the user may select image-forward by pressing a specific button (e.g., pressing once) or by swiping a finger on the touchpad (e.g., from back to front). In this example, the user may select image-rewind (e.g., to view previously viewed or skipped content) by pressing another specific button (e.g., double pressing) or by swiping a finger on the touchpad (e.g., from front to back).

[0093] In block 550, the eye-mounted device 100 identifies additional images responsive to the image advance selection, and in block 552, the eye-mounted device 100 presents the identified images. The image processor 312 and the image display driver 342 present the additional images on the optical assembly 180 of the eye-mounted device 100.

[0094] Any of the methods described herein, such as image capture program programming 344, image retrieval program programming 345, device position / orientation program programming 346, and program programming of a rendering engine 348 for the eye-mounted device 100, mobile device 390, and server system 398, may be embodied as method steps in one or more methods, or may be embodied in one or more applications as described above. According to some examples, an "application" or "firmware" refers to a program that executes the functions defined in a program, such as logic embodied in software or hardware instructions. Various programming languages may be used to generate one or more applications structured in various ways, such as an object-oriented programming language (such as Objective-C, Java, or C++) or a programming language (such as C or assembly language). In a specific example, a third-party application (such as an application developed by an entity other than the vendor of a specific platform using ANDROID TM or IOS TM Software Development Kit (SDK) can be used to develop applications on IOS TM ANDROID TM 、 Mobile operating systems such as iOS TM Mobile software running on or other mobile operating systems. In this example, third-party applications can invoke application programming interfaces (APIs) provided by the operating system to facilitate the functionality described herein. The applications can be stored in any type of computer-readable medium or computer storage device and executed by one or more general-purpose computers. In addition, the methods and processes disclosed herein can optionally be embodied in dedicated computer hardware or application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or complex programmable logic devices (CPLDs).

[0095] The programmatic aspects of this technology may be considered "products" or "articles of manufacture," typically carried or embodied in a machine-readable medium in the form of executable code and / or associated data. For example, program programming code may include code for navigation, eye tracking, or other functionality described herein. "Storage" media includes any or all tangible memory of a computer, processor, or other such device, such as various semiconductor memories, tape drives, disk drives, etc., that readily provides non-transitory storage for software programming. All or portions of the software may sometimes be communicated over the Internet or various other telecommunications networks. For example, such communications may involve loading software from one computer or processor to another, such as from a server system 398 or a service provider's host to the eyewear device 100 and mobile device 390. Other types of media that can carry programming, media content, or metadata files include optical, radio, and electromagnetic waves, such as those used through physical interfaces between local devices via wired and optical landline networks and various airlinks. Physical elements that carry such waves, such as wired or wireless links, optical links, etc., may also be considered media that carry software. As used herein, unless restricted solely to "non-transitory," "tangible," or "storage" media, terms such as computer or machine "readable media" refer to any medium that participates in providing instructions or data to a processor for execution.

[0096] Thus, machine-readable media can take the form of various tangible storage media. For example, non-volatile storage media include optical or magnetic disks, such as any storage device in any computer, and can be used to implement the client devices, media gateways, transcoders, and the like shown in the figure. Volatile storage media include dynamic memory, such as the main memory of a computer platform. Tangible transmission media include coaxial cables; copper wire and fiber optics, including the wires that form a bus in a computer system. Carrier transmission media can take the form of electrical or electromagnetic signals, or acoustic or light waves, such as those generated during radio frequency (RF) and infrared (IR) data communications. Thus, common forms of computer-readable media include floppy disks, diskettes, hard disks, magnetic tape, any other magnetic medium, CD-ROMs, DVDs or DVD-ROMs, any other optical medium, punch cards, paper tape, any other physical storage medium with a pattern of holes, RAM, PROMs and EPROMs, FLASH-EPROMs, any other memory chips or cartridges, carrier waves that transport data or instructions, cables or links that transport such carrier waves, or any other medium from which a computer can read programming code and / or data. Among other things, various forms of computer-readable media may be involved in carrying one or more sequences of one or more instructions to a processor for execution.

[0097] The scope of protection is defined solely by the appended claims. The scope is intended and should be construed to be as broad as when interpreted under this specification, with the ordinary meaning of the language used in the claims consistent with the following examination history being interpreted to include all structural and functional equivalents. Nothing in the claims is intended to include subject matter that fails to meet the requirements of sections 101, 102, or 103 of the Patent Act, nor should they be construed in such a manner. Any unintended coverage of such subject matter is not claimed herein.

[0098] Except as hereinbefore described, what is described and shown is not intended and should not be construed as causing any element, step, function, object, benefit, advantage or equivalent to be exclusive to the public regardless of whether it is recited in the claims.

[0099] It will be understood that the terms and expressions used herein have the ordinary meaning accorded to them in relation to their respective fields of inquiry and study, unless otherwise specified herein. Relational terms such as "first" and "second" may be used solely to distinguish one entity or action from another without necessarily requiring or implying any actual such relationship or order between such entities or actions. The terms "comprises," "comprising," "containing," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises or includes a list of elements or steps includes not only those elements or steps but also other elements or steps not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element preceded by "a" does not preclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0100] Unless otherwise indicated, any and all measurements, values, ratings, positions, magnitudes, dimensions, and the like described in this specification (including the claims that follow) are approximate and not exact. Such quantities are intended to have a reasonable range consistent with the functions to which they relate and consistent with customary practices in the art to which they pertain. For example, parameter values and the like may vary by ±10% from the stated amount unless expressly stated otherwise.

[0101] Furthermore, in the foregoing detailed description, various examples group various features together to simplify this disclosure. This method of disclosure should not be interpreted as reflecting an intention that the claimed examples require more features than are expressly recited in each claim. Rather, as the following claims reflect, protected subject matter does not lie in all the features of any single disclosed example. The following claims are therefore hereby incorporated into the detailed description, with each claim standing on its own as separately claimed subject matter.

[0102] While the foregoing describes what are considered to be the best modes and other examples, it is understood that various modifications may be made in such modes and examples, that the subject matter disclosed herein may be implemented in a variety of forms and examples, and that such modes and examples may be applied to a variety of applications, only some of which have been described herein. The following claims are intended to claim any and all modifications and variations that come within the true scope of the present concepts.

Claims

1. A system configured for use with mobile devices, at least one of the mobile devices being adapted to capture images and associated metadata, the metadata including location coordinates and timestamp information for the captured images, the system comprising: memory, including instructions for performing operations; and A server having a server processor that executes the instructions to perform the operations, the operations comprising: receiving the image and associated metadata from the at least one mobile device; storing the image and associated metadata from the at least one mobile device in an image repository for retrieval; receiving the location coordinates of the requesting mobile device; retrieving from the image repository at least one image and associated metadata captured within a coordinate range surrounding the location coordinates of the requesting mobile device; organizing the retrieved at least one image in chronological order and based on distance from a requesting mobile device such that a most recent image is sent before an older image and before metadata images indicating a location farther from the requesting mobile device than a most recent image; and The at least one retrieved image and associated metadata are sent to the requesting mobile device, the image and metadata being captured within a coordinate range surrounding the location coordinates of the requesting mobile device for display by the requesting mobile device organized according to the orientation of the requesting mobile device within the field of view of the requesting mobile device.

2. The system according to claim 1, characterized in that The requesting mobile device includes at least one optical assembly having a viewing area for viewing a scene and configured to present an overlay image to a user within the viewing area, at least one sensor configured to determine a position and orientation of the requesting mobile device; Also includes: a processor coupled to the requesting mobile device; a memory accessible to the processor; and Programming is performed in the memory, wherein execution of the program by the processor configures the requesting mobile device to perform functions, the functions comprising: monitoring the position and orientation of the mobile device using at least one sensor; requesting from the server at least one image from the image repository, the at least one image captured within a coordinate range surrounding the location coordinates of the requesting mobile device; receiving the retrieved at least one image and associated metadata from the server, the at least one image and associated metadata captured within a coordinate range surrounding the location coordinates of the requesting mobile device; selecting an image from at least one image received from the server in response to the location and orientation of the requesting mobile device; generating an overlay image from the selected at least one image, including at least one image icon associated with the selected at least one image; and An overlay image with at least one image icon is presented on an optical assembly of the requesting mobile device.

3. The system according to claim 2, characterized in that: The processor further executes the program in the memory to configure the requesting mobile device to perform functions, the functions comprising: selecting a plurality of images based on at least one image received by the server; generating an overlay image based on the selected plurality of images, including image icons associated with the selected images; presenting an overlaid image with the image icon on an optical assembly of the requesting mobile device; receiving an image selection identifying one of the image icons in the presented overlay image; and A selected image associated with the identified image icon is displayed within the viewing area via the optical assembly.

4. The system according to claim 3, characterized in that The processor executing the program to configure the requesting mobile device to receive the image selection function includes: monitoring an eye gaze direction relative to an optical component of the requesting mobile device; positioning a cursor on the optical assembly in response to the monitored gaze direction; monitoring a time during which the cursor is positioned over one of the graphical icons; and When the monitored time exceeds a time threshold, the image associated with one of the image icons is presented.

5. The system according to claim 3, wherein: The processor executing the program further configures the requesting mobile device to perform additional functions, wherein the additional functions include: associating a plurality of additional images with the selected image associated with the identified image icon based on the location; monitoring the requesting mobile device for image advance instructions; and A plurality of additional images are repeatedly cycled in reverse chronological order in response to the image advance instructions.

6. The system according to claim 3, wherein: The optical assembly includes a touch screen display and the processor executing the program programming the requesting mobile device to configure the requesting mobile device to receive the image selection includes the following functions: monitoring the touch screen display for user input selection of one of the image icons on the touch screen display; An image associated with one of the image icons is presented in response to user input selection of one of the image icons on the touch screen display.

7. The system according to claim 3, wherein: The processor executing the program to configure the requesting mobile device to select an image includes the following functions: determining a position of the mobile device; determining a field of view of the optical assembly; Calculating a position coordinate range within the determined azimuth field of view; as well as Select images that have position coordinates within the calculated position coordinate range.

8. The system according to claim 3, wherein: The processor executing the program to configure the requesting mobile device to generate an overlay image includes the following functions: monitoring the position of the mobile device; monitoring the field of view of the optical assembly; Calculating the position coordinates within the determined azimuth field of view; mapping the selected image to the calculated position coordinates; Recognize visual cues; as well as An image icon is positioned within the overlay image in response to the calculated position coordinates and the visual cue.

9. The system according to claim 3, wherein: The optical assembly further includes a camera and wherein the processor executing the program further configures the requesting mobile device to perform additional functions, wherein the additional functions include: capturing an image with the camera; obtaining a location of the requesting mobile device at the time the image was captured; associating a location at the time of capture with the image; and The image with the associated location is sent to the server.

10. A method for displaying an image, the method comprising: receiving an image and associated metadata from at least one mobile device, the metadata including location coordinates and timestamp information for the received image; storing the image and associated metadata from the at least one mobile device in an image repository for retrieval; receiving the location coordinates of the requesting mobile device; retrieving from the image repository at least one image and associated metadata captured within a coordinate range surrounding the location coordinates of the requesting mobile device; organizing the retrieved at least one image in chronological order and based on distance from a requesting mobile device, such that a most recent image is sent before an older image and before metadata images indicating a location farther from the requesting mobile device than a most recent image; and The at least one retrieved image and associated metadata are sent to the requesting mobile device, the image and metadata being captured within a coordinate range surrounding the location coordinates of the requesting mobile device for display by the requesting mobile device organized according to the orientation of the requesting mobile device within the field of view of the requesting mobile device.

11. The method according to claim 10, characterized in that The requesting mobile device includes at least one optical assembly having a viewing area for viewing a scene and configured to present an overlay image to a user within the viewing area, at least one sensor configured to determine a position and orientation of the requesting mobile device; The method further comprises: monitoring the position and orientation of the mobile device using at least one sensor; requesting from the server at least one image from the image repository, the at least one image captured within a coordinate range surrounding the location coordinates of the requesting mobile device; receiving the retrieved at least one image and associated metadata from the server, the at least one image and associated metadata captured within a coordinate range surrounding the location coordinates of the requesting mobile device; selecting an image from at least one image received from the server in response to the location and orientation of the requesting mobile device; generating an overlay image from the selected at least one image, including at least one image icon associated with the selected at least one image; and An overlay image with at least one image icon is presented on an optical assembly of the requesting mobile device.

12. The method according to claim 11, further comprising: selecting a plurality of images based on at least one image received by the server; generating an overlay image based on the selected plurality of images, including image icons associated with the selected images; presenting an overlaid image with the image icon on an optical assembly of the requesting mobile device; receiving an image selection identifying one of the image icons in the presented overlay image; as well as A selected image associated with the identified image icon is displayed within the viewing area via the optical assembly.

13. The method according to claim 12, further comprising: monitoring an eye gaze direction relative to an optical component of the requesting mobile device; positioning a cursor on the optical assembly in response to the monitored gaze direction; monitoring a time during which the cursor is positioned over one of the image icons; as well as When the monitored time exceeds a time threshold, the image associated with one of the image icons is presented.

14. The method according to claim 12, further comprising: associating a plurality of additional images with the selected image associated with the identified image icon based on the location; monitoring an image advance instruction on the requesting mobile device; as well as A plurality of additional images are repeatedly cycled in reverse chronological order in response to the image advance instructions.

15. The method according to claim 12, characterized in that The optical assembly includes a touch screen display, further comprising: monitoring the touch screen display for user input selection of one of the image icons on the touch screen display; An image associated with one of the image icons is presented in response to user input selection of one of the image icons on the touch screen display.

16. The method according to claim 12, characterized in that Further including: determining a position of the mobile device; determining a field of view of the optical assembly; Calculating a position coordinate range within the determined azimuth field of view; as well as Perform at least one of the following: (1) select an image having position coordinates within the calculated position coordinate range, or (2) map the selected image to the calculated position coordinates, identify a visual cue, and position an image icon within the overlay image in response to the calculated position coordinates and the visual cue.

17. A non-transitory computer-readable medium storing program code which, when executed, causes an electronic processor to perform the following steps: receiving an image and associated metadata from at least one mobile device, the metadata including location coordinates and timestamp information for the received image; storing the image and associated metadata from the at least one mobile device in an image repository for retrieval; receiving the location coordinates of the requesting mobile device; retrieving from the image repository at least one image and associated metadata captured within a coordinate range surrounding the location coordinates of the requesting mobile device; organizing the retrieved at least one image in chronological order and based on distance from a requesting mobile device, such that a most recent image is sent before an older image and before metadata images indicating a location farther from the requesting mobile device than a most recent image; and The at least one retrieved image and associated metadata are sent to the requesting mobile device, the image and metadata being captured within a coordinate range surrounding the location coordinates of the requesting mobile device for display by the requesting mobile device organized according to the orientation of the requesting mobile device within the field of view of the requesting mobile device.

18. The medium of claim 17, further comprising program code that, when executed, causes the electronic processor to perform the following additional steps: presenting an overlaid image to a user on a scene in a viewing area of at least one optical component of the requesting mobile device; monitoring the location and orientation of the requesting mobile device using at least one sensor; requesting from the server at least one image from the image repository, the at least one image captured within a coordinate range surrounding the location coordinates of the requesting mobile device; receiving the retrieved at least one image and associated metadata from the server, the at least one image and associated metadata captured within a coordinate range surrounding the location coordinates of the requesting mobile device; selecting an image from at least one image received from the server in response to the location and orientation of the requesting mobile device; generating an overlay image from the selected at least one image, including at least one image icon associated with the selected at least one image; as well as An overlay image with at least one image icon is presented on at least one optical component of the requesting mobile device.

19. The medium of claim 18, further comprising program code that, when executed, causes the electronic processor to perform the following additional steps: selecting a plurality of images based on at least one image received by the server; generating an overlay image based on the selected plurality of images, including image icons associated with the selected images; presenting an overlay image having an image icon on at least one optical component of the requesting mobile device; receiving an image selection identifying one of the image icons in the presented overlay image; as well as A selected image associated with the identified image icon is displayed in the viewing area via the at least one optical component.

20. The medium of claim 19, further comprising program code that, when executed, causes the electronic processor to perform the following additional steps: determining a position of the mobile device; determining a field of view of the at least one optical component; Calculating a position coordinate range within the determined azimuth field of view; and Perform at least one of the following: (1) select an image having position coordinates within the calculated position coordinate range, or (2) map the selected image to the calculated position coordinates, identify a visual cue, and position an image icon within the overlay image in response to the calculated position coordinates and the visual cue.

Images