Geospatial image surface processing and selection

Abstract

Systems, methods, and non-transitory computer-readable media for displaying images on a mobile device by monitoring a position and orientation of the mobile device, requesting previously captured images corresponding to positions captured by previously captured images relative to a server location, receiving previously captured images from a server, selecting an image from the requested previously captured images responsive to the position and orientation of the mobile device, generating an overlay image from the selected image (including image icons associated with the selected image), displaying the overlay image on an optical assembly, receiving an image selection identifying one of the image icons in the displayed overlay image, and displaying the selected image associated with the identified image icon in a viewing area of ​​the optical assembly.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Patent Application No. 16 / 821,188, filed on March 17, 2020, entitled “GEOSPATIAL IMAGERY SURFACE PROCESSING AND SELECTION,” the contents of which are incorporated herein by reference in their entirety. Background Art

[0003] Mobile devices, including cell phones and eye-wear devices such as smart glasses, headbands, and headsets, integrate image displays and cameras. Such devices can capture and display images. Many mobile devices also integrate sensors that can determine the physical location of the mobile device. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] The drawings illustrate one or more embodiments by way of example only and not by way of limitation.In the drawings, like reference numerals indicate the same or similar elements.

[0005] Figure 1A is a side view of an example hardware configuration for an eye-mounted device that includes a visible light camera located on the corner of the eyepiece and a speaker located on the temple.

[0006] Figure 1B and Figure 1C yes Figure 1A Rear view of an example hardware configuration for an eye-mounted device in , including two different types of image displays.

[0007] Figure 2A and Figure 2B yes Figure 1A Rear view of an example hardware configuration for a goggles in , including the eye tracking hardware.

[0008] Figure 2C is a diagram describing the technique of tracking eye movements.

[0009] Figure 2D yes Figure 1A A top-down cross-sectional view of the eyewear device from the corner of the eye, showing the visible light camera, head tracker, and circuit board.

[0010] Figure 3 is a high-level functional block diagram of an example image selection and display system that includes an eye-mounted device, a mobile device, and a server system connected via various networks.

[0011] Figure 4 A simplified block diagram showing an example of a mobile device's hardware configuration

[0012] Figure 5A and 5B is used to capture and distribute images for use in Figure 3 Flowchart of example steps used in the image selection and display system.

[0013] Figure 5C , 5D , 5E and 5F are used to display the overlapping of images for Figure 3 Flowchart of example steps used in the image selection and display system.

[0014] Fig. 6A A perspective view of a scene as viewed through the see-through optical assembly of the eyewear device, wherein selectable images correspond to image capture locations relative to the user's physical location.

[0015] Figure 6B is through Fig. 6A A perspective view of a scene as viewed by a perspective optical component of the device, with a selected image displayed by the perspective optical component. DETAILED DESCRIPTION

[0016] The detailed description below includes examples for viewing audio and video content captured by others on a user's mobile device (such as a mobile eye-mounted device), based on a capture location related to the location of the user's mobile device. This enables the user of the mobile device to see activities and details near them, mimicking the function of high-powered binoculars and enabling the user to see content that is blocked from the user's field of view from the current location (such as seeing performers on a stage inside a restaurant when the user is outside the restaurant). In one example, content captured by others (or previously by the user) is tagged with location coordinates (such as GPS coordinates) and stored on a server. The user's mobile device provides its current location to the server, and the server retrieves content corresponding to that location and sends it to the mobile device.

[0017] Mobile device overlays are viewing icons related to scene content on a mobile device for the user to select. When the user selects an icon, the device displays the related content.

[0018] The following detailed description sets forth numerous specific details by way of example in order to provide a thorough understanding of the relevant teachings. However, it should be apparent to those skilled in the art that the teachings can be practiced without such details. In other cases, in order to avoid unnecessarily obscuring various aspects of the teachings, generally recognized methods, processes, components, and circuit descriptions are provided at a relatively high level without detailed description.

[0019] As used herein, the term "coupled" refers to any logical, optical, physical or electrical connection, link, etc., by which an electrical signal generated or provided by one system element is transmitted to another coupled element. Unless otherwise specified, coupled elements or devices are not necessarily directly connected to each other and may be separated by intermediate components, elements or communication media that can modify, manipulate or carry electrical signals. The term "on..." refers to being directly supported by an element, or indirectly supported by the element through another element integrated into or supported by the element. As used herein, the term "about" means ±10% of the amount stated.

[0020] The orientations of the mobile device, eye-mounted device, related components, and any set of devices incorporating a camera shown in any of the figures are examples only and are used for purposes of illustration and discussion. In the operation of a particular program programming, the device can be oriented in any other direction suitable for a particular application, such as up, down, sideways, or any other orientation. In addition, for the purposes of the scope used herein, any directional terms, such as front, back, inward, outward, toward, left, right, lateral, longitudinal, up, down, high, low, top, bottom, and side, are used only as examples and do not limit the direction or orientation of any camera or camera assembly configured as described herein.

[0021] The following description will partially set forth the objects, advantages and novel features of the examples. Such other objects, advantages and novel features will be apparent to those skilled in the art after reviewing the following and accompanying drawings, or may be learned and grasped by making or operating the examples. The objects and advantages of the subject matter may be realized and obtained by the methods, means and combinations particularly pointed out in the appended claims.

[0022] Reference will now be made in detail to the examples illustrated in the accompanying drawings and discussed below.

[0023] Figure 1A An example hardware configuration of a mobile device for capturing and displaying content (e.g., images and visual content such as video and audio) in the form of an eye-mounted device 100 is described. The mobile device may take other forms, such as a mobile phone or a tablet computer. In addition, the eye-mounted device 100 may take other forms and may be combined with other types of frames, such as headgear, headphones, or helmets. The eye-mounted device 100 includes at least one visible light camera 114 located at the corner 110B of the eyeglasses for capturing images in a visible area (e.g., a field of view). The illustrated eye-mounted device 100 also includes a speaker 115 and a microphone 116.

[0024] The visible light camera 114 is sensitive to wavelengths in the visible light range. As shown in this example, the visible light camera 114 has a frontal field of view from the wearer's perspective, which is configured to capture an image of the scene being viewed through the optical assembly 180B. Examples of such visible light cameras 114 include high-resolution complementary metal oxide semiconductor (CMOS) image sensors and video graphics array (VGA) cameras, such as 640p (e.g., 640×480 pixels, a total of 0.3 megapixels), 720p, or 1080p. The eye-mounted device 100 captures image sensor data from the visible light camera 114, and optionally captures other data, such as geolocation data and audio data (via microphone 116), digitizes the data using one or more processors, and stores the digitized data in a memory. The term "field of view" is intended to describe the viewing area that a mobile device user can view with his or her eyes through the optical assembly 180, or to display information captured with the visible light camera 114 on a display of the mobile device.

[0025] The visible light camera 114 may be coupled to an image processor ( Figure 3 Image processor 312 includes circuitry (eg, a processor 314) for digitally processing and adding a timestamp and location coordinates corresponding to when and where the scene image was captured. Image processor 312 includes circuitry (eg, a processor 314) for receiving signals from visible light camera 114 and processing the signals from visible light camera 114 into a format suitable for storage in memory. Figure 3 The timestamp may be added by the image processor 312 or other processor controlling the operation of the visible light camera 114. The image processor 312 may additionally add location coordinates, such as location coordinates received from a global positioning system (GPS). Figure 3 Component 331).

[0026] Microphone 116 may be coupled to an audio processor ( Figure 3 The audio processor 313 includes circuitry that receives signals from the microphone 116 (or from memory) and processes them into a format suitable for storage in the memory 334 and / or presentation using the speaker 115. The timestamp may be added by the audio processor 313 or other processor that controls the operation of the speaker 115 and microphone 116.

[0027] like Figure 1A , 1BAs shown in Figures 1 and 1C, the eyewear device 100 includes a frame 105, wherein a left frame 107A is connected to a right frame 107B via a nose bridge 106 that fits the user's nose. The left and right frames 107A-B include respective apertures 175A-B that accommodate respective optical components 180A-B. For example, the left and right temples 125A-B extend from the sides 170A-B of the frame 105 via respective left and right corners 110A-B, respectively. Each temple 125A-B is connected to the frame 105 via a respective hinge 126A-B. The substrate or material forming the frame 105, the corners 110, and the temples 125A-B may include plastic, cellulose acetate plastic, metal, or a combination thereof. The corners 110A-B may be integrated or connected to the frame 105 and / or the temples 125A-B.

[0028] Although shown as having two optical components 180A-B, the eye-mounted device 100 may also adopt other arrangements, such as a single or three optical components, or may adopt different arrangements of the optical components 180A-B depending on the application of the eye-mounted device 100 or the specific circumstances of the intended user.

[0029] In one example, as shown in FIG. 1B , each optical assembly 180A-B includes a display matrix 171 and one or more optical layers 176A-N. The display matrix 171 may include a liquid crystal display (LCD), an organic light emitting diode (OLED) display, or other such display. The one or more optical layers 176 may include lenses, optical coatings, prisms, reflectors, waveguides, light strips, and other optical components in any combination. The term lens as used herein refers to a curved and / or flat transparent or translucent glass or plastic piece used for covering, with or without light converging / diverging effects.

[0030] The optical layer 176A-N may include a prism having a suitable size and configuration and having a first surface for receiving light from the display matrix and a second surface for emitting light to the user's eyes. The prisms of the optical layer 176A-N may extend to all or at least a portion of the respective openings 175A-B formed in the left and right frames 107A-B so that the user sees the second surface of the prism when looking through the corresponding left and right frames 107A-B with their eyes. The first surface of the prism of the optical layer 176A-N is arranged to face upward along the frame 105, and the display matrix covers the prism to ensure that the photons and light emitted from the display matrix are irradiated to the first surface. The size and shape of the prism can ensure that the light is refracted within the prism and directed to the user's eyes by the second surface of the prism of the optical layer 176A-N. In this regard, the second surface of the prism of the optical layer 176A-N can be convex to direct the light to the center of the eye. The prism size and shape may be selected to magnify the image projected by display matrix 171 and to direct light through the prism so that the image viewed from the second surface is larger than the image emitted from display matrix 171 in one or more dimensions.

[0031] In another example, if Figure 1C As described, the image display device of the optical assembly 180A-B includes a projection image display. The projection image display shown includes a laser projector 150 (e.g., a three-color laser projector using a scanning mirror or a galvanometer) and light strips 155A-N, the laser projector 150 is placed near one of the mirror corners 110A-B of the eye-mounted device 100, and the light strips 155A-N are spaced apart on the lens width of the optical assembly 180A-B or on the lens depth between the front surface and the rear surface of the lens.

[0032] When the photons projected from the laser projector 150 pass through the lenses of the optical components 180A and 180B, the photons encounter the light strips 155A-N. When a particular photon encounters a particular light strip, the photon is either redirected to the user's eyes or passed to the next light strip. The particular photons or beams can be controlled by combining the laser projector 150 modulation and the light strip modulation. In one example, the processor controls the light strips 155A-N by activating mechanical, acoustic, or electromagnetic signals.

[0033] In one example, the visible light output generated by the optical components 180A-B of the eye-mounted device 100 includes an overlay image covering at least a portion of the field of view through the optical components 180A-B. In one example, the optical components 180A-B are see-through displays that present the overlay image as an overlay on the scene (or features within the scene) viewed by the wearer through the lens of the optical components. In another example, the optical components 180A-B are not see-through displays (such as opaque displays), and the overlay image is displayed by combining the overlay layer with the real-time image captured by the camera 114 of the eye-mounted device so as to be presented to the user on the display.

[0034] Figure 2A is a rear view of an example hardware configuration of the eye-mounted device 100, which includes an eye tracker 213 on the frame 105 for tracking eye movements of a user of the eye-mounted device 100. The eye tracker 213 of the eye-mounted device 100 includes an infrared emitter 215 and an infrared camera 220. Visible light cameras typically include a blue light filter to block infrared light detection. In the example, the infrared camera 220 is a visible light camera, such as a low-resolution video graphics array (VGA) camera (e.g., 640×480 pixels, a total of 0.3 megapixels) with the blue light filter removed. The infrared emitter 215 and the infrared camera 220 are co-located on the frame 105, for example, both are shown as being connected to the upper portion of the left frame 107A. As further described below, the frame 105 or one or more left and right corners 110A-B include a circuit board for the infrared emitter 215 and the infrared camera 220. For example, the infrared emitter 215 and the infrared camera 220 can be connected to the circuit board by soldering.

[0035] Other arrangements of the infrared emitter 215 and the infrared camera 220 may be implemented, including an arrangement where both the infrared emitter 215 and the infrared camera 220 are on the right frame 107B, or at different locations on the frame 105, such as the infrared emitter 215 on the left frame 107A and the infrared camera 220 on the right frame 107B. In another example, the infrared emitter 215 is located on the frame 105 and the infrared camera 220 is located on one of the corners 110A-B, or vice versa.

[0036] Basically, the infrared emitter 215 can be connected to any position on the frame 105, the left corner 110A or the right corner 110B to emit a pattern of infrared light 250 on the user's eye 252 ( Figure 2C ). Similarly, infrared camera 220 can be connected to substantially anywhere on frame 105, left corner 110A, or right corner 110B to capture at least one reflection change 254 in the infrared light emission pattern of the user's eye.

[0037] The infrared emitter 215 and the infrared camera 220 are placed inwardly facing the user's eyes, covering part or all of the angle of the eyes, to obtain infrared images of the eyes to track the eye movements of the user's eyes. For example, the infrared emitter 215 and the infrared camera 220 are located directly in front of the eyes, on the upper part of the frame 105, or in the corners 110A-B at any end of the frame 105. The eye movement includes the change of the eye direction on the horizontal axis, the vertical axis, or a combination thereof from the initial eye direction during the presentation of the initial display image on the image display of the optical assembly 180A-B.

[0038] The eye tracker 213 can track eye movement by measuring the point of the eye's gaze direction (the position where the user is looking in the optical components 180A-B of the eye-mounted device 100), comparing the currently captured image with the previously captured calibration image, or detecting the eye situation relative to the head. For example, the eye tracker 213 uses a video image to extract the eye position to non-invasively measure eye movement. As described above, a pattern of infrared light is emitted by the infrared emitter 215, and the infrared light is reflected back from the eye, and its changes are sensed and imaged by a camera, such as the infrared camera 220. The data forming the acquired infrared image is then analyzed to extract the eye rotation from the reflection changes. This video-based eye tracker generally uses corneal reflection (first Purkinje spot diagram) and pupil center as features to be tracked over time. In a second example, a dual Purkinje spot eye tracker uses reflections from the front of the cornea (first Purkinje spot diagram) and the back of the lens (fourth Purkinje spot diagram) as features to be tracked. In a third example, image features from inside the eye, such as retinal blood vessels, are tracked and appear as the user's eye moves.

[0039] Before using the eye tracker 213 to track eye positions, calibration of the eye-mounted device 100 based on the unique anatomical features of the user's eyes may be performed. Typically, the user gazes at a point or a series of points, and the eye tracker 213 records values ​​corresponding to each gaze point. The eye-mounted device 100 calibrates the eye tracker 213 by presenting a series of calibration images via the image display of the optical components 180A-B for viewing by the user's eyes prior to presentation of an initial display image via the image display of the optical components 180A-B. Each calibration image has a respective target point at a respective known fixed position on the horizontal axis and the vertical axis. In response to the user's eyes viewing the respective target points, the eye-mounted device 100 records the position of the eye's anatomical features associated with the respective known fixed points of the respective target points in the eye direction (e.g., scan path) database.

[0040] After calibration, the video-based eye tracker 213 can focus on one or both eyes of a user and record the eye movements of a user (e.g., a wearer of the eye-mounted device 100) as they gaze at the image display of the optical assembly 180A-B. When infrared or near-infrared non-collimated light is shone on the pupil of the eye as pupil-visible infrared light through the infrared emitter 215, a corneal reflection is generated in the reflection variation of the infrared light. The vector between the center of the pupil and the corneal reflection in the captured infrared image contains the reflection variation of the infrared light and can be used to calculate the gaze point on the surface or the direction of the eye's gaze.

[0041] There are two common infrared and near infrared (also called active light) eye tracking techniques that can be used: light pupil and dark pupil. Whether to use light pupil or dark pupil depends on the location of the illumination source (infrared emitter 215) relative to infrared camera 220 and the user's eye. If the illumination from infrared emitter 215 is on-axis with the optical path, when the light reflects off the retina, the eye acts as a reflector, creating a bright pupil effect similar to red eye. If the illumination from infrared emitter 215 is off-path, the pupil appears dark because the return reflection from the retina is directed away from infrared camera 220.

[0042] In one example, the infrared emitter 215 of the eye tracker 213 emits infrared illumination, which may be near infrared light or other short wavelength beams of low energy radiation, at the user's eye. Alternatively or additionally, the eye tracker 213 may include an emitter that emits other wavelengths of light other than infrared light, and the eye tracker 213 also includes a camera sensitive to the wavelength for receiving and capturing images having the wavelength. For example, the eye tracker 213 may include a visible light camera, such as a red-green-blue (RGB) camera, that captures light from the eye in the visible light range.

[0043] As described above, the eye-mounted device 100 is coupled to a processor and a memory, such as in the eye-mounted device 100 itself or in another part of the system. The eye-mounted device 100 or the system may then process the captured eye images, for example, the memory and processor coupled in the system process the captured eye images to track eye movements. This processing of the captured images establishes a scanning path to identify the movement of the user's eyes. The scanning path includes a sequence or series of eye movements based on the captured changes in eye reflections. Eye movements are generally divided into such fixations and saccades - when the eye gaze stays at a certain location, and when the eye moves to another location, respectively. The resulting series of fixations and saccades is called a scanning path. Smooth pursuit describes the eyes following a moving object. Fixed eye movements include micro-saccades: small, involuntary saccades that occur during attempted fixations. The scanning path is then used to determine the field of view adjustment.

[0044] An eye direction database may be built during calibration. Since the known fixed positions of the respective target points are known during calibration, the scan path database may be used to build similarities with previous calibration images. Because the known fixed positions of the target points are known when the calibration images are available and recorded in the eye direction database, the eye-mounted device 100 may determine where the user's eyes are looking by comparing the currently captured eye image to the eye direction database. The calibration image that is most similar to the currently captured image may have the known fixed positions of the target points that serve as a good approximation of the eye direction for the currently captured image.

[0045] Figure 2B 2 is a rear view of another example hardware configuration of the eye-mounted device 200. In this example configuration, the eye-mounted device 200 is depicted as having an eye tracker 213 on the right corner 210B for tracking the eye movements of the user of the eye-mounted device. As shown, an infrared emitter 215 and an infrared camera 220 are co-located on the right corner 210B. The eye tracker 213 or one or more components of the eye tracker 213 may alternatively or additionally be located at the left corner 210A and other locations of the eye-mounted device 200, such as the frame 105. The eye tracker 213 has an infrared emitter 215 and an infrared camera 220 and Figure 2A , but the eye tracker 213 may be modified to be sensitive to different wavelengths of light, as previously described in Figure 2A As described in.

[0046] Figure 2D yes Figure 1A A top cross-sectional view of the corner of the eye-mounted device 100 is shown, showing the right visible light camera 114, the head motion tracker 109 and the microphone 116. The configuration and mounting position of the left visible light camera are substantially similar to the right visible light camera 114, except that it is connected and coupled to the left side 170A. As shown, the eye-mounted device 100 includes a circuit board, which can be a flexible printed circuit board (PCB) 240. The right hinge 126B connects the right corner 110B to the right temple 125B of the eye-mounted device 100. In some examples, the components of the right visible light camera 114, the flexible PCB 240 or other electrical connectors or contacts can be located on the right temple 125B or the right hinge 126B.

[0047] For example, the head motion tracker 109 includes an inertial measurement unit (IMU). An inertial measurement unit is an electronic device that uses a combination of accelerometers and gyroscopes, and sometimes a magnetometer, to measure and report the specific force, angular velocity, and sometimes the magnetic field surrounding the body. The working principle of an inertial measurement unit is to detect linear acceleration using one or more accelerometers and rotation rate using one or more gyroscopes. A typical configuration of an inertial measurement unit includes an accelerometer, a gyroscope, and a magnetometer on each of three axes: a horizontal axis (X) for left-right movement, a vertical axis (Y) for top-bottom movement, and a depth axis or distance axis (Z) for up and down movement. The gyroscope detects the gravity vector. The magnetometer defines rotation in the magnetic field (such as facing south, north, etc.), and acts like a compass that generates a heading reference value. The three accelerometers are used to detect acceleration along the horizontal, vertical, and depth axes defined above, which can be defined relative to the ground, the eye-mounted device 100, or the user wearing the eye-mounted device 100.

[0048] The eye-mounted device 100 detects movement of a user of the eye-mounted device 100 by tracking the user's head movement via the head tracker 109. The head movement includes a change in the head direction along a horizontal axis, a vertical axis, or a combination thereof relative to an initial head direction during presentation of an initial display image on the image display. In an example, tracking the user's head movement via the head tracker 109 includes measuring the initial head direction along a horizontal axis (such as an X-axis), a vertical axis (such as a Y-axis), or a combination thereof (such as lateral or diagonal movement) via the inertial measurement unit 109. Tracking the user's head movement via the head tracker 109 also includes measuring continuous head directions along the horizontal axis, the vertical axis, or a combination thereof via the inertial measurement unit 109 during presentation of the initial display image.

[0049] The right mirror corner 110B includes a mirror corner body 110B and a mirror corner cap. Figure 2D The corner cap is omitted from the cross-section shown. Arranged within the right corner 110B are various interconnect circuit boards, such as a printed circuit board or a flexible printed circuit board, which include controller circuitry for the right visible light camera 114, a microphone, low-power wireless circuitry (such as for communicating via Bluetooth), and a plurality of other circuits. TM wireless short-range network communications), high-speed wireless circuits (such as for wireless LAN communications via WiFi).

[0050] Flexible printed circuit board 240 is disposed within right corner 110B and is coupled to one or more other components within right corner 110B. Although shown as being formed on a circuit board of right corner 110B, right visible light camera 114B may also be formed on a circuit board of left corner 110A, temples 125A-B, or frame 105.

[0051] Figure 3is a high-level functional block diagram of an example image selection and display system 300. The image selection and display system 300 includes a mobile device, which in this example is the eye-mounted device 100. The mobile device can communicate with other mobile devices 390 or server systems 398 via one or more wireless networks or wireless links. The image selection and display system 300 also includes other mobile devices 390 and server systems 398. The mobile device 390 can be a smartphone, tablet computer, laptop computer, access point, or other such device capable of connecting to the eye-mounted device 100 using, for example, a low-power wireless connection 325 and a high-speed wireless connection 337. The mobile device 390 is connected to the server system 398 via a network 395. The network 395 can include any combination of wired and wireless connections.

[0052] The eye-mounted device 100 includes and supports a visible light camera 114 , a speaker 115 , a microphone 116 , a user interface 301 , an image display of an optical assembly 180 , an image display driver 342 , an image processor 312 , an audio processor 313 , a low power circuit 320 , and a high speed circuit 330 . Figure 3 The components for the eye-mounted device 100 shown in the figure are located on one or more circuit boards in the temples, such as printed circuit boards or flexible printed circuit boards. In addition, or in addition, the components shown may be located in the corners, frames, hinges, or bridges of the eye-mounted device 100. The memory 334 includes image capture program programming 344, image retrieval program programming 345, and device position / orientation program programming 346 to perform the functions described herein for image selection and display. The memory 334 further includes a rendering engine 348 that uses the image processor 312 and the image display driver 342 to render overlapping images on the displays 180A and 180B.

[0053] The image capture program 344 executes instructions to enable the eye-mounted device 100 to capture scene images and add time stamps and location coordinates through the visible light camera 114. The image retrieval program 345 executes instructions to enable the eye-mounted device 100 to request images from the server system 398 or the memory 334 based on the location where the image is captured, that is, relative to the current location of the eye-mounted device 100. The device position / orientation program 346 executes instructions to enable the eye-mounted device 100 to determine the current location of the eye-mounted device 100 and determine the orientation of the eye-mounted device (e.g., determine the field of view through the optical component 180).

[0054] like Figure 3As shown, the high-speed circuit 330 includes a high-speed processor 343, a memory 334, and a high-speed wireless circuit 336. In this example, the image display driver 342 is operated by the high-speed processor 343 to drive the image display of the optical assembly 180. The high-speed processor 343 can be any processor capable of implementing any general-purpose computing system high-speed communication and operation required by the eye-mounted device 100. The high-speed processor 343 includes the processing resources required to manage high-speed data transmission to a wireless local area network (WLAN) over a high-speed wireless connection 337 using the high-speed wireless circuit 336. In some examples, the high-speed processor 343 executes an operating system of the eye-mounted device 100, such as a LINUX operating system or other such operating system, and the operating system is stored in the memory 334 for execution. In addition to any other tasks, the high-speed processor 343, which executes the software architecture of the eye-mounted device 100, is used to manage data transmission using the high-speed wireless circuit 336. In some examples, the high-speed wireless circuit 336 is configured to implement the Institute of Electrical and Electronics Engineers (IEEE) 802.11 communication standard, also referred to herein as Wi-Fi. In other examples, high-speed wireless circuitry 336 implements other high-speed communication standards.

[0055] The low power wireless circuit 324 and the high speed wireless circuit 336 of the eye-mounted device 100 may include a short range transceiver (Bluetooth TM ) and a wireless wide area, local area or wide area network transceiver (such as a cellular network or WiFi). Mobile device 390 includes a transceiver that communicates via a low power wireless connection 325 and a high speed wireless connection 337, and can be implemented using details of the architecture of eye-mounted device 100, as can other elements of network 395.

[0056] The memory 334 includes a storage device capable of storing various data and applications, including, among other things, camera data generated by the visible light camera 114 and the image processor 312, as well as images generated by the image display driver 342 for display on the image display of the optical assembly 180, and audio data generated by the microphone 116 and the audio processor 313. Although the memory 334 is shown as being integrated with the high-speed circuit 330, in other examples, the memory 334 may be a separate element of the eye-mounted device 100. In some examples, the circuitry may provide a connection from the image processor 312 / audio processor 313 or the low-power processor 324 to the memory 334 through a chip including the high-speed processor 343. In other examples, the high-speed processor 343 may manage addressing of the memory 334 so that the low-power processor 324 will direct the high-speed processor 343 whenever a read or write operation involving the memory 334 is required.

[0057] The eye-mounted device 100 also includes a global positioning system 331, a compass 332, and an inertial measurement unit 333. The global positioning system 331 is a receiver for a satellite-based radio navigation system that receives geographic location and time information from global positioning system satellites. The compass 332 provides directions relative to geographic cardinal directions (or points). The inertial measurement unit 333 is an electronic device that uses a combination of accelerometers, gyroscopes, and / or magnetometers to measure and report forces, angular rates, and / or orientations.

[0058] The eye-mounted device 100 can be connected to a host. For example, the eye-mounted device 100 can be paired with a mobile device 390 via a high-speed wireless connection 337, or connected to a server system 398 via a network 395. In one example, the eye-mounted device 100 captures scene images through the camera 114 and sends the images (along with location and timestamp information) to the host for forwarding to the server system 398. In another example, the eye-mounted device 100 receives images and / or instructions from the host.

[0059] The eye-mounted device 100 also includes other output components and input components. Other output components include acoustic components (such as speakers 115), tactile components (such as vibration motors), and other signal generators. The input components of the eye-mounted device 100, the mobile device 390, and the server system 398 may include alphanumeric input components (such as keyboards, touch screens configured to receive alphanumeric input, optical keyboards, or other alphanumeric input components), point-based input components (such as mice, touch pads, trackballs, joysticks, motion sensors, or other pointing instruments), tactile input components (such as physical buttons, touch screens or other tactile input components that provide touch position and force or touch gestures), audio input components (such as microphones), and the like.

[0060] The image selection and display system 300 may optionally include additional peripheral device elements 319. Such peripheral device elements 319 may include biometric sensors, additional sensors, or display elements integrated with the eye-mounted device 100. For example, the peripheral device elements 319 may include any input / output components, including output components, motion components, position components, or any other such components described herein.

[0061] For example, the biometric components of the image selection and display system 300 include components for detecting expressions (such as hand expressions, facial expressions, voice expressions, body postures, or eye tracking), measuring biological signals (such as blood pressure, heart rate, body temperature, sweating, or brain waves), identifying people (such as voice recognition, retinal recognition, facial recognition, fingerprint recognition, or electroencephalogram-based recognition), etc. The motion components include acceleration sensor components (such as accelerometers), gravity sensor components, rotation sensor components (such as gyroscopes), etc. The position components include position sensor components for generating position coordinates (such as global positioning system (GPS) receiver components), WiFi or Bluetooth components for generating positioning system coordinates, etc. TM transceiver, altitude sensor components (such as an altimeter or barometer that detects air pressure from which altitude can be derived), direction sensor components (such as a magnetometer), etc. Such positioning system coordinates can also be received from mobile device 390 via low power wireless circuit 324 or high speed wireless circuit 336 via wireless connections 325 and 337.

[0062] In one example, the image processor 312 includes a microprocessor integrated circuit (IC) customized to process image sensor data from the visible light camera 114, and volatile memory used by the microprocessor to operate. To reduce the time it takes for the image processor 312 to process data when powered on, a non-volatile read-only memory (ROM) with instructions for operating or booting the image processor 312 may be integrated on the integrated circuit. The read-only memory may be minimized to match the minimum size required to provide the basic functions of collecting sensor data from the visible light camera 114, so that there are no additional functions that would cause startup boot time delays. The read-only memory may be configured with direct memory access (DMA) to the volatile memory of the microprocessor of the image processor 312. Direct memory access allows data to be transferred from the read-only memory to the memory of the image processor 312 independently of the operation of the main controller of the image processor 312. Providing direct memory access to the boot read-only memory further reduces the amount of time from the image processor 312 being powered on until the sensor data from the visible light camera 114 can be processed and stored. In some examples, camera signals from visible light camera 114 are minimally processed by image processor 312 , and additional processing may be performed by an application running on mobile device 390 or server system 398 .

[0063] The low power circuit 320 includes a low power processor 322 and a low power radio circuit 324. These elements of the low power circuit 320 may be implemented as separate elements or may be implemented on a single integrated circuit as part of a system on a single chip. The low power processor 324 includes logic for managing other elements of the eye-mounted device 100. The low power processor 324 is configured to receive input signals or command communications from the mobile device 390 via a low power wireless connection 325. Additional details related to such instructions are further described below. The low power radio circuit 324 includes circuit elements that implement a low power wireless communication system over a short range network. Bluetooth TM Smart, also known as Bluetooth Low Energy TM , which is a standard implementation of a low power wireless communication system, may be used to implement low power wireless circuit 324. In other examples, other low power communication systems may be used.

[0064] Elements of mobile device 390 and network 395, low power wireless connection 325, and high speed wireless architecture 337 may be implemented using architectural details of mobile device 390, such as using Figure 4 The short-range transceiver and wireless wide area network transceiver of the mobile device 390 described in.

[0065] Figure 4 Is for Figure 3 300 provides a high-level functional block diagram of an example of a mobile device 390 for processing. Shown are elements of a touch screen type mobile device 390 loaded with image capture program programming 344, image retrieval program programming 345, and device location / orientation program programming 346, as well as other applications, such as a chat application. Examples of touch screen type mobile devices that may be used include (but are not limited to) smart phones, personal digital assistants (PDAs), tablet computers, laptop computers, or other portable devices. However, the structure and operation of touch screen type devices are provided by way of example, and the subject technology described herein is not limited thereto. Therefore, for purposes of discussion, Figure 4 A block diagram of an exemplary mobile device 390 is provided, which has a touch screen display for displaying content and receiving user input as (or part of) a user interface. The mobile device 390 also includes a camera 470, such as a visible light camera, and a microphone 471.

[0066] like Figure 4As shown, the mobile device 390 includes at least one digital transceiver (XCVR) 410, shown as a wireless wide area network transceiver, namely a WWAN XCVR, for digital wireless communication via a wide area wireless mobile communication network. The mobile device 390 also includes additional digital or analog transceivers, such as a short-range transceiver 420 for short-range network communication, for example, by means of NFC, VLC, DECT, ZigBee, Bluetooth TM Taking the short-range transceiver 420 as an example, it can take the form of any available two-way wireless local area network (WLAN) transceiver, the type of which is compatible with one or more standard communication protocols implemented in a wireless local area network, such as a Wi-Fi standard according to IEEE 802.11 and WiMAX.

[0067] To generate location coordinates for locating the mobile device 390, the mobile device 390 may include a global positioning system (GPS) receiver 331. Alternatively or additionally, the mobile device 390 may utilize one or both of the short-range transceiver 420 and the wireless wide area network transceiver 410 to generate location coordinates for positioning. For example, a positioning system based on GPS, a cellular network, WiFi, or Bluetooth TM Very accurate position coordinates can be generated, especially when used in combination. Such position coordinates can be transmitted to the eye-mounted device 100 via the transceiver 420 through one or more network connections. In addition, the mobile device 390 may include a compass 332 and an inertial measurement unit 333 for determining direction information.

[0068] Transceivers 410, 420 (network communication interface) conform to one or more digital wireless communication standards used by modern mobile networks. Examples of wireless wide area network transceivers 410 include (but are not limited to) transceivers configured to operate according to code division multiple access (CDMA) and third generation partnership project (3GPP) network technologies, such as but not limited to third generation partnership project 2 (or 3GPP2) and LTE, sometimes referred to as "4G". For example, transceivers 410, 420 provide two-way wireless information communication, including digitized audio signals, still images and video signals, web page information for display and input related to the web page, and various types of mobile message communication between mobile devices 390 for user authorization policies.

[0069] The mobile device 390 also includes a microprocessor, such as the CPU 430 shown. A processor is a circuit whose components are constructed and arranged to perform one or more processing functions, typically various data processing functions. Although discrete logic components can be used, components that constitute a programmable CPU are used in the example. For example, a microprocessor includes one or more integrated circuit (IC) chips that integrate electronic components that perform CPU functions. Taking processor 430 as an example, it can be based on any known or available microprocessor architecture, such as a reduced instruction set computing (RISC) using an ARM architecture, which is very commonly used in today's mobile devices and other portable electronic devices. Other processor circuits can be used to form CPU 430 or processor hardware in smartphones, laptops, and tablets.

[0070] The microprocessor 430 acts as a programmable host controller for the mobile device 390 by configuring the mobile device 390 to perform various operations, such as according to instructions or program programming executable by the processor 430. For example, such operations may include various general operations of the mobile device, as well as operations related to determining the device position when capturing images and determining the device position and orientation when generating and presenting image overlays. Although the processor can be configured by employing hard-wired logic, a typical processor in a mobile device is a general processing circuit that is configured by executing program programming.

[0071] The mobile device 390 includes a memory or storage device system for storing data and program programming. In this example, the memory system may include flash memory 440A and random access memory (RAM) 440B. Random access memory 440B acts as short-term storage for instructions and data processed by processor 430, such as working data processing memory. Flash memory 440A generally provides longer-term storage.

[0072] Depending on the device type, the mobile device 390 stores and runs a mobile operating system through which specific applications are executed, which may include image capture programming 344, image retrieval programming 345, device location / orientation programming 346, and rendering engine 348. However, in some implementations, programming may be implemented in firmware or a combination of firmware and application layers. For example, instructions for capturing scene images, tracking device location and orientation information, and generating overlays may reside in firmware (such as a SOC with a dedicated GPU or VPU). Instructions for generating visible output to the user may reside in an application. Applications, such as audio visualization programming 344 and other applications, may be native applications, hybrid applications, or web applications (such as dynamic web pages executed by a web browser) running on the mobile device 390. Examples of mobile operating systems include Google Android, Apple iOS (I-Phone or iPad devices), Windows Mobile, Amazon Fire OS, RIM Blackberry operating system, etc.

[0073] Figure 5A , 5B 5C, 5D, 5E, and 5F are flowcharts 500, 505, 520, 530, 532, and 536, respectively, illustrating example operations of a mobile device (e.g., eye-mounted device 100) or mobile device 390 and other components of image selection and display system 300. Although shown as appearing sequentially, one or more blocks in flowchart 500, flowchart 505, flowchart 520, flowchart 530, flowchart 532, and / or flowchart 536 may be reordered or parallelized depending on implementation.

[0074] The following flowchart is described with reference to an example in which the mobile device is an eye-mounted device 100 that can capture and display images. It is understood that the functions described with reference to the eye-mounted device 100 can be performed by other eye-mounted devices and other mobile devices, such as mobile phones and tablets. Through the description herein, it can be easily understood that appropriate modifications to implement the following content on other mobile devices (including devices with a see-through display and devices with a non-see-through display, such as a touch screen).

[0075] refer to Figure 5AIn flowchart 500, in block 502, the eye-mounted device 100 captures content (e.g., images, audio, and related metadata). The eye-mounted device 100 captures images, such as using the visible light camera 114, and stores the images in the memory 334. In addition, the eye-mounted device 100 captures location coordinates and timestamp information, such as using the global positioning system 331 and the image processor 312, and stores the location coordinates and timestamp information in the memory 334. The eye-mounted device 100 stores the location coordinates and timestamp information as metadata associated with each image it captures. Images, audio, and metadata are captured by many mobile devices in the outside world.

[0076] In block 504, the eye-mounted device 100 sends the image and associated metadata to the server 398. The eye-mounted device 100 (and other mobile devices) sends the image and metadata captured in block 502 to the server system 398, such as via the network 395 and optionally the mobile device 390, for aggregation and central storage.

[0077] refer to Figure 5B 505, in block 506, the server 398 receives the image and metadata, and in block 508, the server 398 stores the received image and metadata. In one example, the server 398 receives and stores images and metadata from millions of mobile devices. In this way, the server 398 generates a repository of images for subsequent retrieval by mobile devices.

[0078] In block 510, the server 398 receives the location coordinates of the eye-mounted device 100. In one example, the server 398 receives the global positioning system location coordinates from the eye-mounted device 100. The server 398 may receive the location coordinates via the network 395 and optionally the mobile device 390.

[0079] In block 512, the server 398 retrieves images from the memory that correspond to the capture locations associated with the location coordinates of the eye-mounted device 100. In one example, the server identifies a coordinate range surrounding the location coordinates of the eye-mounted device 100, which includes, for example, all global positioning system coordinates received from the mobile device that represent a half-mile global positioning system location coordinate of the current location of the eye-mounted device 100. The server 398 then identifies all images that have location coordinate metadata that are within the coordinate range surrounding the eye-mounted device 100.

[0080] In block 514, the server 398 sends the eye-mounted device 100 images corresponding to the location where they were captured in association with the location coordinates of the eye-mounted device 100. In one example, the server 398 organizes the images in chronological order and based on distance from the eye-mounted device 100 such that recent images that are relatively close to the current location of the eye-mounted device 100 are sent before older images and / or metadata images that indicate relatively distant locations.

[0081] refer to Figure 5C Referring to flowchart 520 of FIG. 5 , in block 522, the eye-mounted device 100 monitors its location. The eye-mounted device 100 can monitor its location using the global positioning system 331. The location coordinates monitored by the eye-mounted device can be used to capture location coordinate metadata added to the image that is sent to the server 398 and used to retrieve and display content as described below.

[0082] In block 524, the eye-mounted device 100 monitors its orientation relative to the field of view. The field of view of the eye-mounted device 100 is the view seen through the optical elements (assuming a see-through display). The eye-mounted device 100 monitors its orientation in three-dimensional space (e.g., two axes X and Y or three axes X, Y, and Z) and rotation about one or more axes (e.g., pitch, yaw, and / or roll). The eye-mounted device 100 can use various sensors to monitor its orientation, such as using a compass 332 to determine direction and an inertial measurement unit 333 to determine orientation. In an example where a tablet is the mobile device, the field of view is the image viewed on the screen, which is captured by the visible light camera of the tablet at substantially the same time.

[0083] In block 526, the eye-mounted device 100 sends its position coordinates (the position coordinates detected in block 522) to the server 398. The eye-mounted device 100 sends its position coordinates to the server system 398 via, for example, the network 395 and the optional mobile device 390.

[0084] In block 528, the eye-mounted device 100 receives an image corresponding to its location from the server 398 (the server 398 determines the image, as described with reference to block 512). The eye-mounted device 100 receives the image from the server system 398 via, for example, the network 395 and optionally the mobile device 390.

[0085] In block 530, the eye-mounted device 100 selects an image corresponding to the position and orientation of the eye-mounted device 100 for display in the viewing area of ​​the eye-mounted device 100. In one example, the eye-mounted device 100 determines the orientation of the mobile device (block 530a; Figure 5D), determine its viewing area (e.g., field of view; box 530b), calculate a position coordinate range within the field of view for the determined orientation (box 530c), and select images having position coordinates within the calculated position coordinate range (box 530d). The orientation can be determined with reference to box 524, as described above. The field of view can then be determined based on an angular value associated with the optical component (e.g., a 30-degree cone around the direction in which the central axis of the optical component points). Then, a coordinate range is calculated for all coordinates within the field of view extending half a mile from the eye-mounted device 100 (e.g., all coordinates within a 30-degree cone with a half-mile distance between the tip of the cone and the bottom of the cone). The eye-mounted device 100 then identifies all images having position coordinate metadata within the calculated coordinate range.

[0086] In block 532, the eye-mounted device 100 generates an overlay image, including an image icon associated with the selected image, for display by the mobile device. In one example, the eye-mounted device 100 generates the overlay image by monitoring the orientation of the mobile device (532a), monitoring the field of view of the mobile device (532b), calculating position coordinates within the field of view for the determined orientation (532c), and mapping the selected image to the calculated position coordinates (532d). Optionally, the eye-mounted device 100 may identify visual cues within the field of view (532e) to improve the mapping and display of content. For example, buildings, restaurants, concert venues, and landmarks may be identified (e.g., using object recognition technology). The eye-mounted device 100 then generates an image icon (the image icon may be a thumbnail of the received content, and the position of the image icon within the overlay image responsive to the calculated position coordinates and the visual cues) (532f).

[0087] The eye-mounted device 100 can aggregate content corresponding to a specific location. In one example, all content within a location range associated with a specific visual cue, such as a restaurant, can be represented by an icon thumbnail image corresponding to the latest content (such as an image or video) within the location range. In another example, an icon can be stacked on top of another icon, with only the icon for the latest content fully visible. In addition, the eye-mounted device 100 can change the icon based on factors such as the distance to the visual cue or the amount of content, where the farther away from the mobile device, the smaller the icon size, and the larger the number, the larger the size.

[0088] In block 534 , the eye-mounted device 100 presents the overlay image on the optical assembly 180 of the eye-mounted device 100 . Fig. 6AAn example scene observed through the optical assembly 180 of the eye-mounted device 100 with an overlay image is depicted. In the example shown, the scene is a pier extending from a beach into the water. The overlay image includes three icons 600. The first icon 600a represents content captured at a specific location on the waterfront beach. The second icon 600b represents content depicting a street artist captured at the midpoint of the pier. The third icon 600c represents content captured at the end of the pier. In the example shown, the first icon 600a is larger than the second icon 600b and the third icon 600c because the content was captured at a point close to the current location of the eye-mounted device 100. In other examples, for example, if videos of 10 street artists were captured in the last hour and only one image was captured at the waterfront, the second icon 600b may be larger than the first icon 600a.

[0089] In block 536, the eye-mounted device 100 receives an image selection identifying one of the image icons in the displayed overlay image. The eye-mounted device may receive the image selection via the user interface 301. In one example, the optical assembly 180 and the eye tracker 213 provide input selections. Fig. 6A A cursor 602 is depicted positioned on the optical assembly 180, indicating the current location of the user interaction. The eye tracker 213 tracks the eyes of the wearer of the eye-mounted device 100 and moves the cursor in response to eye movements (blocks 536a and 536b; Fig. 5F ). To select content, the user may adjust their gaze to one of the icons, which moves the cursor 602 to the icon. In one example, the content is selected when the user gazes at an icon associated with the content for more than a predetermined time (e.g., 250 milliseconds; boxes 536c and 536d). In another example, the content is selected when the user gazes at an icon associated with the content and performs a specific action detected by the eye tracker 213 (e.g., blinking twice in rapid succession). For example, in the example of a mobile device such as a tablet computer with a touch screen display, the icons are displayed superimposed on the touch screen of the mobile device, and the user can select the icon by pressing the icon with his finger.

[0090] In block 538 , the eye-mounted device 100 presents the selected image associated with the identified image icon. The image processor 312 and the image display driver 342 present the image on the optical assembly 180 of the eye-mounted device 100 . Figure 6B Content 604 (eg, video) is depicted on the optical assembly 180. In the example shown, the content 604 is associated with the icon 600b ( Fig. 6A ) and is presented in response to selection of that icon.

[0091] In block 540, the eye-mounted device 100 monitors for an image termination selection. The eye-mounted device may receive the image termination selection via the user interface 301. In one example, the eye tracker 213 provides an image termination selection in response to a specific action detected by the eye tracker 213 (e.g., blinking three times in rapid succession) or gazing in a specific direction for a long time (e.g., looking up and to the right for more than 250 milliseconds). In another example, the user interface 301 may be a physical button or a touch pad on the eye-mounted device 100, and the user may select the image termination by pressing the button or sliding a finger (e.g., downward) on the touch pad.

[0092] In block 542, the eye-mounted device 100 processes a decision based on whether an image termination selection is received. If an image termination selection is received, the process proceeds to block 544, the image display is canceled, and the optical components resume depicting the image. Fig. 6A The icon shown in FIG. 5 and the steps in blocks 522-542 are repeated. If the image termination selection is not received, the process proceeds to block 546.

[0093] In block 546, the eye-mounted device 100 processes a decision based on whether there is additional content corresponding to the location of the selected image associated with the identified image icon. If there is no additional content, the optical assembly resumes depicting the image as shown in FIG. Fig. 6A The icon shown is selected and the steps in blocks 522-542 are repeated. If additional content exists (eg, one or more additional images corresponding to the selected image position associated with the identified image icon), processing is performed in block 548.

[0094] In box 548, the eye-mounted device 100 receives an image advance selection. The additional images may be stored so that they are presented in reverse chronological order (i.e., from newest to oldest). The image advance selection cycles through the ordered images. In one example, the eye tracker 213 provides an image advance selection in response to a specific action detected by the eye tracker 213 (e.g., blinking twice in rapid succession) or gazing in a specific direction for a long time (e.g., down and left for more than 250 milliseconds). In another example, the user interface 301 may be a physical button or touch pad on the eye-mounted device 100, and the user may select image advance by pressing a specific button (e.g., pressing once) or by a sliding gesture with a finger on the touch pad (e.g., from back to front). According to this example, the user may select image rewind (e.g., to view previously viewed or skipped content) by another specific button press (e.g., double press) or another sliding gesture with a finger on the touch pad (e.g., from front to back).

[0095] In block 550, the eye-mounted device 100 identifies an additional image responsive to the image advance selection, and in block 552, the eye-mounted device 100 presents the identified image. The image processor 312 and the image display driver 342 present the additional image on the optical assembly 180 of the eye-mounted device 100.

[0096] Any of the methods described herein, such as the image capture program programming 344, the image retrieval program programming 345, the device position / orientation program programming 346, and the program programming of the rendering engine 348 for the eye-mounted device 100, the mobile device 390, and the server system 398, may be embodied in one or more methods as method steps, or may be embodied in one or more applications as described above. According to some examples, an "application" or "firmware" refers to a program that executes the functions defined in a program, such as logic embodied in software or hardware instructions. Various programming languages ​​may be used to generate one or more applications structured in various ways, such as an object-oriented programming language (such as Objective-C, Java, or C++) or a programming language (such as C or assembly language). In a specific example, a third-party application (such as an application created by an entity other than the vendor of a specific platform using ANDROID TM or IOS TM Software Development Kit (SDK) applications can be developed on IOS TM ANDROID TM , Phone and other mobile operating systems, such as IOS TM The mobile software running on or other mobile operating system. In this example, the third-party application can call the application programming interface (API) call provided by the operating system to facilitate the functions described herein. The application program can be stored in any type of computer-readable medium or computer storage device and executed by one or more general-purpose computers. In addition, the methods and processes disclosed herein can be optionally embodied in dedicated computer hardware or application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or complex programmable logic devices (CPLDs).

[0097] The program aspects of the technology may be considered "products" or "articles of manufacture", which are usually carried or embodied in a machine-readable medium in the form of executable code and / or related data. For example, the program programming code may include code for navigation, eye tracking, or other functions described herein. "Storage" type media include any or all tangible memories of computers, processors, etc., or their related modules, such as various semiconductor memories, tape drives, disk drives, etc., which can provide non-transitory storage for software program programming at any time. All or part of the software can sometimes be communicated via the Internet or various other telecommunications networks. For example, such communications can load software from one computer or processor to another computer or processor, such as from a server system 398 or a service provider's host to the eye-mounted device 100 and the mobile device 390. Another type of media that can carry program programming, media content, or metadata files includes optical waves, radio waves, and electromagnetic waves, such as optical waves, radio waves, and electromagnetic waves used through physical interfaces between local devices via wired and optical landline networks and various air links. Physical elements that carry such waves, such as wired or wireless links, optical links, etc., can also be considered as media that carry software. As used herein, unless restricted to "non-transitory," "tangible," or "storage" media, terms such as computer or machine "readable media" refer to any medium that participates in providing instructions or data to a processor for execution.

[0098] Therefore, machine-readable media can take the form of tangible storage media in many forms. Taking non-volatile storage media as an example, it includes optical or magnetic disks, such as all storage devices in any computer, etc., which can be used to implement the client devices, media gateways, transcoders, etc. shown in the figure. Volatile storage media include dynamic memory, such as the main memory of a certain computer platform. Tangible transmission media include coaxial cables; copper wires and optical fibers, including wires that constitute buses in computer systems. The form of carrier transmission media can take the form of electrical or electromagnetic signals, or sound or light waves, such as sound or light waves generated during radio frequency (RF) and infrared (IR) data communications. Therefore, common forms of computer-readable media include: floppy disks, floppy disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, DVDs or DVD-ROMs, any other optical media, punch cards, paper tapes, any other physical storage media with hole patterns, RAM, PROMs and EPROMs, FLASH-EPROMs, any other memory chips or cassettes, carriers that transmit data or instructions, cables or links that transmit such carriers, or any other media from which a computer can read programming code and / or data. Among other things, computer readable media may be involved in carrying one or more sequences of one or more instructions to a processor for execution.

[0099] The scope of protection is defined solely by the appended claims. The scope is intended and should be interpreted to be as broad as it would be when interpreted in accordance with this specification, with the ordinary meaning of the language used in the claims consistent with the following examination history and interpreted to include all structural and functional equivalents. No claim is intended to include subject matter that fails to meet the requirements of sections 101, 102, or 103 of the Patent Act, nor should they be interpreted in such a manner. Any unintended coverage of such subject matter is not claimed herein.

[0100] Except as described above, what is described and shown is not intended and should not be construed to cause any component, step, function, object, benefit, advantage or equivalent to be exclusive to the public regardless of whether it is recited in the claims.

[0101] It should be understood that the terms and expressions used herein have the ordinary meanings given to such terms and expressions relative to their respective fields of investigation and study, unless otherwise specified herein. Relational terms such as "first" and "second" may be used only to distinguish one entity or action from another entity or action, without necessarily requiring or implying any actual such relationship or order between such entities or actions. The terms "comprise", "include", "contain", "included" or any other variation thereof are intended to cover non-exclusive inclusion, such that a process, method, article or device that includes or contains a series of elements or steps includes not only these elements or steps, but also other elements or steps that are not explicitly listed or inherent to such processes, methods, articles or devices. Without further limitation, an element preceded by "a" does not exclude the presence of other identical elements in the process, method, article or device that includes the element.

[0102] Unless otherwise stated, any and all measurements, values, ratings, positions, magnitudes, dimensions, etc., described in this specification (including the claims that follow) are approximate values, not exact values. Such quantities are intended to have a reasonable range consistent with the functions to which they are related and consistent with the practices in the field to which they belong. For example, unless otherwise expressly stated, parameter values, etc. may vary from the specified amount by ±10%.

[0103] In addition, in the foregoing detailed description, various examples group various features together in order to simplify this disclosure. This method of disclosure should not be interpreted as reflecting an intention that the claimed examples require more features than those explicitly recited in each claim. On the contrary, as reflected in the following claims, the subject matter to be protected does not lie in all the features of any single disclosed example. Therefore, the following claims are hereby incorporated into the detailed description, and each claim stands on its own as a separately claimed subject matter.

[0104] Although the foregoing describes what are believed to be the best modes and other examples, it is understood that various modifications may be made in such modes and examples, that the subject matter disclosed herein may be implemented in various forms and examples, and that such modes and examples may be applied to a variety of applications, only some of which are described herein. The following claims are intended to claim any and all modifications and variations that fall within the true scope of the present concept.

Claims

1. A mobile device suitable for displaying an image, the mobile device comprising: an optical assembly, wherein said optical assembly has a viewing area for viewing a scene and for presenting an overlaid image to a user on said scene in said viewing area; as well as at least one sensor for determining the position and orientation of the mobile device; a processor coupled to the mobile device; a memory, wherein the processor is accessible to the memory; as well as Programming is performed in the memory, wherein the processor executes the program programming to configure the mobile device to perform functions, the functions comprising: using at least one sensor to monitor the position and orientation of the mobile device; requesting a previously captured image from a server relative to a location corresponding to a location captured by the previously captured image; receiving a previously captured image from the server; selecting an image from the requested previously captured images responsive to the mobile device position and orientation; generating an overlay image from the selected image, including an image icon associated with the selected image; displaying the overlapping image on the optical assembly; receiving an image selection identifying one of the image icons in the displayed overlay image; and A selected image associated with the identified image icon is displayed in the viewing area via the optical assembly.

2. The mobile device according to claim 1, characterized in that: The mobile device is an eye-mounted device, comprising: a frame supporting the optical assembly; and temples extending from the sides of the frame; The optical component is a lens optical component supported by the frame.

3. The mobile device according to claim 2, characterized in that: The processor executing the program configures the mobile device to receive the image selection including the following functions: monitoring an eye gaze direction relative to an optical component of the eye-mounted device; positioning a cursor on an optical component responsive to the monitored gaze direction; monitoring the time a cursor is positioned over one of the image icons; and When the monitored time exceeds the time threshold, an image associated with one of the image icons is displayed.

4. The mobile device according to claim 2, characterized in that The processor executes a program to configure the mobile device to perform additional functions, wherein the additional functions include: Monitoring an image termination instruction on the eye-mounted device; and The rendering of a rendered image in response to an image termination instruction is canceled.

5. The mobile device according to claim 2, characterized in that The processor executing the program further configures the mobile device to perform additional functions, wherein the additional functions include: associating a plurality of additional images with the selected image associated with the identified image icon based on the location; Monitoring image advancement instructions on the eye-mounted device; and A plurality of additional images are repeatedly cycled in reverse chronological order in response to the image advance command.

6. The mobile device according to claim 1, characterized in that The optical assembly includes a touch screen display and the processor executes a program to configure the mobile device to receive an image selection including the following functions: monitoring the touch screen display for user input selection of one of the image icons on the touch screen display; An image associated with one of the image icons is presented in response to a user input selection of one of the image icons on the touch screen display.

7. The mobile device according to claim 1, characterized in that The processor executes the program to configure the mobile device to select an image including the following functions: determining a location of the mobile device; determining a field of view of the optical assembly; Calculating the position coordinate range within the determined azimuth field of view; as well as Select images that have position coordinates within the calculated position coordinate range.

8. The mobile device according to claim 1, characterized in that The processor executing the program to configure the mobile device to generate an overlay image includes the following functions: monitoring the location of the mobile device; monitoring a field of view of the optical assembly; Calculating the position coordinates within the determined azimuth field of view; mapping the selected image to the calculated position coordinates; recognize visual cues; as well as An image icon is positioned within the overlay image in response to the computed position coordinates and the visual cue.

9. The mobile device according to claim 1, characterized in that The optical assembly also includes a camera and wherein the processor executing the program further configures the mobile device to perform additional functions, wherein the additional functions include: capturing an image with the camera; obtaining a location of the mobile device when the image is captured; associating a location at which the image was taken with the image; and The image and associated location are sent to the server.

10. A method for displaying an image, the method comprising: using at least one sensor to monitor the position and orientation of the mobile device; requesting a previously captured image from a server relative to a location corresponding to a location captured by the previously captured image; receiving a previously captured image from the server; selecting an image from the requested previously captured images responsive to the mobile device position and orientation; generating an overlay image from the selected image, including an image icon associated with the selected image; displaying the overlapping images on an optical assembly; receiving an image selection identifying one of the image icons in the displayed overlay image; A selected image associated with the identified image icon is displayed in a viewing area of ​​the optical assembly via the optical assembly of the mobile device.

11. The method according to claim 10, characterized in that The mobile device is an eye-mounted device including a see-through optical component.

12. The method according to claim 11, characterized in that The receiving image selection comprises: monitoring an eye gaze direction relative to an optical component of the eye-mounted device; positioning a cursor on an optical component responsive to the monitored gaze direction; monitoring the time a cursor is positioned over one of the image icons; and When the monitored time exceeds the time threshold, an image associated with one of the image icons is displayed.

13. The method according to claim 10, further comprising: monitoring an image termination instruction on the mobile device; The rendering of a rendered image in response to an image termination instruction is canceled.

14. The method according to claim 10, further comprising: associating a plurality of additional images with the selected image associated with the identified image icon based on the location; Monitor image forward instructions; A plurality of additional images are repeatedly cycled in reverse chronological order in response to the image advance command.

15. The method according to claim 10, characterized in that The optical assembly includes a touch screen display, and wherein receiving an image selection includes: monitoring the touch screen display for user input selection of one of the image icons on the touch screen display; An image associated with one of the image icons is presented in response to a user input selection of one of the image icons on the touch screen display.

16. The method according to claim 10, characterized in that The selected image includes: determining a location of the mobile device; determining a field of view of the optical assembly; Calculating the position coordinate range within the determined azimuth field of view; Select images that have position coordinates within the calculated position coordinate range.

17. The method according to claim 10, characterized in that Generating overlapping images comprises: monitoring the location of the mobile device; monitoring a field of view of the optical assembly; Calculate the position coordinates within the monitored azimuth field of view; Mapping the selected image to the calculated position coordinates; recognize visual cues; An image icon is positioned within the overlay image in response to the computed position coordinates and the visual cue.

18. The method according to claim 10, characterized in that The optical assembly further comprises a camera, and wherein the method further comprises: Capturing an image using the camera; obtaining a location of the mobile device when the image is captured; Associating the location at which the image was taken with the image; and The image and associated location are sent to the server.

19. A non-transitory computer readable medium storing program code which, when executed, causes an electronic processor to perform the following steps: using at least one sensor to monitor the position and orientation of the mobile device; requesting a previously captured image from a server relative to a location corresponding to a location captured by the previously captured image; receiving a previously captured image from the server; selecting an image from the requested previously captured images responsive to the mobile device position and orientation; generating an overlay image from the selected image, including an image icon associated with the selected image; displaying the overlaid image on an optical assembly of the mobile device; receiving an image selection identifying one of the image icons in the displayed overlay image; as well as A selected image associated with the identified image icon is displayed in a viewing area of ​​the optical assembly via the optical assembly of the mobile device.

20. The non-transitory computer readable medium of claim 19, wherein when the stored program code is executed, selection of an image may be caused by: determining a location of the mobile device; determining a field of view of the optical assembly; Calculating the position coordinate range within the determined azimuth field of view; Select images that have position coordinates within the calculated position coordinate range.

Images