Medical image overlay for augmented reality experience

Abstract

A medical image overlay application for an augmented reality (AR) eyewear device. When the eyewear device detects that the camera field of view includes a medical image location, the image overlay application enables a user of the eyewear device to activate image overlay on the display. The medical image location is defined relative to the virtual marker. The image overlay includes one or more medical images presented according to configurable transparency values. The image registration tool transforms the location and scale of each medical image into a physical environment such that the medical image presented on the display closely matches the location and size of the real object.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. application serial number 17 / 882,158, filed on August 5, 2022, the contents of which are incorporated herein by reference in their entirety. Technical Field

[0003] Examples set forth in this disclosure relate to the field of augmented reality (AR) experiences for electronic devices including wearable devices such as eye-mounted devices. More specifically, but not limited to, this disclosure describes registering and presenting medical images on a display of an eye-mounted device. Background Art

[0004] Many types of computers and electronic devices available today, such as mobile devices (e.g., smartphones, tablets, and laptops), handheld devices, and wearable devices (e.g., smart glasses, digital eyewear, headgear, headbands, and head-mounted displays), include various cameras, sensors, wireless transceivers, input systems, and displays. Users sometimes refer to information on these devices during physical activities, such as exercise.

[0005] Virtual reality (VR) technology generates a complete virtual environment including realistic images, which is sometimes presented on a VR headset or other head-mounted display. The VR experience allows the user to move around in the virtual environment and interact with virtual objects. AR is a VR technology that combines real objects in the physical environment with virtual objects and displays the combination to the user. The combined display gives the illusion that the virtual objects really exist in the environment, especially when the virtual objects appear and behave like real objects. Cross-reality (XR) is generally understood as an umbrella term for systems that include or combine elements from AR, VR, and MR (mixed reality) environments.

[0006] Automatic speech recognition (ASR) is a field of computer science, artificial intelligence, and linguistics that involves receiving spoken words and converting them into audio data suitable for processing by a computing device. The processed audio frame data can be used to translate the received spoken words into text, or to convert the spoken words into commands for controlling and interacting with various software applications. ASR processing can be used by computers, handheld devices, wearable devices, phone systems, automobiles, and a wide variety of other devices to facilitate human-computer interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The features of the various examples described will be readily understood from the following detailed description with reference to the accompanying drawings. In the description and throughout the several views of the accompanying drawings, each element is referenced by a reference numeral. When multiple similar elements are present, a single reference numeral may be assigned to the similar element, with additional uppercase or lowercase letters referring to the specific element.

[0008] The various elements shown in the figures are not drawn to scale unless otherwise indicated. The sizes of the various elements may be enlarged or reduced for clarity. These figures depict one or more embodiments and are presented by way of example only and should not be construed as limiting. The drawings include the following figures:

[0009] Figure 1A is a side view of an eye-mounted device suitable for use with an example virtual guided fitness system (right);

[0010] Figure 1B yes Figure 1A a partial perspective view of the optical components and electronics in a portion of the eyewear device shown;

[0011] Figure 1C yes Figure 1A Side view of the eyewear device (left);

[0012] Figure 1D yes Figure 1C a partial perspective view of the optical components and electronics in a portion of the eyewear device shown;

[0013] Figure 2A and 2B is a rear view of an eye-mounted device used in an example virtual guided fitness system;

[0014] Figure 3 is a block diagram illustrating an example of capturing visible light using the example eye-mounted device shown in any of the preceding figures;

[0015] Figure 4 is a functional block diagram of an example overlay system including an eye-mounted device and a server system connected via various networks;

[0016] Figure 5 Is configured for Figure 4 An illustration of an example hardware configuration for a mobile device of an example overlay system;

[0017] Figure 6 is a user's perspective in an example environment for describing simultaneous localization and mapping (SLAM);

[0018] Figure 7 is a flow chart listing steps in an example method of presenting a medical image overlay on a display;

[0019] Fig. 8A is an illustration of a medical film regarding an object of interest during an example process of capturing an example medical image;

[0020] Figure 8B is an illustration of an example medical image;

[0021] Figure 8C is a perspective view of an example virtual marker presented on an example display relative to a physical environment;

[0022] Fig.8D is a perspective view of an example medical image overlay including one or more medical images presented as an overlay on a display in a foreground relative to an object of interest; and

[0023] Fig. 8E is a perspective view of another example medical image overlay presented on a display. DETAILED DESCRIPTION

[0024] An overlay application for an AR eye-wear device. The application enables a user of the eye-wear device to view a medical image (e.g., a radiographic image of an object of interest such as a tooth) presented as an overlay relative to a target. An image registration tool converts the position, rotation, and scale of the original medical image so that the image overlay is closely registered with the target from any angle viewed through the eye-wear display.

[0025] Various embodiments and details are described with reference to an example for presenting a medical image overlay in an augmented reality environment. In an example embodiment, a method relates to an eye-mounted device having a camera and a display, and includes storing a virtual marker, the virtual marker including a marker position defined relative to a physical environment, and registering a medical image associated with a target, wherein the medical image includes an image position defined relative to the marker position. The method also includes determining a current eye-mounted device position relative to the marker position based on video frame data captured using the camera, and presenting a medical image overlay at an image position on the display according to the current eye-mounted device position. The medical image overlay includes one or more registered medical images presented on the display according to a transparency value. The medical image overlay including the transparency value can be adjusted and controlled using voice commands, gestures, or touchpad input.

[0026] In another example embodiment, the method includes selectively presenting a medical image overlay comprising a plurality of medical images. In this example, the method includes determining whether a medical image location is detected within a field of view of a camera based on the captured video frame data, and in response, selectively presenting a medical image associated with the detected image location.

[0027] Although various systems and methods are described herein with reference to dental surgery, the techniques described herein can be applied to substantially any type of activity or work requiring image overlays. For example, image overlays are a required tool for activities such as dentistry, medical examinations, surgery, industrial applications, quality control activities, inspections, examinations, and various investigations.

[0028] The following detailed description includes systems, methods, techniques, instruction sequences, and computer program products that illustrate the examples set forth in this disclosure. Many details and examples are included for the purpose of providing a thorough understanding of the disclosed subject matter and its related teachings. However, those skilled in the relevant art can understand how to apply the related teachings without these details. The various aspects of the disclosed subject matter are not limited to the specific devices, systems, and methods described, because the related teachings can be applied or practiced in various ways. The terms and nomenclature used herein are only for the purpose of describing specific aspects and are not intended to be restrictive. In general, well-known instruction instances, protocols, structures, and techniques are not necessarily shown in detail.

[0029] As used herein, the terms "connect," "connected," "coupled," and "coupled" refer to any logical, optical, physical, or electrical connection, including a link or the like, through which an electrical or magnetic signal generated or supplied by one system element is transferred to another coupled or connected system element. Unless otherwise specified, coupled or connected elements or devices are not necessarily directly connected to each other and may be separated by intermediate components, elements, or communication media, one or more of which can modify, manipulate, or carry electrical signals. The term "on" means directly supported by an element, or indirectly supported by an element through another element that is integrated into or supported by the element.

[0030] The term "proximal" is used to describe an item or part of an item that is near, adjacent to, or beside an object or person; or closer relative to the rest of the item, which may be described as "distal." For example, the end of an item closest to the object may be called the proximal end, while the generally opposite end may be called the distal end.

[0031] For purposes of illustration and discussion, the orientation of the eye-mounted device, associated components, and any complete device incorporating an eye scanner and camera (such as shown in any of the figures) is given by way of example only. In operation for a particular variable optical processing application, the eye-mounted device may be oriented in any other orientation (e.g., up, down, sideways, or any other orientation) suitable for the particular application of the eye-mounted device. Additionally, within the scope of use herein, any directional terms, such as front, back, inward, outward, toward, left, right, longitudinal, lateral, up, down, above, below, top, bottom, and side, are used by way of example only and are not limited to the direction or orientation of any optical device or component of an optical device constructed as otherwise described herein.

[0032] Advanced AR technologies, such as computer vision and object tracking, can be used to produce perceptually rich and immersive experiences. Computer vision algorithms extract three-dimensional data about the physical world from data captured in digital images or videos. Object recognition and tracking algorithms are used to detect objects in digital images or videos, estimate their orientation or pose, and track their movement over time.

[0033] Additional objects, advantages and novel features of the examples will be set forth in part in the following description, and in part will become apparent to those skilled in the art upon examination of the following and the accompanying drawings, or may be learned by production or operation of the examples. The objects and advantages of the subject matter may be realized or attained by the methods, tools and combinations particularly pointed out in the appended claims.

[0034] In an example configuration, an eye-mounted device with augmented reality (AR) capabilities is used in the system described herein. It is desirable to use an eye-mounted device in the system described herein because such a device is scalable, customizable, enables a personalized experience, can apply effects anywhere, and ensures user privacy by only allowing the user to view the transmitted information. The eye-mounted device can be used without any dedicated hardware in the sample configuration, such as the SPECTACLES provided by Snap Inc. of Santa Monica, California. TM .

[0035] like Figures 1A-1D As shown, the eyewear device 100 includes a first camera 114A and a second camera 114B. The camera 114 captures image information of a scene from different viewpoints. The captured image can be used to project a three-dimensional display onto an image display for three-dimensional (3D) viewing.

[0036] Cameras 114 are sensitive to wavelengths in the visible light range. Each of cameras 114 defines a different forward-facing field of view that overlaps to generate a 3D depth image; for example, first camera 114A defines a first field of view 111A and second camera 114B defines a second field of view 111B. In general, a "field of view" refers to the portion of a scene visible through a camera at a particular location and orientation in space. Fields of view 111 have overlapping fields of view 304 ( Figure 3 ). When the camera captures an image, objects or object features outside the field of view 111 are not recorded in the original image (e.g., a photograph or picture). The field of view describes the angular range or degree to which the image sensor of the camera 114 picks up electromagnetic radiation of a given scene in the captured image of the given scene. The field of view can be expressed as the angular size of the viewing cone; i.e., the viewing angle. The viewing angle can be measured horizontally, vertically, or diagonally.

[0037] In an example configuration, one or both cameras 114 have a field of view of 100° and a resolution of 480x480 pixels. The "coverage angle" describes the range of angles that the lens of camera 114 can effectively image. Typically, a camera lens produces an image circle large enough to completely cover the camera's film or sensor, possibly including some vignetting (e.g., the image gets darker toward the edges compared to the center). If the camera lens's coverage angle does not fill the sensor, the image circle will be visible, typically with strong vignetting toward the edges, and the effective viewing angle will be limited to the coverage angle.

[0038] Examples of suitable cameras 114 include high-resolution complementary metal oxide semiconductor (CMOS) image sensors and digital VGA cameras (video graphics arrays) capable of 480p (e.g., 640 x 480 pixels), 720p, 1080p, or higher resolutions. Other examples include cameras 114 that can capture high-definition (HD) video at high frame rates (e.g., 30 to 60 frames per second or more) and store recordings at a resolution of 1216 by 1216 pixels (or higher).

[0039] The eye-wearing device 100 can capture image sensor data from the camera 114 and geographic location data digitized by the image processor for storage in a memory. The camera 114 captures corresponding raw images (e.g., left and right raw images) in a two-dimensional spatial domain, which includes a pixel matrix on a two-dimensional coordinate system, which includes an X-axis for horizontal positioning and a Y-axis for vertical positioning. Each pixel includes a color attribute value (e.g., a red pixel light value, a green pixel light value, or a blue pixel light value); and a positioning attribute (e.g., an X-axis coordinate and a Y-axis coordinate).

[0040] To capture stereoscopic images for later display as a 3D projection, image processor 412 ( Figure 4) may be coupled to the camera 114 to receive and store visual image information. The image processor 412 or another processor controls the operation of the camera 114 to act as a stereo camera that simulates human binocular vision, and a time stamp may be added to each image. The time stamp on each pair of images allows the images to be displayed together as part of a 3D projection. 3D projection creates an immersive, lifelike experience that is desirable in a variety of environments, including virtual reality (VR) and video games.

[0041] Figure 1B yes Figure 1A 1 is a perspective cross-sectional view of the right corner 110A of the eyewear device 100 depicting the first camera 114A, additional optical components, and electronics. Figure 1C yes Figure 1A 1 is a side view (left) of an example hardware configuration of the eye-mounted device 100 showing the second camera 114B of the camera system. Figure 1D yes Figure 1C 1 is a perspective cross-sectional view of the left corner 110B of the eyewear device 100 depicting the second camera 114B, additional optical components, and electronics of the camera system.

[0042] like Figure 1B As shown in the example of FIG. 1 , the eye-mounted device 100 includes a first camera 114A and a circuit board 140A, which may be a flexible printed circuit board (PCB). The first hinge 126A connects the right corner 110A to the first temple 125A of the eye-mounted device 100. In some examples, the first camera 114A, the flexible PCB 140A, or other electrical connectors or contacts may be located on the first temple 125A or the first hinge 126A.

[0043] The right corner 110A includes a corner body 190 and a corner cap. Figure 1B The corner cap is omitted in the cross section of FIG. Disposed inside the right corner 110A are various interconnected circuit boards 109 (such as a flexible PCB 140A), which include controller circuitry for the first camera 114A, one or more microphones 139, one or more speakers 191, low-power wireless circuitry (e.g., for communicating via Bluetooth TM for wireless short-range network communications), and high-speed wireless circuits (for example, for wireless LAN communications via Wi-Fi).

[0044] The first camera 114A is coupled to or disposed on the flexible PCB 140A and is covered by a camera cover lens that is aimed through one or more openings formed in the frame 105. For example, Figure 2AAs shown, the right edge 107A of the frame 105 is connected to the right corner 110A and includes one or more openings for the camera cover lens. The frame 105 includes a front face that is configured to face outwardly away from the user's eyes. The opening for the camera cover lens is formed on and through the front or outward-facing face of the frame 105. In this example, the first camera 114A has an outward-facing field of view 111A that has a line of sight or viewing angle associated with the right eye of the user of the eyewear device 100 (e.g., Figure 3 ). The camera cover lens may also be adhered to the front or outward facing surface of the right corner 110A, in which the opening is formed with an outward facing coverage angle, but in a different outward direction. The coupling may also be indirect via an intervening component.

[0045] like Figure 1D As shown in the example of FIG. 1 , the eye-mounted device 100 includes a second camera 114B and a circuit board 140B, which may be a flexible printed circuit board (PCB). The second hinge 126B connects the left corner 110B to the second temple 125B of the eye-mounted device 100. In some examples, the second camera 114B, the flexible PCB 140B, or other electrical connectors or contacts may be located on the second temple 125B or the second hinge 126B.

[0046] The left corner 110B includes a corner body 190 and a corner cap. Figure 1D The corner cap is omitted in the cross section of FIG. Disposed inside the right corner 110A are various interconnected circuit boards (such as a flexible PCB 140B) that include controller circuitry for the second camera 114B.

[0047] The camera 114 is coupled to or disposed on a corresponding flexible PCB 140 and is covered by a camera cover lens that is aimed through one or more openings formed in the frame 105. For example, Figure 2A 10A and 107B are connected to the left corner 110B and include one or more openings for the camera cover lens. The frame 105 includes a front face that is configured to face outward and away from the user's eyes. The opening for the camera cover lens is formed on and through the front or outward-facing face of the frame 105. In this example, the camera 114 has a corresponding outward-facing field of view 111 that has a line of sight or perspective associated with a corresponding eye of the user of the eyewear device 100 (e.g., Figure 3). The camera cover lens may also be adhered to the front or outward facing surface of the corresponding corner 110, in which surface the opening is formed with an outward facing coverage angle, but in a different outward direction. The coupling may also be indirect via an intervening component.

[0048] Figure 2A and 2B An example hardware configuration of an eye-mounted device 100 including two different types of image displays is depicted. The size and shape of the eye-mounted device 100 is configured in a form to be worn by a user. The form of glasses is shown in the example shown. The eye-mounted device 100 can take other forms and can be combined with other types of frames; for example, headgear, headphones, or helmets.

[0049] In the glasses example, the eyewear device 100 includes a frame 105 including a right edge 107A connected to a left edge 107B via a nose bridge 106, the nose bridge 106 being configured to receive the user's nose to support the eyewear device 100 on the user's head. The right edge 107A includes a first aperture 175A that holds a first optical element 180A. The left edge 107B includes a second aperture 175B that holds a second optical element 180B. Figure 2B As shown, in some embodiments, each optical element 180A, 180B includes an integrated image display (e.g., a first display 182A and a second display 182B). As used herein, the term "lens" is intended to include a transparent or translucent glass or plastic sheet having curved and flat surfaces that cause light to converge or diverge or cause little or no convergence or divergence.

[0050] A touch-sensitive input device, such as a touchpad 181, is positioned on the first temple 125A. As shown, the touchpad 181 may have a clearly visible border, or include a raised or other tactile edge that provides feedback to the user about the location and border of the touchpad 181; alternatively, the border may be subtle and not easily seen or felt. The eye-mounted device 100 may include a touchpad on the other side that operates independently or in conjunction with the touchpad 181.

[0051] The surface of the touch pad 181 is configured to detect finger touches, taps, and gestures (e.g., moving touch) for use with a graphical user interface (GUI) displayed by the eye-mounted device on the image display to allow the user to navigate and select menu options in an intuitive manner, which enhances and simplifies the user experience.

[0052] Detecting finger input on the touch pad 181 can enable a variety of functions. For example, touching any location on the touch pad 181 can cause the GUI to display or highlight an item on the image display, which can be projected onto at least one of the optical components 180. Tapping or double-clicking on the touch pad 181 can select an item or icon. Sliding or swiping a finger in a particular direction (e.g., from front to back, back to front, top to bottom, or bottom to top) can cause an item or icon to slide or scroll in a particular direction; for example, move to the next item, icon, video, image, page, or slide. Sliding a finger in another direction can slide or scroll in the opposite direction; for example, move to the previous item, icon, video, image, page, or slide. The touch pad 181 can be positioned substantially anywhere on the eye-wearing device 100.

[0053] In one example, a recognized finger gesture of a single tap on touch pad 181 initiates a selection or press of a GUI element in an image presented on the image display of optical assembly 180. Adjusting the image presented on the image display of optical assembly 180 based on the recognized finger gesture can be the primary action for selecting or submitting a GUI element on the image display of optical assembly 180 for further display or execution.

[0054] Figure 2A 1 is an example hardware configuration of the eye-mounted device 100 in which the right corner 110A supports a microphone 139 and a speaker 191. The microphone 139 includes a transducer that converts sound into a corresponding electrical audio signal. The microphone 139 in the illustrated example is positioned with an opening facing inward toward the wearer to facilitate receiving sound waves, such as human speech including verbal commands and questions. Additional or differently oriented openings may be implemented. In other example configurations, the eye-mounted device 100 is coupled to one or more microphones 139 that are configured to operate together or independently and are positioned at various locations on the eye-mounted device 100.

[0055] The speaker 191 includes an electroacoustic transducer that converts an electrical audio signal into a corresponding sound. The speaker 191 is controlled by one of the processors 422, 432 or by the audio processor 413 ( Figure 4 ). As shown, the speaker 191 in this example includes a series of rectangular holes that face inward to direct sound to the wearer. Additional or differently oriented holes may be implemented. In other example configurations, the eye-mounted device 100 is coupled to one or more speakers 191 that are configured to operate together (e.g., in stereo, to produce surround sound in a zone) or independently and are positioned at various locations on the eye-mounted device 100. For example, one or more speakers 191 may be incorporated into the frame 105, the temple 125, or the corner 110 of the eye-mounted device 100.

[0056] Despite Figure 2A and Figure 2B 170A, 170B, and 170C. The eyewear device 100 is shown as having two optical elements 180, but the eyewear device 100 may include other arrangements, such as a single optical element (or it may not include any optical element 180), depending on the application or intended user of the eyewear device 100. As further shown, the eyewear device 100 includes a right corner 110A adjacent to the right lateral side 170A of the frame 105 and a left corner 110B adjacent to the left lateral side 170B of the frame 105. The corners 110 may be integrated into the frame 105 on the respective sides 170 (as shown), or implemented as separate components attached to the frame 105 on the respective sides 170. Alternatively, the corners 110A, 110B may be integrated into temples (not shown) attached to the frame 105.

[0057] In one example, each image display of optical assembly 180 includes an integrated image display (eg, first display 182A and second display 182B). Figure 2A As shown, each optical assembly 180 has a display 182 that includes a suitable display matrix 177, such as a liquid crystal display (LCD), an organic light emitting diode (OLED) display, or other such display. Each optical assembly 180 also includes one or more optical layers 176, which may include lenses, optical coatings, prisms, reflectors, waveguides, optical strips, and other optical components in any combination. The optical layers (e.g., Figure 2A 176A-176N in the optical layer 176A-176N) may include a prism having a suitable size and configuration and including a first surface for receiving light from the display matrix and a second surface for emitting light toward the user's eyes. The prisms of the optical layer 176A-176N may extend over all or at least a portion of the corresponding apertures 175 formed in the left edge 107 and the right edge 107 to allow the user to see the second surface of the prism when the user's eyes look through the corresponding edge 107. The first surface of the prism of the optical layer 176A-176N faces upward from the frame 105, and the display matrix 177 is overlaid on the prism so that photons and light emitted by the display matrix 177 strike the first surface. The prism may be designed in a size and shape so that light is refracted within the prism and directed toward the user's eyes by the second surface of the prism of the optical layer 176A-176N. In this regard, the second surface of the prism of the optical layer 176A-176N may be convex to direct light toward the center of the eye. The prisms may optionally be sized and shaped to magnify the image projected by the display matrix 177, and light travels through the prisms so that the image viewed from the second surface is larger than the image emitted from the display matrix 177 in one or more dimensions.

[0058] In one example, the optical layers 176A-176N may include a transparent LCD layer (holding the lens open) unless and until a voltage is applied that renders the layer opaque (closing or blocking the lens). The image processor 412 on the eye-mounted device 100 may execute a program to apply a voltage to the LCD layer, thereby creating an active shutter system that makes the eye-mounted device 100 suitable for viewing visual content when displayed as a 3D projection. Technologies other than LCD may be used for active shutter modes, including other types of reactive layers that respond to a voltage or another type of input.

[0059] In another example, the image display device of the optical assembly 180 has a display 182, and the display 182 includes Figure 2B Each optical assembly 180 includes a corresponding laser projector 150, such as a three-color laser projector using a scanning mirror or a galvanometer. Each laser projector 150 is disposed in or on a corresponding temple 125 of the eyewear device 100. In this example, each optical assembly 180 includes one or more optical strips (in Figure 2B 155A-155N), these optical strips are spaced apart and span the width of the lens of each optical assembly 180 or span the depth of the lens between the front and back surfaces of the lens.

[0060] As photons projected by the laser projector 150 travel through the lens of each optical assembly 180, the photons encounter optical strips 155A-155N. When a particular photon encounters a particular optical strip, the photon is redirected to the user's eye, or to the next optical strip. The combination of modulation of the laser projector 150 and modulation of the optical strips controls specific photons or light beams. In an example, the processor controls the optical strips 155A-155N by issuing mechanical, acoustic, or electromagnetic signals. Although shown as having two optical assemblies 180, the eye-mounted device 100 may include other arrangements (such as a single or three optical assemblies), or each optical assembly 180 may have a different arrangement depending on the application or intended user of the eye-mounted device 100.

[0061] Figure 33D scene 306, a first raw image 302A captured using a first camera 114A, and a second raw image 302B captured using a second camera 114B. As shown, the first field of view 111A can overlap with the second field of view 111B. The overlapping field of view 304 represents the portion of the image captured using the two cameras 114. When referring to the field of view, the term "overlap" means that the pixel matrices in the generated raw images overlap by thirty percent (30%) or more. "Substantially overlap" means that the pixel matrices in the generated raw images or infrared images of the scene overlap by fifty percent (50%) or more. As described herein, the two raw images 302 can be processed to include a timestamp, which allows the images to be displayed together as part of a three-dimensional projection.

[0062] To capture stereo images, such as Figure 3 As shown, a pair of raw red, green, and blue (RGB) images of a 3D scene 306 are captured at a given moment—a first raw image 302A captured using a first camera 114A and a second raw image 302B captured using a second camera 114B. When the pair of raw images 302 are processed (e.g., by an image processor 412), a depth image is generated. The generated depth image can be viewed on the optical assembly 180 of the eye-wearing device, on another display (e.g., an image display 580 on the mobile device 401), or on a screen.

[0063] The generated depth image is in a three-dimensional spatial domain and may include a matrix of vertices on a three-dimensional position coordinate system including an X-axis for horizontal positioning (e.g., length), a Y-axis for vertical positioning (e.g., height), and a Z-axis for depth (e.g., distance). Each vertex may include a color attribute (e.g., a red pixel light value, a green pixel light value, or a blue pixel light value); a positioning attribute (e.g., an X position coordinate, a Y position coordinate, and a Z position coordinate); a texture attribute; a reflectivity attribute; or a combination thereof. The texture attribute quantifies the perceived texture of the depth image, such as the spatial arrangement of color or intensity in a vertex region of the depth image.

[0064] Figure 4 4 is a functional block diagram of an example overlay system 400 including an eye-mounted device 100, a mobile device 401, and a server system 498 connected via various networks 495 such as the Internet. As shown, the overlay system 400 includes a low-power wireless connection 425 and a high-speed wireless connection 437 between the eye-mounted device 100 and the mobile device 401.

[0065] The eye-wearing device 100 includes one or more cameras 114 that capture still images, video images, or still and video images, as described herein. The camera 114 can have direct memory access (DMA) to the high-speed circuit 430 and act as a stereo camera. The camera 114 can be used to capture initial depth images, which can be rendered into three-dimensional (3D) models that are texture-mapped images of red, green, and blue (RGB) imaged scenes. The device 100 may also include a depth sensor that uses infrared signals to estimate the location of an object relative to the device 100. In some examples, the depth sensor includes one or more infrared emitters and one or more infrared cameras 410.

[0066] The eye-mounted device 100 also includes two image displays (one associated with the right side 170A and one associated with the left side 170B) of the optical assembly 180. The eye-mounted device 100 also includes an image display driver 442, an image processor 412, a low power circuit 420, and a high speed circuit 430. The image display of the optical assembly 180 is used to present images, including still images, video images, or still and video images. The image display driver 442 is coupled to the image display of the optical assembly 180 to control the display of the image.

[0067] Figure 4 The components of the eye-mounted device 100 shown in FIG. 1 are located on one or more circuit boards, such as a printed circuit board (PCB) or a flexible printed circuit (FPC) located in the edge or temple. Alternatively or additionally, the depicted components can be located in the corners, frame, hinge, or bridge of the eye-mounted device 100. The camera 114 includes a digital camera element, such as a complementary metal oxide semiconductor (CMOS) image sensor, a charge coupled device, a lens, or any other corresponding visible light or light capturing element that can be used to capture data (including still images or video of a scene with an unknown object).

[0068] like Figure 4 As shown, the high-speed circuit 430 includes a high-speed processor 432, a memory 434, and a high-speed wireless circuit 436. In this example, an image display driver 442 is coupled to the high-speed circuit 430 and operated by the high-speed processor 432 to drive the image display of the optical assembly 180. The high-speed processor 432 can be essentially any processor capable of managing high-speed communications and operations of any general-purpose computing system. The high-speed processor 432 includes the processing resources required to manage high-speed data transmission on a high-speed wireless connection 437 to a wireless local area network (WLAN) using a high-speed wireless circuit 436.

[0069] In some examples, the high-speed processor 432 executes an operating system, such as a LINUX operating system or other such operating system of the eye-mounted device 100, and the operating system is stored in the memory 434 for execution. In addition to any other tasks, the high-speed processor 432 executes the software architecture of the eye-mounted device 100, which is used to manage data transmission with the high-speed wireless circuit 436. In some examples, the high-speed wireless circuit 436 is configured to implement the Institute of Electrical and Electronics Engineers (IEEE) 802.11 communication standard, also referred to as Wi-Fi herein. In other examples, other high-speed communication standards can also be implemented by the high-speed wireless circuit 436.

[0070] The low power circuit 420 includes a low power processor 422 and a low power wireless circuit 424. The low power wireless circuit 424 and the high speed wireless circuit 436 of the eyewear device 100 may include a short range transceiver (Bluetooth TM or Bluetooth Low Energy (BLE)) and a wireless wide area network, local area network, or wide area network transceiver (e.g., cellular or Wi-Fi). The mobile device 401 (including a transceiver that communicates via a low power wireless connection 425 and a high speed wireless connection 437) can be implemented using details of the architecture of the eye-wearing device 100, as can other elements of the network 495.

[0071] The memory 434 includes any storage device capable of storing various data and applications, including, among other things, camera data generated by the cameras 114A, 114B, one or more infrared cameras 410, the image processor 412, and images generated by the image display driver 442 for display on the image display of each optical assembly 180. Although the memory 434 is shown as being integrated with the high-speed circuit 430, in other examples, the memory 434 can be a separate, separate element of the eye-mounted device 100. In some such examples, electrical routing lines can provide connections from the image processor 412 or the low-power processor 422 to the memory 434 through a chip including the high-speed processor 432. In other examples, the high-speed processor 432 can manage the addressing of the memory 434 so that the low-power processor 422 will start the high-speed processor 432 any time it is needed to perform a read or write operation involving the memory 434.

[0072] like Figure 4 As shown, various elements of the eye-mounted device 100 can be coupled to the low-power circuit 420, the high-speed circuit 430, or both. For example, the infrared camera 410 (including an infrared transmitter in some embodiments), the user input element 491 (e.g., a button switch, a touch pad 181, a microphone 139), and the inertial measurement unit (IMU) 472 can be coupled to the low-power circuit 420, the high-speed circuit 430, or both.

[0073] like Figure 5 As shown, CPU 540 of mobile device 401 may be coupled to camera system 570, mobile display driver 582, user input layer 591, and memory 540A, discussed in further detail below.

[0074] ]The server system 498 can be one or more computing devices as part of a service or network computing system, for example, including a processor, memory, and a network communication interface to communicate with the eye-wearing device 100 and the mobile device 401 via the network 495.

[0075] The output components of the eyewear device 100 include visual elements, such as reference Figure 2A and 2B The image display associated with each lens or optical component 180 (e.g., a liquid crystal display (LCD), a plasma display panel (PDP), a light emitting diode (LED) display, a projector, or a waveguide display). The eye-mounted device 100 may include an indicator facing the user (e.g., an LED, a speaker 191, or a vibration actuator) or a signal facing the outside (e.g., an LED, a speaker 191). The image display of each optical component 180 is driven by the image display driver 442. In some example configurations, the output components of the eye-mounted device 100 also include additional indicators, such as an audible element (e.g., a speaker 191), a tactile component (e.g., an actuator for generating tactile feedback, such as a vibration motor), and other signal generators. For example, the device 100 may include a set of indicators facing the user and a set of signals facing the outside. The set of indicators facing the user is configured to be seen or otherwise sensed by the user of the device 100. For example, the device 100 may include an LED display positioned so that the user can see it, one or more speakers 191 positioned to generate sounds that the user can hear, or an actuator that provides tactile feedback that the user can feel. The outwardly facing set of signals are configured to be seen or otherwise sensed by an observer in the vicinity of the device 100. Similarly, the device 100 may include an LED, speaker 191, or actuator configured and positioned to be sensed by an observer.

[0076] The user input elements 491 of the eye-mounted device 100 may include alphanumeric input components (e.g., a touch screen or touchpad 181 configured to receive alphanumeric input, an optical keyboard, or other alphanumeric configured elements), pointer-based input components (e.g., a mouse, touchpad 181, trackball, joystick, motion sensor, or other directional instrument), tactile input components (e.g., button switches; touch screen or touchpad 181 or other tactile configured elements that sense the location, force, or location and force of a touch or touch gesture), audio input components (e.g., microphone 139), and the like. The mobile device 401 and the server system 498 may include alphanumeric, pointer-based, tactile, audio, and other input components.

[0077] In some examples, the eye-wearing device 100 includes a collection of motion sensing components referred to as IMU 472. The motion sensing components can be microelectromechanical systems (MEMS) with microscopic moving parts, usually small enough to be part of a microchip. In some example configurations, the IMU 472 includes an accelerometer, a gyroscope, and a magnetometer. The accelerometer senses the linear acceleration of the device 100 relative to three orthogonal axes (x, y, z) (including the acceleration caused by gravity). The gyroscope senses the angular velocity of the device 100 around three rotational axes (pitch, roll, yaw). The accelerometer and gyroscope can together provide positioning, orientation, and motion data about the device relative to six axes (x, y, z, pitch, roll, yaw). The magnetometer (if present) senses the heading of the device 100 relative to magnetic north. The positioning of the device 100 can be determined by a position sensor (such as a GPS unit 473), one or more transceivers for generating relative position coordinates, an altitude sensor or a barometer, and other orientation sensors. Such positioning system coordinates may also be received from mobile device 401 over wireless connections 425 , 437 via low power wireless circuitry 424 or high speed wireless circuitry 436 .

[0078] The IMU 472 may include or cooperate with a digital motion processor or program that collects raw data from the components and calculates a number of useful values ​​regarding the location, orientation, and motion of the device 100. For example, the acceleration data collected from the accelerometers may be integrated to obtain the velocity relative to each axis (x, y, z); and integrated again to obtain the location of the device 100 (in a linear coordinate system x, y, and z). The angular velocity data from the gyroscopes may be integrated to obtain the location of the device 100 (in a spherical coordinate system). The program for calculating these useful values ​​may be stored in the memory 434 and executed by the high-speed processor 432 of the eye-mounted device 100.

[0079] The eye-mounted device 100 may optionally include additional peripheral sensors, such as biometric sensors, application-specific sensors, or display elements integrated with the eye-mounted device 100. For example, the peripheral device elements may include any I / O components, including output components, motion components, positioning components, or any other such components described herein. For example, the biometric sensors may include components for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body postures, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweat, or brain waves), identifying people (e.g., identification based on voice, retinal film, facial features, fingerprints, or electrical biosignals (such as electroencephalogram data)), etc.

[0080] The mobile device 401 may be a smartphone, tablet, laptop, access point, or any other such device capable of connecting to the eye-mounted device 100 using a low-power wireless connection 425 and a high-speed wireless connection 437. The mobile device 401 is connected to a server system 498 and a network 495. The network 495 may include any combination of wired and wireless connections.

[0081] like Figure 4 As shown, the overlay system 400 includes a computing device, such as a mobile device 401, coupled to the eye-mounted device 100 via a network 495. The overlay system 400 includes a memory (e.g., a non-transitory computer-readable medium) for storing instructions and a processor for executing instructions. In some embodiments, the memory and processing functions of the overlay system 400 can be shared or distributed between the processors and memories of the eye-mounted device 100, the mobile device 401, and / or the server system 498.

[0082] In some implementations, overlay system 400 includes one or more elements or modules, referred to herein as overlay application 910 , image registration tool 912 , positioning system 915 , image processing system 920 , and speech recognition module 925 .

[0083] As described herein, in some implementations, the overlay application 910 renders and presents the medical image overlay 800 on the display 182 .

[0084] In some embodiments, the image registration tool 912 includes any of a variety of image registration or image registration algorithms for transforming one or more sets of data (e.g., medical image data) into a single or common coordinate system. The field of image registration applies various transformation models, including linear models, vector space models, elastic or non-rigid models, quadratic forms, parameterizations, differential homeomorphisms, vector space models, and other mathematical and computational models. In the context of medical imaging, in some embodiments, the image registration tool 912 transforms the position, orientation, and size of the medical image 810 of the target 50 according to a single coordinate system (e.g., a physical coordinate system 610 associated with the physical environment 600) so that when presented on a display, the registered medical image appears at a position, orientation, and size corresponding to the position, orientation, and size of the target 50.

[0085] In some embodiments, the positioning system 915 obtains positioning data for determining the positioning of the eye-mounted device 100 relative to the physical environment 600. For example, the positioning system 915 can access the video frame data 900 captured using the camera 114B to determine the eye-mounted device position 840 in three-dimensional coordinates relative to the physical environment (with or without reference to data from other sources (such as an inertial measurement unit or IMU 472)). As used herein, the term "video frame data" refers to video motion data captured using one or more cameras 114A, 114B coupled to the eye-mounted device 100, including images, spatial data, and related information captured using substantially any sensor component of the camera in any form and at any sampling rate. In some embodiments, the positioning data can be derived from the motion frame data captured using the IMU 472, data collected from the GPS unit 473, or from a combination thereof.

[0086] In some embodiments, the image processing system 920 cooperates with the image display driver 442 and the image processor 412 to present a medical image overlay 800 as described herein on the display 182 of the corresponding optical assembly 180. In some embodiments, the medical image overlay 800 includes one or more medical images 810 that have been registered to the physical environment 600 using the image registration tool 912.

[0087] In some implementations, the speech recognition module 925 receives human speech, converts the received speech into audio frame data, recognizes a query or request based on the audio data, and performs actions related to and responsive to the recognized query or request.

[0088] Figure 5 is a high-level functional block diagram of an example mobile device 401. Mobile device 401 includes flash memory 540A that stores programs to be executed by CPU 540 to perform all or a subset of the functionality described herein.

[0089] Mobile device 401 may include camera 570, which includes at least two cameras (e.g., first and second visible light cameras with overlapping fields of view) or at least one camera and a depth sensor with substantially overlapping fields of view. Flash memory 540A may also include multiple images or videos generated via camera 570.

[0090] As shown in the figure, the mobile device 401 includes an image display 580, a mobile display driver 582 for controlling the image display 580, and a display controller 584. Figure 5 In the example of , image display 580 includes a user input layer 591 (eg, a touch screen) that is layered over or otherwise integrated into a screen used by image display 580 .

[0091] Examples of touch screen mobile devices that may be used include, but are not limited to, smartphones, personal digital assistants (PDAs), tablet computers, laptop computers, or other portable devices. However, the structure and operation of touch screen devices are provided by way of example; the subject technology described herein is not intended to be limited thereto. Therefore, for the purposes of this discussion, Figure 5 A block diagram is provided of an example mobile device 401 having a user interface including a touch screen input layer 591 for receiving input (via touch, multi-touch or gestures, etc., by hand, stylus or other implement) and an image display 580 for displaying content.

[0092] like Figure 5 As shown, the mobile device 401 includes at least one digital transceiver (XCVR) 510, such as a WWAN XCVR, for digital wireless communication via a wide area wireless mobile communication network. The mobile device 401 also includes additional digital or analog transceivers, such as short-range transceivers (XCVRs) 520 for short-range network communication, such as via NFC, VLC, DECT, ZigBee, Bluetooth TM For example, the short-range XCVR 520 may take the form of any available two-way wireless local area network (WLAN) transceiver compatible with one or more standard communication protocols implemented in wireless local area networks, such as one of the Wi-Fi standards under IEEE 802.11.

[0093] To generate location coordinates for positioning the mobile device 401, the mobile device 401 may include a global positioning system (GPS) receiver. Alternatively or additionally, the mobile device 401 may utilize any one or both of the short-range XCVR 520 and the WWAN XCVR 510 to generate location coordinates for positioning. For example, based on a cellular network, Wi-Fi, or Bluetooth TM The positioning system of can generate accurate position coordinates, especially when used in combination. Such position coordinates can be sent to the eye-wearing device via XCVR 510, 520 through one or more network connections.

[0094] In some examples, the mobile device 401 includes a collection of motion sensing components referred to as an inertial measurement unit (IMU) 572, which is used to sense the location, orientation, and motion of the mobile device 401. The motion sensing component can be a microelectromechanical system (MEMS) with microscopic moving parts, usually small enough to be part of a microchip. In some example configurations, the inertial measurement unit (IMU) 572 includes an accelerometer, a gyroscope, and a magnetometer. The accelerometer senses the linear acceleration (including the acceleration due to gravity) of the mobile device 401 relative to three orthogonal axes (x, y, z). The gyroscope senses the angular velocity of the mobile device 401 around three rotation axes (pitch, roll, yaw). The accelerometer and gyroscope can provide together the location, orientation, and motion data of the device relative to six axes (x, y, z, pitch, roll, yaw). The magnetometer (if present) senses the heading of the mobile device 401 relative to magnetic north.

[0095] The IMU 572 may include or cooperate with a digital motion processor or program that collects raw data from the components and calculates a number of useful values ​​regarding the location, orientation, and motion of the mobile device 401. For example, the acceleration data collected from the accelerometers may be integrated to obtain the velocity relative to each axis (x, y, z); and integrated again to obtain the location of the mobile device 401 (in linear coordinates x, y, and z). The angular velocity data from the gyroscopes may be integrated to obtain the location of the mobile device 401 (in spherical coordinates). The program for calculating these useful values ​​may be stored in one or more memory elements 540A, 540B, 540C and executed by the CPU 540 of the mobile device 401.

[0096] Transceivers 510, 520 (i.e., network communication interfaces) conform to one or more of the various digital wireless communication standards used by modern mobile networks. Examples of WWAN transceivers 510 include, but are not limited to, transceivers configured to operate according to Code Division Multiple Access (CDMA) and Third Generation Partnership Project (3GPP) network technologies, including, but not limited to, 3GPP Type 2 (or 3GPP2) and LTE, sometimes referred to as "4G". For example, transceivers 510, 520 provide two-way wireless information communication, including digitized audio signals, still images and video signals, web page information for display and network-related input, and various types of mobile messages communicated to / from mobile device 401.

[0097] The mobile device 401 also includes a microprocessor that serves as a central processing unit (CPU); Figure 4 540 in . A processor is a circuit having elements constructed and arranged to perform one or more processing functions (typically various data processing functions). Although discrete logic components may be used, these examples utilize components that form a programmable CPU. For example, a microprocessor includes one or more integrated circuit (IC) chips that contain electronic components to perform the functions of a CPU. For example, CPU 540 may be based on any known or available microprocessor architecture, such as a reduced instruction set computing (RISC) using an ARM architecture, as commonly used in today's mobile devices and other portable electronic devices. Of course, other arrangements of processor circuits may be used to form CPU 540 or processor hardware in smartphones, laptops, and tablets.

[0098] CPU 540 functions as a programmable host controller of mobile device 401 by configuring mobile device 401 to perform various operations (e.g., according to instructions or programs executable by CPU 540). For example, such operations may include various general operations of the mobile device, as well as operations related to programs applied on the mobile device. Although a processor may be configured using hardwired logic, a typical processor in a mobile device is a general-purpose processing circuit that is configured by executing a program.

[0099] The mobile device 401 includes a memory or storage system for storing programs and data. In this example, the memory system may include flash memory 540A, random access memory (RAM) 540B and other memory components 540C as needed. RAM 540B is used as short-term storage of instructions and data processed by CPU 540, such as working data processing memory. Flash memory 540A generally provides long-term storage.

[0100] Thus, in the example of mobile device 401, flash memory 540A is used to store programs or instructions executed by CPU 540. Depending on the type of device, mobile device 401 stores and runs a mobile operating system, through which specific applications are executed. Examples of mobile operating systems include Google Android, Apple iOS (for iPhone or iPad devices), Windows Mobile, Amazon Fire OS, RIM BlackBerry OS, etc.

[0101] The processor 432 within the eye-mounted device 100 can build a map of the environment surrounding the eye-mounted device 100, determine the position of the eye-mounted device within the environment map, and determine the relative positioning of the eye-mounted device to one or more objects in the mapped environment. The processor 432 can use a simultaneous localization and mapping (SLAM) algorithm applied to data received from one or more sensors to build a map and determine the position and positioning information. The sensor data includes images received from one or both of the cameras 114A, 114B, one or more distances received from a laser rangefinder, positioning information received from the GPS unit 473, motion and acceleration data received from the IMU 572, or a combination of data from such sensors or from other sensors that provide data that helps determine positioning information. In the context of augmented reality, the SLAM algorithm is used to build and update the environment map while tracking and updating the position of the device (or user) within the mapped environment. The mathematical solution can be approximated using various statistical methods, such as particle filters, Kalman filters, extended Kalman filters, and covariance intersection. In a system including a high-definition (HD) camera that captures video at a high frame rate (e.g., 30 frames per second), the SLAM algorithm updates the map and object positions at least as frequently as the frame rate; in other words, the map and localization are calculated and updated 30 times per second.

[0102] The sensor data includes one or more images received from one or both cameras 114A, 114B, one or more distances received from a laser rangefinder, positioning information received from a GPS unit 473, motion and acceleration data received from an IMU 472, or a combination of data from such sensors or from other sensors that provide data that contributes to positioning information.

[0103] Figure 6 An example physical environment 600 and elements useful when using SLAM algorithms and other types of tracking applications (e.g., natural feature tracking (NFT), hand tracking, etc.) are depicted. A user 602 of the eyewear device 100 is present in the example physical environment 600 (in Figure 6, the example physical environment is an interior room). The processor 432 of the eye-mounted device 100 uses the captured image data to determine the positioning of the eye-mounted device 100 relative to one or more physical objects 604 within the environment 600, constructs a mapping of the environment 600 using a coordinate system of the environment 600 (e.g., a Cartesian coordinate system (x, y, z)), and determines the positioning relative to the coordinate system. In addition, the processor 432 determines the head posture (roll, pitch, and yaw) of the eye-mounted device 100 within the environment by using two or more location points (e.g., three location points 606a, 606b, and 606c) associated with a single object 604a, or by using one or more location points 606 associated with two or more objects 604a, 604b, 604c. The processor 432 of the eye-mounted device 100 can position a virtual object 608 (such as Figure 6 The key (shown) is positioned within environment 600 for viewing during the augmented reality experience.

[0104] In some examples, the positioning system 915 includes a virtual marker 610a associated with a virtual object 608 in the physical environment 600. In an augmented reality environment, in some embodiments, markers are registered at locations in the physical environment 600 to assist the electronic device in the task of tracking and updating the location of users, devices, and objects (virtual and physical) relative to the physical environment. Markers are sometimes registered to high-contrast physical objects, such as relatively dark objects mounted on a light-colored wall, such as a framed picture 604a, to assist cameras and other sensors in the task of detecting markers. Markers can be assigned and registered in memory by an eye-mounted device 100 operating within the environment. In some embodiments, markers are assigned and registered in memory of other devices in the network.

[0105] The positioning system 915 tracks physical objects and virtual objects within the physical environment 600 relative to the eye-mounted device 100. For physical objects 604 (e.g., safe 604c), the positioning system 915 continuously analyzes captured images of the physical environment 600 to identify the object 604 and determine its position relative to the eye-mounted device 100 (e.g., by applying a SLAM algorithm). The positioning system 915 maintains and updates the determined position information of the physical object 604 in a memory, thereby tracking the physical object 604 as the eye-mounted device 100 moves in the physical environment 600. For virtual objects 608 (e.g., keys), the positioning system 915 establishes or specifies an initial position for the virtual object 608 corresponding to a position in the environment 600 or a physical object 604 (or, in some embodiments, a position relative to the eye-mounted device 100). The positioning system 915 maintains and updates the position information of the virtual object 608 in response to the movement of the eye-mounted device 100 in the environment or a combination thereof, such as according to a movement algorithm associated with the virtual object 608, thereby tracking the virtual object 608 as the eye-mounted device 100 moves in the environment.

[0106] The marker may be encoded with information or otherwise linked to information. The marker may include location information, a physical code (such as a barcode or QR code; visible or hidden to the user), or a combination thereof. A set of data associated with the marker is stored in the memory 434 of the eye-wearing device 100. The set of data includes information about the marker 610a, the location (position and orientation) of the marker, one or more virtual objects, or a combination thereof. The marker location may include three-dimensional coordinates of one or more marker landmarks 616a, such as Figure 6 The corners of the roughly rectangular marker 610a shown in . The marker location can be expressed relative to real-world geographic coordinates, a marker coordinate system, the positioning of the eye-mounted device 100, or other coordinate system. The one or more virtual objects associated with the marker 610a can include any of a variety of materials, including still images, videos, audio, tactile feedback, executable applications, interactive user interfaces and experiences, and combinations or sequences of such materials. In this case, any type of content that can be stored in memory and retrieved when the marker 610a encounters an assigned tag or is associated with an assigned tag can be classified as a virtual object. For example, Figure 6 The virtual key 608 shown in FIG. 6 is a virtual object displayed as a 2D or 3D still image at a marked location.

[0107] In one example, the marker 610a can be registered in memory as being located on the physical object 604a (e.g., Figure 6 In another example, the marker can be registered in memory as a specific location relative to the eye-wearing device 100.

[0108] Figure 7 700 is a flow chart of an example method for presenting a medical image overlay 800 on a display 182B of an eye-mounted device 100. Although the steps are described with reference to the eye-mounted device 100 described herein, those skilled in the art will appreciate other implementations of the steps based on the description herein for use with other types of devices. One or more of the steps shown and described may be performed simultaneously, sequentially, in an order different from that shown and described, or in combination with additional steps. Certain steps may be omitted or repeated in certain applications.

[0109] In some embodiments, the overlay application 910 described herein is launched in response to receiving a selection through a user interface (e.g., selection from a menu, pressing a button, using a touchpad) or through some other input method (e.g., a gesture detected in a captured image, a finger touch 681 on the touchpad 181, a voice command).

[0110] Figure 7 Block 702 in the example method steps of capturing video frame data 900 with the cameras 114A, 114B of the eye-mounted device 100 is described. In some embodiments, the process of capturing video frame data 900 is ongoing during active use of the eye-mounted device 100. In other examples, the capture process begins in response to a selection received through a user interface or through some other input method. In some embodiments, at block 702, the example method steps include at least temporarily storing the captured video frame data 900 in the memory 434 on the eye-mounted device 100 so that the video frame data 900 can be used for purposes including processing and analysis.

[0111] Figure 7 Block 704 in FIG. 1 depicts an example step of storing a virtual marker 801 associated with an object of interest (e.g., a bone or tooth) in the physical environment 600. In some implementations, the process of storing the virtual marker 801 includes the processor of the eyewear device 100 associating the virtual marker 801 with a marker position 790 in the physical environment 600 (as opposed to a position on the display 182B). In this regard, the virtual marker position 790 is relative to the physical environment (e.g., relative to the physical coordinate system 610, such as Figure 8C In this example, the marker position 790 is fixed relative to the surrounding physical environment 600 and is independent of the movement of the display 182B or the eye-mounted device 100 in the physical environment 600.

[0112] Figure 8C182B. In some embodiments, the virtual markers 801A, 801B include the marker locations 790A, 790B and virtual indicators 802A, 802B presented near the virtual markers 801A, 801B. In some embodiments, the virtual indicators 802A, 802B include highlights (e.g., Figure 8C 182B) or a mark (e.g., visible, audible, or tactile) that is selectively presented on the display 182B to facilitate identification of the virtual markers 801A, 801B. In some embodiments, the virtual indicators 802A, 802B include a default element or shape that can be selected or changed through a user interface. Figure 8C As shown, in some embodiments, the marker positions 790A, 790B are located near the virtual markers 801A, 801B, but not necessarily centered. In some embodiments, the virtual markers 801A, 801B do not include virtual indicators 802A, 802B or any other type of indicators.

[0113] In some embodiments, the process of storing the virtual marker 801 includes presenting a virtual indicator 802 on the display 182B (e.g., at or near the center of the display 182B) as a visual guide. In this example, the user can place the virtual indicator 802 on or near any trial location of the marker location 790 (e.g., near a specific tooth). Then, in this example storage process, the overlay application 910 receives input (e.g., a finger tap 681 on the touchpad 181, a voice command, or a gesture) indicating the location to place and store the virtual marker 801. In other embodiments, the process of storing the virtual marker 801 includes receiving a voice command or other input indicating a specific location (e.g., tooth number 14, occlusal surface) relative to the physical environment 600 (e.g., the patient's mouth), which has been mapped and stored in the memory. In some embodiments, the process of storing the virtual marker 801 includes using the touchpad 181 to control the positioning of a cursor (not shown) that is then presented on the display 182B, so that the user can place the cursor at the current location (e.g., near a specific tooth), and perform a selection action and store the virtual marker 801 at the current location.

[0114] In some embodiments, the eye-mounted device 100 includes a speech recognition module 925 as described herein and a microphone 139 coupled to a speaker 191. In some embodiments, the speech recognition module 925 configures the processor 432 to perceive human speech, convert the received speech into audio frame data, recognize a first query based on the converted audio frame data, and then perform an action in response to and according to the recognized first query. For example, the human speech may include a verbal command (e.g., "set marker location"), and in response, the recognized first query causes the overlay application 910 to perform a selection action and store the virtual marker 801 at the current location.

[0115] In some embodiments, the process of storing the virtual marker 801 includes presenting a virtual indicator 802 on the display 182B after the selection has been entered. If the selected marker position 790 (as shown by the virtual indicator 802) is acceptable, the user can accept the position 790; if not, the user can cancel and select again.

[0116] Block 706 depicts an example step of registering a medical image 810 associated with the target 50 , wherein the medical image 810 includes an image location 812 relative to the marker location 790 .

[0117] As used herein, the term "medical image" refers to and includes information or data collected by any of a variety of imaging techniques, whether or not applied in a medical context, including but not limited to radiography (X-ray), computed tomography (CT), magnetic resonance imaging (MRI), ultrasound, endoscopy, elastography, tactile imaging, thermal imaging, functional imaging, positron emission tomography (PET), single photon emission computerized tomography (SPECT), and various types of photography.

[0118] As used herein, the term "target" refers to and includes an object, material, substance, structure, tissue, or cell, and any portion or piece of such an object. The target may be biological in nature (e.g., human or animal tissue, such as bone or tooth). The term "target of interest" refers to and includes a target, and may include an area, site, or volume of space near or around the target.

[0119] Although overlay system 400 and overlay application 910 are described herein in the context of medical images and biological objects, the described techniques may be applied to substantially any type of activity or endeavor in which image overlay is a useful or desired tool.

[0120] Fig. 8A In capturing one or more medical images 810 (such as Figure 8B4. An illustration of a medical film 42 about an object of interest (e.g., upper teeth 50U and lower teeth 50L) during an example procedure (shown in FIG. 4 ). The upper teeth 50U are characterized by an upper tooth axis 54U. The lower teeth 50L are characterized by a lower tooth axis 54L. In this example, the medical film 42 is an occlusal film that includes a film holder 40 placed between the teeth 50U, 50L. The occlusal film 42 includes an upper film portion extending along an upper film axis 814U and a lower film portion extending along a lower film axis 814L. The x-ray beam 30 is directed toward the film 42 along a beam direction 34.

[0121] Figure 8B Included are illustrations of an example occlusal image 810, an upper periapical image 810U, and a lower periapical image 810L. As shown, in some embodiments, the periapical images 810U, 810L include the full length of the tooth from crown to root.

[0122] Referring again to block 706, in some embodiments, the medical image 810 includes an image position 812 relative to the marker position 790. For example, Figure 8C As shown, one of the example virtual markers 801A is associated with a marker location 790A defined (e.g., in three dimensions) relative to the physical environment 600. The image location 812 includes three-dimensional coordinates and, in some embodiments, a vector associated with the medical image 810. In this example, the vector may define a plane (e.g., ten degrees relative to the marker location 790) and a distance (e.g., a length of forty millimeters) such that the orientation and size of the medical image 810 is part of the stored image location 812.

[0123] Figure 8C The marked position 790A shown in FIG. 8 is located near the bottom teeth. In some embodiments, the medical image position 812 is defined relative to the marked position 790. For example, Fig.8D As shown, the lower medical image 810L includes relative to one or more marker locations 790A, 790B (eg, Figure 8C In this regard, the image locations 812U, 812L are defined relative to one or more marked locations 790A, 790B in the physical environment 600, or are otherwise permanently associated with them.

[0124] In some embodiments, the example step of registering the medical image 810 at block 706 includes applying or otherwise using an image registration tool 912 as described herein. The process of image registration includes transforming one or more sets of data (e.g., medical image data) having its own coordinates (e.g., position, orientation, size) to another coordinate system (e.g., a physical coordinate system 610 associated with the physical environment 600). In some embodiments, the image registration tool 912 applies linear transformation models and vector mathematics to transform the position, orientation, and size of the medical image 810 so that when the medical image 810 is presented relative to the physical environment 600 (e.g., on the display 182), its position, orientation, and size appear relatively accurate.

[0125] In some embodiments, the process of registering the medical image 810 includes establishing an image position 812 relative to the target 50 (e.g., relative to the marker position 790 closest to the target 50) so that the registered medical image 810 will be presented as an overlay in the foreground relative to the target 50. Fig. 8A , for example, the medical film 42 is located behind the target 50 (e.g., in the background relative to the upper teeth 50U). If the upper image position 812U is established at or near the location of the medical film 42 (e.g., behind the teeth 50U), the resulting medical image 810U will be presented in the background (e.g., behind the upper teeth 50U), where the medical image 810U will not be visible. Therefore, in order to present the medical image overlay 800 (e.g., the medical image 810u) in the foreground, in some embodiments (e.g., using the image registration tool 912), the process of registering the medical image 810U includes establishing the image position 812U in the foreground relative to the target 50 (e.g., in front of the upper teeth 50U).

[0126] Block 707 depicts an example step of registering the medical image 810 by transforming the medical image 810 according to the physical coordinate system 610. The step at block 707 includes establishing the physical coordinate system 610 relative to the marker location 790, such as Figure 8C As shown. In some embodiments, the process of transforming the medical image 810 includes translation, rotation, and scaling. For example, applying a linear transformation model generally involves adjusting translation (e.g., the position of the medical image 810 defined by two-dimensional or three-dimensional coordinates), rotation (e.g., the angle of the medical image 810), and scaling (e.g., the size of the medical image 810) relative to the physical coordinate system 610.

[0127] Block 708 describes an example step of estimating the eyewear device position 840 relative to the marker position 790 (e.g., where the virtual marker 801 and the selected tooth are located). After the marker registration process, as the eyewear device 100 moves in the physical environment 600, its position changes relative to the marker position 790. In some embodiments, the current electronic eyewear device position 840 is estimated using the positioning system 915 described herein.

[0128] In some embodiments, the positioning system 915 on the eye-mounted device 100 configures the processor 432 of the eye-mounted device 100 to obtain positioning data based on the video frame data 900 captured from the cameras 114A, 114B, and in some embodiments, based on the motion data collected by the IMU 472. In some embodiments, the positioning system 915 uses a SLAM algorithm to build a virtual map of one or more objects within the camera field of view 904, as described herein, and updates the map and the location of the objects at least as frequently as the cameras 114A, 114B capture the video data 900. Frequent analysis of the high-frequency video data 900 helps detect relatively subtle movements of the eye-mounted device 100 over time.

[0129] In some embodiments, the step of estimating the electronic eyewear device position 840 relative to the marker position 790 includes calculating a correlation between the marker position 790 and the current electronic eyewear device position 840. The term correlation refers to and includes one or more vectors, matrices, formulas, or other mathematical expressions sufficient to define a three-dimensional distance between the marker position 790 and the current electronic eyewear device position 840. The current electronic eyewear device position 840 is associated with the three-dimensional position and orientation (e.g., head pose, gaze direction) of the display 182 because the display 182 is supported by the frame of the eyewear device 100. In this regard, the correlation process performs the function of calibrating the movement of the eyewear device 100 with the marker position 790. Because the positioning process occurs frequently, the correlation process between the eyewear device position 840 and the marker position 790 produces accurate and near real-time tracking of the current electronic eyewear device position 840 relative to the marker position 790.

[0130] In some embodiments, the process of estimating the current electronic eyewear device position 840 is based on motion frame data captured using the IMU 472, or based on video frame data 900 captured using the camera 114A coupled to the eyewear device 100, or a combination of both. In some embodiments, the process of estimating the current electronic eyewear device position 840 is performed at a frequency approximately as high as the frequency at which the IMU 472 captures motion data (e.g., 100 times per second based on an IMU sampling rate (samples per second) of 100 Hz). In some embodiments, the process of estimating the current electronic eyewear device position 840 occurs at a predefined and configurable frequency, and the IMU 472 is configured to capture motion frame data at a compatible rate.

[0131] Figure 7 Block 710 in 182B describes example steps of presenting a medical image overlay 800 at an image location 812 based on a current electronic eyewear device location 840 on the display 182B. The medical image overlay 800 is presented at an image location 812 associated with the medical image 810. In some embodiments, the medical image overlay 800 includes one or more registered medical images 810 (e.g., as described in block 706). In some embodiments, the medical image overlay 800 includes a portion, part, or segment of one or more registered medical images 810.

[0132] like Fig.8D As shown, the medical image overlay 800a in this example includes a lower medical image 810L presented at an image position 812L and an upper medical image 810U presented at an image position 812U. As described herein, the medical images 810L, 810U have been registered with the physical environment 600. Fig.8D As shown, many real objects (e.g., mouth and other teeth) in physical environment 600 can be seen through the semi-transparent lens assembly and display 182B. The process of registration and rendering produces a medical image overlay 800a that closely matches the position, orientation, and size of the real objects. For example, Fig.8D As shown, the medical image 810L closely matches the position, orientation, and size of a real object (e.g., four real lower teeth that are the subject of the medical image 810L). In this regard, the medical image overlay 800a provides an accurate view of the normally invisible features of the target (e.g., the dentin, pulp chamber, alveolar bone, and roots of the four lower teeth of the subject) by presenting the medical image 810L in the proper position and scale.

[0133] In some embodiments, the process of presenting the medical image overlay 800 includes presenting the medical images 810L, 810U as an overlay relative to the physical environment 600 such that the medical image overlay 800 is always visible in the foreground relative to real objects in the physical environment. Fig.8D As shown, the medical image overlay 800a is presented as an overlay (eg, in the foreground) relative to the physical environment 600 (eg, in front of the teeth) such that the medical image overlay 800 is always visible relative to other objects in the physical environment 600.

[0134] Fig. 8E 8 is another example medical image overlay 800b, which includes a lower medical image 810L presented at an image location 812L. Fig.8D 8, the eyewear device position 840 has changed compared to the position 840 shown in FIG. 8. As described herein, the medical image 810L has been registered with the physical environment 600. The image position 812L (e.g., position, orientation, and scale) is changed in FIG. Fig. 8E In and Fig.8D Therefore, relative to Fig.8D In its presentation, the medical image 810L is presented on the display 182B at a larger angle orientation (and slightly different scale). In this regard, the image registration tool 912 and the positioning system 915 (e.g., determining the current eye-mounted device position 840) work together to present a medical image overlay 800a that closely matches the position, orientation, and size of the real object (e.g., the four real lower teeth that are the subject of the medical image 810L) from any angle.

[0135] In some embodiments, the process of presenting the medical image overlay 800 includes providing one or more tools to the user to adjust or otherwise configure the medical image overlay 800, as described herein, or to selectively start and stop presentation. For example, the user can pause or stop (or restart) the process of presenting the medical image overlay 800 by speaking a voice command, pressing a button on the eye-wearing device 100, performing a hand gesture, or tapping a finger touch 681 on the touchpad 181.

[0136] In some embodiments, the process of presenting the medical image overlay 800 includes moving or otherwise adjusting the image position 812 relative to the target 50 so that the medical image 810 will be presented as an overlay in the foreground relative to the target 50 when registered. In this process, in some embodiments, the overlay application 910 estimates the image position 812 (e.g., the position at which the medical image overlay 800 is presented) relative to the target 50 (e.g., one or more teeth) based on the one or more marker positions 790. If all or part of the medical image overlay 800 is estimated to be in the background relative to the target (e.g., behind the teeth), the overlay application 910 moves or otherwise adjusts the image position 812 so that the medical image 810 will be presented in the foreground relative to the target 50. In some embodiments, the overlay application 910 automatically moves or otherwise adjusts the image position 812 without user input.

[0137] In some embodiments, the image position 812 is configurable such that the process of presenting the medical image overlay 800 includes providing the user with one or more tools to adjust or otherwise configure the image position 812. In some embodiments, the process of adjusting the image position 812 includes calculating or adjusting the image position 812 based on the length and heading of the segment 521 traversed by the finger along the touch pad (e.g., Fig.8D ). In some embodiments, the process of adjusting the image position 812 includes identifying an original value of the image position 812 (e.g., established when registering the medical image at block 706). In some embodiments, the process of adjusting the image position 812 includes calculating a new image position 812 based on the length and heading of a segment 521 (e.g., the longer the segment 521, the greater the change in image position 812). In some embodiments, the image position 812 includes a sign (positive or negative) based on the heading of the segment 521 (e.g., moving the image position 812 toward the foreground in response to a heading toward the front of the eye-wearing device 100; moving toward the background in response to a heading toward the ear). In some embodiments, the process of adjusting the image position 812 occurs in real time so that a viewer can see the change in the image position 812 as the adjustment is made and stops when the desired new image position 812 is reached (e.g., lifting a finger or performing a tap 681 on the touch pad 181).

[0138] Figure 7 Block 712 in FIG. 8 depicts example steps for presenting the medical image overlay 800 based on the field of view 904 associated with the camera 114B. In some embodiments, Fig.8D As shown, the camera field of view 904 is larger than the display 182B. In this example rendering step, when the image positions 812L, 812U are detected within the field of view 904 of the camera 114B (e.g., based on the captured video frame data 900), the medical image overlay 800a appears. Fig.8D If the eyewear device 100 is moved very close to the upper teeth so that the field of view 904 includes the upper image position 812U, the upper medical image 810U will be presented on the display 182B. On the contrary, if the field of view 904 does not include the upper image position 812U, the upper medical image 810U will not be presented on the display 182B.

[0139] In some embodiments, the medical image overlay 800 includes multiple medical images (e.g., 810L, 810U, and in some cases, many other images). In practice, the example steps of presenting the medical image overlay 800 based on the field of view 904 mean that the medical images will appear (or disappear) as the field of view 904 changes (e.g., when the eye-mounted device 100 moves in the physical environment 600).

[0140] Figure 7 Block 714 in 182B describes an example step of selectively adjusting a transparency value 820 associated with the medical image overlay 800. The transparency value 820 affects how much of the real object is visible when the medical image overlay 800 is presented on the display 182B (e.g., in the foreground relative to the real object). In some embodiments, the transparency value 820 is predefined (e.g., 10% transparent, equivalent to 90% opaque).

[0141] In some embodiments, the transparency value 820 is configurable, such that the process of presenting the medical image overlay 800 includes providing the user with one or more tools to adjust or otherwise configure the transparency value 820. In some embodiments, the process of adjusting the transparency value 820 includes calculating or adjusting the transparency value 820 based on the length and heading of the finger along the path 521 traversed by the touch pad (e.g., Fig.8D As shown). In some embodiments, the process of adjusting the transparency value 820 includes identifying an original value of the transparency value 820 (e.g., a default value or a predefined value). In some embodiments, the process of adjusting the transparency value 820 includes calculating a new transparency value 820 based on the length and heading of -521 (e.g., the longer -521, the greater the change in transparency). In some embodiments, the transparency value 820 includes a sign (positive or negative) based on the heading of -521 (e.g., increasing transparency in response to a heading toward the front of the eye-wearing device 100; decreasing transparency for a heading toward the ear). In some embodiments, the process of adjusting the transparency value 820 occurs in real time so that the viewer can see the change in transparency as the adjustment is made, and stops when the desired transparency value 820 is reached (e.g., lifting a finger or performing a tap 681 on the touch pad 181).

[0142] In some embodiments, the eye-mounted device 100 includes a speech recognition module 925 as described herein and a microphone 139 coupled to the speaker 191. In some embodiments, the speech recognition module 925 configures the processor 432 to perceive human speech, convert the received speech into audio frame data, recognize a first query based on the converted audio frame data, and then perform an action in response to and according to the recognized first query. For example, the human speech may include a verbal command (e.g., "transparency 70%"), in response to which the recognized first query causes the overlay application 910 to adjust the transparency value 820 as described herein.

[0143] Any functionality of the eye-wearing device 100, mobile device 401, and server system 498 described herein may be embodied in one or more computer software applications or programming instruction sets, as described herein. According to some examples, a "function," "multiple functions," "application," "multiple applications," "instructions," "multiple instructions," or "programming" is a program that executes the functions defined in a program. Various programming languages ​​may be employed to generate one or more applications structured in various ways, such as an object-oriented programming language (e.g., Objective-C, Java, or C++) or a procedural programming language (e.g., C or assembly language). In a specific example, a third-party application (e.g., an application created using ANDROID ® by an entity other than the vendor of a particular platform) may be used to create a program that executes the functions defined in the program. TM or IOS TM Applications developed with a software development kit (SDK) can be run on mobile operating systems such as IOS TM ANDROID TM , In this example, the third-party application can call API calls provided by the operating system to facilitate the functions described in this article.

[0144] Therefore, machine-readable media can take the form of tangible storage media in a variety of forms. Non-volatile storage media include, for example, optical or magnetic disks, such as any storage device in any (one or more) computers or the like, such as client devices, media gateways, transcoders, etc. shown in the implementation drawings. Volatile storage media include dynamic memory, such as the main memory of such a computer platform. Tangible transmission media include coaxial cables; copper wires and optical fibers, including wires that contain buses within a computer system. Carrier transmission media can take the form of electrical or electromagnetic signals, or sound or light waves, such as those generated during radio frequency (RF) and infrared (IR) data communications. Therefore, common forms of computer-readable media include, for example: floppy disks, floppy disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, DVDs or DVD-ROMs, any other optical media, punched card tapes, any other physical storage media with hole patterns, RAMs, PROMs and EPROMs, FLASH-EPROMs, any other memory chips or cassettes, carriers that transmit data or instructions, cables or links that transmit such carriers, or any other media from which a computer can read at least one programming code or data. Many of these forms of computer readable media may be involved in carrying one or more sequences of one or more instructions to a processor for execution.

[0145] Except as stated immediately above, nothing that has been stated or shown is intended or should be construed as causing any component, step, feature, object, benefit, advantage, or equivalent to be dedicated to the public, whether or not it is stated in the claims.

[0146] It will be understood that the terms and expressions used in this article have the common meanings given by such terms and expressions relative to their corresponding respective explorations and research fields, unless the specific meanings thereof have been set forth in addition herein. Relational terms (such as first and second and the like) can be used only to distinguish an entity or action from another entity or action, without necessarily requiring or implying any actual such relationship or order between such entities or actions. The terms "include", "comprise", "contain", "cover" or any other variation thereof are intended to cover non-exclusive inclusion, so that the process, method, article or equipment including or including a series of elements or steps not only include those elements or steps, but also may include other elements or steps that are not explicitly listed or inherent to such processes, methods, articles or equipment. In the absence of further constraints, an element starting with "one" or "an" does not exclude the presence of additional identical elements in the process, method, article or equipment including the element.

[0147] Unless otherwise indicated, any and all measurements, values, ratings, positions, amplitudes, dimensions, and other specifications set forth in this specification (including the claims that follow) are approximate and not exact. These quantities are intended to have a reasonable range that is consistent with the functions to which they are related and with the customary practices in the art to which they belong. For example, unless otherwise expressly stated, parameter values ​​or similar values ​​may vary from the stated quantity by as much as ±10%.

[0148] In addition, in the foregoing detailed description, it can be seen that, for the purpose of streamlining the present disclosure, individual features are grouped together in various examples. This method of disclosure should not be interpreted as reflecting an intention that the claimed examples require more features than those explicitly stated in each claim. On the contrary, as reflected in the following claims, the subject matter to be protected lies in less than all the features of any single disclosed example. Therefore, the following claims are hereby incorporated into the detailed description, with each claim standing alone as a separate claimed subject matter.

[0149] Although what is considered to be the best mode and other examples have been described above, it is to be understood that various modifications may be made therein and that the subject matter disclosed herein may be implemented in various forms and examples and that they may be applied to numerous applications, only some of which have been described herein. The following claims are intended to claim any and all modifications and variations that fall within the true scope of the present concept.

Claims

1. A method for presenting a medical image overlay using an eye-mounted device, the eye-mounted device comprising a camera and a display, the method comprising: capturing video frame data using the camera; storing a virtual marker, the virtual marker comprising a marker location relative to a physical environment; registering a medical image associated with the target, wherein the medical image includes an image position relative to the marker position; determining a position of the eye-mounted device relative to the marker position based on the captured video frame data; as well as A medical image overlay is presented on the display at the image location based on the eye-mounted device location, wherein the medical image overlay includes a registered medical image.

2. The method of claim 1, wherein the medical image overlay is associated with a transparency value, and wherein the step of presenting the medical image overlay further comprises: The medical image overlay is generated based on the transparency value.

3. The method of claim 2, wherein the transparency value is selectively configurable, and wherein the step of presenting the medical image overlay further comprises: detecting a finger traversing a segment along a touch pad coupled to the eye-mounted device; adjusting the transparency value based on the detected segment; as well as The adjusted transparency value is applied to the medical image overlay.

4. The method of claim 1 , wherein the marker positions are defined relative to a physical coordinate system, and wherein the step of registering the medical image further comprises: The medical image is transformed according to the physical coordinate system, wherein the transformation comprises one or more of translation, rotation and scaling.

5. The method of claim 1, wherein the medical image is associated with a film located in the background relative to the target, and wherein the step of registering the medical image further comprises: The image position relative to the target is established such that when presented at the image position, the medical image overlay will appear as an overlay in the foreground relative to the target.

6. The method according to claim 1, wherein: The step of presenting the medical image overlay also includes: The image position relative to the target is adjusted so that when presented at the image position, the medical image overlay will appear as an overlay in the foreground relative to the target.

7. The method according to claim 1, wherein: The medical image overlay comprises a plurality of medical images including a first medical image associated with a first image location, wherein the camera defines a field of view, and wherein the step of presenting the medical image overlay further comprises: determining, based on the captured video frame data, whether the first image position is detected within the field of view; and The first medical image is selectively presented in response to detecting the first image location.

8. The method of claim 7, wherein the plurality of medical images comprises a subsequent medical image associated with a subsequent image location, and wherein the step of presenting the medical image overlay further comprises: determining, based on the captured video frame data, whether the subsequent image position is detected within the field of view; In response to failure to detect the first image location within the field of view, inhibiting presentation of the first medical image; as well as In response to detecting the subsequent image location, the subsequent medical image is selectively presented.

9. An eye-worn device for presenting a medical image overlay, comprising: processor; Memory; camera; monitor; as well as A program in the memory, wherein execution of the program by the processor configures the eye-wearing device to perform functions, including functions for: capturing video frame data using the camera; storing a virtual marker, the virtual marker comprising a marker location relative to a physical environment; registering a medical image associated with the target, wherein the medical image includes an image position relative to the marker position; determining a position of the eye-mounted device relative to the marker position based on the captured video frame data; as well as A medical image overlay is presented on the display at the image location based on the eye-mounted device location, wherein the medical image overlay includes a registered medical image.

10. The eyewear device according to claim 9, wherein: The functionality for presenting the medical image overlay also includes functionality for: The medical image overlay is generated according to a transparency value.

11. The eyewear device according to claim 10, wherein: The functionality for presenting the medical image overlay also includes functionality for: detecting a finger traversing a segment along a touch pad coupled to the eye-mounted device; adjusting the transparency value based on the detected segment; and The adjusted transparency value is applied to the medical image overlay.

12. The eyewear device according to claim 9, wherein: The functionality for registering the medical images also includes functionality for: defining the position of the marker relative to a physical coordinate system; and The medical image is transformed according to the physical coordinate system, wherein the transformation function comprises one or more of translation, rotation and scaling.

13. The eye-mounted device of claim 9, wherein the medical image is associated with a film located in the background relative to the target, and wherein the function for registering the medical image further comprises a function for: The image position relative to the target is established such that when presented at the image position, the medical image overlay will appear as an overlay in the foreground relative to the target.

14. The eyewear device according to claim 9, wherein: The functionality for presenting the medical image overlay also includes functionality for: The image position relative to the target is adjusted so that when presented at the image position, the medical image overlay will appear as an overlay in the foreground relative to the target.

15. The eye-mounted device of claim 9, wherein the medical image comprises a plurality of medical images, including a first medical image associated with a first image location, wherein: The camera defines a field of view, and wherein the functionality for presenting the medical image overlay further comprises functionality for: determining whether the first image location is detected within the field of view based on the captured video frame data; and In response to detecting the first image location, the first medical image is selectively presented.

16. A non-transitory computer readable medium storing program code, which when executed is operable to cause a processor of an eye-wearing device to perform the following steps: capturing video frame data using a camera coupled to the eye-worn device; storing a virtual marker, the virtual marker comprising a marker location relative to a physical environment; registering a medical image associated with the target, wherein the medical image includes an image position relative to the marker position; determining, based on the captured video frame data, a position of the eye-mounted device relative to the marker position; and A medical image overlay is presented on a display of the eye-mounted device at the image location based on the eye-mounted device location, wherein the medical image overlay includes a registered medical image.

17. The non-transitory computer readable medium storing program code according to claim 16, wherein the program code when executed is operable to cause the processor to perform the following further steps: generating the medical image overlay according to the transparency value; detecting a finger traversing a segment along a touch pad coupled to the eye-mounted device; adjusting the transparency value based on the detected segment; as well as The adjusted transparency value is applied to the medical image overlay.

18. The non-transitory computer readable medium storing program code according to claim 16, wherein the program code when executed is operable to cause the processor to perform the following further steps: determining whether the medical image is associated with a film located in the background relative to the target; establishing the image position relative to the target so that when presented at the image position, the medical image overlay will appear as an overlay in the foreground relative to the target; as well as The image position relative to the target is adjusted so that when presented at the image position, the medical image overlay will appear as an overlay in the foreground relative to the target.

19. The non-transitory computer readable medium storing program code according to claim 16, wherein the program code when executed is operable to cause the processor to perform the following further steps: determining whether the medical image overlay includes a plurality of medical images including a first medical image associated with a first image location; determining, based on the captured video frame data, whether the first image location is detected within a field of view associated with the camera; and The first medical image is selectively presented in response to detecting the first image location.

20. The non-transitory computer readable medium storing program code according to claim 19, wherein the program code when executed is operable to cause the processor to perform the further steps of: determining whether the plurality of medical images further includes a subsequent medical image associated with a subsequent image location; determining, based on the captured video frame data, whether the subsequent image position is detected within the field of view; In response to failure to detect the first image location within the field of view, inhibiting presentation of the first medical image; as well as In response to detecting the subsequent image location, the subsequent medical image is selectively presented.

Images