Magnification overlay related to virtual markings

Abstract

A magnification application with an augmented reality (AR) eyewear device. The magnification application enables a user of the eyewear device to activate a magnification overlay on the display whenever a camera on the eyewear device detects that the field of view includes the registered virtual indicia. The magnified overlay includes one or more of the captured frames of video data presented according to a predefined and configurable magnification factor. A pointer including a vector and a visual tether directs the user toward the virtual indicia. When the eyewear device is located near the perimeter associated with the virtual indicia, the enlarged overlay appears in a predefined and configurable frame on the display.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. application serial number 17 / 876,334, filed on July 28, 2022, the contents of which are incorporated herein by reference in their entirety. Technical Field

[0003] Examples set forth in this disclosure relate to the field of augmented reality (AR) experiences for electronic devices, including wearable devices such as eyewear devices. More particularly, but not by way of limitation, this disclosure describes presenting a magnified overlay using an eyewear device that selectively appears when a virtual marker is within a field of view. Background Art

[0004] Many types of computers and electronic devices available today, such as mobile devices (e.g., smartphones, tablets, and laptops), handheld devices, and wearable devices (e.g., smart glasses, digital glasses, headwear, hats, and head-mounted displays), include various cameras, sensors, wireless transceivers, input systems, and displays. Users sometimes refer to information on these devices during physical activities (such as exercise).

[0005] Virtual reality (VR) technology generates a complete virtual environment including realistic images, sometimes presented on a VR headset or other head-mounted display. The VR experience allows the user to move through the virtual environment and interact with virtual objects. AR is a type of VR technology that combines real objects in the physical environment with virtual objects and displays the combination to the user. The combined display gives the impression that the virtual objects are actually present in the environment, especially when the virtual objects appear and behave like real objects. Cross reality (XR) is generally understood as a general term referring to systems that include or combine elements from AR, VR and MR (mixed reality) environments.

[0006] Automatic speech recognition (ASR) is a field of computer science, artificial intelligence, and linguistics that involves receiving spoken words and converting them into audio data suitable for processing by a computing device. The processed audio data frames can be used to translate the received spoken words into text or to convert the spoken words into commands for controlling and interacting with various software applications. ASR processing can be used by computers, handheld devices, wearable devices, phone systems, automobiles, and a variety of other devices to facilitate human-computer interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The features of the various examples described will be readily understood from the following detailed description with reference to the accompanying drawings. Reference numerals are used for each element in the description and the reference numerals are used throughout the multiple views of the accompanying drawings. When multiple similar elements are present, a single reference numeral may be assigned to the similar elements with an additional uppercase or lowercase letter to refer to the specific element.

[0008] The various elements shown in the figures are not drawn to scale unless otherwise noted. The sizes of the various elements may be enlarged or reduced for clarity. The various figures depict one or more embodiments and are presented by way of example only and should not be construed as limiting. The drawings include the following figures:

[0009] Figure 1A is a side view of an eyewear device suitable for use in an example virtual guided fitness system (right);

[0010] Figure 1B yes Figure 1A A perspective, partially cross-sectional view of the optical components and electronics in a portion of an eyewear device shown in FIG.

[0011] Figure 1C yes Figure 1A Side view of the eyewear device (left);

[0012] Figure 1D yes Figure 1C A perspective, partially cross-sectional view of the optical components and electronics in a portion of an eyewear device shown in FIG.

[0013] Figure 2A and Figure 2B is a rear view of an eyewear device utilized in an example virtual guided fitness system;

[0014] Figure 3 is a block diagram illustrating an example of capturing visible light using an example eyewear device shown in any of the subsequent figures;

[0015] Figure 4 is a functional block diagram of an example demonstration system including an eyewear device and a server system connected via various networks;

[0016] Figure 5 is a diagrammatic representation of an example hardware configuration for a mobile device that is configured to Figure 4 The example demo system is used;

[0017] Figure 6 is a user's perspective in an example environment for describing simultaneous localization and mapping (SLAM);

[0018] Figure 7is a flow chart listing steps in an example method of presenting a magnification overlay on a display;

[0019] Fig. 8A is a perspective view of an example virtual marker presented on a display;

[0020] Figure 8B are perspective views of example frames and frame positioning relative to a display;

[0021] Figure 8C is a perspective view of an example pointer presented on a display;

[0022] Fig.8D is a perspective view of an example magnified overlay presented on a display; and

[0023] Fig. 8E is a perspective view of another example of a magnification overlay presented on a display. DETAILED DESCRIPTION

[0024] A magnification application utilizing an AR eyewear device. The application enables a user of the eyewear device to view a magnification overlay during close work (e.g., dental work, surgery) based on a virtual marker (e.g., placed on a particular tooth or structure). When the virtual marker is within the field of view, the magnification overlay appears, thereby providing a close, detailed view of the structure, facilitating freedom of movement, and otherwise optimizing the user's performance.

[0025] Various embodiments and details are described with reference to an example for presenting a magnified overlay in an augmented reality environment. In an example embodiment, a method relates to an eyewear device having a camera and a display and includes: registering a virtual marker associated with a marker position relative to a physical environment, determining a current eyewear device position relative to the marker position based on a video data frame captured using the camera, and presenting a magnified overlay on the display based on the marker position and the current eyewear device position. The magnified overlay includes one or more of the captured video data frames, which are presented in a defined frame on the display according to the magnification factor. The magnified overlay can be adjusted and controlled using voice commands, gestures, or input from a touchpad.

[0026] In another example embodiment, the method includes defining a perimeter relative to the virtual marker position, comparing a frame location on the display to the perimeter based on a current eyewear device position, and selectively presenting a magnification overlay whenever the frame location is within the perimeter.

[0027] Another example implementation includes presenting a pointer on a display to guide a user toward a virtual marker. The pointer includes a vector arrow, a visual tether leading to the virtual marker, and a range value.

[0028] Although various systems and methods are described herein with reference to dental procedures, the techniques described herein can be applied to substantially any type of close work in which magnification is desired. For example, magnification is a desired tool for activities such as dentistry, surgery, gemology, watchmaking, jewelry trade, photography, geology, antiquities inspection, and archival preservation.

[0029] The following detailed description includes systems, methods, techniques, instruction sequences, and computer program products that illustrate the examples set forth in this disclosure. Many details and examples are included for the purpose of providing a thorough understanding of the disclosed subject matter and its related teachings. However, those skilled in the relevant art can understand how to apply the related teachings without such details. The aspects of the disclosed subject matter are not limited to the specific devices, systems, and methods described, because the related teachings can be applied or practiced in a variety of ways. The terms and nomenclature used herein are only used for the purpose of describing specific aspects, and are not intended to be restrictive. In general, known instruction instances, protocols, structures, and techniques are not necessarily shown in detail.

[0030] As used herein, the terms "connect," "connected," "coupled," and "coupled" refer to any logical, optical, physical, or electrical connection, including links or the like, by which an electrical or magnetic signal generated or provided by one system element is imparted to another coupled or connected system element. Unless otherwise specified, coupled or connected elements or devices are not necessarily directly connected to each other and may be separated by intermediate components, elements, or communications media, one or more of which can modify, manipulate, or carry electrical signals. The term "on" means supported directly by an element or indirectly supported by an element through another element that is integrated into or supported by the element.

[0031] The term "proximal" is used to describe an item or a portion of an item that is located near, adjacent to, or beside an object or person; or that is closer relative to other portions of the item, which may be described as "distal." For example, the end of the item closest to the object may be referred to as the proximal end, while the generally opposite end may be referred to as the distal end.

[0032] The orientation of the eyeglass device, associated components, and any complete device incorporating an eye scanner and camera, such as shown in any of the accompanying drawings, is given by way of example only for purposes of illustration and discussion. In operation for a particular variable optical processing application, the eyeglass device may be oriented in any other direction suitable for the particular application of the eyeglass device, such as upward, downward, sideways, or any other orientation. In addition, as used herein, any directional terms, such as front, back, inward, outward, toward, left, right, lateral, longitudinal, up, down, above, below, top, bottom, and side, are used by way of example only and do not limit the direction or orientation of any optical element or component of an optical element constructed as described herein.

[0033] Advanced AR technologies, such as computer vision and object tracking, can be used to produce perceptually rich and immersive experiences. Computer vision algorithms extract three-dimensional data about the physical world from data captured in digital images or videos. Object recognition and tracking algorithms are used to detect objects in digital images or videos, estimate their orientation or pose, and track their movement over time. Real-time recognition and tracking of hands and fingers is a challenging and processing-intensive task in the field of computer vision.

[0034] In the context of computer vision, object recognition, and tracking, the term "pose" refers to the static position and orientation of an object at a particular moment in time. The term "gesture" refers to the active movement of an object (such as a hand) through a series of postures, sometimes used to convey a signal or idea. In the fields of computer vision and augmented reality, the terms pose and gesture are sometimes used interchangeably. As used herein, the terms "pose" or "gesture" (or variations thereof) are intended to include both poses and gestures; in other words, the use of one term does not exclude the other.

[0035] Additional objects, advantages and novel features of the embodiments will be set forth in part in the following description, and in part will become apparent to those skilled in the art upon examination of the following and accompanying drawings, or may be learned by production or operation of the examples. The objects and advantages of the subject matter may be realized and obtained by means of the methods, tools and combinations particularly pointed out in the appended claims.

[0036] In an example configuration, an eyewear device with augmented reality (AR) capabilities is used in the system described herein. It is desirable to use an eyewear device in the system described herein because such devices are scalable and customizable to enable personalized experiences, enable effects to be applied anywhere, and ensure user privacy by allowing the user to see only the transmitted information. In a sample configuration, an eyewear device such as the eyewear SPECTACLES available from Snap, Inc. of Santa Monica, California can be used without any specialized hardware. TM.

[0037] like Figures 1A to 1D As shown, the eyewear device 100 includes a first camera 114A and a second camera 114B. The cameras 114 capture image information for a scene from separate viewpoints. The captured images can be used to project a three-dimensional display onto an image display for three-dimensional (3D) viewing.

[0038] Cameras 114 are sensitive to wavelengths in the visible light range. Each of cameras 114 defines a different forward-facing field of view that overlaps to enable generation of a 3D depth image; for example, first camera 114A defines a first field of view 111A and second camera 114B defines a second field of view 111B. In general, a "field of view" is the portion of a scene visible through a camera at a particular location and orientation in space. Fields of view 111 have overlapping fields of view 304 ( Figure 3 ). Objects or object features outside the field of view 111 are not recorded in the original image (e.g., a photograph or picture) when the camera captures the image. The field of view describes the angular range or degree to which the image sensor of the camera 114 picks up electromagnetic radiation of a given scene in a captured image of the given scene. The field of view can be expressed as the angular size of the viewing cone; that is, the viewing angle. The viewing angle can be measured horizontally, vertically, or diagonally.

[0039] In an example configuration, one or both cameras 114 have a 100° field of view and a resolution of 480x480 pixels. The "coverage angle" describes the range of angles that the lens of the camera 114 can effectively image. Typically, a camera lens will produce an image circle large enough to completely cover the camera's film or sensor, possibly including some vignetting (e.g., the image gets darker toward the edges compared to the center). If the camera lens's coverage angle does not fill the sensor, the image circle will be visible, typically with strong vignetting toward the edges, and the effective viewing angle will be limited to the coverage angle.

[0040] Examples of suitable cameras 114 include high-resolution complementary metal-oxide-semiconductor (CMOS) image sensors and digital VGA cameras (video graphics arrays), which can be 480p (e.g., 640x480 pixels), 720p, 1080p, or higher resolution. Other examples include cameras 114 capable of capturing high-definition (HD) video at high frame rates (e.g., thirty to sixty frames per second, or more) and storing recordings at a resolution of 1216 by 1216 pixels (or greater).

[0041] The eyewear device 100 can capture image sensor data from the camera 114 together with the geolocation data digitized by the image processor to be stored in the memory. The camera 114 captures the corresponding original image (e.g., the left original image and the right original image) in a two-dimensional spatial domain, which includes a pixel matrix on a two-dimensional coordinate system, and the two-dimensional coordinate system includes an X-axis for horizontal positioning and a Y-axis for vertical positioning. Each pixel includes a color attribute value (e.g., a red pixel light value, a green pixel light value, or a blue pixel light value); and a positioning attribute (e.g., an X-axis coordinate and a Y-axis coordinate).

[0042] To capture stereoscopic images for later display as a 3D projection, image processor 412 ( Figure 4 ) may be coupled to the camera 114 to receive and store visual image information. The image processor 412 or another processor controls the operation of the camera 114 to act as a stereo camera that simulates human binocular vision and may add a timestamp to each image. The timestamps on each pair of images allow the images to be displayed together as part of a 3D projection. 3D projection can produce an immersive, lifelike experience, which is desirable in a variety of environments, including virtual reality (VR) and video games.

[0043] Figure 1B yes Figure 1A 1 is a perspective, cross-sectional view of the right corner 110A of the eyewear device 100 depicting the first camera 114A, additional optical components and electronics. Figure 1C yes Figure 1A 1 is a side view (left) of an example hardware configuration of the eyewear device 100 showing the second camera 114B of the camera system. Figure 1D yes Figure 1C 10B of the eyewear device 100, depicting a second camera 114B, additional optics, and electronics of the camera system.

[0044] like Figure 1B As shown in the example of FIG. 1 , the eyeglass device 100 includes a first camera 114A and a circuit board 140A, which may be a flexible printed circuit board (PCB). The first hinge 126A connects the right corner 110A to the first temple 125A of the eyeglass device 100. In some examples, the first camera 114A, the flexible PCB 140A, or other electrical connectors or contacts may be located on the first temple 125A or the first hinge 126A.

[0045] The right corner 110A includes a corner body 190 and a corner cap. Figure 1BThe corner cap is omitted in the cross section of FIG. Disposed inside the right corner 110A are various interconnect circuit boards 109 (such as a flexible PCB 140A), which include controller circuits for the first camera 114A, a microphone 139, a speaker 191, low-power wireless circuits (e.g., for communicating via Bluetooth TM for wireless short-range network communications), high-speed wireless circuits (for example, for wireless LAN communications via Wi-Fi).

[0046] The first camera 114A is coupled to or disposed on the flexible PCB 140A and is covered by a camera cover lens that is aimed through an opening formed in the frame 105. For example, Figure 2A As shown, the right rim 107A of the frame 105 is connected to the right corner 110A and includes an opening for the camera cover lens. The frame 105 includes a front side configured to face outward and away from the user's eyes. The opening for the camera cover lens is formed on the front side or outward-facing side of the frame 105 and passes through the front side or outward-facing side. In the example, the first camera 114A has an outward-facing field of view 111A (as shown in Figure 3), and its line of sight or viewing angle is related to the right eye of the user of the eyeglass device 100. The camera cover lens can also be adhered to the front side or outward-facing surface of the right corner 110A, with an opening having an outward-facing coverage angle formed therein, but in a different outward-facing direction. Coupling can also be performed indirectly via an intervening component.

[0047] like Figure 1D As shown, the eyeglass device 100 includes a second camera 114B and a circuit board 140B, which may be a flexible printed circuit board (PCB). A second hinge 126B connects the left corner 110B to a second temple 125B of the eyeglass device 100. In some examples, the second camera 114B, the flexible PCB 140B, or other electrical connectors or contacts may be located on the second temple 125B or the second hinge 126B.

[0048] The left corner 110B includes a corner body 190 and a corner cap. Figure 1D The corner cap is omitted in the cross section of FIG. 110A. Disposed inside the right corner 110A are various interconnect circuit boards 109 (such as a flexible PCB 140B) that include controller circuitry for the second camera 114B.

[0049] The camera 114 is coupled to or disposed on a corresponding flexible PCB 140 and is covered by a camera cover lens that is aimed through an opening formed in the frame 105. For example, Figure 2AAs shown, the right rim 107A of the frame 105 is connected to the right corner 110A and includes an opening for the camera cover lens, and the left rim 107B of the frame 105 is connected to the left corner 110B and includes an opening for the camera cover lens. The frame 105 includes a front side configured to face outward and away from the user's eyes. The opening for the camera cover lens is formed on the front side or the side facing outward of the frame 105 and passes through the front side or the side facing outward. In the example, the camera 114A has an outward-facing field of view 111 (as shown in Figure 3), and its line of sight or viewing angle is related to the corresponding eye of the user of the eyeglass device 100. The camera cover lens can also be adhered to the front side or outward-facing surface of the corresponding corner 110, with an opening with an outward-facing coverage angle formed therein, but in a different outward-facing direction. Coupling can also be performed indirectly via an intervening component.

[0050] Figure 2A and Figure 2B An example hardware configuration of the eyeglass device 100 is depicted, including two different types of image displays. The eyeglass device 100 is sized and shaped in a form configured to be worn by a user. The form of glasses is shown in the illustrated example. The eyeglass device 100 may take other forms and may incorporate other types of frames; for example, headwear, headphones, or a helmet.

[0051] In the eyeglass example, the eyeglass device 100 includes a frame 105 including a right rim 107A connected to a left rim 107B via a bridge 106, the bridge 106 being configured to receive the user's nose to support the eyeglass device 100 on the user's head. The right rim 107A includes a first hole 175A that holds a first optical element 180A. The left rim 107B includes a second hole 175B that holds a second optical element 180B. Figure 2B As shown in , each optical element 180A, 180B includes an integrated image display (e.g., a first display 182A and a second display 182B) in some embodiments. As used herein, the term "lens" is intended to include a transparent or translucent glass or plastic sheet having a curved or flat surface that causes light to converge or diverge or causes little or no convergence or divergence.

[0052] A touch-sensitive input device, such as a touch pad 181, is positioned on the first temple 125A. As shown, the touch pad 181 may have a clearly visible border or include a raised or otherwise tactile edge that provides feedback to the user about the location and border of the touch pad 181; alternatively, the border may be imperceptible and not easily seen or felt. The eyeglass device 100 may include a touch pad on the other side that operates independently or in conjunction with the touch pad 181.

[0053] The surface of touchpad 181 is configured to detect finger touches, taps, and gestures (e.g., moving touch) for use with a graphical user interface (GUI) displayed by the eyewear device on the image display to allow the user to browse and select menu options in an intuitive manner, which enhances and simplifies the user experience.

[0054] Detecting finger input on the touch pad 181 can implement multiple functions. For example, touching anywhere on the touch pad 181 can cause the GUI to display or highlight an item on the image display, which can be projected onto at least one of the optical components 180. Tapping or double-clicking on the touch pad 181 can select an item or icon. Sliding or sliding a finger in a particular direction (e.g., from front to back, from back to front, from top to bottom, or downward) can cause an item or icon to slide or scroll in a particular direction; for example, moving to the next item, icon, video, image, page, or slide. Sliding a finger in another direction can slide or scroll in the opposite direction; for example, moving to the previous item, icon, video, image, page, or slide. The touch pad 181 can be positioned substantially anywhere on the eyewear device 100.

[0055] In one example, a single tap of a recognized finger gesture on touch pad 181 initiates a selection or press of a GUI element in an image presented on the image display of optical assembly 180. Adjustments to the image presented on the image display of optical assembly 180 based on the recognized finger gesture may be the primary action of selecting or submitting a GUI element on the image display of optical assembly 180 for further display or execution.

[0056] Figure 2A 1 is an example hardware configuration for the eyeglass device 100, in which the right corner 110A supports a microphone 139 and a speaker 191. The microphone 139 includes a transducer that converts sound into a corresponding electrical audio signal. The microphone 139 in the example shown is positioned with an opening that faces inward toward the wearer to facilitate the reception of sound waves (such as human speech including verbal commands and questions). Additional or differently oriented openings may be implemented. In other example configurations, the eyeglass device 100 is coupled to one or more microphones 139 that are configured to operate together or independently and are positioned at various locations on the eyeglass device 100.

[0057] The speaker 191 includes an electroacoustic transducer that converts an electrical audio signal into a corresponding sound. The speaker 191 is controlled by one of the processors 422, 432 or by the audio processor 413 ( Figure 4) control. The speaker 191 in this example includes a series of rectangular holes, as shown, which face inward to direct the sound toward the wearer. Additional or differently oriented holes may be implemented. In other example configurations, the eyeglass device 100 is coupled to one or more speakers 191, which are configured to operate together (e.g., in stereo, in a zone to produce surround sound) or independently, and are positioned at various locations on the eyeglass device 100. For example, one or more speakers 191 may be incorporated into the frame 105, the temple 125, or the corner 110 of the eyeglass device 100.

[0058] Although Figure 2A and Figure 2B As shown with two optical elements 180, the eyeglass device 100 may include other arrangements, such as a single optical element (or it may not include any optical element 180), depending on the application or intended user of the eyeglass device 100. As further shown, the eyeglass device 100 includes a right corner 110A adjacent to the right lateral side 170A of the frame 105 and a left corner 110B adjacent to the left lateral side 170B of the frame 105. The corners 110 may be integrated into the frame 105 on the respective sides 170 (as shown) or implemented as separate components attached to the frame 105 on the respective sides 170. Alternatively, the corners 110A, 110B may be integrated into temples (not shown) attached to the frame 105.

[0059] In one example, each image display of optical assembly 180 includes an integrated image display (eg, first display 182A and second display 182B). Figure 2A As shown, each optical assembly 180 has a display 182 including a suitable display matrix 177, such as a liquid crystal display (LCD), an organic light-emitting diode (OLED) display, or other such display. Each optical assembly 180 also includes an optical layer 176, which may include lenses, optical coatings, prisms, reflectors, waveguides, optical strips, and other optical components in any combination. Optical layer ( Figure 2AThe optical layer 176A-N (shown as 176A-N) may include a prism having a suitable size and configuration and including a first surface for receiving light from the display matrix and a second surface for emitting light toward the user's eyes. The prisms of the optical layer 176A-N extend over all or at least a portion of the corresponding holes 175 formed in the left and right rims 107 to allow the user to see the second surface of the prism when the user's eyes are looking through the corresponding rims 107. The first surface of the prism of the optical layer 176A-N faces upward from the frame 105 and the display matrix 177 is pressed against the prism so that photons and light emitted by the display matrix 177 strike the first surface. The size and shape of the prism causes the light to be refracted within the prism and directed toward the user's eyes by the second surface of the prism of the optical layer 176A-N. In this regard, the second surface of the prism of the optical layer 176A-N can be convex to direct the light toward the center of the eye. The prisms are optionally sized and shaped to magnify the image projected by the display matrix 177, and the light traveling through the prisms causes the image viewed from the second surface to be larger in one or more dimensions than the image emitted from the display matrix 177.

[0060] In one example, the optical layers 176A-N may include an LCD layer that is transparent (holding the lenses open) unless and until a voltage is applied to render the layer opaque (closing or blocking the lenses). The image processor 412 on the eyewear device 100 may be programmed to apply a voltage to the LCD layer to create an active shutter system, making the eyewear device 100 suitable for viewing visual content when displayed as a 3D projection. Technologies other than LCD may be used for active shutter modes, including other types of reactive layers that respond to a voltage or another type of input.

[0061] In another example, the image display device of the optical assembly 180 has a display 182, which includes Figure 2B Each optical assembly 180 includes a corresponding laser projector 150, such as a three-color laser projector using a scanning mirror or a galvanometer. Each laser projector 150 is disposed in or on a corresponding temple 125 of the eyeglass device 100. In this example, each optical assembly 180 includes one or more optical strips (in Figure 2B 155A-N), which are spaced apart and span the width of the lens of each optical assembly 180 or span the depth of the lens between the front and back surfaces of the lens.

[0062] As the photons projected by the laser projector 150 travel across the lens of each optical assembly 180, the photons encounter the optical strips 155A-N. When a particular photon encounters a particular optical strip, the photon is either redirected toward the user's eye or passed to the next optical strip. The combination of the modulation of the laser projector 150 and the modulation of the optical strips controls the particular photons or light beams. In an example, the processor controls the optical strips 155A-N by initiating mechanical, acoustic, or electromagnetic signals. Although shown as having two optical assemblies 180, the eyeglass device 100 may include other arrangements, such as a single optical assembly or three optical assemblies, or each optical assembly 180 may have a different arrangement depending on the application or intended user of the eyeglass device 100.

[0063] Figure 3 3D scene 306, a first raw image 302A captured using a first camera 114A, and a second raw image 302B captured using a second camera 114B are illustrated. As shown, the first field of view 111A can overlap with the second field of view 111B. The overlapping field of view 304 represents that portion of the image captured using the two cameras 114. When referring to the field of view, the term "overlap" means that the pixel matrices in the generated raw images overlap by thirty percent (30%) or more. "Substantially overlap" means that the pixel matrices in the generated raw images (or infrared images of the scene) overlap by fifty percent (50%) or more. As described herein, the two raw images 302 can be processed to include a timestamp that allows the images to be displayed together as part of a three-dimensional projection.

[0064] For stereoscopic image capture, such as Figure 3 As shown, a pair of raw red, green, and blue (RGB) images of a 3D scene 306 are captured at a given moment—a first raw image 302A captured using a first camera 114A and a second raw image 302B captured using a second camera 114B. When the raw image pair 302 is processed (e.g., by an image processor 412), a depth image is generated. The generated depth image can be viewed on the optical assembly 180 of the eyewear device, on another display (e.g., an image display 580 on the mobile device 401), or on a screen.

[0065] The generated depth image is in a three-dimensional spatial domain and may include a matrix of vertices on a three-dimensional position coordinate system including an X-axis for horizontal positioning (e.g., length), a Y-axis for vertical positioning (e.g., height), and a Z-axis for depth (e.g., distance). Each vertex may include: a color attribute (e.g., a red pixel light value, a green pixel light value, or a blue pixel light value); a positioning attribute (e.g., an X position coordinate, a Y position coordinate, and a Z position coordinate); a texture attribute; a reflectivity attribute; or a combination thereof. Texture attributes quantify the perceived texture of the depth image, such as the spatial arrangement of colors or intensities in a vertex region of the depth image.

[0066] Figure 4 4 is a functional block diagram of an example magnification system 400, which includes the eyeglass device 100, a mobile device 401, and a server system 498 connected via various networks 495 such as the Internet. As shown, the magnification system 400 includes a low-power wireless connection 425 and a high-speed wireless connection 437 between the eyeglass device 100 and the mobile device 401.

[0067] The eyeglass device 100 includes one or more cameras 114 that capture still images, video images, or both still images and video images, as described herein. The camera 114 may have direct memory access (DMA) to the high-speed circuit 430 and be used as a stereo camera. The camera 114 may be used to capture initial depth images that may be rendered into three-dimensional (3D) models, which are texture-mapped images of red, green, and blue (RGB) imaging scenes. The device 100 may also include a depth sensor that uses infrared signals to estimate the positioning of an object relative to the device 100. The depth sensor includes one or more infrared emitters and an infrared camera 410 in some examples.

[0068] The eyeglass device 100 also includes two image displays (one associated with the right side 170A and one associated with the left side 170B) of the optical assembly 180. The eyeglass device 100 also includes an image display driver 442, an image processor 412, a low power circuit 420, and a high speed circuit 430. The image display of the optical assembly 180 is used to present images, including still images, video images, or still and video images. The image display driver 442 is coupled to the image display of the optical assembly 180 to control the display of the image.

[0069] Figure 4The components for the eyeglass device 100 shown in the figure are located on one or more circuit boards, such as a printed circuit board (PCB) or a flexible printed circuit (FPC) located in the rim or temple. Alternatively or additionally, the depicted components can be located in the corners, frame, hinges, or bridge of the eyeglass device 100. The camera 114 includes a digital camera element, such as a complementary metal oxide semiconductor (CMOS) image sensor, a charge coupled device, a lens, or any other corresponding visible light or light capturing element that can be used to capture data (including still images or video of a scene with unknown objects).

[0070] like Figure 4 As shown, the high-speed circuit 430 includes a high-speed processor 432, a memory 434, and a high-speed wireless circuit 436. In the example, an image display driver 442 is coupled to the high-speed circuit 430 and is operated by the high-speed processor 432 to drive the image display of the optical assembly 180. The high-speed processor 432 can be substantially any processor capable of managing high-speed communications and operations of any general-purpose computing system. The high-speed processor 432 includes processing resources required for managing high-speed data transmission on a high-speed wireless connection 437 to a wireless local area network (WLAN) using a high-speed wireless circuit 436.

[0071] In some examples, the high-speed processor 432 executes an operating system (such as a LINUX operating system or other such operating system of the eyeglass device 100) and stores the operating system in the memory 434 for execution. In addition to any other duties, the high-speed processor 432 executes a software architecture for the eyeglass device 100 that is used to manage data transmission with the high-speed wireless circuit 436. In some examples, the high-speed wireless circuit 436 is configured to implement the Institute of Electrical and Electronics Engineers (IEEE) 802.11 communication standard, also referred to herein as Wi-Fi. In other examples, other high-speed communication standards can be implemented by the high-speed wireless circuit 436.

[0072] The low power circuit 420 includes a low power processor 422 and a low power wireless circuit 424. The low power wireless circuit 424 and the high speed wireless circuit 436 of the eyeglass device 100 may include a short range transceiver (Bluetooth TM or Bluetooth Low-Energy (BLE)) and a wireless wide, local or wide area network transceiver (e.g., cellular or Wi-Fi). Mobile device 401 (including a transceiver that communicates via low power wireless connection 425 and high speed wireless connection 437) can be implemented using details of the architecture of eyewear device 100, as can other elements of network 495.

[0073] The memory 434 includes any storage device capable of storing various data and applications, including, among other things, camera data generated by the cameras 114A, 114B, the infrared camera 410, the image processor 412, and images generated for display by the image display driver 442 on the image display of each optical assembly 180. Although the memory 434 is shown as being integrated with the high-speed circuit 430, the memory 434 can be a separate, independent element of the eyewear device 100 in other examples. In some such examples, circuit routing lines can provide a connection from the image processor 412 or the low-power processor 422 to the memory 434 through a chip including the high-speed processor 432. In other examples, the high-speed processor 432 can manage the addressing of the memory 434 so that the low-power processor 422 will start the high-speed processor 432 at any time involving a read or write operation to be performed on the memory 434.

[0074] like Figure 4 As shown, various elements of the eyewear device 100 can be coupled to the low-power circuit 420, the high-speed circuit 430, or both. For example, the infrared camera 410 (including an infrared transmitter in some embodiments), the user input element 491 (e.g., a button switch, a touch pad 181, a microphone 139), and the inertial measurement unit (IMU) 472 can be coupled to the low-power circuit 420, the high-speed circuit 430, or both.

[0075] like Figure 5 As shown, CPU 540 of mobile device 401 may be coupled to camera system 570, mobile display driver 582, user input layer 591, and memory 540A, as discussed in further detail below.

[0076] The server system 498 may be one or more computing devices as part of a service or network computing system, for example, including a processor, a memory, and a network communication interface to communicate with the eyeglass device 100 and the mobile device 401 over the network 495 .

[0077] The output component of the eyewear device 100 includes a visual element, such as Figure 2A and Figure 2BEach lens or optical component 180 described is associated with an image display (e.g., a display such as a liquid crystal display (LCD), a plasma display panel (PDP), a light emitting diode (LED) display, a projector, or a waveguide). The eyeglass device 100 may include an indicator facing the user (e.g., an LED, a speaker 191, or a vibration actuator), or a signal facing outward (e.g., an LED, a speaker 191). The image display of each optical component 180 is driven by an image display driver 442. In some example configurations, the output components of the eyeglass device 100 also include additional indicators, such as an audible element (e.g., a speaker 191), a tactile component (e.g., an actuator for generating tactile feedback, such as a vibration motor), and other signal generators. For example, the device 100 may include a set of indicators facing the user, and a set of signals facing outward. The set of indicators facing the user is configured to be seen or otherwise sensed by the user of the device 100. For example, the device 100 may include an LED display positioned so that it can be seen by a user, one or more speakers 191 positioned to generate sounds that the user can hear, or an actuator that provides tactile feedback that the user can feel. The outward-facing set of signals is configured to be seen or otherwise sensed by an observer near the device 100. Similarly, the device 100 may include LEDs, speakers 191, or actuators configured and positioned to be sensed by an observer.

[0078] The user input elements 491 of the eyeglass device 100 may include alphanumeric input components (e.g., a touch screen or touchpad 181 configured to receive alphanumeric input, an optical keyboard, or other alphanumeric configured components), pointer-based input components (e.g., a mouse, touchpad 181, trackball, joystick, motion sensor, or other pointing instrument), tactile input components (e.g., a button switch, a touch screen or touchpad 181 that senses the location, force, or location and force of a touch or touch gesture, or other tactile configured elements), and audio input components (e.g., a microphone 139), and the like. The mobile device 401 and the server system 498 may include alphanumeric, pointer-based, tactile, audio, and other input components.

[0079] In some examples, the eyewear device 100 includes a collection of motion sensing components referred to as an IMU 472. The motion sensing components can be micro-electro-mechanical systems (MEMS) with microscopic moving parts, typically small enough to be part of a microchip. The IMU 472 includes an accelerometer, a gyroscope, and a magnetometer in some example configurations. The accelerometer senses the linear acceleration of the device 100 relative to three orthogonal axes (x, y, z) (including acceleration due to gravity). The gyroscope senses the angular velocity of the device 100 about three rotational axes (pitch, roll, yaw). The accelerometer and gyroscope can together provide positioning, orientation, and motion data about the device relative to six axes (x, y, z, pitch, roll, yaw). The magnetometer (if present) senses the heading of the device 100 relative to magnetic north. The location of the device 100 can be determined by a position sensor, such as a GPS unit 473, one or more transceivers for generating relative positioning coordinates, an altitude sensor or a barometer, and other orientation sensors. Such positioning system coordinates can also be received from the mobile device 401 via a wireless connection 425, 437 via a low power wireless circuit 424 or a high speed wireless circuit 436.

[0080] The IMU 472 may include, or cooperate with, a digital motion processor or programming that collects raw data from the components and calculates a number of useful values ​​regarding the location, orientation, and motion of the device 100. For example, the acceleration data collected from the accelerometer may be integrated to obtain the velocity relative to each axis (x, y, z); and integrated again to obtain the location of the device 100 (in linear coordinates, x, y, and z). The angular velocity data from the gyroscope may be integrated to obtain the location of the device 100 (in spherical coordinates). The programming for calculating these useful values ​​may be stored in the memory 434 and executed by the high-speed processor 432 of the eyewear device 100.

[0081] The eyeglass device 100 may optionally include additional peripheral sensors, such as biometric sensors, dedicated sensors, or display elements integrated with the eyeglass device 100. For example, the peripheral device elements may include any I / O components, including output components, motion components, positioning components, or any other such components described herein. For example, the biometric sensors may include components for detecting expressions (e.g., hand expressions, facial expressions, vocal expressions, body postures, or eye tracking), for measuring biosignals (e.g., blood pressure, heart rate, body temperature, perspiration, or brain waves), or for identifying people (e.g., identification based on voice, retina, facial features, fingerprints, or electrical biosignals such as electroencephalogram data), and the like.

[0082] The mobile device 401 may be a smartphone, tablet, laptop, access point, or any other such device capable of connecting to the eyeglass device 100 using both a low power wireless connection 425 and a high speed wireless connection 437. The mobile device 401 is connected to a server system 498 and a network 495. The network 495 may include any combination of wired and wireless connections.

[0083] like Figure 4 As shown, the magnification system 400 includes a computing device, such as a mobile device 401, coupled to the eyeglass device 100 via a network 495. The magnification system 400 includes a memory (e.g., a non-transitory computer-readable medium) for storing instructions and a processor for executing the instructions. In some embodiments, the memory and processing functions of the magnification system 400 can be shared or distributed across the processors and memories of the eyeglass device 100, the mobile device 401, and / or the server system 498.

[0084] In some implementations, the magnification system 400 includes one or more subroutines or modules, referred to herein as a magnification application 910 , a positioning system 915 , an image processing system 920 , and a speech recognition module 925 .

[0085] The magnification application 910 renders and presents the magnification overlay 800 on the display 182 in some implementations, as described herein.

[0086] The positioning system 915 obtains positioning data in some embodiments for determining the positioning of the eyeglass device 100 relative to the physical environment 600. For example, the positioning system 915 can access frames of video data 900 captured using the camera 114B to determine the position 840 of the eyeglass device in three-dimensional coordinates relative to the physical environment (with or without reference to data from other sources, such as an inertial measurement unit or IMU 472). As used herein, the term "video data frame" refers to video motion data captured using one or more cameras 114A, 114B coupled to the eyeglass device 100, including images, spatial data, and related information captured in any form and at any sampling rate using substantially any sensor component of the camera. In some embodiments, the positioning data can be derived from frames of motion data captured using the IMU 472, data collected from the GPS unit 473, or from a combination thereof.

[0087] In some embodiments, the image processing system 920 cooperates with the image display driver 442 and the image processor 412 to present the magnified overlay 800 on the display of the corresponding optical assembly 180, as described herein. The magnified overlay 800, in some embodiments, includes one or more captured frames of video data 900, which are presented in near real time so that the magnified overlay 800 (viewed within the frame 820 on the display) appears nearly simultaneously with the view of the physical environment 600 (viewed outside the frame 820 through the semi-transparent display). In this context, the term "near real time" refers to and means that the image processing system 920 presents the magnified overlay 800 on the display immediately or very quickly after the frame of video data 900 is captured by one or more cameras 114A, 114B. For example, if a frame of video data 900 includes multiple teeth (e.g., tooth numbers seven through nine), a magnified overlay 800 (e.g., tooth number eight, shown at a 2X magnification 755) is presented on the display in near real time such that a user's viewing of the magnified overlay 800 (e.g., tooth number eight) occurs nearly simultaneously with the user's viewing of other portions of the physical environment (e.g., tooth numbers seven and nine, not magnified).

[0088] In some implementations, the speech recognition module 925 receives human speech, converts the received speech into audio data frames, recognizes inquiries or requests based on the audio data, and performs actions related to and responsive to the recognized inquiries or requests.

[0089] Figure 5 is a high-level functional block diagram of an example mobile device 401. Mobile device 401 includes flash memory 540A that stores programming to be executed by CPU 540 to perform all or a subset of the functionality described herein.

[0090] Mobile device 401 may include camera 570, which includes at least two cameras (e.g., a first visible light camera and a second visible light camera with overlapping fields of view) or at least one camera and a depth sensor with substantially overlapping fields of view. Flash memory 540A may further include a plurality of images or videos generated via camera 570.

[0091] As shown in the figure, the mobile device 401 includes an image display 580, a mobile display driver 582 for controlling the image display 580, and a display controller 584. Figure 5 In the example of , image display 580 includes a user input layer 591 (eg, a touch screen) that is layered over or otherwise integrated into the screen used by image display 580 .

[0092] Examples of touch screen mobile devices that may be used include, but are not limited to, smart phones, personal digital assistants (PDAs), tablet computers, laptop computers, or other portable devices. However, the structure and operation of touch screen devices are provided by way of example; the subject technology as described herein is not intended to be limited thereto. For purposes of this discussion, Figure 5 Thus provided is a block diagram of an example mobile device 401 having a user interface including a touch screen input layer 591 for receiving input (via touch, multi-touch or gestures and the like, by hand, stylus or other implement) and an image display 580 for displaying content.

[0093] like Figure 5 As shown, the mobile device 401 includes at least one digital transceiver (XCVR) 510 for digital wireless communication via a wide area wireless mobile communication network, shown as a WWAN XCVR. The mobile device 401 also includes additional digital or analog transceivers, such as for example, via NFC, VLC, DECT, ZigBee, Bluetooth TM The short-range transceiver (XCVR) 520 for short-range network communications using a wireless local area network (WLAN) or Wi-Fi. For example, the short-range XCVR 520 may take the form of any available two-way wireless local area network (WLAN) transceiver of a type compatible with one or more standard protocols for communications implemented in wireless local area networks, such as one of the Wi-Fi standards under IEEE 802.11.

[0094] To generate location coordinates for positioning of mobile device 401, mobile device 401 may include a global positioning system (GPS) receiver. Alternatively or additionally, mobile device 401 may utilize either or both of short-range XCVR 520 and WWAN XCVR 510 to generate location coordinates for positioning. For example, based on a cellular network, Wi-Fi, or Bluetooth TM The positioning system of the present invention can generate accurate position coordinates, especially when used in combination. Such position coordinates can be transmitted to the eyewear device via XCVR 510, 520 through one or more network connections.

[0095] The mobile device 401 includes in some examples a collection of motion sensing components referred to as an inertial measurement unit (IMU) 572 for sensing the position, orientation, and motion of the mobile device 401. The motion sensing components may be microelectromechanical systems (MEMS) with microscopic motion parts, typically small enough to be part of a microchip. The inertial measurement unit (IMU) 572 includes an accelerometer, a gyroscope, and a magnetometer in some example configurations. The accelerometer senses the linear acceleration (including the acceleration due to gravity) of the mobile device 401 relative to three orthogonal axes (x, y, z). The gyroscope senses the angular velocity of the mobile device 401 about three rotational axes (pitch, roll, yaw). The accelerometer and gyroscope can together provide position, orientation, and motion data about the device relative to six axes (x, y, z, pitch, roll, yaw). The magnetometer (if present) senses the heading of the mobile device 401 relative to magnetic north.

[0096] The IMU 572 may include, or cooperate with, a digital motion processor or programming that collects raw data from the components and calculates a number of useful values ​​regarding the location, orientation, and motion of the mobile device 401. For example, the acceleration data collected from the accelerometer may be integrated to obtain the velocity relative to each axis (x, y, z); and integrated again to obtain the location of the mobile device 401 (in linear coordinates, x, y, and z). The angular velocity data from the gyroscope may be integrated to obtain the location of the mobile device 401 (in spherical coordinates). The programming for calculating these useful values ​​may be stored in one or more storage elements 540A, 540B, 540C and executed by the CPU 540 of the mobile device 401.

[0097] The transceivers 510, 520 (i.e., network communication interfaces) conform to one or more of the various digital wireless communication standards utilized by modern mobile networks. Examples of WWAN transceivers 510 include, but are not limited to, transceivers configured to operate according to Code Division Multiple Access (CDMA) and 3rd Generation Partnership Project (3GPP) network technologies, including, for example and without limitation, 3GPP Type 2 (or 3GPP2) and LTE, sometimes referred to as "4G". For example, transceivers 510, 520 provide two-way wireless communication of information (including digitized audio signals, still images and video signals, web page information for display, and web-related input) and various types of mobile message communications to / from mobile device 401.

[0098] The mobile device 401 also includes a microprocessor, which serves as a central processing unit (CPU); Figure 4 540. A processor is a circuit having elements that are structured and arranged to perform one or more processing functions, typically various data processing functions. Although discrete logic components can be used, the example utilizes components that form a programmable CPU. A microprocessor, for example, includes one or more integrated circuit (IC) chips that incorporate electronic components to perform the functions of a CPU. CPU 540, for example, can be based on any known or available microprocessor architecture, such as Reduced Instruction Set Computing (RISC) using an ARM architecture, as commonly used today in mobile devices and other portable electronic devices. Of course, other arrangements of processor circuits can be used to form CPU 540 or processor hardware in smartphones, laptop computers, and tablet computers.

[0099] The CPU 540 acts as a programmable host controller for the mobile device 401 by configuring the mobile device 401 to perform various operations (e.g., according to instructions or programming executable by the CPU 540). For example, such operations may include various general operations of the mobile device, as well as operations related to programming for applications on the mobile device. Although the processor may be configured using hardwired logic, a typical processor in a mobile device is a general processing circuit that is configured by executing programming.

[0100] The mobile device 401 includes a memory or storage system for storing programming and data. In an example, the memory system may include flash memory 540A, random-access memory (RAM) 540B, and other memory components 540C as needed. The RAM 540B acts as short-term storage for instructions and data being processed by the CPU 540, such as working data processing memory. The flash memory 540A typically provides longer-term storage.

[0101] Thus, in the example of mobile device 401, flash memory 540A is used to store programming or instructions executed by CPU 540. Depending on the type of device, mobile device 401 stores and runs a mobile operating system through which specific applications are executed. Examples of mobile operating systems include Google Android, Apple iOS (for iPhone or iPad devices), Windows Mobile, Amazon Fire OS, RIM BlackBerry OS, or the like.

[0102] The processor 432 in the eyeglass device 100 can construct a map of the environment around the eyeglass device 100, determine the position of the eyeglass device in the map of the environment, and determine the relative positioning of the eyeglass device and one or more objects in the mapped environment. The processor 432 can use a simultaneous localization and mapping (SLAM) algorithm applied to data received from one or more sensors to construct a map and determine the position and positioning information. The sensor data includes images received from one or both of the cameras 114A, 114B, distances received from a laser rangefinder, positioning information received from a GPS unit 473, motion and acceleration data received from an IMU 572, or a combination of data from these sensors or from other sensors that provide data that helps determine positioning information. In the context of augmented reality, the SLAM algorithm is used to construct and update a map of the environment while tracking and updating the position of the device (or user) in the mapped environment. The mathematical solution can be approximated using various statistical methods, such as particle filters, Kalman filters, extended Kalman filters, and covariance intersection. In a system that includes a high-definition (HD) video camera that captures video at a high frame rate (e.g., thirty frames per second), the SLAM algorithm updates the map and the positions of objects at least as frequently as the frame rate; in other words, the mapping and localization are calculated and updated thirty times per second.

[0103] The sensor data includes images received from one or both cameras 114A, 114B, distances received from a laser rangefinder, positioning information received from a GPS unit 473, motion and acceleration data received from an IMU 472, or a combination of data from these sensors or from other sensors that provide data that helps determine positioning information.

[0104] Figure 6 An example physical environment 600 and elements useful when using SLAM algorithms and other types of tracking applications (e.g., natural feature tracking (NFT), hand tracking, etc.) are described. A user 602 of the eyewear device 100 is present in the example physical environment 600 (who is in Figure 6is an indoor room). The processor 432 of the eyeglass device 100 uses the captured image data to determine the position of the eyeglass device 100 relative to one or more physical objects 604 within the environment 600, constructs a map of the environment 600 using a coordinate system for the environment 600 (e.g., a Cartesian coordinate system (x, y, z)), and determines the position relative to the coordinate system. In addition, the processor 432 determines the head posture (roll, pitch, and yaw) of the eyeglass device 100 within the environment by using two or more position points (e.g., three position points 606a, 606b, and 606c) associated with a single object 604a, or by using one or more position points 606 associated with two or more objects 604a, 604b, 604c. The processor 432 of the eyeglass device 100 can place virtual objects 608 (such as Figure 6 A key (shown in ) is positioned within environment 600 for viewing during the augmented reality experience.

[0105] The positioning system 915, in some examples, includes a virtual marker 610a associated with a virtual object 608 in the physical environment 600. In an augmented reality environment, in some embodiments, a marker is registered at a location in the physical environment 600 to assist an electronic device in the task of tracking and updating the location of users, devices, and objects (virtual and physical) relative to the physical environment. Markers are sometimes registered to high-contrast physical objects, such as relatively dark objects (such as a framed picture 604a, which is mounted on a lighter-colored wall) to assist cameras and other sensors in the task of detecting markers. Markers can be assigned and registered in memory by the eyewear device 100 operating within the environment. In some embodiments, markers are assigned and registered in the memory of other devices in the network.

[0106] The positioning system 915 tracks physical objects and virtual objects within the physical environment 600 relative to the eyeglass device 100. For physical objects 604 (e.g., safe 604c), the positioning system 915 continuously analyzes captured images of the physical environment 600 to identify the object 604 and determine its position relative to the eyeglass device 100 (e.g., by applying a SLAM algorithm). The positioning system 915 maintains and updates the determined position information for the physical object 604 in a memory, thereby tracking the physical object 604 as the eyeglass device 100 moves through the physical environment 600. For virtual objects 608 (e.g., keys), the positioning system 915 establishes or specifies an initial position of the virtual object 608 corresponding to a position in the environment 600 or the physical object 604 (or, in some embodiments, at a position relative to the eyeglass device 100). The positioning system 915 maintains and updates virtual object 608 position information, for example, according to a movement algorithm associated with the virtual object 608, in response to the eyewear device 100 moving through the environment, or a combination thereof, thereby tracking the virtual object 608 as the eyewear device 100 moves through the environment.

[0107] The marker may be encoded with information or otherwise linked to information. The marker may include positioning information, a physical code (such as a barcode or QR code; visible or hidden to the user), or a combination thereof. A set of data associated with the marker is stored in the memory 434 of the eyewear device 100. The set of data includes information about the marker 610a, the positioning (position and orientation) of the marker, one or more virtual objects, or a combination thereof. The marker positioning may include three-dimensional coordinates for one or more marker landmarks 616a, such as Figure 6 The corners of the generally rectangular marker 610a shown. The marker position can be expressed relative to real-world geographic coordinates, a marker coordinate system, the positioning of the eyewear device 100, or other coordinate systems. One or more virtual objects associated with the marker 610a can include any of a variety of materials, including still images, videos, audio, tactile feedback, executable applications, interactive user interfaces and experiences, and combinations or sequences of such materials. In this case, any type of content that can be stored in memory and retrieved when encountering the marker 610a or associated with an assigned marker can be classified as a virtual object. Figure 6 The virtual key 608 shown in FIG. 6 is, for example, a virtual object displayed as a still image (whether 2D or 3D) at a marked location.

[0108] In one example, the marker 610a can be located near and associated with the physical object 604a (e.g., Figure 6 In another example, the mark may be registered in the memory as a specific location relative to the eyewear device 100.

[0109] Figure 7 700 is a flowchart of an example method of presenting a magnification overlay 800 on the display 182B of the eyeglass device 100. Although the steps are described with reference to the eyeglass device 100 described herein, other implementations of the steps described will be understood by those skilled in the art based on the description herein for other types of devices. One or more of the steps shown and described may be performed simultaneously, in series, in an order other than that shown and described, or in combination with additional steps. Some steps may be omitted or repeated in some applications.

[0110] In some embodiments, the magnification application 910 described herein is initiated in response to receiving a selection through a user interface (e.g., selection from a menu, pressing a button, using a touchpad) or through some other input device (e.g., a gesture detected in a captured image, a finger touch 681 on the touchpad 181, a voice command).

[0111] Figure 7 702 in detail the example steps of capturing frames of video data 900 using the cameras 114A, 114B of the eyeglass device 100. In some embodiments, the process of capturing frames of video data 900 continues during active use of the eyeglass device 100. In other examples, the process of capturing begins in response to receiving a selection through a user interface or through some other input device. In some embodiments, the example method steps at box 702 include at least temporarily storing the captured frames of video data 900 in the memory 434 on the eyeglass device 100 so that the frames of video data 900 can be used for purposes including processing and analysis.

[0112] Figure 7 Detailed in block 704 of FIG. 1 are example steps for registering a virtual marker 802 relative to a structure or object of interest (e.g., a particular tooth) in the physical environment 600. In some embodiments, the process of registering includes the processor of the eyeglass device 100 associating the virtual marker 802 with a marker position 790 (relative to the location on the display 182B) in the physical environment 600. In this regard, the virtual marker position 790 is generally fixed so that it appears at the same location relative to the surrounding physical environment 600, regardless of the movement of the display 182B or the eyeglass device 100 on the physical environment 600.

[0113] Fig. 8A 802 is a perspective illustration of an example virtual marker 802 associated with a marker location 790 and presented on the display 182B. In some embodiments, as shown, the virtual marker 802 includes a target 804 (e.g., a triangle), a perimeter 806, and a virtual object 808 (e.g., a highlight). Fig. 8A As shown, the marker position 790 is located near the target 804 in some embodiments but not necessarily exactly on the target 804. In some embodiments, the virtual marker 802 does not include a target or any other visible clues. In this example, as described herein, the magnification overlay 800 is presented on the display based on the virtual marker position 790 and the current eyeglass position 840 without considering any visible target or its position.

[0114] In some embodiments, the process of registering the virtual marker 802 includes presenting a visual guide on the display 182B, such as a marker indicating the frame location 822 or a marker such as Figure 8B The marking line 824 shown can be placed by the user on or near the suggested marking location 790 (e.g., near a particular tooth). Then, in this example, the registration process receives input indicating where to place and register the virtual marker 802 (e.g., a finger tap 681 on the touchpad 181, a voice command, or a gesture). In other embodiments, the process of registering the virtual marker 802 includes receiving a voice command or other input indicating a specific location (e.g., tooth number 4, buccal surface) relative to the physical environment 600 (e.g., the patient's mouth), which has been mapped and stored in a memory. In some embodiments, the process of registering the virtual marker 802 includes using the touchpad 181 to control the positioning of a cursor (not shown) presented on the display 182B so that the user can place the cursor at a specific location (e.g., near a particular tooth) and perform a selection action to register the virtual marker 802.

[0115] In some embodiments, the process of registering the virtual marker 802 includes presenting a target 804 on the display (as described herein, at block 708) for the user to view. If the selected marker position 790 (as indicated by the target 804) is acceptable, the user can accept the registration; if not, the user can cancel and select again.

[0116] Block 706 details an example step of estimating the eyeglass device position 840 relative to the marker position 790 (e.g., where the virtual marker 802 and the selected tooth are located). After the marker registration process, as the eyeglass device 100 moves through the physical environment 600, its position changes relative to the marker position 790. The current electronic eyeglass device position 840 is estimated in some embodiments using a positioning system 915 as described herein.

[0117] The positioning system 915 on the eyeglass device 100 configures the processor 432 of the eyeglass device 100 in some embodiments to obtain positioning data based on the captured video data frames 900 from the cameras 114A, 114B, and in some embodiments to obtain positioning data based on the motion data collected by the IMU 472. In some embodiments, the positioning system 915 constructs a virtual map of one or more objects within the camera field of view 904 using a SLAM algorithm, as described herein, and updates the map and the locations of the objects at least as frequently as the cameras 114A, 114B capture the video data 900. Frequent analysis of the high-frequency video data 900 facilitates detection of relatively subtle movements of the eyeglass device 100 over time.

[0118] In some embodiments, the step of estimating the position 840 of the electronic eyeglass device relative to the marker position 790 includes calculating a correlation between the marker position 790 and the current electronic eyeglass device position 840. The term correlation refers to and includes one or more vectors, matrices, formulas, or other mathematical expressions sufficient to define a three-dimensional distance between the marker position 790 and the current electronic eyeglass device position 840. The current electronic eyeglass device position 840 is associated with the three-dimensional positioning and orientation (e.g., head pose, gaze direction) of the display 182 because the display 182 is supported by the frame of the eyeglass device 100. In this regard, the correlation process performs the function of calibrating the movement of the eyeglass device 100 using the marker position 790. Because the positioning process occurs frequently, the correlation process between the eyeglass device position 840 and the marker position 790 produces accurate and nearly real-time tracking of the current electronic eyeglass device position 840 relative to the marker position 790.

[0119] In some embodiments, the process of estimating the current electronic eyeglass device position 840 is based on frames of motion data captured using the IMU 472, or based on frames of video data 900 captured using the camera 114A coupled to the eyeglass device 100, or a combination of both. In some embodiments, the process of estimating the current electronic eyeglass device position 840 is performed approximately as often as the IMU 472 captures motion data (e.g., eight hundred times per second, or more, which may be 800 Hz (samples per second) or up to 1600 Hz based on the IMU sampling rate). In some embodiments, the process of estimating the current electronic eyeglass device position 840 occurs at a predefined and configurable frequency, and the IMU 472 is configured to capture frames of motion data at a compatible rate.

[0120] Figure 7 708 in detail presents a target 804 (e.g., a triangle, such as a triangle) on the display 182B based on the marker position 790 and the current electronic eyewear device position 840. Fig. 8AFor example, target 804 includes a graphical element or other indicia, such as a bull's eye, a cross, a dot, an X, a plus sign, or a geometric shape (e.g., Fig. 8A 804) or a combination thereof to identify or otherwise highlight the marker location 790. In some embodiments, the target 804 includes a default element or shape that can be selected or changed through a user interface. In some embodiments, the process of presenting the target 804 includes selecting and generating the target 804 and presenting it on the display as an overlay relative to the physical environment 600, so that the target 804 is always visible in the foreground relative to other parts of the physical environment. In some embodiments, the process of presenting the target 804 may or may not include presenting one or more associated elements, including the perimeter 806 or the virtual object 808.

[0121] Perimeter 806, in some embodiments, defines a proximity (e.g., in two or three dimensions) relative to marker location 790. As described herein, the process of presenting magnified overlay 800 is activated or stopped in some embodiments in response to a related process detecting perimeter 806. For example, Figure 8C 800a is a perspective view in which the pointer 810 is aligned with the frame location 822 located at the center of the crosshairs or reticle 824. At the bottom right, the virtual marker 802 includes a target 804 surrounded by a perimeter 806 (shown as a dashed circle). In some embodiments, the process of presenting a magnification overlay 800a is activated when the frame location 822 is detected at or within the perimeter 806. In this example, the magnification overlay 800a is only active when the frame location 822 points to the virtual marker 802; at least as close as the perimeter 806. For example, Fig.8D The orientation at which the frame location 822 is located and the intersection (eg, dashed circle) detected within the perimeter 806 is illustrated. In this orientation, the magnification overlay 800a is activated, as shown, and is presented on the display 182B within the frame 820.

[0122] Subsequently, if the eyewear device 100 is moved to the left, and the frame position 822 is not detected inside the perimeter 806, then the magnification overlay 800a is deactivated and the view within the frame 820 returns to normal (eg, semi-transparent lens, no magnification).

[0123] like Fig.8D As shown, the magnified overlay 800a includes a view of the physical environment 600 that would normally be viewed through the lens at a magnification factor 755. In some implementations, the magnified overlay 800a includes a view of the virtual marker 802 (e.g., presented at the center of the frame 820), even though such a view would not align with the normal view of the physical environment 600.

[0124] The virtual object 808 , in some implementations, includes a highlight (eg, a circle) or other indicia (eg, visible, audible, or tactile) that is selectively presented on the display 182B to facilitate identification of the target 804 .

[0125] In some implementations, presenting the target 804 includes selectively presenting the target 804 when the marker location 790 is within the field of view 904 associated with the camera 114A based on the current eyewear device position 840. In this regard, the target 804 appears when the field of view 904 includes the marker location 790.

[0126] Figure 7 Block 710 in FIG. 1 details an example step of defining a frame 802 associated with a frame position 822 relative to the display 182B. In some implementations, the frame 802 defines the size and shape of the viewing window in which the magnification overlay 800 is presented. The frame 802 can take any size and shape, such as Figure 8B The generally circular frame shown. In some embodiments, frame 802 includes a default size and shape (e.g., a circular area that occupies approximately half of optical element 180B) and the characteristics of frame 802 can be selected or changed through a user interface. For example, a user can select a shape (e.g., a rectangle) and a size (relative to the size of the lens) depending on the task to be performed. Frame 802 is characterized by an effective radius R1, as shown. As used herein, the term effective radius R1 refers to and includes the relative size of frame 820, whether frame 820 is generally circular or takes some other shape. For example, a rectangular or polygonal frame can be characterized by an effective radius R1 that represents the average distance from the center of frame 820 to its edge. Similarly, the lens effective radius R2 represents the relative size of display 182B, whether circular or some other shape. For example, when the magnification overlay 800 is presented, the border of frame 820 may or may not be presented on display 182B.

[0127] As shown, the frame location 822 may or may not be substantially centered relative to the display 182B. In some embodiments, the frame 820, the effective radius R1, and the frame location 822 are predefined and configurable, so that the process of defining the frame 820 includes providing one or more tools to the user to adjust or otherwise configure the frame 820, as described herein.

[0128] Figure 7 Block 712 in the Fig.8D800a. In some embodiments, the magnification application 810 generates the magnification overlay 800a based on the frame of the captured video data 900, the mark position 790, the magnification factor 755, and the size and shape of the frame 820. For example, the frame of the captured video data 900 may include a plurality of teeth (e.g., seven to nine teeth). The mark position 790 is associated with the number of teeth eight. The magnification factor 755 is 2.5 times. The frame 820 is circular in shape and occupies forty percent of the lens area. In this example, the magnification application 810 uses the frame of the captured video data 900 to generate the magnification overlay 800a including the mark position 790 (e.g., the number of teeth eight) at a magnification factor 755 of 2.5 times, so that the magnification overlay 800a occupies the size and shape of the frame 820. In addition, the magnified overlay 800a is presented according to the current eyewear device position 840, which in this example means that the magnified overlay 800a (e.g., tooth number eight at 2.5 times) is presented on the display only when the eyewear device 100 is facing the marked position 790 in the physical environment 600 (e.g., at tooth number eight). In some embodiments, the magnified overlay 800a is presented on the display 182B within the frame 820 and as an overlay relative to the physical environment 600. In this example, the magnified overlay 800a occupies the frame 820 and is presented in the foreground. Outside of the frame 820, other parts of the physical environment 600 (for clarity, Fig.8D 182B. In this regard, the magnified overlay 800a is, in some embodiments, a nearly instantaneous broadcast of a frame of video data 900 captured using the camera 114A, except that it is presented according to the magnification factor 755. The magnified overlay 800a is presented on the display 182B according to the marker position 790 and the current eyewear device position 840.

[0129] The magnification overlay 800a, in some implementations, includes one or more frames of captured video data 900 presented according to the magnification factor 755. In some implementations, the magnification factor 755 is predefined (e.g., 2x) and configurable, such that the process of presenting the magnification overlay 800a includes providing one or more tools to a user to adjust or otherwise configure the magnification factor 755, as described herein.

[0130] In some embodiments, the magnification overlay 800a includes a portion or cropped segment of one or more frames of the captured video data 900 presented according to the magnification factor 755. In this example, the magnification overlay 800a can be customized in size and shape to include a region of interest around or near the marker location 790, thereby optimizing magnification usage and computational efficiency. In some embodiments, the size and shape of the cropped segment of the magnification overlay 800a is predefined and configurable for a particular application.

[0131] In some embodiments, the magnification 755 is a predefined series of values ​​(e.g., 1.1X, 1.4X, 1.7X, etc.; or 1.10X increasing in increments of 0.05X until a predefined maximum value is reached). In this example, the magnification 755 increases incrementally as the frame position 822 is detected to move closer to the target 804. Conversely, in this example, the magnification 755 decreases incrementally as the frame position 822 moves farther away from the target 804. For example, Fig.8D An orientation is illustrated in which a frame location 822 is located and an intersection (e.g., a dashed circle) is detected inside the perimeter 806. In this orientation, the magnification overlay 800a is activated, as shown, and can be presented on the display 182B at a starting or initial magnification factor 755. As the frame location 822 is detected moving closer to the target 804, the magnification factor 755 increases incrementally. When the frame location 822 approaches the virtual marker 802, as shown in FIG. Fig. 8E As shown, the magnification factor 755 increases towards its predefined maximum value in this example.

[0132] In some embodiments, the process of presenting the magnified overlay 800a includes presenting a view of the frame of the captured video data 900 in a nearly real-time manner, such that the magnified overlay 800a represents a view of structures and objects as they currently appear within the frame 820, with little to no delay. In this regard, the magnified overlay 800 (presented in the frame 820) appears to be nearly simultaneous in time with respect to views of other elements of the physical environment 600 (viewable outside of the frame 820). In this regard, the magnified overlay 800 represents a magnified (or reduced) view of that portion of the physical environment 600 that is most closely associated with the registered marker location 790.

[0133] In some embodiments, the process of presenting the magnification overlay 800a includes selectively presenting the magnification overlay 800a when the marker location 790 (and, accordingly, the virtual marker 802) is within the field of view 904 associated with the camera 114A based on the current eyewear device position 840. In this regard, the magnification overlay 800a appears (e.g., fills all or part of the frame 820) when the field of view 904 includes the marker location 790, and in some embodiments disappears when the field of view 904 does not include the marker location 790. In some embodiments, the process includes analyzing frames of the captured video data 900 to determine whether the virtual marker 802 (at the marker location 790) is located within one or more frames (e.g., within the field of view 904 of the camera) and in response, selecting or activating the process of presenting the magnification overlay 800a.

[0134] When the magnified overlay 800a is not presented, in some embodiments, no other objects, obstacles, or overlays are presented in the frame 820 on the display 182B. Under this condition, the frame 820 behaves like any other part of the display 182B, providing a semi-transparent view of the physical environment 600 in real time.

[0135] The process of presenting the magnification overlay 800a includes, in some embodiments, a routine or utility for adjusting the magnification factor 755 (as described herein), or for selectively starting and stopping the presentation. For example, the user can pause or stop the process of presenting the magnification overlay 800a by speaking a voice command, pressing a button on the eyeglass device 100, performing a gesture, or tapping a finger touch 681 on the touchpad 181.

[0136] Figure 7 Block 714 details example steps for presenting a pointer 810 on the display 182B. Figure 8C 810 is a perspective view of an example pointer 810. In some embodiments, the pointer 810 includes one or more default elements or shapes that can be selected or changed through a user interface. In some embodiments, the process of presenting the pointer 810 includes selecting and generating selected elements of the pointer 810 and presenting the pointer 810 on the display as an overlay relative to the physical environment 600, so that the pointer 810 is always visible in the foreground relative to other parts of the physical environment 600. In some embodiments, the process of presenting the pointer 810 includes presenting a visual guide or guideline 824, such as Figure 8C As shown, it is aligned with the frame location 822. In this example implementation, both the graticule 824 and the pointer 810 are aligned with the frame location 822. The graticule 824 facilitates identification and focus on the center or frame location 822 (e.g., typically near the center of the display 182B) in some implementations.

[0137] In some embodiments, the pointer 810 includes a vector 812, a virtual tether 818, and a range 830. The vector 812 in some embodiments starts at the frame position 822 and points to the target 804. As used herein, the term vector refers to and includes a graphical mark in the shape of an arrow (e.g., a ray) with a length and direction in two or three dimensions. The size and length of the vector 812 are determined to be related to the relative distance (e.g., range 830, as described herein) between the frame position 822 (e.g., it is a known value relative to the current eyewear device position 840) and the target 804 (e.g., it is a known value relative to the marker position 790). For example, the vector 812 is sized to be longer in some embodiments so that when the target 804 is relatively far from the frame position 822, the endpoint arrow is at or near the edge of the frame 820; and the vector 812 is shorter when the target 804 is relatively closer. In this embodiment, the length of the vector 812 decreases as the frame position 822 gets closer to the target 804. The vector 812 in some embodiments includes one or more directional arrows presented with a size and orientation toward the target 804.

[0138] In some embodiments, a virtual tether 818 extends from the frame location 822 (or from the pointer end of the vector 812) to the target 804. As the name implies, the virtual tether 818 acts as an always-visible connector between the frame location 822 and the target 804. The range 830, in some embodiments, includes a display of the current distance between the frame location 822 and the target 804.

[0139] Fig. 8E 800b is a perspective view of another example magnified overlay 800b presented on display 182B. In this view, frame location 822 is nearly coincident with virtual marker 802. In some embodiments, the process of presenting pointer 810 includes selectively presenting pointer 810 only under certain conditions. For example, when frame location 822 is nearly coincident with virtual marker 802, such as Fig. 8E As shown, presenting the pointer 810 becomes less necessary and its presentation is interrupted in favor of a cleaner view of the area of ​​interest (e.g., virtual marker 802). In this regard, the process of presenting the pointer 810 includes selectively interrupting the ongoing process of any or all elements—as the frame location 822 is detected to move proximate to the virtual marker 802—including interrupting the vector 812, the virtual tether 818, and the range 830. In some embodiments, the pointer 810 is automatically interrupted whenever the magnification overlay 800 is presented on the display 182B (e.g., based on the assumption that the pointer 810 is no longer useful when the virtual marker 802 has been located and the magnification overlay 800 has been presented on the display 182B).

[0140] Figure 7Block 716 in details the example steps of detecting a segment 521 traversed by a finger along the touch pad 181 and adjusting one or more values ​​based on the detected segment 521. For example, in some embodiments, the effective radius R1 of the frame 820 is configurable. Figure 8B As shown, segment 521 may be similar to a line segment without satisfying the geometric definition of a line segment. In some embodiments, the process of detecting segment 521 includes constructing a best-fit line segment that approximates segment 521 in length and direction of travel. In some embodiments, the process of adjusting the effective radius R1 includes identifying an original value (e.g., a default value or a predefined value) of the effective radius R1. In some embodiments, the process of adjusting the value includes calculating a magnification factor based on the length and direction of travel of segment 521 (e.g., the longer the segment 521, the higher the value). The magnification factor may include a sign (positive or negative) based on the direction of travel of segment 521 (e.g., amplifying in response to a direction of travel toward the front of the eyewear device 100; reducing a direction of travel toward the ear).

[0141] In another example, in some embodiments, the magnification factor 755 associated with the magnification overlay 800a is configurable, such as Fig.8D As shown. The process of adjusting the magnification 755 includes, in some embodiments, identifying an original value of the magnification 755 (e.g., a default or predefined value). The process of adjusting the magnification 755 includes, in some embodiments, calculating a magnification factor based on the length and the heading of the segment 521 (e.g., the longer the segment 521, the higher the value). The magnification factor may include a sign (positive or negative) based on the heading of the segment 521 (e.g., zooming in (the lens is advanced forward) in response to a heading toward the front of the eyewear device 100; zooming out (the lens is pulled away) to a heading toward the ear).

[0142] Figure 7 718 in detail the example steps of detecting the tap gesture 813, and in response thereto, selectively rendering and presenting one or more features described herein on the display 182B. In some embodiments, as Fig. 8AAs shown, the surface of the touch pad 181 is configured to detect finger tap gestures 813 as described herein (e.g., a fingertip briefly touches the touch pad 181 and releases), other types of finger touches, and moving touches (e.g., sliding). Selectivity in some embodiments includes switching (on or off). In other embodiments, selectivity includes changing the color, intensity (more or less bold), and the relative appearance of the selected feature relative to other features (e.g., moving to the background or foreground). In some embodiments, an example process for selectively presenting a feature includes switching (on or off) the presentation of a virtual marker 802 (e.g., as a whole), a target 804, a perimeter 806, or a virtual object 808 (e.g., highlighting around the target 804). In some embodiments, an example process for selectively presenting a feature includes switching (on or off) the presentation of a frame 820 (e.g., a boundary or space within the frame 820), a reticle 824 (e.g., a crosshair or other visual guide), a pointer 810 (e.g., as a whole), a vector 810, a virtual tether 818, or a range 830.

[0143] The eyewear device 100 includes, in some embodiments, a speech recognition module 925 as described herein, and a microphone 139 coupled to the speaker 191. The speech recognition module 925 configures the processor 432 in some embodiments to perceive human speech, convert the received speech into frames of audio data, identify a first query based on the converted frames of audio data, and then perform an action in response and according to the identified first query. For example, the human speech may include a verbal command (e.g., "stop zooming in" or "zoom factor three point zero"), and in response, the identified first query causes the magnification application 910 to stop or pause the process of presenting the magnification overlay 800a, or change the magnification factor 755 to 3.0, respectively, as described herein.

[0144] Although various systems and methods are described herein with reference to dental procedures, the techniques described herein can be applied to substantially any type of close work in which magnification is desired. For example, magnification is an ideal tool for activities such as dentistry, surgery, gemology, watchmaking, jewelry trade, photography, geology, antiquities inspection, and archival preservation.

[0145] Any of the functionality described herein for the eyewear device 100, the mobile device 401, and the server system 498 may be embodied in one or more computer software applications or sets of programming instructions, as described herein. According to some examples, a "function," "multiple functions," "applications," "multiple applications," "instructions," "multiple instructions," or "programming" is a program that performs the functions defined in the program. One or more applications may be developed in a variety of programming languages ​​and structured in various ways, such as an object-oriented programming language (e.g., Objective-C, Java, or C++) or a procedural programming language (e.g., C or assembly language). In a specific example, a third-party application (e.g., an application created by an entity other than the vendor of a particular platform using ANDROID TM or IOS TM Software Development Kit (SDK) applications can be included in platforms such as IOS TM ANDROID TM , Mobile software running on a mobile operating system such as the iPhone, or another mobile operating system. In this example, the third-party application can call API calls provided by the operating system to facilitate the functions described herein.

[0146] Thus, machine-readable media can take the form of tangible / non-transitory storage media in a variety of forms. Non-volatile storage media include, for example, optical or magnetic disks, such as any of the storage devices in any computer device or the like, such as can be used to implement the client devices, media gateways, code converters, etc. shown in the accompanying drawings. Volatile storage media include dynamic memory, such as the main memory of such computer platforms. Tangible transmission media include coaxial cables; copper wires and optical fibers, including the wires that make up the bus within a computer system. Carrier transmission media can take the form of electrical or electromagnetic signals, or the form of sound or light waves, such as those generated during radio frequency (RF) and infrared (IR) data communications. Common forms of computer-readable media therefore include, for example: floppy disks, flexible disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, DVDs or DVD-ROMs, any other optical media, punched card tapes, any other physical storage media with hole patterns, RAMs, PROMs and EPROMs, FLASH-EPROMs, any other memory chips or boxes, carriers that transmit data or instructions, cables or links that transmit such carriers, or any other media from which a computer can read programming code or data. Many of these forms of computer readable media may be involved in carrying one or more sequences of one or more instructions / program codes to a processor for execution.

[0147] Except as stated immediately above, nothing stated or illustrated is intended or should be construed as causing any element, step, feature, object, benefit, advantage, or equivalent to be dedicated to the public regardless of whether it is recited in the claims.

[0148] It should be understood that the terms and expressions used in this article have the common meaning given by the respective inquiry and research field corresponding to such terms and expressions, unless a specific meaning is set forth in addition in this article. Relational terms such as the first and second can only be used to distinguish an entity or action from another, without necessarily requiring or implying any actual such relationship or order between such entities or actions. The term "comprise", "comprising", "including", "comprising" or any other variation thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device containing or including an element or step list not only includes those elements or steps, but also can include other elements or steps that are not clearly listed or inherent to such process, method, article or device. In the absence of further restrictions, the element starting with "one" or "one" does not exclude the presence of other identical elements in the process, method, article or device containing the element.

[0149] Unless otherwise indicated, any and all measurements, values, ratings, positions, amplitudes, sizes and other specifications in this specification, including those set forth in the claims that follow, are approximate and not exact. Such quantities are intended to have a reasonable range that is consistent with the functions to which they are related and the customary practices in the art to which they belong. For example, parameter values ​​or the like may vary from the stated amount or range by as much as plus or minus ten percent unless expressly stated otherwise.

[0150] In addition, in the foregoing detailed description, it can be seen that various features are grouped together in various examples for the purpose of simplifying the present disclosure. The method of the present disclosure should not be interpreted as reflecting the intention that the claimed examples require more features than those explicitly stated in each claim. Rather, as reflected in the following claims, the subject matter to be protected lies in less than all the features of any single disclosed example. Therefore, the following claims are hereby incorporated into the detailed description, with each claim existing independently as a separately claimed subject matter.

[0151] Although what is considered to be the best mode and other examples have been described above, it should be understood that various modifications may be made therein and the subject matter disclosed herein may be implemented in various forms and examples, only some of which are described herein. It is intended that any and all modifications and variations that fall within the true scope of the present concept be protected by the following claims.

Claims

1. A method for presenting a magnified overlay using an eyewear device including a camera and a display, the method comprising: capturing frames of video data using the camera; registering a virtual marker associated with a marker location relative to a physical environment; Determining a position of the eyewear device relative to the marker position based on the captured video data frame; defining a frame associated with a frame size and a frame positioning relative to the display; as well as A magnification overlay is presented within the frame on the display based on the marker location and the frame positioning, wherein the magnification overlay includes one or more of the captured video data frames magnified by a magnification factor.

2. The method according to claim 1, wherein: The step of presenting the magnified overlay further includes: generating the magnified overlay based on the physical environment and the frame; and The magnified overlay is presented within the frame as an overlay in the foreground relative to the physical environment.

3. The method according to claim 1, wherein: The eyeglass device further comprises a microphone coupled to the speaker and the speech recognition module, and wherein the step of presenting the magnified overlay further comprises: receiving human speech using the microphone; Convert received human speech into audio data frames; recognizing a command based on the converted audio data frames using the speech recognition module; and The magnification overlay is selectively presented based on the identified command.

4. The method according to claim 1, wherein: The magnification factor is selectively configurable, and wherein the step of presenting a magnified overlay further comprises: detecting a segment traversed by a finger along a touch pad coupled to the eyeglass device; adjusting the magnification based on the detected fragments; and The adjusted magnification factor is applied to the magnification overlay.

5. The method according to claim 1, wherein: The virtual marker also includes a perimeter, and wherein the step of registering the virtual marker further includes: displaying a mark aligned with the frame position on the display; receiving input when the reticle is approximately aligned with the suggested marking position; and The marker position is set according to the received input and the virtual marker is stored.

6. The method according to claim 5, wherein: The step of presenting the magnified overlay further includes: defining a perimeter relative to the virtual marker location; comparing the frame location to the perimeter based on a current eyewear device position; and The magnified overlay is selectively presented when the frame location is within the perimeter.

7. The method according to claim 1, wherein: The virtual marker includes a target, and the method further includes: presenting the target on the display according to the marker position and the eyewear device position; presenting on the display a pointer associated with the frame location, wherein the pointer includes a vector, a visual tether, and a range including an estimated distance between the frame location and the target; presenting on the display a visual tether from around the frame location to around the target; adjusting the length of the vector according to the range; and The vector is presented in a direction according to the relative orientation between the frame location and the target.

8. The method according to claim 7, wherein: The step of presenting the pointer further comprises: One or more of the vector, the visual tether, and the range are selectively presented based on at least one of a frame location relative to the target, a frame location relative to the perimeter, the range, the frame, and the magnification overlay.

9. The method according to claim 1, wherein: The frame is characterized by a predefined and configurable validity radius, and wherein the step of defining the frame further comprises: detecting a segment traversed by a finger along a touch pad coupled to the eyeglass device; and The effective radius is adjusted based on the detected fragments.

10. An eyewear device for presenting a magnified overlay, comprising: processor; Memory; Inertial measurement unit; camera; monitor; and Programming is performed in the memory, wherein the processor executes the programming to configure the eyewear device to perform functions, including the following functions: capturing frames of video data using the camera; registering a virtual marker associated with a marker location relative to a physical environment; Determining a position of the eyewear device relative to the marker position based on the captured video data frame; defining a frame associated with a frame positioning relative to the display; and A magnification overlay is presented within the frame on the display based on the marker location and the frame positioning, wherein the magnification overlay includes one or more of the captured video data frames magnified by a magnification factor.

11. The eyewear device according to claim 10, wherein: Functionality for presenting magnified overlays also includes: generating the magnified overlay based on the physical environment and the frame; and The magnified overlay is presented within the frame as an overlay in the foreground relative to the physical environment.

12. The eyewear device according to claim 10, wherein: The eyeglass device further includes a microphone coupled to the speaker and the voice recognition module, and wherein the programming is performed to further configure the eyeglass device to perform the following functions: receiving human speech using the microphone; Convert received human speech into audio data frames; recognizing a command based on the converted audio data frames using the speech recognition module; and The magnification overlay is selectively presented based on the identified command.

13. The eyewear device according to claim 10, wherein: The eyewear device further comprises a touch pad, and wherein the function for presenting the magnification overlay further comprises the following functions: detecting a segment traversed by a finger along the touch pad; adjusting the magnification based on the detected fragments; and The adjusted magnification factor is applied to the magnification overlay.

14. The eyewear device according to claim 10, wherein: The function for registering the virtual token also includes the following functions: displaying a mark aligned with the frame position on the display; receiving input when the reticle is approximately consistent with the suggested marking position; as well as The marker position is set according to the received input and the virtual marker is stored.

15. The eyewear device according to claim 10, wherein: The functionality for presenting the magnified overlay also includes the following functionality: defining a perimeter relative to the virtual marker location; comparing the frame location to the perimeter based on a current eyewear device position; as well as The magnified overlay is selectively presented when the frame location is within the perimeter.

16. The eyewear device according to claim 10, wherein: The virtual marker includes a target, and wherein the programming is performed to further configure the eyewear device to perform the following functions: presenting the target on the display according to the marker position and the eyewear device position; presenting on the display a pointer associated with the frame location, wherein the pointer includes a vector, a visual tether, and a range including an estimated distance between the frame location and the target; presenting on the display a visual tether from around the frame location to around the target; adjusting the length of the vector according to the range; and The vector is presented in a direction according to the relative orientation between the frame location and the target.

17. The eyewear device according to claim 16, wherein: The functions for presenting said pointer also include the following functions: One or more of the vector, the visual tether, and the range are selectively presented based on at least one of a frame location relative to the target, a frame location relative to the perimeter, the range, the frame, and the magnification overlay.

18. A non-transitory computer readable medium storing program code, the program code being operable, when executed, to cause a processor of an eyewear device to perform the following steps: capturing frames of video data using the camera; registering a virtual marker associated with a marker location relative to a physical environment; Determining a position of the eyewear device relative to the marker position based on the captured video data frame; defining a frame associated with a frame positioning relative to the display; as well as A magnified overlay is presented on the display within the frame based on the marker location and the frame positioning, and the magnified overlay is presented as an overlay in the foreground relative to the physical environment, wherein the magnified overlay includes one or more of the captured video data frames presented according to the magnification factor.

19. The non-transitory computer readable medium storing program code according to claim 18, wherein: The program code, when executed, is operable to cause the processor to perform the following further steps: defining a perimeter relative to the virtual marker location; comparing the frame location to the perimeter based on a current eyewear device position; and The magnified overlay is selectively presented when the frame location is within the perimeter.

20. The non-transitory computer readable medium storing program code according to claim 19, wherein: The virtual tag comprises a target, and wherein the program code when executed is operable to cause the processor to perform the following further steps: presenting the target on the display according to the marker position and the eyewear device position; presenting on the display a pointer associated with the frame location, wherein the pointer includes a vector, a visual tether, and a range including an estimated distance between the frame location and the target; presenting on the display a visual tether from around the frame location to around the target; adjusting the length of the vector according to the range; presenting the vector in a direction according to a relative orientation between the frame location and the target; and One or more of the vector, the visual tether, and the range are selectively presented based on at least one of a frame location relative to the target, a frame location relative to the perimeter, the range, the frame, and the magnification overlay.

Images