Augmented reality game using virtual eyewear light beams

Abstract

Interactive augmented reality experiences are disclosed that utilize an eyewear device that includes a virtual eyewear beam. A user can direct the virtual beam by orienting the eyewear device or the user's eye gaze or both. The eyewear device can detect the orientation or eye gaze or both of an opponent's eyewear device. The eyewear device can calculate a score based on the user and the opponent's virtual beams hitting respective target areas, such as another player's head or face.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority to U.S. Application No. 17 / 037,011, filed on September 29, 2020, the contents of which are incorporated herein by reference in their entirety. Technical Field

[0003] The examples set forth in this disclosure relate to the fields of augmented reality (AR) and wearable mobile devices, such as eye-worn devices. More specifically, but not as a limitation, this disclosure describes the display of beams of light from virtual eye-worn devices, including systems for interaction with other users. Background Technology

[0004] Many types of computers and electronic devices available today, such as mobile devices (e.g., smartphones, tablets, and laptops), handheld devices, and wearable devices (e.g., smart glasses, digital eyewear, headbands, head-mounted displays), include various cameras, sensors, wireless transceivers, input systems (e.g., touch-sensitive surfaces, indicators), peripherals, displays, and graphical user interfaces (GUIs) through which users can interact with displayed content.

[0005] Augmented reality (AR) combines real-world objects with virtual objects in the physical environment and displays this combination to the user. The combined display gives the impression that the virtual objects truly exist in the environment, especially when the virtual objects look and behave like real objects.

[0006] Advanced AR technologies, such as computer vision and object tracking, can be used to create perceptually rich and immersive experiences. Computer vision algorithms extract 3D data about the physical world from data captured in digital images or videos. Object recognition and tracking algorithms can be used to detect objects in digital images or videos, estimate their orientation or pose, and track their movement over time. Attached Figure Description

[0007] The features of the various examples described will be readily understood from the following detailed embodiments with reference to the accompanying drawings. In the specification and several views of the drawings, each element is represented by a reference numeral. When multiple similar elements exist, a single reference numeral may be assigned to the similar elements, using a lowercase letter to designate the specific element.

[0008] Unless otherwise noted, the various elements shown in the accompanying drawings are not drawn to scale. The dimensions of the individual elements may be enlarged or reduced for clarity. Several figures depict one or more specific embodiments and are presented by way of example only and should not be construed as limiting. The following figures are included in the accompanying drawings:

[0009] Figure 1A This is a side view (right) of an exemplary hardware configuration for an eye-worn device suitable for an augmented reality generation system;

[0010] Figure 1B yes Figure 1A A partial cross-sectional perspective view of the right corner of the eye-wearing device, depicting the right visible light camera and circuit board;

[0011] Figure 1C yes Figure 1A A side view (left) of an exemplary hardware configuration for an eye-wearing device, showing the left visible light camera;

[0012] Figure 1D yes Figure 1C A partial cross-sectional perspective view of the left corner of the eye-wearing device, depicting the left visible light camera and circuit board;

[0013] Figure 2A and Figure 2B This is a rear view of an exemplary hardware configuration of an eye-wearing device used in an augmented reality generation system;

[0014] Figure 3 It is a graphical depiction of a 3D scene, the left raw image captured by the left visible light camera, and the right raw image captured by the right visible light camera;

[0015] Figure 4 It is a functional block diagram of an exemplary augmented reality generation system that includes wearable devices (e.g., eye-wearing devices) and server systems connected via various networks;

[0016] Figure 5 It is used for Figure 4 A graphical representation of an exemplary hardware configuration of a mobile device that generates augmented reality;

[0017] Figure 6 This is a schematic diagram used to describe a user who is simultaneously locating and mapping in an exemplary environment;

[0018] Figure 7 This is a flowchart outlining the steps in an exemplary method for displaying an eye-worn beam in a physical environment;

[0019] Figure 8 This is a flowchart outlining the steps in an exemplary method for displaying an eye-wear beam in a physical environment, including monitoring the target location of the eye or eye-wear.

[0020] Figure 9 This is a flowchart outlining the steps in an exemplary method for displaying an eyewear beam in a physical environment, including receiving data about the target location of the eye or eyewear;

[0021] Figure 10A , Figure 10B , Figure 10C and Figure 10D It is a perspective view of an augmented reality experience using a virtual eye-wearing beam;

[0022] Figure 11A , Figure 11B and Figure 11C It is a perspective view of an augmented reality experience that uses a virtual eyewear beam to display the user's eyes or the positioning of the eyewear.

[0023] Figure 12A , Figure 12B and Figure 12C It is a perspective view of an augmented reality experience that uses a virtual eyewear beam to monitor the target location of the eye or eyewear. Detailed Implementation

[0024] The reference examples describe various specific implementations and details, including methods for implementing the augmented reality applications described herein on wearable devices (e.g., eye-wearing devices), wherein the applications include providing virtual beams corresponding to the gaze of a user or target / opponent in a virtual game that can be visualized through the eye-wearing device. In operation, the eye-wearing device executes code that utilizes a positioning detection system to register the positioning of a first target and the positioning of the eye-wearing device. The eye-wearing device also utilizes its camera system to capture one or more images of the target. At least one portion of the target is identified in the one or more images. The positioning of one or more eyes of the user of the eye-wearing device, or the positioning of the user's eye-wearing device, or both, is monitored, and at least one virtual beam extending from at least one eye of the user, or from the eye-wearing device, or from both, is calculated. The eye-wearing device then presents a virtual image reflecting the direction of at least one calculated beam for display on the eye-wearing device and determines whether one or more of the calculated beams intersect at least one portion of the target.

[0025] The following detailed description includes systems, methods, techniques, instruction sequences, and computer program products illustrating the examples set forth in this disclosure. Numerous details and examples are included to provide a thorough understanding of the disclosed subject matter and its associated teachings. However, those skilled in the art will understand how to apply the teachings without such details. The aspects of the disclosed subject matter are not limited to the specific devices, systems, and methods described, as the associated teachings can be applied or practiced in various ways. The terminology and naming used herein are for descriptive purposes only and are not intended to be limiting. Typically, well-known examples of instructions, protocols, structures, and techniques are not necessarily shown in detail.

[0026] As used herein, the terms “coupled” or “connected” refer to any logical, optical, physical, or electrical connection, including links, through which electrical or magnetic signals generated or provided by one system element are transmitted to another coupled or connected system element. Unless otherwise stated, coupled or connected elements or devices are not necessarily directly connected to each other and may be separated by intermediate components, elements, or communication media, one or more of which may modify, manipulate, or carry electrical signals. The term “on” means directly supported by an element or indirectly supported by an element through another element integrated into or supported by that element.

[0027] The term "proximal" is used to describe an object or part of an object that is located near, to the left of, or next to an object or person; or it is closer to other parts of the object that can be described as "distal." For example, the end of an object that is closest to an object can be called the proximal end, while the roughly opposite end can be called the distal end.

[0028] For purposes of illustration and discussion, the orientation of eye-wearing devices, other mobile devices, associated components, and any other devices including cameras, inertial measurement units, or both shown in any of the accompanying drawings are given by way of example only. In operation, eye-wearing devices may be oriented in any other direction suitable for the specific application of the eye-wearing device, such as up, down, sideways, or any other orientation. Furthermore, for the purposes of this document, any directional terms such as front, back, inside, outside, towards, left, right, sideways, longitudinal, up, down, high, low, top, bottom, side, horizontal, vertical, and diagonal are used by way of example only and do not limit the orientation or orientation of any camera or inertial measurement unit as constructed or otherwise described herein.

[0029] Other objects, advantages, and novel features of the examples will be set forth in part in the detailed description which follows, and in part will become apparent to those skilled in the art upon examination of the following and the accompanying drawings, or may be learned by means of production or operation of the examples. The objects and advantages of this subject matter may be realized and achieved by means of the methods, means, and combinations particularly pointed out in the appended claims.

[0030] Now refer in detail to the accompanying drawings and the examples discussed below.

[0031] Figure 1A This is a side view (right) of an exemplary hardware configuration of an eye-wearing device 100 including a touch-sensitive input device or touchpad 181. As shown, the touchpad 181 may have a subtle and barely perceptible boundary; alternatively, the boundary may be clearly visible or include raised or otherwise tactile edges that provide feedback to the user about the position and boundary of the touchpad 181. In other embodiments, the eye-wearing device 100 may include a touchpad on the left side.

[0032] The surface of touchpad 181 is configured to detect finger touches, taps, and gestures (e.g., moving touches) for use with the GUI displayed on the image display of the eye-wearing device, thereby allowing users to navigate and select menu options in an intuitive way, which improves and simplifies the user experience.

[0033] Detection of finger input on touchpad 181 enables several functions. For example, touching anywhere on touchpad 181 can cause the GUI to be displayed or an item to be highlighted on an image display, which can be projected onto at least one of optical components 180A, 180B. Double-clicking on touchpad 181 selects an item or icon. Sliding or swiping a finger in a specific direction (e.g., from front to back, from back to front, from top to bottom, or from bottom to top) causes an item or icon to slide or scroll in that direction; for example, to move to the next item, icon, video, image, page, or slideshow. Sliding a finger in another direction allows sliding or scrolling in the opposite direction; for example, to move to the previous item, icon, video, image, page, or slideshow. Touchpad 181 can be located virtually anywhere on the eye-wearing device 100.

[0034] In one example, a recognized finger gesture clicked on touchpad 181 initiates the selection or pressing of a graphical user interface element in an image displayed on the optical components 180A, 180B. Adjustments to the image displayed on the optical components 180A, 180B based on the recognized finger gesture can be a primary action of selecting or submitting a graphical user interface element on the image display of the optical components 180A, 180B for further display or execution.

[0035] As shown in the figure, the eye-wearing device 100 includes a right visible light camera 114B. As further described herein, two cameras 114A and 114B capture image information of the scene from two different viewpoints. The two captured images can be used to project a 3D display onto an image display for viewing using 3D glasses.

[0036] The eye-worn device 100 includes a right optical component 180B, which has an image display for presenting images, such as depth images. Figure 1A and Figure 1B As shown, the eye-wearing device 100 includes a right visible light camera 114B. The eye-wearing device 100 may include multiple visible light cameras 114A, 114B forming a passive three-dimensional camera, such as a stereo camera, wherein the right visible light camera 114B is located at the right corner 110B. Figures 1C to 1D As shown, the eye-wearing device 100 also includes a left visible light camera 114A.

[0037] Left and right visible light cameras 114A and 114B are sensitive to wavelengths within the visible light range. Each visible light camera 114A and 114B has a different forward field of view, which overlaps to enable the generation of a three-dimensional depth image; for example, the right visible light camera 114B depicts a right field of view 111B. Typically, a "field of view" is a portion of a scene in space that is visible to a camera at a specific location and orientation. Fields of view 111A and 111B have an overlapping field of view 304. Figure 3 When a visible light camera captures an image, objects or object features outside the field of view 111A, 111B are not recorded in the original image (e.g., a photograph or picture). The field of view describes the angular range or amplitude of electromagnetic radiation of a given scene picked up by the image sensors of visible light cameras 114A, 114B in an image captured of that scene. The field of view can be expressed as the angular size of the view frustum; i.e., the viewing angle. The viewing angle can be measured horizontally, vertically, or diagonally.

[0038] In the examples, visible light cameras 114A and 114B have a field of view between 15° and 110° (e.g., 24°) and a resolution of 480×480 pixels or greater. "Coverage angle" describes the visible light camera 114A, 114B or the infrared camera 410 that can effectively image the image (see...). Figure 2A The angular range of the lens. Typically, a camera lens produces an image circle large enough to completely cover the camera's film or sensor, which may include some degree of vignetting (e.g., the image darkens towards the edges compared to the center). If the camera lens's coverage angle does not extend across the sensor, the image circle will be visible, typically with strong vignetting towards the edges, and the effective angle of view will be limited to the coverage angle.

[0039] Examples of such visible light cameras 114A and 114B include high-resolution complementary metal-oxide-semiconductor (CMOS) image sensors and digital VGA cameras (video graphics arrays) capable of having resolutions of 640p (e.g., 640 × 480 pixels, totaling 0.3 megapixels), 720p, 1080p, or greater. Other examples of visible light cameras 114A and 114B include those capable of capturing high-definition (HD) still images and storing these images at a resolution of 1642 × 1642 pixels (or greater); or recording high-definition video at a high frame rate (e.g., thirty to sixty frames per second or more) and storing the recording at a resolution of 1216 × 1216 pixels (or greater).

[0040] The eye-wearing device 100 can capture image sensor data from visible light cameras 114A and 114B, as well as geolocation data digitized by an image processor, for storage in memory. The visible light cameras 114A and 114B capture corresponding left and right raw images in a two-dimensional spatial domain. These raw images include a pixel matrix in a two-dimensional coordinate system, which includes an X-axis for horizontal positioning and a Y-axis for vertical positioning. Each pixel includes color attribute values ​​(e.g., red pixel light value, green pixel light value, or blue pixel light value); and positioning attributes (e.g., X-axis coordinates and Y-axis coordinates).

[0041] In order to capture stereoscopic images for later display as a 3D projection, image processor 412 (in...) Figure 4 (As shown in the diagram) Visible light cameras 114A and 114B can be coupled to receive and store visual image information. Image processor 412 or another processor controls the operation of visible light cameras 114A and 114B to act as stereo cameras simulating human binocular vision and can add timestamps to each image. The timestamps on each pair of images allow the images to be displayed together as part of a 3D projection. The 3D projection produces an immersive and realistic experience, which is desired in various scenarios including virtual reality (VR) and video games.

[0042] Figure 1B yes Figure 1A A partial cross-sectional perspective view of the right corner 110B of the eye-wearing device 100, depicting the right visible light camera 114B and the circuit board. Figure 1C yes Figure 1A A side view (left) of an exemplary hardware configuration of an eye-wearing device 100, showing the left visible light camera 114A of the camera system. Figure 1D yes Figure 1C A cross-sectional perspective view of the left corner portion 110A of the eye-wearing device, which depicts the left visible light camera 114A of the three-dimensional camera and the circuit board.

[0043] Except for the connection and coupling located on the left side 170A, the structure and arrangement of the left visible light camera 114A are basically similar to those of the right visible light camera 114B. For example... Figure 1BAs shown in the example, the eye-wearing device 100 includes a right visible light camera 114B and a circuit board 140B, which may be a flexible printed circuit board (PCB). A left hinge 126A connects the left corner portion 110A to the left temple 125A of the eye-wearing device 100. In some examples, components of the left visible light camera 114A, the flexible PCB 140A, or other electrical connectors or contacts may be located on the left temple 125A or the right hinge 126B. The right hinge 126B connects the right corner portion 110B to the right temple 125B of the eye-wearing device 100. In some examples, components of the right visible light camera 114B, the flexible PCB 140B, or other electrical connectors or contacts may be located on the right temple 125B or the right hinge 126B.

[0044] The right corner portion 110B includes a corner body 190 and a corner cover, as shown in 1B. Inside the right corner portion 110B are various interconnected circuit boards, such as PCBs or flexible PCBs, which include controller circuitry for the right visible light camera 114B, a microphone, and low-power wireless circuitry (e.g., for use via Bluetooth). TM Wireless short-range network communication), high-speed wireless circuits (e.g., for wireless local area network communication via Wi-Fi).

[0045] The right visible light camera 114B is coupled to or disposed on the flexible PCB 140B and is covered by a visible light camera lens, which is aimed through an opening formed in the frame 105. For example, the right edge 107B of the frame 105, as... Figure 2A As shown, it connects to the right corner 110B and includes an opening for a visible light camera cover lens. The frame 105 includes a front side configured to face outwards and away from the user's eye. The opening for the visible light camera cover lens is formed on and extends through the front or outer side of the frame 105. In the example, the right visible light camera 114B has an outward-facing field of view 111B. Figure 3 (As shown), its line of sight or viewing angle is related to the right eye of the user of the eye-wearing device 100. The visible light camera cover lens can also be attached to the front side or outward-facing surface of the right corner 110B, wherein the opening forms an outward-facing coverage angle, but in a different outward direction. Coupling can also be achieved indirectly via an intermediary member.

[0046] like Figure 1B As shown, the flexible PCB 140B is disposed within the right corner portion 110B and coupled to one or more other components housed in the right corner portion 110B. Although shown as being formed on a circuit board on the right corner portion 110B, the right visible light camera 114B may be formed on a circuit board on the left corner portion 110A, temples 125A, 125B, or frame 105.

[0047] Figure 2A and Figure 2B This is a rear perspective view of an exemplary hardware configuration of an eye-wearing device 100 that includes two different types of image displays. The size and shape of the eye-wearing device 100 are designed to be configured for wear by a user; in this example, it is in the form of eyeglasses. The eye-wearing device 100 may take other forms and may be combined with other types of frames, such as headbands, headphones, or helmets.

[0048] In the example of eyeglasses, the eye-wearing device 100 includes a frame 105 comprising a left edge 107A connected to the right edge 107B via a nose bridge 106 adapted for support by the user's nose. The left and right edges 107A, 107B include corresponding apertures 175A, 175B that hold corresponding optical elements 180A, 180B, such as lenses and display devices. As used herein, the term "lens" is intended to include a sheet of transparent or translucent glass or plastic having a curved or flat surface that causes light to converge / diverge or to cause little or no convergence or divergence.

[0049] Although shown as having two optical elements 180A, 180B, the eyewear device 100 may include other arrangements, such as a single optical element (or it may not include any optical elements 180A, 180B), depending on the application of the eyewear device 100 or the intended user. As further shown, the eyewear device 100 includes a left corner portion 110A adjacent to the left side face 170A of the frame 105 and a right corner portion 110B adjacent to the right side face 170B of the frame 105. The corner portions 110A, 110B may be integrated into the corresponding sides 170A, 170B of the frame 105 (as shown) or implemented as separate components attached to the corresponding sides 170A, 170B of the frame 105. Alternatively, the corner portions 110A, 110B may be integrated into temples (not shown) attached to the frame 105.

[0050] In one example, the image display of optical components 180A and 180B includes an integrated image display. For example... Figure 2A As shown, each optical component 180A, 180B includes a suitable display matrix 177, such as a liquid crystal display (LCD), an organic light-emitting diode (OLED) display, or any other such display. Each optical component 180A, 180B also includes one or more optical layers 176, which may include any combination of lenses, optical coatings, prisms, mirrors, waveguides, optical strips, and other optical components. Optical layers 176A, 176B, ..., 176N (in... Figure 2AThe optical layer 176A-N (shown as 176A-N) may include a prism having suitable dimensions and construction and including a first surface for receiving light from a display matrix and a second surface for emitting light toward a user's eye. The prism of optical layer 176A-N extends over all or at least a portion of corresponding apertures 175A, 175B formed in the left and right edges 107A, 107B, to allow the user to see the second surface of the prism when viewing through the corresponding left and right edges 107A, 107B. The first surface of the prism of optical layer 176A-N faces upward from frame 105, and display matrix 177 covers the prism such that photons and rays emitted by display matrix 177 illuminate the first surface. The prism is sized and shaped such that light is refracted within the prism and directed to the user's eye by the second surface of the prism of optical layer 176A-N. In this respect, the second surface of the prism of optical layer 176A-N may be convex to direct light toward the center of the eye. The size and shape of the prism can be optionally designed to magnify the image projected by the display matrix 177, and the light travels through the prism such that the image viewed from the second surface is larger than the image emitted from the display matrix 177 in one or more dimensions.

[0051] In one example, optical layers 176A-N may include a transparent LCD layer (keeping the lens open) unless and until a voltage is applied that makes the layer opaque (closing or blocking the lens). Image processor 412 on the eyewear device 100 may execute a program to apply voltage to the LCD layer to create an active shutter system, thereby adapting the eyewear device 100 for viewing visual content displayed as a three-dimensional projection. Technologies other than LCDs may be used in the active shutter mode, including other types of reactive layers that respond to voltage or another type of input.

[0052] In another example, the image display device with optical components 180A and 180B includes, for example... Figure 2B The projected image display shown. Each optical component 180A, 180B includes a laser projector 150, which is a tri-color laser projector using a scanning mirror or galvanometer. During operation, a light source such as the laser projector 150 is positioned within or above one of the temples 125A, 125B of the eyewear device 100. In this example, optical component 180B includes one or more optical strips 155A, 155B, ... 155N (in... Figure 2B (shown as 155A-N), which are spaced apart on the width of the lens of each optical component 180A, 180B, or on the depth of the lens between the front and rear surfaces of the lens.

[0053] As photons projected by the laser projector 150 travel through the lens of each optical component 180A, 180B, they encounter optical strips 155A-N. When a particular photon encounters a particular optical strip, it is either redirected toward the user's eye or propagated to the next optical strip. A combination of modulation of the laser projector 150 and modulation of the optical strips can control a particular photon or beam of light. In the example, the processor controls the optical strips 155A-N by emitting mechanical, acoustic, or electromagnetic signals. Although shown as having two optical components 180A, 180B, the eye-wearing device 100 may include other arrangements, such as single or three optical components, or each optical component 180A, 180B may be arranged in a different configuration, depending on the application of the eye-wearing device 100 or the intended user.

[0054] like Figure 2A and Figure 2B As further shown, the eye-wearing device 100 includes a left corner portion 110A adjacent to the left side surface 170A of the frame 105 and a right corner portion 110B adjacent to the right side surface 170B of the frame 105. The corner portions 110A and 110B can be integrated into the corresponding sides 170A and 170B of the frame 105 (as shown) or implemented as separate components attached to the corresponding sides 170A and 170B of the frame 105. Alternatively, the corner portions 110A and 110B can be integrated into the temples 125A and 125B attached to the frame 105.

[0055] In another example, Figure 2B The eye-wearing device 100 shown may include two projectors, a left projector 150A (not shown) and a right projector 150B (shown as projector 150). The left optical assembly 180A may include a left display matrix 177A (not shown) or left optical strips 155'A, 155'B, ..., 155'N (155', A to N, not shown), configured to interact with light from the left projector 150A. Similarly, the right optical assembly 180B may include a right display matrix 177B (not shown) or right optical strips 155"A, 155"B, ..., 155"N (155", A to N, not shown), configured to interact with light from the right projector 150B. In this example, the eye-wearing device 100 includes a left display and a right display.

[0056] Figure 3This is a graphical depiction of a 3D scene 306, a left raw image 302A captured by a left visible light camera 114A, and a right raw image 302B captured by a right visible light camera 114B. As shown, the left field of view 111A may overlap with the right field of view 111B. The overlapping field of view 304 represents the portion captured by the two cameras 114A and 114B in the image. The term "overlapping" in relation to field of view means that the pixel matrix in the generated raw image overlaps by thirty percent (30%) or more. "Substantially overlapping" means that the pixel matrix in the generated raw image or the pixel matrix in the infrared image of the scene overlaps by fifty percent (50%) or more. As described herein, the two raw images 302A and 302B may be processed to include a timestamp that allows the images to be displayed together as part of a 3D projection.

[0057] To capture stereoscopic images, such as Figure 3 As shown, a pair of raw red-green-blue (RGB) images of a real scene 306 are captured at a given time—a left raw image 302A captured by the left camera 114A and a right raw image 302B captured by the right camera 114B. When the pair of raw images 302A, 302B are processed (e.g., by an image processor 412), a depth image is generated. The generated depth image can be viewed on the optical components 180A, 180B of an eye-wearing device, on another display (e.g., an image display 580 on a mobile device 401), or on a screen.

[0058] The generated depth image is in a three-dimensional spatial domain and may include a vertex matrix in a three-dimensional positional coordinate system, which includes an X-axis for horizontal positioning (e.g., length), a Y-axis for vertical positioning (e.g., height), and a Z-axis for depth (e.g., distance). Each vertex may include color attributes (e.g., red pixel light value, green pixel light value, or blue pixel light value); positional attributes (e.g., X-coordinate, Y-coordinate, and Z-coordinate); texture attributes; reflectance attributes; or combinations thereof. Texture attributes quantify the perceptual texture of the depth image, such as the spatial arrangement of colors or intensities in the vertex regions of the depth image.

[0059] In one example, an interactive augmented reality system 400 ( Figure 4 The device includes an eye-wearing device 100, which includes a frame 105, a left temple 110A extending from the left side 170A of the frame 105, and a right temple 125B extending from the right side 170B of the frame 105. The eye-wearing device 100 may also include at least two visible light cameras 114A, 114B having overlapping fields of view. In one example, the eye-wearing device 100 includes a left visible light camera 114A having a left field of view 111A, such as... Figure 3As shown. The left camera 114A is attached to the frame 105 or the left temple 110A to capture a left raw image 302A from the left side of scene 306. The eye-wearing device 100 also includes a right visible light camera 114B with a right field of view 111B. The right camera 114B is attached to the frame 105 or the right temple 125B to capture a right raw image 302B from the right side of scene 306.

[0060] Figure 4 This is a functional block diagram of an exemplary interactive augmented reality system 400, which includes wearable devices (e.g., eye-wearing device 100), mobile devices 401, and server systems 498 connected via various networks 495 (such as the Internet). The interactive augmented reality system 400 includes a low-power wireless connection 425 and a high-speed wireless connection 437 between the eye-wearing device 100 and the mobile device 401.

[0061] like Figure 4 As shown and described herein, the eye-wearing device 100 includes one or more visible light cameras 114A, 114B that capture still images, video images, or both still and video images. Cameras 114A, 114B may have direct memory access (DMA) to high-speed circuitry 430 and function as stereo cameras. Cameras 114A, 114B can be used to capture initial depth images, which can be rendered into three-dimensional (3D) models, which are texture-mapped images of a red-green-blue (RGB) imaged scene. Device 100 may also include a depth sensor that uses infrared signals to estimate the location of an object relative to device 100. In some examples, the depth sensor includes one or more infrared emitters 415 and an infrared camera 410.

[0062] The eye-wear device 100 also includes two image displays for each optical component 180A, 180B (one associated with the left side 170A and one associated with the right side 170B). The eye-wear device 100 also includes an image display driver 442, an image processor 412, low-power circuitry 420, and high-speed circuitry 430. The image displays for each optical component 180A, 180B are used to present images, including still images, video images, or both still and video images. The image display driver 442 is coupled to the image displays for each optical component 180A, 180B to control the display of the images.

[0063] The eye-wearing device 100 also includes one or more speakers 440 (e.g., one associated with the left side of the eye-wearing device and another associated with the right side of the eye-wearing device). The speakers 440 may be embedded in the frame 105, temple 125, or corner 110 of the eye-wearing device 100. The one or more speakers 440 are driven by an audio processor 443 under the control of low-power circuitry 420, high-speed circuitry 430, or both. The speakers 440 are used to present audio signals, including, for example, beat tracks. The audio processor 443 is coupled to the speakers 440 to control the presentation of sound.

[0064] Figure 4 The components shown for the eye-wearing device 100 are located on one or more circuit boards, such as printed circuit boards (PCBs) or flexible printed circuit boards (FPCs) located in the edges or temples. Alternatively or additionally, the depicted components may be located in the corners, frames, hinges, or bridge of the eye-wearing device 100. The left and right visible light cameras 114A, 114B may include digital camera elements, such as complementary metal-oxide-semiconductor (CMOS) image sensors, charge-coupled devices, lenses, or any other corresponding visible or light-capturing elements that can be used to capture data, including still images or videos of scenes with unknown objects.

[0065] like Figure 4 As shown, the high-speed circuit 430 includes a high-speed processor 432, a memory 434, and a high-speed wireless circuit 436. In this example, an image display driver 442 is coupled to the high-speed circuit 430 and operated by the high-speed processor 432 to drive the left and right image displays of each optical component 180A, 180B. The high-speed processor 432 can be any processor capable of managing the high-speed communication and operation of any general-purpose computing system required by the eye-wear device 100. The high-speed processor 432 includes the processing resources required to use the high-speed wireless circuit 436 to transmit high-speed data from the high-speed wireless connection 437 to a wireless local area network (WLAN).

[0066] In some examples, the high-speed processor 432 executes an operating system, such as the LINUX operating system or other such operating system of the eye-wear device 100, and the operating system is stored in memory 434 for execution. Among other duties, the high-speed processor 432, which executes the software architecture of the eye-wear device 100, also manages data transmissions utilizing the high-speed wireless circuit 436. In some examples, the high-speed wireless circuit 436 is configured to implement the Institute of Electrical and Electronics Engineers (IEEE) 802.11 communication standard, also referred to herein as Wi-Fi. In other examples, the high-speed wireless circuit 436 may implement other high-speed communication standards.

[0067] Low-power circuitry 420 includes a low-power processor 422 and low-power wireless circuitry 424. The low-power wireless circuitry 424 and high-speed wireless circuitry 436 of the eye-wear device 100 may include short-range transceivers (Bluetooth™ or Bluetooth Low Energy (BLE)) and wireless wide-area network, local area network, or wide-area network transceivers (e.g., cellular or Wi-Fi). The mobile device 401, including transceivers communicating via low-power wireless connection 425 and high-speed wireless connection 437, may be implemented using the architectural details of the eye-wear device 100, just like other components of the network 495.

[0068] Memory 434 includes any storage device capable of storing various data and applications, including, among other things, camera data generated by the left and right visible light cameras 114A, 114B, the infrared camera 410, the image processor 412, and images generated by the image display driver 442 for display on the image display of each optical component 180A, 180B. While memory 434 is shown as integrated with high-speed circuitry 430, in other examples, memory 434 may be a separate, independent component of the eye-wearing device 100. In some such examples, electrical wiring may provide a connection from the image processor 412 or the low-power processor 422 to memory 434 via a chip including the high-speed processor 432. In other examples, the high-speed processor 432 may manage addressing of memory 434 such that the low-power processor 422 will initiate the high-speed processor 432 whenever a read or write operation involving memory 434 is required.

[0069] like Figure 4 As shown, the high-speed processor 432 of the eye-wearing device 100 can be coupled to a camera system (visible light cameras 114A, 114B), an image display driver 442, a user input device 491, and a memory 434. Figure 5 As shown, the CPU 530 of the mobile device 401 can be coupled to the camera system 570, IMU 572, mobile display driver 582, user input layer 591 and memory 540A.

[0070] Server system 498 may be one or more computing devices as part of a service or network computing system, including, for example, a processor, memory, and a network communication interface for communicating with eye-wearing device 100 and mobile device 401 via network 495.

[0071] The output components of the eye-worn device 100 include visual elements, such as left and right image displays associated with each lens or optical assembly 180A, 180B, such as Figure 2A and Figure 2BThe device 100 may include user-facing indicators (e.g., LEDs, speakers, or vibration actuators) or outward-facing signals (e.g., LEDs, speakers). The image display of each optical component 180A, 180B is driven by an image display driver 442. In some exemplary configurations, the output components of the eye-wearing device 100 may also include additional indicators, such as audible elements (e.g., speakers), tactile elements (e.g., actuators, such as vibration motors for generating tactile feedback), and other signal generators. For example, the device 100 may include a set of user-facing indicators and a set of outward-facing signals. The user-facing set of indicators is configured to be seen or otherwise perceived by the user of the device 100. For example, the device 100 may include an LED display positioned so that the user can see it, one or more speakers positioned to generate sounds that the user can hear, or actuators providing tactile feedback that the user can feel. A set of outward-facing signals is configured to be seen or otherwise perceived by an observer near device 100. Similarly, device 100 may include LEDs, speakers, or actuators configured and positioned to be perceived by an observer.

[0072] The input components of the eye-wearing device 100 may include alphanumeric input components (e.g., a touchscreen or touchpad configured to receive alphanumeric input, a photographic optical keyboard, or other alphanumeric-configured elements), point-based input components (e.g., a mouse, touchpad, trackball, joystick, motion sensor, or other pointing instrument), haptic input components (e.g., a push-button switch, a touchscreen or touchpad that senses the position, force, or position and force of a touch or touch gesture, or other haptic-configured elements), and audio input components (e.g., a microphone). The mobile device 401 and server system 498 may include alphanumeric, point-based, haptic, audio, and other input components.

[0073] In some examples, the eye-worn device 100 includes a collection of motion-sensing components referred to as an inertial measurement unit (IMU) 472. These motion-sensing components can be microelectromechanical systems (MEMS) with micro-moving parts, typically small enough to be part of a microchip. In some exemplary configurations, the IMU 472 includes an accelerometer, a gyroscope, and a magnetometer. The accelerometer senses the linear acceleration (including acceleration due to gravity) of the device 100 relative to three orthogonal axes (x, y, z). The gyroscope senses the angular velocity of the device 100 about three rotational axes (pitch, roll, yaw). Together, the accelerometer and gyroscope provide positioning, orientation, and motion data about the device relative to six axes (x, y, z, pitch, roll, yaw). If a magnetometer is present, it senses the heading of the device 100 relative to magnetic north. The positioning of device 100 can be determined by position sensors such as GPS unit 473, one or more transceivers for generating relative positioning coordinates, altitude sensors or barometers, and other orientation sensors. Such positioning system coordinates can also be received from mobile device 401 via low-power wireless circuit 424 or high-speed wireless circuit 436 through wireless connections 425 and 437.

[0074] IMU 472 may include, or cooperate with, a digital motion processor or program that acquires raw data from components and calculates multiple useful values ​​regarding the positioning, orientation, and motion of device 100. For example, acceleration data acquired from an accelerometer may be integrated to obtain velocity relative to each axis (x, y, z); and integrated again to obtain the positioning of device 100 (represented in linear coordinates x, y, and z). Angular velocity data from a gyroscope may be integrated to obtain the positioning of device 100 (represented in spherical coordinates). The program used to calculate these effective values ​​may be stored in memory 434 and executed by the high-speed processor 432 of the eye-wearing device 100.

[0075] The eye-worn device 100 may optionally include additional peripheral sensors, such as biometric sensors, characteristic sensors, or display elements integrated with the eye-worn device 100. For example, peripheral device elements may include any I / O components, including output components, motion components, positioning components, or any other such components described herein. For example, biometric sensors may include components that detect facial expressions (e.g., gestures, facial expressions, vocal expressions, body posture, or eye tracking), measure biosignals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), or identify a person (e.g., identification based on voice, retina, facial features, fingerprints, or electrophysiological signals such as electroencephalogram data).

[0076] Mobile device 401 may be a smartphone, tablet, laptop, access point, or any other such device capable of connecting to eye-wearing device 100 using both low-power wireless connection 425 and high-speed wireless connection 437. Mobile device 401 connects to server system 498 and network 495. Network 495 may include any combination of wired and wireless connections.

[0077] like Figure 4 The illustrated interactive augmented reality system 400 includes a computing device, such as a mobile device 401, coupled via a network to an eye-wearing device 100. The interactive augmented reality system 400 includes a memory for storing instructions and a processor for executing the instructions. The processor 432 executes the instructions of the interactive augmented reality system 400 to configure the eye-wearing device 100 to cooperate with the mobile device 401. The interactive augmented reality system 400 may utilize the memory 434 of the eye-wearing device 100 or the memory elements 540A, 540B, 540C of the mobile device 401. Figure 5 Furthermore, the interactive augmented reality system 400 may utilize the processor elements 432, 422 of the eye-wearing device 100 or the central processing unit (CPU) 530 of the mobile device 401. Figure 5 Furthermore, the interactive augmented reality system 400 can further utilize the memory and processor elements of the server system 498. In this respect, the memory and processing capabilities of the interactive augmented reality system 400 can be shared or distributed across the eye-wearing device 100, the mobile device 401, and the server system 498.

[0078] Memory 434 includes song file 482 and virtual object 484. Song file 482 includes rhythms (e.g., beat tracks) and optional note sequences and note values. A note is a symbol representing a specific pitch or other musical sound. Note values ​​include the duration of a played note relative to a rhythm and may include other qualities such as loudness, emphasis, articulation, and phrasing relative to other notes. In some implementations, the rhythm includes a default value and a user interface through which a user can select a specific rhythm to use during song playback. Virtual object 484 includes image data for identifying objects or features in images captured by camera 114. These objects may be physical features, such as known paintings or physical markers used to locate the eye-wearing device 100 within an environment.

[0079] The memory 434 further includes a location detection utility 460, a marker registration utility 462, a location utility 464, a virtual object rendering utility 466, a physics engine 468, and a prediction engine 470, all executed by the processor 432. The location detection utility 460 configures the processor 432 to determine location (position and orientation) within the environment, for example, using the location utility 464. The marker registration utility 462 configures the processor 432 to register markers within the environment. Markers can be predefined physical markers with known locations within the environment, or specific locations specified by the processor 432 relative to the environment in which the eyewear device 100 is operating, or relative to the eyewear device itself. The location utility 464 configures the processor 432 to acquire location data for determining the location of the eyewear device 100, virtual objects rendered by the eyewear device, or combinations thereof. The location data can be derived from a series of images, the IMU unit 472, the GPS unit 473, or combinations thereof. Virtual object rendering utility 466 configures processor 432 to render virtual images for display by image display 180 under the control of image display driver 442 and image processor 412. Physics engine 468 configures processor 432 to apply physical laws (such as gravity and friction) to virtual words, for example, between virtual game objects. Prediction engine 470 configures processor 432 to predict the expected movement of an object (such as eye-wearing device 100) based on its current heading, input from sensors (such as IMU 472), environmental images, or a combination thereof.

[0080] Figure 5 This is a high-level functional block diagram of an exemplary mobile device 401. Mobile device 401 includes flash memory 540A storing programs to be executed by CPU 530 to run all or a subset of the functions described herein.

[0081] The mobile device 401 may include a camera 570, which includes at least two visible light cameras (first and second visible light cameras with overlapping fields of view) or at least one visible light camera with substantially overlapping fields of view and a depth sensor. The flash memory 540A may also include a plurality of images or videos generated via the camera 570.

[0082] As shown in the figure, the mobile device 401 includes an image display 580, a mobile display driver 582 for controlling the image display 580, and a display controller 584. Figure 5 In one example, the image display 580 includes a user input layer 591 (e.g., a touchscreen) that is overlaid on top of the screen used by the image display 580 or otherwise integrated into the screen.

[0083] Examples of usable touchscreen mobile devices include (but are not limited to) smartphones, personal digital assistants (PDAs), tablets, laptops, or other portable devices. However, the structure and operation of touchscreen devices are provided by way of example; the subject matter described herein is not intended to be limited thereto. For ease of discussion, Figure 5 Therefore, a block diagram illustration of an exemplary mobile device 401 with a user interface is provided, the user interface including a touch screen input layer 591 for receiving input (touch via hand, stylus or other tool, multi-touch or gesture, etc.) and an image display 580 for displaying content.

[0084] like Figure 5 As shown, mobile device 401 includes at least one digital transceiver (XCVR) 510 for digital wireless communication via a wide-area wireless mobile communication network, shown as a WWAN XCVR. Mobile device 401 also includes additional digital or analog transceivers, such as those for communication via NFC, VLC, DECT, ZigBee, Bluetooth, etc. TM Or a short-range transceiver (XCVR) 520 for short-range network communication via Wi-Fi. For example, the short-range XCVR 520 may take the form of any available bidirectional wireless local area network (WLAN) transceiver compatible with one or more standard communication protocols implemented in a wireless local area network, such as the Wi-Fi standard compliant with IEEE 802.11.

[0085] To generate location coordinates for locating mobile device 401, mobile device 401 may include a Global Positioning System (GPS) receiver. Alternatively or additionally, mobile device 401 may utilize either or both of a short-range XCVR 520 and a WWAN XCVR 510 to generate location coordinates for positioning. For example, based on cellular networks, Wi-Fi, or Bluetooth. TM The positioning systems can generate very accurate location coordinates, especially when used in combination. These location coordinates can be transmitted to the eye-wearing device via one or more network connections through the XCVR510, 520.

[0086] Transceivers 510 and 520 (i.e., network communication interfaces) conform to one or more of the various digital wireless communication standards utilized by modern mobile networks. Examples of WWAN transceivers 510 include (but are not limited to) transceivers configured to operate according to Code Division Multiple Access (CDMA) and 3rd Generation Partnership Project (3GPP) network technologies, including, for example, but not limited to, 3GPP Type 2 (or 3GPP2) and LTE, sometimes referred to as "4G". For example, transceivers 510 and 520 provide bidirectional wireless communication of information including digitized audio signals, still images and video signals, web page information for display and web-related input, and various types of mobile messaging communications to / from mobile device 401.

[0087] Mobile device 401 also includes a microprocessor used as a central processing unit (CPU); such as Figure 4 The CPU 530 is shown in the figure. A processor is a circuit having elements constructed and arranged to perform one or more processing functions, typically various data processing functions. Although discrete logic components can be used, these examples utilize components that form a programmable CPU. A microprocessor includes, for example, one or more integrated circuit (IC) chips that incorporate electronic components that perform the functions of the CPU. For example, the CPU 530 may be based on any known or available microprocessor architecture, such as Reduced Instruction Set Computing (RISC) using the ARM architecture, as is commonly used today in mobile devices and other portable electronic devices. Of course, other arrangements of the processor circuitry can be used to form the CPU 530 or processor hardware in smartphones, laptops, and tablets.

[0088] By configuring the mobile device 401 to perform various operations, such as instructions or programs executable by the CPU 530, the CPU 530 acts as a programmable host controller for the mobile device 401. Such operations may include, for example, various general operations of the mobile device, as well as operations related to applications used on the mobile device. Although the processor can be configured using hardwired logic, a typical processor in a mobile device is a general-purpose processing circuit configured by executing programs.

[0089] Mobile device 401 includes a memory or storage system for storing programs and data. In this example, the memory system may include, as needed, flash memory 540A, random access memory (RAM) 540B, and other memory components 540C. RAM 540B serves as a short-term storage device for instructions and data processed by CPU 530, for example, as working data processing memory. Flash memory 540A typically provides long-term storage.

[0090] Therefore, in the example of mobile device 401, flash memory 540A is used to store programs or instructions executed by CPU 530. Depending on the type of device, mobile device 401 stores and runs a mobile operating system, through which specific applications are executed. Examples of mobile operating systems include Google Android, Apple iOS (for iPhone or iPad devices), Windows Mobile, Amazon Fire OS, RIM BlackBerry OS, etc.

[0091] The processor 432 within the eye-wearing device 100 can construct a map of the environment surrounding the eye-wearing device 100, determine the position of the eye-wearing device within the mapped environment, and determine the relative position of the eye-wearing device with respect to one or more objects in the mapped environment. The processor 432 can construct the map and use a Simultaneous Localization and Mapping (SLAM) algorithm applied to data received from one or more sensors to determine location and positioning information. In the context of augmented reality, the SLAM algorithm is used to construct and update a map of the environment while tracking and updating the position of the device (or user) within the mapped environment. Mathematical solutions can be approximated using various statistical methods, such as particle filtering, Kalman filtering, extended Kalman filtering, and covariance intersection.

[0092] Sensor data includes images received from one or both of cameras 114A and 114B, distances received from a laser rangefinder, positioning information received from GPS unit 473, or a combination of two or more such sensor data, or data from other sensors, providing data for determining positioning information.

[0093] Figure 6 An exemplary environment 600 and elements for natural feature tracking (NFT; for example, tracking applications using SLAM algorithms) are depicted. A user 602 of the eye-wearing device 100 exists in the exemplary physical environment 600 (in... Figure 6 In the interior room (where the image is located), the processor 432 of the eye-wearing device 100 uses the captured images to determine its position relative to one or more objects 604 within the environment 600, constructs a map of the environment 600 using the coordinate system (x, y, z) of the environment 600, and determines its position within the coordinate system. Additionally, the processor 432 determines the head pose (roll, pitch, and yaw) of the eye-wearing device 100 within the environment by using two or more location points (e.g., three location points 606a, 606b, and 606c) associated with a single object 604a, or by using one or more location points 606 associated with two or more objects 604a, 604b, and 604c. The processor 432 of the eye-wearing device 100 can also locate virtual objects 408 within the environment 600 (such as...). Figure 6(The key shown) is used for augmented reality viewing via image display 180.

[0094] Refer again Figure 6 In this example, the method for implementing the interactive augmented reality application described herein includes a virtual tag 610a associated with a virtual object 608 in environment 600. In an AR system, tags are registered at locations in the environment to assist the device in tracking and updating the positions of the user, device, and objects (virtual and physical) within the mapped environment. Sometimes tags are registered on high-contrast physical objects (such as a relatively dark object 604a mounted on a light-colored wall) to aid cameras and other sensors in detecting the tags. Tags can be pre-specified or can be specified by the eye-wearing device 100 upon entering the environment.

[0095] The tag may be encoded with information or otherwise linked to information. The tag may include location information, physical codes (such as barcodes or QR codes; visible or hidden from the user), or a combination thereof. A set of data associated with the tag is stored in the memory 434 of the eye-wearing device 100. This set of data includes information about the tag 610a, the tag's location (position and orientation), one or more virtual objects, or a combination thereof. Tag location may include the three-dimensional coordinates of one or more tag landmarks 616a, such as... Figure 6 The corners of the roughly rectangular marker 610a are shown. Marker positioning can be represented relative to real-world geographic coordinates, the marker coordinate system, the positioning of the eye-wearing device 100, or other coordinate systems. The one or more virtual objects associated with marker 610a can include any material from a variety of sources, including still images, videos, audio, haptic feedback, executable applications, interactive user interfaces and experiences, and combinations or sequences of such materials. In this context, any type of content that can be stored in memory and retrieved upon encountering marker 610a or associated with the specified marker can be classified as a virtual object. For example, Figure 6 The key 608 shown is a virtual object displayed as a 2D or 3D static image at the marked location.

[0096] In one example, marker 610a may be registered in memory as located at physical object 604a (e.g., Figure 6 The marker is located near and associated with the framed artwork shown. In another example, the marker may be registered in memory as a specific location relative to the eye-wearing device 100.

[0097] Figure 7Flowchart 700 depicts a method for implementing the augmented reality applications described herein on a wearable device (e.g., an eye-wearing device), wherein the application includes providing virtual beams of light corresponding to the user's gaze in a virtual game that can be visualized via the eye-wearing device. Although these steps are described herein with reference to eye-wearing device 100, those skilled in the art will understand from the description herein that other specific implementations of the described steps are for other types of devices. Additionally, it is conceivable that in Figure 7 One or more steps shown in the figures and described herein may be omitted, performed simultaneously or sequentially, performed in a different order than those shown and described, or performed in combination with additional steps.

[0098] At box 702, eye-wearing device 100, or 1002, 1102, or 1202 in the examples described below, monitors the positioning of eye-wearing device 100 and the corresponding first target, for example, Figure 10A The processor 432 can continuously receive input images from the visible light camera 114 and store these images in the memory 434 for processing. Additionally, the eye-wearing device 100 can capture information from other sensors (e.g., location information from the GPS unit 473, orientation information from the IMU 472, or distance information from a laser distance sensor). The eye-wearing device can also utilize wireless transmissions such as short-range transceivers (Bluetooth). TM Location information can be determined by communicating with other devices, such as eye-wearing devices, via Bluetooth Low Energy (BLE) and wireless wide area network, local area network, or wide area network transceivers (e.g., cellular or Wi-Fi). Alternatively, location information can be obtained using a location utility 464 of the user's or the first target's eye-wearing device.

[0099] At box 704, the eye-wearing device 100 identifies the head / face of the first target 1006 in the captured image. The head / face can be identified by recognizing the eye-wearing device worn by the first target. This can be accomplished by transmitting positioning or location information via the wireless circuitry of the eye-wearing device. Alternatively or additionally, the head / face can be identified by comparing the received input image with known faces or with a shape library of general facial shapes, for example using image recognition software such as IBM Watson Visual Recognition, available from IBM in Armonk, NY.

[0100] At box 706, the eye-wearing device 100 monitors the positioning of the user's eyes 1003 or the positioning 1002 of the user's eyewear, or both, and the corresponding head / face of the first target 1006. This can be performed by peripheral biometric sensors of the eye-wearing device 100, such as eye-tracking sensors, which can be used to monitor the positioning of the user's eyes 1003. At box 708, the eye-wearing device 100 calculates at least one virtual beam extending from the user's eyes, or at least one beam extending from the user's eyewear, or both. The calculation of the beams can occur within the interactive augmented reality system 400 using the system's processor 432 or the processor elements 432, 422 of the eye-wearing device 100.

[0101] At box 710, the eye-wearing device 100 generates an overlay image including the calculated beam for display on the user's eyewear. This allows the user to visualize the direction of their gaze or the orientation of the eyewear relative to the first target 1006 in the environment. The overlay image may be generated by the image processor 412 of the eye-wearing device 100. At box 712, the eye-wearing device 100 determines a score based on the positioning of at least one of the at least one virtual beam relative to the first target. The user's score may increase upon hitting the first target 1006. A hit for which the score increases may include an intersection of the virtual beam with the face or head of the first target 1006. Score determination may be performed by the processor 432 of the eye-wearing device 100.

[0102] Figure 8 Flowchart 800 depicts a method for implementing the augmented reality applications described herein on wearable devices (e.g., eye-wearing devices), wherein the applications include providing virtual beams corresponding to the gaze of a user or target / opponent in a virtual game that can be visualized through an eye-wearing device. Figure 8 The steps shown include steps 802, 804, 806, 810, 814, and 816, which are similar to... Figure 7 Steps 702, 704, 706, 708, 710, and 712 are shown. Figure 8The process includes an additional step 808 of monitoring the location of the first target's eye or the location of the first target's eyewear, or both, and an additional step 812 of calculating at least one third virtual beam extending from the target's eye, or at least one third virtual beam extending from the target's eyewear, or both. This monitoring can be performed by peripheral biometric sensors of the eyewear device 100, such as eye-tracking sensors, which can be used to monitor the location of the first target 1006's eye. The calculation of the beam can be performed by the processor 432 of the eyewear device 100. These steps reflect an example where users and targets can score points in a game by hitting target areas with the calculated beams, thereby increasing their respective scores or decreasing their opponents' scores. The target areas, based on the points allocated for hitting, are determined by game rules, which can be implemented programmatically by the eyewear device.

[0103] Figure 9 Flowchart 900 depicts a method for implementing the augmented reality applications described herein on wearable devices (e.g., eye-wearing devices), wherein the applications include providing virtual beams corresponding to the gaze of a user or target / opponent in a virtual game that can be visualized through an eye-wearing device. Figure 9 The steps shown include steps 902, 904, 906, 910, 912, 914, and 916, which are similar to... Figure 8 Steps 802, 804, 806, 810, 812, 814, and 816 are shown. Figure 9 The additional step 908 includes receiving data including the location of the first target eye or the location of the first target eyewear, or both. This data can be received using the wireless capabilities of the eyewear device 100, and can be obtained from a biometric sensor on the first target eyewear device 1007. This example may involve data transmission from an eyewear device worn by the first target, providing information about the location of the first target eye or the eyewear device, which may replace or supplement the location determined based on analysis of received images. As discussed, this may involve the wireless transmission of data as described above.

[0104] Figure 10A This is a perspective view of a virtual augmented reality experience using a virtual eye-wearing beam, where user 1001 is wearing eye-wearing device 1002, which can be the aforementioned eye-wearing device 100. Based on the above... Figure 7 According to the method steps, in step 702, the eye-wearing device 1001 is configured to monitor the positioning of the eye-wearing device 100 and the corresponding first target 1006. The user 1001 can guide the eye-wearing device 1002 or focus their eyes on 1003 in directions 1004, 1005. The first target 1006 may be wearing the eye-wearing device 1007. The eye-wearing device 1002 is configured to monitor its own positioning and the positioning of the first target 1006. (As described above...) Figure 7 As discussed in box 704, the eye-wearing device 1002 is configured to identify the head / face of a first target in the captured image.

[0105] Figure 10B The perspective of a user 1001 wearing an eye-worn device 1002 is depicted. The lens and display device allow the user to view the physical environment within the context of the game, including a primary target 1006 or an opponent. The eye-worn device 1002 calculates, according to step 708, at least one virtual beam extending from the user's eyes, or at least one beam extending from the user's eye-worn device, or both. The eye-worn device 1002 generates an overlay image including the virtual beam 1008 for display on the user's eye-worn device according to step 710. The virtual beam 1008... Figure 10B As shown in the image, the score of user 1009 and the score of opponent 1010, which can be the first target 1006, can also be displayed by the eye-wearing device.

[0106] Figure 10C An example is shown in which the orientations 1004 and 1005 of the user's eye-wearing device 1002 are aligned with the user's eye orientation 1003, and the user's gaze is directed toward the face of the first target 1006. Figure 10D The image shows the perspective of a user 1001 wearing the eye-worn device 1002. A virtual beam of light 1008 is displayed, and it is shown intersecting with the face of a first target 1006. In this example, the user's score has increased due to hitting the first target 1006 or an opponent. The score for a hit can be predetermined according to pre-defined game scoring rules.

[0107] Figure 11A An example is shown in which user 1101 points towards eye-wearing device 1102 in a first direction 1108, but the user's eyes are looking towards a second direction 1103. In this example, the user's eye gaze directions 1104, 1005 intersect with the face of a first target 1106. The first target may be wearing eye-wearing device 1107. Figure 11B The image shows user 1101 pointing eye-wearing device 1102 toward a third direction 1108 that aligns with the face of the first target 1106, but the user's eyes are looking toward a fourth direction 1104, 1105. Figure 11C A first virtual beam 1110 and a second virtual beam 1111 displayed on a user's eye-wearing device are shown. The first and second virtual beams are respectively associated with the direction of the eye-wearing device or the direction of the user's gaze. The first virtual beam 1110 reflects a hit on a first target 1106 or an opponent in the game. The user's score 1112 and the opponent's score 1113 are displayed in the eye-wearing device 1102. In this case, the user's score is shown as an increase due to a hit on the first target 1106 or an opponent.

[0108] Figure 12A An example is shown in which the eye-wearing device 1207 of the first target 1206 is pointed in a direction 1216 toward the user 1201. In this example, the first target's eye gaze 1217 is in directions 1214, 1215 toward the ground. In this example, the user's eye-wearing device 1202 determines the direction 1216 of the first target's eye-wearing device 1207 and the directions 1214, 1215 of the first target's eye gaze 1217. These directions may be determined by analysis of images received by the user's eye-wearing device 1202, or by means of wireless signals received from the eye-wearing device 1217 of the first target 1206, or a combination thereof. Figure 12B The first target is shown to guide the eye to gaze at 1217 on third direction 1214, 1215, while the direction of the eye-wearing device 1207 is pointed to fourth direction 1216.

[0109] Figure 12C It shows in Figure 12A In this scenario, the virtual beam 1219, corresponding to the direction 1216 of the eye-wearing device 1207, intersects with the face of the user 1202. The virtual beam 1219 includes an indication 1220 that a first target has hit the user 1201. The user score 1221 and the opponent's score 1222 are displayed in the eye-wearing device 1202. The user 1201 can deduct points as shown, or can add points to the opponent's score as a result of a hit. The virtual beam 1218, corresponding to the eye directions 1214 and 1215 pointing towards the ground, is shown as an indication of no hit.

[0110] As described herein, any function of the eye-wearing device 100, mobile device 401, and server system 498 can be embodied in one or more computer software applications or sets of programming instructions. According to some examples, a “function,” “application,” “instruction,” or “program” is a program that performs the functions defined in the program. Various programming languages ​​can be used to develop one or more applications that are structured in various ways, such as object-oriented programming languages ​​(e.g., Objective-C, Java, or C++) or procedural programming languages ​​(e.g., C or assembly language). In a particular example, a third-party application (e.g., an entity other than a platform-specific vendor using Android) may be used. TM or iOS TM Applications developed using a Software Development Kit (SDK) can be included in mobile operating systems such as iOS. TM ANDROID TM , A mobile phone or mobile software running on another mobile operating system. In this example, a third-party application may invoke API calls provided by the operating system to facilitate the functionality described herein.

[0111] Therefore, machine-readable media can take many forms of tangible storage media. Non-volatile storage media include, for example, optical discs or disks, any storage device such as any computer device, such as client devices, media gateways, code converters, etc., that can be used to implement the figures shown. Volatile storage media include dynamic memory, such as the main memory of computer platforms. Tangible transmission media include coaxial cables; copper wires and optical fibers, including wires that form buses within a computer system. Carrier transmission media can take the form of electrical or electromagnetic signals, or sound or light waves, such as those generated during radio frequency (RF) and infrared (IR) data communications. Therefore, common forms of computer-readable media include, for example: floppy disks, floppy disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, DVDs or DVD-ROMs, any other optical media, punched card tapes, any other physical storage media with a perforated pattern, RAM, PROMs and EPROMs, FLASH-EPROMs, any other memory chips or cartridges, carrier waves for transmitting data or instructions, cables or links for transmitting such carrier waves, or any other medium from which a computer can read program code or data. Many of these forms of computer-readable media can be used to carry one or more sequences of one or more instructions to a processor for execution.

[0112] In addition to what has just been stated above, whether or not it is stated in the claims, the stated or described content is not intended or should not be construed as causing any part, step, feature, object, benefit, advantage or equivalent to be offered to the public.

[0113] It should be understood that, unless otherwise specified herein, the terms and expressions used herein have the general meaning consistent with those in the corresponding fields of investigation and research. Relational terms such as “first” and “second” are used only to distinguish one entity or action from another, and do not necessarily require or imply any actual such relationship or order between these entities or actions. The terms “comprising,” “including,” “containing,” “having,” or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes or comprises a list of elements or steps includes not only those elements or steps, but may also include other elements or steps not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element prefixed with “a” or “an” does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes that element.

[0114] Unless otherwise stated, any and all measurements, values, ratings, positions, quantities, dimensions, and other specifications set forth in this specification, including those in the appended claims, are approximate, not precise. Such quantities are intended to have a reasonable range consistent with the functions they relate to and the conventions in the fields to which they pertain. For example, unless otherwise expressly stated, parameter values, etc., can vary from said quantity or range by up to plus or minus ten percent.

[0115] Furthermore, as can be seen in the foregoing specific embodiments, various features have been combined in various examples for the purpose of simplifying this disclosure. The disclosed method should not be construed as reflecting an intention to require more features than expressly recited in each claim in the claimed examples. Rather, as reflected in the following claims, the claimed subject matter lies in fewer features than in any single disclosed example. Therefore, the following claims are hereby incorporated into the specific embodiments, wherein each claim exists independently as a separately claimed subject matter.

[0116] While examples considered to be best practices and other examples have been described above, it should be understood that various modifications may be made therein, and the subject matter disclosed herein can be implemented in various forms and examples, and is applicable to many applications, of which only some have been described herein. The appended claims are intended to claim protection for any and all modifications and variations falling within the true scope of the inventive concept.

Claims

1. An augmented reality system, comprising: Positioning detection system; Camera system; Display system; Eye-tracking system; An eye-wearing device, the eye-wearing device including the positioning detection system, the display system, the eye tracking system, a processor, and a memory; and The program in the memory, wherein the processor executes the program to configure the eye-wearing device to perform functions, including functions for the following: The positioning detection system is used to register the target positioning and the positioning of the eye-wearing device; The camera system of the eye-wearing device is used to capture at least one image of the target; Identify at least one part of the target in the at least one image; Monitor the target location, which includes the location of at least one eye of the target or the location of the target's eye-wearing device; Monitor at least one user location, the user location including the location of at least one eye of the user wearing the eye-wearing device, or the location of the user wearing the eye-wearing device; Calculate at least one virtual beam extending from the at least one user location to the target location; Calculate at least one virtual beam extending from the at least one target location to the user location; Present at least one virtual image reflecting one direction of the at least one virtual beam for display by the eye-wearing device; as well as Determine whether the at least one virtual beam intersects with at least one portion of the target.

2. The system of claim 1 further includes the function of determining a score based on whether one or more of the calculated beams intersect the target.

3. The system according to claim 1 further includes the function of receiving data for the target location.

4. The system of claim 1, wherein the camera system is configured to capture a sequence of video data frames, wherein each video data frame includes depth information of a plurality of pixels.

5. The system according to claim 1, wherein the display system comprises: A see-through display, supported by the eye-wearing device, to present the at least one virtual beam of light.

6. An augmented reality method for use with an eye-worn device, the eye-worn device having a positioning detection system, a camera system, and a display system, the method comprising: The positioning detection system is used to register the target positioning and the positioning of the eye-wearing device; At least one image of the target is captured using the eye-wearing device; Identify at least one part of the target in the at least one image; Monitor the target location, which includes the location of at least one eye of the target or the location of the eye-wearing device of the target; Monitor at least one user location, the user location including the location of at least one eye of the user wearing the eye-wearing device, or the location of the user wearing the eye-wearing device; Calculate at least one virtual beam extending from the at least one user location to the target location; Calculate at least one virtual beam extending from the at least one target location to the user location; A virtual image reflecting one direction of the at least one computed beam is presented for display by the eye-wearing device; as well as Determine whether the at least one virtual beam intersects with at least one portion of the target.

7. The augmented reality method according to claim 6, further comprising: The score is determined based on whether one or more of the calculated beams intersect with the target.

8. The augmented reality method according to claim 6, further comprising: Receive the target location data.

9. The augmented reality method of claim 6, wherein the camera system is configured to capture a sequence of video data frames, wherein each video data frame includes depth information of a plurality of pixels.

10. The augmented reality method of claim 6, wherein the display system comprises: A see-through display, supported by the eye-wearing device, to present the at least one virtual beam of light.

11. A non-transitory computer-readable medium storing program code, which, when executed, causes an electronic processor to perform the following steps: registering target positioning and eye-wearing device positioning using a positioning detection system of an eye-wearing device; The camera system of the eye-wearing device is used to capture at least one image of the target; Identify at least one part of the target in the at least one image; Monitor the target location, which includes the location of at least one eye of the target or the location of the eye-wearing device of the target; Monitor at least one user location, the user location including the location of at least one eye of the user wearing the eye-wearing device, or the location of the user wearing the eye-wearing device; Calculate at least one virtual beam extending from the at least one user location to the target location; Calculate at least one virtual beam extending from the at least one target location to the user location; A virtual image reflecting one direction of the at least one computed beam is presented for display by the eye-wearing device; as well as Determine whether the at least one virtual beam intersects with at least one portion of the target.

12. The non-transitory computer-readable medium storing program code according to claim 11, wherein the program code, when executed, causes an electronic processor to perform the additional step of determining a score based on whether one or more of the calculated beams intersect the target.

13. The non-transitory computer-readable medium storing program code according to claim 11, wherein the program code, when executed, causes the electronic processor to perform the additional step of receiving data for at least one target location.

14. The non-transitory computer-readable medium storing program code according to claim 11, wherein the program code, when executed, causes the electronic processor to perform the additional step of: capturing a sequence of video data frames using the camera system, wherein each video data frame includes depth information of a plurality of pixels.

Images