Virtual tasting and guided tours for augmented reality experiences
By employing computer vision and object tracking algorithms, augmented reality systems on wearable devices provide immersive virtual product tastings and guided tours, addressing the limitations of existing AR technologies through enhanced interaction and realism.
Patent Information
- Application Number
- KR1020237041978
- Authority / Receiving Office
- KR · KR
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2021-05-07
- Filing Date
- 2022-05-05
- Publication Date
- 2026-11-06
- Estimated Expiration
- 2042-05-05
AI Technical Summary
Existing augmented reality technologies lack immersive and interactive experiences for virtual product tastings and guided tours, particularly in the context of wearable devices like eyewear, which fail to effectively combine real-world objects with virtual content.
The use of computer vision and object tracking algorithms to detect containers in a physical environment, retrieve relevant data, and present contextual overlays and guided tours through avatars on wearable devices, enhancing interaction with virtual tours using touch-sensitive input devices and stereo cameras for 3D projections.
Enables immersive and interactive virtual product tastings and guided tours by providing perceptually rich experiences that enhance user interaction and navigation, leveraging advanced AR technologies for enhanced immersion and realism.
Smart Images

Figure R1020237041978_ABST
Abstract
Description
Technology Field This application claims priority to U.S. Provisional Application No. 63 / 185,406 filed on May 7, 2021, the contents of which are incorporated herein by reference.
[0001] The examples presented in this disclosure relate to the field of augmented reality experiences for electronic devices, including wearable devices such as eyewear. More specifically, this disclosure describes, but is not limited to, presentations of virtual product tastings and guided tours of related facilities in augmented reality. Background Technology
[0002] Many types of computers and electronic devices available today, such as mobile devices (e.g., smartphones, tablets, laptops), handheld devices, and wearable devices (e.g., smart glasses, digital eyewear, headwear, headgear, and head-mounted displays), include various cameras, sensors, wireless transceivers, input systems, and displays.
[0003] Virtual reality (VR) technology creates a complete virtual environment containing realistic images, which are sometimes presented on VR headsets or other head-mounted displays. VR experiences allow users to move through the virtual environment and interact with virtual objects. Augmented reality (AR) is a type of VR technology that combines real objects within a physical environment with virtual objects and displays that combination to the user. The combined display gives the impression that virtual objects actually exist within the environment, particularly when the virtual objects appear and behave like real objects. Cross reality (XR) is generally understood as a comprehensive term referring to systems that include or combine elements from AR, VR, and mixed reality (MR) environments.
[0004] Graphic user interfaces, including virtual objects and icons, graphic elements such as taskbars, list boxes, menus, and buttons, and selection control elements such as cursors, pointers, handles, and sliders, enable a user to interact with displayed content.
[0005] Automatic Speech Recognition (ASR) is a field of computer science, artificial intelligence, and linguistics that involves receiving spoken words and converting them into audio data suitable for processing by computing devices. Frames of processed audio data can be used to convert received spoken words into text or to convert spoken words into commands for controlling and interacting with various software applications. ASR processing can be used by computers, handheld devices, wearable devices, telephone systems, automobiles, and a wide variety of other devices to facilitate human-computer interactions. Brief explanation of the drawing
[0006] The characteristics of the various examples described will be easily understood from the following detailed description, in which drawings are referenced. Reference numbers are used with each element in the description and throughout several drawings. When multiple similar elements exist, a single reference number may be assigned to similar elements, with an added lowercase letter designating a specific element.
[0007] The various elements depicted in the drawings are not drawn in a constant proportion unless otherwise indicated. The dimensions of the various elements may be enlarged or reduced for clarity. Some drawings depict one or more implementations and are presented merely as examples and should not be interpreted as limiting. The drawings include the following:
[0008] FIG. 1a is a side view (right) of an exemplary hardware configuration of an eyewear device suitable for use in an exemplary virtual guided tour system.
[0009] FIG. 1b is a cross-sectional view of the right corner of the eyewear device of FIG. 1a, depicting the right visible light camera and the circuit board.
[0010] FIG. 1c is a side view (left) of an exemplary hardware configuration of the eyewear device of FIG. 1a showing a left visible light camera.
[0011] FIG. 1d is a cross-sectional view of the left corner of the eyewear device of FIG. 1c, depicting the left visible light camera and the circuit board.
[0012] FIGS. 2a and 2b are rear views of exemplary hardware configurations of an eyewear device used in an exemplary virtual guided tour system.
[0013] Figure 3 is a schematic depiction of a three-dimensional scene, a left raw image captured by a left visible light camera, and a right raw image captured by a right visible light camera.
[0014] FIG. 4 is a functional block diagram of an exemplary virtual guided tour system including an eyewear device and a server system connected through various networks.
[0015] FIG. 5 is a schematic representation of an exemplary hardware configuration for a mobile device suitable for use in the exemplary virtual guided tour system of FIG. 4.
[0016] Figure 6 is a schematic example of a user in an exemplary environment to be used to describe simultaneous localization and mapping.
[0017] Figure 7 is a visual example of an exemplary avatar and situational overlay on a display.
[0018] Figure 8 is a cross-sectional example of an avatar presenting a virtual guided tour on a display.
[0019] Figure 9 is a flowchart listing the steps of an exemplary method for presenting a virtual guided tour experience on a display.
[0020] FIG. 10 is a flowchart listing the steps of an exemplary method for presenting a virtual guided tour experience in response to detecting an action on a display. Specific details for implementing the invention
[0021] Various implementations and details regarding examples for presenting virtual product tastings and virtual guided tours of product-related facilities in augmented reality are described. For instance, containers, such as wine bottles, are detected in video data captured by an eyewear device using computer vision and machine-trained classification models. Relevant data regarding the detected containers (and their contents) is retrieved from libraries and internet searches. Contextual overlays based on the retrieved data are presented on the display along with avatars for playing messages and presenting the guided tours.
[0022] Exemplary methods include using an eyewear device to detect a container in a physical environment, presenting a contextual overlay, and initiating a virtual tour experience. The eyewear device includes a camera system, a microphone, a loudspeaker, a guided tour application, an image processing system, and a display. The method includes detecting a first action in frames of captured video data and, in response, initiating a virtual tour experience—which includes presenting video on a display in correlation with a lesson presented through an avatar. The virtual tour experience also includes animating an avatar to perform one or more demonstrations in correlation with the lesson. A graphical control element controls the progress of the tour experience. In the context of traditionally manufactured beverages, the detected container contains a beverage product such as wine, beer, spirits, or cider. The virtual guided tour experience is associated with a site such as a winery, brewery, spirits distillery, or cider distillery, and may include farm tours, factory tours, or facility tours.
[0023] Although various systems and methods related to containers and beverages are described in this specification, the described technology can be applied to detect any type of container or product in a physical environment, retrieve data regarding the detected product, and present a virtual guided tour on a display.
[0024] The following detailed description includes systems, methods, techniques, instruction sequences, and computing machine program products that exemplify the examples presented in this disclosure. Many details and examples are included to provide a complete understanding of the disclosed claims and their related teachings. However, those skilled in the art can understand how to apply the related teachings without such details. The embodiments of the disclosed claims are not limited to the specific devices, systems, and methods described, because the related teachings can be applied or practiced in various ways. The terms and nomenclature used in this specification are used merely to describe specific embodiments and are not intended to be limiting. In general, well-known instruction instances, protocols, structures, and techniques are not necessarily presented in detail.
[0025] As used herein, the terms “combined” or “connected” refer to any logical, optical, physical, or electrical connection, including links through which electrical or magnetic signals generated or supplied by one system element are transmitted to another combined or connected system element. Unless otherwise described, combined or connected elements or devices are not necessarily directly connected to one another and may be separated by intermediate components, elements, or communication media, one or more of which may modify, manipulate, or transmit electrical signals. The term “on” means being directly supported by an element or being indirectly supported by an element through another element that is integrated into or supported by that element.
[0026] The term "proximal" is used to describe an item or part of an item that is located near, adjacent to, or beside an object or person, or that is relatively closer to other parts of the item compared to other parts of the item, which may be described as "distal." For example, the end of an item closest to an object may be referred to as the proximal end, whereas the end generally opposite may be referred to as the distal end.
[0027] The orientations of eyewear devices, other mobile devices, combined components, and any other devices, such as those depicted in any of the drawings, are given merely as examples for the sake of illustration and discussion. In operation, the eyewear device may be oriented in any other direction suitable for the specific application of the eyewear device; for example, upward, downward, sideways, or any other orientation. Additionally, as used herein, any directional terms such as forward, rear, inner, outer, facing, left, right, horizontal, vertical, up, down, upper, lower, top, bottom, bottom, side, horizontal, vertical, and diagonal are used merely as examples and are not limiting to the orientation or orientation of any camera, inertial measurement unit, or display configured or otherwise described herein.
[0028] Advanced AR technologies, such as computer vision and object tracking, can be used to create perceptually rich and immersive experiences. Computer vision algorithms extract three-dimensional data about the physical world from data captured in digital images or videos. Object recognition and tracking algorithms are used to detect objects in digital images or videos, estimate their orientation or pose, and track their movement over time. Recognizing and tracking hands and fingers in real time is one of the most challenging and processing-intensive tasks in the field of computer vision.
[0029] The term "pose" refers to the static position and orientation of an object at a specific moment. The term "gesture" refers to the active movement of an object, such as a hand, through a series of poses to convey a signal or idea. The terms pose and gesture are sometimes used interchangeably in the fields of computer vision and augmented reality. As used herein, the terms "pose" or "gesture" (or variations thereof) are intended to include both poses and gestures; in other words, the use of one term does not exclude the other.
[0030] Additional purposes, benefits, and novel features of the examples will be presented in part in the following description and will become apparent to those skilled in the art upon review of the following and the accompanying drawings, or can be learned by the production or operation of the examples. The purposes and benefits of the subject matter of the claims may be realized and achieved by the methods, means, and combinations specifically indicated in the appended claims.
[0031] Now, refer in detail to the examples illustrated in the attached drawings and discussed below.
[0032] FIG. 1a is a side view (right) of an exemplary hardware configuration of an eyewear device (100) including a touch-sensitive input device such as a touchpad (181). As illustrated, the touchpad (181) may have a boundary that includes raised or otherwise tactile edges that are clearly visible or provide feedback to the user regarding the location and boundaries of the touchpad (181); alternatively, the boundary may be fine and not clearly visible or tactile. In other embodiments, the eyewear device (100) may include a touchpad (181) on the left side that operates together with or independently of the touchpad (181) on the right side.
[0033] The surface of the touchpad (181) is configured to detect finger touches, taps, and gestures (e.g., movable touches) for use with a GUI displayed by the eyewear device on an image display, thereby allowing the user to navigate and select menu options in an intuitive manner, which enhances and simplifies the user experience.
[0034] Detection of finger inputs on the touchpad (181) can enable various functions. For example, touching anywhere on the touchpad (181) can cause the GUI to display or highlight an item on the image display, which can be projected onto at least one of the optical assemblies (180A, 180B). Tapping or double-tapping the touchpad (181) can select an item or icon. Sliding or swiping a finger in a specific direction (e.g., from front to back, from back to front, from top to bottom, or from bottom to top) can cause items or icons to slide or scroll in that specific direction to move, for example, to the next item, icon, video, image, page, or slide. Sliding a finger in a different direction can cause items or icons to slide or scroll in the opposite direction to move, for example, to the previous item, icon, video, image, page, or slide. The touchpad (181) can be located virtually anywhere on the eyewear device (100).
[0035] In one example, an identified finger gesture of a single tap on the touchpad (181) initiates the selection or pressing of a graphic user interface element within an image presented on an image display of the optical assembly (180A, 180B). Adjustment of the image presented on the image display of the optical assembly (180A, 180B) based on the identified finger gesture may be a primary action of selecting or submitting a graphic user interface element on the image display of the optical assembly (180A, 180B) for additional display or execution.
[0036] As described, the eyewear device (100) includes a right visible light camera (114B). As further described herein, the two cameras (114A, 114B) capture image information of a scene from two distinct perspectives. The two captured images can be used to project a three-dimensional display onto an image display for viewing with 3D glasses.
[0037] The eyewear device (100) includes a right optical assembly (180B) having an image display for presenting images such as depth images. As illustrated in FIGS. 1a and 1b, the eyewear device (100) includes a right visible light camera (114B). The eyewear device (100) may include a plurality of visible light cameras (114A, 114B) forming a passive type three-dimensional camera such as a stereo camera, wherein the right visible light camera (114B) is located at the right corner (110B). As illustrated in FIGS. 1c and 1d, the eyewear device (100) also includes a left visible light camera (114A).
[0038] The left and right visible light cameras (114A, 114B) are sensitive to wavelengths in the visible light range. Each of the visible light cameras (114A, 114B) has an overlapping field of view facing a different front to enable the generation of three-dimensional depth images, for example, the right visible light camera (114B) describes the right field of view (111B). Generally, a "field of view" is a portion of a scene seen through a camera at a specific position and orientation in space. The fields of view (111A and 111B) have an overlapping field of view (304) (Fig. 3). When the visible light cameras capture an image, objects or object features outside the field of view (111A, 111B) are not recorded in the raw image (e.g., a photograph or drawing). The field of view describes an angular range or degree in which the image sensors of the visible light cameras (114A, 114B) pick up electromagnetic radiation of a given scene in a captured image of a given scene. The field of view can be expressed as the angular size of the view cone; that is, the angle of view. The angle of view can be measured horizontally, vertically, or diagonally.
[0039] In an exemplary configuration, one or both of the visible light cameras (114A, 114B) have a field of view of 100° and a resolution of 480x480 pixels. The “angle of coverage” describes the range of angles that the lens of the visible light cameras (114A, 114B) or the infrared camera (410) (see FIG. 2a) can effectively image. Typically, the camera lens produces an image circle large enough to completely cover the camera’s sensor or film, possibly including some vignetting (e.g., the image becoming darker toward the edges compared to the center). If the camera lens’s angle of coverage does not fill the sensor, an image circle with typically strong vignetting toward the edges will be seen, and the effective field of view will be limited by the angle of coverage.
[0040] Examples of such visible light cameras (114A, 114B) include high-resolution CMOS (complementary metal-oxide-semiconductor) image sensors and digital VGA cameras (video graphics arrays) capable of resolutions of 480p (e.g., 640x480 pixels), 720p, 1080p or higher. Other examples include visible light cameras (114A, 114B) capable of capturing high-definition (HD) video at a high frame rate (e.g., 30 to 60 frames per second or higher) and storing recordings at a resolution of 1216x1216 pixels (or higher).
[0041] The eyewear device (100) can capture image sensor data from visible light cameras (114A, 114B) along with geolocation data, which are digitized by an image processor and stored in memory. The visible light cameras (114A, 114B) capture individual left and right raw images in a two-dimensional spatial region comprising a matrix of pixels in a two-dimensional coordinate system including an X-axis for horizontal position and a Y-axis for vertical position. Each pixel includes a color attribute value (e.g., red pixel light value, green pixel light value, or blue pixel light value); and a position attribute (e.g., X-axis coordinate and Y-axis coordinate).
[0042] To capture stereo images for later display as a 3D projection, an image processor (412) (shown in FIG. 4) may be coupled to visible light cameras (114A, 114B) to receive and store visual image information. The image processor (412) or another processor may control the operation of the visible light cameras (114A, 114B) to act as stereo cameras simulating human binocular vision and add a timestamp to each image. The timestamp for each pair of images enables the images to be displayed together as part of a 3D projection. 3D projections provide a desirable immersive and realistic experience in various situations, including virtual reality (VR) and video games.
[0043] FIG. 1b is a partial cross-sectional view of the right corner (110B) of the eyewear device (100) of FIG. 1a, depicting the right visible light camera (114B) of the camera system and the circuit board. FIG. 1c is a side view (left) of an exemplary hardware configuration of the eyewear device (100) of FIG. 1a, showing the left visible light camera (114A) of the camera system. FIG. 1d is a partial cross-sectional view of the left corner (110A) of the eyewear device of FIG. 1c, depicting the left visible light camera (114A) of the three-dimensional camera and the circuit board.
[0044] The configuration and arrangement of the left visible light camera (114A) are substantially similar to the right visible light camera (114B), except that the connections and couplings are located on the left side (170A). As illustrated in the example of FIG. 1b, the eyewear device (100) includes a right visible light camera (114B) and a circuit board (140B), which may be a flexible printed circuit board (PCB). A right hinge (126B) connects the right corner (110B) to the right temple (125B) of the eyewear device (100). In some examples, components of the right visible light camera (114B), the flexible PCB (140B), or other electrical connectors or contacts may be located on the right temple (125B) or the right hinge (126B). The left hinge (126B) connects the left corner (110A) to the left temple (125A) of the eyewear device (100). In some examples, components of a left visible light camera (114A), a flexible PCB (140A), or other electrical connectors or contacts may be located on the left temple (125A) or the left hinge (126A).
[0045] The right corner (110B) includes a corner body (190) and a corner cap, the corner cap being omitted in the cross-section of FIG. 1b. Inside the right corner (110B), various interconnected circuit boards are arranged, such as PCBs or flexible PCBs, including a right visible light camera (114B), microphone(s) (139), loudspeaker(s) (191), and controller circuits for low-power wireless circuits (e.g., for wireless short-range network communication via Bluetooth™) and high-speed wireless circuits (e.g., for wireless short-range network communication via Wi-Fi).
[0046] The right visible light camera (114B) is coupled to or placed on a flexible PCB (140B) and is covered by a visible light camera cover lens that is aimed through an opening(s) formed in the frame (105). For example, the right rim (107B) of the frame (105) shown in FIG. 2a is connected to the right corner (110B) and includes an opening(s) for the visible light camera cover lens. The frame (105) includes a front surface configured to face outward away from the user's eyes. An opening for the visible light camera cover lens is formed on and through the front or outward-facing surface of the frame (105). In this example, the right visible light camera (114B) has an outward-facing field of view (111B) (shown in FIG. 3) having a line of sight or perspective corresponding to the right eye of the user of the eyewear device (100). The visible light camera cover lens may be attached to the front or outward-facing surface of the right corner (110B), which has an outward-facing coverage angle and an opening formed in a different outward direction. The connection may also be made indirectly through interposed components.
[0047] As illustrated in FIG. 1b, the flexible PCB (140B) is placed inside the right corner (110B) and coupled to one or more other components accommodated in the right corner (110B). Although shown as being formed on the circuit boards of the right corner (110B), the right visible light camera (114B) may be formed on the circuit boards of the left corner (110A), the temples (125A, 125B), or the frame (105).
[0048] FIGS. 2a and 2b are rear perspective views of an exemplary hardware configuration of an eyewear device (100) comprising two different types of image displays. The eyewear device (100) is sized and shaped to be configured for a user to wear; in this example, the form of glasses is illustrated. The eyewear device (100) may take other forms and may incorporate other types of frameworks; for example, headgear, a headset, or a helmet.
[0049] In the example of eyeglasses, the eyewear device (100) comprises a frame (105) including a left frame (107A) connected to a right frame (107B) via a bridge (106) adapted to be supported by the user's nose. The left and right frames (107A, 107B) each include individual apertures (175A, 175B) that hold individual optical elements (180A, 180B), such as lenses and display devices. As used herein, the term “lens” is intended to include pieces of transparent or translucent glass or plastic having curved and flat surfaces that cause light to converge or diverge, or cause little or no convergence or divergence.
[0050] FIG. 2a is an exemplary hardware configuration for an eyewear device (100) in which the right corner (110B) supports a microphone (139) and a loudspeaker (191). The microphone (139) includes a transducer that converts sound into a corresponding electrical audio signal. In this example, the microphone (139) is positioned with an opening facing inward toward the wearer to facilitate the reception of sound waves, such as human speech, including verbal commands and queries, as illustrated. Additional or differently oriented openings may be implemented. In other exemplary configurations, the eyewear device (100) is coupled to one or more microphones (139) positioned at various locations on the eyewear device (100), configured to operate together or independently.
[0051] The loudspeaker (191) includes an electroacoustic transducer that converts an electric audio signal into a corresponding sound. The loudspeaker (191) is controlled by one of the processors (422, 432) or by the audio processor (413) (Fig. 4). In this example, the loudspeaker (191) includes a series of inwardly oriented rectangular apertures to direct sound toward the wearer, as illustrated. Additional or differently oriented apertures may be implemented. In other exemplary configurations, the eyewear device (100) is coupled to one or more loudspeakers (191) positioned at various locations of the eyewear device (100), configured to operate together (e.g., stereo, in zones that generate surround sound) or independently. For example, one or more loudspeakers (191) may be integrated into the frame (105), temples (125), or corners (110A, 110B) of the eyewear device (100).
[0052] Although FIGS. 2a and 2b are illustrated as having two optical elements (180A, 180B), the eyewear device (100) may include other arrangements, such as a single optical element, depending on the use or intended user of the eyewear device (100) (or it may not include any optical elements (180A, 180B)). As additionally illustrated, the eyewear device (100) includes a left corner (110A) adjacent to the left side (170A) of the frame (105) and a right corner (110B) adjacent to the right side (170B) of the frame (105). The corners (110A, 110B) may be integrated into the frame (105) at individual sides (170A, 170B) (as illustrated) or implemented as separate components attached to the frame (105) at individual sides (170A, 170B). Alternatively, the corners (110A, 110B) may be integrated into eyeglass temples (not shown) attached to the frame (105).
[0053] In one example, the image display of the optical assembly (180A, 180B) includes an integrated image display. As illustrated in FIG. 2a, each optical assembly (180A, 180B) includes a suitable display matrix (177), such as a liquid crystal display (LCD), an organic light-emitting diode (OLED) display, or any other such display. Each optical assembly (180A, 180B) also includes an optical layer or layers (176) which may include any combination of lenses, optical coatings, prisms, mirrors, waveguides, optical strips, and other optical components. The optical layers (176A, 176B, …176N) (illustrated as 176A-N in FIG. 2a and herein) may include a prism having a suitable size and configuration and comprising a first surface for receiving light from the display matrix and a second surface for radiating light to the user's eye. The prisms of the optical layers (176A-N) extend across all or at least part of the individual apertures (175A, 175B) formed in the left and right frames (107A, 107B) so that the user can see the second surface of the prism when the user's eyes are looking through the corresponding left and right frames (107A, 107B). The first surface of the prism of the optical layers (176A-N) faces upward from the frame (105), and the display matrix (177) is placed across the prism so that photons and light emitted by the display matrix (177) collide with the first surface. The prism is sized and shaped so that light is refracted within the prism and directed toward the user's eyes by the second surface of the prism of the optical layers (176A-N). In this regard, the second surface of the prism of the optical layers (176A-N) may be convex to direct the light toward the center of the eye.The prism may optionally be adjusted in size and shape to enlarge an image projected by the display matrix (177), and light travels through the prism such that the image seen on the second surface is one or more dimensions larger than the image emitted from the display matrix (177).
[0054] In one example, the optical layers (176A-N) may include an LCD layer that is transparent (keeps the lens open) when no voltage is applied that makes the layer opaque (closes or blocks the lens) and until such voltage is not applied. An image processor (412) on the eyewear device (100) may be programmed to apply a voltage to the LCD layer to create an active shutter system that makes the eyewear device (100) suitable for viewing the visual content when the visual content is displayed as a three-dimensional projection. Technologies other than LCD, including other types of response layers that respond to voltage or other types of input, may be used for the active shutter mode.
[0055] In another example, the image display device of the optical assembly (180A, 180B) includes a projection image display as shown in FIG. 2b. Each optical assembly (180A, 180B) includes a laser projector (150), which is a three-color laser projector using a scanning mirror or a galvanometer. During operation, a light source such as the laser projector (150) is placed in or on one of the temples (125A, 125B) of the eyewear device (100). The optical assembly (180B) includes one or more optical strips (155A, 155B, …155N) (shown as 155A-N in FIG. 2b) that are spaced apart across the width of the lens of each optical assembly (180A, 180B) or across the depth of the lens between the front and back of the lens.
[0056] As photons projected by the laser projector (150) travel across the lenses of each optical assembly (180A, 180B), the photons strike the optical strips (155A-N). When a specific photon strikes a specific optical strip, that photon is redirected toward the user's eye, or it is passed to the next optical strip. A combination of modulation of the laser projector (150) and modulation of the optical strips can control specific photons or light beams. In one example, a processor controls the optical strips (155A-N) by initiating mechanical, acoustic, or electromagnetic signals. Although illustrated as having two optical assemblies (180A, 180B), the eyewear device (100) may include other arrangements, such as a single or three optical assemblies, or each optical assembly (180A, 180B) may have an arrangement that differs depending on the use or intended user of the eyewear device (100).
[0057] As additionally illustrated in FIGS. 2a and 2b, the eyewear device (100) includes a left corner (110A) adjacent to the left side (170A) of the frame (105) and a right corner (110B) adjacent to the right side (170B) of the frame (105). The corners (110A, 110B) may be integrated into the frame (105) at individual sides (170A, 170B) (as illustrated) or implemented as separate components attached to the frame (105) at individual sides (170A, 170B). Alternatively, the corners (110A, 110B) may be integrated into temples (125A, 125B) attached to the frame (105).
[0058] In another example, the eyewear device (100) illustrated in FIG. 2b may include two projectors, namely a left projector (not shown) and a right projector (150). The left optical assembly (180A) may include a left display matrix (not shown) or a set of left optical strips (not shown) configured to interact with light from the left projector. Similarly, the right optical assembly (180B) may include a right display matrix (not shown) or a set of right optical strips (155A-N) configured to interact with light from the right projector (150). In this example, the eyewear device (100) includes a left display and a right display.
[0059] FIG. 3 is a schematic depiction of a three-dimensional scene (306), a left raw image (302A) captured by a left visible light camera (114A), and a right raw image (302B) captured by a right visible light camera (114B). The left field of view (111A) may overlap with the right field of view (111B), as illustrated. The overlapping field of view (304) represents the corresponding portion of the image captured by both cameras (114A, 114B). When referring to the field of view, the term 'overlapping' means that the matrix of pixels within the generated raw images overlaps by 30 percent (30%) or more. 'Substantially overlapping' means that the matrix of pixels within the generated raw images—or within the infrared image of the scene—overlaps by 50 percent (50%) or more. As described in this specification, two raw images (302A, 302B) may be processed to include a timestamp that enables them to be displayed together as part of a three-dimensional projection.
[0060] For the capture of stereo images, as illustrated in FIG. 3, a pair of raw red, green, and blue (RGB) images of the actual scene (306) at a given moment is captured—a left raw image (302A) captured by the left camera (114A) and a right raw image (302B) captured by the right camera (114B). When the pair of raw images (302A, 302B) is processed (e.g., by an image processor (412)), depth images are generated. The generated depth images can be viewed on the optical assembly (180A, 180B) of the eyewear device, on another display (e.g., an image display (580) on a mobile device (401)), or on a screen.
[0061] The generated depth images are located in a three-dimensional spatial region and may include a matrix of vertices in a three-dimensional location coordinate system that includes an X-axis for horizontal position (e.g., length), a Y-axis for vertical position (e.g., length), and a Z-axis for depth (e.g., distance). Each vertex may include a color attribute (e.g., red pixel light value, green pixel light value, or blue pixel light value); a position attribute (e.g., X location coordinate, Y location coordinate, and Z location coordinate); a texture attribute; a reflectance attribute; or a combination thereof. The texture attribute quantifies the perceived texture of the depth image, such as the spatial arrangement of intensities or colors in the region of the vertices of the depth image.
[0062] In one example, the virtual guided tour system (400) (Fig. 4) includes an eyewear device (100) comprising a frame (105), a left temple (125A) extending from the left side (170A) of the frame (105), and a right temple (125B) extending from the right side (170B) of the frame (105). The eyewear device (100) may further include at least two visible light cameras (114A, 114B) having overlapping fields of view. In one example, the eyewear device (100) includes a left visible light camera (114A) having a left field of view (111A), as illustrated in Fig. 3. The left camera (114A) is connected to the frame (105) or the left temple (125A) to capture a left raw image (302A) from the left side of the scene (306). The eyewear device (100) additionally includes a right visible light camera (114B) having a right field of view (111B). The right camera (114B) is connected to the frame (105) or the right temple (125B) to capture a right raw image (302B) from the right side of the scene (306).
[0063] FIG. 4 is a functional block diagram of an exemplary virtual guided tour system (400) comprising an eyewear device (100), a mobile device (401), and a server system (498) connected via various networks (495), such as the Internet. As illustrated, the virtual guided tour system (400) includes a low-power wireless connection (425) and a high-speed wireless connection (437) between the eyewear device (100) and the mobile device (401).
[0064] As illustrated in FIG. 4, the eyewear device (100) includes one or more visible light cameras (114A, 114B) that capture still images, video images, or both still and video images as described herein. The cameras (114A, 114B) may function as stereo cameras having direct memory access (DMA) to a high-speed circuit (430). The cameras (114A, 114B) may be used to capture initial depth images that can be rendered into a three-dimensional (3D) model, which are texture-mapped images of a red, green, and blue (RGB) imaged scene. The device (100) may also include a depth sensor that uses infrared signals to estimate the positions of objects relative to the device (100). In some examples, the depth sensor includes one or more infrared radiators and infrared camera(s) (410).
[0065] The eyewear device (100) additionally includes two image displays of each optical assembly (180A, 180B) (one associated with the left side (170A) and the other with the right side (170B)). The eyewear device (100) also includes an image display driver (442), an image processor (412), a low-power circuit (420), and a high-speed circuit (430). The image displays of each optical assembly (180A, 180B) are intended to present images including still images, video images, or still and video images. The image display driver (442) is coupled to the image displays of each optical assembly (180A, 180B) to control the display of images.
[0066] The components illustrated in FIG. 4 for the eyewear device (100) are located on one or more circuit boards, for example, a printed circuit board (PCB) or a flexible printed circuit (FPC), located on the eyeglass frames or temples. Alternatively, or additionally, the described components may be located on the corners, frames, hinges, or bridges of the eyewear device (100). The left and right visible light cameras (114A, 114B) may include any other individual visible or light capture elements that can be used to capture data, such as digital camera elements, for example, a CMOS (complementary metal-oxide-semiconductor) image sensor, a charge coupling device, a lens, or still images or videos of scenes containing unknown objects.
[0067] As illustrated in FIG. 4, the high-speed circuit (430) includes a high-speed processor (432), memory (434), and a high-speed wireless circuit (436). In this example, an image display driver (442) is coupled to the high-speed circuit (430) and operated by the high-speed processor (432) to drive the left and right image displays of each optical assembly (180A, 180B). The high-speed processor (432) may be any processor capable of managing high-speed communication and operation of any general computing system required for the eyewear device (100). The high-speed processor (432) includes processing resources necessary to manage high-speed data transmissions via a high-speed wireless connection (437) to a wireless local area network (WLAN) using the high-speed wireless circuit (436).
[0068] In some examples, the high-speed processor (432) runs an operating system such as a LINUX operating system or another such operating system of the eyewear device (100), and the operating system is stored in memory (434) for execution. In addition to any other responsibilities, the high-speed processor (432) runs a software architecture for the eyewear device (100) used to manage data transmissions using the high-speed wireless circuit (436). In some examples, the high-speed wireless circuit (436) is configured to implement the IEEE (Institute of Electrical and Electronic Engineers) 802.11 communication standards, also referred to herein as Wi-Fi. In other examples, other high-speed communication standards may be implemented by the high-speed wireless circuit (436).
[0069] The low-power circuit (420) includes a low-power processor (422) and a low-power wireless circuit (424). The low-power wireless circuit (424) and the high-speed wireless circuit (436) of the eyewear device (100) may include short-range transceivers (Bluetooth™ or Bluetooth Low-Energy (BLE)) and wireless wide-range, short-range, or wide-range network transceivers (e.g., cellular or Wi-Fi). A mobile device (401) including transceivers communicating via a low-power wireless connection (425) and a high-speed wireless connection (437) may be implemented using the details of the architecture of the eyewear device (100), as well as other elements of the network (495).
[0070] The memory (434) comprises any storage device capable of storing various data and applications, including, among other things, camera data generated by the left and right visible light cameras (114A, 114B), infrared camera(s) (410), and image processor (412), and images generated for display by the image display driver (442) on the image display of each optical assembly (180A, 180B). Although the memory (434) is depicted as being integrated with the high-speed circuit (430), in other examples, the memory (434) may be an independent, standalone element of the eyewear device (100). In such specific examples, electrical routing lines may provide a connection through a chip including the high-speed processor (432) from the image processor (412) or the low-power processor (422) to the memory (434). In other examples, the high-speed processor (432) can manage the addressing of the memory (434) so that the low-power processor (422) boots the high-speed processor (432) whenever a read or write operation involving the memory (434) is required.
[0071] As illustrated in FIG. 4, various elements of the eyewear device (100) may be coupled to the low-power circuit (420), the high-speed circuit (430), or both. For example, an infrared camera (410) (including an infrared radiator in some implementations), user input devices (491) (e.g., a touchpad (181)), microphone(s) (139), and an IMU (472) may be coupled to the low-power circuit (420), the high-speed circuit (430), or both.
[0072] As illustrated in FIG. 5, the CPU (540) of the mobile device (401) can be combined with the camera system (570), mobile display driver (582), user input layer (591), and memory (540A).
[0073] The server system (498) may be one or more computing devices as part of a service or network computing system, for example, including a processor, memory, and a network communication interface for communicating with an eyewear device (100) and a mobile device (401) through a network (495).
[0074] The output components of the eyewear device (100) include visual elements (e.g., a liquid crystal display (LCD), a plasma display panel (PDP), a light-emitting diode (LED) display, a projector, or a waveguide display), such as the optical assemblies (180A, 180B) described in FIG. 2a and 2b or left and right image displays associated with each lens. The eyewear device (100) may include an indicator facing the user (e.g., an LED, a loudspeaker (191), or a vibration actuator) or a signal facing outward (e.g., an LED, a loudspeaker (191)). The image displays of each optical assembly (180A, 180B) are driven by an image display driver (442). In some exemplary configurations, the output components of the eyewear device (100) further include additional indicators such as audible elements (e.g., loudspeakers (191)), tactile components (e.g., actuators such as vibration motors that generate haptic feedback), and other signal generators. For example, the device (100) may include a set of indicators facing the user and a set of signals facing outward. The set of indicators facing the user is configured to be visible to or otherwise perceived by the user of the device (100). For example, the device (100) may include an LED display positioned to be visible to the user, one or more speakers (191) positioned to generate sound that the user can hear, or an actuator that provides haptic feedback that the user can feel. The set of signals facing outward is configured to be visible to or otherwise perceived by an observer near the device (100). Similarly, the device (100) may include an LED, a loudspeaker (191), or an actuator configured and positioned so as to be detected by an observer.
[0075] The input components of the eyewear device (100) may include alphanumeric input components (e.g., a touch screen or touchpad (181) configured to receive alphanumeric input, an optical-optical keyboard, or other alphanumeric configured elements), pointer-based input components (e.g., a mouse, a touchpad (181), a trackball, a joystick, a motion sensor, or other pointing tools), haptic input components (e.g., a button switch, a touch screen or touchpad (181) that detects the location, force, or location and force of touches or touch gestures, or other haptic input components), and audio input components (e.g., a microphone (139)). The mobile device (401) and the server system (498) may include alphanumeric, pointer-based, haptic, audio, and other input components.
[0076] In some examples, the eyewear device (100) includes a collection of motion sensing components referred to as an inertial measurement unit (472). The motion sensing components may be micro-electro-mechanical systems (MEMS) having minute moving parts that are often small enough to be part of a microchip. In some exemplary configurations, the inertial measurement unit (IMU) (472) includes an accelerometer, a gyroscope, and a magnetometer. The accelerometer detects the linear acceleration of the device (100) about three orthogonal axes (x, y, z) (including acceleration due to gravity). The gyroscope detects the angular velocity of the device (100) about three rotational axes (pitch, roll, yaw). Together, the accelerometer and gyroscope can provide position, orientation, and motion data regarding the device about six axes (x, y, z, pitch, roll, yaw). A magnetometer detects the orientation of the device (100) relative to magnetic north, if present. The position of the device (100) may be determined by location sensors such as a GPS unit, one or more transceivers for generating relative position coordinates, altitude sensors or barometers, and other orientation sensors. Such positioning system coordinates may also be received from the mobile device (401) via wireless connections (425 and 437) via a low-power wireless circuit (424) or a high-speed wireless circuit (436).
[0077] The IMU (472) may include or cooperate with a programming or digital motion processor that collects raw data from components and calculates a number of useful values regarding the position, orientation, and motion of the device (100). For example, acceleration data collected from an accelerometer may be integrated to obtain velocity for each axis (x, y, z); and the position of the device (100) (in linear coordinates, x, y, and z) may be obtained by integrating again. Angular velocity data from a gyroscope may be integrated to obtain the position of the device (100) (in spherical coordinates). Programming for calculating these useful values may be stored in memory (434) and executed by a high-speed processor (432) of the eyewear device (100).
[0078] The eyewear device (100) may optionally include additional peripheral sensors, such as biometric sensors, special-purpose sensors, or display elements integrated with the eyewear device (100). For example, peripheral device elements may include any I / O components including output components, motion components, position components, or any other such elements described herein. For example, biometric sensors may include components that detect expressions (e.g., hand expressions, facial expressions, voice expressions, body gestures, or eye tracking), measure biosignals (e.g., blood pressure, heart rate, body temperature, sweat, or brain waves), or identify a person (e.g., identification based on electrical biosignals such as voice, retina, facial features, fingerprints, or electroencephalogram data).
[0079] The mobile device (401) may be a smartphone, tablet, laptop computer, access point, or any other such device capable of connecting to the eyewear device (100) using both a low-power wireless connection (425) and a high-speed wireless connection (437). The mobile device (401) is connected to a server system (498) and a network (495). The network (495) may include any combination of wired and wireless connections.
[0080] The virtual guided tour system (400) includes a computing device, such as a mobile device (401), which is connected to the eyewear device (100) via a network as illustrated in FIG. 4. The virtual guided tour system (400) includes memory for storing instructions and a processor for executing instructions. The execution of instructions of the virtual guided tour system (400) by the processor (432) configures the eyewear device (100) to cooperate with the mobile device (401). The virtual guided tour system (400) may utilize the memory (434) of the eyewear device (100) or memory elements (540A, 540B, 540C) of the mobile device (401) (Fig. 5). Additionally, the virtual guided tour system (400) may utilize processor elements (432, 422) of the eyewear device (100) or the central processing unit (CPU) (540) (Fig. 5) of the mobile device (401). Additionally, the virtual guided tour system (400) may further utilize the memory and processor elements of the server system (498). In this embodiment, the memory and processing functions of the virtual guided tour system (400) may be shared or distributed across the processors and memories of the eyewear device (100), the mobile device (401), and the server system (498).
[0081] In some exemplary implementations, memory (434) includes or is combined with an activity library (480), an object data library (482), and a tour library (484). The activity library (480) includes and stores a plurality of predefined activities (850) as described herein, which is useful for determining whether an action detected in a physical environment substantially matches one of the predefined activities (850) defined and stored in the activity library (480). The object data library (482) includes and stores a plurality of predefined objects, such as containers and vessels, as described herein, which is useful for determining whether a container detected in a physical environment substantially matches one of the predefined objects defined and stored in the object data library (482).
[0082] The tour library (484) contains and stores a number of predefined guided tours. For example, in some implementations, the data record associated with the tour includes a title, one or more segment titles, and an avatar (710); content associated with a contextual overlay (725) that includes links (734) to additional content and interaction elements (730); content associated with a video (750) that includes video segments, still photos, and slide presentations; and lectures (872) that include narrations, lessons, and guides (in both text and audio formats).
[0083] In some implementations, memory (434) includes or is combined with a guide tour application (910), a localization system (915), an image processing system (920), a voice recognition module (925), and an avatar animation engine (930). In a virtual guide tour system (400) in which a camera captures frames of video data (900), the guide tour application (910) configures a processor (432) to detect one or more containers or items, retrieve data (850) associated with the detected containers, and present a contextual overlay (725), as described herein. The localization system (915) configures a processor (432) to obtain localization data for use in determining the position of the eyewear device (100) relative to the physical environment. The localization data may be derived from a series of images, an IMU unit (472), a GPS unit, or a combination thereof. The image processing system (920) configures the processor (432) to work in cooperation with the image display driver (442) and the image processor (412) to present a contextual overlay (725), an avatar (710), and various graphic elements (712, 720, 711, 721, 750) on the display of the optical assembly (180A, 180B).
[0084] The speech recognition module (925) is configured to recognize human speech, convert the received speech into frames of audio data (905), identify a query based on the audio data (905), and assemble a correlated response to respond to the identified query.
[0085] The avatar animation engine (930) works in conjunction with the image display driver (442) and the image processor (412) to configure the processor (432) to render the avatar (710) as a still image or a moving image (e.g., partially or fully animated) for presentation on the display of the optical assembly (180A, 180B). The avatar (710) is a cartoon-like character called Bitmoji® that is rendered to appear three-dimensional in some implementations. Predefined and configurable Bitmoji® avatars are accessible via a network (495) and, in some implementations, are stored in the tour library (484) described herein.
[0086] FIG. 5 is a high-level functional block diagram of an exemplary mobile device (401). The mobile device (401) includes a flash memory (540A) that stores a program executed by a CPU (540) to perform all or part of the functions described herein.
[0087] The mobile device (401) may include at least two visible light cameras (first and second visible light cameras having overlapping fields of view) or at least one visible light camera having substantially overlapping fields of view and a depth sensor, and a camera (570). The flash memory (540A) may additionally include a plurality of images or videos generated through the camera (570).
[0088] As illustrated, the mobile device (401) includes an image display (580), a mobile display driver (582) that controls the image display (580), and a display controller (584). In the example of FIG. 5, the image display (580) includes a user input layer (591) (e.g., a touchscreen) that is layered on top of or otherwise integrated into the screen used by the image display (580).
[0089] Examples of touchscreen type mobile devices that may be used include (but are not limited to) smartphones, PDAs (personal digital assistants), tablet computers, laptop computers, or other portable devices. However, the structure and operation of touchscreen type devices are provided as examples; the technology as described herein is not intended to be limited thereto. Accordingly, for the purposes of this discussion, FIG. 5 provides a block diagram example of an exemplary mobile device (401) having a user interface including a touchscreen input layer (591) for receiving input (by touch, multitouch, or gesture, etc., by hand, stylus, or other tool) and an image display (580) for displaying content.
[0090] As illustrated in FIG. 5, the mobile device (401) includes at least one digital transceiver (XCVR) (510), illustrated as WWAN XCVRs, for digital wireless communication over a wide area wireless mobile communication network. The mobile device (401) also includes additional digital or analog transceivers, such as short-range transceivers (XCVRs) (520) for short-range network communication, such as NFC, VLC, DECT, ZigBee, Bluetooth™, or WiFi. For example, the short-range XCVRs (520) may take the form of any available bidirectional wireless local area network (WLAN) transceiver of a type compatible with one or more standard protocols of communication implemented in wireless local area networks, such as one of the Wi-Fi standards under IEEE 802.11.
[0091] To generate location coordinates for positioning the mobile device (401), the mobile device (401) may include a global positioning system (GPS) receiver. Alternatively, or additionally, the mobile device (401) may use either or both of short-range XCVRs (520) and WWAN XCVRs (510) to generate location coordinates for positioning. For example, cellular network, Wi-Fi, or Bluetooth™-based positioning systems can generate very accurate location coordinates, especially when used in combination. Such location coordinates may be transmitted to the eyewear device via one or more network connections through the XCVRs (510, 520).
[0092] In some examples, the client device (401) includes a collection of motion sensing components referred to as an inertial measurement unit (IMU) (572) for detecting the position, orientation, and motion of the client device (401). The motion sensing components may be micro-electro-mechanical systems (MEMS) having minute moving parts that are often small enough to be part of a microchip. In some exemplary configurations, the inertial measurement unit (IMU) (572) includes an accelerometer, a gyroscope, and a magnetometer. The accelerometer detects the linear acceleration of the client device (401) about three orthogonal axes (x, y, z) (including acceleration due to gravity). The gyroscope detects the angular velocity of the client device (401) about three rotational axes (pitch, roll, yaw). The accelerometer and gyroscope together can provide position, orientation, and motion data regarding the device for six axes (x, y, z, pitch, roll, yaw). The magnetometer, if present, detects the orientation of the client device (401) with respect to magnetic north.
[0093] The IMU (572) may include or cooperate with a programming or digital motion processor that collects raw data from components and calculates a number of useful values regarding the position, orientation, and motion of the client device (401). For example, acceleration data collected from an accelerometer may be integrated to obtain velocity for each axis (x, y, z); and integrated again to obtain the position of the client device (401) (in linear coordinates, x, y, and z). Angular velocity data from a gyroscope may be integrated to obtain the position of the client device (401) (in spherical coordinates). Programming for calculating these useful values may be stored in one or more memory elements (540A, 540B, 540C) and executed by the CPU (540) of the client device (401).
[0094] Transceivers (510, 520) (i.e., network communication interfaces) comply with one or more of the various digital wireless communication standards used by modern mobile networks. Examples of WWAN transceivers (510) include, for example and without limitation, transceivers configured to operate according to Code Division Multiple Access (CDMA) and 3rd Generation Partnership Project (3GPP) network technologies, including 3GPP Type 2 (or 3GPP2) and LTE, sometimes referred to as "4G" (but not limited thereto). For example, transceivers (510, 520) provide bidirectional wireless communication of information including digitized audio signals, still images and video signals, web page information for display as well as web-related inputs, and various types of mobile message communications to / from a mobile device (401).
[0095] The mobile device (401) further includes a microprocessor that functions as a central processing unit (CPU); illustrated as the CPU (540) in FIG. 4. A processor is a circuit having elements structured and arranged to perform one or more processing functions, typically various data processing functions. Although individual logic components may be used, examples utilize components that form a programmable CPU. The microprocessor includes, for example, one or more integrated circuit (IC) chips that integrate electronic elements to perform the functions of a CPU. The CPU (540) may be based on any known or available microprocessor architecture, such as RISC (Reduced Instruction Set Computing) using the ARM architecture, which is commonly used in mobile devices and other portable electronic devices today. Of course, other arrangements of processor circuits may be used to form the CPU (540) or processor hardware in smartphones, laptop computers, and tablets.
[0096] The CPU (540) acts as a programmable host controller for the mobile device (401) by configuring the mobile device (401) to perform various operations according to instructions or programming executable by the CPU (540), for example. For example, such operations include various general operations of the mobile device as well as operations related to programming applications on the mobile device. Although the processor may be configured using hardwired logic, typical processors within mobile devices are general processing circuits configured by the execution of programming.
[0097] A mobile device (401) includes a memory or storage system for storing programming and data. In this example, the memory system includes flash memory (540A), random access memory (RAM) (540B), and other memory components (540C) as needed. The RAM (540B) serves as short-term storage for instructions and data being processed by the CPU (540), for example, as a task data processing memory. The flash memory (540A) typically provides longer-term storage.
[0098] Therefore, in the example of a mobile device (401), flash memory (540A) is used to store programming or instructions for execution by the CPU (540). Depending on the type of device, the mobile device (401) stores and runs a mobile operating system in which specific applications are executed. Examples of mobile operating systems include Google Android, Apple iOS (for iPhone or iPad devices), Windows Mobile, Amazon Fire OS, RIM BlackBerry OS, etc.
[0099] A processor (432) within the eyewear device (100) may construct a map of the environment surrounding the eyewear device (100), determine the location of the eyewear device within the mapped environment, and determine the relative position of the eyewear device with respect to one or more objects within the mapped environment. The processor (432) may construct the map and determine location and position information using a simultaneous localization and mapping (SLAM) algorithm applied to data received from one or more sensors. The sensor data includes images received from one or both of the cameras (114A, 114B), distance(s) received from a laser rangefinder, position information received from a GPS unit, motion and acceleration data received from an IMU (572), or data from such sensors, or a combination of data from other sensors that provide data useful for determining position information. In the context of augmented reality, SLAM algorithms are used to construct and update maps of the environment, while simultaneously tracking and updating the location of a device (or user) within the mapped environment. Mathematical solutions can be approximated using various statistical methods, such as particle filters, Kalman filters, extended Kalman filters, and covariance intersections. In systems including high-definition (HD) video cameras that capture video at high frame rates (e.g., 30 frames per second), SLAM algorithms update the locations of maps and objects at least as frequently as the frame rate; in other words, mapping and localization are calculated and updated 30 times per second.
[0100] Sensor data includes image(s) received from one or both of the cameras (114A, 114B), distance(s) received from a laser rangefinder, position information received from a GPS unit, motion and acceleration data received from an IMU (472), or data from such sensors, or a combination of data from other sensors that provide data useful for determining position information.
[0101] FIG. 6 depicts an exemplary physical environment (600) with elements useful for using SLAM applications and other types of tracking applications (e.g., natural feature tracking (NFT)). A user (602) of an eyewear device (100) is present in the exemplary physical environment (600) (in FIG. 6, indoors). A processor (432) of the eyewear device (100) uses captured images to determine the position of one or more objects (604) within the environment (600), constructs a map of the environment (600) using a coordinate system (x, y, z) for the environment (600), and determines its position within the coordinate system. Additionally, the processor (432) determines the head pose (roll, pitch, and yaw) of the eyewear device (100) in the environment by using two or more location points (e.g., three location points (606a, 606b, 606c)) associated with a single object (604a), or by using one or more location points (606) associated with two or more objects (604a, 604b, 604c). The processor (432) of the eyewear device (100) can position a virtual object (608) (e.g., a key shown in FIG. 6) in the environment (600) to view during an augmented reality experience.
[0102] In some examples, the localization system (915) is a virtual marker (610a) associated with a virtual object (608) within the environment (600). In augmented reality, markers are registered at locations within the environment to assist devices in the task of tracking and updating the locations of users, devices, and objects (virtual and physical) within the mapped environment. Markers are sometimes registered at high-contrast physical objects, such as a relatively dark object like a picture frame (604a) mounted on a brighter colored wall, to assist cameras and other sensors in the task of detecting the markers. Markers can be pre-assigned or can be assigned by the eyewear device (100) upon entering the environment.
[0103] Markers may have information encoded or otherwise linked to information. A marker may include position information, a physical code (e.g., a barcode or QR code; visible or hidden from the user), or a combination thereof. A data set associated with a marker is stored in the memory (434) of the eyewear device (100). The data set includes information regarding a marker (610a), the position (location and orientation) of the marker, one or more virtual objects, or a combination thereof. The marker position may include three-dimensional coordinates for one or more marker landmarks (616a), such as the corner of the generally rectangular marker (610a) shown in FIG. 6. The marker location may be expressed in terms of actual geographic coordinates, a marker coordinate system, the position of the eyewear device (100), or other coordinate systems. One or more virtual objects associated with the marker (610a) may include any of various materials, including still images, video, audio, haptic feedback, executable applications, interactive user interfaces and experiences, and combinations or sequences of such materials. Any type of content that can be stored in memory and retrieved when the marker (610a) encounters or is associated with the assigned marker can be classified as a virtual object in this situation. For example, the key (608) illustrated in FIG. 6 is a virtual object displayed as a still image of either 2D or 3D at the marker location.
[0104] In one example, the marker (610a) may be registered in memory as being located near and associated with a physical object (604a) (e.g., a framed work of art shown in FIG. 6). In another example, the marker may be registered in memory as being a specific position for the eyewear device (100).
[0105] FIG. 9 is a flowchart (940) enumerating steps of an exemplary method for presenting a virtual tour experience (700) on a display (180B) of an eyewear device (100). Although the steps are described in relation to the eyewear device (100) described herein, for other types of devices, other implementations of the described steps will be understood by those skilled in the art from the description herein. One or more of the steps illustrated and described may be performed simultaneously, sequentially, in a different order than illustrated and described, or with additional steps. Some steps may be omitted, or in some applications, repeated.
[0106] The guided tour application (910) described herein is initiated in some implementations in response to receiving a selection through a user interface (e.g., selecting from a menu, pressing a button) or through some other input means (e.g., hand gestures, voice commands). In other examples, the guided tour application (910) is initiated in response to detecting a container (650) as described herein.
[0107] Block 942 of FIG. 9 describes an exemplary step of capturing frames of video data (900) within the field of view (904) of a camera (114B) of an eyewear device (100). In this example, the eyewear device (100) includes a camera (114B), a microphone (139), a loudspeaker (191), a guided tour application (910), and a display (180B). In some implementations, the eyewear device (100) includes one or more cameras (114A, 114B) for capturing still images or frames of video data (900), as described herein. In this example, the eyewear device (100) includes an image processing system (920) and one or more displays (180A, 180B). For example, as illustrated in FIGS. 7 and 8, the eyewear device (100) includes a translucent image display (180B) which may include a translucent lens layer and a display matrix layer configured to present images on the lens of the eyewear device, as described herein. Graphic and virtual elements are part of a contextual overlay (725) that is presented on the display (180B) as an overlay of the surrounding physical environment (600). An effect as illustrated allows the viewer to view and interact with the presented contextual overlay (725) while the surrounding environment (600) also continues to be visible through the display (180B).
[0108] In some implementations, the high-speed processor (432) of the eyewear device (100) stores frames of video data (900) captured by the camera (114B) as the wearer moves through the physical environment (600). As described herein and illustrated in FIG. 7, the camera (114B) typically has a camera field of view (904) capable of capturing images and videos of the environment beyond the limits of the display (180B).
[0109] In some embodiments, the camera system comprises one or more high-resolution digital cameras equipped with CMOS image sensors capable of capturing high-resolution still images and high-resolution video at a relatively high frame rate (e.g., 30 frames per second or more). Each frame of the digital video contains depth information for a plurality of pixels within the image. In this embodiment, the camera system acts as a high-resolution scanner by capturing a detailed input image of the physical environment. In some embodiments, the camera system comprises a pair of high-resolution digital cameras (114A, 114B) that are coupled to the eyewear device (100) and separated to acquire a left camera raw image and a right camera raw image as described herein. When combined, the raw images form an input image comprising a matrix of three-dimensional pixel locations. An exemplary method, in block 942, in some embodiments, includes the step of at least temporarily storing the frames of the captured video data (900) in a memory (434) on the eyewear device (100) so that the frames can be used for analysis.
[0110] The process of capturing frames of the video data (900) described herein is, in some implementations, in progress during active use of the eyewear device (100). In other examples, this capturing process is initiated in response to receiving a selection through a user interface (e.g., selection from a menu, pressing a button) or through some other input means (e.g., hand gesture, voice command).
[0111] Block 944 describes an exemplary step of detecting a container (650) in frames of captured video data (900) using an image processing system (920). The process of detecting the container (650) includes detecting the current container position (705) in three-dimensional coordinates for a display (180B) or alternatively for another known position such as an eyewear location (840), as illustrated. In some exemplary implementations, the image processing system (920) analyzes pixel-level data within frames of the captured video data (900) to determine whether the frame contains one or more containers (650).
[0112] FIG. 7 is a perspective example of an exemplary container (650) held in a hand (10) in a physical environment (600). In this example, the guide tour application (910) detects the current container position (705) as illustrated and sets it near the side of the detected container (650). In other situations, the container position (705) may be set near the center of the detected container (650) or at any other predefined or configurable position for the size and shape of the detected container (650).
[0113] As used herein, the term container (650) refers to and includes sealed containers such as bottles, cans, jars, and flasks; open containers such as kettles, pitchers, decanters, glass bottles, cups, mugs, tanks, barrels, and baskets; and other containers for holding items, including but not limited to bowls, plates, dishes, spoons, bags, pouches, and human hands.
[0114] Those skilled in the art will understand that as the eyewear device (100) moves through the physical environment (600), the process of detecting containers (650) and container positions (705) continues over time. Over time, additional containers (650) may be detected within the field of view (904) of the camera (114B), while other items may be stopped from being detected. Furthermore, one or more detected containers (650) may be moved to a new location in the physical environment (600) and then detected at a different or updated current container position (705). In this embodiment, the detection process is ongoing and substantially continuous, wherein the image processing system (920) detects subsequent containers at subsequent container positions by analyzing pixel-level data within frames of the captured video data (900).
[0115] An exemplary process in block 944 includes determining whether frames of captured video data (900) contain a container (650). In some implementations, the image processing system (920) compares pixel-level data within frames of captured video data (900) with data stored in an object data library (482) for each of a plurality of predefined containers and other objects. For example, data records for an object include a name or other identifier (e.g., bottle, can, bicycle, person, mug, wine glass), shape and size including three-dimensional coordinates for one or more object landmarks, viewpoint (e.g., side, top, bottom), orientation reference (e.g., vertical, horizontal), and other data and descriptors associated with each object.
[0116] An exemplary process in block 944 includes detecting two or more containers in frames of captured video data (900). In this embodiment, over time, the process includes detecting a subsequent container, retrieving data associated with the detected subsequent container, and presenting a subsequent contextual overlay and a subsequent start message based on the subsequently retrieved data.
[0117] Block 946 of FIG. 9 describes an exemplary step of retrieving data (850) associated with a detected container (650) using a guided tour application (910). The retrieved data (850) is stored at least temporarily in memory (434) on the eyewear device (100). In some exemplary implementations, as described in Block 946 of FIG. 9, the process of retrieving data (850) includes collecting information regarding the detected container (650) and its contents from one or more sources including an object data library (482) or a tour library (484), and the one or more sources may further include one or more websites on the Internet. In this embodiment, the process of retrieving data (850) includes assembling search terms, performing a search, and collecting information related to the detected container (650) and its contents. In some implementations, the guided tour application (910) is configured to access one or more preferred search engines, websites, and other internet-based resources. In some implementations, the process in the block (946) for collecting data (850) using internet search involves using machine learning algorithms to select the search engines, web resources, and website data most likely to quickly and efficiently retrieve relevant container information.
[0118] Block 948 of FIG. 9 describes an exemplary step of presenting a contextual overlay (725) on a display (180B). The contextual overlay (725) is based on data (850) retrieved regarding a detected container (650). The contextual overlay (725) is presented at a position adjacent to or otherwise associated with the detected container position (705), as illustrated in FIG. 7. In some implementations, the contextual overlay (725) is rendered in a size and shape according to its content up to a configurable maximum size to fit the display (180B). In other implementations, the content of the contextual overlay (725) is limited in size according to the size and shape of the contextual overlay (725).
[0119] As the eyewear device (100) moves through the physical environment (600), in some implementations, a process called localization will be used to make it appear as though a contextual overlay (725) remains near the current container position (705). In some implementations, the localization system (915) on the eyewear device (100) configures the processor (432) on the eyewear (100) to acquire localization data to be used to determine the current eyewear location (840) for the current container position (705). The localization data may be derived from frames of captured video data (900), an IMU unit (472), a GPS unit, or a combination thereof. The localization system (915) can construct a virtual map of various elements within the camera field of view (904) using a SLAM algorithm as described in this specification, which updates the locations of maps and objects as frequently as the frame rate of the camera (114B) (e.g., frequently calculating and updating the mapping and localization of the current eyewear location (840) at least 30 times per second).
[0120] In some implementations, the localization process involves calculating a correlation between a detected container position (705) and a current eyewear location (840). The term correlation refers to and includes one or more vectors, matrices, formulas, or other mathematical expressions sufficient to define a three-dimensional distance between the detected container position (705) and the eyewear display (180B) according to the current eyewear location (840). The current eyewear location (840) is, of course, related to or continuously associated with the display (180B) supported by the frame of the eyewear device (100). In this embodiment, the correlation performs the function of calibrating the motion of the eyewear (100) with respect to the apparent motion of the detected container (650). Because the localization process occurs frequently, the correlation is calculated frequently, so that the tracking of the detected current container position (705) relative to the current eyewear location (840) is accurate and nearly real-time.
[0121] Because the localization process occurs frequently, the correlation is calculated frequently, so that the tracking of the container (650) for the current eyewear location (840) is accurate and nearly real-time.
[0122] The contextual overlay (725) is based on search data (850) retrieved regarding the detected container (650) and its contents. In the example where the container (650) is a wine bottle, the contextual overlay (725) illustrated in FIG. 7 includes the name of the wine, the winemaker, and the region where the wine was produced. The contextual overlay (725) also includes evaluations, tasting notes, and food pairing recommendations as illustrated.
[0123] In some implementations, the contextual overlay (725) includes one or more interactive graphic elements (730), each of which is associated with additional content. For example, as illustrated in FIG. 7, the contextual overlay (725) includes a slider element (730) and a scroll bar (732). For example, using the scroll bar (732) may reveal additional content including additional or different features, evaluations, notes, and information. Using the interactive graphic elements (730), in some implementations, alternatives (e.g., similar wines) to the contents associated with the detected container (650) will be revealed. Additionally, one or more text elements of the contextual overlay (725) are underlined to indicate clickable links to follow. For example, a first link (734A) associated with a citrus note reveals additional details regarding the taste, flavor, and other characteristics of the contents of the container (650) when selected. A second link (734B) associated with food compatibility, if selected, reveals additional details regarding recommended foods that go well with the contents of the container (650). The links, if selected, may cause all or part of the contextual overlay (725) to be replaced or updated with new and different content. Any of the various other graphic elements (730) may be included in the contextual overlay (725). As used herein, one or more interaction elements (730) mean and include any collection of graphic elements presented on a display, including but not limited to graphic elements such as virtual objects and icons associated with VR experiences, thumbnails, taskbars, and menu items.
[0124] Interactive graphic elements (730) include, in some implementations, a movable element (711) (e.g., a cursor as illustrated in FIG. 7) presented at a current element position (740) on the display. Interacting with the cursor (711) includes, in some implementations, detecting a current fingertip location (681) on a touchpad (181) attached to the eyewear device as illustrated; and then presenting the cursor (711) at a current element position (740) on the display according to the detected current fingertip location (681). In this example, a navigating action may be performed by moving the cursor (711) to one of the graphic elements on the display. For example, navigating actions include placing the cursor (711) near the scroll bar (732), near one of the links (734A, 734B), or near a selectable graphic element (720), tapping, pressing, double-tapping, or otherwise selecting a nearby element.
[0125] As illustrated in FIG. 7, the virtual tour experience (700) includes, in some implementations, a selectable graphic element (720) (e.g., a tour start button). The selectable graphic element (720) is presented on the display at a button position (702) relative to the display. In some implementations, the button position (702) is defined relative to the display (180B), so that the selectable graphic element (720) is presented at a consistent position on the display (e.g., the top-left corner). In some implementations, when a selection action (e.g., placing the cursor (711) near the element (720) and tapping) is detected, the guided tour application (910) begins to present the virtual tour experience (700). In some implementations, the selection action includes receiving a voice command to start the tour.
[0126] Block 950 of FIG. 9 describes an exemplary step of presenting an avatar (710) at an avatar position (701) on a display. The avatar (710) is based on data (850) retrieved regarding a detected container (650) and its contents. According to an example where the container (650) is a wine bottle, the avatar (710) shown in FIG. 7 appears to be holding a wine glass.
[0127] The avatar (710) is presented on the display at the avatar position (701) relative to the display. In some implementations, the avatar position (701) is defined in relation to the display (180B), and accordingly, the avatar (710) will be presented at a consistent position on the display (e.g., the top right corner).
[0128] Block 950 of FIG. 9 also describes an exemplary step of presenting a start message (871) based on data (850) retrieved regarding a detected container (650) and its contents. According to an example where the container (650) is a wine bottle, the start message (871) may include an avatar name (e.g., "Sommelier Megan"), a welcome message (e.g., "Good choice! Cheers!"), and a note regarding one or more elements of a contextual overlay (725) (e.g., "This wine pairs well with shellfish" or "Hirshfeld Vineyards was named the best winemaker of 2024").
[0129] In addition to other messages and responses described herein, the start message (871) may be presented in text form (e.g., inside a speech bubble (712) adjacent to the avatar (710), as illustrated in FIG. 7, or anywhere on the display), or played audibly through a loudspeaker (191), or both. In some implementations, playing the message or response involves animating the avatar (710) so that it appears as if the avatar (710) is speaking the message or response.
[0130] In some implementations, the speech bubble (712) is rendered with a size according to the content of the message or response, up to a configurable maximum size to fit the display (180B). In other implementations, the length of the message or response is limited according to the size and shape of the speech bubble (712). In these embodiments, any message or response described herein may be presented sequentially in segments.
[0131] Block 952 of FIG. 9 describes an exemplary step of identifying a first query (860) based on frames of audio data (905) received by a microphone (139) using a speech recognition module (925). The speech recognition module (925) is coupled to or otherwise accessible by the eyewear device (100). The process (860) of identifying the first query includes receiving human speech through the microphone (139) and then converting the speech into frames of audio data (905). The speech recognition module (925) uses automated speech recognition processing to analyze the frames of audio data (905) and identify the content of the first query (860). In some implementations, automated speech recognition involves using machine learning algorithms trained to rapidly and efficiently detect, decode, and identify the content of human speech.
[0132] Block 954 describes an exemplary step of playing a first response (870) in response to an identified first query (860). Like other messages and responses described herein, the first response (870) may be presented in text form (e.g., inside a speech bubble (712)), played audibly through a loudspeaker (191), or both. The process of assembling and playing the first response (870) by the speech recognition module (925) includes correlating the content of the first response (870) with the content of the first query (860).
[0133] Blocks 952 and 954 of FIG. 9 describe an exemplary process for a guided tour application (910) to conduct an interactive question-and-answer session with an avatar (710) that provides answers related to and responding to the questions. In some implementations, this type of voice-based interactive session is active and available at any time during the virtual tour experience (700) described herein.
[0134] FIG. 10 is a flowchart (960) listing the steps of an exemplary method of presenting a virtual guided tour experience (700) in response to detecting an action on a display.
[0135] Block 962 of FIG. 10 describes an exemplary step of detecting a first action (801) in frames of captured video data (900) using an image processing system (920). In an exemplary implementation, the first action (801) is associated with a detected container (650) and a second container (660). In this embodiment, the detection process includes detecting the second container (660) in frames of captured video data (900).
[0136] FIG. 8 is a spectral example of an exemplary container (650) (e.g., a wine bottle) and a second container (660) (e.g., a wine glass) having a central axis (651). The first action (801) illustrated in FIG. 8 includes pouring the contents of the container (650) into the second container (660). The process of detecting the first action (801) includes comparing the central axis (651) of the bottle container (650) with the orientation of the second container (660). In this example, the first action (801) of pouring wine initiates a virtual guided tour experience (700).
[0137] Block 964 of FIG. 10 describes an exemplary step of determining whether a detected first action (801) matches a first predefined activity (851) (e.g., following contents into a second container) among a plurality of predefined activities (850) stored in an activity library (480). Any of the various predefined activities (850) may be set, defined, and stored in the activity library (480).
[0138] In the example illustrated in FIG. 8, the image processing system (920) analyzes pixel-level data of frames of captured video data (900) to determine whether the container (650) is performing a first action (801) (e.g., following) on a second container (660) in a manner that matches a first predefined activity (851) defined and stored in an activity library (480). In some exemplary implementations, the data record for the predefined activity includes a name or other identifier (e.g., open, follow, swivel), three-dimensional coordinates for one or more container landmarks, a series of shapes representing a moving container, a viewpoint (e.g., side, top, bottom), a direction reference (e.g., vertical, horizontal), and other data and descriptors associated with each activity.
[0139] In some implementations, the guided tour application (700) is configured to detect one or more other actions, each of which can be paired with a corresponding predefined activity, as stored in the activity library (480). For example, the application (700) may be configured to detect an opening action (e.g., removing a cork, bottle stopper, stopper, or other cap from a container (650). In certain situations, such as with wine and certain spirits, the opening of the container (650) may be associated with one or more related activities (e.g., allowing the wine to breathe for a predefined duration). A corresponding predefined opening activity may be defined and stored in the activity library (480).
[0140] In another example, the application (700) may be configured to detect a tasting action (e.g., consuming the contents of the container (650) or the second container (660)). A corresponding predefined tasting activity (e.g., bringing the container (650) or the second container (660) within a predefined threshold distance of the mouth) may be defined and stored in the activity library (480).
[0141] Block 966 of FIG. 10 describes an exemplary step of presenting a virtual guided tour experience (700) on a display in response to detecting a match between a detected first action (801) and a first predefined activity (851).
[0142] In some implementations, the virtual guided tour experience (700) includes a video (750) presented on a display at a position adjacent to or otherwise associated with the avatar position (701), as illustrated in FIG. 8. The video (750) is presented in relation to a lecture (872), so that the lecture (872) corresponds temporally to the content of the video (750). The lecture (872) is presented through the avatar (710). Like other messages and responses, the lecture (872) may be presented in text form (e.g., inside a speech bubble (712) adjacent to the avatar (710), or anywhere on the display), or played audibly through a loudspeaker (191), or both. In some implementations, playing the lecture (872) includes animating the avatar (710) using the avatar animation engine (930) described herein so that the avatar (710) appears to be speaking the content of the lecture (872).
[0143] The lecture (872) includes any of the various claims related to the container (650), its contents, retrieved data (850), and any other elements described herein. In the context of traditionally produced wine, beer, spirits, cider, tea, coffee, chocolate, produce, nuts, cheese, and other products, the lecture (872) may include general claims regarding agriculture, horticulture, farm management, soil preparation and preservation, harvesting, storage, mixing and blending, carbonation, flavorings and additives, preservatives, aging, bottling, labeling, packaging, marketing, sales, food and beverage pairings, serving methods and recommendations, history, tasting procedures and protocols, and tour experiences. The avatar (710) may be presented as an artisan, farmer, historian, business owner, laboratory technician, trained guide, or another person familiar with a particular product or task.
[0144] According to an example where the container (650) is a wine bottle, the lecture (872) covers wine, grape cultivation, winemaking, winemaking, winemaking, Claims may include pressing, fermentation, tannin, corking, bottling, etc. In this example, the avatar (710) may be presented as a sommelier, chef, vineyard owner, or skilled winemaker.
[0145] In the context of beer, the lecture (872) may include beer, brewing, grain and grain harvesting, hops, steeping, mashing, lautering, sparging, capping, and bottling. In this example, the avatar (710) may be presented as a master brewer, a brewery owner, or a trained guide.
[0146] In the context of spirits and distilled spirits, the lecture (872) may include spirits, grains and grain harvesting, distillation, fermentation, filtration, aging, and bottling. In this example, the avatar (710) may be presented as a master distiller, a distillery owner, or a trained guide.
[0147] In the context of apple cider, the lecture (872) may include apple cider, apple cider production, orchard, fruit harvesting, pressing, crushing, fermentation, and bottling. In this example, the avatar (710) may be presented as an apple cider maker, orchard manager, apple cider brewery owner, or a trained guide.
[0148] In the context of tea and situations, the lecture (872) may include tea, tea cultivation, farm management, tea blending and additives, sack packaging, steeping, and rituals. In this example, the avatar (710) may be presented as a tea expert, a tea farm owner, or a trained guide.
[0149] In the context of coffee, the lecture (872) may include coffee, coffee bean cultivation, farm management, bean grading, roasting, blending, brewing, and preparation methods. In this example, the avatar (710) may be presented as a barista, coffee expert, farm owner, or trained guide.
[0150] Block 968 of FIG. 10 describes an exemplary step of animating an avatar (710) to perform a demonstration (760) on a display in relation to a lesson (873). The demonstration (760) includes, in some implementations, one or more ways of pouring and serving the contents of a container (650) (e.g., a preferred way of pouring or pouring the contents into a second container (660)), one or more ways of tasting the contents (e.g., stirring the contents), and any of various other examples or preferred ways of performing operations related to the container (650) or its contents. In some implementations, an avatar animation engine (930) renders and presents the avatar (710) on a display with various poses (e.g., holding, pouring, stirring) and various props (e.g., a decanter, wine glasses, mugs) to perform the demonstration (760). Lesson (873) refers to and includes a speech directly related to the demonstration (760), which may or may not be part of the lecture (872).
[0151] Block 970 of FIG. 10 describes an exemplary step of presenting a graphic control element (721) on a display to control the progress of a virtual guided tour experience (700). The graphic control element (721) may be presented on the display at a position related to or adjacent to the avatar position (702), at a position near the video (750), or at another selected position.
[0152] A graphic control element (721) may be configured to control the progression of the video (750). For example, as illustrated in FIG. 8, an exemplary graphic control element (721) includes selectable buttons for play, go back, pause, skip forward, and stop. Components of the graphic control element (721) may be selected using a touchpad (181) and a cursor (711), as described herein.
[0153] In some implementations, the graphic control element (721) is configured to go back (and repeat parts) or skip forward (skip parts) of the virtual guided tour experience (700). For example, the virtual guided tour experience (700) may start with a tour of a farm or harvest; and the control element (721) may be used to skip forward to a tour of a factory or other facilities.
[0154] Additional or other graphic control elements (721) may be presented on the display and may be configured to perform any of the various activities suitable for handling or processing other activities, such as recording, saving, sharing with friends, deleting, and experiences.
[0155] Blocks 972 and 974 of FIG. 10 describe an exemplary process for conducting an interactive question-and-answer session with an avatar (710) during a lecture (872) or lesson (873). In some implementations, the tour (700) is automatically paused during such an interactive session.
[0156] Block 972 of FIG. 10 describes an exemplary step of identifying a tour query (865) (e.g., during a tour (700), during a demonstration (760)) using a speech recognition module (925). The speech recognition module (925) described herein converts speech into frames of audio data (905). The speech recognition module (925) uses automated speech recognition processing to analyze the frames of audio data (905) and identify the content of the tour query (865).
[0157] Block 974 describes an exemplary step of playing a tour response (875) in response to an identified tour query (865). Like other messages and responses described herein, the tour response (875) may be presented in text form (e.g., inside a speech bubble (712)), played audibly through a loudspeaker (191), or both. The process of assembling and playing the tour response (875) by the speech recognition module (925) includes correlating the content of the tour response (875) with the content of the tour query (865).
[0158] Although various systems and methods are described herein in relation to beverages and containers such as wine, beer, spirits, or cider, the described technology may be applied to detect any type of container or product in a physical environment, retrieve data regarding the detected product, and present a virtual guided tour on a display.
[0159] In some exemplary embodiments, the container (650) described herein may include an optionally removable lid, such as a cork, a screw cap, a bottle stopper, or a stopper. The container (650) may accommodate any of a variety of products, including but not limited to wine, beer, spirits, or cider. The virtual tour experience (700) may be associated with one or more sites, including but not limited to a winery, a beer brewery, a spirits distillery, and an cider house.
[0160] In some implementations, the virtual tour experience (700) includes farm segments and factory segments. The farm segment is associated with one or more farms, including but not limited to vineyards, grain farms, hop farms, and orchards. The factory segment is associated with one or more rooms, spaces, or other areas, including but not limited to barns, harvest areas, pressing rooms, blending rooms, laboratories, fermentation rooms, distillation rooms, cider houses, kettle rooms, barrel rooms, cask rooms, and cellars, bottling facilities, warehouses, retail stores, and tasting areas.
[0161] Some of the processes described herein involve the use of computer vision systems and machine learning algorithms trained to analyze pixel-level information contained in frames of captured video data (900), in some implementations. Machine-trained object classification algorithms are used to identify objects detected in frames of video data (900) and, as the name suggests, classify the objects to belong to one or more individual classes (e.g., containers, bottles, cups, mugs, furniture). For example, classification algorithms can be used to detect containers (650) (as well as subsequent containers) and facilitate the process of detecting navigating actions, selection actions, or other actions.
[0162] Similarly, as described herein, processes for matching detected actions with one or more predefined activities involve, in some implementations, the use of machine training algorithms for comparing captured video data (900) with a plurality of predefined activities for the purpose of finding a match.
[0163] As used herein, the term "matching" is intended to include actual matches or near matches that may be determined by a predetermined confidence value associated with possible or candidate matches. A detected item or action may include three-dimensional coordinates for various locations and landmarks, as found in frames of captured video data (900). In some examples, the detection and matching process includes calculating the sum of geodesic distances between a detected item or action and a set of defined and stored coordinates for an item or object (stored in the object data library (482)) or for a predefined activity (stored in the activity library (480)). In this embodiment, a sum falling within a configurable threshold accuracy value represents a match.
[0164] Additionally, the speech recognition processes described herein involve, in some implementations, the use of a computer-based speech recognition system (including natural language understanding (NLU) techniques) and machine learning algorithms trained to analyze human speech, convert speech into audio data, and assemble responses appropriate to the context.
[0165] Machine learning refers to algorithms that improve progressively through experience. By processing a large number of different input datasets (e.g., photos of various objects, containers, and activities in various environments), machine learning algorithms develop improved generalizations regarding specific datasets and then use those generalizations to generate accurate outputs or solutions when processing new datasets. Broadly speaking, machine learning algorithms include one or more parameters that are adjusted or changed in response to new experiences, thereby progressively improving the algorithm; a process similar to learning.
[0166] In the field of computer vision, mathematical models attempt to mimic tasks performed by the human visual system with the aim of using computers to extract information from images and achieve an accurate understanding of their content. Computer vision algorithms have been developed in various fields, including artificial intelligence and autonomous navigation, to extract and analyze data within digital images and videos.
[0167] Deep learning refers to a class of machine learning methods based on or modeled after artificial neural networks. An artificial neural network is a computing system composed of numerous simple and highly interconnected processing elements (nodes) that process information through their dynamic state responses to external inputs. Large-scale artificial neural networks can have hundreds or thousands of nodes.
[0168] A convolutional neural network (CNN) is a type of neural network frequently applied to analyze visual images, including digital photographs and videos. The connection patterns between nodes within a CNN are typically modeled after the organization of the human visual cortex, which includes individual neurons arranged to respond to overlapping regions within the field of view. A neural network suitable for use in the decision process described herein is based on one of the following architectures: VGG16, VGG19, ResNet50, Inception V3, Xception, or other CNN-compatible architectures.
[0169] In one exemplary implementation, a trained item classification model receives a frame of video data containing a detected item and abstracts the image within the frame into layers for analysis. The data within each layer is compared, layer by layer, with items and objects stored in an object data library (482) based on the trained classification model until a good match is identified.
[0170] In one example, layer-by-layer image analysis is performed using a convolutional neural network. In the first convolutional layer, the CNN identifies learned features (e.g., wine bottle characteristics, wine labels, wine bottle descriptors, etc.). In the second convolutional layer, the image is transformed into multiple images, where the learned features are highlighted in each individual sub-image. In the pooling layer, the sizes and resolutions of the images and sub-images are reduced to separate parts of the image containing possible features of interest (e.g., containers, vessels). The values and comparisons of the images from the non-output layers are used to classify the images within the frame.
[0171] Any of the functions described herein for the eyewear device (100), mobile device (401), and server system (498) may be implemented as one or more computer software applications or sets of programming instructions as described herein. According to some examples, "function," "functions," "application," "applications," "instruction," "instructions," or "programming" is a program(s) that executes functions defined in programs. Various programming languages, such as object-oriented programming languages (e.g., Objective-C, Java, or C++) or procedural programming languages (e.g., C or assembly language), may be employed to develop one or more of applications structured in various ways. In certain examples, a third-party application (e.g., an application developed by an entity other than a vendor of a particular platform using an ANDROID™ or IOS™ software development kit (SDK)) may include mobile software running on a mobile operating system such as IOS™, ANDROID™, WINDOWS® Phone, or other mobile operating systems. In this example, the third-party application may invoke API calls provided by the operating system to facilitate the functions described herein.
[0172] Accordingly, machine-readable media can take many forms of tangible storage media. Non-volatile storage media include optical or magnetic disks, such as any storage devices within any computer devices, such as client devices, media gateways, transcoders, etc., which can be used to implement, for example, the client devices, media gateways, transcoders, etc. illustrated in the drawings. Volatile storage media include dynamic memory, such as the main memory of such computer platforms. Tangible transmission media include copper wires and optical fibers, including coaxial cables; and wires including buses within a computer system. Carrier transmission media can take the form of electrical or electromagnetic signals, or acoustic or light waves, such as those generated during radio frequency (RF) and infrared (IR) data communication. Accordingly, common forms of computer-readable media include, for example: floppy disks, flexible disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, DVDs or DVD-ROMs, any other optical media, punch card paper tapes, any other physical storage media having patterns of holes, RAM, PROMs and EPROMs, FLASH-EPROMs, any other memory chips or cartridges, carriers transmitting data or instructions, cables or links transmitting such carriers, or any other media in which a computer can read programming code or data. Many of these forms of computer-readable media may be involved in transmitting one or more sequences of one or more instructions to a processor for execution.
[0173] Except for what is mentioned immediately above, nothing mentioned or exemplified, whether or not it is described in the claims, shall not be intended or interpreted to cause any component, step, feature, object, benefit, advantage, or equivalent to be made available to the public.
[0174] It will be understood that the terms and expressions used herein have the ordinary meanings given to such terms and expressions in relation to their corresponding individual fields of investigation and research, except where specific meanings are otherwise presented in this specification. Relational terms such as first, second, etc., may be used solely to distinguish one entity or action from another and do not necessarily require or imply any actual such relationship or order between such entities or actions. The terms “comprising,” “comprising,” “including,” “comprising,” or any variations thereof are intended to cover non-exclusive inclusion, and thus a process, method, article, or device that comprises or includes a list of elements or steps may not include only said elements or steps but may include other elements not explicitly enumerated or inherent to such process, method, article, or device. Unless further limitations are provided, a singular element does not exclude the presence of additional identical elements in a process, method, article, or device that includes that element.
[0175] Unless otherwise stated, any and all measurements, values, grades, positions, sizes, dimensions, and other specifications set forth herein, including the following claims, are not precise but approximate. Such quantities are intended to have a reasonable range consistent with the functions associated with them and with the customs of the relevant field. For example, unless explicitly stated otherwise, parameter values, etc., may vary by ±10% from the stated quantities or ranges.
[0176] Furthermore, in the foregoing detailed description, it can be seen that various features are grouped together in various examples for the purpose of simplifying the disclosure. This method of disclosure should not be interpreted as reflecting an intention that the claimed examples require more features than are explicitly stated in each claim. Rather, as reflected in the following claims, the subject matter to be protected is less than all the features of any single disclosed example. Accordingly, the following claims are incorporated into the detailed description, and each claim stands independently as an individually claimed subject matter.
[0177] Although the foregoing has described what are considered to be the best modes and other examples, it is understood that various modifications may be made, that the subject matter of the claims disclosed herein may be embodied in various forms and examples, and that they may be applied in many applications—of which only some are described herein. The following claims are intended to claim any and all modifications and variations that fall within the true scope of the concepts.
Claims
Claim 1 A method for presenting a virtual tour experience using an eyewear device, wherein the eyewear device comprises a camera, a microphone, a loudspeaker, a guided tour application, an image processing system, and a display, and the method comprises: capturing frames of video data within the field of view of the camera; using the image processing system to detect a container in a current container position relative to the display in the frames of the captured video data; searching for data associated with the detected container; presenting a contextual overlay adjacent to the current container position on the display based on the searched data; presenting an avatar in an avatar position relative to the display; playing a start message through the avatar based on the searched data; detecting a first action associated with the container and a second container in the frames of the captured video data; determining whether the detected first action matches a first predefined activity among a plurality of predefined activities stored in an activity library; A method comprising the step of presenting a virtual tour experience on the display according to the first predefined activity, wherein the step of presenting the virtual tour experience includes the step of presenting a video on the display adjacent to the avatar position and in relation to a lecture presented through the avatar. Claim 2 A method according to claim 1, further comprising: a step of detecting a subsequent container in a subsequent current container position; a step of retrieving data associated with the detected subsequent container; a step of presenting a subsequent situational overlay based on the subsequently retrieved data; and a step of playing a subsequent start message based on the subsequently retrieved data. Claim 3 The method of claim 1, wherein the step of presenting the situational overlay comprises: presenting one or more interaction elements — each interaction element is associated with additional content —; detecting a current fingertip location on a touchpad coupled to the eyewear device; presenting a movable element at a current element position on the display according to the detected current fingertip location; detecting a navigation action for a first interaction element positioned on the display closest to the current element position in the frames of the captured video data; and presenting the additional content on the display in response to the detected navigation action. Claim 4 The method of claim 1, wherein the eyewear device further comprises a voice recognition module, and the method further comprises: receiving human speech with the microphone; converting the received speech into frames of audio data; identifying a first query based on the frames of audio data using the voice recognition module; and playing a first response through the loudspeaker in response to the identified first query. Claim 5 A method according to claim 1, further comprising: a step of presenting a selectable graphic element at a button position on the display; a step of detecting a selection action associated with the graphic element; and a step of presenting a virtual tour experience on the display according to the detected selection action. Claim 6 delete Claim 7 A method according to claim 1, further comprising: a step of detecting a subsequent action; a step of determining whether the detected subsequent action matches a subsequent predefined activity among a plurality of predefined activities; and a step of presenting a virtual tour experience on the display according to the subsequent predefined activity. Claim 8 A method according to claim 1, further comprising the step of animating the avatar to perform a demonstration on the display adjacent to the avatar position in relation to the lesson. Claim 9 A method according to claim 1, further comprising: a step of presenting a graphic control element on the display including one or more controls selected from a group consisting of play, pause, back, next, and stop; and a step of controlling the progress of the virtual tour experience according to the graphic control element. Claim 10 A method for presenting a virtual tour experience using an eyewear device, wherein the eyewear device comprises a camera, a microphone, a loudspeaker, a guided tour application, an image processing system, and a display, and the method comprises: capturing frames of video data within the field of view of the camera; using the image processing system to detect a container in a current container position relative to the display in the frames of the captured video data; retrieving data associated with the detected container; presenting a contextual overlay adjacent to the current container position on the display based on the retrieved data; and presenting an avatar in an avatar position relative to the display. A method comprising the steps of: playing a start message based on the retrieved data through the avatar; presenting a virtual tour experience on the display, wherein the step of presenting the virtual tour experience includes presenting a video in relation to a lecture presented through the avatar, wherein the container includes an optionally removable lid and accommodates a product selected from the group consisting of wine, beer, spirits, and cider, wherein the virtual tour experience is associated with a site selected from the group consisting of a winery, a beer brewery, a spirits brewery, and a cider brewery, wherein the virtual tour experience includes a farm segment and a factory segment, wherein the farm segment is associated with a farm selected from the group consisting of a vineyard, a barley farm, a hop farm, a grain farm, and an orchard, and wherein the factory segment is associated with a room selected from the group consisting of a barn, a harvest area, a pressing room, a blending room, a laboratory, a fermentation room, a distillation room, a cider house, a hot water room, a barrel room, a tank room, a bottling room, a tasting area, and a retail store. Claim 11 As a virtual guided tour system, an eyewear device comprising a camera, a microphone, a loudspeaker, a guided tour application, an image processing system, a memory, a processor, and a display; and including programming within the memory, wherein the execution of the programming by the processor configures the eyewear device to perform functions, said functions being: capturing frames of video data within the field of view of the camera; using the image processing system, detecting a container at a current container position relative to the display in the frames of the captured video data; retrieving data associated with the detected container; presenting a contextual overlay adjacent to the current container position on the display based on the retrieved data; presenting an avatar at an avatar position relative to the display; playing a start message through the avatar based on the retrieved data; detecting a first action associated with the container and a second container in the frames of the captured video data; and determining whether the detected first action matches a first predefined activity among a plurality of predefined activities stored in an activity library. A virtual guide tour system comprising functions for presenting a virtual tour experience on the display according to the first predefined activity, wherein the virtual tour experience includes a video presented on the display adjacent to the avatar position and in relation to a lecture presented through the avatar. Claim 12 A virtual guided tour system, wherein, in claim 11, the function of presenting a contextual overlay further comprises: presenting one or more interaction elements — each interaction element is associated with additional content —; detecting a current fingertip location on a touchpad coupled to the eyewear device; presenting a movable element at a current element position on the display according to the detected current fingertip location; detecting a navigation action for a first interaction element positioned on the display closest to the current element position in the frames of the captured video data; and presenting the additional content on the display in response to the detected navigation action. Claim 13 A virtual guided tour system according to claim 11, wherein the eyewear device further comprises a voice recognition module, and the execution of the programming further comprises the eyewear device to perform functions of: receiving human speech with the microphone; converting the received speech into frames of audio data; using the voice recognition module to identify a first query based on the frames of audio data; and playing a first response through the loudspeaker in response to the identified first query. Claim 14 A virtual guided tour system according to claim 11, wherein the execution of the programming by the processor further configures the eyewear device to perform additional functions, the additional functions comprising: detecting a subsequent container in a subsequent current container position; retrieving data associated with the detected subsequent container; presenting a subsequent contextual overlay based on the subsequently retrieved data; and playing a subsequent start message based on the subsequently retrieved data. Claim 15 delete Claim 16 In claim 11, the execution of the above programming further comprises the eyewear device to perform functions of animating the avatar to perform a demonstration on the display adjacent to the avatar position in relation to the lesson, a virtual guided tour system. Claim 17 As a non-transient computer-readable storage medium storing program code, said program code, when executed, causes an electronic processor to: capture frames of video data within the field of view of a camera of an eyewear device — said eyewear device further comprises a microphone, a loudspeaker, a guided tour application, an image processing system, and a display —; using said image processing system to detect a container at a current container position relative to the display in the frames of the captured video data; search for data associated with said detected container; based on said searched data, present a contextual overlay adjacent to said current container position on the display; present an avatar at an avatar position relative to the display; through said avatar, play a start message based on said searched data; detect a first action associated with said container and a second container in the frames of the captured video data; and determine whether said detected first action matches a first predefined activity among a plurality of predefined activities stored in an activity library. A non-transient computer-readable storage medium comprising the step of presenting a virtual tour experience on the display according to the first predefined activity, wherein the step of presenting the virtual tour experience operates to perform the step of presenting a video on the display adjacent to the avatar position and in relation to a lecture presented through the avatar. Claim 18 In claim 17, the step of presenting a contextual overlay comprises: presenting one or more interaction elements — each interaction element is associated with additional content —; detecting a current fingertip location on a touchpad coupled to the eyewear device; presenting a movable element at a current element position on the display according to the detected current fingertip location; detecting a navigating action for a first interaction element positioned on the display closest to the current element position in the frames of the captured video data; and presenting the additional content on the display in response to the detected navigating action, a non-transient computer-readable storage medium. Claim 19 A non-transient computer-readable storage medium, wherein, in paragraph 17, the program code operates to further perform, when executed, the steps of: receiving human speech with the microphone; converting the received speech into frames of audio data; identifying a first query based on the frames of audio data using a speech recognition module coupled to the eyewear device; and playing a first response through the loudspeaker in response to the identified first query. Claim 20 In claim 17, the program code, when executed, causes an electronic processor to additionally perform the step of animating the avatar to perform a demonstration on the display adjacent to the avatar position in relation to the lesson, a non-transient computer-readable storage medium.
Citation Information
Patent Citations
Systems and methods of rewards object spawning and augmented reality commerce platform supporting multiple seller entities
US20190108578A1
Holographic virtual assistant
US20200012916A1
