Interactive recording tool for creating interactive AR stories

Abstract

A recording tool for creating an interactive AR experience. An interaction recording application enables a user with little or no programming skill to perform and record user behaviors associated with reactions between story elements, such as virtual objects and connected IoT devices. The user behavior includes a series of actions, such as speaking a trigger word and significantly touching a virtual object. The corresponding reaction includes an action between starting to record the subsequent scene and executing the story element. The trigger recording interface is presented on the display as an overlay relative to the physical environment.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. application serial number 17 / 901,589, filed on September 1, 2022, the contents of which are incorporated herein by reference in their entirety. Technical Field

[0003] Examples set forth in this disclosure relate to the field of augmented reality (AR) experiences for electronic devices, including portable electronic devices and wearable devices such as eyewear. More particularly, but not by way of limitation, this disclosure describes an interaction recording application that enables a user to perform and record user actions associated with reactions between elements such as virtual objects and connected IoT devices in real time. Background Art

[0004] Many types of computers and electronic devices available today, such as mobile devices (e.g., smartphones, tablets, and laptops), handheld devices, and wearable devices (e.g., smart glasses, digital glasses, headsets, headbands, and head-mounted displays), include various cameras, sensors, wireless transceivers, input systems, and displays. Users sometimes refer to information on these devices during physical activities such as exercise.

[0005] The so-called "Internet of Things" (IoT) refers to and includes the following physical products that are embedded with sensors, software, and other technologies for connecting and exchanging data with other devices in a network (usually via the Internet). For example, IoT products are used for home automation to control lighting, heating and air conditioning, media and security systems, and camera systems. Many IoT-enabled devices have been provided to act as smart home hubs to connect different smart home products. IoT devices are also used in many other applications. Application layer protocols and supporting frameworks have been provided to implement such IoT applications. For example, some IoT products include an application programming interface (API) that allows IoT products to pair with other products and electronic devices (such as portable computers) and communicate with them separately. Artificial intelligence has also been combined with the IoT infrastructure to achieve more efficient IoT network operation, improve human-computer interaction, and enhance data management and analysis.

[0006] Virtual reality (VR) technology generates a complete virtual environment including realistic images, which is sometimes presented on a VR headset or other head-mounted display. The VR experience allows the user to move around in the virtual environment and interact with virtual objects. AR is a VR technology that combines real objects in the physical environment with virtual objects and displays the combination to the user. The combined display gives the impression that the virtual objects are actually present in the environment, especially when the virtual objects appear and behave like real objects. Cross-reality (XR) is generally understood as an umbrella term for systems that include or combine elements from AR, VR, and MR (mixed reality) environments.

[0007] Automatic speech recognition (ASR) is a field of computer science, artificial intelligence, and linguistics that involves receiving spoken words and converting them into audio data suitable for processing by a computing device. Human speech received as an analog audio signal can be converted into a digital audio signal and analyzed using ASR to identify discrete words in the speech. The identified words can be used to perform actions or commands useful in controlling and interacting with computer software applications. ASR processing can be used by computers, handheld devices, wearable devices, telephone systems, automobiles, and a variety of other devices to facilitate human-computer interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The features of the various examples described will be readily understood from the following detailed description with reference to the accompanying drawings. In the description and throughout the several views of the accompanying drawings, reference numerals are used with each element. When multiple similar elements are present, a single reference numeral may be assigned to the similar element, with additional uppercase or lowercase letters referring to the specific element.

[0009] Unless otherwise indicated, the various elements shown in the figures are not drawn to scale. For clarity, the size of the various elements may be enlarged or reduced. These figures depict one or more embodiments and are presented only as examples and should not be construed as limiting. The drawings include the following figures:

[0010] Figure 1A is a side view of an eyewear device suitable for an example story interaction system (right);

[0011] Figure 1B yes Figure 1A a partial perspective cross-sectional view of the optical components and electronics in a portion of the eyewear device shown;

[0012] Figure 1C yes Figure 1A Side view of the eyewear device (left);

[0013] Figure 1D yes Figure 1C a partial perspective cross-sectional view of the optical components and electronics in a portion of the eyewear device shown;

[0014] Figure 2A and Figure 2B is a rear view of the eyewear device used in the example story interaction system;

[0015] Figure 3 is a block diagram illustrating an example of capturing visible light using the example eyewear device shown in any of the preceding figures;

[0016] Figure 4 is a functional block diagram of an example story interaction system including an eyewear device and a server system connected via various networks;

[0017] Figure 5 Is configured for Figure 4 An illustration of an example hardware configuration for a mobile device of an example story interaction system;

[0018] Figure 6 is a stereogram of a user in an example environment, including physical and virtual objects, for reference in describing an example process known as simultaneous localization and mapping (SLAM);

[0019] Fig. 7A is a flowchart listing the steps of an example method for generating an interactive AR experience, including presenting a trigger interface on a display;

[0020] Figure 7B is a flow chart listing steps in an example triggering interface in which an example user action includes speaking a trigger word;

[0021] Figure 7C is a flow chart listing steps in an example triggering interface in which an example user action includes touching a virtual object;

[0022] Fig. 8A is an illustration of an example triggering interface presented on a display of a portable electronic device;

[0023] Figure 8B yes Fig. 8A An illustration of an example triggering interface, including an example question and two example answer buttons;

[0024] Figure 8C yes Fig. 8A An illustration of a portion of an example trigger interface;

[0025] Fig.8D yes Fig. 8A An example of triggering another portion of the interface. DETAILED DESCRIPTION

[0026] A story interaction system for use with AR applications on portable electronic devices (including mobile phones and wearable devices such as electronic glasses devices). The story interaction system enables users of portable electronic devices with little or no programming skills to create and record user actions (such as speaking a word or touching a virtual object) and also record reactions associated with the user actions.

[0027] Reference examples are described for various embodiments and details for generating interactive AR experiences using a portable electronic device including a microphone, a camera, a display, and a memory. In an example embodiment, a method involves presenting a trigger interface on a display. The trigger interface facilitates the creation and recording of user actions and corresponding reactions. In some embodiments, the user actions include speaking a word and touching a virtual object.

[0028] The following detailed description includes systems, methods, techniques, instruction sequences, and computer program products that illustrate the examples set forth in this disclosure. Many details and examples are included for the purpose of providing a thorough understanding of the disclosed subject matter and its related teachings. However, those skilled in the relevant art can understand how to apply the related teachings without these details. The various aspects of the disclosed subject matter are not limited to the specific devices, systems, and methods described, because the related teachings can be applied or practiced in various ways. The terms and nomenclature used herein are only for the purpose of describing specific aspects, and are not intended to be restrictive. In general, well-known instruction instances, protocols, structures, and techniques are not necessarily shown in detail.

[0029] As used herein, the terms "connect," "connected," "coupled," and "coupled" refer to any logical, optical, physical, or electrical connection, including a link or the like, through which an electrical or magnetic signal generated or supplied by one system element is conducted to another coupled or connected system element. Unless otherwise specified, coupled or connected elements or devices are not necessarily directly connected to each other and may be separated by intermediate components, elements, or communication media, one or more of which can modify, manipulate, or carry electrical signals. The term "on" means supported directly by an element, or indirectly supported by an element through another element that is integrated into or supported by the element.

[0030] The term "proximal" is used to describe an item or part of an item that is near, adjacent to, or beside an object or person; or closer relative to other parts of the item may be described as "distal." For example, the end of an item closest to the object may be called the proximal end, while the generally opposite end may be called the distal end.

[0031] For purposes of illustration and discussion, the orientation of the eyeglass device, associated components, and any complete device incorporating an eye scanner and camera, such as those shown in any of the accompanying drawings, is given by way of example only. In operation for a particular variable optical processing application, the eyeglass device may be oriented in any other orientation (e.g., upward, downward, sideways, or any other orientation) suitable for the particular application of the eyeglass device. Additionally, as used herein, any directional terms, such as front, back, inward, outward, toward, left, right, lateral, longitudinal, up, down, upper, lower, top, bottom, and side, are used by way of example only and are not limiting with respect to the direction or orientation of any optical device or component of an optical device constructed as otherwise described herein.

[0032] Advanced AR technologies, such as computer vision and object tracking, can be used to produce perceptually rich and immersive experiences. Computer vision algorithms extract three-dimensional data about the physical world from data captured in digital images or videos. Object recognition and tracking algorithms are used to detect objects in digital images or videos, estimate their orientation or pose, and track their movement over time. Real-time recognition and tracking of body parts, such as hands and fingers, arms and legs, and feet, is one of the most challenging and processing-intensive tasks in the field of computer vision.

[0033] Additional objects, advantages and novel features of the examples will be set forth in part in the following description and in part will become apparent to those skilled in the art upon examination of the following and accompanying drawings, or may be learned by production or operation of the examples. The objects and advantages of the subject matter may be realized or attained by the methods, instrumentalities and combinations particularly pointed out in the appended claims.

[0034] In a sample configuration, an eyewear device with augmented reality (AR) capabilities is used in the system described herein. Eyewear devices are expected to be used in the system described herein because such devices are scalable and customizable to achieve a personalized experience, achieve the effect of being applied anytime, anywhere, and ensure user privacy by only allowing the user to see the transmitted information. In a sample configuration, the eyewear device can be used without any dedicated hardware, such as the SPECTACLES available from Snap, Inc. of Santa Monica, California. TM .

[0035] like Figures 1A-1D As shown, the eyewear device 100 includes a first camera 114A and a second camera 114B. The cameras 114 capture image information of a scene from separate viewpoints. The captured images can be used to project a three-dimensional display onto an image display for three-dimensional (3D) viewing.

[0036] Cameras 114 are sensitive to wavelengths in the visible light range. Each of cameras 114 defines a different forward-facing field of view that overlaps to enable generation of a three-dimensional depth image, e.g., first camera 114A defines a first field of view 111A, and second camera 114B defines a second field of view 111B. In general, a "field of view" is the portion of a scene visible through a camera at a particular location and orientation in space. Fields of view 111 have overlapping fields of view 304 ( Figure 3 ). When the camera captures an image, objects or object features outside the field of view 111 are not recorded in the original image (e.g., a photograph or picture). The field of view describes the angular range or degree to which the image sensor of the camera 114 picks up electromagnetic radiation of a given scene in the captured image of the given scene. The field of view can be expressed as the angular size of the viewing cone (i.e., the viewing angle). The viewing angle can be measured horizontally, vertically, or diagonally.

[0037] In an example configuration, one or both cameras 114 have a field of view of 100° and a resolution of 480 x 480 pixels. The "angle of coverage" describes the range of angles that the lens of the camera 114 can effectively image. Typically, a camera lens produces an image circle large enough to completely cover the film or sensor of the camera, which may include some vignetting (e.g., the image becomes darker toward the edges compared to the center). If the angle of coverage of the camera lens does not fill the sensor, the image circle will be visible, typically with strong vignetting toward the edges, and the effective viewing angle will be limited by the angle of coverage.

[0038] Examples of suitable cameras 114 include high-resolution complementary metal oxide semiconductor (CMOS) image sensors and digital VGA cameras (video graphics arrays) capable of 480p (e.g., 640 x 480 pixels), 720p, 1080p, or higher resolutions. Other examples include cameras 114 that can capture high-definition (HD) video at high frame rates (e.g., 30 to 60 frames per second or more) and store recordings at a resolution of 1216 by 1216 pixels (or higher).

[0039] The eyewear device 100 can capture image sensor data from the camera 114 and geographic location data digitized by the image processor to be stored in a memory. The camera 114 captures corresponding raw images (e.g., left and right raw images) in a two-dimensional spatial domain, which includes a pixel matrix on a two-dimensional coordinate system, which includes an X-axis for horizontal positioning and a Y-axis for vertical positioning. Each pixel includes a color attribute value (e.g., a red pixel light value, a green pixel light value, or a blue pixel light value); and a positioning attribute (e.g., an X-axis coordinate and a Y-axis coordinate).

[0040] To capture stereoscopic images for later display as a 3D projection, image processor 412 ( Figure 4) may be coupled to the camera 114 to receive and store visual image information. The image processor 412 or another processor controls the operation of the camera 114 to act as a stereo camera that simulates human binocular vision, and a time stamp may be added to each image. The time stamp on each pair of images allows the images to be displayed together as part of a 3D projection. 3D projection creates an immersive, lifelike experience that is desirable in a variety of contexts, including virtual reality (VR) and video games.

[0041] Figure 1B yes Figure 1A 1 is a perspective cross-sectional view of the right corner 110A of the eyewear device 100 depicting the first camera 114A, additional optical components and electronics. Figure 1C yes Figure 1A 1 is a side view (left) of an example hardware configuration of the eyewear device 100 showing the second camera 114B of the camera system. Figure 1D yes Figure 1C 1 is a perspective cross-sectional view of a left corner 110B of the eyewear device 100 depicting a second camera 114B, additional optical components, and electronics of the camera system.

[0042] like Figure 1B As shown in the example of FIG. 1 , the eyeglass device 100 includes a first camera 114A and a circuit board 140A, which may be a flexible printed circuit board (PCB). The first hinge 126A connects the right corner 110A to the first temple 125A of the eyeglass device 100. In some examples, the first camera 114A, the flexible PCB 140A, or other electrical connectors or contacts may be located on the first temple 125A or the first hinge 126A.

[0043] The right mirror corner 110A includes a mirror corner body 190 and a mirror corner cap. Figure 1B Disposed within the right corner 110A are various interconnected circuit boards 109, such as a flexible PCB 140A, which include controller circuits for the first camera 114A, (one or more) microphones 139, (one or more) speakers 191, low-power wireless circuit systems (e.g., for wireless short-range network communications via BluetoothTM), high-speed wireless circuit systems (e.g., for wireless local area network communications via Wi-Fi).

[0044] The first camera 114A is coupled to or disposed on the flexible PCB 140A and is covered by a camera cover lens that is aimed through the opening(s) formed in the frame 105. For example, Figure 2AAs shown, the right edge 107A of the eyeglass frame 105 is connected to the right corner 110A and includes (one or more) openings for the camera cover lens. The eyeglass frame 105 includes a front side that is configured to face outward and away from the user's eyes. The opening for the camera cover lens is formed in and through the front side or outward facing side of the eyeglass frame 105. In this example, the first camera 114A has an outward facing field of view 111A (e.g., a field of view 111A) at a line of sight or perspective associated with the right eye of the user of the eyeglass device 100. Figure 3 ). The camera cover lens may also be adhered to the front side or outward facing surface of the right mirror corner 110A, in which the opening is formed with an outward facing coverage angle, but in a different outward direction. The coupling may also be indirect via an intervening component.

[0045] like Figure 1D As shown in the example of , the eyeglass device 100 includes a second camera 114B and a circuit board 140B, which may be a flexible printed circuit board (PCB). The second hinge 126B connects the left corner 110B to the second temple 125B of the eyeglass device 100. In some examples, the second camera 114B, the flexible PCB 140B, or other electrical connectors or contacts may be located on the second temple 125B or the second hinge 126B.

[0046] The left mirror corner 110B includes a mirror corner body 190 and a mirror corner cap. Figure 1D The corner cap is omitted in the cross section of FIG. Disposed within the right corner 110A are various interconnected circuit boards (such as a flexible PCB 140B) that include controller circuitry for the second camera 114B.

[0047] The camera 114 is coupled to or disposed on a corresponding flexible PCB 140 and is covered by a camera cover lens that is aimed through the opening(s) formed in the frame 105. For example, Figure 2A As shown, the right edge 107A of the frame 105 is connected to the right corner 110A and includes (one or more) openings for the camera cover lens, and the left edge 107B of the frame 105 is connected to the left corner 110B and includes (one or more) openings for the camera cover lens. The frame 105 includes a front side that is configured to face outward and away from the user's eyes. The opening for the camera cover lens is formed in and through the front side or outward facing side of the frame 105. In this example, the camera 114 has a corresponding outward facing field of view 111 (e.g., a field of view) at a line of sight or perspective associated with a corresponding eye of a user of the eyewear device 100 Figure 3 ). The camera cover lens may also be adhered to the front side or outward facing surface of the corresponding mirror corner 110, in which the opening is formed with an outward facing coverage angle, but in a different outward direction. The coupling may also be indirect via an intervening component.

[0048] Figure 2A and Figure 2B An example hardware configuration of an eyeglass device 100 including two different types of image displays is depicted. The size and shape of the eyeglass device 100 are configured to be worn by a user. The form of the eyeglasses is shown in the illustrated example. The eyeglass device 100 may take other forms and may be combined with other types of frames, such as headgear, headphones, or helmets.

[0049] In the eyeglass example, the eyeglass device 100 includes a frame 105, which includes a right rim 107A connected to a left rim 107B via a nose bridge 106, and the nose bridge 106 is configured to receive the user's nose to support the eyeglass device 100 on the user's head. The right rim 107A includes a first rim 175A, which holds a first optical element 180A. The left rim 107B includes a second rim 175B, which holds a second optical element 180B. Figure 2B As shown, in some embodiments, each optical element 180A, 180B includes an integrated image display (e.g., a first display 182A and a second display 182B). As used herein, the term "lens" is intended to include a transparent or translucent glass or plastic sheet having a curved or flat surface that causes light to converge or diverge, or to cause little or no convergence or divergence.

[0050] A touch-sensitive input device such as a touch pad 181 is positioned on the first temple 125A. As shown, the touch pad 181 can have a clearly visible border, or include a raised or other tactile edge that provides feedback to the user about the location and border of the touch pad 181; alternatively, the border can be subtle and not easily seen or felt. The eyeglass device 100 can include a touch pad on the other side that operates independently or in conjunction with the touch pad 181.

[0051] The surface of the touch pad 181 is configured to detect finger touches, taps, and gestures (e.g., moving touch) for use with a graphical user interface (GUI) displayed by the eyewear device on the image display to allow the user to browse and select menu options in an intuitive manner, which enhances and simplifies the user experience.

[0052] Detecting finger input on the touchpad 181 can achieve a variety of functions. For example, touching any position on the touchpad 181 can cause the GUI to display or highlight an item on the image display, which can be projected onto at least one of the optical components 180. Tapping or double-clicking the touchpad 181 can select an item or icon. Sliding or swiping a finger in a particular direction (e.g., from front to back, from back to front, from top to bottom, or from bottom to top) can cause an item or icon to slide or scroll in a particular direction; for example, moving to the next item, icon, video, image, page, or slide. Sliding a finger in another direction can slide or scroll in the opposite direction; for example, moving to the previous item, icon, video, image, page, or slide. The touchpad 181 can be positioned substantially anywhere on the eyeglass device 100.

[0053] In one example, an identified single-touch gesture on touch pad 181 initiates a selection or press of a GUI element in an image presented on the image display of optical assembly 180. Adjusting the image presented on the image display of optical assembly 180 based on the identified gesture can be a primary action that selects or commits a GUI element on the image display of optical assembly 180 for further display or execution.

[0054] Figure 2A 1 is an example hardware configuration of the eyeglass device 100 in which the right corner 110A supports a microphone 139 and a speaker 191. The microphone 139 includes a transducer that converts sound into a corresponding electrical audio signal. The microphone 139 in the example shown is positioned to have an opening facing inward toward the wearer to facilitate receiving sound waves, such as human speech including verbal commands and questions. Additional or differently oriented openings may be implemented. In other example configurations, the eyeglass device 100 is coupled to one or more microphones 139 that are configured to operate together or independently and are positioned at various locations on the eyeglass device 100.

[0055] The speaker 191 includes an electroacoustic transducer that converts an electrical audio signal into a corresponding sound. The speaker 191 is controlled by one of the processors 422, 432 or by the audio processor 413 ( Figure 4) control. As shown, the speaker 191 in this example includes a series of rectangular apertures that face inward to direct sound to the wearer. Additional or differently oriented apertures may be implemented. In other example configurations, the eyeglass device 100 is coupled to one or more speakers 191 that are configured to operate together (e.g., stereo, producing surround sound in a zone) or independently and are positioned at various locations on the eyeglass device 100. For example, one or more speakers 191 may be incorporated into the frame 105, temple 125, or corner 110 of the eyeglass device 100.

[0056] Despite Figure 2A and Figure 2B 100 is shown as having two optical elements 180, but depending on the application or intended user of the eyeglass device 100, the eyeglass device 100 may include other arrangements, such as a single optical element (or it may not include any optical element 180). As further shown, the eyeglass device 100 includes a right corner 110A adjacent to the right lateral side 170A of the frame 105 and a left corner 110B adjacent to the left lateral side 170B of the frame 105. The corners 110 may be integrated into the frame 105 on the respective sides 170 (as shown), or may be implemented as separate components attached to the frame 105 on the respective sides 170. Alternatively, the corners 110A, 110B may be integrated into temples attached to the frame 105 (not shown).

[0057] In one example, each image display of optical assembly 180 includes an integrated image display (eg, first display 182A and second display 182B). Figure 2A As shown, each optical assembly 180 has a display 182 including a suitable display matrix 177, such as a liquid crystal display (LCD), an organic light-emitting diode (OLED) display, or other such display. Each optical assembly 180 also includes one or more optical layers 176, which may include any combination of lenses, optical coatings, prisms, reflectors, waveguides, optical strips, and other optical components. The optical layers (e.g., Figure 2AThe optical layer 176A-N (shown as 176A-N in FIG. 104 ) may include a prism having a suitable size and configuration and including a first surface for receiving light from the display matrix and a second surface for emitting light toward the user's eyes. The prisms of the optical layer 176A-N extend over all or at least a portion of the corresponding lens ring 175 formed in the left and right lens edges 107 to allow the user to see the second surface of the prism when the user's eyes look through the corresponding lens edge 107. The first surface of the prism of the optical layer 176A-N faces upward from the frame 105, and the display matrix 177 covers the prism so that photons and light emitted by the display matrix 177 strike the first surface. The prism is designed in size and shape so that light is refracted within the prism and directed toward the user's eyes through the second surface of the prism of the optical layer 176A-N. In this regard, the second surface of the prism of the optical layer 176A-N can be convex to direct light toward the center of the eye. The prisms may optionally be sized and shaped to magnify the image projected by the display matrix 177 , and light travels through the prisms so that the image viewed from the second surface is larger in one or more dimensions than the image emitted from the display matrix 177 .

[0058] In one example, the optical layers 176A-N may include an LCD layer that is transparent (holding the lens open) unless and until a voltage is applied that renders the layer opaque (closing or blocking the lens). The image processor 412 on the eyewear device 100 may be programmed to apply a voltage to the LCD layer, thereby creating an active shutter system, making the eyewear device 100 suitable for viewing visual content when displayed as a 3D projection. Technologies other than LCD may be used for active shutter modes, including other types of reactive layers that respond to a voltage or another type of input.

[0059] In another example, the image display device of the optical assembly 180 has a display 182 that includes a display such as Figure 2B Each optical assembly 180 includes a corresponding laser projector 150, such as a three-color laser projector using a scanning mirror or a galvanometer. Each laser projector 150 is disposed in or on a corresponding temple 125 of the eyeglass device 100. In this example, each optical assembly 180 includes one or more optical strips (such as Figure 2B 155A-N), these optical strips span the width of the lens of each optical component 180 or span the depth of the lens between the front and back surfaces of the lens and are spaced apart.

[0060] As photons projected by the laser projector 150 travel through the lens of each optical assembly 180, the photons encounter optical strips 155A-N. When a particular photon encounters a particular optical strip, the photon is redirected toward the user's eye, or it passes to the next optical strip. The combination of modulation of the laser projector 150 and modulation of the optical strips controls specific photons or light beams. In an example, the processor controls the optical strips 155A-N by issuing a mechanical signal, an acoustic signal, or an electromagnetic signal. Although shown as having two optical assemblies 180, the eyeglass device 100 may include other arrangements, such as a single or three optical assemblies, or each optical assembly 180 may have a different arrangement depending on the application or intended user of the eyeglass device 100.

[0061] Figure 3 3D scene 306, a first raw image 302A captured using a first camera 114A, and a second raw image 302B captured using a second camera 114B. As shown, the first field of view 111A can overlap with the second field of view 111B. The overlapping field of view 304 represents the portion of the image captured using the two cameras 114. When referring to the field of view, the term "overlap" means that the pixel matrices in the generated raw images overlap by thirty percent (30%) or more. "Substantially overlap" means that the pixel matrices in the generated raw images or infrared images of the scene overlap by fifty percent (50%) or more. As described herein, the two raw images 302 can be processed to include a timestamp, which allows the images to be displayed together as part of a three-dimensional projection.

[0062] To capture stereo images, such as Figure 3 As shown, a pair of raw red, green, and blue (RGB) images of a 3D scene 306 are captured at a given moment—a first raw image 302A captured using a first camera 114A and a second raw image 302B captured using a second camera 114B. When the pair of raw images 302 are processed (e.g., by an image processor 412), a depth image is generated. The generated depth image can be viewed on an optical assembly 180 of the eyewear device, on another display (e.g., an image display 580 on a mobile device 401), or on a screen.

[0063] The generated depth image is in a three-dimensional spatial domain and may include a matrix of vertices on a three-dimensional position coordinate system including an X-axis for horizontal positioning (e.g., length), a Y-axis for vertical positioning (e.g., height), and a Z-axis for depth (e.g., distance). Each vertex may include a color attribute (e.g., a red pixel light value, a green pixel light value, or a blue pixel light value); a positioning attribute (e.g., an X position coordinate, a Y position coordinate, and a Z position coordinate); a texture attribute; a reflectivity attribute; or a combination thereof. The texture attribute quantifies the perceived texture of the depth image, such as the spatial arrangement of color or intensity in a vertex region of the depth image.

[0064] Figure 4 4 is a functional block diagram of an example story interaction system 400, which includes a glasses device 100, a mobile device 401, and a server system 498 connected via various networks 495 (such as the Internet). As shown, the story interaction system 400 includes a low-power wireless connection 425 and a high-speed wireless connection 437 between the glasses device 100 and the mobile device 401.

[0065] As described herein, the eyewear device 100 includes one or more cameras 114 that capture still images, video images, or both still images and video images. The camera 114 may have direct memory access (DMA) to the high-speed circuit system 430 and be used as a stereo camera. The camera 114 may be used to capture initial depth images that may be rendered as three-dimensional (3D) models that are texture-mapped images of red, green, and blue (RGB) imaging scenes. The device 100 may also include a depth sensor that uses infrared signals to estimate the position of an object relative to the device 100. In some examples, the depth sensor includes one or more infrared emitters and one or more infrared cameras 410.

[0066] The eyeglass device 100 also includes two image displays (one associated with the right side 170A and one associated with the left side 170B) of the optical assembly 180. The eyeglass device 100 also includes an image display driver 442, an image processor 412, a low power circuit system 420, and a high speed circuit system 430. The image display of the optical assembly 180 is used to present images, including still images, video images, or both still images and video images. The image display driver 442 is coupled to the image display of the optical assembly 180 to control the display of the image.

[0067] Figure 4 The components of the eyeglass device 100 shown in FIG. 1 are located on one or more circuit boards, such as a printed circuit board (PCB) or a flexible printed circuit (FPC) located in the rim or temple. Alternatively or additionally, the depicted components may be located in the corners, frame, hinge, or bridge of the eyeglass device 100. The camera 114 includes a digital camera element, such as a complementary metal oxide semiconductor (CMOS) image sensor, a charge coupled device, a lens, or any other corresponding visible light or light capturing element that can be used to capture data, including still images or video of a scene with an unknown object.

[0068] like Figure 4As shown, the high-speed circuit system 430 includes a high-speed processor 432, a memory 434, and a high-speed wireless circuit system 436. In the example, an image display driver 442 is coupled to the high-speed circuit system 430 and is operated by the high-speed processor 432 to drive the image display of the optical assembly 180. The high-speed processor 432 can be substantially any processor capable of managing high-speed communications and operations of any general-purpose computing system. The high-speed processor 432 includes the processing resources required to manage high-speed data transmission on a high-speed wireless connection 437 to a wireless local area network (WLAN) using the high-speed wireless circuit system 436.

[0069] In some examples, the high-speed processor 432 executes an operating system, such as a LINUX operating system or other such operating system of the eyeglass device 100, and the operating system is stored in the memory 434 for execution. In addition to any other duties, the high-speed processor 432 also executes a software architecture for the eyeglass device 100, which is used to manage data transmission with the high-speed wireless circuit system 436. In some examples, the high-speed wireless circuit system 436 is configured to implement the Institute of Electrical and Electronics Engineers (IEEE) 802.11 communication standard, also referred to as Wi-Fi herein. In other examples, other high-speed communication standards can be implemented by the high-speed wireless circuit system 436.

[0070] The low-power circuit system 420 includes a low-power processor 422 and a low-power wireless circuit system 424. The low-power wireless circuit system 424 and the high-speed wireless circuit system 436 of the eyeglass device 100 may include a short-range transceiver (BluetoothTM or Bluetooth Low Energy (BLE)) and a wireless wide area network, local area network, or wide area network transceiver (e.g., cellular or Wi-Fi). The mobile device 401 (including a transceiver that communicates via a low-power wireless connection 425 and a high-speed wireless connection 437) can be implemented using details of the architecture of the eyeglass device 100, as can other elements of the network 495.

[0071] The memory 434 includes any storage device capable of storing various data and applications, including, among other things, camera data generated by the cameras 114A, 114B, the infrared camera(s) 410, the image processor 412, and images generated for display by the image display driver 442 on the image display of each optical assembly 180. Although the memory 434 is shown as being integrated with the high-speed circuit system 430, in other examples, the memory 434 can be a separate, separate element of the eyewear device 100. In some such examples, electrical routing lines can provide a connection from the image processor 412 or the low-power processor 422 to the memory 434 through a chip including the high-speed processor 432. In other examples, the high-speed processor 432 can manage addressing of the memory 434 so that the low-power processor 422 initiates the high-speed processor 432 whenever a read or write operation involving the memory 434 is to be performed.

[0072] like Figure 4 As shown, various elements of the eyeglass device 100 can be coupled to the low-power circuit system 420, the high-speed circuit system 430, or both. For example, the infrared camera 410 (including an infrared transmitter in some embodiments), the user input element 491 (e.g., a button switch, a touch pad 181, a microphone 139), and the inertial measurement unit (IMU) 472 can be coupled to the low-power circuit system 420, the high-speed circuit system 430, or both.

[0073] like Figure 5 As shown, CPU 540 of mobile device 401 may be coupled to camera system 570, mobile display driver 582, user input layer 591, and memory 540A, discussed in further detail below.

[0074] The server system 498 may be one or more computing devices as part of a service or network computing system, for example, including a processor, a memory, and a network communication interface to communicate with the eyeglass device 100 and the mobile device 401 over the network 495 .

[0075] The output component of the eyewear device 100 includes a visual element, such as a reference Figure 2A and Figure 2BEach lens or optical component 180 described is associated with an image display (e.g., a display such as a liquid crystal display (LCD), a plasma display panel (PDP), a light emitting diode (LED) display, a projector, or a waveguide). The eyeglass device 100 may include an indicator facing the user (e.g., an LED, a speaker 191, or a vibration actuator) or a signal facing outward (e.g., an LED, a speaker 191). The image display of each optical component 180 is driven by an image display driver 442. In some example configurations, the output components of the eyeglass device 100 also include additional indicators, such as an audible element (e.g., a speaker 191), a tactile component (e.g., an actuator for generating tactile feedback, such as a vibration motor), and other signal generators. For example, the device 100 may include a set of indicators facing the user and a set of signals facing outward. The set of indicators facing the user is configured to be seen or otherwise sensed by the user of the device 100. For example, the device 100 may include an LED display positioned so that the user can see it, one or more speakers 191 positioned to generate sounds that the user can hear, or an actuator that provides tactile feedback that the user can feel. The outwardly facing set of signals is configured to be seen or otherwise sensed by an observer near the device 100. Similarly, the device 100 may include an LED, speaker 191, or actuator configured and positioned to be sensed by an observer.

[0076] The user input elements 491 of the eyewear device 100 may include alphanumeric input components (e.g., a touch screen or touchpad 181 configured to receive alphanumeric input, an optical keyboard, or other alphanumeric configured components), pointer-based input components (e.g., a mouse, touchpad 181, trackball, joystick, motion sensor, or other directional instrument), tactile input components (e.g., a button switch, a touch screen or touchpad 181 or other tactile configured element that senses the location, force, or location and force of a touch or touch gesture), audio input components (e.g., a microphone 139), and the like. The mobile device 401 and the server system 498 may include alphanumeric, pointer-based, tactile, audio, and other input components.

[0077] In some examples, the eyeglass device 100 includes a collection of motion sensing components referred to as an IMU 472. The motion sensing component may be a microelectromechanical system (MEMS) with microscopic moving parts (usually small enough to be part of a microchip). In some example configurations, the IMU 472 includes an accelerometer, a gyroscope, and a magnetometer. The accelerometer senses the linear acceleration of the device 100 relative to three orthogonal axes (x, y, z) (including acceleration due to gravity). The gyroscope senses the angular velocity of the device 100 around three rotation axes (pitch, roll, yaw). The accelerometer and gyroscope can together provide position, orientation, and motion data about the device relative to six axes (x, y, z, pitch, roll, yaw). The magnetometer (if present) senses the heading of the device 100 relative to magnetic north. The position of the device 100 can be determined by a position sensor (such as a GPS unit 473), one or more transceivers for generating relative position coordinates, an altitude sensor or a barometer, and other orientation sensors. Such positioning system coordinates may also be received from mobile device 401 over wireless connections 425 , 437 via low power wireless circuitry 424 or high speed wireless circuitry 436 .

[0078] The IMU 472 may include or cooperate with a digital motion processor or programming that collects raw data from the components and calculates a plurality of useful values ​​about the position, orientation, and motion of the device 100. For example, the acceleration data collected from the accelerometer may be integrated to obtain the velocity relative to each axis (x, y, z); and integrated again to obtain the position of the device 100 (in the linear coordinate system x, y, and z). The angular velocity data from the gyroscope may be integrated to obtain the position of the device 100 (in the spherical coordinate system). The programming for calculating these useful values ​​may be stored in the memory 434 and executed by the high-speed processor 432 of the eyewear device 100.

[0079] The eyeglass device 100 may optionally include additional peripheral sensors, such as biometric sensors, dedicated sensors, or display elements integrated with the eyeglass device 100. For example, the peripheral device elements may include any I / O components, including output components, motion components, positioning components, or any other such components described herein. For example, the biometric sensor may include components for detecting expressions (e.g., hand expressions, facial expressions, vocal expressions, body postures, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweat, or brain waves), or identifying people (e.g., identification based on voice, retina, facial features, fingerprints, or electrical biosignals (such as electroencephalogram data)), etc.

[0080] The mobile device 401 may be a smartphone, tablet, laptop, access point, or any other such device capable of connecting to the eyewear device 100 using both a low power wireless connection 425 and a high speed wireless connection 437. The mobile device 401 is connected to a server system 498 and a network 495. The network 495 may include any combination of wired and wireless connections.

[0081] like Figure 4 As shown, the story interaction system 400 includes a computing device, such as a portable electronic device 401 (e.g., a mobile device), coupled to the eyewear device 100 via a network 495. The story interaction system 400 includes a memory (e.g., a non-transitory computer-readable medium) for storing instructions and a processor for executing instructions. In some embodiments, the memory and processing functions of the story interaction system 400 can be shared or distributed between the processors and memories of the eyewear device 100, the portable electronic device 401 (e.g., a mobile device), and / or the server system 498.

[0082] In some implementations, story interaction system 400 includes one or more elements or modules, referred to herein as interaction recording application 910 , positioning system 915 , image processing system 920 , and speech recognition module 925 .

[0083] In some implementations, the interaction recording application 910 presents a trigger interface 800 on a display. As described herein, in some implementations, the trigger interface 800 includes a question 810 and one or more answer buttons 812, each of which is associated with a specific user behavior 830.

[0084] In some embodiments, the positioning system 915 obtains positioning data for determining the current position of the portable electronic device 401 (e.g., a mobile phone) or the electronic eyewear device 100 relative to the physical environment 600. For example, the positioning system 915 can access the video data frame 900 captured using the camera 114B to determine the current eyewear device position 840 relative to the physical environment 600 in three-dimensional coordinates (with or without reference to data from other sources, such as an inertial measurement unit or IMU 472). As used herein, the term "video data frame" refers to video motion data captured using one or more cameras 114A, 114B coupled to the eyewear device 100, including images, spatial data, and related information captured using substantially any sensor component of the camera in any form and at any sampling rate. In some embodiments, the positioning data can be derived from the motion data frame captured using the IMU 472, from data collected by the GPS unit 473, or from a combination thereof.

[0085] In some embodiments, the image processing system 920 cooperates with the image display driver 442 and the image processor 412 to generate and present a trigger interface 800 as described herein on a display of a portable electronic device 401 (e.g., a mobile phone) or an electronic eyewear device 100. In some embodiments, the trigger interface 800 includes elements generated on the display and virtually presented as an overlay (e.g., in the foreground) relative to real objects in the physical environment 600.

[0086] In some embodiments, the speech recognition module 925 receives human speech using a microphone, converts the received speech into digital audio signals using a sound card, transmits the digital audio signals to a processor, identifies one or more discrete words that form a command, query, or other instruction, and coordinates or performs actions based on the identified instructions.

[0087] Figure 5 is a high-level functional block diagram of an example mobile device 401. Mobile device 401 includes flash memory 540A that stores programming to be executed by CPU 540 to perform all or a subset of the functionality described herein.

[0088] Mobile device 401 may include camera 570, which includes at least two cameras (e.g., a first visible light camera and a second visible light camera with overlapping fields of view) or at least one camera and a depth sensor with substantially overlapping fields of view. Flash memory 540A may also include multiple images or videos generated via camera 570.

[0089] As shown, the mobile device 401 includes an image display 580, a mobile display driver 582 that controls the image display 580, and a display controller 584. In some embodiments, the mobile device 401 includes a microphone 539 and a speaker 592. Figure 5 In the example of , image display 580 includes a user input layer 591 (eg, a touch screen) that is layered over or otherwise integrated into the screen used by image display 580 .

[0090] Examples of touch screen mobile devices that may be used include, but are not limited to, smartphones, personal digital assistants (PDAs), tablet computers, laptop computers, or other portable devices. However, the structure and operation of touch screen devices are provided by way of example; the subject technology as described herein is not intended to be limited thereto. For purposes of this discussion, therefore, Figure 5A block diagram illustration of an example mobile device 401 is provided with a user interface including a touch screen input layer 591 for receiving input (through touch, multi-touch, gestures, etc., by hand, stylus, or other implement) and a graphics display 580 for displaying content.

[0091] like Figure 5 As shown, the mobile device 401 includes at least one digital transceiver (XCVR) 510, shown as a WWAN XCVR, for digital wireless communication via a wide area wireless mobile communication network. The mobile device 401 also includes additional digital or analog transceivers, such as a short-range transceiver (XCVR) 520 for short-range network communication (such as via NFC, VLC, DECT, ZigBee, Bluetooth™ or Wi-Fi). For example, the short-range XCVR 520 can take the form of any available two-way wireless local area network (WLAN) transceiver, the type of which is compatible with one or more standard communication protocols implemented in wireless local area networks, such as one of the Wi-Fi standards under IEEE 802.11.

[0092] In order to generate location coordinates for locating the mobile device 401, the mobile device 401 may include a global positioning system (GPS) receiver. Alternatively or additionally, the mobile device 401 may utilize either or both of the short-range XCVR 520 and the WWAN XCVR 510 to generate location coordinates for positioning. For example, a positioning system based on a cellular network, Wi-Fi, or Bluetooth™ may generate accurate location coordinates, especially when used in combination. Such location coordinates may be transmitted to the eyewear device via the XCVR 510, 520 through one or more network connections.

[0093] In some examples, the mobile device 401 includes a collection of motion sensing components called an inertial measurement unit (IMU) 572 for sensing the position, orientation, and motion of the mobile device 401. The motion sensing component can be a microelectromechanical system (MEMS) with microscopic moving parts (usually small enough to be part of a microchip). In some example configurations, the inertial measurement unit (IMU) 572 includes an accelerometer, a gyroscope, and a magnetometer. The accelerometer senses the linear acceleration of the mobile device 401 relative to three orthogonal axes (x, y, z) (including the acceleration caused by gravity). The gyroscope senses the angular velocity of the mobile device 401 around three rotational axes (pitch, roll, yaw). The accelerometer and gyroscope can provide together the position, orientation, and motion data of the device relative to six degrees of freedom (x, y, z, pitch, roll, yaw) (sometimes referred to as 6DOF). The magnetometer (if present) senses the heading of the mobile device 401 relative to magnetic north.

[0094] The IMU 572 may include or cooperate with a digital motion processor or programming that collects raw data from the components and calculates a number of useful values ​​about the position, orientation, and motion of the device mobile device 401. For example, the acceleration data collected from the accelerometer may be integrated to obtain the velocity relative to each axis (x, y, z); and integrated again to obtain the position of the mobile device 401 (in the linear coordinate system x, y, and z). The angular velocity data from the gyroscope may be integrated to obtain the position of the mobile device 401 (in the spherical coordinate system). The programming for calculating these useful values ​​may be stored in one or more memory elements 540A, 540B, 540C and executed by the CPU 540 of the mobile device 401.

[0095] The transceivers 510, 520 (i.e., network communication interfaces) conform to one or more of the various digital wireless communication standards utilized by modern mobile networks. Examples of WWAN transceivers 510 include, but are not limited to, transceivers configured to operate in accordance with Code Division Multiple Access (CDMA) and 3rd Generation Partnership Project (3GPP) network technologies (including, for example, but not limited to, 3GPP Type 2 (or 3GPP2) and LTE, sometimes referred to as "4G"). For example, transceivers 510, 520 provide two-way wireless communication of information (including digitized audio signals, still images and video signals, web page information for display, and web-related input), as well as various types of mobile message communications to / from mobile device 401.

[0096] The mobile device 401 also includes a microprocessor that acts as a central processing unit (CPU), such as Figure 4 540 in . A processor is a circuit having the following elements that are structured and arranged to perform one or more processing functions, typically performing various data processing functions. Although discrete logic components can be used, the example utilizes components that constitute a programmable CPU. For example, a microprocessor includes one or more integrated circuit (IC) chips that combine electronic components for performing the functions of a CPU. For example, CPU 540 can be based on any known or available microprocessor architecture, such as Reduced Instruction Set Computing (RISC) using an ARM architecture, as commonly used today in mobile devices and other portable electronic devices. Of course, other arrangements of processor circuit systems can be used to constitute CPU 540 or processor hardware in smart phones, laptop computers, and tablet computers.

[0097] The CPU 540 acts as a programmable main controller for the mobile device 401 by configuring the mobile device 401 to perform various operations, for example, according to instructions or programming executable by the CPU 540. For example, such operations may include various general operations of the mobile device, as well as operations related to programming for applications on the mobile device. Although the processor can be configured by using hard-wired logic, a typical processor in the mobile device is a general-purpose processing circuit that is configured by executing programming.

[0098] The mobile device 401 includes a memory or storage system for storing programming and data. In this example, the memory system may include flash memory 540A, random-access memory (RAM) 540B and other memory components 540C as needed. RAM 540B serves as short-term storage for instructions and data processed by CPU 540, for example, as working data processing memory. Flash memory 540A typically provides long-term storage.

[0099] Thus, in the example of mobile device 401, flash memory 540A is used to store programming or instructions for execution by CPU 540. Depending on the type of device, mobile device 401 stores and runs a mobile operating system through which specific applications are executed. Examples of mobile operating systems include Google Android, Apple iOS (for iPhone or iPad devices), Windows Mobile, Amazon Fire OS, RIM BlackBerry OS, or similar systems.

[0100] The processor 432 in the eyeglass device 100 can construct a map of the environment around the eyeglass device 100, determine the position of the eyeglass device in the map of the environment, and determine the relative positioning of the eyeglass device and one or more objects in the mapped environment. The processor 432 can use a simultaneous localization and mapping (SLAM) algorithm applied to data received from one or more sensors to construct a map and determine the position and positioning information. The sensor data includes images received from one or two of the cameras 114A, 114B, (one or more) distances received from the laser rangefinder, positioning information received from the GPS unit 473, motion and acceleration data received from the IMU 572, or a combination of data from such sensors or from other sensors that provide data that helps determine positioning information. In the context of augmented reality, the SLAM algorithm is used to build and update a map of the environment while tracking and updating the position of the device (or user) in the mapped environment. The mathematical solution can be approximated using various statistical methods (such as particle filters, Kalman filters, extended Kalman filters, and covariance intersection). In a system including a high-definition (HD) camera that captures video at a high frame rate (e.g., 30 frames per second), the SLAM algorithm updates the map and the position of objects at least as frequently as the frame rate; in other words, the map and localization are calculated and updated 30 times per second.

[0101] The sensor data includes one or more images received from one or both cameras 114A, 114B, (one or more) distances received from a laser rangefinder, positioning information received from a GPS unit 473, motion and acceleration data received from an IMU 472, or a combination of data from such sensors or from other sensors that provide data that helps determine positioning information.

[0102] Figure 6 An example physical environment 600 and elements useful when using SLAM algorithms (simultaneous localization and mapping) and other types of tracking applications (e.g., natural feature tracking (NFT), hand tracking, foot tracking, etc.) are depicted. A portable electronic device 401 (e.g., a mobile phone) and a user 602 of the electronic eyewear device 100 are present in the example physical environment 600 (in Figure 6, the example physical environment is an indoor room). The processor 432 of the eyeglass device 100 (or the processor of the mobile phone 401) uses the captured image data to determine the position of the electronic device 100, 401 relative to one or more physical objects 604 (e.g., including the IoT object 50) within the environment 600, constructs a map of the environment 600 using a coordinate system of the environment 600 (e.g., a Cartesian coordinate system (x, y, z)), and determines the position of the device 100, 410 relative to the coordinate system. In addition, in some embodiments, the processor determines the precise orientation and position (x-prime, y-prime, z-prime, roll, pitch, and yaw) of the portable electronic device 401 or the eyeglass device 100 in the environment by using two or more position points (e.g., three position points 606a, 606b, and 606c) associated with a single object 604a, or by using one or more position points 606 associated with two or more objects 604a, 604b, 604c. An application on the portable electronic device 401 or the eyewear device 100 can present a virtual object 608 (such as Figure 6 keys shown) to view them on the display during the augmented reality experience.

[0103] In some examples, the positioning system 915 includes a virtual marker 610a associated with a virtual object 608 in the physical environment 600. In an augmented reality environment, in some embodiments, markers are registered at locations in the physical environment 600 to assist the electronic device in the task of tracking and updating the location of users, devices, and objects (virtual and physical) relative to the physical environment. Markers are sometimes registered to high-contrast physical objects (such as relatively dark objects mounted on a light-colored wall, such as framed pictures 604a) to assist cameras and other sensors in the task of detecting markers. Markers can be assigned and registered in memory by the eyewear device 100 operating in the environment. In some embodiments, markers are assigned and registered in the memory of other devices in the network.

[0104] The positioning system 915 tracks physical objects and virtual objects within the physical environment 600 relative to the eyeglass device 100. For physical objects 604 (e.g., example floor safe 604c, example IoT object 50), the positioning system 915 continuously analyzes the captured images of the physical environment 600 to identify the object 604 and determine its position relative to the eyeglass device 100 (e.g., by applying a SLAM algorithm). The positioning system 915 maintains and updates the determined position information of the physical object 604 in a memory so that the physical object 604 is tracked with the eyeglass device 100 if and when it moves through the physical environment 600. For virtual objects 608 (e.g., keys), the positioning system 915 establishes or specifies an initial position of the virtual object 608 corresponding to a position in the environment 600 or the physical object 604 (or, in some embodiments, a position relative to the eyeglass device 100). The positioning system 915 maintains and updates position information of the virtual object 608, such as according to a movement algorithm associated with the virtual object 608, in response to the eyewear device 100 moving through the environment, or a combination thereof, thereby tracking the virtual object 608 as the eyewear device 100 moves through the environment.

[0105] The marker may be encoded with information or otherwise linked to information. The marker may include positioning information, a physical code (such as a barcode or QR code; visible or hidden to the user), or a combination thereof. A set of data associated with the marker is stored in the memory 434 of the eyewear device 100. The set of data includes information about the marker 610a, the positioning (position and orientation) of the marker, one or more virtual objects, or a combination thereof. The marker positioning may include three-dimensional coordinates of one or more marker landmarks 616a, such as Figure 6 The corners of the roughly rectangular marker 610a shown in . The marker location can be expressed relative to real-world geographic coordinates, a marker coordinate system, the positioning of the eyewear device 100, or other coordinate systems. One or more virtual objects associated with the marker 610a can include any of a variety of materials (including still images, videos, audio, tactile feedback, executable applications, interactive user interfaces, and experiences) and combinations or sequences of such materials. In this context, any type of content that can be stored in memory and retrieved when the marker 610a is encountered or associated with an assigned marker can be classified as a virtual object. For example, Figure 6 The virtual key 608 shown in FIG. 6 is a virtual object displayed as a 2D or 3D still image at a marked location.

[0106] In one example, the marker 610a can be registered in memory as being located on the physical object 604a (e.g., Figure 6In another example, the marker may be registered in memory as a specific location relative to the eyewear device 100.

[0107] 7A to 7C A flowchart 700 of an example method for generating an interactive AR experience using a portable electronic device 401 is depicted. Although these steps are described with reference to the portable electronic device 401 (e.g., a mobile phone) or the eyewear device 100 described herein, those skilled in the art will understand other implementations of the steps for other types of devices based on the description herein. One or more of the steps shown and described may be performed simultaneously, in series, in an order different from that shown and described, or in combination with additional steps. Some steps may be omitted or repeated in some applications.

[0108] In some embodiments, the interaction recording application 910 described herein is launched in response to receiving a selection through a user interface (e.g., selecting from a menu, pressing a button, using a touchpad) or through some other input method (e.g., a gesture detected in a captured image, a finger touch on the touchpad 181, a voice command).

[0109] Fig. 7A Block 702 in FIG. 5 depicts example steps for presenting on the display 580 a trigger interface 800 associated with a record 840 of a story 70 comprised of one or more scenes 60. In some implementations, the trigger interface 800 facilitates the creation of a record 840 associated with a story 70 that includes one or more scenes 60. During the creation of each scene 60 in the story 70, the interaction record application 910 creates a record (e.g., record 840) of the various steps involved and the input received from the user. The example method uses, for example, Figure 5 The discussion is shown in the context of portable electronic device 401 , which includes microphone 539 , speaker 592 , camera 570 , display 580 , and memory 540 .

[0110] Fig. 7A Block 704 in describes example steps for presenting a question 810 and one or more answer buttons 812 on the display 580. For example, Figure 8B An example question 810 (eg, "What should trigger the next scene?") and two example answer buttons 812 are shown; namely, a first answer button 812a displaying "Speak a keyword" and a second answer button 812b displaying "Touch a virtual object."

[0111] Fig. 7ABlock 706 in illustrative embodiments depicts example steps of receiving user input associated with selecting an answer 814 from among one or more answer buttons 812 (eg, a tap on a display screen proximate to a desired option).

[0112] Fig. 7A Block 708 in describes an example step of recording user behavior 830 based on the answer 814. In some implementations, the user behavior includes speaking words and touching virtual objects.

[0113] Fig. 7A Block 710 in depicts example steps of storing user behavior 830 and answers 814 associated with scenario 60 and record 840 in memory 540 .

[0114] Fig. 7A 830, wherein the reaction 860 includes starting a subsequent recording 842 associated with the subsequent scene 62. In some embodiments, the process of starting the subsequent recording 842 includes presenting a scene identifier 835 (e.g., "scene 2 / 2") on the display 480, such as Fig.8D In some embodiments, the scene identifier 835 includes a reminder phrase (e.g., "hear: small") containing a transcription of the keyword. In this regard, the user is reminded that the current scene being created begins with someone speaking the keyword (e.g., "small").

[0115] Fig. 7A Block 714 in describes example steps of storing a record 840 and a subsequent record 842 in the memory 540, wherein the records 840, 842 include instructions associated with the story 70. In context, in some implementations, the records 840, 842, et seq., constitute a single continuous record.

[0116] Figure 7B Block 720 in describes example steps for presenting trigger interface 800 when user behavior 830 includes “speaking the word” (eg, based on answer 814 ).

[0117] Figure 7B Box 722 in FIG. 1 describes an example step of receiving a spoken word 820 via a microphone. The spoken word 820 may include multiple words, sentences, or even longer speech passages. In the scenario, the user takes the action of speaking the spoken word 820 in response to a question, such as Figure 8C The example question shown is “What is the keyword?” In some embodiments, the process of receiving the spoken word 820 includes presenting a graphical response 833 (e.g., an audio wave animation, such as Figure 8Cone shown) to provide feedback to the user that the spoken words 820 are being detected by microphone 539.

[0118] Figure 7B 724 in the example steps of identifying trigger words 830 based on the spoken words 820. In some embodiments, the process of identifying trigger words 830 includes using a speech recognition module 925 as described herein. In some embodiments, the speech recognition module receives the spoken words 820 through the microphone 539, converts the spoken words 820 into digital audio signals, and analyzes using ASR to identify trigger words 830 (e.g., one or more discrete words included in the spoken word(s) 820).

[0119] Figure 7B 861. In some embodiments, the word response 861 includes presenting a transcription 831 of the identified trigger word 830 on the display 580. In some embodiments, the word response 861 includes presenting a highlight on the display 580. In some embodiments, the word response 861 includes playing a sound or message through a speaker. In some embodiments, the word response 861 includes generating a tactile movement of the device, such as a vibration.

[0120] Figure 7B 728 in the example steps of storing the response 833 and the trigger word 830 associated with the record 840 in the memory 540. The trigger word 830 is stored as part of the record 840 of the current scene 60 for later use. In some implementations, during playback of the story 70, when the user speaks the trigger word 830 (e.g., speaks the word "red balloon"), the story will advance from the current scene 60 to the next or subsequent scene 62. In this example, the trigger word 830 triggers the progression of the story 70 from one scene to the next, thereby providing an interactive experience for readers and viewers of the story 70.

[0121] Figure 7C Box 730 in describes example steps for presenting the trigger interface 800 when the user behavior 830 includes “touching the virtual object” (eg, based on the answer 814 ).

[0122] Figure 7CBlock 732 in the example method describes example steps of capturing a video data frame 900 using a camera 570 coupled to a portable electronic device 410. In some embodiments, the process of capturing a video data frame 900 is ongoing during active use of the portable device 401. In other examples, the capture process begins in response to receiving a selection through a user interface or through some other input means. In some embodiments, the example method includes at least temporarily storing the captured video data frame 900 in a memory on the portable electronic device 401 so that the video data frame 900 is available for use including processing and analysis.

[0123] Figure 7C Block 734 in describes example steps for detecting a virtual object 840 at an object location 790 in the physical environment 600 using the captured frame of video data 900 .

[0124] In some implementations, the process of detecting the virtual object 840 includes determining or estimating the virtual object's position relative to the physical environment 600. In some implementations, the current object position 790 is estimated using a positioning system 915 as described herein.

[0125] In some embodiments, the positioning system 915 configures the interaction recording application 910 to obtain positioning data based on the captured video data frames 900, and in some embodiments, based on motion data collected by an inertial measurement unit (IMU). In some embodiments, the positioning system 915 uses a SLAM algorithm to build a virtual map of one or more objects within the camera field of view 304, as described herein, and the SLAM algorithm updates the map and the position of the objects at least as frequently as the camera captures the video data 900.

[0126] In some embodiments, the process of determining or estimating the virtual object position 790 relative to the physical environment 600 includes calculating the correlation between the object position 790 and the current portable electronic device position 440 (eg, Fig. 8A ). The term correlation refers to and includes one or more vectors, matrices, formulas, or other mathematical expressions sufficient to define the three-dimensional distance between the object location 790 and the current portable electronic device location 440. Because the positioning process occurs frequently, the correlation process between the object location 790 and the portable electronic device location 440 produces accurate and near real-time tracking of the object location 790 relative to the portable electronic device location 440.

[0127] Figure 7CBlock 736 in illustrative embodiments depicts example steps of using captured video data frame 900 to determine portable electronic device position 440 relative to virtual object position 790. As portable electronic device 401 moves through physical environment 600, its position relative to virtual object position 790 changes.

[0128] In some implementations, portable electronic device location 440 is estimated using positioning system 915 as described herein.

[0129] In some embodiments, the process of determining or estimating the portable electronic device position 440 relative to the virtual object position 790 includes calculating a correlation between the two positions. The term correlation refers to and includes one or more vectors, matrices, formulas, or other mathematical expressions sufficient to define a three-dimensional distance between the current portable electronic device position 440 and the current virtual object position 790. Because the positioning process occurs frequently, the correlation process produces accurate and near real-time tracking of the portable electronic device position 440 relative to the virtual object position 790. In some embodiments, the process of estimating the current portable electronic device position 440 is based on motion data frames captured using an IMU, or based on video data frames 900 captured using a camera, or a combination of both.

[0130] Figure 7C 738 in the figure describes an example step of detecting an intersection 835 between the user 602 and the virtual object 840. In some embodiments, the intersection 835 occurs between the hand of the user 602 (in the physical environment 600) and the virtual object 840 (in the virtual environment) (e.g., as mathematically defined by the virtual object position 790). The process of detecting the intersection 835 includes using a SLAM algorithm (simultaneous localization and mapping) or other types of tracking applications (e.g., natural feature tracking (NFT), hand tracking, etc.), as described herein. Fig.8D An example intersection point 835 between a hand and a virtual object 840 located at coordinates 791 is shown in FIG.

[0131] Figure 7C Box 740 in describes example steps for performing a touch response 862. In some embodiments, the touch response 862 includes presenting an object name 841 associated with the virtual object 840 on the display 580. In some embodiments, the touch response 862 includes presenting a highlight near the virtual object 840 on the display 580. In some embodiments, the touch response 862 includes playing a sound or message through a speaker. In some embodiments, the touch response 862 includes generating a tactile movement of the device.

[0132] Figure 7CBlock 742 in describes example steps of storing touch responses 862 and intersection points 835 associated with scene 60 and record 840 in memory 540 .

[0133] These steps are described with reference to a portable electronic device 401 (e.g., a mobile phone). In some embodiments, the methods and systems described herein can be executed and operated on the eyeglass device 100, the mobile phone 401, or another electronic device. For example, in some embodiments, the portable electronic device 401 is an electronic eyeglass device 100 as described herein. In this example, the camera 570 is the eyeglass camera 114B, the display 580 is the eyeglass display 182B, the IMU 572 is the eyeglass IMU 472, the memory elements 540A, 540B, 540C are the eyeglass memory 434, the microphone 539 is the eyeglass microphone 139, the speaker 592 is the eyeglass speaker 191, etc.

[0134] In some embodiments, the methods and systems described herein are processed and executed using a portable electronic device 401 (e.g., a mobile phone), and the trigger interface 800 is presented on the display 580 of the mobile phone 401, or on the display 182B of the eyeglass device 100, or both.

[0135] Any functions of the eyewear device 100, mobile device 401, and server system 498 described herein may be embodied in one or more computer software applications or programming instruction sets, as described herein. According to some examples, a "function," "multiple functions," "application," "multiple applications," "instruction," "multiple instructions," or "programming" is a program (or programs) that executes the functions defined in the program. Various programming languages ​​may be used to generate one or more applications that are structured in a variety of ways, such as an object-oriented programming language (e.g., Objective-C, Java, or C++) or a procedural programming language (e.g., C or assembly language). In a specific example, a third-party application (e.g., an application created by an entity other than the vendor of a particular platform using ANDROID TM or IOS TM Software development kit (SDK) applications can be developed on mobile operating systems such as IOS TM ANDROID TM , Mobile software running on a mobile phone or other mobile operating system. In this example, the third-party application can call API calls provided by the operating system to facilitate the functions described in this article.

[0136] Thus, machine-readable media can take the form of tangible / non-temporary storage media in a variety of forms. Non-volatile storage media include, for example, optical or magnetic disks, such as any storage device in any computer device or the like, such as can be used to implement the client device, media gateway, transcoder, etc. shown in the accompanying drawings. Volatile storage media include dynamic memory, such as the main memory of such a computer platform. Tangible transmission media include coaxial cables; copper wires and optical fibers, including wires that contain buses within a computer system. Carrier transmission media can take the form of electrical or electromagnetic signals or sound or light waves, such as those generated during radio frequency (RF) and infrared (IR) data communications. Therefore, common computer-readable media forms include, for example: floppy disks, floppy disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, DVDs or DVD-ROMs, any other optical media, punched card tapes, any other physical storage media with hole patterns, RAMs, PROMs and EPROMs, FLASH-EPROMs, any other memory chips or cassettes, carriers that transmit data or instructions, cables or links that transmit such carriers, or any other medium from which a computer can read programming code or data. Many of these forms of computer readable media may be involved in carrying one or more sequences of one or more instructions / program codes to a processor for execution.

[0137] Except as stated immediately above, nothing that has been stated or shown is intended or should be construed as conferring any component, step, feature, object, benefit, advantage, or equivalent to the public, whether or not recited in the claims.

[0138] It will be understood that the terms and expressions used herein have the common meanings given by such terms and expressions relative to their corresponding corresponding explorations and research fields, unless the specific meanings thereof have been set forth in addition herein. Relational terms (such as first and second, etc.) can be used only to distinguish one entity or action from another entity or action, without necessarily requiring or implying any actual such relationship or order between such entities or actions. The terms "include", "comprise", "contain", "cover" or any other variation thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including or including a series of elements or steps not only includes those elements or steps, but also may include other elements or steps that are not explicitly listed or inherent to such processes, methods, articles or devices. In the absence of further constraints, an element beginning with "one" or "an" does not exclude the presence of additional identical elements in the process, method, article or device including the element.

[0139] Unless otherwise indicated, any and all measurements, values, ratings, positions, amplitudes, dimensions, and other specifications set forth in this specification (including the claims that follow) are approximate and not exact. Such quantities are intended to have a reasonable range that is consistent with the functions to which they are related and with the customary practices in the art to which they belong. For example, unless otherwise expressly stated, a parameter value or similar value may vary from a stated amount or range by as much as ±10%.

[0140] In addition, in the foregoing detailed description, it can be seen that, for the purpose of streamlining the present disclosure, individual features are combined together in various examples. This method of disclosure should not be interpreted as reflecting such an intention that the examples claimed for protection require more features than those explicitly stated in each claim. On the contrary, as reflected in the following claims, the subject matter to be protected lies in less than all the features of any single disclosed example. Therefore, the following claims are hereby incorporated into the detailed description, with each claim existing independently as a separate claimed subject matter.

[0141] While the foregoing has described what are considered to be the best modes and other examples, it should be understood that various modifications may be made therein, and the subject matter disclosed herein may be implemented in various forms and examples, and they may be applied in many applications, only some of which are described herein. The following claims are intended to claim any and all modifications and variations that fall within the true scope of the present concept.

Claims

1. A method for generating an interactive AR experience using a portable electronic device, the portable electronic device comprising a microphone, a camera, a display, and a memory, the method comprising: Presenting, on the display, a trigger interface associated with a record, the record being associated with a story including one or more scenes, wherein the trigger interface includes: presenting a question and one or more answer buttons on the display; receiving user input associated with an answer selected from among the one or more answer buttons; Recording user behavior based on said answers; and The user behavior and the answer associated with the scenario and the record are stored in the memory.

2. The method according to claim 1, further comprising: executing a reaction associated with the user action, wherein the reaction includes starting a subsequent recording associated with a subsequent scenario; as well as The record and the subsequent record are stored in the memory, wherein the record includes instructions associated with the story.

3. The method according to claim 2, wherein: Performing the reaction includes an effect selected from the group consisting of moving through a physical environment, inserting a virtual object, interacting with a virtual object, deleting a virtual object, and connecting to an IoT product.

4. The method according to claim 2, wherein: The user behavior corresponds to an IoT product, and wherein performing the reaction further comprises: A control signal is sent to the IoT product, wherein the control signal includes an instruction for executing an IoT action corresponding to the user behavior.

5. The method according to claim 2, further comprising: presenting a play button on the display; as well as The story is played back based on the stored recording.

6. The method according to claim 1, wherein: The user behavior includes speaking a word, and the method further includes: receiving spoken words via the microphone; Identify trigger words based on the words spoken; performing a word response, the word response comprising an action selected from the group consisting of presenting a transcription of the identified trigger word on the display, presenting a highlight on the display, playing a sound through a speaker, playing a message through a speaker, and generating a haptic movement of the device; and A term response associated with the record and the trigger term are stored in the memory.

7. The method according to claim 6, wherein: Receiving spoken words also includes: presenting on the display a graphical response to the spoken words; and A next button is presented on the display, the next button configured to receive a selection to accept the trigger word and perform at least one of a reaction or the word response.

8. The method according to claim 1, wherein: The user behavior includes touching a virtual object, and the method further includes: capturing frames of video data using the camera; detecting the virtual object at an object position relative to a physical environment using the captured frames of video data; using the captured frames of video data to determine a position of the portable electronic device relative to the position of the object; detecting an intersection between the user and the virtual object; performing a touch response, the touch response comprising an action selected from the group consisting of presenting an object name associated with the virtual object on the display, presenting a highlight near the virtual object on the display, playing a sound through a speaker, playing a message through a speaker, and generating a tactile movement of the device; and The touch responses and the intersection points are stored in the memory in association with the scene and the record.

9. A portable electronic device comprising: microphone, camera, monitor, Processor; and a memory comprising instructions that, when executed by the processor, configure the portable electronic device to: presenting, on the display, a trigger interface associated with a recording, the recording being associated with a story including one or more scenes, wherein the trigger interface configures the portable electronic device to: presenting a question and one or more answer buttons on the display; receiving user input associated with an answer selected from among the one or more answer buttons; Recording user behavior based on said answers; as well as The user behavior and the answer associated with the scenario and the record are stored in the memory.

10. The portable electronic device according to claim 9, wherein: The instructions further configure the portable electronic device to: executing a reaction associated with the user action, wherein the reaction comprises starting a subsequent recording associated with a subsequent scenario; and The record and the subsequent record are stored in the memory, wherein the record includes instructions associated with the story.

11. The portable electronic device according to claim 10, wherein: Executing the reaction includes executing an effect selected from the group consisting of moving through a physical environment, inserting a virtual object, interacting with a virtual object, deleting a virtual object, and connecting to an IoT product.

12. The portable electronic device according to claim 11, wherein: The user behavior corresponds to an IoT product, and wherein in order to perform the reaction, the portable electronic device is configured to: A control signal is sent to the IoT product, wherein the control signal includes an instruction for executing an IoT action corresponding to the user behavior.

13. The portable electronic device according to claim 11, wherein: The instructions further configure the portable electronic device to: presenting a play button on the display; and The story is played back based on the stored recording.

14. The portable electronic device according to claim 9, wherein: The user action comprises speaking a word, and wherein the instructions further configure the portable electronic device to: receiving spoken words via the microphone; Identify trigger words based on the words spoken; performing a word response, the word response comprising an action selected from the group consisting of presenting a transcription of the identified trigger word on the display, presenting a highlight on the display, playing a sound through a speaker, playing a message through a speaker, and generating a haptic movement of the device; and A term response associated with the record and the trigger term are stored in the memory.

15. The portable electronic device according to claim 14, wherein: To receive the spoken words, the portable electronic device is configured to: presenting on the display a graphical response to the spoken words; and A next button is presented on the display, the next button configured to receive a selection to accept the trigger word and perform at least one of a reaction or the word response.

16. The portable electronic device according to claim 9, wherein: The user action includes touching a virtual object, and wherein the instructions further configure the portable electronic device to: capturing frames of video data using the camera; detecting the virtual object at an object position relative to a physical environment using the captured frames of video data; using the captured frames of video data to determine a position of the portable electronic device relative to the position of the object; detecting an intersection between the user and the virtual object; performing a touch response, the touch response comprising an action selected from the group consisting of presenting an object name associated with the virtual object on the display, presenting a highlight near the virtual object on the display, playing a sound through a speaker, playing a message through a speaker, and generating a tactile movement of the device; and The touch responses and the intersection points are stored in the memory in association with the scene and the record.

17. A non-transitory computer-readable medium storing instructions for generating an interactive AR experience using a portable electronic device including a microphone, a camera, a display, and a memory, the instructions, when executed by a processor, configuring the portable electronic device to perform the following functions, the functions comprising: Presenting, on the display, a trigger interface associated with a record, the record being associated with a story including one or more scenes, wherein the trigger interface includes: presenting a question and one or more answer buttons on the display; receiving user input associated with an answer selected from among the one or more answer buttons; Recording user behavior based on said answers; and The user behavior and the answer associated with the scenario and the record are stored in the memory.

18. The non-transitory computer readable medium of claim 17, wherein: The instructions, when executed, further configure the portable electronic device to perform the following functions: executing a reaction associated with the user action, wherein the reaction comprises starting a subsequent recording associated with a subsequent scenario; and The record and the subsequent record are stored in the memory, wherein the record includes instructions associated with the story.

19. The non-transitory computer readable medium of claim 18, wherein: Executing the reaction includes executing an effect selected from the group consisting of moving through a physical environment, inserting a virtual object, interacting with a virtual object, deleting a virtual object, and connecting to an IoT product.

20. The non-transitory computer readable medium of claim 18, wherein: The user behavior corresponds to an IoT product, and wherein performing the reaction further comprises: A control signal is sent to the IoT product, wherein the control signal includes an instruction for executing an IoT action corresponding to the user behavior.

Images