Authoring tool for creating interactive AR experience

Abstract

An authoring tool for creating an interactive AR experience is described. The story authoring application enables users with little or no programming skills to create interactive stories, including recording voice commands for advancing to the next scene, inserting and manipulating virtual objects in a mixed reality environment, and recording various interactions with connected IoT devices. The story creation interface is presented on the display as a virtual object in the AR environment.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. application serial number 17 / 901,540, filed on September 1, 2022, the contents of which are incorporated herein by reference in their entirety. Technical Field

[0003] Examples set forth in this disclosure relate to the field of augmented reality (AR) experiences for electronic devices, including portable electronic devices and wearable devices such as eyewear. More particularly, but not by way of limitation, this disclosure describes a story creation application that enables users to create interactive AR experiences that include interacting with virtual objects and with connected IoT devices. Background Art

[0004] Many types of computers and electronic devices available today, such as mobile devices (e.g., smartphones, tablets, and laptops), handheld devices, and wearable devices (e.g., smart glasses, digital glasses, headsets, headbands, and head-mounted displays), include various cameras, sensors, wireless transceivers, input systems, and displays. Users sometimes refer to information on these devices during physical activities such as exercise.

[0005] The so-called "Internet of Things" (IoT) refers to and includes the following physical products that are embedded with sensors, software, and other technologies for connecting and exchanging data with other devices in a network (usually via the Internet). For example, IoT products are used for home automation to control lighting, heating and air conditioning, media and security systems, and camera systems. Many IoT-enabled devices have been provided to act as smart home hubs to connect different smart home products. IoT devices are also used in many other applications. Application layer protocols and supporting frameworks have been provided to implement such IoT applications. For example, some IoT products include an application programming interface (API) that allows IoT products to pair with other products and electronic devices (such as portable computers) and communicate with them separately. Artificial intelligence has also been combined with the IoT infrastructure to achieve more efficient IoT network operation, improve human-computer interaction, and enhance data management and analysis.

[0006] Virtual reality (VR) technology generates a complete virtual environment including realistic images, which is sometimes presented on a VR headset or other head-mounted display. The VR experience allows the user to move around in the virtual environment and interact with virtual objects. AR is a VR technology that combines real objects in the physical environment with virtual objects and displays the combination to the user. The combined display gives the impression that the virtual objects are actually present in the environment, especially when the virtual objects appear and behave like real objects. Cross-reality (XR) is generally understood as an umbrella term for systems that include or combine elements from AR, VR, and MR (mixed reality) environments.

[0007] Automatic speech recognition (ASR) is a field of computer science, artificial intelligence, and linguistics that involves receiving spoken words and converting them into audio data suitable for processing by a computing device. Human speech received as an analog audio signal can be converted into a digital audio signal and analyzed using ASR to recognize discrete words in the speech. The recognized words can be used to perform actions or commands that are useful in controlling and interacting with computer software applications. ASR processing can be used in computers, handheld devices, wearable devices, phone systems, automobiles, and a variety of other devices to facilitate human-computer interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The features of the various examples described will be readily understood from the following detailed description with reference to the accompanying drawings. In the description and throughout the several views of the accompanying drawings, reference numerals are used with each element. When multiple similar elements are present, a single reference numeral may be assigned to the similar element, with additional uppercase or lowercase letters referring to the specific element.

[0009] Unless otherwise indicated, the various elements shown in the figures are not drawn to scale. For clarity, the size of the various elements may be enlarged or reduced. These figures depict one or more embodiments and are presented only as examples and should not be construed as limiting. The drawings include the following figures:

[0010] Figure 1A is a side view of an eyewear device suitable for use with an example story creation system (right);

[0011] Figure 1B yes Figure 1A a perspective partial cross-sectional view of the optical components and electronics in a portion of the eyewear device shown;

[0012] Figure 1C yes Figure 1A Side view of the eyewear device (left);

[0013] Figure 1D yes Figure 1Ca cross-sectional perspective view of the optical components and electronics in a portion of the eyewear device shown;

[0014] Figure 2A and 2B is a rear view of the eyewear device used in the example story creation system;

[0015] Figure 3 is a block diagram illustrating an example of using the example eyewear device shown in any of the preceding figures to capture visible light;

[0016] Figure 4 is a functional block diagram of an example story creation system including an eyewear device and a server system connected via various networks;

[0017] Figure 5 Is configured for Figure 4 An illustration of an example hardware configuration for a mobile device of an example story creation system;

[0018] Figure 6 is a stereogram of a user in an example environment, including physical and virtual objects, for reference in describing an example process known as simultaneous localization and mapping (SLAM);

[0019] Fig. 7A is a flow chart listing steps in an example method of generating an interactive AR experience, the method comprising presenting, on a display, a story creation interface including a keyword module;

[0020] Figure 7B is a flow chart listing steps in an example story creation interface including a virtual object module;

[0021] Figure 7C is a flowchart listing steps in an example story creation interface including an IoT product module;

[0022] Fig. 8A is an illustration of an example story creation interface presented on a display of a portable electronic device;

[0023] Figure 8B yes Fig. 8A An illustration of the example story creation interface, including a section of the example keyword module;

[0024] Figure 8C yes Fig. 8A An illustration of the example story creation interface, including another portion of the example keyword module;

[0025] Fig.8D yes Fig. 8A An illustration of an example story creation interface, including yet another portion of an example keyword module;

[0026] Fig. 9Ais an illustration of an example story creation interface presented on a display of a portable electronic device, including portions of an example IoT product module;

[0027] Fig. 9B yes Fig. 9A An illustration of the example story creation interface, including another section of the example IoT product module;

[0028] Fig. 9C yes Fig. 9A An illustration of an example story creation interface for , including yet another portion of an example IoT product module; and

[0029] Fig.9D yes Fig. 9A Illustration of the example story creation interface for , including another view of the example IoT product module. DETAILED DESCRIPTION

[0030] A story creation system for use with AR applications on portable electronic devices, including mobile phones and wearable devices such as electronic glasses. The story creation system enables users of portable electronic devices to author interactive stories for AR experiences that include voice commands, virtual objects, and interactions with connected IoT devices with little or no programming skills.

[0031] Various embodiments and details are described with reference to an example of generating an interactive AR experience using a portable electronic device including a microphone, a camera, a display, and a memory. In an example embodiment, the method involves presenting a story creation interface on a display. The story creation interface facilitates the creation of a record associated with a story including a continuous series of scenes. In some embodiments, the story creation interface includes a keyword module, a virtual object module for inserting virtual objects, and an IoT product module for interacting with an IoT product.

[0032] The example keyword module uses ASR to detect and recognize keywords, which are stored as part of the recording of the current scene. During story playback, when the user speaks the keyword (e.g., the word "small"), the story will advance from the current scene to the next or subsequent scene. In this example, speaking the keyword triggers the progression of the story from one scene to the next, providing an interactive experience for readers and viewers of the story.

[0033] The following detailed description includes systems, methods, techniques, instruction sequences, and computer program products that illustrate the examples set forth in this disclosure. Many details and examples are included for the purpose of providing a thorough understanding of the disclosed subject matter and its related teachings. However, those skilled in the relevant art can understand how to apply the related teachings without these details. The various aspects of the disclosed subject matter are not limited to the specific devices, systems, and methods described, because the related teachings can be applied or practiced in various ways. The terms and nomenclature used herein are only for the purpose of describing specific aspects, and are not intended to be restrictive. In general, well-known instruction instances, protocols, structures, and techniques are not necessarily shown in detail.

[0034] As used herein, the terms "connect," "connected," "coupled," and "coupled" refer to any logical, optical, physical, or electrical connection, including a link or the like, through which an electrical or magnetic signal generated or supplied by one system element is conducted to another coupled or connected system element. Unless otherwise specified, coupled or connected elements or devices are not necessarily directly connected to each other and may be separated by intermediate components, elements, or communication media, one or more of which can modify, manipulate, or carry electrical signals. The term "on" means supported directly by an element, or indirectly supported by an element through another element that is integrated into or supported by the element.

[0035] The term "proximal" is used to describe an item or part of an item that is near, adjacent to, or beside an object or person; or closer relative to other parts of the item may be described as "distal." For example, the end of an item closest to the object may be called the proximal end, while the generally opposite end may be called the distal end.

[0036] For purposes of illustration and discussion, the orientation of the eyeglass device, associated components, and any complete device incorporating an eye scanner and camera, such as those shown in any of the accompanying drawings, is given by way of example only. In operation for a particular variable optical processing application, the eyeglass device may be oriented in any other orientation (e.g., upward, downward, sideways, or any other orientation) suitable for the particular application of the eyeglass device. Additionally, as used herein, any directional terms, such as front, back, inward, outward, toward, left, right, lateral, longitudinal, up, down, upper, lower, top, bottom, and side, are used by way of example only and are not limiting with respect to the direction or orientation of any optical device or component of an optical device constructed as otherwise described herein.

[0037] Advanced AR technologies, such as computer vision and object tracking, can be used to produce perceptually rich and immersive experiences. Computer vision algorithms extract three-dimensional data about the physical world from data captured in digital images or videos. Object recognition and tracking algorithms are used to detect objects in digital images or videos, estimate their orientation or pose, and track their movement over time. Real-time recognition and tracking of body parts, such as hands and fingers, arms and legs, and feet, is one of the most challenging and processing-intensive tasks in the field of computer vision.

[0038] Additional objects, advantages and novel features of the examples will be set forth in part in the following description and in part will become apparent to those skilled in the art upon examination of the following and accompanying drawings, or may be learned by production or operation of the examples. The objects and advantages of the subject matter may be realized or attained by the methods, instrumentalities and combinations particularly pointed out in the appended claims.

[0039] In a sample configuration, an eyewear device with augmented reality (AR) capabilities is used in the system described herein. Eyewear devices are expected to be used in the system described herein because such devices are scalable and customizable to achieve a personalized experience, achieve the effect of being applied anytime, anywhere, and ensure user privacy by only allowing the user to see the transmitted information. In a sample configuration, the eyewear device can be used without any dedicated hardware, such as the SPECTACLES available from Snap, Inc. of Santa Monica, California. TM .

[0040] like Figures 1A-1D As shown, the eyewear device 100 includes a first camera 114A and a second camera 114B. The cameras 114 capture image information of a scene from separate viewpoints. The captured images can be used to project a three-dimensional display onto an image display for three-dimensional (3D) viewing.

[0041] Cameras 114 are sensitive to wavelengths in the visible light range. Each of cameras 114 defines a different forward-facing field of view that overlaps to enable generation of a three-dimensional depth image, e.g., first camera 114A defines a first field of view 111A, and second camera 114B defines a second field of view 111B. In general, a "field of view" is the portion of a scene visible through a camera at a particular location and orientation in space. Fields of view 111 have overlapping fields of view 304 ( Figure 3 ). When the camera captures an image, objects or object features outside the field of view 111 are not recorded in the original image (e.g., a photograph or picture). The field of view describes the angular range or degree to which the image sensor of the camera 114 picks up electromagnetic radiation of a given scene in the captured image of the given scene. The field of view can be expressed as the angular size of the viewing cone (i.e., the viewing angle). The viewing angle can be measured horizontally, vertically, or diagonally.

[0042] In an example configuration, one or both cameras 114 have a field of view of 100° and a resolution of 480x480 pixels. The "coverage angle" describes the range of angles that the lens of the camera 114 can effectively image. Typically, a camera lens produces an image circle large enough to completely cover the camera's film or sensor, which may include some vignetting (e.g., the image gets darker toward the edges compared to the center). If the coverage angle of the camera lens does not fill the sensor, the image circle will be visible, typically with strong vignetting toward the edges, and the effective viewing angle will be limited to the coverage angle.

[0043] Examples of suitable cameras 114 include high-resolution complementary metal oxide semiconductor (CMOS) image sensors and digital VGA cameras (video graphics arrays) capable of 480p (e.g., 640 x 480 pixels), 720p, 1080p, or higher resolutions. Other examples include cameras 114 that can capture high-definition (HD) video at high frame rates (e.g., 30 to 60 frames per second or more) and store recordings at a resolution of 1216 by 1216 pixels (or higher).

[0044] The eyewear device 100 can capture image sensor data from the camera 114 and geographic location data digitized by the image processor to be stored in a memory. The camera 114 captures corresponding raw images (e.g., left and right raw images) in a two-dimensional spatial domain, which includes a pixel matrix on a two-dimensional coordinate system, which includes an X-axis for horizontal positioning and a Y-axis for vertical positioning. Each pixel includes a color attribute value (e.g., a red pixel light value, a green pixel light value, or a blue pixel light value); and a positioning attribute (e.g., an X-axis coordinate and a Y-axis coordinate).

[0045] To capture stereoscopic images for later display as a 3D projection, image processor 412 ( Figure 4 ) may be coupled to the camera 114 to receive and store visual image information. The image processor 412 or another processor controls the operation of the camera 114 to act as a stereo camera that simulates human binocular vision, and a time stamp may be added to each image. The time stamp on each pair of images allows the images to be displayed together as part of a 3D projection. 3D projection creates an immersive, lifelike experience that is desirable in a variety of environments, including virtual reality (VR) and video games.

[0046] Figure 1B yes Figure 1A 1 is a perspective cross-sectional view of the right corner 110A of the eyewear device 100 depicting the first camera 114A, additional optical components and electronics. Figure 1C yes Figure 1A1 is a side view (left) of an example hardware configuration of the eyewear device 100 showing the second camera 114B of the camera system. Figure 1D yes Figure 1C 1 is a cross-sectional perspective view of a left corner 110B of the eyewear device 100 depicting a second camera 114B, additional optics, and electronics of the camera system.

[0047] like Figure 1B As shown in the example of FIG. 1 , the eyeglass device 100 includes a first camera 114A and a circuit board 140A, which may be a flexible printed circuit board (PCB). The first hinge 126A connects the right corner 110A to the first temple 125A of the eyeglass device 100. In some examples, the first camera 114A, the flexible PCB 140A, or other electrical connectors or contacts may be located on the first temple 125A or the first hinge 126A.

[0048] The right corner 110A includes a corner body 190 and a corner cap. Figure 1B The corner cap is omitted in the cross section of FIG. Disposed within the right corner 110A are various interconnected circuit boards 109, such as a flexible PCB 140A, which include circuit boards for the first camera 114A, (one or more) microphones 139, (one or more) speakers 191, low-power wireless circuit systems (e.g., for communicating via Bluetooth TM wireless short-range network communications), high-speed wireless circuit systems (for example, controller circuits for wireless LAN communications via Wi-Fi).

[0049] The first camera 114A is coupled to or disposed on the flexible PCB 140A and is covered by a camera cover lens that is aimed through the opening(s) formed in the frame 105. For example, Figure 2A As shown, the right edge 107A of the eyeglass frame 105 is connected to the right corner 110A and includes (one or more) openings for the camera cover lens. The eyeglass frame 105 includes a front side that is configured to face outward and away from the user's eyes. The opening for the camera cover lens is formed in and through the front side or outward facing side of the eyeglass frame 105. In this example, the first camera 114A has an outward facing field of view 111A (e.g., a line of sight or perspective associated with the right eye of the user of the eyeglass device 100) Figure 3 ). The camera cover lens may also be adhered to the front side or outward facing surface of the right corner 110A where the opening is formed with an outward facing coverage angle, but in a different outward direction. The coupling may also be indirect via an intervening component.

[0050] like Figure 1DAs shown in the example of FIG. 1 , the eyeglass device 100 includes a second camera 114B and a circuit board 140B, which may be a flexible printed circuit board (PCB). The second hinge 126B connects the left corner 110B to the second temple 125B of the eyeglass device 100. In some examples, the second camera 114B, the flexible PCB 140B, or other electrical connectors or contacts may be located on the second temple 125B or the second hinge 126B.

[0051] The left corner 110B includes a corner body 190 and a corner cap. Figure 1D The corner cap is omitted in the cross section of FIG. Disposed inside the right corner 110A are various interconnected circuit boards (such as a flexible PCB 140B) that include controller circuitry for the second camera 114B.

[0052] The camera 114 is coupled to or disposed on a corresponding flexible PCB 140 and is covered by a camera cover lens that is aimed through the opening(s) formed in the frame 105. For example, Figure 2A As shown, the right edge 107A of the frame 105 is connected to the right corner 110A and includes (one or more) openings for the camera cover lens and the left edge 107B of the frame 105 is connected to the left corner 110B and includes (one or more) openings for the camera cover lens. The frame 105 includes a front side that is configured to face outward and away from the user's eyes. The opening for the camera cover lens is formed in and through the front side or outward facing side of the frame 105. In this example, the camera 114 has a corresponding outward facing field of view 111 (e.g., a line of sight or perspective associated with a corresponding eye of a user of the eyewear device 100) Figure 3 ). The camera cover lens may also be adhered to the front side or outward facing surface of the corresponding corner 110, in which the opening is formed with an outward facing coverage angle, but in a different outward direction. The coupling may also be indirect via an intervening component.

[0053] Figure 2A and 2B An example hardware configuration of an eyeglass device 100 including two different types of image displays is depicted. The size and shape of the eyeglass device 100 are configured to be worn by a user. The form of the eyeglasses is shown as an example. The eyeglass device 100 may take other forms and may be combined with other types of frames, such as headgear, headphones, or helmets.

[0054] In the eyeglass example, the eyeglass device 100 includes a frame 105, which includes a right rim 107A connected to a left rim 107B via a nose bridge 106, and the nose bridge 106 is configured to receive the user's nose to support the eyeglass device 100 on the user's head. The right rim 107A includes a first rim 175A, which holds a first optical element 180A. The left rim 107B includes a second rim 175B, which holds a second optical element 180B. Figure 2B As shown, in some embodiments, each optical element 180A, 180B includes an integrated image display (e.g., a first display 182A and a second display 182B). As used herein, the term "lens" is intended to include a transparent or translucent glass or plastic sheet having a curved or flat surface that causes light to converge / diverge or causes little or no convergence / divergence.

[0055] A touch-sensitive input device such as a touch pad 181 is located on the first temple 125A. As shown, the touch pad 181 can have a clearly visible border, or include a raised or otherwise tactile edge that provides feedback to the user about the location and border of the touch pad 181; alternatively, the border can be subtle and not easily seen or felt. The eyeglass device 100 can include a touch pad on the other side that operates independently or in conjunction with the touch pad 181.

[0056] The surface of the touch pad 181 is configured to detect finger touches, taps, and gestures (e.g., moving touch) for use with a graphical user interface (GUI) displayed by the eyewear device on the image display to allow the user to browse and select menu options in an intuitive manner, which enhances and simplifies the user experience.

[0057] Detecting finger input on the touchpad 181 can achieve a variety of functions. For example, touching any position on the touchpad 181 can cause the GUI to display or highlight an item on the image display, which can be projected onto at least one of the optical components 180. Tapping or double-clicking the touchpad 181 can select an item or icon. Sliding or swiping a finger in a particular direction (e.g., from front to back, from back to front, from top to bottom, or from bottom to top) can cause an item or icon to slide or scroll in a particular direction; for example, move to the next item, icon, video, image, page, or slide. Sliding a finger in another direction can slide or scroll in the opposite direction; for example, move to the previous item, icon, video, image, page, or slide. The touchpad 181 can be positioned substantially anywhere on the eyeglass device 100.

[0058] In one example, a recognized single tap gesture on touch pad 181 initiates selection or pressing of a GUI element in an image presented on the image display of optical assembly 180. Adjusting the image presented on the image display of optical assembly 180 based on the recognized gesture can be a primary action that selects or submits a GUI element on the image display of optical assembly 180 for further display or execution.

[0059] Figure 2A 1 is an example hardware configuration of the eyeglass device 100 in which the right corner 110A supports a microphone 139 and a speaker 191. The microphone 139 includes a transducer that converts sound into a corresponding electrical audio signal. The microphone 139 in the example shown is positioned to have an opening facing inward toward the wearer to facilitate receiving sound waves, such as human speech including verbal commands and questions. Additional or differently oriented openings may be implemented. In other example configurations, the eyeglass device 100 is coupled to one or more microphones 139 that are configured to operate together or independently and are located at various locations on the eyeglass device 100.

[0060] The speaker 191 includes an electroacoustic transducer that converts an electrical audio signal into a corresponding sound. The speaker 191 is controlled by one of the processors 422, 432 or by the audio processor 413 ( Figure 4 ) control. As shown, the speaker 191 in this example includes a series of rectangular apertures that face inward to direct sound to the wearer. Additional or differently oriented apertures may be implemented. In other example configurations, the eyeglass device 100 is coupled to one or more speakers 191 that are configured to operate together (e.g., in stereo, to produce surround sound in a zone) or independently and are located at various locations on the eyeglass device 100. For example, one or more speakers 191 may be incorporated into the frame 105, temple 125, or corner 110 of the eyeglass device 100.

[0061] Despite Figure 2A and Figure 2B 100 is shown as having two optical elements 180, but depending on the application or intended user of the eyeglass device 100, the eyeglass device 100 may include other arrangements, such as a single optical element (or it may not include any optical element 180). As further shown, the eyeglass device 100 includes a right corner 110A adjacent to the right lateral side 170A of the frame 105 and a left corner 110B adjacent to the left lateral side 170B of the frame 105. The corners 110 may be integrated into the frame 105 on the respective sides 170 (as shown), or may be implemented as separate components attached to the frame 105 on the respective sides 170. Alternatively, the corners 110A, 110B may be integrated into temples attached to the frame 105 (not shown).

[0062] In one example, each image display of optical assembly 180 includes an integrated image display (eg, first display 182A and second display 182B). Figure 2A As shown, each optical assembly 180 has a display 182 including a suitable display matrix 177, such as a liquid crystal display (LCD), an organic light-emitting diode (OLED) display, or other such display. Each optical assembly 180 also includes one or more optical layers 176, which may include any combination of lenses, optical coatings, prisms, reflectors, waveguides, optical strips, and other optical components. The optical layers (e.g., Figure 2A The optical layer 176A-N (shown as 176A-N in FIG. 104 ) may include a prism having a suitable size and configuration and including a first surface for receiving light from the display matrix and a second surface for emitting light toward the user's eyes. The prisms of the optical layer 176A-N extend over all or at least a portion of the corresponding lens ring 175 formed in the left and right lens edges 107 to allow the user to see the second surface of the prism when the user's eyes look through the corresponding lens edge 107. The first surface of the prism of the optical layer 176A-N faces upward from the frame 105, and the display matrix 177 covers the prism so that photons and light emitted by the display matrix 177 strike the first surface. The prism is designed in size and shape so that light is refracted within the prism and directed toward the user's eyes through the second surface of the prism of the optical layer 176A-N. In this regard, the second surface of the prism of the optical layer 176A-N can be convex to direct light toward the center of the eye. The prisms may optionally be sized and shaped to magnify the image projected by the display matrix 177 , and light travels through the prisms so that the image viewed from the second surface is larger in one or more dimensions than the image emitted from the display matrix 177 .

[0063] In one example, the optical layers 176A-N may include an LCD layer that is transparent (holding the lens open) unless and until a voltage is applied that renders the layer opaque (closing or blocking the lens). The image processor 412 on the eyewear device 100 may be programmed to apply a voltage to the LCD layer, thereby creating an active shutter system, making the eyewear device 100 suitable for viewing visual content when displayed as a 3D projection. Technologies other than LCD may be used for active shutter modes, including other types of reactive layers that respond to a voltage or another type of input.

[0064] In another example, the image display device of the optical assembly 180 has a display 182 that includes a display such as Figure 2BEach optical assembly 180 includes a corresponding laser projector 150, such as a three-color projector using a scanning mirror or a galvanometer. Each laser projector 150 is disposed in or on a corresponding temple 125 of the eyeglass device 100. In this example, each optical assembly 180 includes one or more optical strips (such as Figure 2B 155A-N), these optical strips span the width of the lens of each optical component 180 or span the depth of the lens between the front and back surfaces of the lens and are spaced apart.

[0065] As the photons projected by the laser projector 150 travel through the lens of each optical assembly 180, the photons encounter the optical strips 155A-N. When a particular photon encounters a particular optical strip, the photon is redirected toward the user's eye, or it passes to the next optical strip. The combination of modulation of the laser projector 150 and modulation of the optical strips controls specific photons or light beams. In an example, the processor controls the optical strips 155A-N by issuing a mechanical signal, an acoustic signal, or an electromagnetic signal. Although shown as having two optical assemblies 180, the eyeglass device 100 may include other arrangements, such as a single or three optical assemblies, or each optical assembly 180 may have a different arrangement depending on the application or intended user of the eyeglass device 100.

[0066] Figure 3 3D scene 306, a first raw image 302A captured using a first camera 114A, and a second raw image 302B captured using a second camera 114B. As shown, the first field of view 111A can overlap with the second field of view 111B. The overlapping field of view 304 represents the portion of the image captured using the two cameras 114. When referring to the field of view, the term "overlap" means that the pixel matrices in the generated raw images overlap by thirty percent (30%) or more. "Substantially overlap" means that the pixel matrices in the generated raw images or infrared images of the scene overlap by fifty percent (50%) or more. As described herein, the two raw images 302 can be processed to include a timestamp, which allows the images to be displayed together as part of a three-dimensional projection.

[0067] To capture stereo images, such as Figure 3 As shown, a pair of raw red, green, and blue (RGB) images of a 3D scene 306 are captured at a given moment—a first raw image 302A captured using a first camera 114A and a second raw image 302B captured using a second camera 114B. When the pair of raw images 302 are processed (e.g., by an image processor 412), a depth image is generated. The generated depth image can be viewed on an optical assembly 180 of the eyewear device, on another display (e.g., an image display 580 on a mobile device 401), or on a screen.

[0068] The generated depth image is in a three-dimensional spatial domain and may include a matrix of vertices on a three-dimensional position coordinate system including an X-axis for horizontal positioning (e.g., length), a Y-axis for vertical positioning (e.g., height), and a Z-axis for depth (e.g., distance). Each vertex may include a color attribute (e.g., a red pixel light value, a green pixel light value, or a blue pixel light value); a positioning attribute (e.g., an X position coordinate, a Y position coordinate, and a Z position coordinate); a texture attribute; a reflectivity attribute; or a combination thereof. The texture attribute quantifies the perceived texture of the depth image, such as the spatial arrangement of color or intensity in a vertex region of the depth image.

[0069] Figure 4 4 is a functional block diagram of an example story creation system 400, which includes a glasses device 100, a mobile device 401, and a server system 498 connected via various networks 495 (such as the Internet). As shown, the story creation system 400 includes a low-power wireless connection 425 and a high-speed wireless connection 437 between the glasses device 100 and the mobile device 401.

[0070] As described herein, the eyewear device 100 includes one or more cameras 114 that capture still images, video images, or both still images and video images. The camera 114 may have direct memory access (DMA) to the high-speed circuit system 430 and be used as a stereo camera. The camera 114 may be used to capture initial depth images that may be rendered as three-dimensional (3D) models that are texture-mapped images of red, green, and blue (RGB) imaging scenes. The device 100 may also include a depth sensor that uses infrared signals to estimate the position of an object relative to the device 100. In some examples, the depth sensor includes one or more infrared emitters and one or more infrared cameras 410.

[0071] The eyeglass device 100 also includes two image displays (one associated with the right side 170A and one associated with the left side 170B) of the optical assembly 180. The eyeglass device 100 also includes an image display driver 442, an image processor 412, a low power circuit system 420, and a high speed circuit system 430. The image display of the optical assembly 180 is used to present images, including still images, video images, or both still images and video images. The image display driver 442 is coupled to the image display of the optical assembly 180 to control the display of the image.

[0072] Figure 4The components of the eyeglass device 100 shown in FIG. 1 are located on one or more circuit boards, such as a printed circuit board (PCB) or a flexible printed circuit (FPC) located in the rim or temple. Alternatively or additionally, the depicted components may be located in the corners, frame, hinge, or bridge of the eyeglass device 100. The camera 114 includes a digital camera element, such as a complementary metal oxide semiconductor (CMOS) image sensor, a charge coupled device, a lens, or any other corresponding visible light or light capturing element that can be used to capture data, including still images or video of a scene with an unknown object.

[0073] like Figure 4 As shown, the high-speed circuit system 430 includes a high-speed processor 432, a memory 434, and a high-speed wireless circuit system 436. In the example, an image display driver 442 is coupled to the high-speed circuit system 430 and is operated by the high-speed processor 432 to drive the image display of the optical assembly 180. The high-speed processor 432 can be substantially any processor capable of managing high-speed communications and operations of any general-purpose computing system. The high-speed processor 432 includes the processing resources required to manage high-speed data transmission on a high-speed wireless connection 437 to a wireless local area network (WLAN) using the high-speed wireless circuit system 436.

[0074] In some examples, the high-speed processor 432 executes an operating system, such as a LINUX operating system or other such operating system of the eyeglass device 100, and the operating system is stored in the memory 434 for execution. In addition to any other duties, the high-speed processor 432 also executes a software architecture for the eyeglass device 100, which is used to manage data transmission with the high-speed wireless circuit system 436. In some examples, the high-speed wireless circuit system 436 is configured to implement the Institute of Electrical and Electronics Engineers (IEEE) 802.11 communication standard, also referred to as Wi-Fi herein. In other examples, other high-speed communication standards can be implemented by the high-speed wireless circuit system 436.

[0075] The low power circuit system 420 includes a low power processor 422 and a low power wireless circuit system 424. The low power wireless circuit system 424 and the high speed wireless circuit system 436 of the eyeglass device 100 may include a short range transceiver (Bluetooth TM or Bluetooth Low Energy (BLE)) and a wireless wide area network, local area network, or wide area network transceiver (e.g., cellular or Wi-Fi). Mobile device 401 (including a transceiver that communicates via low power wireless connection 425 and high speed wireless connection 437) can be implemented using details of the architecture of eyewear device 100, as can other elements of network 495.

[0076] The memory 434 includes any storage device capable of storing various data and applications, including, among other things, camera data generated by the cameras 114A, 114B, one or more infrared cameras 410, the image processor 412, and images generated for display by the image display driver 442 on the image display of each optical assembly 180. Although the memory 434 is shown as being integrated with the high-speed circuit system 430, in other examples, the memory 434 can be a separate, separate element of the eyewear device 100. In some such examples, electrical routing lines can provide a connection from the image processor 412 or the low-power processor 422 to the memory 434 through a chip including the high-speed processor 432. In other examples, the high-speed processor 432 can manage addressing of the memory 434 so that the low-power processor 422 initiates the high-speed processor 432 whenever a read or write operation involving the memory 434 is to be performed.

[0077] like Figure 4 As shown, various elements of the eyeglass device 100 can be coupled to the low-power circuit system 420, the high-speed circuit system 430, or both. For example, the infrared camera 410 (including an infrared transmitter in some embodiments), the user input element 491 (e.g., a button switch, a touch pad 181, a microphone 139), and the inertial measurement unit (IMU) 472 can be coupled to the low-power circuit system 420, the high-speed circuit system 430, or both.

[0078] like Figure 5 As shown, CPU 540 of mobile device 401 may be coupled to camera system 570, mobile display driver 582, user input layer 591, and memory 540A, discussed in further detail below.

[0079] The server system 498 may be one or more computing devices as part of a service or network computing system, for example, including a processor, a memory, and a network communication interface to communicate with the eyeglass device 100 and the mobile device 401 over the network 495 .

[0080] The output component of the eyewear device 100 includes a visual element, such as Figure 2A and 2BEach lens or optical component 180 described is associated with an image display (e.g., a display such as a liquid crystal display (LCD), a plasma display panel (PDP), a light emitting diode (LED) display, a projector, or a waveguide). The eyeglass device 100 may include an indicator facing the user (e.g., an LED, a speaker 191, or a vibration actuator) or a signal facing outward (e.g., an LED, a speaker 191). The image display of each optical component 180 is driven by an image display driver 442. In some example configurations, the output components of the eyeglass device 100 also include additional indicators, such as an audible element (e.g., a speaker 191), a tactile component (e.g., an actuator for generating tactile feedback, such as a vibration motor), and other signal generators. For example, the device 100 may include a set of indicators facing the user and a set of signals facing outward. The set of indicators facing the user is configured to be seen or otherwise sensed by the user of the device 100. For example, the device 100 may include an LED display positioned so that the user can see it, one or more speakers 191 positioned to generate sounds that the user can hear, or an actuator that provides tactile feedback that the user can feel. The outwardly facing set of signals is configured to be seen or otherwise sensed by an observer near the device 100. Similarly, the device 100 may include an LED, speaker 191, or actuator configured and positioned to be sensed by an observer.

[0081] The user input elements 491 of the eyewear device 100 may include alphanumeric input components (e.g., a touch screen or touchpad 181 configured to receive alphanumeric input, an optical keyboard, or other alphanumeric configured components), pointer-based input components (e.g., a mouse, touchpad 181, trackball, joystick, motion sensor, or other directional instrument), tactile input components (e.g., a button switch, a touch screen or touchpad 181 or other tactile configured element that senses the location, force, or location and force of a touch or touch gesture), audio input components (e.g., a microphone 139), and the like. The mobile device 401 and the server system 498 may include alphanumeric, pointer-based, tactile, audio, and other input components.

[0082] In some examples, the eyeglass device 100 includes a collection of motion sensing components referred to as an IMU 472. The motion sensing component may be a microelectromechanical system (MEMS) with microscopic moving parts (usually small enough to be part of a microchip). In some example configurations, the IMU 472 includes an accelerometer, a gyroscope, and a magnetometer. The accelerometer senses the linear acceleration of the device 100 relative to three orthogonal axes (x, y, z) (including acceleration due to gravity). The gyroscope senses the angular velocity of the device 100 around three rotation axes (pitch, roll, yaw). The accelerometer and gyroscope can together provide position, orientation, and motion data about the device relative to six axes (x, y, z, pitch, roll, yaw). The magnetometer (if present) senses the heading of the device 100 relative to magnetic north. The position of the device 100 can be determined by a position sensor (such as a GPS unit 473), one or more transceivers for generating relative position coordinates, an altitude sensor or a barometer, and other orientation sensors. Such positioning system coordinates may also be received from mobile device 401 over wireless connections 425 , 437 via low power wireless circuitry 424 or high speed wireless circuitry 436 .

[0083] The IMU 472 may include or cooperate with a digital motion processor or programming that collects raw data from the components and calculates a plurality of useful values ​​about the position, orientation, and motion of the device 100. For example, the acceleration data collected from the accelerometer may be integrated to obtain the velocity relative to each axis (x, y, z); and integrated again to obtain the position of the device 100 (in the linear coordinate system x, y, and z). The angular velocity data from the gyroscope may be integrated to obtain the position of the device 100 (in the spherical coordinate system). The programming for calculating these useful values ​​may be stored in the memory 434 and executed by the high-speed processor 432 of the eyewear device 100.

[0084] The eyeglass device 100 may optionally include additional peripheral sensors, such as biometric sensors, dedicated sensors, or display elements integrated with the eyeglass device 100. For example, the peripheral device element may include any I / O component including an output component, a motion component, a positioning component, or any other such component described herein. For example, the biometric sensor may include components for detecting expressions (e.g., hand expressions, facial expressions, vocal expressions, body postures, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweat, or brain waves), or identifying people (e.g., recognition based on voice, retina, facial features, fingerprints, or electrical biosignals (such as electroencephalogram data)), etc.

[0085] The mobile device 401 may be a smartphone, tablet, laptop, access point, or any other such device capable of connecting to the eyewear device 100 using both a low power wireless connection 425 and a high speed wireless connection 437. The mobile device 401 is connected to a server system 498 and a network 495. The network 495 may include any combination of wired and wireless connections.

[0086] like Figure 4 As shown, the story creation system 400 includes a computing device, such as a portable electronic device 401 (e.g., a mobile device), coupled to the eyewear device 100 via a network 495. The story creation system 400 includes a memory (e.g., a non-transitory computer-readable medium) for storing instructions and a processor for executing instructions. In some embodiments, the memory and processing functions of the story creation system 400 can be shared or distributed between the processors and memories of the eyewear device 100, the portable electronic device 401 (e.g., a mobile device), and / or the server system 498.

[0087] In some implementations, story creation system 400 includes one or more elements or modules, referred to herein as story authoring application 910 , positioning system 915 , image processing system 920 , and speech recognition module 925 .

[0088] In some implementations, the story creation application 910 presents the story creation interface 800 on the display. As described herein, in some implementations, the story creation interface 800 includes a keyword module 891, a virtual object module 892, and an IoT product module 893.

[0089] In some embodiments, the positioning system 915 obtains positioning data for determining the current position of the portable electronic device 401 (e.g., a mobile phone) or the electronic eyewear device 100 relative to the physical environment 600. For example, the positioning system 915 can access the video data frame 900 captured using the camera 114B to determine the current eyewear device position 840 relative to the physical environment 600 in three-dimensional coordinates (with or without reference to data from other sources, such as an inertial measurement unit or IMU 472). As used herein, the term "video data frame" refers to video motion data captured using one or more cameras 114A, 114B coupled to the eyewear device 100, including images, spatial data, and related information captured using substantially any sensor component of the camera in any form and at any sampling rate. In some embodiments, the positioning data can be derived from the motion data frame captured using the IMU 472, from data collected by the GPS unit 473, or from a combination thereof.

[0090] In some embodiments, the image processing system 920 cooperates with the image display driver 442 and the image processor 412 to generate and present the story creation interface 800 as described herein on the display of the portable electronic device 401 (e.g., a mobile phone) or the electronic eyewear device 100. In some embodiments, the story creation interface 800 includes elements generated relative to real objects in the physical environment 600 and virtually presented on the display as an overlay (e.g., in the foreground).

[0091] In some embodiments, the speech recognition module 925 receives human speech using a microphone, converts the received speech into digital audio signals using a sound card, sends the digital audio signals to a processor, recognizes one or more discrete words that form a command, query, or other instruction, and coordinates or performs actions based on the recognized instructions.

[0092] Figure 5 is a high-level functional block diagram of an example mobile device 401. Mobile device 401 includes flash memory 540A that stores programming to be executed by CPU 540 to perform all or a subset of the functionality described herein.

[0093] Mobile device 401 may include camera 570, which includes at least two cameras (e.g., a first visible light camera and a second visible light camera with overlapping fields of view) or at least one camera and a depth sensor with substantially overlapping fields of view. Flash memory 540A may also include multiple images or videos generated via camera 570.

[0094] As shown, the mobile device 401 includes an image display 580, a mobile display driver 582 that controls the image display 580, and a display controller 584. In some embodiments, the mobile device 401 includes a microphone 539 and a speaker 592. Figure 5 In the example of , image display 580 includes a user input layer 591 (eg, a touch screen) that is layered over or otherwise integrated into the screen used by image display 580 .

[0095] Examples of touch screen type mobile devices that may be used include, but are not limited to, smartphones, personal digital assistants (PDAs), tablet computers, laptop computers, or other portable devices. However, the structure and operation of touch screen type devices are provided by way of example; the subject technology as described herein is not intended to be limited thereto. For purposes of this discussion, therefore, Figure 5A block diagram illustration of an example mobile device 401 is provided with a user interface including a touch screen input layer 591 for receiving input (through touch, multi-touch, gestures, etc., by hand, stylus, or other implement) and a graphics display 580 for displaying content.

[0096] like Figure 5 As shown, the mobile device 401 includes at least one digital transceiver (XCVR) 510, shown as a WWAN (Wireless Wide Area Network) XCVR, for digital wireless communication via a wide area wireless mobile communication network. The mobile device 401 also includes additional digital or analog transceivers, such as for short-range network communication (such as via NFC, VLC, DECT, ZigBee, Bluetooth TM For example, the short-range XCVR 520 may take the form of any available two-way wireless local area network (WLAN) transceiver of a type compatible with one or more standard communication protocols implemented in wireless local area networks, such as one of the Wi-Fi standards under IEEE 802.11.

[0097] To generate location coordinates for locating the mobile device 401, the mobile device 401 may include a global positioning system (GPS) receiver. Alternatively or additionally, the mobile device 401 may utilize either or both of the short-range XCVR 520 and the WWAN XCVR 510 to generate location coordinates for positioning. For example, based on a cellular network, Wi-Fi, or Bluetooth TM The positioning system of the present invention can generate accurate position coordinates, especially when used in combination. Such position coordinates can be sent to the eyewear device via XCVR 510, 520 through one or more network connections.

[0098] In some examples, the mobile device 401 includes a collection of motion sensing components called an inertial measurement unit (IMU) 572 for sensing the position, orientation, and motion of the mobile device 401. The motion sensing component can be a microelectromechanical system (MEMS) with microscopic moving parts (usually small enough to be part of a microchip). In some example configurations, the inertial measurement unit (IMU) 572 includes an accelerometer, a gyroscope, and a magnetometer. The accelerometer senses the linear acceleration of the mobile device 401 relative to three orthogonal axes (x, y, z) (including the acceleration caused by gravity). The gyroscope senses the angular velocity of the mobile device 401 around three rotational axes (pitch, roll, yaw). The accelerometer and gyroscope can provide together the position, orientation, and motion data of the device relative to six degrees of freedom (x, y, z, pitch, roll, yaw) (sometimes referred to as 6DOF). The magnetometer (if present) senses the heading of the mobile device 401 relative to magnetic north.

[0099] The IMU 572 may include or cooperate with a digital motion processor or programming that collects raw data from the components and calculates a number of useful values ​​about the position, orientation, and motion of the device mobile device 401. For example, the acceleration data collected from the accelerometer may be integrated to obtain the velocity relative to each axis (x, y, z); and integrated again to obtain the position of the mobile device 401 (in the linear coordinate system x, y, and z). The angular velocity data from the gyroscope may be integrated to obtain the position of the mobile device 401 (in the spherical coordinate system). The programming for calculating these useful values ​​may be stored in one or more memory elements 540A, 540B, 540C and executed by the CPU 540 of the mobile device 401.

[0100] The transceivers 510, 520 (i.e., network communication interfaces) conform to one or more of the various digital wireless communication standards utilized by modern mobile networks. Examples of WWAN transceivers 510 include, but are not limited to, transceivers configured to operate in accordance with Code Division Multiple Access (CDMA) and 3rd Generation Partnership Project (3GPP) network technologies (including, for example, but not limited to, 3GPP Type 2 (or 3GPP2) and LTE, sometimes referred to as "4G"). For example, transceivers 510, 520 provide two-way wireless communication of information (including digitized audio signals, still images and video signals, web page information for display, and web-related input), as well as various types of mobile message communications to / from mobile device 401.

[0101] The mobile device 401 also includes a microprocessor that acts as a central processing unit (CPU), such as Figure 4 540 in . A processor is a circuit having the following elements that are structured and arranged to perform one or more processing functions, typically performing various data processing functions. Although discrete logic components can be used, the example utilizes components that constitute a programmable CPU. For example, a microprocessor includes one or more integrated circuit (IC) chips that combine electronic components for performing the functions of a CPU. For example, CPU 540 can be based on any known or available microprocessor architecture, such as Reduced Instruction Set Computing (RISC) using an ARM architecture, as commonly used today in mobile devices and other portable electronic devices. Of course, other arrangements of processor circuit systems can be used to constitute CPU 540 or processor hardware in smart phones, laptop computers, and tablet computers.

[0102] The CPU 540 acts as a programmable main controller for the mobile device 401 by configuring the mobile device 401 to perform various operations, for example, according to instructions or programming executable by the CPU 540. For example, such operations may include various general operations of the mobile device, as well as operations related to programming for applications on the mobile device. Although the processor can be configured by using hard-wired logic, a typical processor in the mobile device is a general-purpose processing circuit that is configured by executing programming.

[0103] The mobile device 401 includes a memory or storage system for storing programming and data. In this example, the memory system may include flash memory 540A, random-access memory (RAM) 540B and other memory components 540C as needed. RAM 540B serves as short-term storage for instructions and data processed by CPU 540, for example, as working data processing memory. Flash memory 540A typically provides long-term storage.

[0104] Thus, in the example of mobile device 401, flash memory 540A is used to store programming or instructions for execution by CPU 540. Depending on the type of device, mobile device 401 stores and runs a mobile operating system through which specific applications are executed. Examples of mobile operating systems include Google Android, Apple iOS (for iPhone or iPad devices), Windows Mobile, Amazon Fire OS, RIM BlackBerry OS, or similar systems.

[0105] The processor 432 in the eyeglass device 100 can construct a map of the environment around the eyeglass device 100, determine the position of the eyeglass device in the map of the environment, and determine the relative positioning of the eyeglass device and one or more objects in the mapped environment. The processor 432 can use a simultaneous localization and mapping (SLAM) algorithm applied to data received from one or more sensors to construct a map and determine the position and positioning information. The sensor data includes images received from one or two of the cameras 114A, 114B, one or more distances received from the laser rangefinder, positioning information received from the GPS unit 473, motion and acceleration data received from the IMU 572, or a combination of data from such sensors or from other sensors that provide data that helps determine positioning information. In the context of augmented reality, the SLAM algorithm is used to build and update a map of the environment while tracking and updating the position of the device (or user) in the mapped environment. The mathematical solution can be approximated using various statistical methods (such as particle filters, Kalman filters, extended Kalman filters, and covariance intersection). In a system including a high-definition (HD) camera that captures video at a high frame rate (e.g., 30 frames per second), the SLAM algorithm updates the map and the location of objects at least as frequently as the frame rate; in other words, the map and localization are calculated and updated 30 times per second.

[0106] The sensor data includes one or more images received from one or both cameras 114A, 114B, one or more distances received from a laser rangefinder, positioning information received from a GPS unit 473, motion and acceleration data received from an IMU 472, or a combination of data from such sensors or from other sensors that provide data that helps determine positioning information.

[0107] Figure 6 An example physical environment 600 and elements useful when using SLAM algorithms (simultaneous localization and mapping) and other types of tracking applications (e.g., natural feature tracking (NFT), hand tracking, foot tracking, etc.) are depicted. A portable electronic device 401 (e.g., a mobile phone) and a user 602 of the electronic eyewear device 100 are present in the example physical environment 600 (in Figure 6, the example physical environment is an interior room). The processor 432 of the eyeglass device 100 (or the processor of the mobile phone 401) uses the captured image data to determine the position of the electronic device 100, 401 relative to one or more physical objects 604 (e.g., including the IoT object 50) within the environment 600, constructs a map of the environment 600 using a coordinate system of the environment 600 (e.g., a Cartesian coordinate system (x, y, z)), and determines the position of the device 100, 410 relative to the coordinate system. In addition, in some embodiments, the processor determines the precise orientation and position (x-prime, y-prime, z-prime, roll, pitch, and yaw) of the portable electronic device 401 or the eyeglass device 100 in the environment by using two or more position points (e.g., three position points 606a, 606b, and 606c) associated with a single object 604a, or by using one or more position points 606 associated with two or more objects 604a, 604b, 604c. The application on the portable electronic device 401 or the eyewear device 100 can present a virtual object 608 (eg, Figure 6 keys shown) to view them on the display during the augmented reality experience.

[0108] In some examples, the positioning system 915 includes a virtual marker 610a associated with a virtual object 608 in the physical environment 600. In an augmented reality environment, in some embodiments, markers are registered at locations in the physical environment 600 to assist the electronic device in the task of tracking and updating the location of users, devices, and objects (virtual and physical) relative to the physical environment. Markers are sometimes registered to high-contrast physical objects (such as relatively dark objects mounted on a light-colored wall, such as framed pictures 604a) to assist cameras and other sensors in the task of detecting markers. Markers can be assigned and registered in memory by the eyewear device 100 operating in the environment. In some embodiments, markers are assigned and registered in the memory of other devices in the network.

[0109] The positioning system 915 tracks physical objects and virtual objects within the physical environment 600 relative to the eyeglass device 100. For physical objects 604 (e.g., example floor safe 604c, example IoT object 50), the positioning system 915 continuously analyzes the captured images of the physical environment 600 to identify the object 604 and determine its position relative to the eyeglass device 100 (e.g., by applying a SLAM algorithm). The positioning system 915 maintains and updates the determined position information of the physical object 604 in a memory so that the physical object 604 is tracked as the eyeglass device 100 moves through the physical environment 600. For virtual objects 608 (e.g., keys), the positioning system 915 establishes or specifies an initial position of the virtual object 608 corresponding to a position in the environment 600 or the physical object 604 (or, in some embodiments, a position relative to the eyeglass device 100). The positioning system 915 maintains and updates position information of the virtual object 608, such as according to a movement algorithm associated with the virtual object 608, in response to movement of the eyewear device 100 in the environment, or a combination thereof, thereby tracking the virtual object 608 as the eyewear device 100 moves through the environment.

[0110] The marker may be encoded with information or otherwise linked to information. The marker may include positioning information, a physical code (such as a barcode or QR code; visible or hidden to the user), or a combination thereof. A set of data associated with the marker is stored in the memory 434 of the eyewear device 100. The set of data includes information about the marker 610a, the positioning (position and orientation) of the marker, one or more virtual objects, or a combination thereof. The marker positioning may include three-dimensional coordinates of one or more marker landmarks 616a, such as Figure 6 The marker location can be expressed relative to real-world geographic coordinates, a marker coordinate system, the positioning of the eyewear device 100, or other coordinate systems. One or more virtual objects associated with the marker 610a may include any of a variety of materials (including still images, videos, audio, tactile feedback, executable applications, interactive user interfaces, and experiences) and combinations or sequences of such materials. In this case, any type of content that can be stored in memory and retrieved when the marker 610a is encountered or associated with an assigned marker can be classified as a virtual object. For example, Figure 6 The virtual key 608 shown in FIG. 6 is a virtual object displayed as a 2D or 3D still image at a marked location.

[0111] In one example, the marker 610a can be registered in memory as being located on the physical object 604a (e.g., Figure 6 In another example, the mark can be registered in the memory as a specific location relative to the eyewear device 100.

[0112] Figures 7A-7C A flowchart 700 of an example method for generating an interactive AR experience using a portable electronic device 401 is depicted. Although the steps are described with reference to the portable electronic device 401 (e.g., a mobile phone) or the eyewear device 100 described herein, for other types of devices, those skilled in the art will understand other implementations of the steps based on the description herein. One or more of the steps shown and described may be performed simultaneously, in series, in an order different from that shown and described, or in combination with additional steps. Certain steps may be omitted or repeated in certain applications.

[0113] In some implementations, the story creation application 910 described herein is launched in response to receiving a selection through a user interface (e.g., selection from a menu, pressing a button, using a touchpad) or through some other input method (e.g., a gesture detected in a captured image, a finger touch 681 on a touchpad 181, a voice command).

[0114] Fig. 7A Block 701 in FIG. 4 describes example steps for presenting a story creation interface 800 on the display 580 of the portable electronic device 401. In some implementations, the story creation interface 800 facilitates creating a record 840 associated with a story 70 that includes one or more scenes 60. The story creation application 910 creates a record (e.g., record 840) of the various steps involved in creating each scene 60 in the story 70 and the input received from the user. Figure 5 As shown, the example methods are discussed in the context of using a portable electronic device 401 that includes a microphone 539 , a speaker 592 , a camera 570 , a display 580 , and a memory 540 .

[0115] In some implementations, the story creation interface 800 includes a keyword module 891 , a virtual object module 892 , and an IoT product module 893 .

[0116] Fig. 7A Block 702 in illustrative embodiment depicts an example keyword module 891 of the story creation interface 800 , which begins with the example step of presenting a question 810 and one or more answer buttons 812 on the display 580 .

[0117] Fig. 8A is an illustration of an example story creation interface 800 presented on display 580 of portable electronic device 401 .

[0118] Figure 8B 8 is a diagram of an example story creation interface 800 including a portion of an example keyword module 891. As shown, Figure 8BAn example step is shown of presenting a question 810 (e.g., "What triggers the next scene?") and one or more answer buttons 812 (e.g., a single answer button 812, "Say a keyword") on the display 580. In some implementations, more than one answer button 812 is presented on the display.

[0119] In some implementations, the process of presenting the question 810 and the answer buttons 812 includes receiving user input, such as a finger tap on the answer button 812 associated with the desired selection.

[0120] Fig. 7A Block 704 in Figure 5 describes an example step of receiving a spoken word 820 (e.g., "little") via microphone 539. Spoken word 820 may include multiple words, sentences, or even longer speech passages. In this case, the user takes the action of speaking spoken word 820 in response to question 810 (and selected answer button 812). In some embodiments, the process of receiving spoken word 820 includes presenting a graphical response 833 (e.g., an audio wave animation, such as Fig. 9C ), to provide feedback to the user that the spoken word 820 is being detected by the microphone 539.

[0121] Fig. 7A Block 706 in describes an example step of identifying keywords 830 based on spoken words 820. In some embodiments, the process of identifying keywords 830 includes using a speech recognition module 925 as described herein. In some embodiments, the speech recognition module receives the spoken words 820 via a microphone 539, converts the spoken words 820 into digital audio signals, and analyzes them using ASR to identify keywords 830 (e.g., one or more discrete words contained in the spoken words 820). Figure 8C In the example shown, keyword 830 has been identified as the single word "small".

[0122] Fig. 7A Block 708 in FIG. 5 describes an example step of presenting a transcription 831 of the identified keyword 830 on the display 580. Figure 8C As shown, transcript 831 includes the word "small" as identified keyword 830. In some embodiments, presenting transcript 831 also includes presenting a "next" button, such as Figure 8C as shown, and the Repeat button (in the upper right corner of the display, as shown Figure 8C ), which, if selected, will allow the system to repeat the process of receiving another spoken word 820.

[0123] Fig. 7ABlock 710 in 70 describes an example step of storing a keyword 830 in association with a record 840 in the memory 540. The keyword 830 is stored as part of the record 840 of the current scene 60 for later use. In some implementations, during playback of the story 70, when the user speaks the keyword 830 (e.g., the word "small"), the story will advance from the current scene 60 to the next or subsequent scene 62. In this example, the keyword 830 triggers the progression of the story 70 from one scene to the next, providing an interactive experience for readers and viewers of the story 70.

[0124] Fig. 7A Block 712 in 400 describes example steps for starting a subsequent recording 842 associated with the subsequent scene 62. In some embodiments, the process of starting the subsequent recording 842 includes presenting a scene identifier 835 (e.g., "scene 2 / 2") on the display 480, such as Fig.8D In some embodiments, the scene identifier 835 includes a reminder phrase (e.g., "hear: small") that contains a transcription 831 of the identified keyword 830. In this regard, the user is reminded that the current scene being created begins with someone saying the keyword 830 (e.g., "small").

[0125] Fig. 7A Block 714 in describes an example step of storing a record 840 and a subsequent record 842 in the memory 540, wherein the records 840, 842 together and in sequence constitute instructions associated with the story 70. In this case, in some embodiments, the records 840, 842, etc. constitute a single continuous record.

[0126] Fig. 7A Block 716 in FIG. 70 describes an example step of playing back the story 70 based on the stored recordings 840, 842. In some embodiments, the process of playing back includes pressing or selecting a play button 818 (eg, Fig.9D shown).

[0127] In some embodiments, the story creation interface 800 includes a virtual object module 892, an example of which is Fig. 9A Start showing.

[0128] like Fig. 9A As shown, scene identifier 835 indicates that this is "Scene 3", and the reminder phrase (e.g., "Hear: Red Balloon") indicates that the current scene will begin (during playback) when someone says keyword 830 (e.g., "Red Balloon"). Scene identifier 835 also includes a new scene button 819 (e.g., a plus sign in a circle) and a previous scene button 811. In some embodiments, tapping the new scene button 819 allows the system to initiate the keyword module 891 and request the use of keyword 830 when starting the next or new scene.

[0129] Figure 7B Block 718 in describes example steps for presenting the story creation interface 800 including the virtual object module 892 .

[0130] Figure 7B Block 720 in FIG. 5 describes presenting on display 580 in the lower right corner of display 580 as shown in FIG. Fig. 9A 815 (eg, a cube-shaped icon in a circle). In some implementations, presenting the virtual object icon 815 includes receiving user input, such as a finger tapping the icon 815.

[0131] Figure 7B Block 722 in FIG. 8 depicts an example step of receiving a spoken name 832 via a microphone. The spoken name 832 may include multiple words, sentences, or even longer speech passages. In one example, the user responds to a prompt or question 830 (e.g., Fig. 9B The action of speaking the spoken name 832 is taken by the user by responding to the user by saying “say the name of the model (e.g., “balloon”)” as shown, or in another example, “say the name of the virtual object you want to insert (e.g., cloud)”. In some embodiments, the process of receiving the spoken name 832 includes presenting a graphical response 833 (e.g., an audio wave animation, such as Fig. 9C ), to provide feedback to the user that the spoken name 832 is being detected by the microphone 539.

[0132] Figure 7B Block 724 in describes an example step of identifying a virtual object name 834 based on a spoken name 832. In some implementations, the process of identifying the keyword 830 includes using a speech recognition module 925 as described herein. Fig. 9C In the example shown, the virtual object name 834 has been identified as a single word "balloon".

[0133] Figure 7B Block 726 in describes example steps of retrieving a virtual object 840 from memory based on the virtual object name 834 .

[0134] Figure 7B Block 728 in FIG. 5 describes example steps for presenting a virtual object 840 on the display 580 at an object location 790 relative to the physical environment 600. Fig. 9C As shown, a virtual object 840 (e.g., a balloon) has been presented on the display at the object position 790. In some embodiments, as shown, a transcription 834 of the virtual object name 834 (e.g., "Balloon") is also presented on the display. In some embodiments, the process of presenting the virtual object 840 also includes presenting a pair of buttons, such as Figure 8C"Say Again" and "Confirm" are shown. The "Say Again" button acts as a repeat request and, if selected, will allow the system to repeat the process of receiving another spoken name 832.

[0135] Figure 7B Block 730 in describes example steps of storing a virtual object name 834 , a virtual object 840 , and an object position 790 in association with a record 840 of the current scene 60 in a memory.

[0136] Fig.9D A virtual object 840 (eg, a balloon) is shown that has been moved relative to its original position on the display. In some implementations, the user can move or otherwise manipulate the virtual object 840 as part of the current scene being created.

[0137] In some implementations, the story creation interface 800 includes an IoT product module 893 .

[0138] Figure 7C Block 732 in describes example steps for presenting a story creation interface 800 including an IoT product module 893 . Fig.8D An example IoT product 50 (eg, a lamp) is shown located at an object location 791 .

[0139] In some embodiments, an IoT product 50 (e.g., a light, a speaker, a fan) is paired with a portable electronic device 401 or an electronic eyeglass device 100. Some IoT products 50 include an application programming interface (API) (e.g., a product API) that allows the IoT product 50 to be paired with and otherwise communicate with other products and electronic devices (such as a portable electronic device 401 and an electronic eyeglass device 100). In some embodiments, the product API includes information such as the product type (e.g., a light, a speaker, a fan), and in some embodiments, a list of one or more controllable features associated with the IoT product 50 (e.g., on, off, brightness, color temperature, tint, volume, speed).

[0140] Figure 7C Block 734 in describes example steps of capturing a video data frame 900 with a camera 570 coupled to a portable electronic device 410. In some embodiments, the process of capturing the video data frame 900 is ongoing during active use of the portable device 401. In other examples, the capture process begins in response to receiving a selection through a user interface or through some other input means. In some embodiments, the example method includes at least temporarily storing the captured video data frame 900 in a memory on the portable electronic device 401 so that the video data frame 900 can be used for purposes including processing and analysis.

[0141] Figure 7C Block 736 in the example steps of detecting the IoT product 50 at the IoT product location 791 in the physical environment 600 using the captured video data frame 900. As described herein, in some embodiments, the IoT product 50 has been paired with or otherwise wirelessly communicated with the portable electronic device 401. In this regard, the detection process occurs when (and only if) the IoT product 50 is within a threshold proximity of the portable electronic device 401.

[0142] In some embodiments, the process of detecting the IoT product 50 includes determining or estimating the IoT product location 791 relative to the physical environment 600. In some embodiments, the current IoT product location 791 is estimated using the positioning system 915 described herein.

[0143] In some implementations, the positioning system 915 configures the story creation application 910 to obtain positioning data based on the captured video data frames 900, and in some implementations, based on motion data collected by an inertial measurement unit (IMU). In some implementations, the positioning system 915 constructs a virtual map of one or more objects within the camera field of view 304 using a SLAM algorithm, as described herein, and updates the map and the positions of the objects at least as frequently as the camera captures the video data 900.

[0144] In some embodiments, the process of determining or estimating the IoT product location 791 relative to the physical environment 600 includes calculating a correlation between the IoT product location 791 and the current portable electronic device location 440 (e.g., Fig. 8A ). The term correlation refers to and includes one or more vectors, matrices, formulas, or other mathematical expressions sufficient to define the three-dimensional distance between the IoT product location 791 and the current portable electronic device location 440. Because the positioning process occurs frequently, the correlation process between the IoT product location 791 and the portable electronic device location 440 produces accurate and near real-time tracking of the IoT product location 791 relative to the portable electronic device location 440.

[0145] Figure 7C Block 738 in describes example steps of using the captured video data frame 900 to determine the portable electronic device location 440 relative to the IoT product location 791. As the portable electronic device 401 moves in the physical environment 600, its location relative to the IoT product location 791 changes.

[0146] In some implementations, portable electronic device location 440 is estimated using positioning system 915 as described herein.

[0147] In some embodiments, the process of determining or estimating the portable electronic device location 440 relative to the IoT product location 791 includes calculating the correlation between the two locations. The term correlation refers to and includes one or more vectors, matrices, formulas, or other mathematical expressions sufficient to define the three-dimensional distance between the current portable electronic device location 440 and the current IoT product location 791. Because the positioning process occurs frequently, the correlation process produces accurate and near real-time tracking of the portable electronic device location 440 relative to the IoT product location 791. In some embodiments, the process of estimating the current portable electronic device location 440 is based on motion data frames captured using an IMU, or based on video data frames 900 captured using a camera, or a combination of both.

[0148] Figure 7C Block 740 in describes example steps for presenting an IoT interface when the device location 440 is detected within a threshold proximity of the IoT product 50. In some embodiments, the process of presenting the IoT interface includes presenting the interface when the electronic device location 440 is within a threshold proximity of the IoT product 50 (e.g., by mathematically evaluating the distance between the electronic device location 440 and the IoT product location 791). In this example, the IoT interface appears when the electronic device 401 is close to the IoT product 50.

[0149] Figure 7C Block 742 in describes example steps for interaction between detection device 401 and IoT interface. In some embodiments, IoT product 50 includes a product API (application programming interface) or is associated with a product API. In some embodiments, the product API includes controllable features (e.g., brightness, volume) and variable value ranges (e.g., from zero to one hundred percent, from one to eleven, etc.). The product API includes or can be used to construct an IoT interface suitable for receiving input and selections associated with the controllable features.

[0150] Figure 7C Block 744 in describes example steps of storing the IoT product 50 , the IoT product location 791 , and the detected interaction in association with the record 840 in the memory 540 .

[0151] These steps are described with reference to a portable electronic device 401 (e.g., a mobile phone). In some embodiments, the methods and systems described herein can be executed and operated on the eyeglass device 100, the mobile phone 401, or another electronic device. For example, in some embodiments, the portable electronic device 401 is an electronic eyeglass device 100 as described herein. In this example, the camera 570 is the eyeglass camera 114B, the display 580 is the eyeglass display 182B, the IMU 572 is the eyeglass IMU 472, the memory elements 540A, 540B, 540C are the eyeglass memory 434, the microphone 539 is the eyeglass microphone 139, the speaker 592 is the eyeglass speaker 191, etc.

[0152] In some embodiments, the methods and systems described herein are processed and executed using a portable electronic device 401 (e.g., a mobile phone), and the story creation interface 800 is presented on the display 580 of the mobile phone 401, or on the display 182B of the eyeglass device 100, or both.

[0153] Any functions of the eyewear device 100, mobile device 401, and server system 498 described herein may be embodied in one or more computer software applications or programming instruction sets, as described herein. According to some examples, a "function," "multiple functions," "application," "multiple applications," "instruction," "multiple instructions," or "programming" is a program (or programs) that executes the functions defined in the program. Various programming languages ​​may be used to generate one or more applications that are structured in a variety of ways, such as an object-oriented programming language (e.g., Objective-C, Java, or C++) or a procedural programming language (e.g., C or assembly language). In a specific example, a third-party application (e.g., an application created by an entity other than the vendor of a particular platform using ANDROID TM or IOS TM Software development kit (SDK) applications can be developed on mobile operating systems such as IOS TM ANDROID TM , Mobile software running on a mobile phone or other mobile operating system. In this example, the third-party application can call API calls provided by the operating system to facilitate the functions described in this article.

[0154] Thus, machine-readable media can take the form of tangible / non-temporary storage media in a variety of forms. Non-volatile storage media include, for example, optical or magnetic disks, such as any storage device in any computer device or the like, such as can be used to implement the client device, media gateway, transcoder, etc. shown in the accompanying drawings. Volatile storage media include dynamic memory, such as the main memory of such a computer platform. Tangible transmission media include coaxial cables; copper wires and optical fibers, including wires that contain buses within a computer system. Carrier transmission media can take the form of electrical or electromagnetic signals or sound or light waves, such as those generated during radio frequency (RF) and infrared (IR) data communications. Therefore, common computer-readable media forms include, for example: floppy disks, floppy disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, DVDs or DVD-ROMs, any other optical media, punched card tapes, any other physical storage media with hole patterns, RAMs, PROMs and EPROMs, FLASH-EPROMs, any other memory chips or cassettes, carriers that transmit data or instructions, cables or links that transmit such carriers, or any other medium from which a computer can read programming code or data. Many of these forms of computer readable media may be involved in carrying one or more sequences of one or more instructions / program codes to a processor for execution.

[0155] Except as stated immediately above, nothing that has been stated or shown is intended or should be construed as conferring any component, step, feature, object, benefit, advantage, or equivalent to the public, whether or not recited in the claims.

[0156] It will be understood that the terms and expressions used herein have the common meanings given by such terms and expressions relative to their corresponding corresponding explorations and research fields, unless the specific meanings thereof have been set forth in addition herein. Relational terms (such as first and second, etc.) can be used only to distinguish one entity or action from another entity or action, without necessarily requiring or implying any actual such relationship or order between such entities or actions. The terms "include", "comprise", "contain", "cover" or any other variation thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including or including a series of elements or steps not only includes those elements or steps, but also may include other elements or steps that are not explicitly listed or inherent to such processes, methods, articles or devices. In the absence of further constraints, an element beginning with "one" or "an" does not exclude the presence of additional identical elements in the process, method, article or device including the element.

[0157] Unless otherwise indicated, any and all measurements, values, ratings, positions, amplitudes, dimensions, and other specifications set forth in this specification (including the claims that follow) are approximate and not exact. Such quantities are intended to have a reasonable range that is consistent with the functions to which they are related and with the customary practices in the art to which they belong. For example, unless otherwise expressly stated, a parameter value or similar value may vary from a stated amount or range by as much as ±10%.

[0158] In addition, in the foregoing detailed description, it can be seen that, for the purpose of streamlining the present disclosure, individual features are grouped together in various examples. This method of disclosure should not be interpreted as reflecting an intention that the examples claimed require more features than those explicitly stated in each claim. On the contrary, as reflected in the following claims, the subject matter to be protected lies in less than all the features of any single disclosed example. Therefore, the following claims are hereby incorporated into the detailed description, with each claim existing independently as a separate claimed subject matter.

[0159] While the foregoing has described what are considered to be the best modes and other examples, it should be understood that various modifications may be made therein, and the subject matter disclosed herein may be implemented in various forms and examples, and they may be applied in many applications, only some of which are described herein. The following claims are intended to claim any and all modifications and variations that fall within the true scope of the present concept.

Claims

1. A method for generating an interactive AR experience using a portable electronic device including a microphone, a camera, a display, and a memory, the method comprising: Presenting a story creation interface on the display to create a record associated with a story including one or more scenes, wherein the story creation interface includes a keyword module, a virtual object module, and an IoT product module, and wherein the keyword module includes: presenting a question and answer buttons on said display; receiving spoken words via said microphone; identifying keywords based on the spoken words; presenting a transcription of the identified keyword on the display; storing the keyword in association with the recording in the memory, wherein detection of sound of the keyword during playback triggers the start of a subsequent scene; commencing a subsequent recording associated with the subsequent scene; and The record and the subsequent record are stored in the memory, wherein the records together include instructions associated with the story.

2. The method of claim 1, wherein starting the subsequent recording further comprises: presenting on said display a scene identifier and said transcription of the identified keywords; presenting a new scene button on the display; and The keyword module associated with the subsequent record is started.

3. The method of claim 1 , wherein receiving the spoken word further comprises: presenting on said display a graphical response to said spoken word; and A next button is presented on the display, the next button being configured to receive a selection to accept the keyword and to perform the step of starting the subsequent recording.

4. The method according to claim 1, further comprising: presenting a play button on the display; and The story is played back based on the stored recording.

5. The method according to claim 1, wherein the story creation interface further comprises a virtual object module, wherein the virtual object module comprises: presenting a virtual object icon on the display; presenting instructions on the display; receiving a spoken name via the microphone; identifying a virtual object name based on the spoken name; retrieving a virtual object from the memory based on the virtual object name; presenting the virtual object on the display at the virtual object's location relative to a physical environment; and The virtual object name, the virtual object, and the object position are stored in the memory in association with the record.

6. The method according to claim 5, wherein presenting the virtual object further comprises: presenting candidate virtual objects for consideration; and A transcription of the virtual object name is presented at a default location relative to the virtual object location.

7. The method according to claim 1, wherein the story creation interface further comprises an IoT product module, wherein the IoT product module comprises: capturing frames of video data with the camera; detecting the IoT product at its location in the physical environment using the captured video data frames; Using the captured video data frames to determine a location of the portable electronic device relative to the location of the IoT product; When the device location is detected within a threshold proximity of an IoT product, presenting an IoT interface; detecting an interaction between the device and the IoT interface; and The IoT product, the IoT product location, and the detected interactions associated with the record are stored in the memory.

8. A portable electronic device comprising: microphone, camera, monitor, processor, and a memory storing instructions for execution by the processor, the instructions, when executed, configuring the portable electronic device to: Presenting a story creation interface on the display for creating a record associated with a story including one or more scenes, wherein the story creation interface includes a keyword module, a virtual object module, and an IoT product module, and wherein the keyword module further configures the portable electronic device to: presenting a question and answer buttons on said display; receiving spoken words via said microphone; identifying keywords based on the spoken words; presenting a transcription of the identified keyword on the display; storing the keyword in association with the recording in the memory, wherein detecting the sound of the keyword during playback triggers the start of a subsequent scene; commencing a subsequent recording associated with the subsequent scene; and The record and the subsequent record are stored in the memory, wherein the records together include instructions associated with the story.

9. The portable electronic device according to claim 8, wherein: To initiate the subsequent recording, the keyword module further configures the portable electronic device to: presenting on said display a scene identifier and said transcription of the identified keywords; presenting a new scene button on the display; and The keyword module associated with the subsequent record is started.

10. The portable electronic device according to claim 8, wherein: To receive the spoken word, the keyword module further configures the portable electronic device to: presenting on said display a graphical response to said spoken word; and A next button is presented on the display, the button being configured to receive a selection accepting the keyword and executing to start the subsequent recording.

11. The portable electronic device according to claim 8, wherein: The instructions, when executed, further configure the portable electronic device to: presenting a play button on the display; and The story is played back based on the stored recording.

12. The portable electronic device of claim 8, wherein the story creation interface further comprises a virtual object module, the virtual object module further configuring the portable electronic device to: presenting a virtual object icon on the display; presenting instructions on the display; receiving a spoken name via the microphone; identifying a virtual object name based on the spoken name; retrieving a virtual object from the memory based on the virtual object name; presenting the virtual object on the display at the virtual object's location relative to a physical environment; and The virtual object name, the virtual object, and the object position are stored in the memory in association with the record.

13. The portable electronic device according to claim 12, wherein: To present the virtual object, the virtual object module further configures the portable electronic device to: presenting candidate virtual objects for consideration; and A transcription of the virtual object name is presented at a default location relative to the virtual object location.

14. The portable electronic device of claim 8, wherein the story creation interface further comprises an IoT product module, wherein the IoT product module configures the portable electronic device to: capturing frames of video data with the camera; detecting the IoT product at its location in the physical environment using the captured video data frames; Using the captured video data frames to determine a location of the portable electronic device relative to the location of the IoT product; When the device location is detected within a threshold proximity of an IoT product, presenting an IoT interface; detecting an interaction between the device and the IoT interface; and The IoT product, the IoT product location, and the detected interaction are stored in the memory in association with the record.

15. A non-transitory computer-readable medium storing instructions for generating an interactive AR experience using a portable electronic device including a microphone, a camera, a display, and a memory, the instructions, when executed by a processor, configure the portable electronic device to perform functions, the functions comprising: Presenting a story creation interface on the display for creating a record associated with a story including one or more scenes, wherein the story creation interface includes a keyword module, a virtual object module, and an IoT product module, and wherein the keyword module includes: presenting a question and answer buttons on said display; receiving spoken words via said microphone; identifying keywords based on the spoken words; presenting a transcription of the identified keyword on the display; storing the keyword in association with the recording in the memory, wherein detecting the sound of the keyword during playback triggers the start of a subsequent scene; commencing a subsequent recording associated with the subsequent scene; and The record and the subsequent record are stored in the memory, wherein the records together include instructions associated with the story.

16. The non-transitory computer readable medium of claim 15, wherein: The function of starting the subsequent recording includes the following functions: presenting on the display a transcription of the scene identifier and the identified keyword; presenting a new scene button on the display; and The keyword module associated with the subsequent record is started.

17. The non-transitory computer readable medium of claim 15, wherein: The functionality of receiving the spoken word includes the following functions: presenting on said display a graphical response to said spoken word; and A next button is presented on the display, the next button being configured to receive a selection to accept the keyword and to execute a function to start the subsequent recording.

18. The non-transitory computer readable medium of claim 15, wherein: When the instructions are executed by the processor, the portable electronic device is further configured to perform additional functions: presenting a play button on the display; and The story is played back based on the stored recording.

19. The non-transitory computer-readable medium of claim 15, wherein the story creation interface further comprises a virtual object module, the virtual object module comprising instructions for configuring the portable electronic device to perform additional functions, the additional functions comprising: presenting a virtual object icon on the display; presenting instructions on the display; receiving a spoken name via the microphone; identifying a virtual object name based on the spoken name; retrieving a virtual object from the memory based on the virtual object name; presenting the virtual object on the display at the virtual object's location relative to a physical environment; and The virtual object name, the virtual object, and the object position are stored in the memory in association with the record.

20. The non-transitory computer readable medium of claim 19, wherein: The function of presenting the virtual object includes the following functions: presenting candidate virtual objects for consideration; and A transcription of the virtual object name is presented at a default location relative to the virtual object location.

Images