Generating time shifts for collaborative targets

Abstract

A collaboration session (e.g., a virtual time capsule) in which access to a collaboration target and added virtual content is selectively provided to a participant / user. In one example of a collaborative session, a participant (host) creates a new session and invites the participant to join. Session creators (i.e., hosts) and other approved participants may access the content of the session (e.g., they may be recorded using applications such as shot cloud features; lens cloud features are available from the Snap Company of the Sanmeniaca California). A timestamp is associated with each received virtual content, and the root provides the user with a time shift of the collaboration target according to the timestamp.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Serial No. 17 / 900,840, filed on August 31, 2022, the contents of which are incorporated herein by reference in their entirety. Technical Field

[0003] Examples set forth in this disclosure relate to the field of virtual reality for electronic devices, including mobile devices and wearable devices, such as goggle devices. More specifically, but not by way of limitation, this disclosure describes a collaborative method with selective access. Background Art

[0004] Many types of computers and electronic devices available today, such as mobile devices (e.g., smartphones, tablets, and laptops), handheld devices, and wearable devices (e.g., smart glasses, digital goggles, headgear, head caps, and head-mounted displays), include various cameras, sensors, wireless transceivers, input systems, and displays.

[0005] A graphical user interface allows a user to interact with displayed content, including virtual objects and graphical elements such as icons, task bars, list boxes, menus, buttons, and selection control elements like cursors, pointers, handles, and slides.

[0006] Virtual reality (VR) technology generates a complete virtual environment including real images, which is sometimes presented on a VR headset or other head-mounted display. The VR experience allows the user to move through the virtual environment and interact with virtual objects. Augmented reality (AR) is a type of VR technology that combines real objects in the physical environment with virtual objects and displays the combination to the user. The display of the combination gives the impression that the virtual objects are truly present in the environment, especially when the virtual objects look and behave like real objects. Cross-reality (XR) is generally understood to refer to a general term for systems that include or combine elements from AR, VR, and MR (mixed reality) environments.

[0007] Collaboration tools are available for users of VR technology. Collaboration tools enable users to meet virtually in a collaborative session. During a collaborative session, users can communicate with each other in a virtual environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The features of the various examples described will be readily understood from the following detailed description with reference to the accompanying drawings. Reference numerals are used with each element in this specification and in the multiple views of the accompanying drawings. When multiple similar elements are present, a single reference numeral may be assigned to the similar elements, with an added lower case letter indicating a specific element.

[0009] Unless otherwise indicated, the various elements shown in the drawings are not drawn to scale. For clarity, the sizes of the various elements may be exaggerated or reduced. Several drawings depict one or more embodiments and are presented by way of example only and should not be construed as limiting. The drawings include the following figures:

[0010] Figure 1A is a side view of an example hardware configuration of an eyewear device suitable for use in an example collaborative system (right);

[0011] Figure 1B yes Figure 1A a perspective partial cross-sectional view of a right corner of the goggle apparatus, depicting a right visible light camera and a circuit board;

[0012] Figure 1C yes Figure 1A A side view (left) of an example hardware configuration of a goggle device showing a left visible light camera;

[0013] Figure 1D yes Figure 1C a perspective partial cross-sectional view of a left corner of the goggle apparatus, depicting a left visible light camera and a circuit board;

[0014] Figure 2A and Figure 2B is a rear view of an example hardware configuration of an eyewear device for use in an example collaborative system;

[0015] Figure 3 is a pictorial illustration of a three-dimensional scene, a left original image captured by a left visible light camera, and a right original image captured by a right visible light camera;

[0016] Figure 4 is a functional block diagram of an example collaborative system including a wearable device (e.g., a goggle device) and a server system connected via various networks;

[0017] Figure 5 Is suitable for Figure 4 pictorial representation of an example hardware configuration for a mobile device used in an example system of;

[0018] Figure 6 is a schematic diagram of a user in an example environment for use in describing simultaneous localization and mapping;

[0019] Figure 7 is a perspective view of an example collaborative target in the form of a box that can be manipulated by hand;

[0020] Figure 8 is used to open Figure 7 A perspective view of a first example hand gesture associated with the opening gesture of the box depicted in FIG.

[0021] Fig. 9 Is used for closing Figure 7 a perspective view of a second example hand gesture associated with the closing gesture of the box depicted in FIG.

[0022] Fig.10 is a flowchart listing the steps in an example collaborative method;

[0023] Fig.11 is a flow chart listing the steps of an example selective collaborative target access method; and

[0024] Fig.12 is a flow chart listing the steps of associating a time stamp with each received virtual content and providing a time shift of a collaborative goal. DETAILED DESCRIPTION

[0025] A collaborative session (e.g., a virtual time capsule) with access to collaborative goals and added virtual content is selectively provided to participants / users. In one example of a collaborative session, a participant (host) creates a new session and invites participants to join. The session creator (i.e., host) and other approved participants can access the content of the session (e.g., which can be recorded using an application such as the Lens Cloud feature; the Lens Cloud feature is commercially available from Snap, Inc. of Santa Monica, California). A timestamp is associated with each received virtual content, and a time shift of the collaborative goal according to the timestamp is provided to the user.

[0026] The following detailed description includes systems, methods, techniques, instruction sequences, and computing machine program products that illustrate the examples set forth in this disclosure. In order to provide a thorough understanding of the disclosed subject matter and its related teachings, many details and examples are included. However, those skilled in the relevant art can understand how to apply the related teachings without such details. The various aspects of the disclosed subject matter are not limited to the specific devices, systems, and methods described, because the related teachings can be applied or practiced in a variety of ways. The terms and nomenclature used in this article are only used for the purpose of describing specific aspects and are not intended to be limiting. In general, well-known instruction examples, protocols, structures, and techniques are not necessarily shown in detail.

[0027] As used herein, the term "coupled" or "connected" refers to any logical, optical, physical, or electrical connection, including links through which electrical or magnetic signals generated or provided by one system element are imparted to another coupled or connected system element, etc. Unless otherwise described, coupled or connected elements or devices are not necessarily directly connected to each other and may be separated by intermediate components, elements, or communication media, one or more of which can modify, manipulate, or carry electrical signals. The term "on" means supported directly by an element or indirectly supported by an element through another element integrated into or supported by the element.

[0028] The term "proximal" is used to describe an object or part of an object that is near, adjacent to, or close to an object or person; or an object or part of an object that is closer relative to other parts of the object may be described as "distal." For example, the end of the object closest to the object may be called the proximal end, and the generally opposite end may be called the distal end.

[0029] The orientation of the goggle device, other mobile devices, associated components, and any other device incorporating a camera, an inertial measurement unit, or both (such as shown in any of the figures in the accompanying drawings) is given by way of example only for purposes of illustration and discussion. In operation, the goggle device may be oriented in any other direction suitable for the particular application of the goggle device; for example, upward, downward, sideways, or any other orientation. Moreover, within the scope used herein, any directional terms (such as front, back, inside, outside, toward, left, right, lateral, longitudinal, up, down, upper, lower, top, bottom, side, horizontal, vertical, and diagonal) are used only as examples and are not limited to the direction or orientation of any camera or inertial measurement unit as constructed or otherwise described herein.

[0030] Advanced AR techniques, such as computer vision and object tracking, can be used to produce perceptually rich and immersive experiences. Computer vision algorithms extract three-dimensional data about the physical world from data captured in digital images or videos. Object recognition and tracking algorithms are used to detect objects in digital images or videos, estimate their orientation or pose, and track their movement over time. Real-time hand and finger recognition and tracking is one of the most challenging and processing-intensive tasks in computer vision.

[0031] The term "pose" refers to the static position and orientation of an object at a particular moment. The term "gesture" refers to the active movement of an object (such as a hand) through a series of postures, sometimes used to convey a signal or idea. The terms, pose and gesture, are sometimes used interchangeably in the fields of computer vision and augmented reality. As used herein, the terms "pose" or "gesture" (or variations thereof) are intended to include both poses and gestures; in other words, the use of one term does not exclude the other.

[0032] Additional objects, advantages and novel features of the examples will be set forth in part in the following description and will become apparent to those skilled in the art upon examination of the following and accompanying drawings, or may be learned by the making or operation of the examples. The objects and advantages of the subject matter may be realized and obtained by the methods, means and combinations particularly pointed out in the appended claims.

[0033] Reference is now made in detail to the examples illustrated in the accompanying drawings and discussed below.

[0034] Figure 1A is a side view (right) of an example hardware configuration of the goggle device 100 including a touch-sensitive input device or touchpad 181, and Figure 1C is a side view (left) of an example hardware configuration of the goggle device 100 including a touch-sensitive input device or touchpad 181. As shown, the touchpad 181 may have a subtle and not easily visible border; alternatively, the border may be clearly visible or include a raised or otherwise tactile edge that provides feedback to the user regarding the position and border of the touchpad 181. In other embodiments, the goggle device 100 may include a touchpad on the left side.

[0035] The surface of the touch pad 181 is configured to detect finger touches, taps, and gestures (e.g., moving touch) for use with a GUI displayed by the goggle device on an image display to allow the user to navigate and select menu options in an intuitive manner, which enhances and simplifies the user experience.

[0036] Detecting finger input on the touch pad 181 can enable several functions. For example, touching anywhere on the touch pad 181 can cause the GUI to display or highlight an item on the image display, which can be projected onto at least one of the optical components 180A, 180B. A double tap on the touch pad 181 can select an item or icon. Sliding or tapping a finger in a particular direction (e.g., front to back, back to front, up to down, or down) can cause an item or icon to slide or scroll in a particular direction; for example, to move to the next item, icon, video, image, page, or slide. Sliding a finger in another direction can slide or scroll in the opposite direction; for example, to move to the previous item, icon, video, image, page, or slide. The touch pad 181 can be virtually anywhere on the goggle device 100.

[0037] In one example, a recognized finger gesture of a single tap on touch pad 181 initiates selection or pressing of a graphical user interface element in an image presented on the image display of optical assembly 180A, 180B. Adjustment of the image presented on the image display of optical assembly 180A, 180B based on the recognized finger gesture may be the primary action of selecting or submitting a graphical user interface element on the image display of optical assembly 180A, 180B for further display or execution.

[0038] As shown, the goggle device 100 includes a left visible light camera 114A and a right visible light camera 114B. As further described herein, the two cameras 114A, 114B capture image information of the scene from two separate perspectives. The two captured images can be used to project a three-dimensional display onto an image display for viewing with 3D glasses.

[0039] The eyewear device 100 includes a right optical assembly 180B having an image display for presenting an image, such as a depth image. Figure 1A and Figure 1C As shown, the goggle device 100 may include a plurality of visible light cameras 114A, 114B, wherein the plurality of visible light cameras 114A, 114B form a passive type three-dimensional camera, such as a stereo camera, wherein the right visible light camera 114B is located at the right corner 110B, and as shown in FIG. Figure 1C to Figure 1D As shown, left visible light camera 114A is located at left corner 110A.

[0040] Left visible light camera 114A and right visible light camera 114B are sensitive to wavelengths in the visible light range. Each of visible light cameras 114A, 114B has a different forward-facing field of view that overlaps to enable the generation of a three-dimensional depth image, e.g., left visible light camera 114A captures left field of view 111A, and right visible light camera 114B captures right field of view 111B. In general, a "field of view" is the portion of a scene visible through a camera at a particular location and orientation in space. Field of view 111A and field of view 111B have overlapping fields of view 304 ( Figure 3 ). Objects or object features outside of field of view 111A, 111B are not recorded in the original image (e.g., a photograph or picture) when the visible light camera captures the image. The field of view describes the angular range or degree to which the image sensor of visible light camera 114A, 114B picks up electromagnetic radiation of a given scene in the captured image of the given scene. The field of view can be expressed as the angular size of the viewing cone (i.e., the viewing angle). The viewing angle can be measured horizontally, vertically, or diagonally.

[0041] In an example configuration, one or both of visible light cameras 114A, 114B has a field of view of 100° and a resolution of 480×480 pixels. “Angle of coverage” describes the angle of coverage of visible light camera 114A, 114B or infrared camera 410 (see FIG. Figure 2A ) is the range of angles that a lens can effectively image. Typically, a camera lens produces an image circle that is large enough to completely cover the film or sensor of the camera, possibly including some vignetting (e.g., the image gets darker toward the edges when compared to the center). If the coverage angle of a camera lens does not fill the sensor, the image circle will be visible, typically with strong vignetting toward the edges, and the effective viewing angle will be limited to the coverage angle.

[0042] Examples of such visible light cameras 114A, 114B include digital camera elements such as high-resolution complementary metal oxide semiconductor (CMOS) image sensors and digital VGA cameras (video graphics arrays) capable of resolutions of 640p (e.g., 640×480 pixels for a total of 300,000 pixels), 720p, or 1080p. Other examples include visible light cameras 114A, 114B that can capture high-definition (HD) video at high frame rates (e.g., 30-60 frames per second or greater) and store recordings at a resolution of 1216×1216 pixels (or greater).

[0043] The goggle device 100 can capture image sensor data from the visible light cameras 114A, 114B and geographic location data digitized by the image processor for storage in a memory. The visible light cameras 114A, 114B capture corresponding left and right original images in a two-dimensional spatial domain, and the left and right original images include a pixel matrix on a two-dimensional coordinate system, and the two-dimensional coordinate system includes an X-axis for horizontal position and a Y-axis for vertical position. Each pixel includes a color attribute value (e.g., a red pixel light value, a green pixel light value, or a blue pixel light value) and a position attribute (e.g., an X-axis coordinate and a Y-axis coordinate).

[0044] In order to capture stereoscopic images for later display as a three-dimensional projection, the image processor 412 ( Figure 4 ) may be coupled to visible light cameras 114A, 114B to receive and store visual image information. Image processor 412 or another processor controls the operation of visible light cameras 114A, 114B to act as a stereo camera that simulates human binocular vision, and may add a timestamp to each image. The timestamps on each pair of images allow the images to be displayed together as part of a three-dimensional projection. The three-dimensional projection produces an immersive, realistic experience that is desired in a variety of contexts including virtual reality (VR) and video games.

[0045] Figure 1Byes Figure 1A 10B of the goggle apparatus 100, depicting the right visible light camera 114B and circuit board of the camera system. Figure 1C yes Figure 1A 1 is a side view (left) of an example hardware configuration of the eyewear apparatus 100, showing the left visible light camera 114A of the camera system. Figure 1D yes Figure 1C A perspective cross-sectional view of the left corner 110A of the goggle apparatus, depicting the left visible light camera 114A and circuit board of the 3D camera.

[0046] like Figure 1B As shown in the example of , the goggle device 100 includes a right visible light camera 114B and a circuit board 140B, which can be a flexible printed circuit board (PCB). The right hinge 126B connects the right corner 110B to the right temple 125B of the goggle device 100. In some examples, the right visible light camera 114B, the flexible PCB 140B, or other electrical connectors or contacts can be located on the right temple 125B or the right hinge 126B.

[0047] The construction and placement of the left visible light camera 114A is substantially similar to the right visible light camera 114B, except that the connection and coupling is located on the left outer side portion 170A. The left hinge 126B connects the left corner 110A to the left temple 125A of the eyewear device 100. In some examples, components of the left visible light camera 114A, the flexible PCB 140A, or other electrical connectors or contacts can be located on the left temple 125A or the left hinge 126A.

[0048] The left corner 110A and the right corner 110B each include a corner body 190 and a corner cap. Figure 1B and Figure 1D Disposed within the left and right corners 110A, 110B are various interconnected circuit boards 140A, 140B, such as PCBs or flexible PCBs, which include controller circuits for the left and right visible light cameras 114A, 114B, microphones, low-power wireless circuits (e.g., for communicating via Bluetooth TM The corners 110A, 110B may be integrated into the frame 105 on the respective outer sides 170A, 170B (as shown) or implemented as separate components attached to the frame 105 on the respective sides 170A, 170B. Alternatively, the corners 110A, 110B may be integrated into the temples 125A, 125B attached to the frame 105.

[0049] The left visible light camera 114A and the right visible light camera 114B are connected to or disposed on the respective flexible PCBs 140A, 140B and are covered by visible light camera lens covers that are aimed through (multiple) openings formed in the frame 105. For example, the left frame 107A and the right frame 107B of the frame 105 are connected to the left corner 110A and the right corner 110B and include openings for the visible light camera lens covers. The frame 105 includes a front side configured to face outward and away from the user's eyes. The opening for the visible light camera lens cover is formed on and through the front side or outward-facing side of the frame 105. In the example, left visible light camera 114A and right visible light camera 114B each have respective outward-facing fields of view 111A, 111B having lines of sight or viewing angles associated with respective left and right eyes of a user of eyewear device 100. A visible light camera lens cover may also be adhered to the front side or outward-facing surface of right corner 110B, wherein the opening is formed with an outward-facing coverage angle, but in a different outward direction. The coupling may also be indirect via an intermediate component.

[0050] Figure 2A and Figure 2B is a perspective view from behind of an example hardware configuration of the eyewear apparatus 100 including two different types of image displays. The eyewear apparatus 100 is sized and shaped in a form configured for being worn by a user; in the example shown is the form of eyewear. The eyewear apparatus 100 may take other forms and may be combined with other types of frames; for example, a head cap, a headset, or a helmet.

[0051] In the goggle example, the goggle device 100 includes a frame 105 including a left frame 107A connected to a right frame 107B via a bridge 106, which is adapted to be supported by the user's nose. The left frame 107A and the right frame 107B include corresponding apertures 175A, 175B, which hold corresponding optical elements 180A, 180B, such as lenses and display devices. As used herein, the term "lens" is intended to include a transparent or translucent piece of glass or plastic with a curved or flat surface that causes light to converge / diverge, or causes little or no convergence / divergence.

[0052] Although shown as having two optical elements 180A, 180B, the eyewear device 100 may include other arrangements, such as a single optical element (or it may not include any optical elements 180A, 180B), depending on the application or intended user of the eyewear device 100. As further shown, the eyewear device 100 includes a left corner 110A adjacent to the left outer side 170A of the frame 105 and a right corner 110B adjacent to the right outer side 170B of the frame 105.

[0053] In one example, the image display of the optical components 180A, 180B includes an integrated image display. Figure 2A As shown, each optical assembly 180A, 180B includes a suitable display matrix 177, such as a liquid crystal display (LCD), an organic light emitting diode (OLED) display, or any other such display. Each optical assembly 180A, 180B also includes one or more optical layers 176, which may include lenses, optical coatings, prisms, reflectors, waveguides, optical strips, and other optical components in any combination. Optical layers 176A, 176B, ... 176N (in Figure 2A The optical layer 176A-N (shown herein as 176A-N) may include a prism having a suitable size and configuration and including a first surface for receiving light from the display matrix and a second surface for emitting light toward the user's eye. The prism of the optical layer 176A-N extends over all or at least a portion of the corresponding apertures 175A, 175B formed in the left frame 107A and the right frame 107B to allow the user to see the second surface of the prism when the user's eye is looking through the corresponding left frame 107A and the right frame 107B. The first surface of the prism of the optical layer 176A-N faces upward from the frame 105, and the display matrix 177 overlies the prism so that photons and light emitted by the display matrix 177 strike the first surface. The size and shape of the prism are determined so that the light beam is refracted within the prism and directed toward the user's eye by the second surface of the prism of the optical layer 176A-N. In this regard, the second surface of the prism of the optical layer 176A-N can be convex so as to direct light toward the center of the eye. The prisms may optionally be sized and shaped to magnify the image projected by the display matrix 177, and the light beams travel through the prisms so that the image viewed from the second surface is larger than the image emitted from the display matrix 177 in one or more dimensions.

[0054] In one example, the optical layers 176A-N may include an LCD layer that is transparent (keeps the lenses open) unless and until a voltage is applied that renders the layer opaque (closes or blocks the lenses). Figure 4) can be programmed to apply voltage to the LCD layer to create an active shutter system, thereby making the goggle device 100 suitable for viewing visual content when displayed as a three-dimensional projection. Technologies other than LCD can be used for active shutter mode, including other types of reactive layers that respond to voltage or another type of input.

[0055] In another example, the image display device of the optical assembly 180A, 180B includes Figure 2B Each optical assembly 180A, 180B includes a laser projector 150, which is a three-color laser projector using a scanning mirror or a galvanometer. During operation, a light source (such as a laser projector 150) is disposed in or on one of the temples 125A, 125B of the eyewear device 100. The optical assembly 180B in this example includes one or more optical strips 155A, 155B, ... 155N (such as Figure 2B 155A-N), these optical bands 155A, 155B, ... 155N are spaced apart and span the width of the lens of each optical component 180A, 180B or span the depth between the front and back surfaces of the lens.

[0056] As the photons projected by the laser projector 150 travel through the lenses of each optical assembly 180A, 180B, the photons encounter the optical bands 155A-N. When a particular photon encounters a particular optical band, the photon is either redirected toward the user's eye, or it passes to a lower optical band. The combination of the modulation of the laser projector 150 and the modulation of the optical bands can control specific photons or light beams. In an example, the processor controls the optical bands 155A-N by initiating mechanical, acoustic, or electromagnetic signals. Although shown as having two optical assemblies 180A, 180B, the eyewear device 100 can include other arrangements, such as a single or three optical assemblies, or each optical assembly 180A, 180B can be arranged in a different arrangement depending on the application or intended user of the eyewear device 100.

[0057] In another example, Figure 2B The goggle device 100 shown in the figure may include two projectors: a left projector (not shown) and a right projector 150. The left optical assembly 180A may include a left display matrix 177 or a set of left optical strips (not shown) configured to interact with the light from the left projector. Similarly, the right optical assembly 180B may include a right display matrix (not shown) or a set of right optical strips 155A, 155B, ... 155N configured to interact with the light from the right projector 150. In this example, the goggle device 100 includes a left display and a right display.

[0058] Figure 3306, a left raw image 302A captured by a left visible light camera 114A, and a right raw image 302B captured by a right visible light camera 114B. The left field of view 111A may overlap with the right field of view 111B as shown. The overlapping field of view 304 represents a portion of the image captured by the two cameras 114A, 114B. When referring to the field of view, the term "overlap" refers to the overlap of the pixel matrices in the generated raw images by thirty percent (30%) or more. "Approximately overlap" means that the pixel matrices in the generated raw images or infrared images of the scene overlap by fifty percent (50%) or more. As described herein, the two raw images 302A, 302B can be processed to include a timestamp that allows the images to be displayed together as part of a three-dimensional projection.

[0059] To capture stereo images, Figure 3 As shown, a pair of raw red, green, and blue (RGB) images of a real scene 306 are captured at a given moment - a left raw image 302A captured by the left camera 114A and a right raw image 302B captured by the right camera 114B. When the pair of raw images 302A, 302B are processed (e.g., by the image processor 412), a depth image is generated. The generated depth image can be viewed on the optical components 180A, 180B of the goggle device, on another display (e.g., the image display 580 on the mobile device 401), or on a screen.

[0060] The generated depth image is in a three-dimensional spatial domain and may include a matrix of vertices on a three-dimensional position coordinate system including an X-axis for horizontal position (e.g., length), a Y-axis for vertical position (e.g., height), and a Z-axis for depth (e.g., distance). Each vertex may include a color attribute (e.g., a red pixel light value, a green pixel light value, or a blue pixel light value); a position attribute (e.g., an X position coordinate, a Y position coordinate, and a Z position coordinate); a texture attribute; a reflectivity attribute; or a combination thereof. The texture attribute quantifies the perceived texture of the depth image, such as the spatial arrangement of color or intensity in a vertex region of the depth image.

[0061] Figure 44 is a functional block diagram of an example collaborative system 400, which includes a wearable device (e.g., goggle device 100), a mobile device 401, and a server system 498 connected via various networks 495 (such as the Internet). The server system 498 can be one or more computing devices as part of a service or network computing system, and the computing device includes, for example, a processor, a memory, and a network communication interface to communicate with the goggle device 100 and the mobile device 401 through the network 495. The server system 498 includes a server processor 499, which can be configured to host a collaborative session. The functions of the goggle device 100 or the mobile device 401 described herein (such as collaborative processing and serving collaborative goals to users) can be performed by the processor 499 of the server system 498.

[0062] The goggle device 100 includes one or more visible light cameras 114A, 114B that capture static images, video images, or both static images and video images, as described herein. Cameras 114A, 114B may have direct memory access (DMA) to high-speed circuits 430 and act as stereo cameras. Cameras 114A, 114B may be used to capture an initial depth image that can be reproduced as a three-dimensional (3D) model, which is an image of a texture map of a red, green, and blue (RGB) imaging scene. The device 100 may also include a depth sensor 213 that uses infrared signals to estimate the position of a target relative to the device 100. In some examples, the depth sensor includes one or more infrared emitters 215 and an infrared camera 410.

[0063] The eyewear device 100 further includes two image displays (one associated with the left side portion 170A and one associated with the right side portion 170B) for each optical assembly 180A, 180B. The eyewear device 100 also includes an image display driver 442, an image processor 412, a low power circuit 420, and a high speed circuit 430. The image display of each optical assembly 180A, 180B is used to present an image including a static image, a video image, or a static image and a video image. The image display driver 442 is coupled to the image display of each optical assembly 180A, 180B to control the display of the image.

[0064] The eyewear device 100 additionally includes one or more microphones (not shown) and one or more speakers 413 (e.g., one associated with the left side portion of the eyewear device and another associated with the right side portion of the eyewear device). The speakers 413 may be incorporated into the frame 105, the temple 125, or the corner 110 of the eyewear device 100. The one or more speakers 413 are driven by an audio processor 414 and an audio driver 415 under the control of the low power circuit 420, the high speed circuit 430, or both. The speakers 413 are used to present audio signals, including, for example, a beat track. The audio processor 414 is coupled to the microphone and the speakers 413 to control the corresponding capture and presentation of the sound.

[0065] Figure 4 The components for the eyewear device 100 shown in the figure are located on one or more circuit boards (e.g., printed circuit boards (PCBs) or flexible printed circuits (FPCs)) in the frame or temples. Alternatively or additionally, the depicted components can be located in the corners, frames, hinges, or bridges of the eyewear device 100.

[0066] like Figure 4 As shown, the high-speed circuit 430 includes a high-speed processor 432, a memory 434, and a high-speed wireless circuit 436. In this example, an image display driver 442 is coupled to the high-speed circuit 430 and is operated by the high-speed processor 432 to drive the left image display and the right image display of each optical assembly 180A, 180B. The high-speed processor 432 can be any processor capable of managing high-speed communications and operations of any general-purpose computing system required for the eyewear device 100. The high-speed processor 432 includes the processing resources required to manage high-speed data transmission to a wireless local area network (WLAN) on a high-speed wireless connection 437 using a high-speed wireless circuit 436.

[0067] In some examples, the high-speed processor 432 executes an operating system, such as a LINUX operating system or other such operating system of the goggle device 100, and the operating system is stored in the memory 434 for execution. In addition to any other duties, the high-speed processor 432 also executes a software architecture for the goggle device 100, which is used to manage data transmission through the high-speed wireless circuit 436. In some examples, the high-speed wireless circuit 436 is configured to implement the Institute of Electrical and Electronics Engineers (IEEE) 802.11 communication standard, also referred to herein as Wi-Fi. In other examples, other high-speed communication standards can be implemented by the high-speed wireless circuit 436.

[0068] The low power circuit 420 includes a low power processor 422 and a low power wireless circuit 424. The low power wireless circuit 424 and the high speed wireless circuit 436 of the goggle device 100 may include a short range transceiver (Bluetooth TM or Bluetooth Low Energy (BLE)) and a wireless wide area network, local area network, or wide area network transceiver (e.g., cellular or Wi-Fi). A mobile device 401 including a transceiver for communicating via a low power wireless connection 425 and a high speed wireless connection 437 can be implemented using details of the architecture of the goggle device 100, as can other elements of the network 495.

[0069] The memory 434 includes any storage device capable of storing various data and applications, including, among other things, camera data generated by the left visible light camera 114A and the right visible light camera 114B, the infrared camera(s) 410, the image processor 412, and the images generated by the image display driver 442 for display on the image display of each optical assembly 180A, 180B. Although the memory 434 is shown as being integrated with the high-speed circuit 430, in other examples the memory 434 can be a separate, individual element of the goggle device 100. In some such examples, electrical routing lines can provide a connection from the image processor 412 or the low-power processor 422 to the memory 434 through a chip including the high-speed processor 432. In other examples, the high-speed processor 432 can manage the addressing of the memory 434 so that the low-power processor 422 will start the high-speed processor 432 at any time when a read or write operation involving the memory 434 is required.

[0070] like Figure 4 As shown, the high-speed processor 432 of the goggle device 100 can be connected to the camera system (visible light cameras 114A, 114B), the image display driver 442, the user input device 491 and the memory 434.

[0071] The output components of the eyewear device 100 include visual elements, such as Figure 2A and Figure 2BThe left image display and the right image display associated with each lens or optical component 180A, 180B described in the embodiment of the present invention are (e.g., a display such as a liquid crystal display (LCD), a plasma display panel (PDP), a light emitting diode (LED) display, a projector, or a waveguide). The goggle device 100 may include an indicator facing the user (e.g., an LED, a speaker 413, or a vibration actuator), or a signal facing outward (e.g., an LED, a speaker 413). The image display of each optical component 180A, 180B is driven by an image display driver 442. In some example configurations, the output components of the goggle device 100 also include additional indicators, such as audible elements (e.g., speakers 413), tactile components (e.g., actuators such as vibration motors to generate tactile feedback), and other signal generators. For example, the device 100 may include a set of indicators facing the user and a set of signals facing outward. The set of indicators facing the user is configured to be seen or otherwise sensed by the user of the device 100. For example, the device 100 may include an LED display positioned so that it can be seen by a user, one or more speakers 413 positioned to generate sounds that can be heard by the user, or an actuator that provides tactile feedback that can be felt by the user. The outward-facing signal group is configured to be seen or otherwise sensed by an observer near the device 100. Similarly, the device 100 may include LEDs, speakers, or actuators that are configured and positioned to be sensed by an observer.

[0072] The input components of the goggle device 100 may include alphanumeric input components (e.g., a touch screen or touchpad configured to receive alphanumeric input, an optical keyboard, or other alphanumeric configured element), pointer-based input components (e.g., a mouse, touchpad, trackball, joystick, motion sensor, or other pointing instrument), tactile input components (e.g., a button switch, a touch screen or touchpad or other tactile configured element that senses the position, force, or position and force of a touch or touch gesture), visual input components (e.g., a camera 114 / 420), and audio input components (e.g., a microphone), etc. The mobile device 401 and the server system 498 may include alphanumeric, pointer-based, tactile, audio, and other input components.

[0073] In some examples, the goggle device 100 includes a collection of motion sensing components referred to as an inertial measurement unit 472. The motion sensing component may be a microelectromechanical system (MEMS) with microscopic moving parts, typically small enough to be part of a microchip. In some example configurations, the inertial measurement unit (IMU) 472 includes an accelerometer, a gyroscope, and a magnetometer. The accelerometer senses the linear acceleration (including the acceleration due to gravity) of the device 100 relative to three orthogonal axes (x, y, z). The gyroscope senses the angular velocity of the device 100 about three rotation axes (pitch, roll, yaw). The accelerometer and gyroscope can provide together position, orientation, and motion data about the device relative to six axes (x, y, z, pitch, roll, yaw). The magnetometer (if present) senses the direction of the device 100 relative to magnetic north. The position of the device 100 can be determined by a position sensor (such as a GPS unit), one or more transceivers for generating relative position coordinates, an altitude sensor or a barometer, and other orientation sensors. Such positioning system coordinates may also be received from the mobile device 401 via the low power wireless circuitry 424 or the high speed wireless circuitry 436 over the wireless connections 425 , 437 .

[0074] The IMU 472 may include or cooperate with a digital motion processor or program that collects raw data from the components and calculates a number of useful values ​​regarding the position, orientation, and motion of the device 100. For example, the acceleration data collected from the accelerometer may be integrated to obtain the velocity relative to each axis (x, y, z); and integrated again to obtain the position of the device 100 (in linear coordinates x, y, and z). The angular velocity data from the gyroscope may be integrated to obtain the position of the device 100 (in spherical coordinates). The programming for calculating these useful values ​​may be stored in the memory 434 and executed by the high-speed processor 432 of the goggle device 100.

[0075] The eyewear device 100 may optionally include additional peripheral sensors, such as biometric sensors, dedicated sensors, or display elements integrated with the eyewear device 100. For example, the peripheral device elements may include any I / O components, including output components, motion components, position components, or any other such components described herein. For example, the biometric sensor may include components for detecting expressions (e.g., hand expressions, facial expressions, vocal expressions, body postures, or eye tracking), for measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), or for identifying people (e.g., based on voice, retina, facial features, fingerprints, or electrical biosignals such as electroencephalogram data), etc.

[0076] The mobile device 401 may be a smartphone, tablet, laptop, access point, or any other such device capable of connecting to the goggle device 100 using both a low power wireless connection 425 and a high speed wireless connection 437. The mobile device 401 is connected to a server system 498 and a network 495. The network 495 may include any combination of wired and wireless connections.

[0077] like Figure 4 As shown, the illustrated collaborative system 400 includes a computing device, such as a mobile device 401, connected to the goggle device 100 via a network. The collaborative system 400 includes a memory for storing instructions and a processor for executing instructions. The execution of the instructions of the collaborative system 400 by the processor 432 configures the goggle device 100 to cooperate with the mobile device 401. The collaborative system 400 can utilize the memory 434 of the goggle device 100 or the memory elements 540A, 540B, 540C ( Figure 5 ). Moreover, the collaborative system 400 can utilize the processor elements 432, 422 of the goggle device 100 or the central processing unit (CPU) 540 ( Figure 5 ). In addition, the collaborative system 400 can further utilize the memory and processor elements of the server system 498. In this regard, the memory and processing functions of the collaborative system 400 can be shared or distributed across the processors and memories of the goggle device 100, the mobile device 401, and the server system 498 to implement the functions described herein.

[0078] In some example embodiments, the memory 434 includes or is coupled to a hand gesture library 480, as described herein. In some implementations, the process of detecting a hand shape or gesture involves comparing pixel-level data in one or more frames of video data captured by the goggles 100 or mobile device 401 with hand shapes and gestures stored in the library 480 until a good match is found. A gesture can be a static pose that can be detected in one or more frames of data, or a dynamic pose that is detected during the process of two or more frames of data.

[0079] In some example embodiments, the memory 434 further includes an element animation application 910, a positioning system 915, an image processing system 920, and a collaborative application 925. In the collaborative system 400 where the camera is capturing a frame of video data, the element animation application 910 configures the processor 432 to control the movement of a series of virtual things 700 on the display in response to detecting one or more inputs (e.g., IMU data, captured images, and hand shapes or gestures). The positioning system 915 configures the processor 432 to obtain positioning data for determining the position of the goggle device 100 relative to the physical environment. The positioning data can be derived from a series of images, an IMU unit 472, a GPS unit, or a combination thereof. The image processing system 920 configures the processor 432 to present the captured images on the display of the optical assembly 180A, 180B in coordination with the image display driver 442 and the image processor 412. The collaborative application 925 configures the processor 432 to implement the collaborative functions described herein.

[0080] Figure 5 is a high-level functional block diagram of an example mobile device 401. Mobile device 401 includes flash memory 540A that stores programming to be executed by CPU 540 to perform all or a subset of the functionality described herein.

[0081] The mobile device 401 may include a camera 570 including at least two visible light cameras (a first visible light camera and a second visible light camera having overlapping fields of view) or at least one visible light camera and a depth sensor having substantially overlapping fields of view. The mobile device 401 may additionally include a speaker 571. The flash memory 540A may further include a plurality of images or videos generated via the camera 570.

[0082] As shown in the figure, the mobile device 401 includes an image display 580, a mobile display driver 582 for controlling the image display 580, and a display controller 584. Figure 5 In the example of , image display 580 includes a user input layer 591 (eg, a touch screen) layered on top of or otherwise integrated into a screen used by image display 580 .

[0083] Examples of touch screen type mobile devices that may be used include, but are not limited to, smart phones, personal digital assistants (PDAs), tablet computers, laptop computers, or other portable devices. However, the structure and operation of touch screen type devices are provided by way of example; the subject technology as described herein is not intended to be limited thereto. For purposes of discussion, Figure 5Thus, a block diagram illustration of an example mobile device 401 having a user interface is provided, the user interface including a touch screen input layer 591 for receiving input (via touch, multi-touch or gestures, etc., by hand, stylus or other tool), a camera 570 for capturing images of targets (including the user's hand and potentially virtual content), and an image display 580 for displaying content.

[0084] like Figure 5 As shown, the mobile device 401 includes at least one digital transceiver (XCVR) 510 for digital wireless communication via a wide area wireless mobile communication network, shown as a WWAN XCVR. The mobile device 401 also includes additional digital or analog transceivers, such as a short-range transceiver (XCVR) 520 for short-range network communication, such as via NFC, VLC, DECT, ZigBee, Bluetooth TM For example, the short-range XCVR 520 may take the form of any available two-way wireless local area network (WLAN) transceiver of a type compatible with one or more standard protocols for communications implemented in wireless local area networks, such as one of the Wi-Fi standards under IEEE 802.11.

[0085] To generate location coordinates for positioning the mobile device 401, the mobile device 401 may include a global positioning system (GPS) receiver. Alternatively or additionally, the goggle device 100 or the mobile device 401 may utilize either or both of the short-range XCVR 520 and the WWAN XCVR 510 to generate location coordinates for positioning. For example, a cellular network, Wi-Fi, or Bluetooth TM The base positioning system can generate very accurate position coordinates, especially when used in combination. Such position coordinates can be transmitted between the goggle device 100 or the mobile device 401 via one or more network connections via XCVR 510, 520.

[0086] In some examples, the mobile device 401 includes a collection of motion sensing components for sensing the position, orientation, and motion of the client device 401, referred to as an inertial measurement unit (IMU) 572. The motion sensing components may be microelectromechanical systems (MEMS) with microscopic moving parts, typically small enough to be part of a microchip. In some example configurations, the inertial measurement unit (IMU) 572 includes an accelerometer, a gyroscope, and a magnetometer. The accelerometer senses the linear acceleration (including acceleration due to gravity) of the client device 401 relative to three orthogonal axes (x, y, z). The gyroscope senses the angular velocity of the client device 401 about three rotational axes (pitch, roll, yaw). Together, the accelerometer and gyroscope can provide position, orientation, and motion data about the device relative to six axes (x, y, z, pitch, roll, yaw). The magnetometer (if present) senses the direction of the client device 401 relative to magnetic north.

[0087] The IMU 572 may include or cooperate with a digital motion processor or program that collects raw data from the components and calculates a number of useful values ​​regarding the position, orientation, and motion of the client device 401. For example, the acceleration data collected from the accelerometer may be integrated to obtain the velocity with respect to each axis (x, y, z); and integrated again to obtain the position of the client device 401 (in linear coordinates x, y, and z). The angular velocity data from the gyroscope may be integrated to obtain the position of the client device 401 (in spherical coordinates). The programming for calculating these useful values ​​may be stored on one or more memory elements 540A, 540B, 540C and executed by the CPU 540 of the client device 401.

[0088] The transceivers 510, 520 (i.e., network communication interfaces) conform to one or more of the different digital wireless communication standards used by modern mobile networks. Examples of WWAN transceivers 510 include, but are not limited to, transceivers configured to operate according to Code Division Multiple Access (CDMA) and Third Generation Partnership Project (3GPP) network technologies, including, for example, but not limited to, 3GPP Type 2 (or 3GPP2) and LTE, sometimes referred to as "4G". For example, transceivers 510, 520 provide two-way wireless communication of information, including digitized audio signals, still images and video signals, web page information for display and network-related input, and different types of mobile messaging communications to / from mobile device 401.

[0089] The mobile device 401 further includes a microprocessor serving as a central processing unit (CPU); Figure 5540. A processor is a circuit having elements configured and arranged to perform one or more processing functions (typically different data processing functions). Although discrete logic components may be used, the example utilizes components forming a programmable CPU. A microprocessor, for example, includes one or more integrated circuit (IC) chips that combine electronic components to perform the functions of a CPU. For example, CPU 540 may be based on any known or available microprocessor architecture, such as a reduced instruction set computing (RISC) using an ARM architecture, as commonly used today in mobile devices and other portable electronic devices. Of course, other arrangements of processor circuits may be used to form CPU 540 or processor hardware in smart phones, laptop computers, and tablet computers.

[0090] The CPU 540 functions as a programmable host controller for the mobile device 401 by configuring the mobile device 401 to perform various operations, for example, according to instructions or programming executable by the CPU 540. Such operations may include, for example, various conventional operations of the mobile device, as well as operations related to the programming of applications on the mobile device. Although a processor may be configured using hardwired logic, a typical processor in a mobile device is a conventional processing circuit that is configured by the execution of programming.

[0091] The mobile device 401 includes a memory or storage system for storing programming and data. In this example, the memory system may include flash memory 540A, random access memory (RAM) 540B, and other memory components 540C as needed. RAM 540B is used as short-term storage of instructions and data processed by CPU 540, for example, as working data processing memory. Flash memory 540A generally provides long-term storage.

[0092] Thus, in the example of mobile device 401, flash memory 540A is used to store programs or instructions for execution by CPU 540. Depending on the type of device, mobile device 401 stores and runs a mobile operating system through which specific applications are executed. Examples of mobile operating systems include Google Android, Apple iOS (for iPhone or iPad devices), Windows Mobile, Amazon Fire OS, RIM BlackBerry OS, etc.

[0093] like Figure 5As shown, the CPU 540 of the mobile device 401 can be connected to the camera system 570, the mobile display driver 582, the user input layer 591, and the memory 540A. The various components and functions of the goggle device 100 described herein can be incorporated into the mobile device 401. Likewise, the components and functions of the mobile device 401 described herein can be incorporated into the goggle device 100.

[0094] The processor 432 within the goggle device 100 or the processor 540 within the mobile device 401 can build a map of the environment surrounding the corresponding device, determine the location of the device within the map environment, and determine the relative position of the device to one or more targets in the map environment. The processor 432 / 540 can use a conventional simultaneous localization and mapping (SLAM) algorithm applied to data received from one or more sensors to build a map and determine location and position information. The sensor data includes images received from one or both of the cameras 114A, 114B, or the camera 570, distances received from a laser rangefinder, location information received from a GPS unit, motion and acceleration data received from an IMU 472 / 572, or a combination of data from such sensors or from other sensors that provide data useful in determining location information.

[0095] In the context of augmented reality, SLAM algorithms are used to build and update a map of the environment while tracking and updating the position of the device (or user) within the mapped environment. The mathematical solution can be approximated using various statistical methods, such as particle filters, Kalman filters, extended Kalman filters, and covariance intersection. In a system including a high-definition (HD) camera that captures video at a high frame rate (e.g., thirty frames per second), the SLAM algorithm updates the map and the location of the target at least as frequently as the frame rate; in other words, the map and positioning are calculated and updated thirty times per second.

[0096] Figure 6 An example physical environment 600 and elements useful when using SLAM applications and other types of tracking applications (e.g., natural feature tracking (NFT)) are depicted. Although the following example is provided with reference to the goggle device 100, the example can be implemented in a similar manner in the mobile device 401. A user 602 of the goggle device 100 is present in the example physical environment 600 (in Figure 6The processor 432 of the goggle device 100 uses the captured images to determine its position relative to one or more targets 604 within the environment 600, uses the coordinate system (x, y, z) of the environment 600 to build a map of the environment 600, and determines its position within the coordinate system. In addition, the processor 432 determines the head posture (roll, pitch, and yaw) of the goggle device 100 within the environment by using two or more location points (e.g., three location points 606a, 606b, and 606c) associated with a single target 604a or by using one or more location points 606 associated with two or more targets 604a, 604b, 604c. The processor 432 of the goggle device 100 can place a virtual target 608 (such as Figure 6 ) are positioned within environment 600 for viewing during an augmented reality experience (such as a collaborative augmented reality experience), where each user has a corresponding augmented reality device (e.g., goggle device 100 or mobile device 401).

[0097] In some examples, the positioning system 915 associates the virtual marker 610a with the virtual target 608 in the environment 600. In augmented reality, the task of registering markers at locations in the environment to help the device track and update the locations of the user, the device, and the objects (virtual and physical) in the map environment. Sometimes markers are registered to high contrast physical targets (such as relatively dark targets, such as framed pictures 604a mounted on lighter walls) to help cameras and other sensors detect the markers. These markers can be pre-assigned or can be assigned by the goggle device 100 when entering the environment.

[0098] The marker may be encoded with information or otherwise linked to information. The marker may include location information, a physical code (such as a barcode or QR code; visible or hidden to the user), or a combination thereof. A set of data associated with the marker is stored in the memory 434 of the eyewear device 100. The set of data includes information about the marker 610a, the location (position and orientation) of the marker, one or more virtual targets, or a combination thereof. The marker location may include three-dimensional coordinates of one or more marker landmarks 616a, such as Figure 6The marker location can be expressed relative to real-world geographic coordinates, a marker coordinate system, the location of the goggle device 100, or other coordinate systems. The one or more virtual targets associated with the marker 610a may include any of a variety of materials (including static images, videos, audio, tactile feedback, executable applications, interactive user interfaces, and experiences) and combinations or arrangements of such materials. Any type of content that can be stored in memory and retrieved when the marker 610a is encountered or associated with an assigned marker may be classified as a virtual target in this context. For example, Figure 6 The key 608 shown in FIG. 6 is a virtual target displayed as a static image (2D or 3D) at the marked location.

[0099] In one example, the marker 610a can be registered in memory as being located at the physical target 604a (e.g., Figure 6 In another example, the mark can be registered in the memory as a specific position relative to the goggle device 100.

[0100] Figure 7 , Figure 8 and Fig. 9 is an illustration of an example collaborative goal 700 developed by adding virtual content 702 during a collaborative session for describing the following Fig.10 and Fig.11 700). Although a box is used for collaborative target 700 in many of the examples described herein, any virtual target may be selected to be used as collaborative target 700. In addition, although the illustration depicts the goggle device 100 as a physical remote device, it should be understood that the functionality described below may be implemented using other physical remote devices (e.g., mobile device 401).

[0101] Figure 7 A perspective view of an example collaborative target 700 in the form of a box in a first state (closed) is provided, which can be manipulated in three dimensions 701 based on a hand 651 (e.g., by a gesture detected in an image or touch input on a touch screen) for corresponding movements in three dimensions 681. For example, the hand 651 can be rotated to rotate the collaborative target. In some examples, an extended index finger can be detected near the collaborative target, and when the user makes a tap gesture, a corresponding audible signal will be presented via a speaker. The position of the goggle device can also be tracked in three dimensions 840 within the environment 600, so that the coloring generated for the presentation of the display 180B is more realistic. The hand 651 can be predefined as a left hand, as shown in the figure. In some embodiments, the system includes a process for selecting and setting the left and right hands that will serve as the hand 651 to be detected.

[0102] Figure 8 A perspective view of an example collaborative target 700 in a second state (open) is provided, wherein associated virtual content 702 (dial 702a, pot 702b, book 702c, other virtual content 702d-f) is added during the collaborative period. Hand 652 is shown in the open position. This position of hand 652 or hand 651 in the relaxed position ( Figure 7 ) to the transition of the hand 652 in the open position can be set to correspond to opening the collaborative target 700, so that when the hand position or hand gesture is detected, the collaborative target 700 transitions to the open state.

[0103] Fig. 9 A perspective view of an example collaborative target 700 in a first state (closed) is provided, wherein virtual content 703a-703c is added to the outer surface of the collaborative target 700. The hand 653 is illustrated in a closed position. This position of the hand 653 or the hand 651 in a relaxed position ( Figure 7 ) to the transition of the hand 653 in the closed position can be set to correspond to the closing of the collaborative target 700, so that when the hand position or hand gesture is detected, the collaborative target 700 transitions to the closed state.

[0104] The process of detecting and tracking includes detecting hands 651 / 652 / 653 in a set or series of captured video data frames over time, in various poses. In this context, detection refers to and includes detecting hands in as few as one frame of video data, as well as detecting hands in a subset or series of frames of video data over time. Thus, in some embodiments, the process includes detecting hands 651 in a particular pose in one or more of the captured frames of video data. In other embodiments, the process includes detecting hands 651 / 652 / 653 in different poses over time in a subset or series of captured video data frames.

[0105] Fig.10 1000 is a flowchart depicting an example method of developing a collaborative goal 700 during a collaborative session of a collaborative session including multiple physical remote devices (e.g., goggle device 100, mobile device 401, or a combination thereof). In the example, Fig.10 The steps are performed by a server system 498 (see Figure 4) is performed by a processor 499 of a physical remote device. In other examples, one or more steps may be performed by processors 432 and 540 of a physical remote device, or a combination of a processor of server system 498 and a physical remote device (acting as a processor for implementing the steps). One or more of the steps shown and described may be performed simultaneously, in series, in an order different from that shown and described, or in conjunction with additional steps. Some steps may be omitted, or repeated in some applications.

[0106] At block 1002, the processor receives user parameters for a collaborative session. In an example, the processor receives the user parameters from a physical remote device of a host user, wherein the host user specifies the user parameters through their physical remote device during a connection to the server system 498. The user parameters include identification codes of users that are allowed to access the collaborative session. The user parameters may also include access levels that identify what individuals may access during the collaborative session. Referring now to flow chart 1100 ( Fig.11 ) steps to describe additional details about setting and maintaining access levels.

[0107] At block 1004, the processor receives target parameters. In an example, the processor receives the target parameters from a physical remote device of a host user (or other user with an appropriate access level), wherein the user specifies the target parameters through their physical remote device during a connection to the server system 498. The target parameters include an identification code (e.g., such as Figure 7-Figure 9 Other target parameters may include the material of the target (e.g., cardboard, metal, glass) or a time parameter that provides a time window 710 or deadline 712 during which virtual content 702 may be added to collaborative target 700 (after which virtual content 702 may no longer be added to collaborative target 700).

[0108] The processor of the server system 498 may present to the physical remote device via the network 495 a list of available virtual targets 702 for the user to select through their device, which list, when selected, is received by the processor of the server system 498. Alternatively, the user may send a virtual target 702 (e.g., a 3D image) that they generate on their physical remote device to the server system 498, wherein the processor 499 of the server system 498, upon receipt, designates the received virtual target 702 as a collaborative target 700.

[0109] At box 1006, the processor provides access to the collaborative target 700. The processor provides access to the collaborative target 700 through a server connection with the physical remote device based on the access level associated with each device. In the example, the processor 499 develops the collaborative target 700 in response to the received target parameters and stores the collaborative target 700 in a location accessible to the physical remote device. The processor provides access to the collaborative target 700 based on the access level associated with the user of the physical remote device. When the user accesses the collaborative target 700, the processor sends a file including the collaborative target 700 to the mobile device 401, and the physical remote device uses the file to generate a rendering for presentation on a display of the physical remote device (such as the display 180A-180B or the display 580 of the goggle device 100). The user can then interact with the representation of the collaborative target 700 on their display (e.g., using a browser such as Figure 7 hand gestures shown).

[0110] At block 1008, the processor receives the design parameters. The processor receives the design parameters from a physical remote device that has an appropriate permission level to access the collaborative target 700. In an example, the processor sends a file to the physical remote device that includes the collaborative target 700. The physical remote device generates a rendering on a display that interacts with the user to add the design parameters to the collaborative target 700. The user may then, for example, select an image (e.g., from their camera, such as camera 114A-114B or camera 570) and select a surface of the collaborative target 700, where upon selecting the surface, the selected image is associated with the collaborative target 700 on the physical remote device. As the user makes changes on their device, the added / changed design parameters are transmitted by the physical remote device to the server system 498 via the network 498.

[0111] At block 1010, the processor updates the collaborative target 700 in response to the design parameters. The processor updates the collaborative target 700 in response to the changes received from the physical remote device via the network 495. In an example, upon receiving the added / changed design parameters from the physical remote device, the processor associates the added / changed design parameters with the collaborative target 700 in a location accessible to the physical remote device.

[0112] At block 1012, the processor receives virtual content 702. The processor receives virtual content 702 from a physical remote device having an appropriate permission level for accessing the collaborative target 700. In an example, the processor sends a file to the physical remote device that includes the collaborative target 700. The physical remote device generates a coloring on a display that interacts with the user to add virtual content 702 to the collaborative target 700. The user can then add visual virtual content 702, for example, by selecting an image (e.g., from its camera) and performing an action (e.g., dragging and dropping the image onto the collaborative target 700 or double-clicking the target) to associate the virtual content 702 with the collaborative target 700 on the physical remote device. In addition, audio virtual content can be added to the video virtual content by, for example, pressing and holding the video virtual content and speaking into the microphone, wherein the audio received while pressing the video virtual content is associated with the video virtual content. When the user makes changes on their device, the added virtual content 702 is transmitted to the server system 498 by the physical remote device via the network 495.

[0113] In an example, the user may associate the virtual content 702 with the collaborative target 700 by, for example, dragging and dropping the virtual content 702 onto the surface of the collaborative target 700. In one example, using the goggle device 100, the goggle device 100 may recognize hand gestures, and the user may manipulate the collaborative target 700 displayed on the displays 180A-180B and select the virtual content 702 via hand gestures captured and processed by the goggle device 100. In another example, using the mobile device 401, the mobile device 401 may interpret instructions received via the touch screen 580 of the mobile device 401. The user may manipulate the collaborative target 700 and select the virtual content 702 by touching / tapping the touch screen 580 with their finger to select the virtual content 702 and by dragging their finger to move the virtual content 702 onto the collaborative target 700 (which may associate the virtual content 702 with the collaborative target 700).

[0114] At block 1014, the processor associates the virtual content 702 with the collaborative target 700. The processor associates the virtual content 702 with the collaborative target 700 by updating the collaborative target 700 in response to the changes received from the physical remote device. In an example, upon receiving the added virtual content 702 from the physical remote device, the processor associates the added virtual content 702 with the collaborative target 700 in a location accessible to the physical remote device.

[0115] At block 1016, the processor stores the collaborative goal 700. The processor 499 stores the collaborative goal 700 in a memory accessible to the physically remote device via the network 495 during the collaborative session.

[0116] At block 1018, the processor provides access to the collaborative target 700. The processor 499 provides access to the collaborative target 700 in memory accessible to the physically remote device via the network 495. In an example, the processor checks the credentials (e.g., user ID) of the user requesting access and allows access if the credentials match the credentials associated with the collaborative session of the collaborative target 700.

[0117] At block 1020, the processor presents the collaborative target 700. The processor 499 presents the collaborative target 700 to the physical remote device via the network 495. In an example, the processor, in response to a request from the physical remote device, sends a file including the collaborative target 700 (and associated virtual content or a link to such content) to the physical remote device that has access to the collaborative session, and the physical remote device uses the file to generate a rendering including the collaborative target 700 and the associated virtual content for presentation on a display of the remote physical device. In one example, when the collaborative target 700 is placed in an open state, the associated virtual content is presented all at once. In another example, the associated virtual content is presented in sequence based on a timestamp added when the virtual content is associated with the collaborative target 700.

[0118] Fig.11 is a flowchart listing the steps of an example selective cooperative target access method. In the example, Fig.11 The steps are performed by a server system 498 (see Figure 4 ) is performed by the processor 499 of the server system 498. In other examples, one or more steps may be performed by the processors 432 and 540 of the physical remote device, or a combination of the processor 499 of the server system 498 and the physical remote device (acting as a processor for implementing the steps). One or more of the steps shown and described may be performed simultaneously, in series, in an order different from that shown and described, or in conjunction with additional steps. Some steps may be omitted, or repeated in some applications.

[0119] At block 1102, the processor receives a user identification code. The processor receives a user identification code of a user to be associated with a collaborative session. In one example, a user (host) accesses the server system 498 using a physical remote device. Using the physical remote device, the user creates a collaborative session and specifies other users to participate in the collaboration. For example, the host (user A) may invite another user (user B) to participate in the collaboration to prepare content for another user (user C) with the intention of providing the content to the user at a later time.

[0120] At box 1104, the processor receives access parameters for the user. The processor receives the access parameters for the user from the host or another user with an acceptable access level. The access parameters indicate the corresponding access level of each user's collaborative target 700, which allows the corresponding access level of at least one user to be different from the corresponding access level of another user. For example, the host may have a first access level (allowing access to access the collaborative target 700 to associate virtual content 704 and view the associated virtual content 704 during the collaborative period), and another user may have a second access level to the collaborative target 700 that is less than the first access level (for example, it only allows access after the collaborative period ends; or it permits access to the collaborative target 700 during the collaborative period, but does not access the associated virtual content 704 until after the collaborative period ends). In one example, the host receives an access level that enables access by default, and the host can grant other users access rights by providing access rights to the processor 499 of the server system 498 via a physical remote device.

[0121] At block 1106, the processor maintains an access parameter table. In an example, the processor maintains a table in the cloud storage that includes an identification code for each user and their respective access levels based on the access parameters provided by the host. In an example where the host (user A; A_ID) invites another user (user B; B_ID) to collaborate on a project for another user (user C; C_ID), the processor may initially create a table including the information identified in Table 1 below:

[0122] Table 1

[0123] user license A_ID yes B_ID no C_ID no

[0124] In Table 1, permission level "yes" indicates first level access and permission level "no" indicates second level access. As shown in Table 1, the host (A_ID) is initially the only user with access rights to provide access to, for example, virtual content 704 associated with collaborative target 700 or the ability to add virtual content 704.

[0125] At block 1108, the processor maintains a timer. The processor may receive a timer value from the host. The timer may be used to track the time left during the collaboration period.

[0126] At block 1110, the processor provides access to the collaborative target 700. The processor provides access to the collaborative target 700 based on the user's respective access level. In the example, when a user attempts to access the collaborative target 700, the processor compares the user's user certification (ID) to the value in the table. If the user's ID (e.g., D_ID) is not found on the table, the user will not be able to access the collaborative target 700. If another user (e.g., C_ID) is found in the table, but has a "no" permission level, the user will only be provided with access equivalent to that access level. If another user (e.g., A_ID) is found in the table with a "yes" permission level, the user will be provided with access to the collaborative target 700 equivalent to that access level.

[0127] At box 1112, the processor identifies the access level change. The processor identifies the access level change of at least one other user. In one example, the host grants another user (e.g., B_ID) access rights by sending an access rights change request including the user ID to be changed and the new access level to the processor 499 of the server system 498 via a physical remote device. In another example, all users with a "yes" permission level must request an access level change for a user with another permission level. The access level change is identified by the processor when the request is received. In another example, the processor can identify the change based on the expiration of the collaborative period or a preset "show" time (e.g., based on a monitored timer). For example, in the case where user A and user B prepare content for user C, user A and user B can initially have a "yes" permission level, and once the collaborative period has ended or the show time has arrived, the processor can automatically identify the access level change request of user C.

[0128] At block 1114, the processor changes the corresponding access level. The processor changes the corresponding access level of at least one other user in response to the access level change. In response to the identified access level change (e.g., changing the B_ID permission level to "yes"), the processor updates the table as shown in Table 2 below:

[0129] Table 2

[0130] user license A_ID yes B_ID yes C_ID no

[0131] Fig.121200 is a flowchart including steps of a method for use in a collaborative application 925. The processor 499 of the server system 498 presents a time shift of the collaborative target 700 during the collaborative period according to the timestamp on the display of its physical remote device. The time shift is a series of images including editing of the collaborative target 700 during the collaborative period. In the example, images are generated when virtual content 702 (collectively referred to as events) is added, modified and deleted, and each time the collaborative target 700 is edited. The time shift is a set of images (such as a video) presented sequentially in the order in which the events occurred. At the end of the collaborative period, the time shift is presented to the users on the displays of their physical remote devices via the network 495 as a presentation of the collective work during the collaborative session that the users can enjoy and appreciate. The collaborative period can be a target parameter specified by the user (e.g., see Fig.10 1004 and related description).

[0132] At block 1202, the processor 499 provides editing access to the collaborative target 700 to the user via the network 495 during the collaborative period. The processor 499 receives a request to join the collaborative session from a user of a plurality of users via the network 495. In an example, the request is a message and includes an authentication identification code. The processor 499 provides the user access to the collaborative target 700 when the user joins the collaborative session (e.g., by changing access permissions). The processor 499 presents the collaborative target 700 to the user via the network 495, for example, as a coloring on the display 180A-180B of the goggle device 100 or the display 580 of the mobile device 401. In one example, the user cannot edit the collaborative target 700 after the collaborative period. The processor 499 allows the user to access the collaborative target 700 using a corresponding physical remote device, such as the goggle device 100 and the mobile device 401, via the network 495. The processor 499 provides a time indicator for display on the physical remote device, which indicates when the collaborative period ends. In one example, the processor 499 automatically allows the requesting user to access the collaborative session. In another example, the processor 499 may receive a message from the host of the collaborative session and then authorize access to the collaborative session.

[0133] At block 1204, the processor 499 associates the virtual content 702 received from the user via the network 495 with the collaborative target 700 during the collaborative period, such as Figure 7-Figure 9 In the example, the user facilitates joining the collaboration by using their physical remote device to add and modify virtual content 702 at a selected location of the collaboration target 700, and is shown in the display of their respective device, such as the display 180A-180B of the goggle device 100 and the display 580 of the mobile device 401. In the example, the virtual content 702 may include a virtual image of the target and audio associated with the virtual image.

[0134] At block 1206, the processor 499 associates a timestamp with each virtual content 702 received via the network 495. The processor 499 associates the timestamp with the user corresponding to the received request. The timestamp is a time indicator identifying the time when the user requested access to join the collaborative session. If the timestamp is during the collaborative period, the processor 499 provides viewing access to the collaborative target 700.

[0135] At box 1208, the processor 499 generates and provides to the user the generated time-shift of the collaborative target 700 according to the timestamps. The time-shift is a series of images including editing of the collaborative target 700 during the collaborative period. In the example, images are generated and stored in memory each time the collaborative target 700 is edited, including when virtual content 702 is added, modified, and deleted (collectively referred to as events). The processor 499 records the images in memory at each event. The time-shift is a set of continuous images (e.g., as a video) or a stepped sequence of displayed images presented in the order in which the events occurred. The time-shift is presented to the user at the end of the collaborative period as a presentation that the users can enjoy and appreciate the collective work during the collaborative session.

[0136] Machine learning refers to algorithms that improve incrementally through experience. By processing a large number of different input data sets, a machine learning algorithm can develop improved generalizations about a specific data set, and then use those generalizations to produce accurate outputs or solutions when processing new data sets. Broadly speaking, a machine learning algorithm includes one or more parameters that will be adjusted or changed in response to new experiences, thereby incrementally improving the algorithm; a process similar to learning.

[0137] In the context of computer vision, mathematical models attempt to simulate the tasks performed by the human visual system, with the goal of using a computer to extract information from an image and achieve an accurate understanding of the image's content. Computer vision algorithms have been developed for use in a variety of fields, including artificial intelligence and autonomous navigation, to extract and analyze data from digital images and videos.

[0138] Deep learning refers to a class of machine learning methods based on or modeled after artificial neural networks. Artificial neural networks are computing systems composed of multiple simple, highly interconnected processing elements (nodes) that process information through their dynamic state responses to external inputs. Large artificial neural networks may have hundreds or thousands of nodes.

[0139] Convolutional neural networks (CNNs) are a type of neural network that is often used to analyze visual images, including digital photos and videos. The connectivity patterns between nodes in a CNN are typically modeled after the organization of the human visual cortex, which includes individual neurons arranged to respond to overlapping areas in the field of view. Neural networks suitable for use in the determination process described herein are based on one of the following architectures: VGG16, VGG19, ResNet50, Inception V3, Xception, or other CNN-compatible architectures.

[0140] In the machine learning example, at blocks 1008 and 1012, the processor 432 uses a machine trained algorithm known as a hand feature model to determine whether the detected series of hand shapes substantially match a predefined hand gesture. The processor 432 is configured to access the hand feature model trained by machine learning and apply the hand feature model to identify and locate features of hand shapes in one or more frames of video data.

[0141] In an example embodiment, the trained hand feature model receives a frame of video data including a detected hand shape and abstracts the image in the frame into layers for analysis. Based on the trained hand feature model, the data in each layer is compared layer by layer with the hand gesture data stored in the hand gesture library 480 until a good match is identified.

[0142] In one example, a layer-by-layer image analysis is performed using a convolutional neural network. In the first convolutional layer, the CNN identifies the learned features (e.g., hand landmarks, joint coordinate sets, etc.). In the second convolutional layer, the image is transformed into multiple images, wherein the learned features are each highlighted in the corresponding sub-images. In the merging layer, the size and resolution of the images and sub-images are reduced to isolate the portion of each image that includes features of possible interest (e.g., possible palm shapes, possible finger joints). The values ​​and comparisons of the images from the non-output layers are used to classify the images in the frame. Classification as used herein refers to the process of classifying images according to the detected hand shapes using a trained model. For example, if a series of detected two-handed hand shapes match the touch gestures stored in library 480, the image can be classified as a "touch action."

[0143] As described herein, any of the functions described for the goggle device 100, the mobile device 401, and the server system 498 may be embodied in one or more computer software applications or programming instruction sets. According to some examples, a "function," "functions," "application," "applications," "instruction," "instruction," or "programming" is a program that executes the functions defined in the program. One or more applications structured in different ways may be developed using different programming languages, such as a target-oriented programming language (e.g., Objective-C, Java, or C++) or a procedural programming language (e.g., C or assembly language). In a specific example, a third-party application (e.g., an application developed using ANDROID by an entity other than the vendor of a particular platform) may be used to develop an application. TM or IOS TM Software Development Kit (SDK) can be included in mobile operating systems (such as IOS TM ANDROID TM , Mobile software running on a mobile phone or another mobile operating system. In this example, the third-party application can call API calls provided by the operating system to facilitate the functions described herein.

[0144] Therefore, machine-readable media can take many forms of tangible storage media. Non-temporary storage media include, for example, optical or magnetic disks, such as any storage device in any computer device, etc., such as can be used to implement the client device shown in the accompanying drawings, media gateways, transcoders, etc. Temporary storage media include dynamic memory, such as the main memory of such a computer platform. Tangible transmission media include coaxial cables, copper wires and optical fibers, including wires containing buses within a computer system. Carrier transmission media can take the form of electrical or electromagnetic signals, or sound or light waves such as those generated during radio frequency (RF) and infrared (IR) data communications. Therefore, common forms of computer-readable media include, for example: floppy disks, flexible disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, DVDs or DVD-ROMs, any other optical media, punched card paper tapes, any other physical storage media with hole patterns, RAMs, PROMs and EPROMs, FLASH-EPROMs, any other memory chips or boxes, carriers that transmit data or instructions, cables or links that transmit such carriers, or any other media from which a computer can read programming codes or data. Many of these forms of computer readable media may be involved in carrying one or more sequences of one or more instructions to a processor for execution.

[0145] Except as immediately stated above, nothing stated or shown is intended or should be construed as conferring dedication or public equivalent to any component, step, feature, object, benefit, advantage, whether or not stated in the claims.

[0146] It should be understood that the terms and expressions used herein have ordinary meanings, as given to these terms and expressions for their corresponding corresponding queries and research fields, unless the specific meaning is otherwise set forth herein. Relational terms such as first and second, etc. can be used only to distinguish one entity or action from another entity or action, without requiring or implying any actual such relationship or order between these entities or actions. The term "comprises", "comprising", "includes", "including" or any other variation thereof is intended to cover non-exclusive inclusions, so that the process, method, thing or equipment including or including a series of elements or steps not only includes those elements or steps, but also includes other elements or steps that are not explicitly listed or inherent to such processes, methods, things or equipment. In the absence of further restrictions, an element preceded by "one" or "a kind of" does not exclude the presence of other identical elements in the process, method, thing or equipment including the element.

[0147] Unless otherwise stated, any and all measurements, values, ratings, positions, amplitudes, sizes, and other specifications set forth in this specification (including in the following claims) are approximate and imprecise. Such quantities are intended to have reasonable ranges consistent with the functions to which they are related and with the conventional art to which they belong. For example, parameter values, etc. may vary from the stated amount or range by up to ±10%, unless otherwise expressly stated.

[0148] In addition, in the above detailed description, it can be seen that for the purpose of streamlining the present disclosure, various features are grouped together in various examples. This method of the present disclosure should not be interpreted as reflecting the intention that the examples claimed for protection require more features than the features explicitly stated in each claim. On the contrary, as reflected in the following claims, the subject matter to be protected lies in fewer features than all the features of any single disclosed example. Therefore, the following claims are hereby incorporated into the detailed description, with each claim independently serving as a separately claimed subject matter.

[0149] While the foregoing has described what are considered to be the best modes and other examples, it should be understood that various modifications may be made therein and that the subject matter disclosed herein may be implemented in different forms and examples and that they may be applied to many applications, only some of which are described herein. The appended claims are intended to claim any and all modifications and variations that fall within the true scope of the present concept.

Claims

1. A server having a processor configured to perform an interactive augmented reality method, the processor being configured to: Providing users with access to collaborative objects; associating virtual content received from the user with the collaboration target during a collaboration period; associating a timestamp with each received virtual content; and A time shift of the collaboration target according to the time stamp is provided to the user.

2. The server according to claim 1, wherein: The processor is configured to allow the user to see the time shift of the collaboration target when the collaboration period ends.

3. The server according to claim 2, wherein: The user cannot access the associated virtual content received from other users until the collaborative period ends.

4. The server according to claim 1, wherein: The processor is configured to record the location of the collaboration target for each timestamp.

5. The server according to claim 4, wherein: The processor is configured to allow the user to see the collaboration target from different perspectives.

6. The server according to claim 1, wherein: The processor is configured to allow the user to access the collaboration target using a corresponding physical remote device.

7. The server according to claim 6, wherein: The processor is configured to provide the collaboration target to the physically remote device. 8 . The server of claim 1 , further comprising a maintenance timer including a countdown indicating when the collaboration period ends for associating virtual content with the collaboration target.

9. An interactive augmented reality method for use with a processor, comprising: Providing users with access to collaborative objects; associating virtual content received from the user with the collaborative target during a collaborative period; associating a timestamp with each received virtual content; as well as A time shift of the collaborative target with associated virtual content according to the timestamp is provided to the user.

10. The method according to claim 9, wherein: The steps to provide include: The user is allowed to see a time shift of the collaborative target with associated virtual content at the end of the collaborative period.

11. The method according to claim 10, wherein: The user cannot access the associated virtual content received from other users until the collaborative period ends.

12. The method according to claim 9, further comprising: The location of the collaborative target is recorded for each timestamp.

13. The method according to claim 12, wherein: The processor allows the user to see the collaborative target from different perspectives.

14. The method according to claim 9, wherein: The processor allows the user to access the collaborative target using a corresponding physical remote device.

15. The method according to claim 14, wherein: The processor provides the cooperation target to the physically remote device.

16. The method according to claim 9, further comprising: A maintenance timer including a countdown indicating when the collaboration period ends for associating virtual content with the collaboration target.

17. A non-transitory computer readable medium storing program code, the program code, when executed by an electronic processor of an electronic device having a display, configuring the electronic device to: Providing users with access to collaborative objects; associating virtual content received from the user with the collaborative target during a collaborative period; associating a timestamp with each received virtual content; and A time shift of the generated collaborative target with associated virtual content according to the time stamp is provided to the user.

18. The non-transitory computer readable medium storing program code according to claim 17, wherein the program code, when executed by the electronic processor, further configures the electronic device to: The user is allowed to see a time shift of the generated collaboration goal at the end of the collaboration period.

19. The non-transitory computer readable medium storing program code according to claim 18, wherein the program code, when executed by the electronic processor, further configures the electronic device to: The location of the collaborative target is recorded for each timestamp.

20. The non-transitory computer readable medium storing program code according to claim 19, wherein the program code, when executed by the electronic processor, further configures the electronic device to: See the collaborative goal from a different perspective.

Images