Collaborative targets associated with geographic locations

Abstract

And a collaborative session in which access to the added virtual content is selectively provided to the participant / user. A participant (host) creates a new session and invites the participant to join. The invited participant receives an invitation joining session. Content of the session may be accessed by a session creator (i.e., a host) and other approved participants. When a new participant joins the session, the session identifies the new participant, and the session concurrently notifies other participants in the session of the permission that the new participant is waiting for access to the added virtual content. The host or approved participant may set permission for the new participant to access the added virtual content.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Serial No. 17 / 900,846, filed August 31, 2022, the contents of which are incorporated herein by reference in their entirety. Technical Field

[0003] The examples set forth in this disclosure relate to the field of virtual reality for electronic devices, including mobile devices and wearable devices such as goggle devices. More specifically, but not in a limiting manner, this disclosure describes a collaborative method with selective access. Background Technology

[0004] Many types of computers and electronic devices available today (such as mobile devices (e.g., smartphones, tablets, and laptops), handheld devices, and wearable devices (e.g., smart glasses, digital goggles, headgear, hats, and head-mounted displays)) include a variety of cameras, sensors, wireless transceivers, input systems, and displays.

[0005] A graphical user interface allows users to interact with the displayed content, including virtual targets and graphical elements such as icons, taskbars, list boxes, menus, buttons, and selection control elements like cursors, pointers, handles, and slideshows.

[0006] Virtual reality (VR) technology generates a complete virtual environment that includes real-world images, sometimes displayed on VR headsets or other head-mounted displays. VR experiences allow users to move through the virtual environment and interact with virtual objects. Augmented reality (AR) is a type of VR technology that combines real-world objects in the physical environment with virtual objects and displays this combination to the user. The combined display gives the impression that the virtual objects truly exist in the environment, especially when the virtual objects look and behave like real objects. Cross-reality (XR) is generally understood to refer to a system that includes or combines elements from AR, VR, and MR (mixed reality) environments.

[0007] Collaboration tools can be used by VR technology users. These tools allow users to meet virtually in collaborative sessions. During these sessions, users can communicate with each other in a virtual environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The features of the various examples described will be readily understood from the following detailed description with reference to the accompanying drawings. Reference numerals are used in this specification and in the various views of the drawings with each element. When multiple similar elements exist, a single reference numeral may be assigned to the similar elements, with an additional lowercase letter indicating the specific element.

[0009] Unless otherwise specified, the various elements shown in the accompanying drawings are not drawn to scale. For clarity, the dimensions of the various elements may be enlarged or reduced. Several figures depict one or more embodiments and are presented by way of example only and should not be construed as limiting. The following figures are included in the accompanying drawings:

[0010] Figure 1A This is a side view (right) of an example hardware configuration suitable for use in an example collaborative system for goggle devices;

[0011] Figure 1B yes Figure 1A A partial cross-sectional view of the right corner of the goggle assembly, depicting the right visible light camera and circuit board;

[0012] Figure 1C yes Figure 1A A side view (left) of an example hardware configuration of the goggle device, showing the left visible light camera;

[0013] Figure 1D yes Figure 1C A partial cross-sectional view of the left corner of the goggle device, depicting the left visible light camera and circuit board;

[0014] Figure 2A and Figure 2B This is a rear view of an example hardware configuration for the goggle device used in the example collaborative system;

[0015] Figure 3 It is an illustrative description of a 3D scene, the left original image taken by the left visible light camera, and the right original image taken by the right visible light camera;

[0016] Figure 4 It is a functional block diagram of an example collaborative system that includes wearable devices (e.g., goggle devices) and server systems connected via various networks;

[0017] Figure 5 Is it suitable for Figure 4 A graphical representation of the example hardware configuration of the mobile device used in the example system;

[0018] Figure 6 This is a schematic diagram for use in an example environment that describes simultaneous location and mapping for users;

[0019] Figure 7 It is a perspective view of an example collaborative target in a box shape that can be manipulated by hand;

[0020] Figure 8 It is used to open Figure 7 A perspective view of the first example hand gesture associated with the opening gesture of the box depicted in the image;

[0021] Figure 9 It is used for closing Figure 7 A perspective view of a second example hand gesture associated with the box-closing gesture depicted in the image;

[0022] Figure 10 It is a flowchart that lists the steps in the example cooperative method; and

[0023] Figure 11 It is a flowchart that outlines the steps of the example selective collaborative target access method;

[0024] Figure 12 It is a flowchart outlining the steps for associating virtual collaborative goals with geographic locations; and

[0025] Figure 13 It depicts virtual time capsules hidden for users to find, such as near a tree. DETAILED DESCRIPTION

[0026] A collaborative session (e.g., a virtual time capsule) with access to a collaborative goal and added virtual content is selectively offered to participants / users. In one example of a collaborative session, a participant (the host) creates a new session and invites other participants to join. The session creator (i.e., the host) and other approved participants can access the session's content (e.g., which can be recorded using an application such as Lens Cloud features; Lens Cloud features are available from Snap Inc. in Santa Monica, California). During the collaborative session, virtual content received from users is associated with a virtual collaborative goal. The virtual collaborative goal is associated with a geographic location, and access to the virtual collaborative goal is permitted based on proximity to a physically remote geographic location.

[0027] The following detailed description includes systems, methods, techniques, instruction sequences, and computer program products illustrating examples set forth in this disclosure. Numerous details and examples are included to provide a thorough understanding of the disclosed subject matter and its associated teachings. However, those skilled in the art will understand how the associated teachings can be applied without such details. The aspects of the disclosed subject matter are not limited to the specific apparatus, system, and method described, as the associated teachings can be applied or practiced in many ways. The terminology and naming used herein are for the purpose of describing particular aspects only and are not intended to be limiting. In general, well-known examples of instructions, protocols, structures, and techniques are not necessarily shown in detail.

[0028] As used herein, the terms “connection” or “link” mean any logical, optical, physical, or electrical connection, including links through which electrical or magnetic signals generated or provided by one system element are imparted to another connected or linked system element. Unless otherwise described, connected or linked elements or devices are not necessarily directly connected to each other and can be separated by intermediate components, elements, or communication media, one or more of which can modify, manipulate, or carry electrical signals. The term “on” means directly supported by an element or indirectly supported by an element through another element integrated into or supported by the element.

[0029] The term "proximal" is used to describe a thing or part of a thing that is located near, adjacent to, or close to a target or person; or a part of a thing that is closer to other parts of a thing, which may be described as "distal". For example, the end of a thing that is closest to the target may be called the proximal end, while the generally opposite end may be called the distal end.

[0030] The orientations of the goggle device, other moving devices, associated components, and any other device incorporating a camera, inertial measurement unit, or both (as shown in any of the figures in the accompanying drawings) are given by way of example only for illustrative and discussion purposes. In operation, the goggle device may be oriented in any other direction suitable for the particular application of the goggle device; for example, upward, downward, lateral, or any other orientation. Moreover, within the scope used herein, any directional terms (such as front, back, inside, outside, facing, left, right, lateral, longitudinal, up, down, upper, lower, top, bottom, side, horizontal, vertical, and diagonal) are used by way of example only and are not limited to the orientation or orientation of any camera or inertial measurement unit as constructed or otherwise described herein.

[0031] Advanced AR technologies, such as computer vision and object tracking, can be used to create perceptually rich and immersive experiences. Computer vision algorithms extract 3D data about the physical world from data captured in digital images or videos. Object recognition and tracking algorithms are used to detect objects in digital images or videos, estimate their orientation or pose, and track their movement over time. Real-time hand and finger recognition and tracking is one of the most challenging and processing-intensive tasks in computer vision.

[0032] The term "pose" refers to the static position and orientation of a target at a given moment. The term "gesture" refers to the active movement of a target (such as a hand) through a series of poses, sometimes used to convey signals or ideas. The terms pose and gesture are sometimes used interchangeably in the fields of computer vision and augmented reality. As used herein, the terms "pose" or "gesture" (or variations thereof) are intended to include both pose and gesture; in other words, the use of one term does not exclude the other.

[0033] Further objects, advantages, and novel features of the example will be set forth in part in the description which follows, and will become partly apparent to those skilled in the art upon examination of the following and the accompanying drawings, or may be learned by the generation or operation of the example. The objects and advantages of this subject matter may be realized and obtained by means and combinations of methods, means, and combinations thereof specifically pointed out in the appended claims.

[0034] Now refer in detail to the examples illustrated in the accompanying drawings and discussed below.

[0035] Figure 1A This is a side view (right) of an example hardware configuration of the goggle device 100, including a touch-sensitive input device or touchpad 181, and Figure 1C This is a side view (left) of an example hardware configuration of a goggle device 100 including a touch-sensitive input device or touchpad 181. As shown, the touchpad 181 may have a subtle and not easily visible boundary; alternatively, the boundary may be clearly visible or include a raised or otherwise tactile edge that provides feedback to the user about the position and boundary of the touchpad 181. In other embodiments, the goggle device 100 may include a touchpad on the left side.

[0036] The surface of the touchpad 181 is configured to detect finger touches, taps, and gestures (e.g., movement touches) for use with a GUI displayed on an image display by the goggle device, allowing users to navigate and select menu options intuitively, which enhances and simplifies the user experience.

[0037] Detecting finger input on touchpad 181 can enable several functions. For example, touching anywhere on touchpad 181 can cause an item to be displayed in a GUI or highlighted on an image display, which can be projected onto at least one of optical components 180A, 180B. Double-tapping on touchpad 181 can select an item or icon. Sliding or tapping a finger in a specific direction (e.g., from front to back, from back to front, up to down, or down) can cause an item or icon to slide or scroll in that direction; for example, moving to the next item, icon, video, image, page, or slideshow. Sliding a finger in another direction can slide or scroll in the opposite direction; for example, moving to the previous item, icon, video, image, page, or slideshow. Touchpad 181 can be located virtually anywhere on goggle device 100.

[0038] In one example, a recognized finger gesture of a single tap on touchpad 181 initiates the selection or pressing of a graphical user interface element in an image displayed on the image displays of optical components 180A and 180B. Adjustments to the image displayed on the image displays of optical components 180A and 180B based on the recognized finger gesture can be the selection or submission of graphical user interface elements on the image displays of optical components 180A and 180B for further display or execution of primary actions.

[0039] As shown, the goggle device 100 includes a left visible light camera 114A and a right visible light camera 114B. As further described herein, the two cameras 114A and 114B capture image information of the scene from two separate perspectives. The captured images can be used to project a 3D display onto an image display for viewing with 3D glasses.

[0040] The goggle device 100 includes a right optical assembly 180B, which has an image display for presenting images (such as depth images). Figure 1A and Figure 1C As shown, the goggle device 100 may include a plurality of visible light cameras 114A and 114B, which form a passive type of three-dimensional camera, such as a stereo camera, with the right visible light camera 114B located at the right corner 110B, and as shown... Figures 1C to 1D As shown, the left visible light camera 114A is located at the left corner 110A.

[0041] Left visible light camera 114A and right visible light camera 114B are sensitive to wavelengths within the visible light range. Each of the visible light cameras 114A and 114B has a different forward-facing field of view, which overlaps to generate a three-dimensional depth image; for example, left visible light camera 114A captures a left field of view 111A, and right visible light camera 114B captures a right field of view 111B. Generally, a "field of view" is the portion of a scene visible through a camera at a specific location and orientation in space. Fields of view 111A and 111B have an overlapping field of view 304. Figure 3 When a visible light camera captures an image, targets or target features outside the fields of view 111A and 111B are not recorded in the original image (e.g., photograph or picture). The field of view describes the angular range or extent to which the image sensors of visible light cameras 114A and 114B pick up the electromagnetic radiation of a given scene in an image of that scene. The field of view can be expressed as the angular size of the view frustum (i.e., the viewing angle). The viewing angle can be measured horizontally, vertically, or diagonally.

[0042] In the example configuration, one or both of the visible light cameras 114A and 114B have a 100° field of view and a resolution of 480×480 pixels. "Angle of coverage" describes the visible light camera 114A, 114B, or the infrared camera 410 (see [link to relevant documentation]). Figure 2A The effective field of view is the range of angles within which a camera lens can image. Typically, a camera lens produces an image circle that is large enough to completely cover the camera's film or sensor, and may include some vignetting (e.g., the image darkens towards the edges when compared to the center). If the camera lens's coverage angle does not fill the sensor, the image circle will be visible, typically with strong vignetting towards the edges, and the effective field of view will be limited by the coverage angle.

[0043] Examples of such visible light cameras 114A and 114B include digital camera elements, such as high-resolution complementary metal-oxide-semiconductor (CMOS) image sensors and digital VGA cameras (video graphics arrays) capable of achieving resolutions of 640p (e.g., 640×480 pixels, totaling 300,000 pixels), 720p, or 1080p. Other examples include visible light cameras 114A and 114B that can capture high-definition (HD) video at high frame rates (e.g., 30-60 frames per second or greater) and store the recordings at a resolution of 1216×1216 pixels (or greater).

[0044] The goggle device 100 can capture image sensor data from visible light cameras 114A and 114B, as well as geographic location data digitized by an image processor, for storage in memory. The visible light cameras 114A and 114B capture corresponding left and right raw images in a two-dimensional spatial domain. These images comprise pixel matrices in a two-dimensional coordinate system, which includes an X-axis for horizontal position and a Y-axis for vertical position. Each pixel includes color attribute values ​​(e.g., red pixel light value, green pixel light value, or blue pixel light value) and position attributes (e.g., X-axis coordinates and Y-axis coordinates).

[0045] In order to capture stereoscopic images for later display as a 3D projection, image processor 412 ( Figure 4 The image processor 412 or another processor can be connected to visible light cameras 114A and 114B to receive and store visual image information. The image processor 412 or another processor controls the operation of the visible light cameras 114A and 114B to act as stereo cameras simulating human binocular vision, and can add timestamps to each image. The timestamps on each pair of images allow the images to be displayed together as part of a 3D projection. The 3D projection produces an immersive and realistic experience desired in various contexts, including virtual reality (VR) and video games.

[0046] Figure 1Byes Figure 1A A perspective cross-sectional view of the right corner 110B of the goggle device 100 depicts the right visible light camera 114B and the circuit board of the camera system. Figure 1C yes Figure 1A A side view (left) of an example hardware configuration of the goggle device 100, showing the left visible light camera 114A of the camera system. Figure 1D yes Figure 1C A perspective cross-sectional view of the left corner 110A of the goggle device, depicting the left visible light camera 114A of the three-dimensional camera and the circuit board.

[0047] like Figure 1B As shown in the example, the goggle assembly 100 includes a right visible light camera 114B and a circuit board 140B, which may be a flexible printed circuit board (PCB). A right hinge 126B connects the right corner 110B to the right temple 125B of the goggle assembly 100. In some examples, the right visible light camera 114B, the flexible PCB 140B, or other components with electrical connectors or contacts may be located on the right temple 125B or the right hinge 126B.

[0048] The left visible light camera 114A is constructed and positioned similarly to the right visible light camera 114B, except that the connection and coupling are located on the left outer side 170A. The left hinge 126B connects the left corner 110A to the left temple 125A of the goggle assembly 100. In some examples, components such as the left visible light camera 114A, the flexible PCB 140A, or other electrical connectors or contacts may be located on the left temple 125A or the left hinge 126A.

[0049] Left corner 110A and right corner 110B each include a corner body 190 and a corner cap, wherein in Figure 1B and Figure 1D The corner caps are omitted in the cross-sectional view. Located inside the left corner 110A and right corner 110B are various interconnecting circuit boards 140A and 140B, such as PCBs or flexible PCBs, which include controller circuitry for the left visible light camera 114A and the right visible light camera 114B, microphones, and low-power wireless circuitry (e.g., for use via Bluetooth). TM The corners 110A and 110B can be integrated into the frame 105 on the corresponding outer sides 170A and 170B (as shown) or implemented as separate components attached to the frame 105 on the corresponding sides 170A and 170B. Alternatively, the corners 110A and 110B can be integrated into the temples 125A and 125B attached to the frame 105.

[0050] The left visible light camera 114A and the right visible light camera 114B are coupled to or disposed on corresponding flexible PCBs 140A and 140B and covered by visible light camera lens caps, which are aimed through multiple openings formed in the frame 105. For example, the left frame 107A and the right frame 107B of the frame 105 are connected to the left corner 110A and the right corner 110B and include openings for the visible light camera lens caps. The frame 105 includes a front side configured to face outwards and away from the user's eye. The openings for the visible light camera lens caps are formed on the front side or the outward-facing side of the frame 105 and extend through the front side or the outward-facing side of the frame 105. In the example, the left visible light camera 114A and the right visible light camera 114B each have corresponding outward-facing fields of view 111A and 111B, which have lines of sight or viewing angles associated with the corresponding left and right eyes of the user of the goggle device 100. The visible light camera lens cover may also be adhered to the front side or outward-facing surface of the right corner 110B, wherein the opening forms an outward-facing coverage angle, but in different outward directions. This connection may also be indirect via an intermediate component.

[0051] Figure 2A and Figure 2B This is a rearward perspective view of an example hardware configuration of a goggle device 100 including two different types of image displays. The size and shape of the goggle device 100 are determined by its configuration for use by a user; the form of the goggles is shown in the example. The goggle device 100 may take other forms and may be combined with other types of frames; for example, a hood, headphones, or a helmet.

[0052] In the example of goggles, goggle device 100 includes a frame 105 that includes a left frame 107A connected to a right frame 107B via a bridging member 106 adapted to be supported by a user's nose. The left frame 107A and right frame 107B include corresponding apertures 175A, 175B that hold corresponding optical elements 180A, 180B, such as lenses and display devices. As used herein, the term "lens" is intended to include transparent or translucent sheets of glass or plastic having curved or flat surfaces that cause light to converge / diverge, or cause little or no convergence / divergence.

[0053] Although shown as having two optical elements 180A and 180B, depending on the application or intended user of the goggle device 100, the goggle device 100 may include other arrangements, such as a single optical element (or it may not include any optical elements 180A and 180B). As further shown, the goggle device 100 includes a left corner 110A adjacent to the left outer portion 170A of the frame 105 and a right corner 110B adjacent to the right outer portion 170B of the frame 105.

[0054] In one example, the image display of optical components 180A and 180B includes an integrated image display. For example... Figure 2A As shown, each optical component 180A, 180B includes a suitable display matrix 177, such as a liquid crystal display (LCD), an organic light-emitting diode (OLED) display, or any other such display. Each optical component 180A, 180B also includes one or more optical layers 176, which may include lenses, optical coatings, prisms, mirrors, waveguides, optical strips, and other optical components in any combination. Optical layers 176A, 176B, ... 176N (in... Figure 2A The optical layer 176A-N (shown herein as 176A-N) may include a prism having suitable dimensions and configuration and including a first surface for receiving light from a display matrix and a second surface for emitting light toward a user's eye. The prism of the optical layer 176A-N extends over all or at least a portion of corresponding apertures 175A, 175B formed in the left frame 107A and right frame 107B, allowing the user to see the second surface of the prism when the user's eye is viewing through the corresponding left frame 107A and right frame 107B. The first surface of the prism of the optical layer 176A-N faces upward from the frame 105, and the display matrix 177 covers the prism such that photons emitted by the display matrix 177 strike the first surface. The size and shape of the prism are determined such that the light beam is refracted within the prism and guided toward the user's eye by the second surface of the prism of the optical layer 176A-N. In this respect, the second surface of the prism of the optical layer 176A-N may be convex to direct light toward the center of the eye. The prism can optionally be sized and shaped to magnify the image projected by the display matrix 177, and the beam travels through the prism such that the image viewed from the second surface is larger than the image emitted from the display matrix 177 in one or more dimensions.

[0055] In one example, optical layers 176A-N may include a transparent (keeping the lens open) LCD layer unless and until a voltage is applied that makes the layer opaque (closing or blocking the lens). The image processor 412 on the goggle device 100 ( Figure 4Programming can be performed to apply voltage to the LCD layer to generate an active shutter system, thereby making the goggle device 100 suitable for viewing visual content when displayed as a three-dimensional projection. Technologies other than LCD can be used for the active shutter mode, including other types of reactive layers that respond to voltage or another type of input.

[0056] In another example, the image display device with optical components 180A and 180B includes, for example... Figure 2B The image projection display shown. Each optical component 180A, 180B includes a laser projector 150, which is a three-color laser projector using a scanning mirror or a galvanometer. During operation, a light source (such as the laser projector 150) is positioned in or above one of the temples 125A, 125B of the goggle assembly 100. In this example, the optical component 180B includes one or more optical bands 155A, 155B, ... 155N (e.g., ...). Figure 2B (shown as 155A-N), these optical bands 155A, 155B, ... 155N are spaced apart and span the width of the lens of each optical component 180A, 180B or the depth between the front and rear surfaces of the lens.

[0057] As photons projected by the laser projector 150 travel through the lenses of each optical component 180A, 180B, they encounter optical bands 155A-N. When a particular photon encounters a particular optical band, the photon is either redirected toward the user's eye, or it is passed to the lower optical band. A combination of modulation of the laser projector 150 and modulation of the optical bands can control specific photons or beams. In the example, the processor controls the optical bands 155A-N by initiating mechanical, acoustic, or electromagnetic signals. Although shown as having two optical components 180A, 180B, the goggle device 100 may include other arrangements, such as single or three optical components, or each optical component 180A, 180B may be arranged differently depending on the application of the goggle device 100 or the intended user.

[0058] In another example, Figure 2B The goggle device 100 shown may include two projectors: a left projector (not shown) and a right projector 150. The left optical assembly 180A may include a left display matrix 177 or a set of left optical strips (not shown) configured to interact with light from the left projector. Similarly, the right optical assembly 180B may include a right display matrix (not shown) or a set of right optical strips 155A, 155B, ..., 155N configured to interact with light from the right projector 150. In this example, the goggle device 100 includes a left display and a right display.

[0059] Figure 3This is an illustrative description of a 3D scene 306, a left original image 302A captured by a left visible light camera 114A, and a right original image 302B captured by a right visible light camera 114B. The left field of view 111A may overlap with the right field of view 111B as shown. The overlapping field of view 304 represents portions of the images captured by the two cameras 114A and 114B. When referring to field of view, the term "overlap" means that the pixel matrix in the resulting original image overlaps by thirty percent (30%) or more. "Approximately overlap" means that the pixel matrix in the resulting original image or infrared image of the scene overlaps by fifty percent (50%) or more. As described herein, the two original images 302A and 302B can be processed to include a timestamp that allows the images to be displayed together as part of a 3D projection.

[0060] To capture stereoscopic images, such as Figure 3 As shown, a pair of raw red, green, and blue (RGB) images of a real scene 306 are captured at a given moment—a left raw image 302A captured by the left camera 114A and a right raw image 302B captured by the right camera 114B. When the pair of raw images 302A and 302B are processed (e.g., by an image processor 412), a depth image is generated. The generated depth image can be viewed on the optics of the goggle device 180A and 180B, on another display (e.g., an image display 580 on the mobile device 401), or on a screen.

[0061] The generated depth image exists in a three-dimensional spatial domain and may comprise a vertex matrix in a three-dimensional positional coordinate system, including an X-axis for horizontal position (e.g., length), a Y-axis for vertical position (e.g., height), and a Z-axis for depth (e.g., distance). Each vertex may include color attributes (e.g., red pixel light value, green pixel light value, or blue pixel light value); positional attributes (e.g., X-coordinate, Y-coordinate, and Z-coordinate); texture attributes; reflectance attributes; or combinations thereof. Texture attributes quantify the perceptual texture of the depth image, such as the spatial arrangement of colors or intensities in the vertex regions of the depth image.

[0062] Figure 4This is a functional block diagram of an example collaborative system 400, which includes a wearable device (e.g., goggles device 100), a mobile device 401, and a server system 498 connected via various networks 495 (such as the Internet). The server system 498 may be one or more computing devices as part of a service or network computing system, including, for example, a processor, memory, and a network communication interface for communicating with the goggles device 100 and the mobile device 401 via the network 495. The server system 498 includes a server processor 499, which may be configured to host collaborative sessions. The functions of the goggles device 100 or the mobile device 401 described herein (such as collaborative processing and serving collaborative goals to the user) may be performed by the processor 499 of the server system 498.

[0063] The goggle device 100 includes one or more visible light cameras 114A, 114B, as described herein, for capturing still images, video images, or both. Cameras 114A, 114B may have direct memory access (DMA) to high-speed circuitry 430 and act as stereo cameras. Cameras 114A, 114B can be used to capture initial depth images reproducible as a three-dimensional (3D) model, which is an image of a texture map of a red, green, and blue (RGB) imaged scene. The device 100 may also include a depth sensor 213 that uses infrared signals to estimate the position of a target relative to the device 100. In some examples, the depth sensor includes one or more infrared emitters 215 and an infrared camera 410.

[0064] The goggle device 100 further includes two image displays for each optical component 180A, 180B (one associated with the left side 170A and one associated with the right side 170B). The goggle device 100 also includes an image display driver 442, an image processor 412, low-power circuitry 420, and high-speed circuitry 430. The image displays for each optical component 180A, 180B are used to display images including still images, video images, or a combination of still and video images. The image display driver 442 is coupled to the image displays for each optical component 180A, 180B to control the display of the images.

[0065] The goggle device 100 further includes one or more microphones (not shown) and one or more speakers 413 (e.g., one associated with the left side of the goggle device and another associated with the right side of the goggle device). The speakers 413 may be integrated into the frame 105, temple 125, or corner 110 of the goggle device 100. One or more speakers 413 are driven by an audio processor 414 and an audio driver 415 under the control of a low-power circuit 420, a high-speed circuit 430, or both. The speakers 413 are used to present audio signals, including, for example, beat tracks. The audio processor 414 is coupled to the microphones and speakers 413 to control the appropriate capture and presentation of sound.

[0066] Figure 4 The components shown for the goggle device 100 are located on one or more circuit boards (e.g., printed circuit boards (PCBs) or flexible printed circuit boards (FPCs)) in the frame or temples. Alternatively or additionally, the depicted components may be located in corners, frames, hinges, or bridging elements of the goggle device 100.

[0067] like Figure 4 As shown, the high-speed circuit 430 includes a high-speed processor 432, a memory 434, and a high-speed wireless circuit 436. In this example, an image display driver 442 is coupled to the high-speed circuit 430 and operated by the high-speed processor 432 to drive the left and right image displays of each optical component 180A, 180B. The high-speed processor 432 can be any processor capable of managing the high-speed communication and operation of any general-purpose computing system required by the goggle device 100. The high-speed processor 432 includes the processing resources required to manage high-speed data transmission to a wireless local area network (WLAN) over a high-speed wireless connection 437 using the high-speed wireless circuit 436.

[0068] In some examples, the high-speed processor 432 executes an operating system, such as the LINUX operating system or other such operating system for the goggle device 100, and this operating system is stored in memory 434 for execution. Among other duties, the high-speed processor 432 also executes a software architecture for the goggle device 100, which manages data transmission via high-speed wireless circuit 436. In some examples, the high-speed wireless circuit 436 is configured to implement the Institute of Electrical and Electronics Engineers (IEEE) 802.11 communication standard, also known herein as Wi-Fi. In other examples, other high-speed communication standards may be implemented by the high-speed wireless circuit 436.

[0069] Low-power circuitry 420 includes a low-power processor 422 and a low-power wireless circuitry 424. The low-power wireless circuitry 424 and high-speed wireless circuitry 436 of the goggle device 100 may include a short-range transceiver (Bluetooth). TM Or Bluetooth Low Energy (BLE) and wireless wide area network, local area network or wide area network transceivers (e.g. cellular or Wi-Fi). The mobile device 401, which includes transceivers that communicate via low-power wireless connection 425 and high-speed wireless connection 437, can be implemented using the architectural details of the goggle device 100, as can other elements of the network 495.

[0070] Memory 434 includes any storage device capable of storing various data and applications, including, among other things, camera data generated by the left visible light camera 114A and the right visible light camera 114B, the infrared camera(s) 410, the image processor 412, and the generated images displayed on the image display of each optical component 180A, 180B by the image display driver 442. Although memory 434 is shown as integrated with high-speed circuitry 430, in other examples memory 434 may be a separate, independent element of the goggle device 100. In some such examples, electrical routing lines may provide a connection from the image processor 412 or the low-power processor 422 to memory 434 via a chip including the high-speed processor 432. In other examples, the high-speed processor 432 may manage addressing of memory 434 such that the low-power processor 422 will activate the high-speed processor 432 whenever a read or write operation involving memory 434 is required.

[0071] like Figure 4 As shown, the high-speed processor 432 of the goggle device 100 can be connected to the camera system (visible light cameras 114A, 114B), image display driver 442, user input device 491, and memory 434.

[0072] The output component of the goggle device 100 includes a visual element, such as... Figure 2A and Figure 2BEach lens or optical component 180A, 180B described herein is associated with a left and right image display (e.g., a display such as a liquid crystal display (LCD), a plasma display panel (PDP), a light-emitting diode (LED) display, a projector, or a waveguide). The goggle device 100 may include user-facing indicators (e.g., LEDs, speakers 413, or vibration actuators) or outward-facing signals (e.g., LEDs, speakers 413). The image display of each optical component 180A, 180B is driven by an image display driver 442. In some example configurations, the output components of the goggle device 100 may also include additional indicators, such as audible elements (e.g., speakers 413), tactile elements (e.g., actuators such as vibration motors to generate tactile feedback), and other signal generators. For example, the device 100 may include a set of user-facing indicators and a set of outward-facing signals. The user-facing set of indicators is configured to be seen or otherwise sensed by a user of the device 100. For example, device 100 may include an LED display positioned so that it is visible to a user, one or more speakers 413 positioned to generate audible sounds to the user, or actuators providing tactile feedback that the user can feel. Outward-facing signal arrays are configured to be seen or otherwise sensed by an observer near device 100. Similarly, device 100 may include LEDs, speakers, or actuators configured and positioned to be sensed by an observer.

[0073] The input components of the goggle device 100 may include alphanumeric input components (e.g., a touchscreen or touchpad configured to receive alphanumeric input, an optical keyboard, or other alphanumeric configuration element), pointer-based input components (e.g., a mouse, touchpad, trackball, joystick, motion sensor, or other indicating instrument), haptic input components (e.g., a push-button switch, a touchscreen or touchpad, or other haptic configuration element that senses the position, force, or position and force of a touch or touch gesture), visual input components (e.g., cameras 114 / 420), and audio input components (e.g., a microphone), etc. The mobile device 401 and the server system 498 may include alphanumeric, pointer-based, haptic, audio, and other input components.

[0074] In some examples, the goggle device 100 includes a collection of motion-sensing components referred to as an inertial measurement unit (IMU) 472. The motion-sensing components can be microelectromechanical systems (MEMS) with microscopic moving parts, typically small enough to be part of a microchip. In some example configurations, the IMU 472 includes an accelerometer, a gyroscope, and a magnetometer. The accelerometer senses the linear acceleration (including acceleration due to gravity) of the device 100 relative to three orthogonal axes (x, y, z). The gyroscope senses the angular velocity of the device 100 about three rotational axes (pitch, roll, yaw). The accelerometer and gyroscope can together provide position, orientation, and motion data about the device relative to six axes (x, y, z, pitch, roll, yaw). The magnetometer (if present) senses the orientation of the device 100 relative to magnetic north. The position of the device 100 can be determined by a position sensor (such as a GPS unit), one or more transceivers for generating relative position coordinates, an altitude sensor or barometer, and other orientation sensors. Such positioning system coordinates can also be received from mobile device 401 via low-power wireless circuit 424 or high-speed wireless circuit 436 through wireless connections 425 and 437.

[0075] IMU 472 may include, or work in conjunction with, a digital motion processor or program that collects raw data from components and calculates multiple useful values ​​regarding the position, orientation, and motion of device 100. For example, acceleration data collected from an accelerometer may be integrated to obtain velocity relative to each axis (x, y, z); and integrated again to obtain the position of device 100 (in linear coordinates x, y, and z). Angular velocity data from a gyroscope may be integrated to obtain the position of device 100 (in spherical coordinates). Programming for calculating these useful values ​​may be stored in memory 434 and executed by the high-speed processor 432 of the goggle device 100.

[0076] The goggle device 100 may optionally include additional peripheral sensors, such as biometric sensors, dedicated sensors, or display elements integrated with the goggle device 100. For example, peripheral device elements may include any I / O components, including output components, motion components, position components, or any other such components described herein. For example, biometric sensors may include components for detecting facial expressions (e.g., hand expressions, facial expressions, vocal expressions, body posture, or eye tracking), for measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), or for identifying a person (e.g., identification based on voice, retina, facial features, fingerprints, or electrobiosignals such as electroencephalogram data).

[0077] Mobile device 401 may be a smartphone, tablet computer, laptop computer, access point, or any other such device capable of connecting to goggle device 100 using both low-power wireless connection 425 and high-speed wireless connection 437. Mobile device 401 is connected to server system 498 and network 495. Network 495 may include any combination of wired and wireless connections.

[0078] like Figure 4 As shown, the illustrated collaborative system 400 includes a computing device, such as a mobile device 401, connected via a network to the goggle device 100. The collaborative system 400 includes a memory for storing instructions and a processor for executing those instructions. The execution of instructions by the processor 432 by the collaborative system 400 configures the goggle device 100 to collaborate with the mobile device 401. The collaborative system 400 may utilize the memory 434 of the goggle device 100 or the memory elements 540A, 540B, 540C of the mobile device 401. Figure 5 Furthermore, the collaborative system 400 can utilize the processor elements 432, 422 of the goggle device 100 or the central processing unit (CPU) 540 of the mobile device 401. Figure 5 Furthermore, the collaborative system 400 can further utilize the memory and processor elements of the server system 498. In this respect, the memory and processing capabilities of the collaborative system 400 can be shared or distributed across the processors and memory of the goggle device 100, the mobile device 401, and the server system 498 to achieve the functions described herein.

[0079] In some example embodiments, memory 434 includes or is coupled to hand gesture library 480, as described herein. In some implementations, the process of detecting hand shape or gesture involves comparing pixel-level data from one or more video data frames captured by goggles 100 or mobile device 401 with hand shapes and gestures stored in library 480 until a good match is found. A gesture can be a static posture that can be detected in one or more data frames, or a dynamic posture that can be detected during a process of two or more data frames.

[0080] In some example embodiments, memory 434 further includes an element animation application 910, a positioning system 915, an image processing system 920, and a collaboration application 925. In the collaboration system 400, where the camera is capturing video data frames, the element animation application 910 configures processor 432 to control the movement of a series of virtual objects 700 on the display in response to the detection of one or more inputs (e.g., IMU data, captured images, and hand shapes or gestures). The positioning system 915 configures processor 432 to obtain positioning data for determining the position of the goggle device 100 relative to the physical environment. Positioning data can be derived from a series of images, IMU unit 472, GPS unit, or a combination thereof. The image processing system 920 configures processor 432 to collaborate with image display driver 442 and image processor 412 to present captured images on the displays of optical components 180A, 180B. The collaboration application 925 configures processor 432 to implement the collaboration functions described herein.

[0081] Figure 5 This is a high-level functional block diagram of an example mobile device 401. The mobile device 401 includes flash memory 540A, which stores programming to be executed by CPU 540 to perform all or a subset of the functions described herein.

[0082] Mobile device 401 may include camera 570, which includes at least two visible light cameras (a first visible light camera and a second visible light camera with overlapping fields of view) or at least one visible light camera and a depth sensor with substantially overlapping fields of view. Mobile device 401 may additionally include speaker 571. Flash memory 540A may further include a plurality of images or videos generated via camera 570.

[0083] As shown in the figure, the mobile device 401 includes an image display 580, a mobile display driver 582 for controlling the image display 580, and a display controller 584. Figure 5 In one example, the image display 580 includes a user input layer 591 (e.g., a touchscreen), which is layered on top of the screen used by the image display 580 or otherwise integrated into the screen used by the image display 580.

[0084] Examples of touchscreen mobile devices that can be used include (but are not limited to) smartphones, personal digital assistants (PDAs), tablet computers, laptop computers, or other portable devices. However, the structure and operation of touchscreen devices are provided by way of example; the subject matter described herein is not intended to be limited thereto. For the purposes of discussion, Figure 5Therefore, a block diagram illustration of an example mobile device 401 with a user interface is provided, the user interface including a touch screen input layer 591 for receiving input (by touch, multi-touch or gesture, etc., by hand, stylus or other tools), a camera 570 for capturing images of targets (including the user's hand and potential virtual content), and an image display 580 for displaying content.

[0085] like Figure 5 As shown, mobile device 401 includes at least one digital transceiver (XCVR) 510 for digital wireless communication via a wide-area wireless mobile communication network, shown as a WWAN XCVR. Mobile device 401 also includes additional digital or analog transceivers, such as a short-range transceiver (XCVR) 520 for short-range network communication, such as via NFC, VLC, DECT, ZigBee, Bluetooth. TM Or Wi-Fi. For example, the short-range XCVR 520 can take the form of any available bidirectional wireless LAN (WLAN) transceiver that is compatible with one or more standard protocols for communication implemented in a wireless LAN, such as one of the Wi-Fi standards under IEEE 802.11.

[0086] To generate location coordinates for locating mobile device 401, mobile device 401 may include a Global Positioning System (GPS) receiver. Alternatively or additionally, goggle device 100 or mobile device 401 may utilize either or both of a short-range XCVR 520 and a WWAN XCVR 510 to generate location coordinates for positioning. For example, cellular networks, Wi-Fi, or Bluetooth. TM The base positioning system can generate very accurate position coordinates, especially when used in combination. Such position coordinates can be transmitted between the goggle device 100 or the mobile device 401 via one or more network connections through XCVRs 510, 520.

[0087] In some examples, the mobile device 401 includes a collection of motion sensing components, referred to as an inertial measurement unit (IMU) 572, for sensing the position, orientation, and motion of the client device 401. The motion sensing components can be microelectromechanical systems (MEMS) with microscopic moving parts, typically small enough to be part of a microchip. In some example configurations, the inertial measurement unit (IMU) 572 includes an accelerometer, a gyroscope, and a magnetometer. The accelerometer senses the linear acceleration (including acceleration due to gravity) of the client device 401 relative to three orthogonal axes (x, y, z). The gyroscope senses the angular velocity of the client device 401 about three rotational axes (pitch, roll, yaw). The accelerometer and gyroscope can together provide position, orientation, and motion data about the device relative to six axes (x, y, z, pitch, roll, yaw). The magnetometer (if present) senses the orientation of the client device 401 relative to magnetic north.

[0088] The IMU 572 may include, or work in conjunction with, a digital motion processor or program that collects raw data from components and calculates multiple useful values ​​regarding the position, orientation, and motion of the client device 401. For example, acceleration data collected from an accelerometer may be integrated to obtain velocity relative to each axis (x, y, z); and integrated again to obtain the position of the client device 401 (in linear coordinates x, y, and z). Angular velocity data from a gyroscope may be integrated to obtain the position of the client device 401 (in spherical coordinates). The programming used to calculate these useful values ​​may be stored on one or more memory elements 540A, 540B, 540C and executed by the CPU 540 of the client device 401.

[0089] Transceivers 510 and 520 (i.e., network communication interfaces) conform to one or more of the different digital wireless communication standards used in modern mobile networks. Examples of WWAN transceivers 510 include (but are not limited to) transceivers configured to operate according to Code Division Multiple Access (CDMA) and 3rd Generation Partnership Project (3GPP) network technologies, including, for example, but not limited to, 3GPP Type 2 (or 3GPP2) and LTE, sometimes referred to as "4G". For example, transceivers 510 and 520 provide bidirectional wireless communication of information, including digitized audio signals, still images and video signals, web page information for display and network-related input, and different types of mobile message communication to / from mobile device 401.

[0090] Mobile device 401 further includes a microprocessor serving as a central processing unit (CPU); Figure 5The example shown is CPU 540. A processor is a circuit having elements configured and arranged to perform one or more processing functions (typically different data processing functions). While discrete logic components can be used, this example utilizes parts that form a programmable CPU. A microprocessor, for example, includes one or more integrated circuit (IC) chips that combine electronic components to perform the functions of the CPU. For example, CPU 540 may be based on any known or available microprocessor architecture, such as Reduced Instruction Set Computing (RISC) using the ARM architecture, as is commonly used today in mobile devices and other portable electronic devices. Of course, other arrangements of the processor circuitry can be used to form CPU 540 or processor hardware in smartphones, laptops, and tablets.

[0091] CPU 540 acts as a programmable host controller for mobile device 401 by configuring mobile device 401 to perform various operations, for example, according to instructions executable by CPU 540 or programmed. Such operations may include various general operations of the mobile device, as well as operations related to programming of applications on the mobile device. While a processor can be configured using hardwired logic, a typical processor in a mobile device is a conventional processing circuit configured through the execution of programming.

[0092] Mobile device 401 includes a memory or storage system for storing programming and data. In this example, the memory system may include, as needed, flash memory 540A, random access memory (RAM) 540B, and other memory components 540C. RAM 540B serves as short-term storage for instructions and data processed by CPU 540, for example, as working data processing memory. Flash memory 540A typically provides long-term storage.

[0093] Therefore, in the example of mobile device 401, flash memory 540A is used to store programs or instructions for execution by CPU 540. Depending on the type of device, mobile device 401 stores and runs a mobile operating system, through which specific applications are executed. Examples of mobile operating systems include Google Android, Apple iOS (for iPhone or iPad devices), Windows Mobile, Amazon Fire OS, RIM BlackBerry OS, etc.

[0094] like Figure 5As shown, the CPU 540 of the mobile device 401 can be connected to the camera system 570, the mobile display driver 582, the user input layer 591, and the memory 540A. Many components and functions of the goggle device 100 described herein can be incorporated into the mobile device 401. Similarly, the components and functions of the mobile device 401 described herein can be incorporated into the goggle device 100.

[0095] The processor 432 within the goggle device 100 or the processor 540 within the mobile device 401 can construct a map of the environment surrounding the device, determine the device's position within the map environment, and determine the device's relative position to one or more targets in the map environment. The processors 432 / 540 can use conventional Simultaneous Localization and Mapping (SLAM) algorithms applied to data received from one or more sensors to construct the map and determine location and position information. Sensor data includes images received from one or both of cameras 114A, 114B, or 570; distances received from a laser rangefinder; position information received from a GPS unit; motion and acceleration data received from IMU 472 / 572; or combinations of data from such sensors or from other sensors that provide data useful in determining position information.

[0096] In the context of augmented reality, SLAM algorithms are used to build and update maps of the environment while simultaneously tracking and updating the location of a device (or user) within the map environment. Mathematical solutions can be approximated using various statistical methods, such as particle filters, Kalman filters, extended Kalman filters, and covariance intersection. In systems including high-definition (HD) cameras that capture video at high frame rates (e.g., thirty frames per second), SLAM algorithms update the map and target location at least as frequently as the frame rate; in other words, the map and localization are calculated and updated thirty times per second.

[0097] Figure 6 An example physical environment 600 and useful elements are depicted when using SLAM applications and other types of tracking applications (e.g., Natural Feature Tracking (NFT)). Although the following example is provided with reference to the goggle device 100, this example can be implemented in a similar manner in the mobile device 401. The user 602 of the goggle device 100 exists in the example physical environment 600 (in... Figure 6In an indoor environment, the processor 432 of the goggle device 100 uses captured images to determine its position relative to one or more targets 604 within the environment 600, constructs a map of the environment 600 using the coordinate system (x, y, z) of the environment 600, and determines its position within the coordinate system. Additionally, the processor 432 determines the head posture (roll, pitch, and yaw) of the goggle device 100 within the environment by using two or more location points (e.g., three location points 606a, 606b, and 606c) associated with a single target 604a, or by using one or more location points 606 associated with two or more targets 604a, 604b, and 604c. The processor 432 of the goggle device 100 can also use virtual targets 608 (such as...) Figure 6 The buttons shown are positioned within environment 600 for viewing during augmented reality experiences (such as collaborative augmented reality experiences), where each user has a corresponding augmented reality device (e.g., goggle device 100 or mobile device 401).

[0098] In some examples, the positioning system 915 associates a virtual tag 610a with a virtual target 608 in the environment 600. In augmented reality, registering tags at locations in the environment helps the device track and update the locations of targets (virtual and physical) in the user, device, and map environment. Sometimes tags are registered to high-contrast physical targets (such as relatively dark targets, such as a framed picture 604a mounted on a light-colored wall) to help cameras and other sensors detect the tags. These tags can be pre-assigned or can be assigned by the goggle device 100 upon entering the environment.

[0099] The marker can be encoded with information or otherwise linked to information. The marker may include location information, physical codes (such as barcodes or QR codes; visible or hidden from the user), or a combination thereof. A set of data associated with the marker is stored in the memory 434 of the goggle device 100. This set of data includes information about the marker 610a, the marker's location (position and orientation), one or more virtual targets, or a combination thereof. The marker location may include the three-dimensional coordinates of one or more marker landmarks 616a, such as... Figure 6The corner of the roughly rectangular marker 610a shown. The marker position can be expressed relative to real-world geographic coordinates, a marker coordinate system, the position of the goggle device 100, or other coordinate systems. One or more virtual targets associated with marker 610a can include any of a variety of materials, including still images, videos, audio, haptic feedback, executable applications, interactive user interfaces, and experiences, as well as combinations or arrangements of such materials. Any type of content that can be stored in memory and retrieved upon encountering marker 610a or associated with an assigned marker can be classified as a virtual target in this context. For example, Figure 6 The key 608 shown is a virtual target displayed as a static image (2D or 3D) at the marked location.

[0100] In one example, tag 610a can be registered in memory as located at physical target 604a (e.g., Figure 6 The mark is located near and associated with the framed artwork shown. In another example, the mark may be registered in memory at a specific location relative to the goggle device 100.

[0101] Figure 7 , Figure 8 and Figure 9 The illustration for example collaboration objective 700 is developed by adding virtual content 702 during the collaboration session to describe the following. Figure 10 and Figure 11 The steps of the method illustrated herein (e.g., for creating a virtual time capsule). Although the box is used as the cooperative target 700 in many of the examples described herein, any virtual target can be selected to be used as the cooperative target 700. Additionally, although the goggle device 100 is depicted as a physical remote device in the illustrations, it should be understood that the functionality described below can be implemented using other physical remote devices (e.g., mobile device 401).

[0102] Figure 7 A perspective view of an example cooperative target 700 in a box-like form in a first state (closed) is provided. This cooperative target can be manipulated in three dimensions 601 based on a corresponding movement of a hand 651 (e.g., through gestures detected in an image on a touchscreen or touch input). For example, the hand 651 can be rotated to rotate the cooperative target. In some examples, an extended index finger can be detected near the cooperative target, and a corresponding audible signal will be presented via a speaker when the user makes a tapping gesture. The position of the goggle device can also be tracked in three dimensions 840 within the environment 600, making the shading generated for presentation on the display 180B more realistic. The hand 651 can be predefined as the left hand, as shown. In some embodiments, the system includes a process for selecting and setting the left or right hand to act as the detected hand 651.

[0103] Figure 8 A perspective view of an example collaborative target 700 in a second state (open) is provided, where associated virtual content 702 (dial 702a, pot 702b, book 702c, other virtual content 702d-f) is added during the collaborative period. Hand 652 is shown in the open position. This position of hand 652, or hand 651 in the relaxed position (… Figure 7 The transition of hand 652 to the open position can be set to correspond to the open cooperative target 700, such that when the hand position or hand gesture is detected, the cooperative target 700 transitions to the open state.

[0104] Figure 9 A perspective view of an example cooperative target 700 in its first state (closed) is provided, where virtual content 703a-703c is added to the outer surface of the cooperative target 700. Hand 653 is illustrated in the closed position. This position of hand 653, or hand 651 in the relaxed position... Figure 7 The transition of hand 653 to the closed position can be set to correspond to the closed cooperative target 700, such that when the hand position or hand gesture is detected, the cooperative target 700 transitions to the closed state.

[0105] The detection and tracking process includes detecting hands 651 / 652 / 653 over time, in various poses, and in a set or series of captured video data frames. In this context, detection refers to and includes detecting hands in as few as one video data frame, as well as detecting hands in a subset or series of frames of video data over time. Therefore, in some embodiments, the process includes detecting a hand 651 in a specific pose in one or more captured frames of video data. In other embodiments, the process includes detecting hands 651 / 652 / 653 over time in different poses within a subset or series of captured video data frames.

[0106] Figure 10 This is a flowchart 1000 depicting an example method for developing a collaborative objective 700 during a collaborative session comprising multiple physical remote devices (e.g., goggle device 100, mobile device 401, or combinations thereof). In the example, Figure 10 The steps are performed by a server system 498 that can be accessed by a physically remote device (see...). Figure 4The steps are executed by processor 499. In other examples, one or more steps may be executed by processors 432 and 540 of the physical remote device, or by a combination of the processor of server system 498 and the physical remote device (acting as the processor for implementing the steps). One or more of the steps shown and described may be executed simultaneously, sequentially, in an order different from that shown and described, or in combination with additional steps. Some steps may be omitted, or these steps may be repeated in some applications.

[0107] At box 1002, the processor receives user parameters for the collaborative session. In this example, the processor receives user parameters from the host user's physical remote device, where the host user specifies the user parameters via their physical remote device during the connection to server system 498. The user parameters include an identifier code of the user allowed to access the collaborative session. The user parameters may also include an access level identifying which individual is allowed access during the collaborative session. Refer below to flowchart 1100 ( Figure 11 The steps describe additional details about setting and maintaining access levels.

[0108] At box 1004, the processor receives target parameters. In this example, the processor receives the target parameters from the physical remote device of the host user (or another user with an appropriate access level), where the user specifies the target parameters via their physical remote device during a connection to server system 498. The target parameters include an identifier that identifies the target to be used as the cooperative target 700 (e.g., such as...). Figure 7-Figure 9 (as shown in the box). Other target parameters may include the target's material (e.g., cardboard, metal, glass) or time parameters that provide a time window or period (e.g., clock value 710 or timeline 712), during which virtual content 702 may be added to collaborative target 700 (after which virtual content 702 may no longer be added to collaborative target 700).

[0109] The processor of server system 498 may present a list of available virtual targets 702 for a user to select via their device to a physical remote device via network 495, the list being received by the processor of server system 498 when selected. Alternatively, the user may send virtual targets 702 (e.g., 3D images) generated on their physical remote device to server system 498, wherein the processor 499 of server system 498 designates the received virtual target 702 as a cooperative target 700 upon receipt.

[0110] At box 1006, the processor provides access to the collaborative target 700. The processor provides access to the collaborative target 700 via a server connection to the physical remote device, based on an access level associated with each device. In the example, processor 499 develops the collaborative target 700 in response to received target parameters and stores the collaborative target 700 in a location accessible to the physical remote device. The processor provides access to the collaborative target 700 based on the access level associated with the user of the physical remote device. When the user accesses the collaborative target 700, the processor sends a file including the collaborative target 700 to mobile device 401, which the physical remote device uses to generate a coloring for rendering on the display of the physical remote device (such as displays 180A-180B of the goggle device 100 or display 580). The user can then interact with the representation of the collaborative target 700 on their display (e.g., using...). Figure 7 (The hand gesture shown).

[0111] At box 1008, the processor receives design parameters. The processor receives the design parameters from a physical remote device with an appropriate permission level to access the collaborative target 700. In this example, the processor sends a file to the physical remote device that includes the collaborative target 700. The physical remote device generates shading on a display that interacts with the user to add design parameters to the collaborative target 700. The user can then select an image (e.g., from their camera, such as cameras 114A-114B or 570) and select a surface of the collaborative target 700, where the selected image is associated with the collaborative target 700 on the physical remote device after the surface is selected. When the user makes changes on their device, the added / changed design parameters are transmitted from the physical remote device to the server system 498 via network 498.

[0112] At box 1010, the processor updates the collaborative target 700 in response to design parameters. The processor updates the collaborative target 700 in response to changes received from a physical remote device via network 495. In the example, when receiving added / changed design parameters from the physical remote device, the processor associates the added / changed design parameters with the collaborative target 700 in a location accessible to the physical remote device.

[0113] At box 1012, the processor receives virtual content 702. The processor receives virtual content 702 from a physical remote device having an appropriate permission level for accessing the collaborative target 700. In the example, the processor sends a file to the physical remote device that includes the collaborative target 700. The physical remote device generates shading on a display that interacts with the user to add virtual content 702 to the collaborative target 700. The user can then associate the virtual content 702 with the collaborative target 700 on the physical remote device, for example, by selecting an image (e.g., from their camera) and performing an action (e.g., dragging and dropping the image onto the collaborative target 700 or double-clicking the target). Furthermore, audio virtual content can be added to video virtual content by, for example, pressing and holding video virtual content and speaking into a microphone, where audio received while pressing the video virtual content is associated with the video virtual content. When the user makes changes on their device, the added virtual content 702 is transmitted from the physical remote device to the server system 498 via network 495.

[0114] In one example, a user can associate virtual content 702 with a collaborative target 700 by, for example, dragging and dropping virtual content 702 onto the surface of the collaborative target 700. In another example, using a goggle device 100 that can recognize hand gestures, a user can manipulate the collaborative target 700 displayed on displays 180A-180B and select virtual content 702 via hand gestures captured and processed by the goggle device 100. In yet another example, using a mobile device 401 that can interpret instructions received via its touchscreen 580, a user can manipulate the collaborative target 700 and select virtual content 702 by touching / tapping the touchscreen 580 with their fingers and by dragging their fingers to move virtual content 702 onto the collaborative target 700 (which associates virtual content 702 with the collaborative target 700).

[0115] At box 1014, the processor associates virtual content 702 with cooperative target 700. The processor associates virtual content 702 with cooperative target 700 by updating cooperative target 700 in response to changes received from the physical remote device. In the example, when added virtual content 702 is received from the physical remote device, the processor associates the added virtual content 702 with cooperative target 700 in a location accessible to the physical remote device.

[0116] At box 1016, the processor stores the cooperative target 700. During a cooperative session, the processor 499 stores the cooperative target 700 in a memory accessible to the physical remote device via network 495.

[0117] At box 1018, the processor provides access to the cooperative target 700. Processor 499 provides access to the cooperative target 700 in memory accessible via network 495 from a physically remote device. In the example, the processor checks the credentials (e.g., user ID) of the user requesting access, and allows access if the credentials match the credentials associated with the cooperative session of the cooperative target 700.

[0118] At box 1020, the processor renders the collaborative target 700. Processor 499 renders the collaborative target 700 to a physical remote device via network 495. In the example, in response to a request from the physical remote device, the processor sends a file including the collaborative target 700 (and associated virtual content or links to such content) to the physical remote device that has access to the collaborative session. The physical remote device uses this file to generate colorings including the collaborative target 700 and the associated virtual content for rendering on the display of the remote physical device. In one example, when the collaborative target 700 is in an open state, the associated virtual content is rendered all at once. In another example, the associated virtual content is rendered sequentially based on timestamps added when the virtual content is associated with the collaborative target 700.

[0119] Figure 11 This is a flowchart outlining the steps of an example selective cooperative target access method. In the example, Figure 11 The steps are performed by a server system 498 that can be accessed by a physically remote device (see...). Figure 4 The steps are executed by processor 499. In other examples, one or more steps may be executed by processors 432 and 540 of the physical remote device, or by a combination of processor 499 of server system 498 and the physical remote device (acting as processors for implementing the steps). One or more steps shown and described may be executed simultaneously, sequentially, in an order different from that shown and described, or in combination with additional steps. Some steps may be omitted, or these steps may be repeated in some applications.

[0120] At box 1102, the processor receives a user identification code. The processor receives the user identification code of the user to be associated with the collaboration session. In one example, the user (host) accesses server system 498 using a physical remote device. Using the physical remote device, the user creates a collaboration session and designates other users to participate in the collaboration. For example, the host (user A) may invite another user (user B) to participate in the collaboration to prepare content for another user (user C), with the intention of providing the content to that user later.

[0121] At box 1104, the processor receives user access parameters. The processor receives user access parameters from the host or another user with an acceptable access level. The access parameters indicate the corresponding access level for each user's collaborative target 700, which allows at least one user's corresponding access level to differ from another user's corresponding access level. For example, the host may have a first access level (allowing access to the collaborative target 700 to associate and view the associated virtual content 702 during a collaborative session), and another user may have a second access level to the collaborative target 700 that is lower than the first access level (e.g., allowing access only after the collaborative session ends; or allowing access to the collaborative target 700 during the collaborative session but not to the associated virtual content 702 until after the collaborative session ends). In one example, the host receives an access level that is enabled by default, and the host may grant access to other users by providing access to the processor 499 of the server system 498 via a physical remote device.

[0122] At box 1106, the processor maintains an access parameter table. In the example, the processor maintains a table in cloud storage that includes an identifier for each user based on access parameters provided by the host, along with their respective access levels. In the example where a host (user A; A_ID) invites another user (user B; B_ID) to collaborate on a project for another user (user C; C_ID), the processor may initially create a table that includes the information identified in Table 1 below:

[0123] Table 1

[0124] user license A_ID yes B_ID no C_ID no

[0125] In Table 1, the permission level "Yes" indicates Level 1 access, and the permission level "No" indicates Level 2 access. As shown in Table 1, the host (A_ID) is initially a unique user with access rights to provide access to, for example, virtual content 702 associated with the collaborative target 700 or the ability to add virtual content 702.

[0126] At box 1108, the processor maintains a timer. The processor can receive an initial time value from the host. The timer can be used to track the remaining time during a cooperative period. In one example, the processor provides a time value corresponding to the initial time or the tracked time to a physical remote device, which can use the time value to generate a coloring that depicts or represents the remaining time (e.g., clock value 710 or time axis 712).

[0127] At box 1110, the processor provides access to collaborative target 700. The processor provides access to collaborative target 700 based on each user's respective access level. In the example, when a user attempts to access collaborative target 700, the processor compares the user's user credentials (ID) with values ​​in a table. If the user's ID is not found in the table (e.g., D_ID), the user will not be able to access collaborative target 700. If another user (e.g., C_ID) is found in the table but has a "No" permission level, that user will only be provided with access equivalent to that access level. If another user (e.g., A_ID) with a "Yes" permission level is found in the table, that user will be provided with access to collaborative target 700 equivalent to that access level.

[0128] At box 1112, the processor identifies access level changes. The processor identifies access level changes for at least one other user. In one example, the host grants access to another user (e.g., B_ID) by sending an access permission change request, including the user ID to be changed and the new access level, to the processor 499 of server system 498 via a physical remote device. In another example, all users with a "yes" permission level must request an access level change from a user with another permission level. The access level change is identified by the processor upon receiving the request. In another example, the processor may identify the change based on the expiration of a collaborative period or a preset "presentation" time (e.g., based on a monitored timer). For example, in the case where user A and user B are preparing content for user C, user A and user B may initially have a "yes" permission level, and once the collaborative period has ended or the presentation time has arrived, the processor may automatically identify user C's access level change request.

[0129] At box 1114, the processor changes the corresponding access level. In response to the access level change, the processor also changes the corresponding access level of at least one other user. In response to the identified access level change (e.g., changing the B_ID permission level to "Yes"), the processor updates the table as shown in Table 2 below:

[0130] Table 2

[0131]

[0132]

[0133] Figure 12 This is a flowchart 1200 including steps of a method used in the collaborative application 925. The processor 499 of the server system 498 enables users to associate virtual collaborative targets 700 with geographical locations, and the processor 499 allows users to access virtual collaborative targets 700 based on the user's proximity to the geographical location.

[0134] At box 1202, during a collaborative session, processor 499 provides a user with editable access to collaborative target 700 via network 495. Processor 499 receives a request to join a collaborative session from one of multiple users via network 495. In this example, the request is a message and includes an authentication identifier. Processor 499 provides user access to collaborative target 700 when the user joins the collaborative session (e.g., by changing access permissions). Processor 499 presents collaborative target 700 to the user via network 495, for example, as a coloring on displays 180A-180B of goggles device 100 or display 580 of mobile device 401. In one example, the user cannot edit collaborative target 700 after the collaborative session. Processor 499 allows the user to access collaborative target 700 via network 495 using appropriate physical remote devices, such as goggles device 100 and mobile device 401. Processor 499 provides a time indicator for display on the physical remote device, indicating when the collaborative session ends. In one example, processor 499 automatically allows the requesting user to access the collaborative session. In another example, processor 499 can receive messages from the host of the collaboration session and then authorize access to the collaboration session.

[0135] At box 1204, processor 499 associates virtual content 702 received from the user via network 495 with cooperative target 700 during the cooperative period, such as Figure 7-Figure 9 As shown in the example, in this case, users facilitate joining a collaboration by adding and modifying virtual content 702 at a selected location of the collaborative target 700 using their physical remote devices, and this is displayed on the monitors of their respective devices, such as displays 180A-180B of the goggle device 100 and display 580 of the mobile device 401. In this example, the virtual content 702 may include a virtual image of the target and audio associated with the virtual image.

[0136] At box 1206, the user associates the virtual collaborative target 700 with a geographic location (such as defined using latitude and longitude coordinates). The geographic location can also be a GPS location, a location address, or some other form of location. The geographic location is chosen by one or more users (such as the host user or all users) to mutually agree on the selected geographic location. Proximity to a geographic location is defined as a location within a predetermined radius (10 yards in the example). The collaborative target 700 becomes accessible to the user when they approach the geographic location. In the example, as... Figure 7-Figure 9 As shown, the collaborative goal 700 has a theme, such as a time capsule, which includes virtual content 702 along with a timer contributed by the user. In this example, the user virtually hides the time capsule for the user to find, such as in Figure 13 Near tree 1212 shown.

[0137] At box 1208, when a user approaches a geographic location, processor 499 allows the user to access collaborative target 700. In this example, when a device approaches a geographic location, the user can access collaborative target 700 using a corresponding physical remote device. IMU 472 of goggle device 100 and IMU 572 of mobile device 401 determine the user's location, and this location is uploaded to server network 498 via network 495. In another example, the physical remote device may have a Global Positioning System (GPS) receiver, which determines when the device approaches a geographic location. When processor 499 of server network 498 determines that the physical remote device is approaching a geographic location, processor 499 sends authorization (such as a message) to the physical remote device and also downloads collaborative target 700 to the authorized device. The physical remote device then displays collaborative target 700 on a display of the device, such as on displays 180A-B of goggle device 100 and on touchscreen display 580 of mobile device 401.

[0138] Machine learning refers to algorithms that improve incrementally through experience. By processing a large number of different input datasets, machine learning algorithms can develop improved generalizations about a particular dataset, and then use those generalizations to produce accurate outputs or solutions when processing new datasets. More broadly, machine learning algorithms involve adjusting or changing one or more parameters in response to new experiences, thereby progressively improving the algorithm; similar to a learning process.

[0139] In the context of computer vision, mathematical models attempt to simulate tasks performed by the human visual system, with the goal of using computers to extract information from images and achieve accurate understanding of image content. Computer vision algorithms have been developed for various fields, including artificial intelligence and autonomous navigation, to extract and analyze data from digital images and videos.

[0140] Deep learning refers to a class of machine learning methods based on or modeling artificial neural networks. An artificial neural network is a computational system composed of multiple simple, highly interconnected processing elements (nodes) that process information through their dynamic state responses to external inputs. Large artificial neural networks can have hundreds or even thousands of nodes.

[0141] Convolutional Neural Networks (CNNs) are a type of neural network frequently used to analyze visual images, including digital photographs and videos. The connectivity patterns between nodes in a CNN are typically modeled after the human visual cortex is organized, comprising individual neurons arranged to respond to overlapping regions in the visual field. Neural networks suitable for the determination process described in this paper are based on one of the following architectures: VGG16, VGG19, ResNet50, Inception V3, Xception, or other CNN-compatible architectures.

[0142] In the machine learning example, at boxes 1008 and 1012, processor 432 uses a machine-trained algorithm called a hand feature model to determine whether a range of detected hand shapes roughly match a predefined hand gesture. Processor 432 is configured to access the hand feature model trained by machine learning and applies the hand feature model to identify and locate features of hand shapes in one or more frames of video data.

[0143] In one example embodiment, a trained hand feature model receives video data frames including detected hand shapes and abstracts the images in those frames into layers for analysis. Based on the trained hand feature model, the data in each layer is compared layer by layer with hand gesture data stored in a hand gesture library 480 until a good match is identified.

[0144] In one example, a convolutional neural network (CNN) is used to perform layer-by-layer image analysis. In the first convolutional layer, the CNN identifies learned features (e.g., hand landmarks, joint coordinate sets, etc.). In the second convolutional layer, the image is transformed into multiple images, where the learned features are highlighted in their respective sub-images. In the merging layer, the size and resolution of the images and sub-images are reduced to isolate portions of each image that include features of interest (e.g., possible hand shapes, possible finger joints). Values ​​from the images in the non-output layers are compared to classify the images in the frame. As used in this paper, classification refers to the process of classifying an image based on detected hand shapes using a trained model. For example, if a series of detected hand hand shapes match touch gestures stored in library 480, the image can be classified as a "touch action".

[0145] As described herein, any of the functions described for the goggle device 100, mobile device 401, and server system 498 may be embodied in one or more computer software applications or sets of programming instructions. According to some examples, “function,” “functions,” “application,” “instruction,” or “programming” are programs that execute functions defined in a program. One or more applications structured in different ways can be developed using different programming languages, such as object-oriented programming languages ​​(e.g., Objective-C, Java, or C++) or procedural programming languages ​​(e.g., C or assembly language). In specific examples, third-party applications (e.g., those using Android by entities from vendors different from those on a particular platform) may be used. TM or iOS TM Applications developed using a Software Development Kit (SDK) can be included in mobile operating systems such as iOS. TM ANDROID TM , Mobile software running on a mobile phone (Phone) or another mobile operating system. In this example, a third-party application may invoke API calls provided by the operating system to facilitate the functionality described herein.

[0146] Therefore, machine-readable media can take many forms of tangible storage media. Non-transitory storage media include, for example, optical discs or disks, any storage device such as any computer device, such as those used to implement client devices, media gateways, transcoders, etc., as shown in the accompanying drawings. Transitory storage media include dynamic memory, such as the main memory of such a computer platform. Tangible transmission media include coaxial cables, copper wires, and optical fibers, including wires containing buses within a computer system. Carrier transmission media can take the form of electrical or electromagnetic signals, or sound waves or light waves such as those generated during radio frequency (RF) and infrared (IR) data communications. Therefore, common forms of computer-readable media include, for example: floppy disks, flexible disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, DVDs or DVD-ROMs, any other optical media, punched card tapes, any other physical storage media with a perforated pattern, RAM, PROMs and EPROMs, FLASH-EPROMs, any other memory chips or cartridges, carrier waves for transmitting data or instructions, cables or links for transmitting such carrier waves, or any other media from which a computer can read programming code or data. Many of these forms of computer-readable media may involve carrying one or more sequences of instructions to a processor for execution.

[0147] Apart from what is immediately stated above, nothing that has been stated or shown is intended or should be construed as causing any contribution or equivalence to any part, step, feature, object, benefit, advantage, or similarity to the public, whether or not it is stated in the claims.

[0148] It should be understood that the terms and expressions used herein have their general meanings as assigned to their respective corresponding queries and fields of study, unless otherwise specified herein. Relational terms such as "first" and "second" may be used merely to distinguish one entity or action from another, without necessarily requiring or implying any actual such relationship or order between these entities or actions. The terms "comprises," "comprising," "includes," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, thing, or apparatus that includes or comprises a list of elements or steps includes not only those elements or steps, but may also include other elements or steps not expressly listed or inherent to such process, method, thing, or apparatus. Without further limitation, an element preceded by "an" or "a" does not exclude the presence of additional identical elements in the process, method, thing, or apparatus that includes that element.

[0149] Unless otherwise stated, any and all measurements, values, ratings, locations, amplitudes, sizes, and other specifications set forth in this specification (including in the following claims) are approximate and imprecise. Such quantities are intended to have a reasonable range consistent with the function they relate to and with the conventions of the field to which they belong. For example, unless expressly stated otherwise, parameter values, etc., may differ from the stated quantities or ranges by up to ±10%.

[0150] Furthermore, as can be seen from the above specific embodiments, for the purpose of simplification, various features are combined in various examples. The method of this disclosure should not be construed as reflecting an intention to require more features than are expressly stated in each claim. Rather, as reflected in the following claims, the subject matter to be protected lies in fewer features than all the features of any single disclosed example. Therefore, the following claims are thus incorporated into the detailed description, wherein each claim is independently claimed as a separate subject matter.

[0151] While the foregoing has described what is considered the best model and other examples, it should be understood that various modifications may be made therein, and the subject matter disclosed herein can be implemented in different forms and examples, and these can be applied to many applications, of which only a few have been described herein. The appended claims are intended to claim protection for any and all modifications and variations that fall within the true scope of this concept.

Claims

1. A system comprising a server having a processor configured to perform an interactive augmented reality method, the processor being configured to: Providing users with access to virtual collaborative objects; associating virtual content received from the user with the virtual collaboration target during a collaboration period; associating the virtual collaborative target with a geographic location; as well as Based on proximity to the geographic location, access to the virtual collaborative target is permitted.

2. The system according to claim 1, wherein: The processor is configured to: The geographic location is identified in response to input from at least one of the users.

3. The system according to claim 2, wherein: At least one of the users is all users.

4. The system of claim 2, further comprising a remote physical device having a camera, wherein: The remote physical device: Capture images of physical environments; identifying surfaces in the environment; recognizing a gesture in proximity to the surface; determining a physical location in response to the gesture approaching the surface; as well as The physical location is sent to a processor of the server, wherein the physical location is assigned as the geographic location.

5. The system according to claim 1, wherein: The virtual collaborative target is associated with the geographic location in response to input from one of the users, and wherein the processor is further configured to: In response to input from another of the users, the virtual collaboration target is associated with another geographic location.

6. The system according to claim 1, wherein: The processor is configured to determine proximity of a device to the geographic location and trigger access to the virtual collaboration target on the device.

7. The system according to claim 1, wherein: The processor is configured to allow a user to access the virtual collaboration target using a corresponding physical remote device.

8. The system according to claim 7, wherein: The processor is configured to provide the virtual collaboration target to the physical remote device.

9. The system according to claim 8, wherein: The processor is configured to provide a time indicator for display on a physical remote device of the user, the time indicator representing when the collaborative period ends.

10. An interactive augmented reality method performed by a processor, the method comprising: Providing users with access to virtual collaborative objects; associating virtual content received from the user with the virtual collaboration target during a collaboration period; associating the virtual collaborative target with a geographic location; as well as Based on proximity to the geographic location, access to the virtual collaborative target is permitted.

11. The method according to claim 10, further comprising: The geographic location is identified in response to input from at least one of the users.

12. The method according to claim 11, wherein: At least one of the users is all users.

13. The method of claim 11, further comprising a remote physical device having a camera, the remote physical device: Capture images of physical environments; identifying surfaces in the environment; recognizing a gesture in proximity to the surface; determining a physical location in response to the gesture approaching the surface; as well as The physical location is sent to a processor of a server, wherein the physical location is assigned as the geographic location.

14. The method according to claim 10, wherein: The virtual collaborative target is associated with the geographic location in response to input from one of the users, the method further comprising: In response to input from another of the users, the virtual collaboration target is associated with another geographic location.

15. The method according to claim 10, further comprising: Proximity of a device to the geographic location is determined and access to the virtual collaboration target on the device is triggered.

16. The method according to claim 10, further comprising: The user is allowed to access the virtual collaboration target using a corresponding physical remote device.

17. The method according to claim 16, further comprising: The virtual collaboration target is provided to the physical remote device.

18. The method according to claim 17, further comprising: A time indicator is provided for display on the physically remote device, the time indicator representing when the collaborative period ends.

19. A non-transitory computer readable medium storing program code, the program code, when executed by an electronic processor of an electronic device having a display, configuring the electronic device to: Providing users with access to virtual collaborative objects; associating virtual content received from the user with the virtual collaboration target during a collaboration period; associating the virtual collaborative target with a geographic location; as well as Based on proximity to the geographic location, access to the virtual collaborative target is permitted.

20. The non-transitory computer readable medium storing program code according to claim 19, wherein the program code, when executed by the electronic processor, further configures the electronic device to: The geographic location is identified in response to input from at least one of the users.

Images