Selective collaborative object access

Abstract

A collaboration session in which participants / users may selectively access the added virtual content. A participant (host) creates a new session and invites the participant to join. An invited participant receives an invitation to join a session. A session creator (i.e., host) and other approved participants may access the content of the session. The session identifies a new participant when the new participant joins the session and simultaneously notifies other participants in the session that the new participant is waiting for permission to access the added virtual content. The host or approved participant may set permission for the new participant to access the added virtual content.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. application serial number 17 / 900,783, filed on August 31, 2022, the contents of which are incorporated herein by reference in their entirety. Technical Field

[0003] Examples set forth in this disclosure relate to the field of virtual reality including electronic devices such as mobile devices and wearable devices such as eye-mounted devices. More specifically, but not by way of limitation, this disclosure describes a collaborative method with selective access. Background Art

[0004] Many types of computers and electronic devices available today, such as mobile devices (e.g., smartphones, tablets, and laptops), handheld devices, and wearable devices (e.g., smart glasses, digital eyewear, headsets, headgear, and head-mounted displays), include various cameras, sensors, wireless transceivers, input systems, and displays.

[0005] A graphical user interface allows a user to interact with displayed content, including virtual objects and graphical elements such as icons, task bars, list boxes, menus, buttons, and selection control elements (e.g., cursors, pointers, handles, and sliders).

[0006] Virtual reality (VR) technology generates a complete virtual environment including realistic images, which is sometimes presented on a VR headset or other head-mounted display. The VR experience allows the user to move around in the virtual environment and interact with virtual objects. Augmented reality (AR) is a VR technology that combines real objects in the physical environment with virtual objects and displays the combination to the user. The combined display gives the impression that the virtual objects are actually present in the environment, especially when the virtual objects appear and behave like real objects. Cross reality (XR) is generally understood as an umbrella term for systems that include or combine elements from AR, VR, and MR (mixed reality) environments.

[0007] Users of VR technology can use collaboration tools. Collaboration tools enable users to meet virtually in a collaboration session. During a collaboration session, users can communicate with each other in a virtual environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The features of the various examples described will be readily understood from the following detailed description with reference to the accompanying drawings. In the description and throughout the several views of the accompanying drawings, reference numerals are used with each element. When multiple similar elements are present, a single reference numeral may be assigned to the similar element, with the additional lower case letter referring to the specific element.

[0009] Unless otherwise indicated, the various elements shown in the figures are not drawn to scale. For clarity, the size of the various elements may be enlarged or reduced. These figures depict one or more embodiments and are presented only as examples and should not be construed as limiting. The drawings include the following figures:

[0010] Figure 1A is a side view of an example hardware configuration of an eye-mounted device suitable for use in an example collaborative system (right);

[0011] Figure 1B yes Figure 1A a partial cross-sectional perspective view of a right corner of an eye-mounted device depicting a right visible light camera and a circuit board;

[0012] Figure 1C yes Figure 1A A side view (left) of an example hardware configuration of an eye-mounted device of FIG. 1 , showing a left visible light camera;

[0013] Figure 1D yes Figure 1C a partial cross-sectional perspective view of a left corner of an eye-mounted device depicting a left visible light camera and a circuit board;

[0014] Figure 2A and 2B is a rear view of an example hardware configuration of an eye-mounted device used in an example collaborative system;

[0015] Figure 3 is a schematic diagram of a three-dimensional scene, a left original image captured by a left visible light camera, and a right original image captured by a right visible light camera;

[0016] Figure 4 is a functional block diagram of an example collaborative system including a wearable device (e.g., an eye-worn device) and a server system connected via various networks;

[0017] Figure 5 Is applicable to Figure 4 An illustration of an example hardware configuration for a mobile device of an example system;

[0018] Figure 6 is a schematic diagram for describing a user in an example environment that is simultaneously located and mapped;

[0019] Figure 7 is a perspective view of an example collaborative object in the form of a box that can be manipulated by hand;

[0020] Figure 8 is used to open Figure 7 A perspective view of an example first gesture associated with an opening gesture of a box depicted in FIG.

[0021] Fig. 9 Yes and for closing Figure 7 A perspective view of an example second gesture associated with the closing gesture of the box depicted in FIG.

[0022] Fig.10 is a flowchart listing the steps in an example collaborative method; and

[0023] Fig.11 is a flow chart listing the steps of an example selective collaborative object access method. DETAILED DESCRIPTION

[0024] A collaborative session (e.g., a virtual time capsule) in which access to collaborative objects and added virtual content is selectively provided to participants / users. In one example of a collaborative session, a participant (host) creates a new session and invites participants to join. The invited participants receive an invitation to join the session. The session creator (i.e., the host) and other approved participants can access the content of the session (e.g., which can be recorded using an application such as Lens Cloud Function; available from Snap, Inc. of Santa Monica, California). The session identifies the new participant when the new participant joins the session and simultaneously notifies the other participants in the session that the new participant is waiting for permission to access the added virtual content. The host or approved participants can set permissions for the new participant to access the added virtual content.

[0025] The following detailed description includes systems, methods, techniques, instruction sequences, and computer program products that illustrate the examples set forth in this disclosure. Many details and examples are included for the purpose of providing a thorough understanding of the disclosed subject matter and its related teachings. However, those skilled in the relevant art can understand how to apply the related teachings without these details. The various aspects of the disclosed subject matter are not limited to the specific devices, systems, and methods described, because the related teachings can be applied or practiced in various ways. The terms and nomenclature used herein are only for the purpose of describing specific aspects, and are not intended to be restrictive. In general, well-known instruction instances, protocols, structures, and techniques are not necessarily shown in detail.

[0026] As used herein, the terms "coupled" or "connected" refer to any logical, optical, physical, or electrical connection, including a link or the like, through which an electrical or magnetic signal generated or supplied by one system element is conducted to another coupled or connected system element. Unless otherwise specified, coupled or connected elements or devices are not necessarily directly connected to each other and may be separated by intermediate components, elements, or communication media, one or more of which can modify, manipulate, or carry electrical signals. The term "on" means supported directly by an element, or indirectly supported by an element through another element that is integrated into or supported by the element.

[0027] The term "proximal" is used to describe an item or part of an item that is near, adjacent to, or beside an object or person; or closer relative to other parts of the item, which may be described as "distal." For example, the end of an item closest to the object may be referred to as the proximal end, while the generally opposite end may be referred to as the distal end.

[0028] For purposes of illustration and discussion, orientations of eye-mounted devices, other mobile devices, associated components, and any other devices incorporating a camera, an inertial measurement unit, or such as those shown in any of the accompanying figures are given by way of example only. In operation, the eye-mounted device may be oriented in any other direction (e.g., upward, downward, sideways, or any other orientation) suitable for the particular application of the eye-mounted device. Additionally, as used herein, any directional terms, such as front, rear, inward, outward, toward, left, right, lateral, longitudinal, up, down, above, below, top, bottom, side, lateral, longitudinal, and diagonal, are used by way of example only and are not limiting with respect to the direction or orientation of any camera or inertial measurement unit constructed as otherwise described herein.

[0029] Advanced AR technologies, such as computer vision and object tracking, can be used to produce perceptually rich and immersive experiences. Computer vision algorithms extract three-dimensional data about the physical world from data captured in digital images or videos. Object recognition and tracking algorithms are used to detect objects in digital images or videos, estimate their orientation or pose, and track their movement over time. Recognizing and tracking hands and fingers in real time is one of the most challenging and processing-intensive tasks in the field of computer vision.

[0030] The term "pose" refers to the static position and orientation of an object at a particular moment. The term "gesture" refers to the active movement of an object (such as a hand) through a series of postures, sometimes to convey a signal or idea. The terms pose and gesture are sometimes used interchangeably in the fields of computer vision and augmented reality. As used herein, the terms "pose" or "gesture" (or variations thereof) are intended to include both poses and gestures; in other words, the use of one term does not exclude the other.

[0031] Additional objects, advantages and novel features of the examples will be set forth in part in the following description, and in part will become apparent to those skilled in the art upon examination of the following and the accompanying drawings, or may be learned by production or operation of the examples. The objects and advantages of the subject matter may be realized or attained by the methods, tools and combinations particularly pointed out in the appended claims.

[0032] Reference is now made in detail to the examples illustrated in the accompanying drawings and discussed below.

[0033] Figure 1Ais a side view (right) of an example hardware configuration of an eye-mounted device 100 including a touch-sensitive input device or touchpad 181, Figure 1C is a side view (left) of an example hardware configuration thereof. As shown, the touch pad 181 can have a subtle and not easily visible border; alternatively, the border can be clearly visible, or include a raised or otherwise tactile edge that provides feedback to the user about the position and border of the touch pad 181. In other embodiments, the eye-mounted device 100 can include a touch pad on the left side.

[0034] The surface of the touchpad 181 is configured to detect finger touches, taps, and gestures (e.g., moving touch) for use with a GUI displayed by the eye-mounted device on an image display to allow a user to navigate and select menu options in an intuitive manner, which enhances and simplifies the user experience.

[0035] Detecting finger input on the touch pad 181 can enable a variety of functions. For example, touching anywhere on the touch pad 181 can cause the GUI to display or highlight an item on the image display, which can be projected onto at least one of the optical components 180A, 180B. Double-clicking the touch pad 181 can select an item or icon. Sliding or swiping a finger in a particular direction (e.g., from front to back, back to front, top to bottom, or bottom to top) can cause an item or icon to slide or scroll in a particular direction; for example, move to the next item, icon, video, image, page, or slide. Sliding a finger in another direction can slide or scroll in the opposite direction; for example, move to the previous item, icon, video, image, page, or slide. The touch pad 181 can be almost anywhere on the eye-wearing device 100.

[0036] In one example, a recognized gesture of a single tap on touch pad 181 initiates selection or pressing of a graphical user interface element in an image presented on the image display of optical assembly 180A, 180B. Adjusting the image presented on the image display of optical assembly 180A, 180B based on the recognized gesture can be a primary action that selects or submits a graphical user interface element on the image display of optical assembly 180A, 180B for further display or execution.

[0037] As shown, the eye-wearing device 100 includes a left visible light camera 114A and a right visible light camera 114B. As further described herein, the two cameras 114A, 114B capture image information of a scene from two separate viewpoints. The two captured images can be used to project a three-dimensional display onto an image display for viewing with 3D glasses.

[0038] The eye-mounted device 100 includes a right optical assembly 180B having an image display to present an image, such as a depth image. Figure 1A and 1C As shown, the eye-mounted device 100 may include a plurality of visible light cameras 114A, 114B forming a passive three-dimensional camera (such as a stereo camera), wherein the right visible light camera 114B is located at the right corner 110B, as shown in FIG. Figure 1C -D, the left visible light camera 114A is located at the left corner 110A.

[0039] Left and right visible light cameras 114A, 114B are sensitive to wavelengths in the visible light range. Each of visible light cameras 114A, 114B has a different forward-facing field of view that overlaps to generate a three-dimensional depth image, e.g., left visible light camera 114A captures left field of view 111A, and right visible light camera 114B captures right field of view 111B. In general, a "field of view" is the portion of a scene visible through a camera at a particular position and orientation in space. Fields of view 111A and 111B have overlapping fields of view 304 ( Figure 3 ). When visible light camera captures an image, objects or object features outside field of view 111A, 111B are not recorded in the original image (e.g., a photograph or picture). The field of view describes the angular range or degree to which the image sensor of visible light camera 114A, 114B picks up electromagnetic radiation of a given scene in a captured image of the given scene. The field of view can be expressed as the angular size of the viewing cone (i.e., the viewing angle). The viewing angle can be measured horizontally, vertically, or diagonally.

[0040] In an example configuration, one or both visible light cameras 114A, 114B have a field of view of 100° and a resolution of 480 x 480 pixels. “Coverage angle” describes the angle of coverage of visible light camera 114A, 114B or infrared camera 410 (see FIG. Figure 2A ) is the range of angles that a lens can image effectively. Typically, a camera lens produces an image circle large enough to completely cover the camera's film or sensor, which may include some vignetting (e.g., the image gets darker toward the edge of the lens compared to the center). If the camera lens's coverage angle does not fill the sensor, the image circle will be visible, often with strong vignetting toward the edge of the lens, and the effective viewing angle will be limited to the coverage angle.

[0041] Examples of such visible light cameras 114A, 114B include digital camera elements such as high-resolution complementary metal oxide semiconductor (CMOS) image sensors and digital VGA cameras (video graphics arrays) capable of 480p (e.g., 640 x 480 pixels), 720p, 1080p, or higher resolutions. Other examples include visible light cameras 114A, 114B that can capture high-definition (HD) video at high frame rates (e.g., 30 to 60 frames per second or more) and store recordings at a resolution of 1216 by 1216 pixels (or higher).

[0042] The eye-wearing device 100 can capture image sensor data from the visible light cameras 114A, 114B and geographic location data digitized by the image processor for storage in a memory. The visible light cameras 114A, 114B capture corresponding left and right raw images in a two-dimensional spatial domain, which includes a pixel matrix on a two-dimensional coordinate system, which includes an X-axis for horizontal positioning and a Y-axis for vertical positioning. Each pixel includes a color attribute value (e.g., a red pixel light value, a green pixel light value, or a blue pixel light value); and a positioning attribute (e.g., an X-axis coordinate and a Y-axis coordinate).

[0043] To capture stereoscopic images for later display as a three-dimensional projection, image processor 412 ( Figure 4 ) may be coupled to visible light cameras 114A, 114B to receive and store visual image information. Image processor 412 or another processor controls the operation of visible light cameras 114A, 114B to act as a stereo camera that simulates human binocular vision, and may add a timestamp to each image. The timestamps on each pair of images allow the images to be displayed together as parts of a three-dimensional projection. Three-dimensional projection creates an immersive, lifelike experience that is desirable in a variety of environments, including virtual reality (VR) and video games.

[0044] Figure 1B yes Figure 1A FIG. 1 is a cross-sectional perspective view of the right corner 110B of the eye-mounted device 100 depicting the right visible light camera 114B and circuit board of the camera system. Figure 1C yes Figure 1A 1 is a side view (left) of an example hardware configuration of the eye-mounted device 100, showing the left visible light camera 114A of the camera system. Figure 1D yes Figure 1C 1 is a cross-sectional perspective view of the left corner 110A of the eye-mounted device, depicting the left visible light camera 114A and circuit board of the three-dimensional camera.

[0045] like Figure 1BAs shown in the example of , the eye-mounted device 100 includes a right visible light camera 114B and a circuit board 140B, which can be a flexible printed circuit board (PCB). The right hinge 126B connects the right corner 110B to the first temple 125B of the eye-mounted device 100. In some examples, the right visible light camera 114B, the flexible PCB 140B, or other electrical connectors or contacts can be located on the right temple 125B or the right hinge 126B.

[0046] The left visible light camera 114A is constructed and placed substantially similarly to the right visible light camera 114B, except that it is connected and coupled to the left side 170A. The left hinge 126B connects the left corner 110A to the left temple 125A of the eye-wearing device 100. In some examples, components of the left visible light camera 114A, the flexible PCB 140A, or other electrical connectors or contacts can be located on the left temple 125A or the left hinge 126A.

[0047] Each of the left corner portion 110A and the right corner portion 110B includes a corner portion body 190 and a corner portion cap. Figure 1B and Figure 1D Disposed within the left corner 110A and the right corner 110B are various interconnected circuit boards 140A and 140B (such as PCBs and flexible PCBs), which include controller circuits for the left visible light camera 114A and the right visible light camera 114B, one or more microphones, low power wireless circuitry (e.g., for communicating via Bluetooth TM The corners 110A, 110B may be integrated into the frame 105 on the respective lateral sides 170A, 170B (as shown), or implemented as separate components attached to the frame 105 on the respective sides 170A and 170B. Alternatively, the corners 110A, 110B may be integrated into the temples 125A, 125B attached to the frame 105.

[0048] The left visible light camera 114A and the right visible light camera 114B are coupled to or disposed on respective flexible PCBs 140A and 140B and are covered by visible light camera cover lenses that are aimed through one or more openings formed in the frame 105. For example, the left and right edges 107A and 107B of the frame 105 are connected to the left and right corners 110A and 110B and include openings for visible light camera cover lenses. The frame 105 includes a front side configured to face outward and away from the user's eyes. The opening for the visible light camera cover lens is formed on or outwardly upwardly through the front side of the frame 105. In this example, each of the left and right visible light cameras 114A and 114B has a line of sight or perspective associated with the respective left and right eyes of the user of the eyewear device 100 with respective outwardly facing fields of view 111A and 111B. The visible light camera cover lens may also be adhered to the front side or outward facing surface of the right corner 110B where the opening is formed with an outward facing coverage angle, but in a different outward direction. The coupling may also be indirect via intervening components.

[0049] Figure 2A and 2B 1 is a perspective view of an example hardware configuration of an eye-mounted device 100 including two different types of image displays, viewed from the rear. The size and shape of the eye-mounted device 100 is configured to be worn by a user. The form of glasses is shown as an example. The eye-mounted device 100 can take other forms and can be combined with other types of frames; for example, a headband, headphones, or a helmet.

[0050] In the eyeglass example, the eyewear device 100 includes a frame 105 including a left rim 107A connected to a right rim 107B via a bridge 106 adapted to be supported by the user's nose. The left rim 107A and the right rim 107B include respective rims 175A, 175B that hold respective optical elements 180A, 180B, such as lenses and display devices. As used herein, the term "lens" means a transparent or translucent glass or plastic sheet having curved and flat surfaces that cause light to converge or diverge or cause little or no convergence or divergence.

[0051] Although shown with two optical elements 180A, 180B, the eye-mounted device 100 may include other arrangements, such as a single optical element (or it may not include any optical elements 180A, 180B), depending on the application or intended user of the eye-mounted device 100. As further shown, the eye-mounted device 100 includes a left corner 110A adjacent to the left lateral side 170A of the frame 105 and a right corner 110B adjacent to the right lateral side 170B of the frame 105.

[0052] In one example, the image display of the optical components 180A, 180B includes an integrated image display. Figure 2A As shown, each optical assembly 180A, 180B includes a suitable display matrix 177, such as a liquid crystal display (LCD), an organic light emitting diode (OLED) display, or any other such display. Each optical assembly 180A, 180B also includes one or more optical layers 176, which may include lenses, optical coatings, prisms, reflectors, waveguides, optical strips, and other optical components in any combination. Optical layers 176A, 176B...176N (such as Figure 2A The optical layers 176A-176N (shown as 176A-176N in FIG. 105 ) may include prisms having a suitable size and configuration and including a first surface for receiving light from the display matrix and a second surface for emitting light toward the user's eyes. The prisms of the optical layers 176A-176N extend over all or at least a portion of the corresponding lens rings 175A, 175B formed in the left lens edge 107A and the right lens edge 107B to allow the user to see the second surface of the prisms when the user's eyes look through the corresponding left lens edge 107A and the right lens edge 107B. The first surfaces of the prisms of the optical layers 176A-176N face upward from the frame 105, and the display matrix 177 overlies the prisms so that photons and light emitted by the display matrix 177 strike the first surface. The prisms may be sized and shaped so that light is refracted within the prisms and directed toward the user's eyes by the second surfaces of the prisms of the optical layers 176A-176N. In this regard, the second surface of the prisms of the optical layers 176A-176N can be convex to direct light toward the center of the eye. The prisms can optionally be sized and shaped to magnify the image projected by the display matrix 177, and the light travels through the prisms so that the image viewed from the second surface is larger than the image emitted from the display matrix 177 in one or more dimensions.

[0053] In one example, the optical layers 176A-176N may include a transparent LCD layer (holding the lens open) unless and until a voltage is applied that renders the layer opaque (closing or blocking the lens). The image processor 412 on the eye-mounted device 100 may be programmed to apply a voltage to the LCD layer, thereby creating an active shutter system that makes the eye-mounted device 100 suitable for viewing visual content when displayed as a three-dimensional projection. Technologies other than LCD may be used for active shutter modes, including other types of reactive layers that respond to a voltage or another type of input.

[0054] In another example, the image display device of the optical assembly 180A, 180B includes Figure 2BEach optical assembly 180A, 180B includes a laser projector 150, which is a three-color laser projector using a scanning mirror or a galvanometer. During operation, a light source (such as a laser projector 150) is disposed in or on one of the temples 125A, 125B of the eye-mounted device 100. In this example, the optical assembly 180B includes one or more optical strips 155A, 155B ... 155N (in Figure 2B 155A-155N), these optical strips are spaced across the width of the lens of each optical assembly 180A, 180B or across the depth of the lens between the front and back surfaces of the lens.

[0055] As photons projected by the laser projector 150 travel through the lens of each optical component 180A, 180B, the photons encounter optical strips 155A-155N. When a particular photon encounters a particular optical strip, the photon is turned toward the user's eye, or passed to the next optical strip. A combination of modulation of the laser projector 150 and modulation of the optical strips can control specific photons or light beams. In an example, the processor controls the optical strips 155A-155N by initiating a mechanical signal, an acoustic signal, or an electromagnetic signal. Although shown as having two optical components 180A, 180B, the eye-mounted device 100 may include other arrangements (such as a single or three optical components), or each optical component 180A, 180B may have a different arrangement depending on the application or intended user of the eye-mounted device 100.

[0056] In another example, Figure 2B The eye-mounted device 100 shown may include two projectors, a left projector (not shown) and a right projector 150. The left optical assembly 180A may include a left display matrix 177 or a left optical strip group (not shown) configured to interact with light from the left projector. Similarly, the right optical assembly 180B may include a right display matrix (not shown) or a right optical strip group 155A, 155B, ... 155N configured to interact with light from the right projector 150. In this example, the eye-mounted device 100 includes a left display and a right display.

[0057] Figure 3306, a left raw image 302A captured by a left visible light camera 114A, and a right raw image 302B captured by a right visible light camera 114B. As shown, the left field of view 111A can overlap with the right field of view 111B. The overlapping field of view 304 represents the portion of the image captured by the two cameras 114A, 114B. When referring to the field of view, the term "overlap" means that the pixel matrices in the generated raw images overlap by thirty percent (30%) or more. "Substantially overlap" means that the pixel matrices in the generated raw images or infrared images of the scene overlap by fifty percent (50%) or more. As described herein, the two raw images 302A, 302B can be processed to include a timestamp, which allows the images to be displayed together as parts of a three-dimensional projection.

[0058] To capture stereo images, such as Figure 3 As shown, a pair of raw red, green, and blue (RGB) images of a real scene 306 are captured at a given moment—a left raw image 302A captured by the left camera 114A and a right raw image 302B captured by the right camera 114B. When the pair of raw images 302A, 302B are processed (e.g., by the image processor 412), a depth image is generated. The generated depth image can be viewed on the optical components 180A, 180B of the eye-wearing device, on another display (e.g., the image display 580 on the mobile device 401), or on a screen.

[0059] The generated depth image is in a three-dimensional spatial domain and may include a matrix of vertices on a three-dimensional position coordinate system including an X-axis for horizontal positioning (e.g., length), a Y-axis for vertical positioning (e.g., height), and a Z-axis for depth (e.g., distance). Each vertex may include a color attribute (e.g., a red pixel light value, a green pixel light value, or a blue pixel light value); a positioning attribute (e.g., an X position coordinate, a Y position coordinate, and a Z position coordinate); a texture attribute; a reflectivity attribute; or a combination thereof. The texture attribute quantifies the perceived texture of the depth image, such as the spatial arrangement of color or intensity in a vertex region of the depth image.

[0060] Figure 44 is a functional block diagram of an example collaboration system 400, which includes a wearable device (e.g., eye-mounted device 100), a mobile device 401, and a server system 498 connected via various networks 495 (such as the Internet). The server system 498 can be one or more computing devices as a component of a service or network computing system, for example, including a processor, a memory, and a network communication interface to communicate with the eye-mounted device 100 and the mobile device 401 through the network 495. The server system 498 includes a server processor 499 that can be configured to host a collaboration session. The functions of the eye-mounted device 100 or the mobile device 401 described herein (such as collaboration processing and providing collaboration objects to users) can be performed by the processor 499 of the server system 498.

[0061] As described herein, the eye-wearing device 100 includes one or more visible light cameras 114A, 114B that capture still images, video images, or still and video images. The cameras 114A, 114B can have direct memory access (DMA) to the high-speed circuit system 430 and act as stereo cameras. The cameras 114A, 114B can be used to capture initial depth images that can be rendered into three-dimensional (3D) models that are texture mapped images of red, green, and blue (RGB) imaged scenes. The device 100 may also include a depth sensor 213 that uses infrared signals to estimate the location of an object relative to the device 100. In some examples, the depth sensor includes one or more infrared emitters 215 and one or more infrared cameras 410.

[0062] The eye-mounted device 100 also includes two image displays for each optical assembly 180A, 180B (one associated with the left side 170A and one associated with the right side 170B). The eye-mounted device 100 also includes an image display driver 442, an image processor 412, a low-power circuit system 420, and a high-speed circuit system 430. The image display of each optical assembly 180A, 180B is used to present an image, including a still image, a video image, or a still and video image. The image display driver 442 is coupled to the image display of each optical assembly 180A, 180B to control the display of the image.

[0063] The eye-mounted device 100 also includes one or more microphones (not shown) and one or more speakers 413 (e.g., one associated with the left side of the eye-mounted device and another associated with the right side of the eye-mounted device). The speakers 413 may be incorporated into the frame 105, the temples 125, or the corners 110 of the eye-mounted device 100. The one or more speakers 413 are driven by an audio processor 414 and an audio driver 415 under the control of the low-power circuit system 420, the high-speed circuit system 430, or both. The speakers 413 are used to present audio signals including, for example, a beat track. The audio processor 414 is coupled to the microphone and the speaker 413 to control the corresponding capture and presentation of sound.

[0064] Figure 4 The components of the eyewear device 100 shown in the figure are located on one or more circuit boards, such as a printed circuit board (PCB) or a flexible printed circuit (FPC) located in the frame or temples. Alternatively or additionally, the depicted components can be located in the corners, rims, hinges, or bridge of the eyewear device 100.

[0065] like Figure 4 As shown, the high-speed circuit system 430 includes a high-speed processor 432, a memory 434, and a high-speed wireless circuit system 436. In this example, an image display driver 442 is coupled to the high-speed circuit system 430 and operated by the high-speed processor 432 to drive the left image display and the right image display of each optical assembly 180A, 180B. The high-speed processor 432 can be any processor capable of managing high-speed communications and operations of any general-purpose computing system required by the eye-wearing device 100. The high-speed processor 432 includes the processing resources required to manage high-speed data transmission on a high-speed wireless connection 437 to a wireless local area network (WLAN) using the high-speed wireless circuit system 436.

[0066] In some examples, the high-speed processor 432 executes an operating system, such as a LINUX operating system or other such operating system of the eye-mounted device 100, and the operating system is stored in the memory 434 for execution. In addition to any other duties, the high-speed processor 432 that executes the software architecture of the eye-mounted device 100 is also used to manage data transmission with the high-speed wireless circuit system 436. In some examples, the high-speed wireless circuit system 436 is configured to implement the Institute of Electrical and Electronics Engineers (IEEE) 802.11 communication standard, also referred to as Wi-Fi herein. In other examples, the high-speed wireless circuit system 436 can also implement other high-speed communication standards.

[0067] The low-power circuit system 420 includes a low-power processor 422 and a low-power wireless circuit system 424. The low-power wireless circuit system 424 and the high-speed wireless circuit system 436 of the eye-wearing device 100 may include a short-range transceiver (Bluetooth TM or Bluetooth Low Energy (BLE)) and a wireless wide area network, local area network, or wide area network transceiver (e.g., cellular or Wi-Fi). The mobile device 401 (including a transceiver that communicates via a low power wireless connection 425 and a high speed wireless connection 437) can be implemented using details of the architecture of the eye-wearing device 100, as can other elements of the network 495.

[0068] The memory 434 includes any storage device capable of storing various data and applications, including, among other things, camera data generated by the left visible light camera 114A, the right visible light camera 114B, one or more infrared cameras 410, the image processor 412, and images generated by the image display driver 442 for display on the image display of each optical assembly 180A, 180B. Although the memory 434 is shown as being integrated with the high-speed circuit system 430, in other examples, the memory 434 can be a separate, separate element of the eye-mounted device 100. In some such examples, electrical routing lines can provide connections from the image processor 412 or the low-power processor 422 to the memory 434 through a chip including the high-speed processor 432. In other examples, the high-speed processor 432 can manage the addressing of the memory 434 so that the low-power processor 422 will start the high-speed processor 432 any time a read or write operation involving the memory 434 is required.

[0069] like Figure 4 As shown, the high-speed processor 432 of the eye-mounted device 100 can be coupled to the camera system (visible light cameras 114A, 114B), the image display driver 442, the user input device 491 and the memory 434.

[0070] The output components of the eyewear device 100 include visual elements, such as Figure 2A and 2B4. The eye-mounted device 100 may include an indicator facing the user (e.g., an LED, a speaker 413, or a vibration actuator) or a signal facing outward (e.g., an LED, a speaker 413). The image display of each optical component 180A, 180B is driven by an image display driver 442. In some example configurations, the output components of the eye-mounted device 100 also include additional indicators, such as an audible element (e.g., a speaker 413), a tactile component (e.g., an actuator for generating tactile feedback, such as a vibration motor), and other signal generators. For example, the device 100 may include a set of indicators facing the user and a set of signals facing outward. The set of indicators facing the user is configured to be seen or otherwise sensed by the user of the device 100. For example, the device 100 may include an LED display positioned so that it can be seen by a user, one or more speakers 413 positioned to produce sounds that the user can hear, or an actuator that provides tactile feedback that the user can feel. The outwardly facing set of signals is configured to be seen or otherwise sensed by an observer near the device 100. Similarly, the device 100 may include an LED, a speaker, or an actuator that is configured and positioned to be sensed by an observer.

[0071] The input components of the eye-mounted device 100 may include alphanumeric input components (e.g., a touch screen or touchpad configured to receive alphanumeric input, an optical keyboard, or other alphanumeric configured element), a point-based input component (e.g., a mouse, a touchpad, a trackball, a joystick, a motion sensor, or other directional instrument), a tactile input component (e.g., a button switch, a touch screen or touchpad or other tactile configured element that senses the location, force, or location and force of a touch or touch gesture), a visual input component (e.g., a camera 114 / 420), an audio input component (e.g., a microphone), and the like. The mobile device 401 and the server system 498 may include alphanumeric, point-based, tactile, audio, and other input components.

[0072] In some examples, the eye-wearing device 100 includes a collection of motion sensing components referred to as an inertial measurement unit 472. The motion sensing components can be micro-electromechanical systems (MEMS) with microscopic moving parts (usually parts small enough to be microchips). In some example configurations, the inertial measurement unit (IMU) 472 includes an accelerometer, a gyroscope, and a magnetometer. The accelerometer senses the linear acceleration of the device 100 relative to three orthogonal axes (x, y, z) (including acceleration due to gravity). The gyroscope senses the angular velocity of the device 100 around three rotational axes (pitch, roll, yaw). The accelerometer and gyroscope can together provide positioning, orientation, and motion data about the device relative to six axes (x, y, z, pitch, roll, yaw). The magnetometer (if present) senses the heading of the device 100 relative to magnetic north. The positioning of the device 100 can be determined by a position sensor (such as a GPS unit), one or more transceivers for generating relative positioning coordinates, an altitude sensor or a barometer, and other orientation sensors. Such positioning system coordinates may also be received from mobile device 401 over wireless connections 425 , 437 via low power wireless circuitry 424 or high speed wireless circuitry 436 .

[0073] The IMU 472 may include or cooperate with a digital motion processor or program that collects raw data from the components and calculates a number of useful values ​​regarding the location, orientation, and motion of the device 100. For example, the acceleration data collected from the accelerometers may be integrated to obtain the velocity relative to each axis (x, y, z); and integrated again to obtain the location of the device 100 (in a linear coordinate system x, y, and z). The angular velocity data from the gyroscopes may be integrated to obtain the location of the device 100 (in a spherical coordinate system). The program for calculating these useful values ​​may be stored in the memory 434 and executed by the high-speed processor 432 of the eye-mounted device 100.

[0074] The eye-mounted device 100 may optionally include additional peripheral sensors, such as biometric sensors, application-specific sensors, or display elements integrated with the eye-mounted device 100. For example, the peripheral device elements may include any I / O components including output components, motion components, positioning components, or any other such components described herein. For example, the biometric sensors may include components for detecting expressions (e.g., hand expressions, facial expressions, vocal expressions, body postures, or eye tracking), measuring bio-signals (e.g., blood pressure, heart rate, body temperature, sweat, or brain waves), identifying people (e.g., identification based on voice, retinal film, facial features, fingerprints, or electrical bio-signals (such as electroencephalogram data)), etc.

[0075] The mobile device 401 may be a smartphone, tablet, laptop, access point, or any other such device capable of connecting to the eye-mounted device 100 using a low-power wireless connection 425 and a high-speed wireless connection 437. The mobile device 401 is connected to a server system 498 and a network 495. The network 495 may include any combination of wired and wireless connections.

[0076] like Figure 4 As shown, the collaboration system 400 shown includes a computing device, such as a mobile device 401, coupled to the eye-wearing device 100 via a network. The collaboration system 400 includes a memory for storing instructions and a processor for executing instructions. The instructions of the collaboration system 400 are executed by the processor 432 to configure the eye-wearing device 100 to collaborate with the mobile device 401. The collaboration system 400 can utilize the memory 434 of the eye-wearing device 100 or the memory elements 540A, 540B, 540C ( Figure 5 ). In addition, the collaboration system 400 can utilize the processor elements 432, 422 of the eye-wearing device 100 or the central processing unit (CPU) 540 ( Figure 5 ). In addition, the collaboration system 400 can also utilize memory and processor elements of the server system 498. In this regard, the memory and processing functions of the collaboration system 400 can be shared or distributed between the eye-wearing device 100, the mobile device 401, and the processors and memories of the server system 498 to implement the functions described herein.

[0077] In some example embodiments, the memory 434 includes or is coupled to a gesture library 480, as described herein. In some embodiments, the process of detecting a hand shape or gesture involves comparing pixel-level data in one or more frames of video data captured by the eyewear 100 or mobile device 401 to the hand shapes and gestures stored in the library 480 until a good match is found. A gesture can be a static gesture that can be detected in one or a few frames of data, or a dynamic gesture that can be detected over the course of two or more frames of data.

[0078] In some example embodiments, the memory 434 also includes an element animation application 910, a positioning system 915, an image processing system 920, and a collaboration application 925. In the collaboration system 400 where the camera is capturing a frame of video data, the element animation application 910 configures the processor 432 to control the movement of a series of virtual items 700 on the display in response to detecting one or more inputs (e.g., IMU data, captured images, and hand shapes or gestures). The positioning system 915 configures the processor 432 to obtain positioning data for determining the position of the eye-wearing device 100 relative to the physical environment. The positioning data can be derived from a series of images, an IMU unit 472, a GPS unit, or a combination thereof. The image processing system 920 configures the processor 432 to collaborate with the image display driver 442 and the image processor 412 to present the captured image on the display of the optical assembly 180A, 180B. The collaboration application 925 configures the processor 432 to implement the collaboration functions described herein.

[0079] Figure 5 is a high-level functional block diagram of an example mobile device 401. Mobile device 401 includes flash memory 540A that stores programs to be executed by CPU 540 to perform all or part of the functionality described herein.

[0080] Mobile device 401 may include a camera 570 including at least two visible light cameras (e.g., first and second visible light cameras with overlapping fields of view) or at least one visible light camera and a depth sensor with substantially overlapping fields of view. Mobile device 401 may additionally include a speaker 571. Flash memory 540A may also include a plurality of images or videos generated via camera 570.

[0081] As shown in the figure, the mobile device 401 includes an image display 580, a mobile display driver 582 for controlling the image display 580, and a display controller 584. Figure 5 In the example of , the image display 580 includes a user input layer 591 (eg, a touch screen) that is layered over or otherwise integrated into a screen used by the image display 580 .

[0082] Examples of touch screen mobile devices that may be used include, but are not limited to, smartphones, personal digital assistants (PDAs), tablet computers, laptop computers, or other portable devices. However, the structure and operation of touch screen devices are provided by way of example; the subject technology described herein is not intended to be limited thereto. Therefore, for the purposes of this discussion, Figure 5A block diagram is provided of an example mobile device 401 having a user interface that includes a touch screen input layer 591 for receiving input (via touch, multi-touch, or gestures, etc., by a hand, stylus, or other tool), a camera 570 for capturing images of objects (including a user's hand and potentially virtual content), and an image display 580 for displaying content.

[0083] like Figure 5 As shown, the mobile device 401 includes at least one digital transceiver (XCVR) 510, such as a WWAN XCVR, for digital wireless communication via a wide area wireless mobile communication network. The mobile device 401 also includes additional digital or analog transceivers, such as for short-range network communication (such as via NFC, VLC, DECT, ZigBee, Bluetooth TM For example, the short-range XCVR 520 may take the form of any available two-way wireless local area network (WLAN) transceiver compatible with one or more standard communication protocols implemented in a wireless local area network, such as one of the Wi-Fi standards under the IEEE 802.11 standard.

[0084] To generate location coordinates for positioning the mobile device 401, the mobile device 401 may include a global positioning system (GPS) receiver. Alternatively or additionally, the eyewear device 100 or the mobile device 401 may utilize any one or both of the short-range XCVR 520 and the WWAN XCVR 510 to generate location coordinates for positioning. For example, based on a cellular network, Wi-Fi, or Bluetooth TM The positioning system can generate very accurate position coordinates, especially when used in combination. Such position coordinates can be sent between the eye-wearing device 100 or the mobile device 401 via the XCVR 510, 520 through one or more network connections.

[0085] In some examples, the mobile device 401 includes a collection of motion sensing components referred to as an inertial measurement unit (IMU) 572 for sensing the position, orientation, and motion of the client device 401. The motion sensing components can be micro-electromechanical systems (MEMS) with microscopic moving parts (usually parts small enough to be microchips). In some example configurations, the inertial measurement unit (IMU) 572 includes an accelerometer, a gyroscope, and a magnetometer. The accelerometer senses the linear acceleration (including acceleration due to gravity) of the client device 401 relative to three orthogonal axes (x, y, z). The gyroscope senses the angular velocity of the client device 401 around three rotational axes (pitch, roll, yaw). The accelerometer and gyroscope can together provide data about the position, orientation, and motion of the device relative to six axes (x, y, z, pitch, roll, yaw). The magnetometer (if present) senses the heading of the client device 401 relative to magnetic north.

[0086] The IMU 572 may include or cooperate with a digital motion processor or program that collects raw data from the components and calculates a number of useful values ​​regarding the location, orientation, and motion of the client device 401. For example, the acceleration data collected from the accelerometers may be integrated to obtain the velocity relative to each axis (x, y, z); and integrated again to obtain the location of the client device 401 (in a linear coordinate system x, y, and z). The angular velocity data from the gyroscopes may be integrated to obtain the location of the client device 401 (in a spherical coordinate system). The program for calculating these useful values ​​may be stored in one or more memory elements 540A, 540B, 540C and executed by the CPU 540 of the client device 401.

[0087] The transceivers 510, 520 (i.e., network communication interfaces) conform to one or more of the various digital wireless communication standards used by modern mobile networks. Examples of WWAN transceivers 510 include, but are not limited to, transceivers configured to operate in accordance with Code Division Multiple Access (CDMA) and Third Generation Partnership Project (3GPP) network technologies (including, for example, but not limited to, 3GPP Type 2 (or 3GPP2) and LTE, sometimes referred to as "4G"). For example, transceivers 510, 520 provide two-way wireless communication of information (including digitized audio signals, still images and video signals, web page information for display, and network-related input), as well as various types of mobile message communications to / from mobile device 401.

[0088] The mobile device 401 also includes a microprocessor that serves as a central processing unit (CPU); Figure 5540 in . A processor is a circuit having elements constructed and arranged to perform one or more processing functions (typically various data processing functions). Although discrete logic components may be used, these examples utilize components that form a programmable CPU. For example, a microprocessor includes one or more integrated circuit (IC) chips that contain electronic components to perform the functions of a CPU. For example, CPU 540 may be based on any known or available microprocessor architecture, such as a reduced instruction set computing (RISC) using an ARM architecture, as commonly used in today's mobile devices and other portable electronic devices. Of course, other arrangements of processor circuits may be used to form CPU 540 or processor hardware in smartphones, laptops, and tablets.

[0089] CPU 540 functions as a programmable host controller of mobile device 401 by configuring mobile device 401 to perform various operations (e.g., according to instructions or programming executable by CPU 540). For example, such operations may include various general operations of the mobile device, as well as operations related to programming of applications on the mobile device. Although processors can be configured using hardwired logic, typical processors in mobile devices are general-purpose processing circuits that are configured by executing programming.

[0090] The mobile device 401 includes a memory or storage system for storing programming and data. In this example, the memory system may include flash memory 540A, random access memory (RAM) 540B, and other memory components 540C as needed. RAM 540B is used as short-term storage for instructions and data processed by CPU 540, such as working data processing memory. Flash memory 540A generally provides long-term storage.

[0091] Thus, in the example of mobile device 401, flash memory 540A is used to store programming or instructions for execution by CPU 540. Depending on the type of device, mobile device 401 stores and runs a mobile operating system through which specific applications are executed. Examples of mobile operating systems include Google Android, Apple iOS (for iPhone or iPad devices), Windows Mobile, Amazon Fire OS, RIM BlackBerry OS, or similar systems.

[0092] like Figure 5As shown, the CPU 540 of the mobile device 401 can be coupled to the camera system 570, the mobile display driver 582, the user input layer 591, and the memory 540A. The components and functions of the eye-mounted device 100 described herein can be incorporated into the mobile device 401. Likewise, the components and functions of the mobile device 401 described herein can be incorporated into the eye-mounted device 100.

[0093] The processor 432 within the eye-mounted device 100 or the processor 540 within the mobile device 401 can build a map of the environment surrounding the respective device, determine the location of the device within the mapped environment, and determine the relative positioning of the device to one or more objects in the mapped environment. The processor 432 / 540 can build the map and determine the position and positioning information using a conventional simultaneous localization and mapping (SLAM) algorithm applied to data received from one or more sensors. The sensor data includes images received from one or both of the cameras 114A, 114B or one or more cameras 570, one or more distances received from a laser rangefinder, positioning information received from a GPS unit, motion and acceleration data received from an IMU 472 / 572, or a combination of data from such sensors or from other sensors that provide data that helps determine positioning information.

[0094] In the context of augmented reality, SLAM algorithms are used to build and update a map of the environment while tracking and updating the position of the device (or user) in the mapped environment. The mathematical solution can be approximated using various statistical methods (such as particle filters, Kalman filters, extended Kalman filters, and covariance intersection). In a system including a high-definition (HD) camera that captures video at a high frame rate (e.g., 30 frames per second), the SLAM algorithm updates the map and object position at least as frequently as the frame rate; in other words, the map and positioning are calculated and updated 30 times per second.

[0095] Figure 6 An example physical environment 600 and elements useful when using SLAM algorithms and other types of tracking applications (e.g., natural feature tracking (NFT)) are depicted. Although the following example is provided with reference to the eye-wearing device 100, the example can be implemented in a similar manner in the mobile device 401. A user 602 of the eye-wearing device 100 is present in the example physical environment 600 (in Figure 6, the example physical environment is an interior room). The processor 432 of the eye-mounted device 100 uses the captured images to determine the position of the eye-mounted device 100 relative to one or more objects 604 within the environment 600, constructs a map of the environment 600 using a coordinate system (x, y, z) of the environment 600, and determines its position within the coordinate system. In addition, the processor 432 determines the head pose (roll, pitch, and yaw) of the eye-mounted device 100 in the environment by using two or more location points (e.g., three location points 606a, 606b, and 606c) associated with a single object 604a, or by using one or more location points 606 associated with two or more objects 604a, 604b, 604c. The processor 432 of the eye-mounted device 100 may position a virtual object 608 (such as Figure 6 The buttons shown) are positioned within the environment 600 for viewing during an augmented reality experience (such as a collaborative augmented reality experience) in which each user has a corresponding augmented reality device (e.g., eye-mounted device 100 or mobile device 401).

[0096] In some examples, the positioning system 915 associates a virtual marker 610a with a virtual object 608 in the physical environment 600. In augmented reality, markers are registered at locations in the environment to assist the device in the task of tracking and updating the locations of users, devices, and objects (virtual and physical) in the mapped environment. Markers are sometimes registered to high-contrast physical objects, such as relatively dark objects mounted on a light-colored wall, such as a framed picture 604a, to assist cameras and other sensors in the task of detecting the marker. Markers can be pre-assigned, or can be assigned by the eye-mounted device 100 when entering the environment.

[0097] The marker may be encoded with information or otherwise linked to information. The marker may include location information, a physical code (such as a barcode or QR code; visible or hidden to the user), or a combination thereof. A set of data associated with the marker is stored in the memory 434 of the eye-wearing device 100. The set of data includes information about the marker 610a, the location (position and orientation) of the marker, one or more virtual objects, or a combination thereof. The marker location may include three-dimensional coordinates of one or more marker landmarks 616a, such as Figure 6, the corners of the roughly rectangular marker 610a shown in . The marker location can be expressed relative to real-world geographic coordinates, a marker coordinate system, the positioning of the eye-mounted device 100, or other coordinate systems. One or more virtual objects associated with the marker 610a can include any of a variety of materials, including still images, videos, audio, tactile feedback, executable applications, interactive user interfaces and experiences, and combinations or sequences of such materials. In this case, any type of content that can be stored in memory and retrieved when the marker 610a is encountered or associated with an assigned marker can be classified as a virtual object. For example, Figure 6 The button 608 shown in FIG. 6 is a virtual object displayed as a 2D or 3D still image at the marked position.

[0098] In one example, the marker 610a can be registered in memory as being located on the physical object 604a (e.g., Figure 6 In another example, the marker can be registered in memory as a specific location relative to the eye-wearing device 100.

[0099] Figure 7 , 8 9 is a diagram of an example collaboration object 700 developed by adding virtual content 702 during a collaboration period of a collaboration session, for describing the following Fig.10 and 11 700). Although in many of the examples described herein, a box is used for the collaboration object 700, any virtual object may be selected for use as the collaboration object 700. Furthermore, although the illustration depicts the eye-mounted device 100 as a physically remote device, it should be understood that the functionality described below may be implemented using other physically remote devices, such as the mobile device 401.

[0100] Figure 7 A perspective view of an example collaborative object 700 in the form of a box in a first state (closed) is provided, which can be manipulated in three dimensions 701 with a hand 651 based on corresponding movements in three dimensions 681 (e.g., by gestures in a detected image or touch input on a touch screen). For example, the hand 651 can be rotated to rotate the collaborative object. In some examples, an extended index finger can be detected near the collaborative object, and when the user makes a tap gesture, a corresponding audible signal will be presented via a speaker. The position of the eye-wearing device can also be tracked in three dimensions 840 within the environment 600, making the overlay generated for the presentation of the display 180B more realistic. The hand 651 can be predefined as a left hand, as shown. In some embodiments, the system includes a process for selecting and setting the right hand or left hand to be used as the hand 651 to be detected.

[0101] Figure 8 A perspective view of an example collaboration object 700 in a second state (open) is provided, wherein associated virtual content 702 (dial 702a, urn 702b, book 702c, other virtual content 702d-f) is added during the collaboration period. Hand 652 is shown in an open position. The position of hand 652 or hand 651 in a relaxed position ( Figure 7 ) to the hand 652 in the open position can be set to correspond to opening the collaboration object 700, so that when the hand position or gesture is detected, the collaboration object 700 transitions to the open state.

[0102] Fig. 9 A perspective view of an example collaborative object 700 in a first state (closed) is provided, wherein virtual content 702 is added to the outer surface of the collaborative object 700. The hand 653 is shown in a closed position. The position of the hand 653 or the hand 651 in a relaxed position ( Figure 7 ) to the transition of the hand 653 in the closed position can be set to correspond to the closing of the collaboration object 700, so that when the hand position or gesture is detected, the collaboration object 700 transitions to the closed state.

[0103] The process of detecting and tracking includes detecting a hand 651 / 652 / 653 in various poses over time in a set or series of captured video data frames. In this case, detection refers to and includes detecting a hand in at least one frame of video data, as well as detecting a hand in a subset or series of frames of video data over time. Thus, in some embodiments, the process includes detecting a hand 651 in a particular pose in one or more captured frames of video data. In other embodiments, the process includes detecting a hand 651 / 652 / 653 in various poses over time in a subset or series of captured frames of video data.

[0104] Fig.10 1000 is a flowchart depicting an example method of developing a collaboration object 700 during a collaboration period of a collaboration session including multiple physically remote devices (e.g., eyewear device 100, mobile device 401, or a combination thereof). In the example, Fig.10 The steps are performed by a server system 498 (see Figure 4 ) is performed by a processor 499 of a physical remote device. In other examples, one or more steps may be performed by a combination of processors 432 and 540 of a physical remote device or a processor of a server system 498 and one or more physical remote devices (acting as processors to implement one or more steps). One or more steps shown and described may be performed simultaneously, in series, in an order different from that shown and described, or in combination with additional steps. Certain steps may be omitted or repeated in certain applications.

[0105] At block 1002, the processor receives user parameters for the collaboration session. In an example, the processor receives the user parameters from a physical remote device of a host user, wherein the host user specifies the user parameters through his physical remote device during a connection to the server system 498. The user parameters include identifiers of users who are allowed to access the collaboration session. The user parameters may also include access levels that identify what content an individual can access during the collaboration session. Referring now to flowchart 1100 ( Fig.11 ) describes additional details about setting and maintaining access levels.

[0106] At block 1004, the processor receives object parameters. In an example, the processor receives the object parameters from a physical remote device of a host user (or other user with an appropriate access level), wherein the user specifies the object parameters through his physical remote device during a connection to the server system 498. The object parameters include an identifier (e.g., a URL) that identifies the object to be used as the collaboration object 700. Figure 7-9 Other object parameters may include the material of the object (e.g., cardboard, metal, glass) or a time parameter (e.g., in the form of a clock value 710 or a time bar 712) that provides a time window or deadline during which virtual content 702 may be added to collaborative object 700 (after which virtual content 702 may no longer be added to collaborative object 700).

[0107] The processor of the server system 498 can present a list of available virtual objects 702 to the physical remote device via the network 495 for the user to select through their device, which list is received by the processor of the server system 498 when selected. Alternatively, the user can send the virtual object 702 (e.g., 3D image) they generated on the physical remote device to the server system 498, where the processor 499 of the server system 498 designates it as a collaborative object 700 when the virtual object 702 is received.

[0108] At block 1006, the processor provides access to the collaboration object 700. The processor provides access to the collaboration object 700 through a server connection with the physical remote device based on the access level associated with each device. In an example, the processor 499 develops the collaboration object 700 in response to the received object parameters and stores the collaboration object 700 in a location accessible to the physical remote device. The processor provides access to the collaboration object 700 based on the access level associated with the user of the physical remote device. When the user accesses the collaboration object 700, the processor sends a file containing the collaboration object 700 to the mobile device 401, which the physical remote device uses to generate an overlay for presentation on a display of the physical remote device, such as the display 180A-B or the display 580 of the eye-mounted device 100. The user can then interact with the representation of the collaboration object 700 on their display (e.g., using a browser such as Figure 7 gesture shown)

[0109] At block 1008, the processor receives the design parameters. The processor receives the design parameters from a physical remote device that has an appropriate permission level to access the collaboration object 700. In an example, the processor sends a file containing the collaboration object 700 to the physical remote device. The physical remote device generates an overlay for display, with which the user can interact to add the design parameters to the collaboration object 700. Then, for example, the user can select an image (e.g., from their camera, such as camera 114A-B or camera 570) and select a surface of the collaboration object 700, upon selecting the surface, the selected image is associated with the collaboration object 700 on the physical remote device. As the user makes changes on their device, the added / changed design parameters are communicated by the physical remote device to the server system 498 via the network 498.

[0110] At block 1010, the processor updates the collaboration object 700 in response to the design parameters. The processor updates the collaboration object 700 in response to the changes received from the physical remote device via the network 495. In an example, upon receiving the added / changed design parameters from the physical remote device, the processor associates the added / changed design parameters with the collaboration object 700 in a location accessible to the physical remote device.

[0111] At block 1012, the processor receives virtual content 702. The processor receives virtual content 702 from a physical remote device with an appropriate permission level for accessing the collaborative object 700. In an example, the processor sends a file containing the collaborative object 700 to the physical remote device. The physical remote device generates an overlay for display, and the user can interact with the overlay to add virtual content 702 to the collaborative object 700. Then, for example, the user can add visual virtual content 702 by selecting an image (e.g., from their camera) and performing an action (e.g., dragging and dropping the image onto the collaborative object 700 or double-clicking the object) to associate the virtual content 702 with the collaborative object 700 on the physical remote device. In addition, video virtual content can be added to the video virtual content by, for example, pressing and holding the video virtual content and speaking into a microphone, where the audio received when pressing the video virtual content is associated with the video virtual content. When the user makes changes on their device, the added virtual content 702 is communicated to the server system 498 by the physical remote device via the network 495.

[0112] In an example, the user may associate the virtual content 702 with the collaboration object 700 by, for example, dragging and dropping the virtual content 702 onto the surface of the collaboration object 700. In one example, using the eye-mounted device 100, the eye-mounted device 100 may recognize a gesture, and the user may manipulate the collaboration object 700 displayed on the display 180A-B and select the virtual content 702 via the gesture captured and processed by the eye-mounted device 100. In another example, using the mobile device 401, the mobile device 401 may interpret instructions received via the touch screen 580 of the mobile device 401. The user may manipulate the collaboration object 700 and select the virtual content 702 by touching / tapping the touch screen 580 with his finger, and select the virtual content 702 by dragging his finger to move the virtual content 702 onto the collaboration object 700 (which may associate the virtual content 702 with the collaboration object 700).

[0113] At block 1014, the processor associates the virtual content 702 with the collaboration object 700. The processor associates the virtual content 702 with the collaboration object 700 by updating the collaboration object 700 in response to the changes received from the physical remote device. In an example, upon receiving the added virtual content 702 from the physical remote device, the processor associates the added virtual content 702 with the collaboration object 700 in a location accessible to the physical remote device.

[0114] At block 1016, the processor stores the collaboration object 700. The processor 499 stores the collaboration object 700 in a memory accessible to the physically remote device via the network 495 during the collaboration session.

[0115] At block 1018, the processor provides access to the collaboration object 700. The processor 499 provides access to the collaboration object 700 in a memory accessible to the physically remote device via the network 495. In an example, the processor checks the credentials (e.g., user ID) of the user requesting access and allows access if the credentials match the credentials associated with the collaboration session of the collaboration object 700.

[0116] At block 1020, the processor presents the collaboration object 700. The processor 499 presents the collaboration object 700 to the physical remote device via the network 495. In an example, the processor, in response to a request from the physical remote device, sends a file including the collaboration object 700 (and associated virtual content or a connection to such content) to the physical remote device that can access the collaboration session, and the physical remote device uses the file to generate an overlay including the collaboration object 700 and the associated virtual content for presentation on the display of the remote physical device. In one example, when the collaboration object 700 is in an open state, the associated virtual content is presented all at once. In another example, the associated virtual content is presented sequentially based on a timestamp added when the virtual content is associated with the collaboration object 700.

[0117] Fig.11 is a flowchart listing the steps of an example selective collaboration object access method. In the example, Fig.11 The steps are performed by a server system 498 (see Figure 4 ) is performed by a processor 499 of a physical remote device. In other examples, one or more steps may be performed by a combination of processors 432 and 540 of a physical remote device or processor 499 of server system 498 and one or more physical remote devices (acting as processors to implement one or more steps). One or more steps shown and described may be performed simultaneously, in series, in an order different from that shown and described, or in combination with additional steps. Certain steps may be omitted or may be repeated in certain applications.

[0118] At block 1102, the processor receives a user identifier. The processor receives a user identifier of a user associated with a collaboration session. In one example, a user (host) accesses the server system 498 using a physical remote device. Using the physical remote device, the user creates a collaboration session and specifies other users to participate in the collaboration. For example, the host (user A) can invite another user (user B) to participate in the collaboration to prepare content for another user (user C) so that the content can be provided to the user at a later time.

[0119] At block 1104, the processor receives the user's access parameters. The processor receives the user's access parameters from the host or another user with an acceptable access level. The access parameters indicate the corresponding access level of each user's collaborative object 700, which allows the corresponding access level of at least one user to be different from the corresponding access level of another user. For example, the host can have a first access level (allowing access to the collaborative object 700, so as to associate virtual content 704 and view the associated virtual content 704 during the collaborative period), and another user can have a second access level to the collaborative object 700 that is less than the first access level (for example, it only allows access after the collaborative period ends; or it allows access to the collaborative object 700 during the collaborative period, but does not allow access to the associated virtual content 702 until the collaborative period ends). In one example, the host receives the access level that can be accessed by default, and the host can authorize other user access rights by providing access rights to the processor 499 of the server system 498 via a physical remote device.

[0120] At block 1106, the processor maintains an access parameter table. In an example, the processor maintains a table in the cloud storage that includes an identifier for each user and their respective access levels based on the access parameters provided by the host. In an example where the host (user A; A_ID) invites another user (user B; B_ID) to collaborate on a project for yet another user (user C; C_ID), the processor may initially create a table that includes the information identified in Table 1 below:

[0121] user license A_ID yes B_ID no C_ID no

[0122] Table 1

[0123] In Table 1, a permission level of "yes" indicates a first access level, and a permission level of "no" indicates a second access level. As shown in Table 1, the host (A_ID) is initially the only user with access rights that provide, for example, access to virtual content 704 associated with collaborative object 700 or the ability to add virtual content 704.

[0124] At block 1108, the processor maintains a timer. The processor may receive an initial time value from the host. The timer may be used to track the time remaining during the collaboration period. In one example, the processor provides a time value corresponding to the initial time or the tracked time to the physical remote device, which may use the time value to generate an overlay (e.g., clock value 710 or time bar 712) depicting or representing the remaining time.

[0125] At block 1110, the processor provides access to the collaboration object 700. The processor provides the user access to the collaboration object 700 based on the user's corresponding access level. In the example, when a user attempts to access the collaboration object 700, the processor compares the user identification (ID) of the user with the value in the table. If the user's ID (e.g., D_ID) is not found on the table, the user will not be able to access the collaboration object 700. If another user (e.g., C_ID) is found in the table, but it has a "no" permission level, the user will only be provided with access rights commensurate with the access level. If another user (e.g., A_ID) is found in the table, it has a "yes" permission level, the user will be provided with access to the collaboration object 700 commensurate with the access level.

[0126] At block 1112, the processor identifies the access level change. The processor identifies the access level change of at least one other user. In one example, the host authorizes another user (e.g., B_ID) access rights by sending an access rights change request to the processor 499 of the server system 498 via a physical remote device, the request including the user ID to be changed and the new access level. In another example, all users with a "yes" permission level must request an access level change for a user with another permission level. The processor identifies the access level change by the processor after receiving one or more requests. In another example, the processor can identify the change based on the expiration of the collaboration period or a preset "show" time (e.g., based on a monitored timer). For example, in the case where user A and user B are preparing content for user C, user A and user B may initially have a "yes" permission level, and once the collaboration period has ended or the show time has been reached, the processor can automatically identify the access level change request of user C.

[0127] At block 1114, the processor changes the corresponding access level. The processor changes the corresponding access level of at least one other user in response to the access level change. In response to the identified access level change (e.g., changing the B_ID permission level to "yes"), the processor updates the table shown in Table 2 below:

[0128] user license A_ID yes B_ID yes C_ID no

[0129] Table 2

[0130] Machine learning refers to algorithms that gradually improve through experience. By processing a large number of different input data sets, machine learning algorithms can make improved generalizations about a specific data set and then use these generalizations to produce accurate outputs or solutions when processing new data sets. Broadly speaking, a machine learning algorithm includes one or more parameters that will adjust or change in response to new experience, thereby gradually improving the algorithm; similar to the process of learning.

[0131] In the context of computer vision, mathematical models attempt to simulate the tasks performed by the human visual system, with the goal of using a computer to extract information from an image and achieve an accurate understanding of the image's content. Computer vision algorithms have been developed for use in a variety of fields (including artificial intelligence and autonomous navigation) to extract and analyze data from digital images and videos.

[0132] Deep learning refers to a class of machine learning methods that are based on or imitate artificial neural networks. Artificial neural networks are computing systems composed of many simple, highly interconnected processing elements (nodes) that process information by dynamically responding to external inputs. Large artificial neural networks may have hundreds or thousands of nodes.

[0133] A convolutional neural network (CNN) is a type of neural network commonly used to analyze visual images, including digital photos and videos. The connection patterns between nodes in a CNN are often modeled after the organization of the human visual cortex, which includes individual neurons arranged to respond to overlapping regions in the visual field. The neural network suitable for the determination process described in this article is based on one of the following architectures: VGG16, VGG19, ResNet50, Inception V3, Xception, or other CNN-compatible architectures.

[0134] In the machine learning example, at blocks 1008 and 1012, the processor 432 uses a machine-trained algorithm called a hand feature model to determine whether the detected series of hand shapes substantially matches a predefined hand gesture. The processor 432 is configured to access the hand feature model trained by machine learning and apply the hand feature model to identify and locate hand shape features in one or more frames of video data.

[0135] In one example implementation, the trained hand feature model receives a video data frame containing a detected hand shape and abstracts the image in the frame into layers for analysis. Based on the trained hand feature model, the data in each layer is compared with the gesture data stored in the gesture library 480 layer by layer until a good match is identified.

[0136] In one example, a layer-by-layer image analysis is performed using a convolutional neural network. In the first convolutional layer, the CNN identifies learned features (e.g., hand landmarks, joint coordinate sets, etc.). In the second convolutional layer, the image is transformed into multiple images in which the learned features are emphasized in the corresponding sub-images. In the pooling layer, the size and resolution of the image and sub-images are reduced to isolate the parts of each image that include possible target features (e.g., possible palm shapes, possible finger joints). The values ​​and comparisons of the images from the non-output layers are used to classify the images in the frame. As used herein, classification refers to the process of classifying images based on the detected hand shape using a trained model. For example, if the detected series of two-handed shapes matches the touch gestures stored in library 480, the image can be classified as a "touch action."

[0137] Any functionality of the eye-wearing device 100, mobile device 401, and server system 498 described herein may be embodied in one or more computer software applications or programming instruction sets, as described herein. According to some examples, a "function," "multiple functions," "application," "multiple applications," "instructions," "multiple instructions," or "programming" is one or more programs that execute functions defined in a program. Various programming languages ​​may be employed to develop one or more applications structured in various ways, such as an object-oriented programming language (e.g., Objective-C, Java, or C++) or a procedural programming language (e.g., C or assembly language). In a specific example, a third-party application (e.g., an application created using ANDROID ® by an entity other than the vendor of a particular platform) may be used to develop one or more applications structured in various ways. TM or IOS TM Software development kit (SDK) applications can be included in mobile operating systems (such as IOS TM ANDROID TM , Mobile software running on an Android phone or other mobile operating system. In this example, third-party applications can call API calls provided by the operating system to facilitate the functions described in this article.

[0138] Therefore, machine-readable media can take the form of tangible storage media in various forms. Non-volatile storage media include, for example, optical or magnetic disks, such as any storage device in any computer device or the like, such as client devices, media gateways, transcoders, etc. shown in the accompanying drawings that can be used to implement them. Volatile storage media include dynamic memory, such as the main memory of such a computer platform. Tangible transmission media include coaxial cables; copper wires and optical fibers, including wires that contain buses within a computer system. Carrier transmission media can take the form of electrical or electromagnetic signals, or the form of sound or light waves, such as those generated during radio frequency (RF) and infrared (IR) data communications. Therefore, common forms of computer-readable media include, for example: floppy disks, floppy disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, DVDs or DVD-ROMs, any other optical media, punched card tapes, any other physical storage media with hole patterns, RAMs, PROMs and EPROMs, FLASH-EPROMs, any other memory chips or cassettes, carriers that transmit data or instructions, cables or links that transmit such carriers, or any other media from which a computer can read programming code or data. Many of these forms of computer readable media may be involved in carrying one or more sequences of one or more instructions to a processor for execution.

[0139] Except as stated immediately above, nothing that has been stated or shown is intended or should be construed as causing any component, step, feature, object, benefit, advantage, or equivalent to be dedicated to the public, whether or not it is stated in the claims.

[0140] It will be understood that the terms and expressions used in this article have the common meanings given by such terms and expressions relative to their corresponding respective explorations and research fields, unless the specific meanings thereof have been set forth in addition herein. Relational terms (such as first and second and the like) can be used only to distinguish an entity or action from another entity or action, without necessarily requiring or implying any actual such relationship or order between such entities or actions. The terms "include", "comprise", "contain", "cover" or any other variation thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including or including a series of elements or steps not only include those elements or steps, but also may include other elements or steps that are not explicitly listed or inherent to such processes, methods, articles or devices. In the absence of further constraints, an element starting with "one" or "an" does not exclude the presence of additional identical elements in the process, method, article or device including the element.

[0141] Unless otherwise indicated, any and all measurements, values, ratings, positions, amplitudes, dimensions, and other specifications set forth in this specification (including the claims that follow) are approximate and not exact. These quantities are intended to have a reasonable range, i.e., consistent with the functions to which they are related and the customary practices in the art to which they belong. For example, unless expressly stated otherwise, a parameter value or similar value may vary from the stated amount or range by as much as ±10%.

[0142] In addition, in the foregoing detailed description, it can be seen that, for the purpose of streamlining the present disclosure, individual features are grouped together in various examples. This method of disclosure should not be interpreted as reflecting an intention that the claimed examples require more features than those explicitly stated in each claim. On the contrary, as reflected in the following claims, the subject matter to be protected lies in less than all the features of any single disclosed example. Therefore, the following claims are hereby incorporated into the detailed description, with each claim standing alone as a separate claimed subject matter.

[0143] Although what is considered to be the best mode and other examples have been described above, it is to be understood that various modifications may be made therein and that the subject matter disclosed herein may be implemented in various forms and examples and that they may be applied to numerous applications, only some of which have been described herein. The following claims are intended to claim any and all modifications and variations that fall within the true scope of the present concept.

Claims

1. A system for selectively authorizing access to a collaboration object, the system comprising: a memory for storing collaborative session instructions, and A processor coupled to the memory, the processor being configured to execute the collaborative session instructions to perform functions including the following functions: receiving a user identifier of a user associated with the collaboration session; receiving access parameters of the users, the access parameters indicating respective access levels of each user to the collaboration object, the respective access level of at least one user providing a first access level to the collaboration object, and the respective access level of at least one other user providing a second access level to the collaboration object, the second access level being less than the first access level; providing users with access to the collaboration object based on their respective access levels; identifying a change in access level of the at least one other user having the second access level to the collaboration object; and A corresponding access level of the at least one other user is changed in response to the access level change.

2. The system according to claim 1, wherein: The functionality to identify said access level changes includes the following: An access level change request is received from at least one of the at least one user having the first access level.

3. The system according to claim 1, wherein: The functionality to identify said access level changes includes the following: Access level change requests are received from all of the at least one user having the first access level.

4. The system according to claim 1, wherein the functions further include the following functions: maintaining a timer indicating when a collaboration period of the collaboration session ends; in, The ability to identify access level changes includes the following features: A determination is made in response to the timer that the collaboration period has ended.

5. The system according to claim 1, wherein the functions further include the following functions: receiving a display time of the collaborative session; Maintenance timer; in, The functionality to identify said access level changes includes the following: Reaching the display time is identified in response to the timer.

6. The system according to claim 1, wherein: The function of changing the corresponding access level of the at least one other user includes the following functions: The corresponding access level of the at least one other user is increased to the first access level.

7. The system according to claim 1, wherein the functions further include the following functions: associating virtual content received from a user having the first access level with the collaboration object during a collaboration period of the collaboration session; A user having the first access level can access the virtual content, and a user having the second access level cannot access the virtual content.

8. The system according to claim 7, wherein: The function of changing the corresponding access level of the at least one other user includes the following functions: The corresponding access level is changed to a new level capable of accessing the virtual content.

9. The system of claim 1, wherein the functions further include the following functions: maintaining in said memory a table comprising said user identifiers and corresponding access levels of said users; and The table is updated in response to changing the corresponding access level.

10. A method for selectively authorizing access to a collaboration object, the method comprising: receiving a user identifier of a user associated with the collaboration session; receiving access parameters of the users, the access parameters indicating respective access levels of each user to the collaboration object, the respective access level of at least one user providing a first access level to the collaboration object, and the respective access level of at least one other user providing a second access level to the collaboration object, the second access level being less than the first access level; providing users with access to the collaboration object based on their respective access levels; identifying a change in access level of the at least one other user having a second access level to the collaboration object; and A corresponding access level of the at least one other user is changed in response to the access level change.

11. The method according to claim 10, wherein: Identifying the access level change includes: An access level change request is received from at least one of the at least one user having the first access level.

12. The method according to claim 10, wherein: Identifying the access level change includes: Access level change requests are received from all of the at least one user having the first access level.

13. The method according to claim 10, further comprising: maintaining a timer indicating when a collaboration period of the collaboration session ends; Among other things, identifying access level changes includes: A determination is made in response to the timer that the collaboration period has ended.

14. The method according to claim 10, further comprising: receiving a display time of the collaborative session; Maintenance timer; Wherein, identifying the access level change includes: Reaching the display time is identified in response to the timer.

15. The method according to claim 10, wherein: Changing the corresponding access level of the at least one other user includes: The corresponding access level of the at least one other user is increased to the first access level.

16. The method according to claim 10, further comprising: associating virtual content received from a user having the first access level with the collaboration object during a collaboration period of the collaboration session; A user having the first access level can access the virtual content, and a user having the second access level cannot access the virtual content.

17. The method according to claim 16, wherein: Changing the corresponding access level of the at least one other user includes: The corresponding access level is changed to a new level capable of accessing the virtual content.

18. The method according to claim 10, further comprising: maintaining in a memory a table comprising said user identifiers and corresponding access levels of said users; and The table is updated in response to changing the corresponding access level.

19. A non-transitory computer-readable storage medium comprising instructions for selectively authorizing access to a collaboration object, the instructions, when executed by a processor, configure a system to perform functions comprising: receiving a user identifier for a user associated with the collaboration session; receiving access parameters of the users, the access parameters indicating respective access levels of each user to the collaboration object, the respective access level of at least one user providing a first access level to the collaboration object, and the respective access level of at least one other user providing a second access level to the collaboration object, the second access level being less than the first access level; providing users with access to the collaboration object based on their respective access levels; identifying a change in access level of the at least one other user having a second access level to the collaboration object; and A corresponding access level of the at least one other user is changed in response to the access level change.

20. The non-transitory computer-readable storage medium of claim 19, wherein the functions further comprise the following functions: maintaining in a memory a table including the user identifiers and corresponding access levels for the users; and updating the table in response to a change in the corresponding access level.

Images