Real work response capability

Abstract

A collaborative session (e.g., a virtual time capsule) in which a user is provided with access to collaborative objects with related materials and added virtual content. In a collaboration session example, a user selects an associated material of a collaboration object. When manipulating the collaboration object, physical features are assigned to the collaboration object in accordance with the associated material perceived by the participant. In one example, a material associated with a collaborative object is a metal, where an interaction between a user and the collaborative object generates a response of the collaborative object that is indicative of physical characteristics of the metal, such as inertia, acoustics, and ductility.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. application serial number 17 / 900,807, filed on August 31, 2022, the contents of which are incorporated herein by reference in their entirety. Technical Field

[0003] Examples set forth in this disclosure relate to the field of virtual reality for electronic devices, including mobile devices and wearable devices such as eye-mounted devices. More specifically, but not by way of limitation, this disclosure describes a collaborative method with selective access. Background Art

[0004] Many types of computers and electronic devices available today, such as mobile devices (e.g., smartphones, tablets, and laptops), handheld devices, and wearable devices (e.g., smart glasses, digital eyewear, head-mounted devices, helmets, and head-mounted displays), include various cameras, sensors, wireless transceivers, input systems, and displays.

[0005] A graphical user interface allows the user to interact with displayed content, including virtual objects and graphical elements such as icons, task bars, list boxes, menus, buttons, and selection control elements like cursors, pointers, handles, and sliders.

[0006] Virtual reality (VR) technology generates a complete virtual environment that includes realistic images, sometimes presented on a VR headset or other head-mounted display. VR experiences allow users to move around in a virtual environment and interact with virtual objects. Augmented reality (AR) is a VR technology that combines real objects in the physical environment with virtual objects and displays this combination to the user. The combined display gives the impression that the virtual objects really exist in the environment, especially when the virtual objects look and behave like real objects. Extended reality (XR) is generally understood as an umbrella term referring to systems that include or combine elements from AR, VR, and MR (mixed reality) environments.

[0007] Users of VR technology can use collaboration tools. Collaboration tools enable users to meet virtually in a collaboration session. During a collaboration session, users can communicate with each other in a virtual setting. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The features of the various examples described can be easily understood from the following detailed description, in which reference is made to the accompanying drawings. In the several views of the specification and the accompanying drawings, each element uses a reference numeral. When there are multiple similar elements, a single reference numeral can be assigned to the similar elements, and the added lowercase letter indicates the specific element.

[0009] Unless otherwise noted, the various elements shown in the figures are not drawn to scale. For clarity, the sizes of the various elements may be enlarged or reduced. Several figures depict one or more embodiments and are presented by way of example only and should not be construed as limiting. The drawings include the following figures:

[0010] Figure 1A is a side view of an example hardware configuration suitable for an eye-mounted device in an example collaborative system (right);

[0011] Figure 1B yes Figure 1A A perspective partial cutaway view of a right corner of an eye-mounted device depicting a right visible light camera and a circuit board;

[0012] Figure 1C yes Figure 1A A side view of an example hardware configuration of an eye-mounted device of FIG. 1 (left), showing a left visible light camera;

[0013] Figure 1D yes Figure 1C A perspective partial cutaway view of a left corner of an eye-mounted device;

[0014] Figure 2A and Figure 2B is a rear view of an example hardware configuration of an eye-mounted device used in an example collaborative system;

[0015] Figure 3 is a schematic diagram of a three-dimensional scene, a left original image taken by a left visible light camera, and a right original image taken by a right visible light camera;

[0016] Figure 4 is a functional block diagram of an example collaborative system including a wearable device (e.g., an eye-mounted device) and a server system connected via various networks;

[0017] Figure 5 Is applicable to Figure 4 An illustration of an example hardware configuration for a mobile device in an example system;

[0018] Figure 6 is a schematic diagram of a user in an example environment used to describe simultaneous localization and mapping;

[0019] Figure 7 is a perspective view of an example collaborative object in the form of a box that can be manipulated by hand;

[0020] Figure 8 is used to open Figure 7 a perspective view of an example of a first hand gesture associated with an opening gesture of a box shown;

[0021] Fig. 9 Yes and for closing Figure 7a perspective view of an example of a second hand gesture associated with a closing gesture of a box shown;

[0022] Fig.10 is a flow chart listing the steps in an example collaborative approach;

[0023] Fig.11 is a flow chart listing the steps of an example selective collaborative object access method;

[0024] Fig.12 is a flow chart listing steps for an example interaction of a collaboration object associated with a material;

[0025] Fig.13A is an example of a collaboration object associated with a metal material; and

[0026] Fig. 13B is an example of a collaborating object associated with an elastic material. DETAILED DESCRIPTION

[0027] A collaborative session (e.g., a virtual time capsule) in which a user is provided access to a collaborative object having an associated material and added virtual content. In an example collaborative session, a user selects an associated material of a collaborative object. As the collaborative object is manipulated, physical characteristics are assigned to the collaborative object based on the associated material as perceived by the participant. In one example, the material associated with the collaborative object is a metal, wherein the interaction between the user and the collaborative object generates a response of the collaborative object that indicates physical properties of the metal such as inertia, acoustics, and ductility.

[0028] The following detailed description includes systems, methods, techniques, instruction sequences, and computer program products that explain the examples set forth in this disclosure. In order to provide a thorough understanding of the disclosed subject matter and its related teachings, many details and examples are included. However, those skilled in the relevant art can understand how to apply the related teachings without these details. The various aspects of the disclosed subject matter are not limited to the specific devices, systems, and methods described, because the related teachings can be applied or practiced in various ways. The terms and nomenclature used herein are only for describing specific aspects, not for limiting. In general, it is not necessary to show in detail the well-known instruction examples, protocols, structures, and techniques.

[0029] As used herein, the terms "coupled" or "connected" refer to any logical, optical, physical, or electrical connection, including links that transmit electrical or magnetic signals generated or supplied by one system element to another coupled or connected system element, etc. Unless otherwise specified, coupled or connected elements or devices are not necessarily directly connected to each other and may be separated by intermediate components, elements, or communication media, one or more of which can modify, manipulate, or carry electrical signals. The term "on" means supported by an element directly or supported by the element through another element that is integrated into or supported by the element.

[0030] The term "proximal" is used to describe an item or a portion of an item that is near, adjacent to, or beside an object or person; or closer to another portion of the item that may be described as "distal." For example, the end of an item that is closest to the object may be called the proximal end, while the generally opposite end may be called the distal end.

[0031] The orientation of eye-mounted devices, other mobile devices, related components, and any other devices incorporating cameras, inertial measurement units, or both, as shown in any of the figures, are given as examples only, for purposes of illustration and discussion. In operation, the eye-mounted device may be oriented in any other orientation suitable for the particular application of the eye-mounted device; for example, up, down, sideways, or any other orientation. Furthermore, for the purposes of use herein, any directional terms, such as front, back, inward, outward, toward, left, right, lateral, longitudinal, up, down, upper, lower, top, bottom, sideways, horizontal, vertical, and diagonal are used only as examples and do not limit the orientation or orientation of any camera or inertial measurement unit constructed or otherwise described herein.

[0032] Advanced AR technologies such as computer vision and object tracking can be used to produce perceptually rich and immersive experiences. Computer vision algorithms extract three-dimensional data about the physical world from data captured in digital images or videos. Object recognition and tracking algorithms are used to detect objects in digital images or videos, estimate their orientation or pose, and track their motion over time. Real-time recognition and tracking of hands and fingers is one of the most challenging and processing-intensive tasks in the field of computer vision.

[0033] The term "pose" refers to the static position and orientation of an object at a particular moment. The term "gesture" refers to the active movement of an object (such as a hand) through a series of gestures, sometimes to convey a signal or idea. In the fields of computer vision and augmented reality, the terms pose and gesture are sometimes used interchangeably. As used herein, the terms "pose" or "gesture" (or variations thereof) are intended to include both poses and gestures; in other words, the use of one term does not exclude the other.

[0034] Other purposes, advantages and novel features of the examples will be explained in part in the following description, and in part will become apparent to those skilled in the art after studying the following and the accompanying drawings, or can be understood through the production or operation of the examples. The purposes and advantages of the subject matter can be realized and obtained by the methods, means and combinations particularly pointed out in the appended claims.

[0035] Reference will now be made in detail to examples illustrated in the accompanying drawings and discussed below.

[0036] Figure 1A is a side view (right) of an example hardware configuration of the eye-mounted device 100, Figure 1C is a side view (left) of an example hardware configuration of the eye-mounted device 100, which includes a touch-sensitive input device or touchpad 181. As shown, the touchpad 181 can have a subtle and not easily visible border; in addition, the border can be clearly visible or include a raised or other tactile edge that provides feedback to the user about the position and border of the touchpad 181. In other embodiments, the eye-mounted device 100 can include a touchpad located on the left side.

[0037] The surface of the touch pad 181 is configured to detect finger touches, taps, and gestures (e.g., moving touch) for use with a graphical user interface displayed by the eye-mounted device on the image display, thereby allowing the user to browse and select menu options in an intuitive manner, which enhances and simplifies the user experience.

[0038] Detecting finger input on the touch pad 181 can enable a variety of functions. For example, touching anywhere on the touch pad 181 can cause the GUI to display or highlight an item on the image display, which can be projected onto at least one of the optical components 180A, 180B. Double-clicking on the touch pad 181 can select an item or icon. Sliding or swiping a finger in a particular direction (e.g., from front to back, from back to front, from top to bottom, or from bottom to top) can cause an item or icon to slide or scroll in a particular direction; for example, move to the next item, icon, video, image, page, or slide. Sliding a finger in another direction can slide or scroll in the opposite direction; for example, move to the previous item, icon, video, image, page, or slide. The touch pad 181 can be located almost anywhere on the eye-mounted device 100.

[0039] In one example, a single tap finger gesture recognized on touch pad 181 initiates selection or pressing of a graphical user interface element in an image presented on the image display of optical assembly 180A, 180B. Adjustment of the image presented on the image display of optical assembly 180A, 180B based on the recognized finger gesture may be the primary action for selecting or submitting a graphical user interface element on the image display of optical assembly 180A, 180B for further display or execution.

[0040] As shown, the eye-mounted device 100 includes a left visible light camera 114A and a right visible light camera 1141B. As further described herein, the two cameras 114A, 114B capture image information of a scene from two separate viewpoints. The two captured images can be used to project a three-dimensional display onto an image display for viewing with 3D glasses.

[0041] The eye-mounted device 100 includes a right optical assembly 180B having an image display that presents an image (such as a depth image). Figure 1A and Figure 1C As shown, the eye-mounted device 100 may include a plurality of visible light cameras 114A and 114B, wherein the visible light cameras 114A and 114B form a passive type three-dimensional camera (such as a stereo camera), wherein the right visible light camera 114B is located at the right corner 110B, and as shown in FIG. Figure 1C-Figure 1D As shown, left visible light camera 114A is located at left corner 110A.

[0042] Left and right visible light cameras 114A, 114B are sensitive to wavelengths in the visible light range. Each of visible light cameras 114A, 114B has a different forward field of view that overlaps with each other to generate a three-dimensional depth image, for example, left visible light camera 114A captures left field of view 111A, and right visible light camera 114B captures right field of view 111B. In general, a "field of view" is a portion of a scene visible through a camera at a specific location and orientation in space. Fields of view 111A and 111B have overlapping fields of view 304 ( Figure 3 ). When visible light camera captures an image, objects or object features outside field of view 111A, 111B are not recorded in the original image (e.g., a photograph or picture). Field of view describes the angular range or degree to which the image sensor of visible light camera 114A, 114B picks up electromagnetic radiation of a given scene in a captured image of the given scene. Field of view can be expressed as the angular size of the viewing cone; i.e., the viewing angle. The viewing angle can be measured horizontally, vertically, or diagonally.

[0043] In an example configuration, one or both visible light cameras 114A, 114B have a 100° field of view and a resolution of 480×480 pixels. “Coverage angle” describes the angle of coverage of visible light camera 114A, 114B or infrared camera 410 (see FIG. Figure 2A) is the range of angles over which a lens can effectively image. Typically, a camera lens produces an image circle large enough to completely cover the camera's film or sensor, possibly including some vignetting (e.g., the image gets darker toward the edges compared to the center). If the camera lens's angular coverage does not fill the sensor, the image circle will be visible, usually with strong vignetting toward the edges, and the effective viewing angle will be limited to the angular coverage.

[0044] Examples of such visible light cameras 114A, 114B include digital camera elements, such as high-resolution complementary metal oxide semiconductor (CMOS) image sensors and digital VGA cameras (video graphics arrays), with resolutions up to 480p (e.g., 640×480 pixels), 720p, 1080p, or higher. Other examples include visible light cameras 114A, 114B that can capture high-definition (HD) video at high frame rates (e.g., thirty to sixty frames per second, or higher) and store recordings at a resolution of 1216×1216 pixels (or higher).

[0045] The eye-mounted device 100 can capture image sensor data and geographic location data from the visible light cameras 114A and 114B, which are digitized by the image processor for storage in the memory. The visible light cameras 114A and 114B capture corresponding left and right raw images in a two-dimensional spatial domain, which include a pixel matrix on a two-dimensional coordinate system, which includes an X-axis for horizontal position and a Y-axis for vertical position. Each pixel includes a color attribute value (e.g., a red pixel light value, a green pixel light value, or a blue pixel light value) and a position attribute (e.g., an X-axis coordinate and a Y-axis coordinate).

[0046] To capture stereoscopic images for later display as a three-dimensional projection, image processor 412 ( Figure 4 ) can be coupled to visible light cameras 114A, 114B to receive and store visual image information. Image processor 412 or another processor controls the operation of visible light cameras 114A, 114B to act as a stereo camera that simulates human binocular vision, and can add a time stamp to each image. The time stamp on each pair of images allows the images to be displayed together as part of a three-dimensional projection. The three-dimensional projection produces an immersive, lifelike experience that is ideal in a variety of environments including virtual reality (VR) and video games.

[0047] Figure 1B yes Figure 1A 1 is a perspective cross-sectional view of the right corner 110B of the eye-mounted device 100, depicting the right visible light camera 114B and circuit board of the camera system. Figure 1C yes Figure 1A 1 is a side view (left) of an example hardware configuration of the eye-mounted device 100, showing the left visible light camera 114A of the camera system. Figure 1D yes Figure 1C A perspective cross-sectional view of the left corner 110A of the eye-mounted device depicting the left visible light camera 114A and circuit board of the 3D camera.

[0048] like Figure 1B As shown in the example of FIG. 1 , the eye-mounted device 100 includes a right visible light camera 114B and a circuit board 140B, which may be a flexible printed circuit board (PCB). The right hinge 126B connects the right corner 110B to the right temple 125B of the eye-mounted device 100. In some examples, the right visible light camera 114B, the flexible PCB 140B, or other electrical connectors or contacts may be located on the right temple 125B or the right hinge 126B.

[0049] The left visible light camera 114A is constructed and arranged substantially similarly to the right visible light camera 114B, except that the connection and coupling is located at the left outer side 170A. The left hinge 126B connects the left corner 110A to the left temple 125A of the eye-mounted device 100. In some examples, components of the left visible light camera 114A, the flexible PCB 140A, or other electrical connectors or contacts can be located on the left temple 125A or the left hinge 126A.

[0050] The left and right corners 110A and 110B each include a corner body 190 and a corner cover. Figure 1B and Figure 1D Corner covers are omitted in the cross-section of FIG. Arranged within left corner 110A and right corner 110B are various interconnected circuit boards 140A and 140B, such as PCBs or flexible PCBs, including controller circuits for left visible light camera 114A and right visible light camera 114B, microphones, low power wireless circuits (e.g., for communicating via Bluetooth TM The corners 110A, 110B may be integrated into the frame 105 located on the respective outer sides 170A, 170B (as shown), or implemented as separate components connected to the frame 105 on the respective sides 170A, 170B. In addition, the corners 110A, 110B may be integrated into the temples 125A, 125B connected to the frame 105.

[0051] The left and right visible light cameras 114A and 114B are coupled to or disposed on the corresponding flexible PCBs 140A and 140B, respectively, and are covered by cover lenses of the visible light cameras, which are aligned through openings formed in the frame 105. For example, the left and right frames 107A and 107B of the frame 105 are connected to the left and right corners 110A and 110B and include openings for the visible light camera cover lenses. The frame 105 includes a front side configured to face outward and away from the user's eyes. The opening of the visible light camera cover lens is formed and passes through the front side or outward-facing side of the frame 105. In this example, the left and right visible light cameras 114A and 114B each have an outward-facing field of view 111A and 111B, whose line of sight or viewing angle is associated with the corresponding left eye and right eye of the user of the eye-mounted device 100. The visible light camera cover lens may also be adhered to the front or outward facing surface of the right corner 110B, with the opening formed with the outward facing cover corner, but in a different outward direction. The coupling may also be an indirect coupling via an intermediate component.

[0052] Figure 2A and Figure 2B 1 is a perspective view of an example hardware configuration of an eye-mounted device 100 shown from the rear, including two different types of image displays. The size and shape of the eye-mounted device 100 is configured to be worn by a user; the shape of glasses is shown in the example. The eye-mounted device 100 can take other forms and can be combined with other types of frames; for example, a helmet, a head-mounted device, or a head cover.

[0053] In the glasses example, the eyewear device 100 includes a frame 105, which includes a left frame 107A connected to a right frame 107B via a bridge 106, and the bridge 106 is suitable for being supported by the user's nose. The left and right frames 107A, 107B include corresponding holes 175A, 175B, and the holes 175A, 175B accommodate corresponding optical elements 180A, 180B, such as lenses and display devices. As used herein, the term "lens" refers to a transparent or translucent glass or plastic sheet having a curved or flat surface that converges or diverges light or converges or diverges little or no.

[0054] Although shown as having two optical elements 180A, 180B, the eye-mounted device 100 may include other arrangements such as a single optical element (or it may not include any optical elements 180B, 180A), depending on the application or intended user of the eye-mounted device 100. As further shown, the eye-mounted device 100 includes a left corner 110A adjacent to a left outer side 170A of the frame 105 and a right corner 110B adjacent to a right outer side 170B of the frame 105.

[0055] In one example, the image display of the optical components 180A, 180B includes an integrated image display. Figure 2A As shown, each optical assembly 180A, 180B includes a suitable display matrix 177, such as a liquid crystal display (LCD), an organic light emitting diode (OLED) display, or any other such display. Each optical assembly 180A, 180B also includes one or more optical layers 176, which may include lenses, optical coatings, prisms, reflectors, waveguides, light strips, and other optical components in any combination. Optical layers 176A, 176B, ..., 176N (in Figure 2A The optical layers 176A-176N (shown herein as 176A-176-N) may include prisms having a suitable size and configuration and including a first surface for receiving light from the display matrix and a second surface for emitting light toward the user's eyes. The prisms of the optical layers 176A-176N extend over all or at least a portion of the corresponding holes 175A, 175B formed in the left and right frames 107A, 107B to allow the user to see the second surface of the prisms when the user's eyes look through the corresponding left and right frames 107A, 107B. The first surfaces of the prisms of the optical layers 176A-176N face upward from the frame 105, and the display matrix 177 covers the prisms so that photons and light emitted by the display matrix 177 strike the first surface. The size and shape of the prisms are such that light is refracted within the prisms and directed to the user's eyes through the second surfaces of the prisms of the optical layers 176A-176N. In this regard, the prismatic second surface of the optical layers 176A-176N can be convex to direct light toward the center of the eye. The size and shape of the prisms can optionally magnify the image projected by the display matrix 177, and the light passes through the prisms so that the image viewed from the second surface is larger in one or more dimensions than the image emitted from the display matrix 177.

[0056] In one example, the optical layers 176A-176N may include an LCD layer that is transparent (keeping the lenses open) unless / until a voltage is applied to make the layer opaque (closing or blocking the lenses). Figure 4 ) can be programmed to apply a voltage to the LCD layer, thereby creating an active shutter system, making the eye-mounted device 100 suitable for viewing visual content when displayed as a three-dimensional projection. Technologies other than LCD can be used for active shutter mode, including other types of reactive layers that respond to voltage or another type of input.

[0057] In another example, the image display device of the optical assembly 180A, 180B includes Figure 2BThe projected image display shown. Each optical assembly 180A, 180B includes a laser projector 150, which is a three-color laser projector using a scanning mirror or a galvanometer. During operation, a light source such as a laser projector 150 is set in one of the temples 125A, 125B of the eye-mounted device 100 or in it. The optical assembly 180B in this example includes one or more light strips 155A, 155B, ..., 155N (in Figure 2B 155A-155N), these light bands are spaced apart and span the width of the lens of each optical component 180A, 180B or span the depth of the lens between the front and back surfaces of the lens.

[0058] When the photons projected by the laser projector 150 pass through the lenses of each optical component 180A, 180B, the photons encounter light strips 155A-155N. When a particular photon encounters a particular light strip, the photon is either redirected to the user's eye or passed to the next light strip. A combination of the modulation of the laser projector 150 and the modulation of the light strips can control specific photons or light beams. In one example, the processor controls the light strips 155A-155N by initiating mechanical, acoustic, or electromagnetic signals. Although shown as having two optical components 180A, 180B, the eye-mounted device 100 may include other arrangements, such as a single or three optical components, or each optical component 180A, 180B may have a different arrangement depending on the application or intended user of the eye-mounted device 100.

[0059] In another example, Figure 2B The eye-mounted device 100 shown may include two projectors, a left projector (not shown) and a right projector 150. The left optical component 180A may include a left display matrix 177 or a set of left light strips (not shown) configured to interact with the light from the left projector. Similarly, the right optical component 180B may include a right display matrix (not shown) or a set of right light strips 155A, 155B, ..., 155N configured to interact with the light from the right projector 150. In this example, the eye-mounted device 100 includes a left display and a right display.

[0060] Figure 3306, a left raw image 302A captured by a left visible light camera 114A, and a right raw image 302B captured by a right visible light camera 1141B. As shown, the left field of view 111A can overlap with the right field of view 111B. Overlapping fields of view 304 represent portions of images captured by two cameras 114A, 114B. The term "overlap" when referring to the field of view means that the pixel matrices in the generated raw images overlap by thirty percent (30%) or more. "Substantially overlap" means that the pixel matrices in the generated raw images or infrared images of the scene overlap by fifty percent (50%) or more. As described herein, the two raw images 302A, 302B can be processed to include a timestamp, which allows the images to be displayed together as part of a three-dimensional projection.

[0061] For stereoscopic image capture, such as Figure 3 As shown, at a given moment, a pair of raw red, green, and blue (RGB) images of a real scene 306 are captured, with the left camera 114A capturing the left raw image 302A and the right camera 114B capturing the right raw image 302B. When the pair of raw images 302A, 302B are processed (e.g., by the image processor 412), a depth image is generated. The generated depth image can be viewed on the optical components 180A, 180B of the eyewear device, on another display (e.g., the image display 580 on the mobile device 401), or on a screen.

[0062] The generated depth image is in a three-dimensional spatial domain and may include a vertex matrix on a three-dimensional position coordinate system including an X-axis (e.g., length) of horizontal position, a Y-axis (e.g., height) of vertical position, and a Z-axis (e.g., distance) of depth. Each vertex may include a color attribute (e.g., a red pixel light value, a green pixel light value, or a blue pixel light value), a position attribute (e.g., an X position coordinate, a Y position coordinate, and a Z position coordinate), a texture attribute, a reflectivity attribute, or a combination thereof. The texture attribute quantifies the perceived texture of the depth image, such as the spatial arrangement of color or intensity in an area of ​​a vertex of the depth image.

[0063] Figure 44 is a functional block diagram of an example collaboration system 400, which includes a wearable device (e.g., eye-mounted device 100), a mobile device 401, and a server system 498 connected via various networks 495 (such as the Internet). The server system 498 can be one or more computing devices as part of a service or network computing system, for example, including a processor, a memory, and a network communication interface to communicate with the eye-mounted device 100 and the mobile device 401 through the network 495. The server system 498 includes a server processor 499, which can be configured to host a collaboration session. The functions of the eye-mounted device 100 or the mobile device 401 described herein, such as collaboration processing and providing collaboration objects to users, can be performed by the processor 499 of the server system 498.

[0064] As described herein, the eye-mounted device 100 includes one or more visible light cameras 114A, 114B that capture still images, video images, or still images and video images. Cameras 114A, 114B may have direct memory access (DMA) to high-speed circuits 430 and function as stereo cameras. Cameras 114A, 114B may be used to capture an initial depth image that may be rendered into a three-dimensional (3D) model that is a texture-mapped image of a red, green, and blue (RGB) imaging scene. The device 100 may also include a depth sensor 213 that uses infrared signals to estimate the position of an object relative to the device 100. The depth sensor in some examples includes one or more infrared emitters 215 and an infrared camera 410.

[0065] The eye-mounted device 100 also includes two image displays for each optical assembly 180A, 180B (one associated with the left side 170A and one associated with the right side 170B). The eye-mounted device 100 also includes an image display driver 442, an image processor 412, a low power circuit 420, and a high speed circuit 430. The image display of each optical assembly 180A, 180B is used to present an image, including a still image, a video image, or a still and video image. The image display driver 442 is coupled to the image display of each optical assembly 180A, 180B to control the display of the image.

[0066] The eye-mounted device 100 additionally includes one or more microphones (not shown) and one or more speakers 413 (e.g., one associated with the left side of the eye-mounted device and another associated with the right side of the eye-mounted device). The speakers 413 can be incorporated into the frame 105, the temple 125, or the corner 110 of the eye-mounted device 100. The one or more speakers 413 are driven by the audio processor 414 and the audio driver 415 under the control of the low power circuit 420, the high speed circuit 430, or both. The speakers 413 are used to present audio signals including, for example, a click track. The audio processor 414 is coupled to the microphone and the speaker 413 to control the corresponding capture and presentation of sound.

[0067] Figure 4 The components of the eye-mounted device 100 shown are located on one or more circuit boards, such as a printed circuit board (PCB) or a flexible printed circuit (FPC) located in the frame or temple. Alternatively or additionally, the components shown can be located in the corners, frame, hinges, or bridge of the eye-mounted device 100.

[0068] like Figure 4 As shown, the high-speed circuit 430 includes a high-speed processor 432, a memory 434, and a high-speed wireless circuit 436. In this example, an image display driver 442 is coupled to the high-speed circuit 430 and operated by the high-speed processor 432 to drive the left and right image displays of each optical assembly 180A, 180B. The high-speed processor 432 can be any processor capable of managing high-speed communications and operations of any general-purpose computing system required for the eye-mounted device 100. The high-speed processor 432 includes the processing resources required to manage high-speed data transmission on a high-speed wireless connection 437 to a wireless local area network (WLAN) using a high-speed wireless circuit 436.

[0069] In some examples, the high-speed processor 432 executes a LINUX operating system or other such operating system such as the eye-mounted device 100, and the operating system is stored in the memory 434 for execution. In addition to any other duties, the high-speed processor 432 also executes the software architecture of the eye-mounted device 100 for managing data transmission through the high-speed wireless circuit 436. In some examples, the high-speed wireless circuit 436 is configured to implement the Institute of Electrical and Electronics Engineers (IEEE) 802.11 communication standard, also referred to herein as Wi-Fi. In other examples, the high-speed wireless circuit 434 can implement other high-speed communication standards.

[0070] The low power circuit 420 includes a low power processor 422 and a low power wireless circuit 424. The low power wireless circuit 424 and the high speed wireless circuit 436 of the eyewear device 100 may include a short range transceiver (Bluetooth TMor Bluetooth Low Energy (BLE)) and a wireless wide area network, local area network, or wide area network transceiver (e.g., cellular or Wi-Fi). The mobile device 401 including a transceiver for communication via a low power wireless connection 425 and a high speed wireless connection 437 can be implemented using details of the architecture of the eye-mounted device 100, and other elements of the network 495 can also implement the mobile device 401.

[0071] The memory 434 includes any storage device capable of storing various data and applications, including, among other things, camera data generated by the left and right visible light cameras 114A, 114B, the infrared camera 410, the image processor 412, and the image display driver 442 to display the generated images on the image display of each optical assembly 180A, 180B. Although the memory 434 is shown as being integrated with the high-speed circuit 430, the memory 434 in other examples can be a separate element of the eye-mounted device 100. In some such examples, the circuit wiring line can provide a connection from the image processor 412 or the low-power processor 422 to the memory 434 through a chip including the high-speed processor 432. In other examples, the high-speed processor 432 can manage the addressing of the memory 434 so that whenever a read or write operation involving the memory 434 is required, the low-power processor 422 will start the high-speed processor 432.

[0072] like Figure 4 As shown, the high-speed processor 432 of the eye-mounted device 100 can be coupled to the camera system (visible light cameras 114A, 114B), the image display driver 442, the user input device 491 and the memory 434.

[0073] The output components of the eye-mounted device 100 include visual elements, such as Figure 2A and Figure 2B4. The eye-mounted device 100 may include a left and right image display associated with each lens or optical component 180A, 180B described in the embodiment of the present invention (e.g., a display such as a liquid crystal display (LCD), a plasma display panel (PDP), a light emitting diode (LED) display, a projector, or a waveguide). The eye-mounted device 100 may include an indicator facing the user (e.g., an LED, a speaker 413, or a vibration actuator), or an outward-facing signal (e.g., an LED, a speaker 413). The image display of each optical component 180A, 180B is driven by an image display driver 442. In some example configurations, the output components of the eye-mounted device 100 also include additional indicators, such as auditory elements (e.g., speakers 413), tactile components (e.g., actuators such as vibration motors that generate tactile feedback), and other signal generators. For example, the device 100 may include a set of indicators facing the user and a set of signals facing the outside. The set of indicators facing the user is configured to be seen or otherwise sensed by the user of the device 100. For example, the device 100 may include an LED display positioned so that the user can see it, one or more speakers 413 positioned to generate sounds that the user can hear, or an actuator that provides tactile feedback that the user can feel. The outward-facing set of signals is configured to be seen or otherwise sensed by an observer near the device 100. Similarly, the device 100 may include an LED, a speaker, or an actuator configured and positioned to be sensed by an observer.

[0074] The input components of the eye-mounted device 100 may include alphanumeric input components (e.g., a touch screen or touchpad configured to receive alphanumeric input, an optical keyboard, or other alphanumeric configuration elements), pointer-based input components (e.g., a mouse, touchpad, trackball, joystick, motion sensor, or other pointing instrument), tactile input components (e.g., a button switch, a touch screen or touchpad that senses the position, force, or position and force of a touch or touch gesture, or other tactile configuration elements), visual input components (e.g., a camera 114 / 420), and audio input components (e.g., a microphone), etc. The mobile device 401 and the server system 498 may include alphanumeric, pointer-based, tactile, audio, and other input components.

[0075] In some examples, the eye-mounted device 100 includes a collection of motion sensing components called an inertial measurement unit 472. The motion sensing components can be micro-electromechanical systems (MEMS) with tiny moving parts, which are usually small enough to be part of a microchip. In some example configurations, the inertial measurement unit (IMU) 472 includes an accelerometer, a gyroscope, and a magnetometer. The accelerometer senses the linear acceleration (including gravity acceleration) of the device 100 relative to three orthogonal axes (x, y, z). The gyroscope senses the angular velocity of the device 100 around three rotation axes (pitch, roll, yaw). The accelerometer and gyroscope together can provide the position, orientation, and motion data of the device relative to six axes (x, y, z, pitch, roll, yaw). The magnetometer (if present) senses the direction of the device 100 relative to magnetic north. The position of the device 100 can be determined by a position sensor such as a GPS unit, one or more transceivers that generate relative position coordinates, an altitude sensor or a barometer, and other orientation sensors. Such positioning system coordinates may also be received from mobile device 401 over wireless connections 425 , 437 via low power wireless circuitry 424 or high speed wireless circuitry 436 .

[0076] The IMU 472 may include or cooperate with a digital motion processor or program that collects raw data from the components and calculates a number of useful values ​​about the position, orientation, and motion of the device 100. For example, acceleration data collected from the accelerometer can be integrated to obtain the velocity relative to each axis (x, y, z); and integrated again to obtain the position of the device 100 (in linear coordinates x, y, and z). Angular velocity data from the gyroscope can be integrated to obtain the position of the device 100 (in spherical coordinates). The program for calculating these useful values ​​can be stored in the memory 434 and executed by the high-speed processor 432 of the eye-mounted device 100.

[0077] The eye-mounted device 100 may optionally include additional peripheral sensors, such as biosensors, specialized sensors, or display elements integrated with the eye-mounted device 100. For example, the peripheral device elements may include any I / O components, including output components, motion components, position components, or any other such components described herein. For example, the biosensor may include components that detect expressions (e.g., hand expressions, facial expressions, voice expressions, body postures, or eye tracking), measure biosignals (e.g., blood pressure, heart rate, body temperature, sweat, or brain waves), or recognize people (e.g., recognition based on voice, retina, facial features, fingerprints, or electrical biosignals such as EEG data).

[0078] The mobile device 401 can be a smartphone, tablet, laptop, access point, or any other device capable of connecting to the eye-mounted device 100 using a low-power wireless connection 425 and a high-speed wireless connection 437. The mobile device 401 is connected to a server system 498 and a network 495. The network 495 can include any combination of wired and wireless connections.

[0079] like Figure 4 As shown, the cooperation system 400 includes a computing device (such as a mobile device 401) coupled to the eye-mounted device 100 through a network. The cooperation system 400 includes a memory for storing instructions and a processor for executing instructions. The processor 432 executes the instructions of the cooperation system 400 to configure the eye-mounted device 100 to cooperate with the mobile device 401. The cooperation system 400 can utilize the memory 434 of the eye-mounted device 100 or the storage elements 540A, 540B, 540C ( Figure 5 In addition, the collaboration system 400 can utilize the processor elements 432, 422 of the eye-mounted device 100 or the central processing unit (CPU) 540 ( Figure 5 ). In addition, the collaboration system 400 can also utilize the memory and processor elements of the server system 498. In this regard, the memory and processing functions of the collaboration system 400 can be shared or distributed between the eye-mounted device 100, the mobile device 401, and the processor and memory of the server system 498 to implement the functions described herein.

[0080] In some example embodiments, the memory 434 includes or is coupled to a hand gesture library 480, as described herein. In some embodiments, the process of detecting a hand shape or gesture involves comparing pixel-level data in one or more frames of video data captured by the eye-mounted device 100 or the mobile device 401 to the hand shapes and gestures stored in the library 480 until a good match is found. A gesture can be a static gesture that can be detected in one or a few frames of data, or a dynamic gesture that can be detected over the course of two or more frames of data.

[0081] In some example embodiments, the memory 434 additionally includes an element animation application 910, a positioning system 915, an image processing system 920, and a collaboration application 925. In the collaboration system 400 where the camera is capturing frames of video data, the element animation application 910 configures the processor 432 to control the movement of a series of virtual items 700 on the display in response to detecting one or more inputs (e.g., IMU data, captured images, and hand shapes or gestures). The positioning system 915 configures the processor 432 to obtain positioning data for determining the position of the eye-mounted device 100 relative to the physical environment. The positioning data can be derived from a series of images, an IMU unit 472, a GPS unit, or a combination thereof. The image processing system 920 configures the processor 432 to cooperate with the image display driver 442 and the image processor 412 to present the captured image on the display of the optical assembly 180A, 180B. The collaboration application 925 configures the processor 432 to implement the collaboration functions described herein.

[0082] Figure 5 is a high-level functional block diagram of an example mobile device 401. The mobile device 401 includes a flash memory 540A that stores programs executed by the CPU 540 to perform all or a subset of the functions described herein.

[0083] Mobile device 401 may include camera 570 including at least two visible light cameras (first and second visible light cameras with overlapping fields of view) or at least one visible light camera and a depth sensor with substantially overlapping fields of view. Mobile device 401 may also include speaker 571. Flash memory 540A may also include multiple images or videos generated via camera 570.

[0084] As shown in the figure, the mobile device 401 includes an image display 580, a mobile display driver 582 for controlling the image display 580, and a display controller 584. Figure 5 In the example of , image display 580 includes a user input layer 591 (eg, a touch screen) over or otherwise integrated into the screen used by image display 580 .

[0085] Examples of touch screen type mobile devices that may be used include, but are not limited to, smartphones, personal digital assistants (PDAs), tablet computers, laptop computers, or other portable devices. However, the structure and operation of touch screen type devices are provided by way of example; the subject technology described herein is not intended to be limited thereto. For purposes of this discussion, Figure 5Thus, a block diagram illustration of an example mobile device 401 having a user interface is provided, the user interface including a touch screen input layer 591 for receiving input (via touch, multi-touch, or gestures, etc., by a hand, stylus, or other tool), a camera 570 for capturing images of objects (including a user's hand and potentially virtual content), and an image display 580 for displaying content.

[0086] like Figure 5 As shown, the mobile device 401 includes at least one digital transceiver (XCVR) 510, shown as a WWAN XCVR, for digital wireless communication via a wide area wireless mobile communication network. The mobile device 401 also includes additional digital or analog transceivers, such as for short-range network communication (such as via NFC, VLC, DECT, ZigBee, Bluetooth TM For example, the short-range XCVR 520 may take the form of any available two-way wireless local area network (WLAN) transceiver of a type compatible with one or more standard communication protocols implemented in wireless local area networks (e.g., one of the Wi-Fi standards under IEEE 802.11).

[0087] To generate the location coordinates of the mobile device 401, the mobile device 401 may include a global positioning system (GPS) receiver. Alternatively or additionally, the eye-mounted device 100 or the mobile device 401 may utilize one or both of the short-range XCVR 520 and the WWAN XCVR 510 to generate the location coordinates for positioning. For example, a cellular network, Wi-Fi, or Bluetooth-based TM The positioning system can generate very accurate position coordinates, especially when used in combination. Such position coordinates can be transmitted between the eye-mounted device 100 or the mobile device 401 via one or more network connections via the XCVR 510, 520.

[0088] In some examples, the mobile device 401 includes a collection of motion sensing components called an inertial measurement unit (IMU) 572 for sensing the position, orientation, and motion of the client device 401. The motion sensing components can be micro-electromechanical systems (MEMS) with tiny moving parts, which are usually small enough to be part of a microchip. In some example configurations, the inertial measurement unit (IMU) 572 includes an accelerometer, a gyroscope, and a magnetometer. The accelerometer senses the linear acceleration (including the acceleration of gravity) of the client device 401 relative to three orthogonal axes (x, y, z). The gyroscope senses the angular velocity of the client device 401 around three rotational axes (pitch, roll, yaw). The accelerometer and gyroscope together can provide the position, orientation, and motion data of the device relative to six axes (x, y, z, pitch, roll, yaw). The magnetometer (if present) senses the direction of the client device 401 relative to magnetic north.

[0089] The IMU 572 may include or cooperate with a digital motion processor or program that collects raw data from the components and calculates a number of useful values ​​about the position, orientation, and motion of the client device 401. For example, acceleration data collected from an accelerometer may be integrated to obtain velocity relative to each axis (x, y, z); and integrated again to obtain the position of the client device 401 (in linear coordinates x, y, and z). Angular velocity data from a gyroscope may be integrated to obtain the position of the client device 401 (in spherical coordinates). The program for calculating these useful values ​​may be stored in one or more storage elements 540A, 540B, 540C and executed by the CPU 540 of the client device 401.

[0090] The transceivers 510, 520 (i.e., network communication interfaces) conform to one or more of the various digital wireless communication standards used by modern mobile networks. Examples of WWAN transceivers 510 include, but are not limited to, transceivers configured to operate in accordance with Code Division Multiple Access (CDMA) and Third Generation Partnership Project (3GPP) network technologies (including, for example, but not limited to, 3GPP Type 2 (or 3GPP2) and LTE, sometimes referred to as "4G"). For example, transceivers 510, 520 provide two-way wireless communication of information, including digitized audio signals, still images and video signals, web page information for display and network-related input, and various types of mobile message communications to / from mobile device 401.

[0091] The mobile device 401 also includes Figure 5A microprocessor used as a central processing unit (CPU) as shown in the CPU 540 in FIG. A processor is a circuit having elements that are constructed and arranged to perform one or more processing functions, typically various data processing functions. Although discrete logic components can be used, these examples utilize components that form a programmable CPU. For example, a microprocessor includes one or more integrated circuit (IC) chips that contain electronic components that perform CPU functions. For example, CPU 540 can be based on any known or available microprocessor architecture, such as a reduced instruction set computing (RISC) using an ARM architecture, as is commonly used today in mobile devices and other portable electronic devices. Of course, other arrangements of processor circuits can be used to form CPU 540 or processor hardware in smart phones, laptop computers, and tablet computers.

[0092] The CPU 540 serves as a programmable main controller of the mobile device 40, which performs various operations by configuring the mobile device 401, for example, according to instructions or programs executable by the CPU 540. For example, such operations may include various general operations of the mobile device, as well as operations related to programs applied on the mobile device. Although the processor can be configured by using hard-wired logic, a typical processor in a mobile device is a general-purpose processing circuit that is configured by executing a program.

[0093] The mobile device 401 includes a memory or storage system for storing programs and data. In this example, the memory system may include flash memory 540A, random access memory (RAM) 540B, and other memory components 540C as needed. RAM 540B is used as a short-term storage device for instructions and data processed by CPU 540, for example, as a working data processing memory. Flash memory 540A generally provides long-term storage.

[0094] Thus, in the example of mobile device 401, flash memory 540A is used to store programs or instructions executed by CPU 540. Depending on the type of device, mobile device 401 stores and runs a mobile operating system through which specific applications are executed. Examples of mobile operating systems include Google Android, Apple iOS (for iPhone or iPad devices), Windows mobile, Amazon Fire OS, RIM BlackBerry OS, etc.

[0095] like Figure 5As shown, the CPU 540 of the mobile device 401 can be coupled to the camera system 570, the mobile display driver 582, the user input layer 591, and the memory 540A. The components and functions of the eye-mounted device 100 described herein can be incorporated into the mobile device 401. Likewise, the components and functions of the mobile device 401 described herein can be incorporated into the eye-mounted device 100.

[0096] The processor 432 in the eye-mounted device 100 or the processor 540 in the mobile device 401 can construct a map of the environment around the corresponding device, determine the position of the device in the mapped environment, and determine the relative position of the device to one or more objects in the mapped environment. The processor 432 / 540 can construct the map and determine the location and position information using a conventional simultaneous localization and mapping (SLAM) algorithm applied to data received from one or more sensors. The sensor data includes images received from the camera 570 or one or both of the cameras 114A, 114B, distances received from a laser rangefinder, position information received from a GPS unit, motion and acceleration data received from an IMU 472 / 572, or a combination of data from these sensors or from other sensors that provide data for determining position information.

[0097] In the context of augmented reality, SLAM algorithms are used to build and update environmental maps while tracking and updating the position of the device (or user) in the mapped environment. Various statistical methods such as particle filtering, Kalman filtering, extended Kalman filtering, and covariance intersection can be used to approximate the mathematical solution. In a system including a high-definition (HD) video camera that captures video at a high frame rate (e.g., thirty frames per second), the SLAM algorithm updates the position and mapping of the object at least as frequently as the frame rate; in other words, the mapping and position are calculated and updated thirty times per second.

[0098] Figure 6 An example physical environment 600 and elements useful when using SLAM applications and other types of tracking applications (e.g., natural feature tracking (NFT)) are depicted. Although the following example is provided with reference to the eye-mounted device 100, the example can be implemented in a mobile device 401 in a similar manner. A user 602 of the eye-mounted device 100 is present in the example physical environment 600 (in Figure 6The processor 432 of the eye-mounted device 100 uses the captured images to determine its position relative to one or more objects 604 in the environment 600, constructs a map of the environment 600 using the coordinate system (x, y, z) of the environment 600, and determines its position within the coordinate system. In addition, the processor 432 determines the head posture (roll, pitch, and yaw) of the eye-mounted device 100 in the environment by using two or more position points (e.g., three position points 606a, 606b, and 606c) associated with a single object 604a, or by using one or more position points 606 associated with two or more objects 604a, 604b, and 604c. The processor 432 of the eye-mounted device 100 can place a virtual object 608 (such as Figure 6 The key shown) is positioned within environment 600 for viewing during an augmented reality experience (main figure collaborative augmented reality experience), wherein each user has a respective augmented reality device (e.g., eye-mounted device 100 or mobile device 401).

[0099] The positioning system 915 in some examples associates a virtual marker 610a with a virtual object 608 in the environment 600. In augmented reality, markers are registered at locations in the environment to help the device complete the task of tracking and updating the location of users, devices, and (virtual and physical) objects in the mapped environment. Markers are sometimes registered to high-contrast physical objects, such as relatively dark objects, such as framed pictures 604a mounted on light-colored walls, to help cameras and other sensors complete the task of detecting markers. Markers can be pre-specified, or can be specified by the eye-mounted device 100 when entering the environment.

[0100] The tag may be encoded with the information or otherwise linked to the information. The tag may include location information, a physical code (such as a barcode or QR code; visible or hidden to the user), or a combination thereof. A set of data associated with the tag is stored in the memory 434 of the eye-mounted device 100. The set of data includes information about the tag 610a, the location (position and orientation) of the tag, one or more virtual objects, or a combination thereof. The tag location may include three-dimensional coordinates of one or more tag markers 616a, such as Figure 6The corners of the generally rectangular marker 610a shown. The marker location can be expressed relative to real-world geographic coordinates, a marker coordinate system, the location of the eye-mounted device 100, or other coordinate systems. The one or more virtual objects associated with the marker 610a can include any of a variety of materials, including still images, videos, audio, tactile feedback, executable applications, interactive user interfaces and experiences, and combinations or sequences of such materials. Any type of content that can be stored in memory and can be retrieved when the marker 610a is encountered or the marker 610a is associated with a specified marker can be classified as a virtual object in the environment. For example, Figure 6 The key 608 shown is a virtual object displayed as a still image (2D or 3D) at the marked location.

[0101] In one example, the marker 610a can be registered in memory as being located at the physical object 604a (e.g., Figure 6 In another example, the marker can be registered in memory as a specific location relative to the eye-mounted device 100.

[0102] Figure 7 , Figure 8 and Fig. 9 are illustrations of example collaboration objects 700 developed by adding virtual content 702 during collaboration in a collaboration session, which are used to describe the following Fig.10 and Fig.11 The steps of the method shown (e.g., creating a virtual time capsule). Although in many examples described herein, a box is used for the collaboration object 700, any virtual object may be selected for use as the collaboration object 700. In addition, although the illustration depicts the eye-mounted device 100 as a physical remote device, it should be understood that other physical remote devices such as the mobile device 401 may be used to implement the functions described below.

[0103] Figure 7 A perspective view of an example collaborative object 700 in the form of a box in a first state (closed) is provided, which can be manipulated in three dimensions 701 using a hand 651 (e.g., through gestures detected in the image or touch input on a touch screen) based on corresponding movements in three dimensions 681. For example, the hand 651 can be rotated to rotate the collaborative object. In some examples, an extended index finger can be detected near the collaborative object, and when the user makes a tap gesture, a corresponding auditory signal will be presented via a speaker. The position of the eye-mounted device can also be tracked in three dimensions 840 within the environment 600, so that the overlay generated for the presentation of the display 180B is more realistic. As shown, the hand 651 can be predefined as a left hand. In some embodiments, the system includes a process for selecting and setting a hand (left and right) that will be used as the hand 651 to be detected.

[0104] Figure 8 A perspective view of an example collaboration object 700 in a second state (open) is provided with associated virtual content 702 (dial 702a, vase 702b, book 702c, other virtual content 702d-702f) added during the collaboration period. The hand 652 is shown in the open position. This position of the hand 652 or the hand 651 in the relaxed position ( Figure 7 ) to the hand 652 in the open position can be set to correspond to opening the collaboration object 700, so that when this hand position or hand gesture is detected, the collaboration object 700 transitions to the open state.

[0105] Fig. 9 A perspective view of an example collaboration object 700 in a first state (closed) is provided with an outer surface of the collaboration object 700 having been added. The hand 653 is shown in a closed position. This position of the hand 653 or the hand 651 in a relaxed position ( Figure 7 ) to the hand 653 in the closed position can be set to correspond to closing the collaboration object 700, so that when this hand position or hand gesture is detected, the collaboration object 700 transitions to the closed state.

[0106] The process of detecting and tracking includes detecting a hand 651 / 652 / 653 in various poses over time in a set or series of captured video data frames. In this case, detection refers to and includes detecting a hand in only one frame of video data, as well as detecting a hand in a subset or series of frames of video data over time. Thus, in some embodiments, the process includes detecting a hand 651 in a particular pose in one or more of the captured video data frames. In other embodiments, the process includes detecting a hand 651 / 652 / 653 in various poses over time in a subset or series of captured video data frames.

[0107] Fig.10 1000 is a flowchart depicting an example method of developing a collaboration object 700 during a collaboration session of a collaboration session including multiple physically remote devices (e.g., eye-mounted devices 100, mobile devices 401, or a combination thereof). In the example, Fig.10 The steps are performed by a server system 498 (see Figure 4 ) is performed by a processor 499 of a physical remote device. In other examples, one or more steps may be performed by a combination of processors 432 and 540 of a physical remote device or a processor of a server system 498 and a physical remote device (acting as a processor implementing the steps). One or more steps shown and described may be performed simultaneously, in succession, in an order different from that shown and described, or in combination with additional steps. Some steps may be omitted, or may be repeated in some applications.

[0108] At block 1002, the processor receives user parameters for a collaboration session. In an example, the processor receives the user parameters from a physical remote device of a host user, wherein the host user specifies the user parameters through their physical remote device during a connection to the server system 498. The user parameters include identifiers of users who are allowed to access the collaboration session. The user parameters may also include access levels that identify content that individuals may access during the collaboration session. Referring now to flow chart 1100 ( Fig.11 ) describes additional details about setting and maintaining access levels.

[0109] At block 1004, the processor receives object parameters. In an example, the processor receives the object parameters from a physical remote device of a host user (or other user with an appropriate access level), wherein the user specifies the object parameters through their physical remote device during a connection period with the server system 498. The object parameters include an identifier (e.g., a URL) that identifies the object to be used as the collaboration object 700. Figure 7-Figure 9 Other object parameters may include the material of the object (e.g., cardboard, metal, glass) or a time parameter 710 that provides a time window 712 or deadline within which virtual content 702 may be added to the collaboration object 700 (after which virtual content 702 may no longer be added to the collaboration object 700).

[0110] The processor of the server system 498 can present a list of available virtual objects 702 to the physical remote device via the network 495 for the user to select through their device, and the selection is received by the processor of the server system 496. In addition, the user can send the virtual object 702 (e.g., 3D image) they generated on their physical remote device to the server system 498, wherein the processor 499 of the server system 498 designates the received virtual object 702 as the collaboration object 700 when received.

[0111] At box 1006, the processor provides access to the collaboration object 700. The processor provides access to the collaboration object 700 through a server connection with the physical remote device based on the access level associated with each device. In an example, the processor 499 develops the collaboration object 700 in response to the received object parameters and stores the collaboration object 700 in a location accessible to the physical remote device. The processor provides access to the collaboration object 700 based on the access level associated with the user of the physical remote device. When the user accesses the collaboration object 700, the processor sends a file containing the collaboration object 700 to the mobile device 401, and the physical remote device uses the file to generate an overlay for presentation on a display of the physical remote device (such as the display 180A-180B or the display 580 of the eye-mounted device 100). The user can then interact with the representation of the collaboration object 700 on their display (for example, using a browser such as Figure 7 hand gestures shown).

[0112] At block 1008, the processor receives the design parameters. The processor receives the design parameters from a physical remote device that has an appropriate level of permissions to access the collaboration object 700. In an example, the processor sends a file to the physical remote device that contains the collaboration object 700. The physical remote device generates an overlay for display, with which the user can interact to add the design parameters to the collaboration object 700. The user can then, for example, select an image (e.g., from their camera, such as camera 114A-114B or camera 570) and select a surface of the collaboration object 700, wherein upon selecting the surface, the selected image is associated with the collaboration object 700 on the physical remote device. As the user makes changes on their device, the added / changed design parameters are transmitted by the physical remote device to the server system 498 via the network 495.

[0113] At block 1010, the processor updates the collaboration object 700 in response to the design parameters. The processor updates the collaboration object 700 in response to changes received from the physical remote device via the network 495. In an example, upon receiving the added / changed design parameters from the physical remote device, the processor associates the added / changed design parameters with the collaboration object 700 in a location accessible to the physical remote device.

[0114] At block 1012, the processor receives virtual content 702. The processor receives virtual content 702 from a physical remote device with an appropriate level of permission to access the collaborative object 700. In an example, the processor sends a file to the physical remote device containing the collaborative object 700. The physical remote device generates an overlay for display, and the user can interact with the overlay to add virtual content 702 to the collaborative object 700. Then, for example, the user can add visual virtual content 702 by selecting an image (e.g., from their camera) and performing an action (e.g., dragging and dropping on the collaborative object 700 or double-clicking on the object), thereby associating the virtual content 702 with the collaborative object 700 on the physical remote device. In addition, audio virtual content can be added to the video virtual content by, for example, pressing and holding the video virtual content to speak into the microphone, wherein the audio received when the video virtual content is released is associated with the video virtual content. When the user makes changes on their device, the added virtual content 702 is transmitted to the server system 498 by the physical remote device via the network 495.

[0115] In an example, a user may associate virtual content 702 with a collaborative object 700 by, for example, dragging and dropping virtual content 702 onto a surface of collaborative object 700. In one example, using eye-mounted device 100, eye-mounted device 100 may recognize hand gestures, and a user may manipulate collaborative object 700 displayed on display 180A-180B, and select virtual content 702 via hand gestures captured and processed by eye-mounted device 100. In another example, using mobile device 401, mobile device 401 may interpret instructions received via touch screen 580 of mobile device 401. A user may manipulate collaborative object 700, and select virtual content 702 by touching / tapping touch screen 580 with their finger, and by dragging their finger to move virtual content 702 onto collaborative object 700 (which may associate virtual content 704 with collaborative object 700).

[0116] At block 1014, the processor associates the virtual content 702 with the collaboration object 700. The processor associates the virtual content 702 with the collaboration object 700 by updating the collaboration object 700 in response to changes received from the physical remote device. In an example, upon receiving the added virtual content 702 from the physical remote device, the processor associates the added virtual content 702 with the collaboration object 700 in a location accessible to the physical remote device.

[0117] At block 1016, the processor stores the collaboration object 700. The processor 499 stores the collaboration object 700 in a memory accessible to the physically remote device via the network 495 during the collaboration session.

[0118] At block 1018, the processor provides access to the collaboration object 700. The processor 499 provides access to the collaboration object 700 in memory accessible to the physically remote device via the network 495. In an example, the processor checks the credentials (e.g., user ID) of the user requesting access and allows access if the credentials match the credentials associated with the collaboration session of the collaboration object 700.

[0119] At box 1020, the processor presents the collaboration object 700. The processor 499 presents the collaboration object 700 to the physical remote device via the network 495. In one example, the processor, in response to a request from the physical remote device, sends a file including the collaboration object 700 (and associated virtual content or a link to such content) to the physical remote device that can access the collaboration session, and the physical remote device uses the file to generate an overlay including the collaboration object 700 and the associated virtual content for presentation on the display of the remote physical device. In one example, when the collaboration object 700 is placed in an open state, the associated virtual content is presented simultaneously. In another example, the associated virtual content is presented in a continuous order based on a timestamp added when the virtual content is associated with the collaboration object 700.

[0120] Fig.11 is a flowchart listing the steps of an example selective collaborative object access method. In one example, Fig.11 The steps are performed by a server system 498 (see Figure 4 ) is performed by the processor 499 of the server system 498. In other examples, one or more steps may be performed by the processors 432 and 540 of the physical remote device or a combination of the processor 499 of the server system 498 and the physical remote device (acting as the processor implementing the steps). One or more steps shown and described may be performed simultaneously, in succession, in an order different from that shown and described, or in combination with additional steps. Some steps may be omitted, or may be repeated in some applications.

[0121] At block 1102, the processor receives a user identifier. The processor receives a user identifier of a user to be associated with a collaboration session. In one example, a user (host) accesses the server system 498 using a physical remote device. Using the physical remote device, the user creates a collaboration session and specifies other users to participate in the collaboration. For example, the host (user A) may invite another user (user B) to participate in the collaboration to prepare content for another user (user C) with the intention of providing the content to the user at a later time.

[0122] At box 1104, the processor receives the user's access parameters. The processor receives the user's access parameters from the host or another user with an acceptable access level. The access parameters indicate the corresponding access level of each user to the collaborative object 700, which allows the corresponding access level of at least one user to be different from the corresponding access level of another user. For example, the host can have a first access level (allowing access to access the collaborative object 700, so as to associate virtual content 702 and view the associated virtual content 702 during the collaborative period), while another user can have a second access level to the collaborative object 700, which is lower than the first access level (for example, it only allows access after the collaborative period ends; or it allows access to the collaborative object 700 during the collaborative period, but does not allow access to the associated virtual content 702 until after the collaborative period ends). In one example, the host receives an access level that enables access by default, and the host can grant other users access rights by providing access rights to the processor 499 of the server system 498 via a physical remote device.

[0123] At block 1106, the processor maintains an access parameter table. In an example, the processor maintains a table in the cloud storage that includes an identifier for each user and their respective access levels based on the access parameters provided by the host. In an example where the host (user A; A_ID) invites another user (user B; B_ID) to collaborate on a project for another user (user C; C_ID), the processor may initially create a table including the information identified in Table 1 below:

[0124] user Permissions A_ID yes B_ID no C_ID no

[0125] In Table 1, the permission level "Yes" indicates the first level of access, and the permission level "No" indicates the second level of access. As shown in Table 1, the host (A_ID) is initially the only user with access rights that can access, for example, the virtual content 702 associated with the collaboration object 700 or add virtual content 702.

[0126] At block 1108, the processor maintains a timer. The processor may receive a timer value from the host. The timer may be used to track the time remaining in the collaboration period.

[0127] At box 1110, the processor provides access to the collaboration object 700. The processor provides users with access to the collaboration object 700 based on their respective access levels. In one example, when a user attempts to access the collaboration object 700, the processor compares the user identification (ID) of the user with the value in the table. If the user's ID (e.g., D_ID) is not found on the table, the user will not be able to access the collaboration object 700. If another user (e.g., C_ID) is found in the table, but the user has a "no" permission level, the user will only be provided with access rights commensurate with that access level. If another user (e.g., A_ID) with a "yes" permission level is found in the table, the user is provided with access to the collaboration object 700 commensurate with that access level.

[0128] At box 1112, the processor identifies the access level change. The processor identifies the access level change of at least one other user. In one example, the host grants another user (e.g., B_ID) access rights by sending an access rights change request including the user ID to be changed and the new access level to the processor 499 of the server system 498 via a physical remote device. In another example, all users with the "yes" permission level must request an access level change for a user with another permission level. The processor identifies the access level change when receiving the request. In another example, the processor can identify the change based on the expiration of the collaboration period or the preset "reveal" time (e.g., based on a monitored timer). For example, in the case where user A and user B are preparing content for user C, users A and B may initially have the "yes" permission level, and once the collaboration period has ended or the reveal time has arrived, the processor can automatically identify the access level change request of user C.

[0129] At block 1114, the processor changes the corresponding access level. In response to the access level change, the processor changes the corresponding access level of at least one other user. In response to the identified access level change (e.g., changing the B_ID permission level to "yes"), the processor updates the table as shown in Table 2 below:

[0130] user Permissions A_ID yes B_ID yes C_ID no

[0131] Table 2

[0132] Fig.12 1200 is a flowchart including steps of a method used in a collaboration application 925. The processor 499 of the server system 498 enables a user to associate virtual content 702 with a collaboration object 700 via the network 495. In one example, the collaboration object 700 is a container (e.g., Figure 7-Figure 9). In one example, the steps of flowchart 1200 are performed by multiple processors distributed between server system 498 and the user's physical remote device (such as processor 499, processor 432, and processor 540). Processor 499 associates materials with collaboration objects 700 to provide realistic interactions between the user and collaboration objects 700 in collaboration application 925. Each material assigned to collaboration object 700 has physical properties associated with the selected material. This allows the user to have a different experience with collaboration object 700 depending on the associated material. In one example, the material associated with collaboration object 700 is cardboard, where the interaction between the user and collaboration object 700 generates a response from the collaboration object that indicates the physical properties of the cardboard such as inertia, acoustics, and ductility.

[0133] At block 1202, the processor 499 provides the user with access to the collaboration object 700 during a session period. Figure 7-9 . In an example, authorization is provided to the user via the network 495 to join the session and collaborate to generate a collaboration object 700. The processor 499 provides the collaboration object 700 to a physical remote device via the network 495, which presents the collaboration object 700 to the user (e.g., as an overlay on the display 180 of the eye-mounted device 100 or the display 580 of the mobile device 401). In one example, the user cannot access the associated virtual content 702 received from other users until the collaboration period ends. In another example, a subset of users can access the associated virtual content 702 added by other users in the subset of users.

[0134] At block 1204 , the processor 499 enables the user to associate the virtual content 702 with the collaboration object 700 . Figure 7-Figure 9 In one example, users contribute to the joint collaboration by using their physical remote devices to add and modify virtual content 702 at selected locations of the collaborative object 700 and display it on the displays of their respective devices (e.g., the display 180 of the eye-mounted device 100 and the display 580 of the mobile device 401).

[0135] At block 1206, the processor 499 associates a material with the collaboration object 700. In an example, the material may be selected and changed by a user who has access to the collaboration object 700. Additionally, the processor 499 may select the material as a default material setting when there is no input from the user. The visual appearance of the collaboration object 700 indicates the material with which it is associated. In an example, the material associated with the collaboration object 700 is cardboard. The collaboration object 700 may appear to have a brown cardboard color with a matte surface with details indicating the corrugations commonly seen in cardboard materials. In another example, as Fig.13AAs shown, the material of the collaboration object 700 is metal. The collaboration object 700 may then appear to have a smooth, shiny surface 770 of metallic coloring, such as silver. The material of the collaboration object 700 may also be glass, wherein the collaboration object 700 is transparent and any virtual content 702 within the collaboration object 700 may also be visible. Other materials may be used, and the examples given are not intended to be a limiting list.

[0136] At box 1208, the processor associates physical properties with the collaboration object 700 representing the selected material. In an example, the physical properties include inertial properties, acoustic properties, and ductility properties. Inertial properties describe the material's resistance to motion or acceleration, and therefore describe the density associated with the material or the total mass associated with the collaboration object 700. Acoustic properties describe the sounds associated with a particular material. For example, a metal material is associated with a metallic clanking sound, a cardboard material is associated with the sound of cardboard being folded and knocked, a glass material is associated with a clanking sound such as two glasses hitting each other, and an elastic material is associated with a sound indicating rubber stretching. Ductile properties describe the response of the associated material to deformation of the collaboration object 700. For example, a rubber material is associated with allowing the collaboration object 700 to be stretched 780, such as Fig. 13B As shown, the cardboard material can be folded or wrinkled and then remain deformed, and the metal material can be as shown in 772 ( Fig.13A ) as shown and remains deformed.

[0137] At block 1210, the collaborative object 700 is manipulated in its environment. Manipulation may occur due to an action by a user (such as interacting with the collective object 700) or due to an event created by a processor (such as the processor implementing a predefined modification of an object). In one example, the user moves the position of the collaborative object 700 from a first position to a second position, or from a first state to a second state. In another example, the collaborative object 700 is a box, and manipulation occurs when virtual content 702 is added to the box.

[0138] At box 1212, the collaborative object 700 responds to the manipulation representing the physical properties associated by the material. In the example where the user moves the position of the collaborative object 700, the collaborative object 700 moves at a speed representing the manipulation and the inertial mass of the material. For example, a collaborative object 700 associated with a metal material has a higher inertia value than a collective object associated with a cardboard material. Therefore, the movement and acceleration of the metal material will be slower than the cardboard material. In the example where the virtual content 702 is added to the box (collaborative object 700), when the virtual content 702 is added, the metal material produces a metallic jingle, while the cardboard material produces a sound indicative of cardboard. In another example, if the collaborative object 700 is associated with a cloth-like material, the bottom of the material will sag when the virtual content 702 is associated to the inside of the collaborative object 700.

[0139] Another example of a response indicative of the associated material of the collaboration object 700 is when the material is glass. If the collaboration object 700 is manipulated in a rough manner, the collaboration object 700 may appear cracked or shattered due to its associated material being glass.

[0140] Machine learning refers to algorithms that gradually improve through experience. By processing a large number of different input data sets, machine learning algorithms can develop improved generalization properties about a specific data set, and then use these generalization properties to produce accurate outputs or solutions when processing new data sets. In general, machine learning algorithms include one or more parameters that are adjusted or changed based on new experience, thereby gradually improving the algorithm; similar to the process of learning.

[0141] In the context of computer vision, mathematical models attempt to simulate the tasks performed by the human visual system, with the goal of using computers to extract information from images and achieve an accurate understanding of the image content. Computer vision algorithms have been developed for use in a variety of fields, including artificial intelligence and autonomous navigation, to extract and analyze data from digital images and videos.

[0142] Deep learning refers to a class of machine learning methods based on artificial neural networks or artificial neural networks as models. Artificial neural networks are computing systems composed of many simple, highly interconnected processing elements (nodes) that process information through their dynamic responses to external inputs. Large artificial neural networks can have hundreds or thousands of nodes.

[0143] A convolutional neural network (CNN) is a type of neural network that is often used to analyze visual images, including digital photos and videos. The connection patterns between nodes in a CNN are often modeled after the organization of the human visual cortex, which includes individual neurons arranged to respond to overlapping areas in the field of view. A neural network suitable for the determination process described in this article is based on one of the following architectures: VGG16, VGG19, ResNet50, InceptionV3, Xception, or other CNN-compatible architectures.

[0144] In the machine learning example, at blocks 1008 and 1012, the processor 432 determines whether the detected series of hand shapes substantially matches a predefined hand gesture using a machine trained algorithm called a hand feature model. The processor 432 is configured to access the hand feature model trained by machine learning and apply the hand feature model to identify and locate features of hand shapes in one or more frames of video data.

[0145] In one example implementation, the trained hand feature model receives a video data frame containing a detected hand shape and extracts the image in the frame into layers for analysis. Based on the trained hand feature model, the data in each layer is compared with the hand gesture data stored in the hand gesture library 480 layer by layer until a good match is identified.

[0146] In one example, a layer-by-layer image analysis is performed using a convolutional neural network. In the first convolutional layer, CNN identifies learned features (e.g., hand landmarks, joint coordinate groups, etc.). In the second convolutional layer, the image is converted into multiple images, wherein each learned feature is emphasized in the corresponding sub-image. In the pooling layer, the size and resolution of the image and its sub-images are reduced to isolate the parts of each image that may contain features of interest (e.g., possible palm shapes, possible finger joints). The values ​​and comparisons of the images from the non-output layers are used to classify the images in the frame. As used herein, classification refers to the process of classifying images using a training model based on the detected hand shapes. For example, if a series of detected two-handed hand shapes match the touch gestures stored in library 480, the image can be classified as a "touch action".

[0147] As described herein, any functions of the eye-mounted device 100, mobile device 401, and server system 498 described herein can be embodied in one or more computer software applications or programming instruction sets. According to some examples, "function", "multiple functions", "application", "applications", "instruction", "instructions", or "programming" is a program that executes the functions defined in the program. Various programming languages ​​can be used to develop one or more of the applications constructed in various ways, such as object-oriented programming languages ​​(e.g., Objective-C, Java, or C++) or procedural programming languages ​​(e.g., C or assembly language). In a specific example, a third-party application (e.g., an application developed by an entity other than the vendor of a particular platform using ANDROID TM or IOS TM Software Development Kit (SDK) applications can be included in iOS TM ANDROID TM , Mobile software running on a mobile operating system such as iPhone or other mobile operating systems. In this example, third-party applications can call API calls provided by the operating system to facilitate the functions described herein.

[0148] Therefore, machine-readable media can adopt tangible storage media in various forms. Non-volatile storage media include, for example, optical disks or disks, such as any storage device in any computer device, such as can be used to implement client devices, media gateways, code converters, etc., as shown in the figure. Volatile storage media include dynamic memory, such as the main memory of such a computer platform. Tangible transmission media include coaxial cables, copper wires and optical fibers, including wires that form the bus in the computer system. Carrier transmission media can take the form of electrical signals or electromagnetic signals, or the form of sound waves or light waves generated during radio frequency (RF) and infrared (IR) data communication. Therefore, the common form of computer-readable media includes, for example: floppy disk, flexible disk, hard disk, magnetic tape, any other magnetic medium, CD-ROM, DVD or DVD-ROM, any other optical medium, punched card tape, any other physical storage medium with hole patterns, RAM, PROM and EPROM, FLASH-EPROM, any other storage chip or box, a carrier wave that transmits data or instructions, a cable or link that transmits such a carrier wave, or any other medium from which a computer can read programming code or data. Many of these forms of computer readable media may be involved in carrying one or more sequences of one or more instructions to a processor for execution.

[0149] Except as stated above, nothing stated or described is intended or should be construed as conferring any element, step, feature, object, benefit, advantage, or equivalent to the public, regardless of whether it is recited in the claims.

[0150] It should be understood that, unless otherwise provided herein, the terms and expressions used herein have the common meaning consistent with these terms and expressions in their respective investigation and research fields. Relational terms such as first and second can be used only to distinguish an entity or action from another entity or action, without requiring or implying any actual such relationship or order between these entities or actions. The term "comprises", "comprising", "includes", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including or comprising a series of elements or steps not only includes those elements or steps, but also can include other elements or steps that are not clearly listed or inherent to such process, method, article or equipment. In the absence of further restrictions, the element starting with "a" or "an" does not exclude the existence of additional identical elements in the process, method, article or device including the element.

[0151] Unless otherwise indicated, any and all measurements, values, levels, positions, amplitudes, dimensions and other specifications listed in this specification (including the appended claims) are approximate and not exact. These quantities are intended to have a reasonable range that is consistent with the functions to which they are related and the customary practices in the field to which they belong. For example, unless expressly stated, parameter values, etc. may vary by plus or minus ten percent from the stated amount or range.

[0152] Furthermore, in the foregoing detailed description, it can be seen that various features are grouped together in various examples for the purpose of streamlining the disclosure. This method of disclosure should not be interpreted as reflecting an intention that the claimed examples require more features than are expressly recited in each claim. On the contrary, as the following claims reflect, the subject matter to be protected is less than all features of any single disclosed example. Therefore, the following claims are hereby incorporated into the detailed description, with each claim standing on its own as separately claimed subject matter.

[0153] Although the best mode and other examples have been described above, it should be understood that various modifications may be made thereto, and the subject matter disclosed herein may be implemented in various forms and examples and may be applied to a variety of applications, only some of which are described herein. The following claims are intended to claim any and all modifications and variations that fall within the true scope of the present concept.

Claims

1. A system comprising a server and a client, the system having a processor configured to perform an interactive augmented reality method, the processor being configured to: Providing users with access to collaborative objects; associating virtual content received from the user with the collaboration object; associating the material with said collaborative object; associating a physical property with said collaboration object representing said material; receiving a manipulation instruction for the collaboration object; A response to the manipulation instruction is generated, wherein the response to the manipulation instruction is representative of the physical property.

2. The system according to claim 1, wherein: The processor is configured to, in response to the association of the virtual content, modify the object in a manner corresponding to a physical attribute of the collaboration object.

3. The system according to claim 1, wherein: The collaboration object is a virtual container configured to contain the virtual content.

4. The system according to claim 1, wherein: The processor is configured to receive a material selection of the material from one or more of the users.

5. The system according to claim 2, wherein: The physical property is an inertial property, wherein the cooperating object is modified according to the inertial property by moving the cooperating object.

6. The system according to claim 2, wherein: The physical property is an acoustic property, wherein the collaboration object is altered according to the acoustic property by generating a sound associated with the collaboration object.

7. The system according to claim 2, wherein: The physical property is a ductility property, wherein the cooperating object is altered according to the ductility property by deforming the cooperating object.

8. The system according to claim 1, wherein: The processor is configured to allow users to access the collaboration object using respective physically remote devices.

9. The system according to claim 8, wherein: The processor is configured to provide the collaboration object to the physically remote device.

10. An interactive augmented reality method performed by a processor, the method comprising: Providing users with access to collaborative objects; associating virtual content received from the user with the collaboration object; associating the material with said collaborative object; associating a physical property with said collaboration object representing said material; receiving a manipulation instruction for the collaboration object; as well as A response to the manipulation instruction is generated, wherein the response to the manipulation instruction is indicative of the physical property.

11. The method according to claim 10, wherein: The processor modifies the collaboration object in a manner corresponding to a physical attribute of the collaboration object in response to the association of the virtual content.

12. The method according to claim 10, wherein: The collaboration object is a virtual container configured to contain the virtual content.

13. The method according to claim 10, wherein: The processor receives a material selection of the material from one or more of the users.

14. The method according to claim 11, wherein: The physical property is an inertial property, wherein the cooperating object is modified according to the inertial property by moving the cooperating object.

15. The method according to claim 11, wherein: The physical property is an acoustic property, wherein the collaboration object is altered according to the acoustic property by generating a sound associated with the collaboration object.

16. The method according to claim 11, wherein: The physical property is a ductility property, wherein the cooperating object is altered according to the ductility property by deforming the cooperating object.

17. The method according to claim 10, wherein: The users access the collaboration object using respective physical remote devices, and the method further comprises: The collaboration object is provided to the physically remote device.

18. A non-transitory computer readable medium storing program code which, when executed by an electronic processor of a system including a server and a client, configures the processor to: Providing users with access to collaborative objects; associating virtual content received from the user with the collaboration object; associating the material with said collaborative object; associating a physical property with said collaboration object representing said material; receiving a manipulation instruction for the collaboration object; A response to the manipulation instruction is generated, wherein the response to the manipulation instruction is indicative of the physical property.

19. The non-transitory computer readable medium of claim 18, wherein: The code configures the processor to modify the collaboration object in a manner corresponding to a physical property of the collaboration object in response to the association of the virtual content.

20. The non-transitory computer readable medium of claim 19, wherein: The code configures the processor to receive a material selection of the material from one or more of the users.

Images