Augmented Reality

Abstract

An augmented reality (AR) system receives multiple local 3D models of a portion of a real-world environment, each of which has been generated by a different AR device when located in the real-world environment. The local 3D models are combined to generate a global 3D model, at least a portion of which is sent to a device remote from the real-world environment. The global 3D model individually represents a larger portion of the real environment than any of the local 3D models. The AR system receives rendering data from a remote device and sends it to the AR device when the AR device is located in the real-world environment. Alternatively, the rendering data can be sent directly from the remote device to the AR device via a network. The rendering data is used when rendering a virtual object at the AR device in the real-world environment.

Description

[0001] This application is a divisional application of the invention patent application "Augmented Reality" with application date of August 19, 2016 and application number 201680048059.5. Background Art

[0002] The display system can be used to make a desired image visible to the user. The wearable display system can be implemented in a wearable head-mounted device, which can be configured to display images within a short distance from the human eye. Such a wearable head-mounted device may sometimes be referred to as a head-mounted display. The optical components are arranged in the wearable headband to display the desired image within a few centimeters of the user's eyes.

[0003] Some types of display systems provide a view through the optics so that the generated image displayed to the user is overlaid on the real world view. This is called augmented reality ("AR"), or equivalently, mixed reality. An example is the Microsoft HoloLens device.

[0004] As with wearable display systems, augmented reality devices may include other components, such as a processing unit, a computer interface to provide network connectivity, and a camera, etc. These components may be housed in the headband, such as in the HoloLens, or in a separate housing connected to the headband by wireless or wired means. Summary of the invention

[0005] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0006] In one aspect, an augmented reality system includes a computer interface, a global model generation module, and an augmentation sharing module.

[0007] The computer interface is configured to receive multiple local 3D models of a portion of a real-world environment, wherein each local 3D model has been generated by a different augmented reality device when located in the real-world environment. The global model generation module is configured to combine the local 3D models to generate a global 3D model, and send at least a portion of the global 3D model to a device away from the real-world environment. The global 3D model individually represents a larger portion of the real environment than any local 3D model in the local 3D models. This allows a remote device to remotely explore and enhance a virtual version of the real-world environment. The enhanced sharing module is configured to receive rendering data from a remote device and send it to the augmented reality device when the augmented reality device is located in the real-world environment. The rendering data is used when rendering a virtual object at the augmented reality device in the real-world environment.

[0008] In another aspect, the rendering data may be sent directly from the remote device to the AR device via a network (ie, not via the augmented reality system). BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to better understand the subject matter and to show how embodiments of the subject matter may be implemented, reference is made to the following drawings, in which:

[0010] Figure 1A shows a perspective view of an augmented reality device, and Figure 1B A block diagram of the device is shown;

[0011] Figure 2 The functional modules of the augmented reality device are shown;

[0012] Figure 3 A communication system is shown;

[0013] Figure 4 A method of enabling AR-based communication between a user and another user is shown;

[0014] Figure 5 An improved method of enabling AR-based communication between a user and another user according to the present subject matter is shown;

[0015] Figure 6 An exemplary environment is shown in which the theme may be applied. DETAILED DESCRIPTION

[0016] Figure 1A A perspective view of a wearable augmented reality (“AR”) device 2 is shown from the perspective of a wearer 22 of the device 2 (“AR user”). Figure 1B A schematic block diagram of an AR device 2 is shown. The AR device 2 is a computer device in the form of a wearable head-mounted device. Figure 1Aor Figure 1B , but is shown in some of the subsequent figures.

[0017] The augmented reality device 2 includes a headband 6, which is a headband arranged to be worn on the wearer's head. The headband 6 has a central portion 4 intended to fit over the wearer's nose bridge, and has an inward curve intended to wrap around the wearer's head over the wearer's ears.

[0018] The headband 3 supports left and right optical assemblies, designated 10L and 10R, which are waveguides. For ease of reference herein, the optical assembly 10 will be considered to be either the left or right assembly, since the assemblies are essentially identical except that they are mirror images of each other. Thus, all descriptions that apply to the left-hand assembly also apply to the right-hand assembly. The central portion 4 houses at least one light engine 17, which is not Figure 1A but in Figure 1B Depicted in.

[0019] The light engine 17 includes a microdisplay and imaging optics in the form of a collimating lens (not shown). The microdisplay may be any type of image source, such as a liquid crystal on silicon (LCOS) display, a transmissive liquid crystal display (LCD), a matrix array of LEDs (whether organic or inorganic), and any other suitable display. The display is provided by Figure 1A and Figure 1B The optical engine 4 is driven by invisible circuitry that activates individual pixels of the display to generate an image. The substantially collimated light from each pixel falls on the exit pupil of the light engine 4. At the exit pupil, the collimated light beam is coupled into each optical component 10L, 10R and in turn coupled into a corresponding coupling region 12L, 12R provided on each component. Figure 1A 14L and 14R, and also downwardly to respective exit regions 16L and 16R where it leaves the assembly 10 to the user's eye. The optical assembly 10 is located between the light engine 13 and the eye, i.e. the display system configuration is of a so-called transmissive type.

[0020] The collimating lens calibrates the image into multiple beams, which form a virtual version of the displayed image, which is a virtual image at infinity that is optically sensed. The light leaves as multiple beams that correspond to the input beams and form substantially the same virtual image, which the lens of the eye projects onto the retina to form a real image visible to the AR user 22. In this way, the optical assembly 10 projects the displayed image onto the wearer's eyes. The optical assemblies 10L, 10R and the light engine 17 constitute the display device of the AR device 2.

[0021] The regions 12, 14, 16 may be, for example, appropriately arranged diffraction gratings or holograms.The optical component 10 has a refractive index n such that total internal reflection occurs to direct a light beam from the light engine along the intermediate expansion region 314 and downwardly towards the exit region 316.

[0022] The optical assembly 10 is substantially transparent so that the wearer can see through it to view their real world surroundings simultaneously with the projected image, thereby providing an augmented reality experience.

[0023] In order to provide a stereoscopic image (i.e., an image perceived by the user as having a 3D structure), slightly different versions of the 2D image may be projected onto each eye, for example from different light engines 17 in the central portion 4 (i.e., two microdisplays) or from the same light engine (i.e., one microdisplay), using appropriate optics to separate the light output from a single display.

[0024] exist Figure 1A The wearable AR device 2 shown in is merely an example configuration. For example, where two light engines are used, they may alternatively be located at separate locations on the right and left sides of the device (close to the wearer's ears). Furthermore, while in this example the input beam forming the virtual image is generated by calibrating the light from the display, an alternative light engine based on so-called scanning may replicate the effect with a single beam whose orientation is rapidly adjusted while simultaneously adjusting its intensity and / or color. The virtual image may be simulated in a manner equivalent to a virtual image created by calibrating the light of a (real) image on a display using collimating optics. Alternatively, a similar AR experience may be provided by embedding substantially transparent pixels in a glass or polymer plate in front of the wearer's eyes, having a similar configuration to the optical assemblies 10A, 10L, but without the need for the regional structures 12, 14, 16.

[0025] Other headbands 6 are also within the scope of the present subject matter. For example, the display optics can be equally attached to the user's head using a frame (in the manner of traditional glasses), a helmet, or other fitting systems. The fitting system can be designed to meet user groups across anthropometric ranges and head morphologies, and provide comfortable support for the display system.

[0026] The AR device 2 also includes one or more cameras 19 - stereo cameras 10L, 10R fixed to the headband 3 and configured in this example to capture an approximate field of view ("field of view") from the user's left and right eyes, respectively. The cameras are placed on the headband 3 to either side of the user's head, and thus capture images of the scene in front of the device from slightly different perspectives. In combination, the stereo cameras capture stereoscopic moving images of the real-world environment as the device moves through the real-world environment. Stereoscopic moving images refer to two moving images of slightly different perspectives of the same scene, each formed by a time sequence of frames that are broadcast closely to replicate the motion. When combined, the two images give the impression of a moving 3D structure.

[0027] As in Figure 1B As shown in FIG. 1 , the AR device 2 further includes: one or more speakers 11; one or more microphones 13; a memory 5; a processing device in the form of one or more processing units 3 (e.g., a CPU, a GPU, and / or a customized processing unit optimized for a specific function such as an AR-related function); and one or more computer interfaces for communicating with other computer devices, such as a Wifi interface 7a, a Bluetooth interface 7b, etc. The wearable device 3 may include other components not shown, such as a dedicated depth sensor, additional interfaces, etc.

[0028] As in Figure 1A As shown in FIG, a left microphone 11L and a right microphone 13R are located at the front of the headband (from the wearer's perspective), while left and right channel speakers, headphones, or other audio output transducers are located on the left and right sides of the headband 3. These are in the form of a pair of bone conduction audio transducers 11L, 11R, which act as left and right audio channel output speakers.

[0029] Despite Figure 1AAs is not apparent in the figure, the processing device 3, the memory 5, and the interfaces 7a, 7b are encapsulated in the headband 3. Alternatively, these may be encapsulated in a housing separate from the components connected to the headband 3 by wired and / or wireless means. For example, the separate housing may be designed to be worn or a belt, or fit in the pocket of the wearer, or one or more of these components may be encapsulated in a separate computer device (e.g., a smartphone, tablet computer, laptop or desktop computer, etc.), which wirelessly communicates with the display and camera device in the AR wireless headset 2, whereby the wireless headset and the separate device constitute an augmented reality device.

[0030] The memory holds executable code 9 which the processor means 3 is configured to execute. In some cases, different parts of the code 9 may be executed by different processing units of the processor means 3. The code 9 includes code for an operating system, and code for one or more applications configured to run on the operating system.

[0031] Figure 2 1 shows a functional block diagram of an AR device 2. An AR system 40 is shown, which includes the following functional modules: a rendering module 42, a local enhancement module 44, a local modeling module 46, a model enhancement module 48, a device tracking module 50, an encoder 52, and a decoder 54. Each of these functional modules can be implemented in software, that is, they can be code modules of code 9, dedicated hardware of the AR device 2, or a combination of software and dedicated hardware.

[0032] The rendering module 42 controls the light engine 17 to generate stereoscopic images visible to the wearer, ie to generate slightly different 2D images projected by the optical assemblies 10L, 10R onto different eyes substantially simultaneously, so as to create the impression of a 3D structure.

[0033] The stereoscopic image is formed by a rendering module 42 rendering at at least one virtual display element (“augmented”) that is perceived as a 3D element, ie having a 3D structure as perceived by the AR user in the real world in 3D space.

[0034] The enhancement is defined by rendering data in the form of an enhancement data object stored in the memory 5. The data object comprises: position data defining the desired position of the virtual element in 3D space (e.g. as (x, y, z) Cartesian coordinates); structure data defining the 3D surface structure of the virtual element, i.e. a 3D model of the virtual element; and image data defining a 2D surface texture of the virtual element to be applied to the surface defined by the 3D model. The data object may include additional information, such as the desired orientation of the enhancement.

[0035] The perceived 3D effect is achieved by appropriate rendering of the augmented data objects. To give the impression of augmentations with 3D structures, a stereo image is generated based on the 2D surfaces in the data objects and the 3D augmentation model data, where the augmentations are rendered to appear at the desired locations in the stereo image.

[0036] The 3D model of the portion of the real world near the AR device 2 is used to give the impression of the real world having the desired tangible effect on the augmentation in the way it would be a real world object. The 3D model represents the surface structure that exists in the real world, and the information it provides about the surface structure allows the augmentation to be displayed as if it were a real world 3D object, thereby providing an immersive augmented reality experience. The 3D model is in the form of a 3D mesh.

[0037] For example, based on a model of the real world, one can give the impression of being obscured by an augmentation that is, from the user's perspective, in front of their perceived location; dynamically interact with real-world objects, such as by moving around the objects; statically interact with them, such as by sitting on top of the real-world objects, etc.

[0038] Whether or not real-world surface structures affect, augmentation can be determined based on suitable rendering criteria. For example, by creating a 3D model of the perceived AR world that includes the real-world surface structures and any augmentations, and projecting it onto a plane along the line of sight of the AR user determined using gesture tracking (see below), a suitable criterion for determining whether a real-world object should be perceived as partially obscuring an augmentation is whether the projection of the real-world object in the plane overlaps with the projection of the augmentation, which can be further refined to take into account transparent or opaque real-world structures. In general, the criteria may depend on the position and / or orientation of the augmented reality device 2, and / or the real-world structure in question.

[0039] Enhancements may also be mapped to a grid, in the sense that their desired position and / or orientation is defined relative to certain structures in the grid. If the structures move and / or rotate to cause corresponding changes in the grid, this will cause corresponding changes in the position and / or orientation of the enhancements when rendered appropriately. For example, the desired position of an enhancement may be defined on and relative to a tabletop structure; if the table is moved, the enhancement moves with it. Object recognition may be used for this purpose, for example to recognize a table of known shape and thereby use its recognizable structure to detect when the table is moved. Such object recognition techniques are known in the art.

[0040] An enhancement that is mapped to a mesh or otherwise associated with a segment of a surface structure implemented in a 3D model in this way is referred to as an "annotation" of that segment of the surface structure. In order to annotate a segment of a real-world surface structure, it is necessary that the surface structure is represented by the 3D model in question - without this, the real-world structure cannot be annotated.

[0041] The local modeling module 46 generates a local 3D model “LM” of the environment in the memory 5 using the AR device's own sensors (eg, the camera 18 ) and / or any dedicated depth sensors, etc. The local modeling module 46 and the sensors constitute the sensing means of the AR device 2 .

[0042] For reasons explained in detail below, an externally generated 3D model “GM” (“Global Mesh”) of the same environment is received from the network 26 .

[0043] The model aggregation module selectively uses data of the local module LM and data of the externally generated model GM to generate an aggregated 3D model "AM" for use by the rendering component 42 when rendering the enhancements. That is, the enhancements are rendered based on at least the local mesh data LM and the global mesh data GM.

[0044] The device tracking module 50 uses local sensor data captured at the AR device 2 to track the position and orientation of the AR device 2. The sensor data can be captured in a variety of ways, such as using the camera 18 and / or other sensors of the AR device 2. The tracking module indicates the current position and orientation of the AR device 2 to the rendering module 42, for example, by outputting a current "pose vector" of the calculated AR device. The pose vector is a six-dimensional vector, such as (x, y, z, P, R, Y), where (x, y, z) are the Cartesian coordinates of the device about a suitable origin, and (P, R, Y) are the pitch, yaw, and roll angles of the device about a suitable reference axis.

[0045] The rendering module 42 adjusts the augmentations based on the tracking to take into account the movement of the device, i.e. to maintain the perception of 3D elements occupying the real world, for example ensuring that static augmentations appear to remain static (which would actually be achieved by scaling or rotating them as the environment moves relative to them from the perspective of the AR user).

[0046] Returning briefly to FIG. 1 , code 9 includes a communication client 36 (see FIG. 1 ) for implementing a communication event in the communication system 20 via the communication network 26. Figure 3When executed on the AR device 2, the communication client 36 enables the AR device 2 to operate over the network 26 to establish a real-time video communication event with other users of the communication system 20 (who operate their own devices running their own versions of the communication client software), such as a real-time video call, instant messaging session, screen sharing or whiteboard session, etc.

[0047] Figure 3 Another user 30 is shown, who is a remote user from the perspective of the AR user 22. The remote user 30 is operating a user device ("partner device") 32, which is shown connected to the network 26 and is executing its own version of client software 37 ("partner client"). The partner device 32 may be, for example, a smartphone, a tablet computer, a laptop or desktop computer device, a smart TV or set-top box, a game controller, etc.

[0048] Call audio and video data are sent and received over the network 26 during an established communication event. The communication is "real time" in the sense that there is only a short delay (e.g., about 2 seconds or less) between the audio / video captured at the near-end device and the audio / video output by the far-end device. The client can be, for example, a stand-alone software application that executes on a processor of the associated device and runs on an operating system, or a plug-in to another application (e.g., a web browser) that executes on the processor.

[0049] The communication system may be a voice or video over Internet Protocol (VoIP) system. These systems are beneficial to users because they often have significantly lower costs than traditional fixed-line or mobile cellular networks, particularly for long-distance communications. The client software 36, 37 establishes a VoIP connection and provides other functions, such as registration and user authentication, based on, for example, login credentials (e.g., a username and associated password).

[0050] return Figure 2 , Figure 3 The encoder and decoder modules 53, 54 represent functions implemented by executing the AR client 36 on the processing device 3. The encoder module 52 receives image data from the camera 18 and audio data from the microphone 13, and sends these data as an outbound data stream 55 to the remote partner device 32 via the network 26. The decoder 54 receives the incoming data stream 53 from the remote device 32 and extracts the call audio and video data of the remote user therefrom.

[0051] The AR client 36 and the partner client 37 are able to include AR elements into real-time communication events.

[0052] In a video call or other video-based communication event, the AR client 36 displays the video elements of the communication event as 3D enhancements. For example, the 3D enhancements may be rendered visible to the AR user of the AR device 2, containing the remote user's call video, or an image of what is currently displayed on their screen or shared whiteboard. For example, the element may be rendered so that it is perceived as being projected on a real-world surface (e.g., a table or wall); or it may be rendered as an apparently independent element at a location that the AR user may walk toward, walk away from, or walk around. The decoder 54 generates one or more call enhancement data objects "CA" from the video data (which is extracted from the incoming data stream), which the decoder 54 provides to the rendering module 42. The 2D surface data of the call object CA includes at least a portion of the video data, which enables the video data to be displayed as a surface texture of the call object CA when rendered by the rendering module 42.

[0053] The remote user can also add their own enhancements, such as annotations, via the user interface of the partner device 32 by creating a suitable augmentation data object “RA” at the partner device 32 via the user interface of the partner client 37. The data object RA is sent to the AR device 2 in the incoming video stream 53. The decoder 54 extracts the remotely generated augmentation data object RA and provides it to the rendering module 42 for rendering in the same manner.

[0054] To be able to add annotations in this way, the partner device 32 also needs access to the surface structure of the part of the environment being annotated. The decoder 54 therefore includes in the outgoing stream 55 a version of the local mesh LM.

[0055] AR users can also create their own augmentations via the user interface of the AR device 2. The user interface of the AR device may include, for example, a "natural" user interface (NUI). The NUI enables users to interact with the device in a natural way without being subject to artificial constraints imposed by input devices such as a mouse, keyboard, remote control, etc. Examples of NUI methods include those that utilize the following technologies: touch-sensitive displays, voice and speech recognition, intent and target understanding, motion gesture detection using depth cameras (e.g., stereo or time-of-flight camera systems, infrared camera systems, RGB camera systems, and combinations thereof), motion gesture detection using accelerometers / gyroscopes, facial recognition, 3D displays, head, eye and gaze tracking, etc.

[0056] In response to the user input, the local enhancement module 44 creates a local enhancement data object "LA" according to the user input. The data object LA is provided to the rendering module 42 for rendering in the same manner as the call and remote data objects CA, RA. The local enhancement object is also provided to the decoder 52, which includes it in the external data stream.

[0057] In a traditional video call, each device would normally capture video of its respective user and send it to the other call participants. However, here, the AR client 36 instead captures a moving image of the real-world view currently visible to the AR user, as captured using the cameras 18L, 18R on the headband 3, and sends the moving image as a video signal to the partner device 32 via the network 26. The partner client 37 receives the video signal and displays it on the display of the partner device 32. The partner client 37 also renders any augmentations currently perceived by the AR user created by the remote user 30 in the following manner: they are currently perceived, and displays them on the display 34 superimposed on the real-world image to provide a complete representation of the AR "world" to the AR user.

[0058] The local data objects AR sent to the partner device 32 in the outbound data stream are rendered by the partner device and overlaid on the real world image on the display 34. The pose vector of the device as generated by the tracking module 50 is also transmitted to the partner device 32 in the outbound stream for use by the partner device in displaying the AR user's world.

[0059] As with the real-time communication event functionality, the clients 36, 37 also provide for AR-based asynchronous communication, where a partner device user 30 can view the AR user's representation of the world "offline" (i.e., when the AR user is not located in the environment), and add annotations using the 3D model of the environment, which become visible to the AR user via their head mounted device 2 after a period of time of their choosing. The remote user's annotation data objects are uploaded to the backend computer system 28, and can be downloaded by the AR device 2 when the AR device is later activated in the environment.

[0060] return Figure 3 , a backend computer system ("augmented reality system") 28 is shown connected to the network 26 and is formed by one or more servers. The backend computer system 28 is a cloud system in this example, ie implemented on a cloud computing platform based on hardware virtualization.

[0061] Also connected to the network 26 are a plurality of AR devices 2a, 2b of the type described above. Each is worn by a respective AR user 22a, 22b located in the same real-world environment 21 as the other. The present subject matter is particularly, but not exclusively, useful in the context of a large environment 21 occupied by a relatively large number of users 22a, 22b over time (at the same time or at different times), such as a football stadium (and the users 22a, 22b may be watching a football game), a museum, a gallery, etc.

[0062] Through the 3D data collection of multiple AR devices 2, the system can achieve a fully immersive scaled-up 3D view of the environment 21 with 3D annotations and augmented reality content. Another application of device awareness is the real-time sharing and editing of 3D CGI in the film industry.

[0063] Figure 6 An exemplary environment 21 is shown in which a large number of AR users 22 gather about a large real-world structure 60 with complex surface geometry. By combining the local mesh data LM from all users' AR devices, a complex 3D model of the structure 60 can be generated.

[0064] This disclosure addresses the problem of having incomplete 3D mesh data for a space 21. A single AR device 2 may only capture a limited amount of 3D data.

[0065] The mixed reality space can be extended with this data by using the local mesh LM collected from a set of AR devices 2a, 2b, ... in a common space 21. This allows users to extend 3D interactions across larger spaces beyond current capabilities.

[0066] This also allows AR device users 22 with access to cloud data to be able to interact with a wide variety of large augmented reality spaces.

[0067] The data may be collected by multiple users 22 collectively but asynchronously (ie, at different times, such as different hours, days, weeks, months, etc.).

[0068] By collecting data in the cloud, partner devices 32 can also access and explore the shared space.

[0069] Multiple AR devices 2a, 2b collect 3D volumetric data of a space 21 at one time (in the form of their local meshes LM) for upload to a cloud system 28. By collecting data from multiple AR users 22a, 22b of the same space 21, it is possible to stitch the data together based on a common reference point at the cloud system 28. When an AR user 22 accesses data of a space from the cloud system 28, it is possible to identify relevant data and allow the user 22 to explore a large number of large spaces as needed. The depth perception of the AR device 22 will be limited in practice, but more details of the space 21 can be rendered in more detail as the user moves through the space 21 (see below).

[0070] This is achieved through the following mechanisms implemented at the cloud system:

[0071] 3D and real-world scene capture of the space by multiple AR devices 22a, 22b;

[0072] identifying that the collection of data (i.e., different local grids LM) is related by a common reference point such as location (e.g., latitude and longitude data), time, date, etc.;

[0073] The data sets (ie local meshes LM) are stitched together and cover the common points. That is, multiple local models LM are combined to form a global model GM based on position data, time data, and deleted common structures.

[0074] The local model LM generated locally at the AR device 2 will in some scenarios only provide limited structural information about the environment. Whatever sensing device is used to generate the local mesh LM, it will have a limited sensor range in practice. For example, the sensor will only be able to sense surfaces visible to the user, i.e. the sensor cannot "see" behind objects. In addition, the sensing device will have a limited field of view and limited depth perception - limited physical limitations of the sensing device, such as the sensor hardware optics and the processing power of the device, and / or environmental factors such as background light levels or disruptive lighting. In practice, this means that when the user 22 stands in a specific position and looks in a specific direction, the only surface structures that can be detected will be those visible to the user 22 and in a limited volume of space near the sensing device (outside of which the sensing device cannot detect surfaces) - which in most cases is more limited than the user's visual range.

[0075] If the AR user 22 moves or turns his head, new surface structures become detectable. The local modeling module 46 updates the local model LM to add the newly detected surface structures. Thus, as the user moves or looks around the environment, a more complete local model LM is built over time.

[0076] However, the problem of incomplete local data still exists, and this is particularly serious when AR users are facing a previously unknown environment for the first time.

[0077] As indicated above, this problem is solved by using an externally generated global mesh GM of the same environment 21. The model aggregation module selectively uses the data of the local model LM and the data of the externally generated model GM to generate an aggregated 3D model AM, which has more resulting information content than the local model LM, to be used by the rendering component 42 when rendering enhancement.

[0078] This allows the use of mesh data previously collected by other AR devices that are or have been in the same environment 21 to effectively expand the depth perception and field of view of the sensor device of the AR device 2. In turn, this allows a wider range of AR functions that are not constrained by the physical limitations of the sensor device of the AR device 2 or the conditions of the environment 21.

[0079] To illustrate this, two scenarios will now be considered in which the AR functionality of the AR device 2 is employed to allow a companion device user 30 to annotate the AR user's perceived environment with virtual augmentations.

[0080] In the first scenario, the AR function of the AR device 2 is based solely on the local grid LM, and it does not utilize the global grid of the present disclosure. Figure 4 To describe the method of implementing AR-based communication in this scenario. Figure 4 On the left hand side a flow chart for the method is shown, while pictorial representations of the method steps are on the right.

[0081] At step S2, the AR device 2 is activated. A volume R of space is shown, in front of the sensor arrangement of the AR device. Due to limitations of the device and / or environmental factors, the AR device 2 cannot detect surface structures outside this volume R, and therefore cannot detect the surface structures of the two (physical) objects Ob1, Ob2, even if they are visible to the user 22, due to the position of the AR user 22 and the direction they are facing. Therefore, at this point in time, it is not possible to annotate one of the visible objects Ob1, Ob2, because the surface structures of the visible objects Ob1, Ob2 are not captured as mesh data of the local mesh LM.

[0082] Only when the user moves within the environment (S4), for example by turning his head to face the first object Ob1, mesh data MD1 representing some of its surface structures can be captured and added to the local model LM. Furthermore, if the AR user 22 does not move close to the more distant second object Ob2 so that it comes into the range of the sensing device, it will not be possible to detect its surface structure and therefore not to annotate it.

[0083] So the partner device user 30 can add their own annotations to the AR user's previewed world, at step S6 the AR device 2 sends the updated local mesh LM to the partner device 32 via the network 26, e.g. in a real-time AR communication event, so that the partner user 30 can annotate the AR user's world in real time, and / or to the cloud system 28 for storage there so that the partner device 32 can retrieve it and make such annotations offline when they choose. An image of the movement of the environment 21 as captured by at least one of the cameras 18L, 18R is sent in association with the model so that the partner device user 30 can also see the 2D texture of the environment as well as its 3D structure.

[0084] At step S8, in real time or offline, the partner device displays the moving image on its display 32. The partner user 30 can pause or rewind the image using appropriate user input at the partner device 34.

[0085] At step S10, the partner user 30 adds their own annotation RA1 to the moving image, which in this example is a three-dimensional ring around the first object Ob1. Because the AR user has not moved close enough to the second object Ob2, its surface structure is not encapsulated in the local mesh LM, so the partner user 30 cannot annotate it.

[0086] The partner device 32 transmits the annotation as an annotation data object via the network 26:

[0087] sent back to the AR device 2, for example in real time via a network, and / or

[0088] Sent to the cloud system 28 for storage therein such that:

[0089] a. which can be retrieved by the AR user 22 at the time of selection by the AR user 22 (ie, offline), and / or

[0090] b. So that the media relay server of the system 28 can relay the data object to the AR device 2 in real time, that is, the media relay server receives the data object and sends it directly to the AR device 2.

[0091] At step S10 , the received or retrieved data object is used to render the companion user's annotation visible to the AR user 22 .

[0092] As described above, real-time in this context means that there is only a short delay (e.g., approximately two seconds or less) between rendering data sent by partner device 32 and data received by AR device 2, including when it is related via media relay of cloud system 28.

[0093] Note that steps S10-S12 are only possible after the user 22 has spent some time navigating the environment, building up the required level of detail in the local grid LM.

[0094] Now consider a scenario where a global mesh generated by stitching together multiple meshes from multiple Hololens is used instead.

[0095] Figure 5 A flow chart of a method for implementing AR-based communication within a communication system 20 according to the present subject matter is shown. Over time, local grids are sent from multiple AR devices 2 to a cloud system 28. Each local grid LM is sent in association with location and time information indicating where and when the local LM was captured. The cloud system identifies when multiple local grids fit similar locations based on location filtering criteria. They detect common structures in the grids and combine local grids LM based on these to generate a global grid GM of the environment 21. When new local grids are received, the global grid is updated accordingly. The global grid can be implemented between communication events (e.g., calls), i.e., a communication event between users can use a previous version of the global grid for annotation, update the global grid after the communication event ends, and use the updated version to annotate in subsequent communication events between users or between different users. A time constraint can be imposed so that the local grid data contained in the global grid is removed a predetermined amount of time after it is captured. In this way, it is possible to establish a comprehensive global model GM of the environment 21 that would be difficult to generate using only a single AR device. This comprises full mesh data representations MD1 ′, MD2 of the surface structure of the first and second objects Ob1 , Ob2 . The global mesh GM is stored in the cloud system 28 in association with location metadata identifying the location of the environment 21 .

[0096] At step S16, the partner device user 30 uses their device 32 to retrieve the global grid from the cloud system 28 by downloading it via an appropriate request 21 to the cloud system 28, and annotates in the same manner as described above, but based on the downloaded global grid GM instead of the single local grid LM. This can be offline, or during a real-time communication event (e.g., the global grid GM can be downloaded by the partner device 32 at the beginning of the real-time communication event for use therein). The partner device then sends their annotations as a data object to the target AR user 22T using the target AR device 2T, or to the cloud system 28 for storage therein, so that the target device 22T can retrieve it later.

[0097] At step S18, the target user retrieves the global grid GM from the cloud system 28. This step may be performed, for example, when the target user device 2T is activated for the first time in the environment, or when a real-time communication event begins. The target user device sends an indication of its current location to the cloud system 22, which is matched to the location metadata associated with the global grid GM at the cloud system 28, so that the global grid GM for the environment 21 can be retrieved.

[0098] At step S20, the target AR device 2T uses the global mesh GM to render the annotations received from the partner device 32. These may be received directly from the partner device 32 during a real-time communication event, or downloaded from the cloud system 28 in an offline annotation scenario.

[0099] It should be noted that annotations are rendered as soon as they are received, before the target user 22T walks through the environment (since the annotation capabilities are no longer limited by the local mesh data of the target device subset). Thus, for example, when an AR user first walks into a previously unknown AR environment, they can immediately be presented with a wide range of virtual elements throughout the environment in a manner that would not be possible using only the local mesh - such as annotations of distant objects, or more generally, augmentations that are perceived as being far away from the real-world environment 21 and yet still interacting with the real-world environment 21.

[0100] As mentioned above, the model aggregation module 44 of the target device 2T combines the local grids LM of the target device subset with the global grid GM (see Figure 2 ) to generate an aggregated mesh AM that may contain mesh data from both. For example, in the event that some of the surface structures in the local mesh LM do not match the corresponding surface structures in the global mesh GM, the surface structures in the local mesh LM may overwrite the global mesh GM because it is more likely to be up to date. The local mesh LM may still be sent to the partner device 32, which enables the partner device 32 to create its own aggregated mesh to provide a more precise annotation function; or the aggregated mesh AM may be sent to the partner device 32 for the same purpose.

[0101] The target partner device also sends its own local grid LM in association with the event and location information to the cloud system 28 for use in updating the version of the global grid GM stored there.

[0102] Although in the above, the global grid GM is received from a server via the Internet 26, alternatively, local grid data can be exchanged between AR devices in the environment in a decentralized manner, for example by establishing an ad hoc mesh network between AR devices in the environment (e.g., using Bluetooth or other wireless NFC technology) so that they can exchange their local grids via the ad hoc network.

[0103] In a first aspect, an augmented reality device comprises: a stereoscopic display device configured to provide a user of the augmented reality device with a view of a real-world environment in which the user is located, wherein the display device is configured to generate a stereoscopic image visible to the user simultaneously with the real-world view; a computer interface configured to receive externally generated 3D model data of the real-world environment in which the user is located from a network; and a rendering module configured to use the externally generated 3D model data to control the display device to render virtual elements in a manner such that they are perceived by the user as 3D elements located at desired locations in the real-world environment.

[0104] In an embodiment, the augmented reality device may also include: a memory; and a sensor device for detecting surface structures in a real-world environment, wherein the sensor device is configured to generate local 3D model data of the real-world environment implementing the surface structures of the real-world environment in the memory; and update the local 3D model to implement new surface structures as they become detectable due to movement of the sensor device within the environment; wherein the externally generated 3D model data received from the network is used to render the virtual element in the manner described at least when the surface structure implemented in the local 3D model data is insufficient for the local 3D model data to be used.

[0105] The computer interface may be configured to retrieve externally generated 3D model data from a server via the network.

[0106] For example, the externally generated 3D model may be retrieved from the server by sending the position of the real world environment from the augmented reality device to the server for comparison therewith with position data associated with the externally generated 3D model data.

[0107] Alternatively or additionally, the computer interface may be configured to send the local 3D model data to the server via the network for use by the server in generating an updated version of the externally generated 3D model data. For example, the local 3D model data may be sent in association with position information and / or time information related to the local 3D model data.

[0108] The rendering module may be configured to receive element position data defining a desired position from the network or another network.

[0109] For example, the element position data may be received from another user via the network or the other network in a real-time communication event between the user and the other user, or retrieved from a server by the augmented reality device.

[0110] For example, the element position data may be received in a virtual element data object defining a virtual element, the data object further comprising: structure data defining a 3D surface structure of the virtual element, and / or image data defining a 2D surface texture of the virtual element.

[0111] The augmented reality device may be configured to determine the desired location based on user input received from a user of the augmented reality device.

[0112] The rendering module may be configured to apply at least one rendering criterion to the externally generated 3D model to detect surface structures in the real world environment that the user would like to influence the virtual element, wherein the rendering of the virtual element is based on the detection.

[0113] For example, at least one influencing criterion may be based on the position and / or orientation of the detected surface structure as indicated by externally generated 3D model data; and / or on the position and / or orientation of the display device.

[0114] For example, the rendering of the virtual element based on the detection may include at least one of the following:

[0115] suppressing a portion of the virtual element such that a surface structure detected at a location between the virtual element and the user is perceived as partially obscuring the virtual element;

[0116] applying animation effects to the virtual element so that it is perceived as dynamically interacting with the detected surface structure; and

[0117] The desired position at which the virtual element is perceived to be located is modified to take into account the detected surface structure.

[0118] The element position data may define a desired position of a segment relative to a surface structure implemented by externally generated 3D model data.

[0119] At least the display device may be implemented in the wearable head mounted device. For example, the augmented reality device may be implemented in the wearable head mounted device (ie, the entire device), or another function may be implemented in a component (eg, a smart phone) separate from the head mounted device.

[0120] The display device may include two substantially transparent viewing portions through which the real world is visible and thereby provides a real-world view, and each substantially transparent viewing portion may be arranged to project a respective image component of a stereoscopic image onto a respective eye of a user.

[0121] For example, the display device may include a light engine device capable of being controlled by a rendering module to emit image light, and two optical components are mounted on the head-mounted device, each positioned to project a different portion of the image light onto a respective eye of the user thereby generating a stereoscopic image; the optical components may be substantially transparent and thereby provide a real-world view.

[0122] In a second aspect, a method for controlling an augmented reality device is provided, the augmented reality device comprising a display device configured to provide a field of view of a real-world environment in which the augmented reality device is located, the code method comprising: controlling a stereoscopic display device of the augmented reality device to generate a stereoscopic image visible to a user simultaneously with the real-world field of view; receiving externally generated 3D model data of the real-world environment in which the user is located from a network; and using the externally generated 3D model data to control the display device to render virtual elements in such a way that they are perceived by the user as 3D elements at desired locations in the real-world environment.

[0123] According to a third aspect, an augmented reality system comprises: a computer interface configured to receive multiple local 3D models of portions of a real-world environment, each local 3D model being generated by a different augmented reality device when the different augmented reality device is located in the real-world environment; a global modeling module configured to: combine the local 3D models to generate a global 3D model that individually represents a larger portion of the real environment than any of the local 3D models; and send at least a portion of the global 3D model to a target augmented reality device currently located in the real-world environment for use when rendering virtual elements at the augmented reality device.

[0124] In an embodiment, the global modeling module may be configured to transmit at least a portion of the global 3D model to a device remote from the real-world environment, thereby allowing the remote device to remotely explore and enhance the virtual version of the real-world environment.

[0125] For example, the augmented reality system may further include an augmented sharing model configured to: receive rendering data from the remote device, the rendering data being used when rendering a virtual object at a target augmented reality device; and send the rendering data to the target augmented reality device.

[0126] In some cases, the global 3D model may be sent to the remote device in response to a download request from the remote device, and / or the rendering data may be sent to the target augmented reality device in response to a download request from the augmented reality device.

[0127] The remote device may be, for example, a smartphone or a tablet computer.

[0128] In a third aspect, an augmented reality system comprises: a computer interface configured to receive a plurality of local 3D models of portions of a real-world environment, each local 3D model being generated by a different augmented reality device when the different augmented reality device is located in the real-world environment; a global model generation module configured to: combine the local 3D models to generate a global 3D model that individually represents a larger portion of the real environment than any of the local 3D models; and send at least a portion of the global 3D model to a device remote from the real-world environment, thereby allowing the remote device to remotely explore and augment a virtual version of the real-world environment; and an augmented sharing model configured to: receive rendering data from the remote device for use in rendering virtual objects at the augmented reality device in the real-world environment; and send the rendering data to the augmented reality device when the augmented reality device is located in the real-world environment.

[0129] In an embodiment, the rendering data is received from the remote device and sent to the augmented reality device in real-time.

[0130] The rendering data is received from the remote device and sent to the augmented reality device during a communication event between a user of the remote device and a user of the augmented reality device.

[0131] For example, the computer interface may be configured to receive at least one additional local 3D model from at least one additional augmented reality device when the at least one additional augmented reality device is located in the real-world environment; the augmented sharing module may be configured to update the global 3D model based on the at least one additional local 3D model at a time after the communication event ends, and the updated global 3D model is used for use in future communication events between users.

[0132] The augmented sharing module may be configured to receive additional rendering data from the augmented reality device and send it to the remote device, the additional rendering data being used in rendering another virtual object at the remote device using the global 3D model.

[0133] For example, the additional rendering data is received from the augmented reality device and sent to the remote device in real time.

[0134] Alternatively, the augmented sharing module may be configured to store the rendering data in a computer storage of the augmented reality system and to send it to the augmented reality device in response to a download request from the augmented reality device.

[0135] The computer interface may be configured to receive at least one additional local 3D model from at least one additional augmented reality device when the at least one additional augmented reality device is located in the real-world environment, wherein the augmented reality module is configured to update the global 3D model based on the at least one additional local 3D model at a time after the global 3D model is generated.

[0136] Each local 3D model is received in association with position information relating to the local 3D model, wherein the global model generation module is configured to identify the multiple local 3D models as being related to the (same) real-world environment based on their associated position information, and select them for use in generating the global 3D model based on the identification.

[0137] Alternatively or additionally, each local 3D model is received in association with time information relating to the local 3D model data, wherein the global model generation module is configured to identify the plurality of local 3D models as relating to a predetermined time interval and select them for use in generating the global 3D model based on the identification.

[0138] For example, the global model generation module is configured to determine the age of at least one of the local 3D models based on time information associated with at least one of the local 3D models, and in response to the age reaching an age threshold, update the global 3D model to remove data derived from the at least one local 3D model from the global 3D model.

[0139] The global 3D model is also sent to the augmented reality device for use when rendering virtual elements at the augmented reality device.

[0140] A fourth aspect is directed to a method for implementing communication between an augmented reality device in a real-world environment and another device remote from the real-world environment via a network, the method comprising implementing the following steps at the other device: receiving a global 3D model, the global 3D model being generated by combining multiple local 3D models of parts of the real-world environment, each local 3D model being generated by a different augmented reality device when the different augmented reality device is located in the real-world environment, wherein the global 3D model individually represents a larger portion of the real environment than any of the local 3D models; sending at least a portion of the global 3D model to the remote device, thereby allowing the remote device to remotely explore and enhance a virtual version of the real-world environment; using the global 3D model to generate rendering data for use in rendering virtual objects at the augmented reality device in the real-world environment; and sending the rendering data to the augmented reality device and / or to an augmented reality system via the network when the augmented reality device is located in the real-world environment.

[0141] In an embodiment, the communication is real-time communication between a user of the augmented reality device and a user of the other device via the network, and the rendering data is sent from the other device to the augmented reality device in real time.

[0142] For example, the rendering data may be sent from the other device to the augmented reality device via the augmented reality system, the rendering data being relayed in real time between the devices by the augmented reality system.

[0143] The augmented reality data may be sent to the augmented reality system for storage therein.

[0144] The method may include: receiving, at the other device via the network, additional rendering data from the augmented reality device; and using, by the other device, the additional rendering data to render additional virtual elements on a display of the other device.

[0145] The method may include:

[0146] a. receiving a video signal from the augmented reality device via the network at the other device;

[0147] b. Displaying a moving image derived from the video signal on the display by the other device, wherein the additional virtual element is overlaid on the moving image on the display.

[0148] In some cases, the global 3D model may be used by the other device to render the additional virtual elements.

[0149] In a fifth aspect, a communication system comprises: a device remote from a real-world environment; and an augmented reality system comprising: (i) a computer interface configured to receive a plurality of local 3D models of portions of the real-world environment, each local 3D model being generated by a different augmented reality device when located in the real-world environment; and (ii) a global model generation module configured to: combine the local 3D models to generate a global 3D model that individually represents a larger portion of the real environment than any of the local 3D models; and send at least a portion of the global 3D model to a device remote from the real-world environment, thereby allowing the remote device to remotely explore and augment a virtual version of the real-world environment; wherein the remote device is configured to use the global 3D data to generate rendering data for use in rendering virtual objects at the augmented reality device in the real-world environment, and to send the rendering data to the augmented reality device and / or to the augmented reality system (e.g., for storage therein and / or for association with the augmented reality device) when the augmented reality device is located in the real-world environment.

[0150] It should be noted that any of the system functions described may be implemented in corresponding method steps of a method for implementing shared reality communication.

[0151] In a sixth aspect, a computer-readable medium stores executable code that, when executed on a computer system (eg, implemented in a user device or a distributed cloud system), implements any of the system functions or method steps disclosed herein.

[0152] Generally speaking, any of the functions described herein can be implemented using software, firmware, hardware (e.g., fixed logic circuitry), or a combination of these implementations. As used herein, the terms "module," "function," "component," and "logic" (e.g., Figure 2 The term "functional module" generally refers to software, firmware, hardware, or a combination thereof. In the case of a software implementation, a module, function, or logic represents a program code that performs a specified task when executed on a processor (e.g., a CPU or multiple CPUs). The program code may be stored in one or more computer-readable memory devices. The features of the technology described below are platform-independent, which means that the technology can be implemented on a variety of commercial computing platforms with a variety of processors.

[0153] For example, the AR or partner device (user terminal) may also include an entity (e.g., software) that causes the hardware of the user terminal to perform an operation, such as a processor function block, etc. For example, the user terminal may include a computer-readable medium, which may be configured to store instructions that cause the user terminal and more specifically the operating system and associated hardware of the user terminal to perform an operation. Therefore, the instructions are used to configure the operating system and associated hardware to perform the operation, and in this way cause the conversion of the operating system and associated hardware to perform the function. The instructions may be provided to the user terminal by a computer-readable medium through a variety of different configurations.

[0154] One such configuration of a computer-readable medium is a signal bearing medium, and is therefore configured to send instructions (e.g., as a carrier wave) to a computing device, such as via a network. A computer-readable medium may also be configured as a computer-readable storage medium, and is therefore not a signal bearing medium. Examples of computer-readable storage media include random access memory (RAM), read-only memory (ROM), optical disks, flash memory, hard disk storage, and other memory devices that may use magnetic, optical, and other technologies to store instructions and other data.

[0155] Although the subject matter has been described in language specific to structural features and / or methodological acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Instead, the specific features and acts described above are disclosed as example forms of implementing the claims.

Claims

1. An augmented reality device, comprising: processor; as well as a memory in communication with the processor, the memory comprising executable instructions that, when executed by the processor, cause the augmented reality device to: receiving a plurality of local 3D models of the real-world environment from a plurality of second remote devices located at the real-world environment, each local 3D model representing a surface structure of a different portion of the real-world environment captured by each respective second remote device; generating, based on the plurality of local 3D models, a global 3D model representing a larger portion of the real-world environment than any of the local 3D models independently represent; generating augmentation data for rendering a 3D virtual object to be placed at a location in the global 3D model; as well as Sending to a first remote device located at the real-world environment (i) the augmented data for rendering the 3D virtual object and (ii) global 3D model data representing at least a portion of the global 3D model, so that the 3D virtual object is rendered by the first remote device based on the global 3D model data even when the surface structure of the real-world environment captured by the first remote device is insufficient for the first remote device to render the 3D virtual object.

2. The device according to claim 1, wherein: To generate the global 3D model, the instructions further include instructions that, when executed by the processor, cause the processor to control the device to perform the following function: combining the received local 3D models.

3. The device of claim 1, wherein: The plurality of second remote devices are respectively located at a plurality of different locations of the real-world environment.

4. The device of claim 1, wherein: The portion of the real-world environment represented by the global 3D model is larger than the portion of the real-world environment represented by any of the local 3D models.

5. The device of claim 1, wherein: The instructions also include instructions that, when executed by the processor, cause the processor to control the device to perform the following functions: receiving an updated local 3D model or additional local 3D models from at least one of the plurality of second remote devices; and The global 3D model is updated based on the received updated local 3D model or additional local 3D models.

6. The device of claim 1, wherein: In order to generate the enhancement data, the instructions also include instructions that, when executed by the processor, cause the processor to control the device to perform the following functions: receiving local enhancement data from the second remote device for creating the 3D virtual object in the local 3D model generated by the second remote device, and the position of the 3D virtual object in the global 3D model is determined based on the received local enhancement data.

7. A method of operating an augmented reality device, the method comprising: receiving a plurality of local 3D models of the real-world environment from a plurality of second remote devices located at the real-world environment, each local 3D model representing a surface structure of a different portion of the real-world environment captured by each respective second remote device; generating, based on the plurality of local 3D models, a global 3D model representing a larger portion of the real-world environment than any of the local 3D models independently represent; generating augmentation data for rendering a 3D virtual object to be placed at a location in the global 3D model; as well as Sending to a first remote device located at the real-world environment (i) the augmented data for rendering the 3D virtual object and (ii) global 3D model data representing at least a portion of the global 3D model, so that the 3D virtual object is rendered by the first remote device based on the global 3D model data even when the surface structure of the real-world environment captured by the first remote device is insufficient for the first remote device to render the 3D virtual object.

8. The method of claim 7, wherein: Generating the global 3D model includes combining the received local 3D models.

9. The method of claim 7, wherein: The plurality of second remote devices are respectively located at a plurality of different locations of the real-world environment.

10. The method of claim 7, wherein: The portion of the real-world environment represented by the global 3D model is larger than the portion of the real-world environment represented by any of the local 3D models.

11. The method of claim 7, further comprising: receiving an updated local 3D model or additional local 3D models from at least one of the plurality of second remote devices; as well as The global 3D model is updated based on the received updated local 3D model or additional local 3D models.

12. The method of claim 7, wherein: Generating the augmentation data includes receiving local augmentation data from the second remote device for creating the 3D virtual object in the local 3D model generated by the second remote device, and the position of the 3D virtual object in the global 3D model is determined based on the received local augmentation data.

13. A non-transitory computer-readable medium comprising instructions that, when executed by a processor, cause an augmented reality device to: receiving a plurality of local 3D models of the real-world environment from a plurality of second remote devices located at the real-world environment, each local 3D model representing a surface structure of a different portion of the real-world environment captured by each respective second remote device; generating, based on the plurality of local 3D models, a global 3D model representing a larger portion of the real-world environment than any of the local 3D models independently represent; generating augmentation data for rendering a 3D virtual object to be placed at a location in the global 3D model; as well as Sending to a first remote device located at the real-world environment (i) the augmented data for rendering the 3D virtual object and (ii) global 3D model data representing at least a portion of the global 3D model, so that the 3D virtual object is rendered by the first remote device based on the global 3D model data even when the surface structure of the real-world environment captured by the first remote device is insufficient for the first remote device to render the 3D virtual object.

14. The non-transitory computer readable medium of claim 13, wherein: To generate the global 3D model, the instructions further include instructions that, when executed by the processor, cause the processor to control the device to perform the following function: combining the received local 3D models.

15. The non-transitory computer readable medium of claim 13, wherein: The plurality of second remote devices are respectively located at a plurality of different locations of the real-world environment.

16. The non-transitory computer readable medium of claim 13, wherein: The portion of the real-world environment represented by the global 3D model is larger than the portion of the real-world environment represented by any of the local 3D models.

17. The non-transitory computer readable medium of claim 13, wherein: The instructions also include instructions that, when executed by the processor, cause the processor to control the device to perform the following functions: receiving an updated local 3D model or additional local 3D models from at least one of the plurality of second remote devices; and The global 3D model is updated based on the received updated local 3D model or additional local 3D models.

18. The non-transitory computer readable medium of claim 13, wherein: In order to generate the enhancement data, the instructions also include instructions that, when executed by the processor, cause the processor to control the device to perform the following functions: receiving local enhancement data from the second remote device for creating the 3D virtual object in the local 3D model generated by the second remote device, and the position of the 3D virtual object in the global 3D model is determined based on the received local enhancement data.

Images