Apparatus and method for providing augmented reality interaction - Patents.com

Abstract

The communication device (1) is configured to obtain (S1) location information of a first user, send (S2) a query request to a computerized virtual real estate directory system (4), and receive (S3) a query response therefrom. The query response includes one or more virtual real estate objects assigned to the virtual real estate object location. The communication device (1) sends (S5) an interaction request to one of the augmented reality interaction servers (5) and receives (S6) access permissions for the particular virtual real estate object. The communication device (1) sends (S7) first user data via the augmented reality interaction server (5) to a communication device of a second user associated with the particular virtual real estate object, and receives (S8) second user data from the communication device of the second user.

Description

[Technical field]

[0001] The present invention relates to an apparatus and method for providing augmented reality interaction, in particular to an apparatus, method and computer program product for providing augmented reality interaction. [Background technology]

[0002] Augmented reality (AR) is a technology that augments a real-world environment with computer-generated virtual information. Unlike virtual reality (VR), AR does not replace the real world with a computer-generated world, but rather adds computer-generated information to the real world, often by overlaying virtual content on real-world objects. Virtual content can include not only visual content (such as digital images or models) that are overlaid on real-world objects to enhance the real-world visual experience, but also auditory, tactile, olfactory, and other sensory modalities. Additionally, augmented reality can include representations of other users, enabling interaction between users beyond typical video or phone conferences by introducing spatial and contextual aspects to the representations and interactions.

[0003] Communication devices for playing augmented reality content are diverse and constantly under development. Communication devices can be stationary or portable or worn by the user. These communication devices include holographic projection systems, video conferencing systems, personal computers, and mobile communication devices. Communication devices can include various sensors and displays, as well as wearable modules such as eyepiece headsets, smart glasses, etc., that present augmented reality content to the user in addition to real-world content. Communication devices can be supported by additional connectivity devices for augmented reality interaction, such as headphones for auditory content or gloves to provide precise motion tracking and haptic feedback.

[0004] To ensure privacy of augmented reality interactions, it is essential to be able to control which other users a particular user or a particular server can interact with. In large-scale augmented reality environments, especially those involving federated systems, centralized control of augmented reality interactions is impractical or undesirable, especially since low-latency, high-data-rate connections are critical to enable smooth and natural interactions between users and servers. Summary of the Invention [Problem to be solved by the invention]

[0005] It is an object of the present invention to provide an apparatus and method for providing augmented reality interaction. [Means for solving the problem]

[0006] In particular, the present invention relates to an apparatus, method and computer program product for providing augmented reality interaction to a user of a communication device. According to the invention, these objects are achieved by means of the features of the independent claims, further advantageous embodiments follow from the dependent claims and the description.

[0007] According to the invention, the above object is achieved in particular by a communication device, the communication device including an electronic circuit configured to obtain location information of a first user of the communication device. The location information can be related to a physical location, for example a geographical location determined by a GPS. The location information can also be related to an emulated location, i.e. a location in the real or virtual world different from the physical location of the user. The communication device is configured to send a query request via a communication network to a computerized virtual real estate directory system, the query request including at least the location information of the first user. The communication device is configured to receive a query response via the communication network from the virtual real estate directory system, the query response including one or more virtual real estate objects assigned to a virtual real estate object location within a predefined area defined by the location information of the first user, each of the virtual real estate objects including an identification and a network address of an augmented reality interaction server. The communication device is configured to send an interaction request via the communication network to one of the augmented reality interaction servers for at least one particular virtual real estate object, using the network address of the augmented reality interaction server respectively, and to receive access permissions for the particular virtual real estate object from the augmented reality interaction server via the communication network, each interaction request including the identification information of the particular virtual real estate object. The communication device is configured to send first user data associated with the particular virtual real estate object to a communication device of a second user via the augmented reality interaction server, and to receive second user data from the communication device of the second user via the augmented reality interaction server.

[0008] In one embodiment, the communication device is configured to transmit the first user data to the communication device of the second user and receive the second user data in real time from the communication device of the second user via an augmented reality interaction server so as to enable natural interaction between the first user and the second user. In one embodiment, the electronic circuitry is configured to transmit a public user identifier of the first user to the augmented reality interaction server, the public user identifier including one or more of the first user's public name or the first user's encryption key. In one embodiment, the electronic circuitry is configured to receive from the augmented reality interaction server a public user identifier of the second user comprising one or more of a public name of the second user or an encryption key of the second user. As an example, the encryption key of the first user is the public key of a public / private key pair generated in the electronic circuit of the communication device of the first user, and the encryption key of the second user is the public key of a public / private key pair generated in the communication device of the second user.

[0009] In one embodiment, the electronic circuitry is configured to receive from the augmented reality interaction server a session key generated at the augmented reality interaction server and encrypted by the augmented reality interaction server with the public key of the first user. The session key is decrypted at the communication device of the first user with the private key of the first user. The electronic circuitry is configured to encrypt data to be sent to the augmented reality interaction server with the session key and to decrypt data received from the augmented reality interaction server with the session key. In one embodiment, the electronic circuit is configured to encrypt data transmitted to the communication device of the second user via the augmented reality interaction server using the public key of the second user. In one embodiment, the electronic circuit is configured to decrypt data received from the communication device of the second user via the augmented reality interaction server, i.e. data encrypted using the public key of the first user, using the private key of the first user stored in the communication device of the first user.

[0010] In one embodiment, the electronic circuit is configured to digitally sign data sent to the augmented reality interaction server or to the communication device of the second user via the augmented reality interaction server, in particular configured to sign the sent data with the private key stored in the communication device of the first user, i.e. such that this signature allows the augmented reality server and / or the communication device of the second user to verify the authenticity of the sent data by using the public key of the communication device of the first user. In one embodiment, the electronic circuit is configured to verify the authenticity of data received from the augmented reality interaction server and / or data received from the communication device of the second user using a public key received from the augmented reality interaction server and / or a public key received from the communication device of the second user, respectively.

[0011] In one embodiment, the communication device includes one or more cameras and a rendering system. The electronic circuitry is configured to determine visual representation data of the first user using one or more images of the first user captured by the one or more cameras. The electronic circuit is configured to transmit the visual representation data of the first user included in the first user data to the communication device of the second user via the augmented reality interaction server, receive the visual representation data of the second user included in the second user data from the communication device of the second user via the augmented reality interaction server, and render augmented reality content including the visual representation of the second user using the rendering system and the visual representation data of the second user. In one embodiment, the electronic circuitry is further configured to determine a field of view of the first user using the one or more cameras, determine from the field of view coordinates of real-world features within the field of view, and render the augmented reality content using the rendering system such that the augmented reality content has a fixed position relative to the coordinates of the real-world features.

[0012] In one embodiment, the electronic circuitry is further configured to verify whether the augmented reality content is completely contained within a bounding volume of the virtual real estate object, and in response to a positive verification, render the augmented reality content, and in response to a negative verification, either not render the augmented reality content or modify the augmented reality content to render modified augmented reality content that fits within the bounding volume of the virtual real estate object. In one embodiment, the communication device further includes a motion tracker, and the electronic circuitry is further configured to record user motion data relating to bodily movements of the first user using the motion tracker, and the electronic circuitry is further configured to transmit the user motion data included in the first user data to the augmented reality interaction server.

[0013] In one embodiment, the communication device further includes one or more microphones and one or more speakers, and the electronic circuit is configured to record voice data of the first user using the one or more microphones, transmit the voice data of the first user included in the first user data to the augmented reality interaction server, receive the voice data of the second user included in the second user data from the communication device of the second user, and play the voice data of the second user on the one or more speakers. In one embodiment, the electronic circuitry is further configured to receive an augmented reality object from the augmented reality interaction server and render augmented reality content including the augmented reality object using the rendering system. The electronic circuitry is configured to determine user interaction data of the first user interacting with the augmented reality object and transmit the user interaction data to the augmented reality interaction server. The electronic circuitry is configured to receive an updated augmented reality object from the augmented reality interaction server and render the augmented reality content including the updated augmented reality object using the rendering system.

[0014] In addition to the communication device, the invention further relates to a method of interaction in augmented reality for a first user of a communication device having an electronic circuit, the method comprising: obtaining location information of the first user of the communication device; and sending a query request from the communication device via a communication network to a computerized virtual real estate directory system, the query request including the location information of at least the first user (2). The method comprises receiving a query response from the virtual real estate directory system via the communication network in the communication device, the query response including one or more virtual real estate objects assigned to virtual real estate object locations within a predefined area defined by the location information of the first user. Each of the virtual real estate objects includes an identification and a network address of an augmented reality interaction server. The method comprises sending an interaction request from the communication device via the communication network for at least one particular virtual real estate object to one of the augmented reality interaction servers using the network address of the augmented reality interaction server, each interaction request including the identification of the particular virtual real estate object. The method includes receiving, at the communication device, a permission for accessing the particular virtual real estate object from the augmented reality interaction server via the communication network, transmitting, from the communication device, via the augmented reality interaction server, first user data associated with the particular virtual real estate object to a communication device of a second user, and receiving, at the communication device, via the augmented reality interaction server, second user data from the communication device of the second user.

[0015] In one embodiment, the method further includes transmitting the first user data from the communication device to the communication device of the second user, and receiving the second user data from the communication device of the second user in real time via the augmented reality interaction server. In one embodiment, the method further includes transmitting a public user identifier of the first user from the communication device of the first user to the augmented reality interaction server, the public user identifier including one or more of the first user's public name or the first user's encryption key. In one embodiment, the method further includes receiving, at the communication device of the first user, a public user identifier of the second user from the augmented reality interaction server, the public user identifier including one or more of the public name of the second user or an encryption key of the second user. As an example, the encryption key of the first user is the public key of a public / private key pair generated in the electronic circuit of the communication device of the first user, and the encryption key of the second user is the public key of a public / private key pair generated in the communication device of the second user.

[0016] In one embodiment, the method includes receiving, at the communication device of the first user, from the augmented reality interaction server a session key generated at the augmented reality interaction server and encrypted by the augmented reality interaction server using the public key of the first user. The session key is decrypted at the communication device of the first user using a private key of the first user. The electronic circuitry is configured to encrypt data to be sent to the augmented reality interaction server using the session key and to decrypt data received from the augmented reality interaction server using the session key. In one embodiment, the method includes encrypting, at the communication device of the first user, data to be sent to the communication device of the second user via the augmented reality interaction server using the public key of the second user. In one embodiment, the method includes decrypting, at the communication device of the first user, data received from the communication device of the second user via the augmented reality interaction server, i.e. data encrypted using the public key of the first user, using a private key of the first user stored in the communication device of the first user.

[0017] In one embodiment, the method comprises digitally signing data sent to the augmented reality interaction server or data sent via the augmented reality interaction server to the communication device of the second user. In particular, the method comprises digitally (signing) the sent data with the private key stored in the communication device of the first user, i.e. allowing the augmented reality server and / or the communication device of the second user to verify the authenticity of the sent data by using the public key of the communication device of the first user. In one embodiment, the method includes verifying authenticity of data received from the augmented reality interaction server and / or data received from the communication device of the second user using a public key received from the augmented reality interaction server and / or a public key received from the communication device of the second user, respectively.

[0018] In one embodiment, the communication device includes one or more cameras and a rendering system, and the method further includes determining, at the communication device, visual representation data of the first user using one or more images of the first user captured by the one or more cameras. The method includes transmitting the visual representation data of the first user included in the first user data from the communication device of the first user to the communication device of the second user via the augmented reality interaction server. The method includes receiving, at the communication device of the first user, visual representation data of the second user included in the second user data from the communication device of the second user via the augmented reality interaction server. The method includes rendering augmented reality content including a visual representation of the second user using the rendering system and the visual representation data of the second user. In one embodiment, the method further includes determining, at the communication device, a field of view of the first user using the one or more cameras. The method includes determining, at the communication device, coordinates of real-world features in the field of view from the field of view, and rendering the augmented reality content using the rendering system such that the augmented reality content has a fixed position relative to the coordinates of the real-world features. In particular, the coordinates of the augmented reality content, e.g., the coordinates of a center of the augmented reality content, are related to the coordinates of the real-world feature such that as the real-world feature moves or rotates, the augmented reality content moves or rotates, respectively, with the real-world feature to remain in a fixed position and orientation relative to the position and orientation of the real-world feature. This provides a visually consistent and stable augmented reality environment.

[0019] In one embodiment, the method further includes verifying, at the communication device, whether the augmented reality content is entirely contained within a bounding volume of the virtual real estate object. In response to a positive verification, the method includes rendering the augmented reality content, and in response to a negative verification, either not rendering the augmented reality content or modifying the augmented reality content at the communication device to render modified augmented reality content that fits within the bounding volume of the virtual real estate object. In one embodiment, the communication device further includes a motion tracker, and the method further includes recording, at the communication device, user motion data relating to bodily movements of the first user using the motion tracker, and transmitting the user motion data included in the first user data from the communication device to the augmented reality interaction server. In one embodiment, the communication device further includes one or more microphones and one or more speakers, and the method includes recording voice data of the first user using the one or more microphones at the communication device. The method includes transmitting the voice data of the first user included in the first user data from the communication device to the augmented reality interaction server. The method includes receiving voice data of the second user included in the second user data from the communication device of the second user at the communication device, and playing the voice data of the second user on the one or more speakers.

[0020] In one embodiment, the method further includes receiving an augmented reality object from the augmented reality interaction server at the communication device and rendering augmented reality content including the augmented reality object using the rendering system. The method includes determining user interaction data of the first user interacting with the augmented reality object at the communication device and transmitting the user interaction data from the communication device to the augmented reality interaction server. The method includes receiving an updated augmented reality object from the augmented reality interaction server at the communication device and rendering the augmented reality content including the updated augmented reality object using the rendering system.

[0021] In addition to the computer device and the method of interaction in augmented reality, the present invention further relates to a computer program product including a non-transitory computer readable medium having computer program code stored thereon, the computer program code being configured to control electronic circuitry of a communication device such that the communication device performs the above-mentioned method.

[0022] The invention will now be described in more detail, by way of example only, with reference to the drawings, in which: [Brief description of the drawings]

[0023] [Figure 1] FIG. 1 is a block diagram that illustrates a schematic diagram of a communication device for providing augmented reality interaction. [Diagram 2] 1 is a schematic diagram illustrating a first user with a first communication device, a computerized virtual real estate directory system, an augmented reality interaction server, and a network therebetween. [Diagram 3] 1 is a schematic diagram showing a first user having a first communication device, an augmented reality interaction server, a second user having a second communication device, and a network therebetween; [Figure 4] 1 is a schematic diagram illustrating a planar view of a user in an urban environment of virtual real estate objects. [Diagram 5] 1 is a schematic diagram illustrating a field of view of a first user in an indoor environment consisting of a communication device, a visual representation of a second user, and an augmented reality object. [Figure 6] FIG. 1 is a block diagram illustrating a virtual real estate directory system. [Figure 7] FIG. 2 is a schematic block diagram of an augmented reality interaction server including augmented reality objects; [Figure 8] 1 is a flow diagram illustrating a series of steps for providing an augmented reality interaction. [Figure 9] 11 is a flow diagram illustrating a series of steps for rendering a visual representation of a second user. [Figure 10] 1 is a flow diagram illustrating a series of steps for rendering an augmented reality object to a first and second user. [Figure 11] 1 is a flow diagram illustrating modification of augmented reality content. [Figure 12] FIG. 1 is a block diagram illustrating modification of augmented reality content. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0024] In FIG. 1, reference number 1 denotes a communication device 1 including an electronic circuit 13. Depending on the embodiment, the communication device 1 includes various further modules such as a rendering system 11, one or more cameras 12, one or more speakers 4, one or more microphones 15, a motion tracker 16, a memory, a communication interface, a battery, a direction module, and / or a user interface. The rendering system 11 is configured to display visual content in two and / or three dimensions. Depending on the embodiment, the rendering system 11 may include a holographic projection system, one or more projectors, and / or a display. A user interface may be integrated in the rendering system 11 in the form of a touch-sensitive display. Depending on the embodiment, the communication device 1 may include further modules such as a GPS module, an accelerometer, and an environmental module (for measuring temperature and / or humidity). Furthermore, the communication device 1 may be connected to a building automation system, for example for controlling lights, doors, and / or HVAC systems. The communication device 1 may also be connected to other Internet of Things (IoT) devices. The modules of the communication device 1 are connected to each other via a data connection mechanism so as to be able to transmit and receive data. Alternatively or additionally, one or more of the modules of the communication device 1 are integrated in an electronic circuit 13. The communication interface of the communication device 1 is configured for wireless communication with a remote server via a communication network 3. The communication network 3 includes the Internet as well as other intermediate networks. The wireless communication is performed using a mobile data network such as GSM, CDMA, LTE network, and / or a short-range wireless communication interface using a Wi-Fi network, Bluetooth, and / or other wireless network types and standards. In one embodiment, the communication device 1 further includes an accessory module including a headphone, a head-mounted display and / or a display such as smart glasses, and / or a haptic glove.These accessory modules are configured to exchange data with the communication device 1 via a data connection mechanism. A data connection is a term that refers to a mechanism that facilitates data communication between two components, devices, systems, or other entities. In one embodiment, the data connection is a wire, such as a cable or a system bus. In one embodiment, the data connection includes wireless communication, where some components of a communication device are physically separated from each other and exchange data with each other wirelessly, for example using the Bluetooth standard.

[0025] Depending on the embodiment, the electronic circuitry 13 may include a System on a Chip (SoC), a Central Processing Unit (CPU), and / or a Graphical Processing Unit (GPU), other more specific processing units such as Application Specific Integrated Circuits (ASICs), reprogrammable processing units such as Field Programmable Gate Arrays (FPGAs), and processing units specifically configured to accelerate specific applications, such as Artificial Intelligence (AI) accelerators for accelerating neural network and / or machine learning processing. The memory includes one or more volatile and / or non-volatile storage components. The storage components may be removable and / or non-removable and may be integrated in whole or in part with the communication device 1. Examples of storage components include a random access memory (RAM), a flash memory, a hard disk, a data memory, and / or other data stores. The data storage system includes a non-transitory computer-readable medium having stored thereon computer program code configured to control the electronic circuitry 13 such that the communication device 1 performs one or more steps and / or functions described herein. Depending on the embodiment, the computer program code is compiled or non-compiled program logic and / or machine code. In this way, the communication device 1 is configured to perform one or more steps and / or functions. The computer program code defines and / or is part of a separate software application. Those skilled in the art will appreciate that the computer program code may also be distributed among multiple software applications. In one embodiment, the computer program code further provides interfaces, such as Application Programming Interfaces (APIs), to allow functionality and / or data of the mobile communication device 1 to be accessed remotely, such as via a client application or a web browser. In one embodiment, the computer program code is configured such that one or more steps and / or functions are performed not in the communication device 1, but in another computer system remote from the communication device 1.

[0026] In one embodiment, the communication device 1 is a personal computer. The personal computer, for example a laptop running the Windows operating system, is configured to install computer program code, for example from a portable medium or a server. In one embodiment, the communication device 1 is a mobile phone. A mobile phone, for example a smartphone running the Android® operating system or the iOS® operating system, is configured to download and install computer program code from a server, for example from an App Store. Further examples of mobile communication devices 1 are tablet computers, smart watches, etc.

[0027] In one embodiment, one or more speakers 14 are configured to reproduce auditory content and include electro-acoustic transducers, such as loudspeakers or headphones. The auditory content may be reproduced by speakers 14 arranged in mono, stereo, or multi-channel configurations. Additionally, depending on the auditory content and the configuration of speakers 14, spatial or stereoscopic (3D) audio may be reproduced. The orientation module of the communication device 1 captures the orientation (heading) of the first user 2 and / or the communication device 1 and may include a compass, an accelerometer, a gyro sensor, etc. The user interface allows the first user 2 to interact with the communication device 1. In particular, the user interface allows the first user 2 to input data into the mobile communication device 1. Examples of user interfaces include hardware buttons, such as a touch-sensitive display, a keyboard, etc. One or more microphones 15 are configured to capture sound, e.g., to capture voice commands of the first user 2. In one embodiment, the user interface further uses other modules, such as one or more cameras 15 and / or a motion tracker 16, to capture the movements and / or gestures of the first user 2 as user input. The motion tracker 16 may include a holding or wearing device that tracks the movements of the first user 2 or the movements of a body part of the first user 2.

[0028] Figure 2 shows a schematic diagram of a first user 2 with a communication device 1, a computerised virtual real estate directory system 4, an augmented reality interaction server 5 and a communication network 3 over which communication takes place between the communication device 1 and the virtual real estate directory system 4 and between the communication device 1 and the augmented reality interaction server 5. Also shown in the figure are transmissions T2, T3, T5 and S6, which will be explained in more detail below with reference to Figure 8. The first user 2 carries the communication device 1 with him. In one embodiment where the communication device 1 is a wearable device, the first user 2 wears the communication device 1, for example as a headpiece or eyepiece. In one embodiment where the communication device 1 is not portable, for example fixedly installed, the first user 2 is located in the vicinity of the communication device 1. By separating the virtual real estate directory system 4 and the augmented reality interaction server 5, the virtual real estate directory system 4 can be configured to quickly and efficiently determine the virtual real estate objects 41 (not shown). Thus, the ownership and governance of the virtual real estate directory system 4 is decoupled from the ownership and governance of the augmented reality interaction server 5. The virtual real estate directory system 4 provides a relatively small amount of data to the communication device 1, whereas the augmented reality interaction server 5 exchanges a relatively large amount of data with the communication device 1. Thus, providing the augmented reality interaction server 5 relatively densely and more locally than the servers of the virtual real estate directory system 4 allows for an efficient distribution of the servers, and allows a relatively large amount of data to be exchanged with the communication device 1 via the augmented reality interaction server 5 with low transmission latency and high bandwidth, allowing for more natural and responsive interactions. In particular, the augmented reality interaction server 5 is located at a physical location corresponding to the location of the virtual real estate object. In this way, a user physically located in the same place as the augmented reality interaction server 5 benefits from a low latency connection.

[0029] 3 shows a schematic diagram of a first user 2 having a first communication device 1 connected to an augmented reality interaction server 5 via a communication network 3, and a second user 2B having a second communication device 1B also connected to the augmented reality interaction server 5 via the communication network 3. Both the first user 2 and the second user 2B are located in the same virtual real estate object 41. In one embodiment, the first user 2 and the second user 2B are located in different virtual real estate objects 41 adjacent or proximate to each other. Also shown in the figure are transmissions T7, T8, T7B, and T8B, which are described in more detail below with reference to FIG.

[0030] FIG. 4 is a diagrammatical illustration of a floor plan of an urban environment, comprising a virtual real estate object 41 and a first user 2 located at a position represented by location information 21. As shown in FIG. In one embodiment, the location information 21 is the physical geographic location of the first user 2, for example expressed by coordinates in a coordinate system. For example, the coordinate system may be a geographic coordinate system expressed in spherical coordinates, i.e. latitude and longitude, a projected coordinate system such as Universal Transverse Mercator (UTM), etc. It is also possible to express the location information 21 of the first user 2 using a national coordinate system, such as the Swiss coordinate system. In one embodiment, a correction is provided to the location information 21 of the first user 2 to compensate for continental drift. In one embodiment, the location information 21 is for an emulated location of the first user 2, meaning that the first user 2 is not actually physically present at the location, but instead a representation of the first user 2 is located at the physical location. For example, the first user 2 selects a representation of the first user 2 to be virtually located at the location 2. For example, the first user 2 may be physically at home, but the emulated location information 21 of the first user 2 is located at the user's workplace. Region R around the first user 2 is defined as the two-dimensional or three-dimensional space in which the first user 2 is located. In some embodiments or configurations, region R is a geometric object, such as a sphere centered on the position represented by the position information 21 of the first user 2, or a rectangle with the first user 2 located inside. In one embodiment, region R is defined with respect to the direction in which the first user 2 is facing, such that region R extends further from the first user 2 in the direction in which the first user 2 is facing (e.g., facing and / or looking) than in directions in which the first user 2 is not facing.

[0031] The real-world space is virtually divided into a hierarchy of virtual real estate objects 41 arranged similarly to a tree data structure, with parent virtual real estate objects containing child virtual real estate objects. The hierarchy includes multiple levels. Higher level virtual real estate objects 41 cover larger geographic areas, such as a country, state, or city. Lower level virtual real estate objects 41 cover smaller areas, such as city squares, blocks, buildings, and so on. At even lower levels, virtual real estate objects 41 cover even smaller spaces, such as floors of a building, stores, individual rooms, and / or individual sections or shelves of a room. For illustrative purposes, multiple virtual real estate objects 41 are shown in FIG. 4. In one example, the virtual real estate objects 41 are three-dimensional (3D) objects defined by a set of polygons, a set of edges and vertices, and / or voxels. In another example, the virtual real estate objects 41 are two-dimensional (2D) objects represented by a set of lines and / or a set of points that define a virtual fence around the virtual real estate objects 41. Each virtual real estate object 41 is assigned a virtual real estate object location 42 in a virtual real estate directory system 4 that associates the virtual real estate object 41 with a real world location, such that the virtual real estate object location 42 is represented by coordinates in a coordinate system. As above for the location information 21 of the first user 2, the coordinate system may be, for example, spherical or projected coordinates based on a regional or national coordinate system. In one embodiment, the coordinates are further corrected to account for continental drift and other error factors.

[0032] Each virtual real estate object 41 has a bounding volume V, which is a closed volume that contains the virtual real estate object 41. The bounding volume V is defined using polygons, vertices and edges, and / or other geometric objects. A bounding volume V being a closed volume means that a point cannot move from inside to outside the bounding volume V without passing through the exterior surface of the bounding volume V. As one example, the bounding volume V exactly matches the shape of the virtual real estate object 41. In another example, the bounding volume V is a relatively simpler shape than the relatively more complex virtual real estate object 41, meaning that it is represented with fewer polygons and / or fewer vertices and edges than the virtual real estate object 41, but is guaranteed to completely contain the virtual real estate object 41, e.g., no point of the virtual real estate object 41 is outside the bounding volume V. In one embodiment, the bounding volume V is a rectangular parallelepiped.

[0033] FIG. 5 is a diagram showing a schematic view F of a first user 2 in an indoor environment. The view F is shown by a dashed line. The view F is defined, for example, by a solid angle whose apex is at the first user 2 or the communication device 1. Alternatively, the view F is defined as a square pyramid with horizontal and vertical angles and whose apex is at the first user 2. The position of the first user 2 is defined by the position information 21. The first user 2 is located in front of a table. Multiple cameras 12 take images of the first user 2 from multiple angles. One or more microphones 15 capture voice data of the first user 2. The rendering system 11 renders an augmented reality content 51 including an augmented reality content 53 and user data received from a user with whom the first user 2 is interacting, in particular a second user 2B. The rendered augmented reality content 51 also includes visual augmented reality content and auditory augmented reality content. The augmented reality content 51 may further include other sensory modalities, for example touch, smell, or taste, depending on the capabilities of the particular rendering system 11. Depending on the embodiment, the received user data includes a visual representation 54 of the second user 2B rendered by the rendering system 11 and auditory augmented reality content played by one or more speakers 14. In one embodiment, the augmented reality object 53 includes visual content, e.g. a 3D model represented by a set of polygons, together with auditory content, e.g. represented by an audio track. In one embodiment, besides the 3D model, the augmented reality content 51 includes dynamic content, interactive content with which the first user 2 can interact, etc.

[0034] The augmented reality content 51 is rendered for the first user 2 using the communication device 1. Depending on the particular hardware configuration of the communication device 1, e.g., what type of device the communication device 1 is, what modules it includes, and what accessory modules are connected to it, the rendering of the augmented reality content 51 may involve rendering and / or playing one or more types of sensory content. The augmented reality content 51 is rendered by the rendering system 11 so that it remains in a fixed position relative to the table. In particular, the camera 12 extracts coordinates C of the environment of the first user 2, e.g. the table, from the field of view F of the first user 2. The communication device 1 then renders (draws) the augmented reality content 51 in the air so that it remains stationary relative to real-world objects in the environment, such as the table. In one embodiment, the rendered augmented reality content 51 is anchored to real world objects, e.g. building features such as floors, walls, or other surfaces of a building, or items of furniture such as a table. The augmented reality content 51 is rendered to maintain a fixed position relative to the real world objects, such that the augmented reality content 51 remains in place (position) even as the first user moves their head or body.

[0035] Using the field of view F, in one embodiment, coordinates C are overlaid onto the real world, allowing coordinates of real-world objects and features to be extracted from the field of view F. For example, the camera 12 of the communication device 1 captures the field of view F as one or more images, and the communication device 1 extracts points, edges, and other features of real-world objects, such as building corners, street edges, skyline, and other people, from the one or more images and stores these extracted features. The extracted features are used by the communication device 1, for example, to improve the calculated position of the first user 2, to improve the determined orientation of the first user 2, and / or to improve the positioning and rendering of the augmented reality content 51. In one embodiment, the field of view F is defined using the field of view of the camera 12 of the mobile communication device 1. Furthermore, the accelerometer and / or gyro sensor of the communication device 1 is used by the communication device 1 to determine the orientation of the communication device 1. The orientation of the communication device 1 is then used to determine the field of view F. In one embodiment, the field of view F is defined as the field of view of the first user 2 by using an orientation module of the communication device 1 to determine an orientation of the first user 2, such as an orientation of the body of the first user 2, an orientation of the head of the first user 2, and / or an orientation of the eyes of the first user 2. In some embodiments, the orientation module of the communication device 1 includes a head tracker, an eye tracker, or other component(s) for detecting the orientation of the first user 2.

[0036] FIG. 6 is a block diagram that illustrates a schematic of the virtual real estate directory system 4. The virtual real estate directory system 4 is realized using a server computer system including one or more server computers. The server computers of the virtual real estate directory system 4 may be co-located or distributed, i.e., remotely located from each other at different locations. The server computers of the virtual real estate directory system 4 include one or more processors, memory, and a communication interface for exchanging data with other systems and / or devices using the communication network 3. For example, the virtual real estate directory system 4 exchanges data with the mobile communication device 1 using the communication network 3. The virtual real estate directory system 4 is implemented on the server computer as a hierarchical system and / or database that includes virtual real estate objects 41.

[0037] As described above in connection with FIG. 4, the real-world space is virtually divided into a hierarchy of virtual real estate objects 41, with higher-level virtual real estate objects 41 covering larger geographic areas, such as countries, states, or cities, and lower-level virtual real estate objects 41 covering smaller areas. The virtual real estate directory system 4 is arranged in a similar hierarchical structure with one or more higher-level server computers containing the higher-level virtual real estate objects 41 and one or more lower-level server computers containing the lower-level virtual real estate objects 41. In one embodiment, the server computers are located in the same location or area as the virtual real estate objects 41 that they store. For example, a server computer containing a virtual real estate object 41 for a particular country is located in that same country, and a server computer containing a virtual real estate object 41 for a particular city is located in that same city.

[0038] The virtual real estate object 41 includes a virtual real estate object location 42, an identification 43 that uniquely identifies the virtual real estate object 41, a network address of an augmented reality interaction server 44, and a bounding volume V. The identification 43 includes at least an identifier that uniquely identifies the virtual real estate object 41, or a hash value of the virtual real estate object 41. The hash value is generated by hashing the virtual real estate object 41 or one or more components thereof. In addition, the virtual real estate object 41 includes information about the owner and / or assignee of the virtual real estate object, and 3D content related to a more complex boundary of the virtual real estate object 41. Depending on the hierarchical level, a given virtual real estate object 41 includes identification information of child virtual real estate objects 41 included in the given virtual real estate object 41, and identification information of parent virtual real estate objects 41 of which the given virtual real estate object 41 is itself a child virtual real estate object 41. This allows efficient mapping and traversal of the virtual real estate objects 41 in the virtual real estate directory system 4. In one embodiment, the identification information 43 includes and / or defines a bounding volume V. In further embodiments, the identification information 43 includes one or more of an identifier, a hash value, a bounding volume V, or a virtual real estate object location 42.

[0039] FIG. 7 is a block diagram that illustrates a schematic representation of an augmented reality interaction server 5 including an augmented reality object 53. The augmented reality interaction server 5 includes one or more server computers. The server computer of the augmented reality interaction server 5 includes one or more processors, a memory, and a communication interface for exchanging data with other systems and / or devices using the communication network 3. For example, the augmented reality interaction server 5 exchanges data with the communication device 1 using the communication network 3. The augmented reality interaction server 5 is separate from the virtual real estate directory system 4. For example, the augmented reality interaction server 5 is located near the location where the virtual real estate object 41 is assigned. The augmented reality interaction 51 includes sensory content of one or more sensory modalities, as described above in relation to FIG. 4. In one embodiment, the augmented reality interaction server 5 is co-located with the virtual real estate directory system 4 and / or implemented on a server of the virtual real estate directory system 4 .

[0040] In one embodiment, the augmented reality interaction server 5 hosts one or more augmented reality rooms. In particular, one or more processors of the augmented reality interaction server 5 are configured to generate one or more augmented reality rooms. An augmented reality room is a virtual three-dimensional space configured to allow a first user 2 and a second user 2B to participate in the augmented reality room. In one embodiment, the augmented reality room corresponds to a physical space such as one or more of an estate, a building, an auditorium, or a room. The augmented reality interaction server 5 is configured for secure data exchange, such that the first user 2, the second user 2B, and / or further users can securely exchange data with each other in the first augmented reality room. In this case, any data exchanged in the augmented reality room is not known or accessible by other users in other augmented reality rooms. In a first embodiment, the augmented reality interaction server 5 is configured to generate an encryption key for the augmented reality room. In a first example, the augmented reality interaction server 5 generates a public / private key pair and transmits the public key to the first user 2 and the second user 2B. The communication device 1 of the first user 2 is configured to receive the public key of the augmented reality room from the augmented reality interaction server 5. The communication device 1 of the first user 2 is configured to transmit the public key of the communication device 1 to the augmented reality room of the augmented reality interaction server 5. The communication device 1 is configured to encrypt data to be transmitted to the augmented reality room using the public key of the augmented reality room and to decrypt data received from the augmented reality room using the private key of the communication device 1. In this way, a high level of security is provided for data transmitted between the communication device 1 and the augmented reality room. The augmented reality room encrypts data to be transmitted to the communication device 1 of the first user 2 using the public key of the communication device 1 and encrypts data to be transmitted to the communication device 1B of the second user 2B using the public key of the communication device 1B of the second user 2B.

[0041] In one embodiment, the data transmitted from the communication device 1 of the first user 2 to the augmented reality room comprises address information consisting of one or more of the following: the public name of the second user 2B, one or more public names of the further users or a public flag. The address information is, in some embodiments, encrypted with the public encryption key of the augmented reality room. The augmented reality interaction server 5 is configured to forward the data transmitted from the communication device 1 depending on the address information, in particular to all users in the augmented reality room if the address information comprises a public flag or to a set of users according to a list of public names contained in the address information. In some embodiments, the first user data transmitted by the communication device 1 of the first user 2 to the communication device 1B of the second user 2B is encrypted with a public key or a symmetric session key established between the communication device 1 of the first user 2 and the communication device 1B of the second user 2B, so that neither the augmented reality room nor any further user can access the unencrypted first user data. The above arrangement of communication between the augmented reality interaction server 5, the first user 1 and the second user 1B ensures a high degree of security since the data is encrypted end-to-end between the first user 1 and the second user 1B.

[0042] In one embodiment, the communication device 1 receives an encryption key from the augmented reality room. The encryption key is securely encrypted using the public key of the communication device 1 received in the augmented reality room. The communication device 1 is configured to decrypt the encryption key using the private key of the communication device 1. Data transmitted by the communication device 1 to the augmented reality room are then encrypted in the communication device 1 using the encryption key. In this embodiment, the encryption key of the augmented reality room serves as a symmetric session key that is securely transmitted to the first user 1, the second user 1B and further users logged in to the augmented reality room. Further users participating in the augmented reality room receive the encryption key from the augmented reality room and can then participate in the data of the augmented reality room. This configuration allows fast and secure data exchange between all participants of the augmented reality room. At the same time, the communication device 1 is also configured to be able to perform end-to-end encrypted data exchange with one or more participants of the augmented reality room.

[0043] In one embodiment, the augmented reality room is configured to generate a new encryption key and distribute the new encryption key to the communication device 1 of the first user 2 and the communication device 1B of the second user 2B when a further participant of the augmented reality room leaves the augmented reality room. This ensures the continued security of the augmented reality room and the data exchanged therein. In particular, the communication device 1 is configured to receive from the augmented reality interaction server 5 the new encryption key encrypted with the public key of the communication device 1, to decrypt the new encryption key using the private key stored in the communication device 1, to replace the stored encryption key with the new encryption key, and to encrypt subsequent data to be sent to the augmented reality interaction server 5 with the new encryption key.

[0044] FIG. 8 is a flow diagram illustrating a series of steps for providing an augmented reality interaction 51 between a first user 2 of a first communication device 1 and a second user 2B of a second communication device 1B. In step S1, location information 21 of the first user 2 is obtained. The location information 21 of the first user 2 is obtained by the electronic circuit 13 of the communication device 1. The location information 21 relates to the physical location of the first user 2 or to an emulated location of the first user 2. The emulated location of the first user 2 corresponds to a real-world location selected by the first user 2, for example by choosing a location on a map, or a real-world location assigned to the first user 2, for example by belonging to a particular group, for example their company. For example, if the first user 2 is at home, the first user 2 can select his / her workplace as his / her emulated location, and the location information 21 of the first user 2 is henceforth related to that emulated location. In one embodiment, the location information 21 relates to the physical location of the first user 2, and the location module of the mobile communication device 1 determines the physical location using trilateration based on signals from global navigation satellite systems, e.g. Global Positioning System (GPS), Glonass, and / or Galileo. In one embodiment, the location module further uses the presence (location) of other wireless networks, e.g. mobile radio or WLAN networks, to determine the physical location of the first user 2, e.g. by trilateration. In one embodiment, the location module further uses beacons, e.g. Bluetooth or Ultra Wide Band (UWB) beacons, to determine the physical location of the first user 2. This is particularly advantageous indoors, where the location information 21 of the first user 2 may not be accurately determined using satellite-based systems. The electronic circuit 13 obtains the location from the location module. In one embodiment, a device separate from the communication device 1 of the first user 2 determines location information 21 of the first user 2 and transmits the location information 21 to the mobile communication device 2 .

[0045] In step S2, the communication device 1 transmits a query request to the virtual real estate directory system 4 using the communication network 3 in a transmission T2. ​​The query request includes the location information 21 of the first user 2. In one embodiment, the query request further includes the orientation of the first user 2, the field of view F of the first user 2, a user identifier of the first user 2, a device identifier of the communication device 1, a network address of the communication device 1, a device type of the communication device 1, a hardware configuration of the communication device 1, and / or a movement speed of the first user 2 of the communication device 1. The field of view F of the first user 2 and the device type of the communication device 1 enable, among other things, the recipient of the query request to provide an appropriate and complete query response to the communication device 1, as will be described below. In step S201, the virtual real estate directory system 4 receives a query request from the communication device 1. The query request is received at a server computer system. In particular, the query request is received at a particular server computer of the server computer system.

[0046] In step S202, the virtual real estate directory system 4 processes the query request. The query request is processed by a server computer system. The processing of the query request includes checking whether a particular server computer of the virtual real estate directory system 4 that received the query request should forward the query request to another server computer of the virtual real estate directory system 4. The query request is forwarded depending on one or more of the following factors: the physical location to which the location information 21 of the first user 2 relates, the user identifier of the first user 2, the device identifier of the communication device 1, and the network address of the communication device 1. In particular, depending on the location information 21 of the first user 2, the query request is forwarded to one or more further server computers of the virtual real estate directory system 4 that are responsible for the geographical area corresponding to the location information 21 of the first user 2, i.e., that contain the virtual real estate objects 41 that are assigned to the virtual real estate positions 42 within the geographical area indicated by the location information 21 of the first user 2. In one embodiment, forwarding of a query request in the virtual real estate directory system 4 includes one or more separate forwarding steps in which a given receiving server computer forwards the query request to one or more additional server computers which process and forward the query results to other server computers. In this manner, the query request traverses (traverses) server computers arranged in a hierarchical system and / or database.

[0047] In step S203, the virtual real estate directory system 4 generates a query response. The query response includes a virtual real estate object 41 located near a location defined by the location information 21 (physical location or emulated location) of the first user 2. In particular, the query response includes a virtual real estate object 41 assigned to a virtual real estate location 42 within a region R around the location defined by the location information 21 of the first user 2. In one embodiment, the query response includes a virtual real estate object 41 located within a field of view F of the first user 1. For example, the query response includes a virtual real estate object 41 stored in one or more server computers of the virtual real estate directory system 4. In particular, the query response can include multiple query response messages from multiple server computers of the virtual real estate directory system 4, each query response message including a virtual real estate object 41 assigned to a virtual real estate location 42 within a region R around the location of the first user 2. For example, a server computer of the virtual real estate directory system 4 responsible for a city may generate a query response message with large virtual real estate objects 41 corresponding to city blocks, landmarks, and / or natural geographic features, while another server computer of the virtual real estate directory system 4 responsible for a particular building may generate another query response message with smaller virtual real estate objects 41 corresponding to the particular building along with a particular floor, unit, and / or area of ​​the building.

[0048] In step S204, a query response is sent in transmission T3 from the virtual real estate directory system 4 to the communication device 1 via the communication network 3. The query response is sent from a server computer system of the virtual real estate directory system 4 and includes one or more individual query response messages from one or more server computers of the virtual real estate directory system 4. In step S3, at the communication device 1, a query response is received. The query response comprises one or more virtual real estate objects 41. Each virtual real estate object 41 comprises a virtual real estate object position 42, an identification 43, a network address 44 of the augmented reality interaction server 5, and a bounding volume V. In one embodiment, at least one of the query request or the query response is encrypted. In one example, the query request is encrypted by the communication device 1 using a public key of the virtual real estate directory system 4 obtained by the communication device 1. In another example, the query response is encrypted by the virtual real estate directory system 4 using a public key of the communication device 1 obtained by the virtual real estate directory system 4. In another example, the communication device 1 and the virtual real estate directory system 4 establish a secure connection, for example using the Secure Sockets Layer (SSL) protocol, when the communication device 1 transmits the query request.

[0049] In step S4, the query response is processed and stored in the communication device 1. In some embodiments and / or configurations, processing the query response includes the communication device 1 using the orientation and / or field of view F of the first user 2 to generate a subset of virtual real estate objects 41 having virtual real estate object positions 42 located in the direction of the orientation of the first user 2 and / or located within the field of view F of the first user 2. Additionally, processing the query response may include the communication device 1 using a movement speed of the first user 2 to generate a subset of virtual real estate objects 41 having virtual real estate object positions 42 located in a direction in which the first user 2 moves. In one embodiment, the virtual real estate object 41 is displayed using the rendering system 11 of the communication device 1 such that the virtual real estate object 41 appears on the rendering system 11 overlaid on the real world in a semi-transparent or semi-transparent manner at the virtual real estate object location 42. In one embodiment, this includes displaying a bounding volume V of the virtual real estate object 41. In particular, when the communication device 1 is in an indoor environment, the rendering system 11 is used to display the virtual real estate object 41 around the communication device 1. For example, the virtual real estate object 41 may encompass the first user 2 if the location information 21 of the first user 2 is associated with an emulated location. The communication device 1 generates an interaction request for each virtual real estate object 41 that the communication device 1 receives in the query response. In one embodiment, the communication device 1 generates interaction requests only for those virtual real estate objects 41 of the subset of the virtual real estate objects 41. Each specific interaction request includes an identification 43 of the specific virtual real estate object 41. Additionally, each specific interaction request may include a user identifier of the first user 2, a device identifier of the communication device 1, a network address of the communication device 1, a device type of the communication device 1, and / or a hardware configuration of the communication device 1.

[0050] In step S5, the communication device 1 sends one or more interaction requests to one or more augmented reality interaction servers 5 in a transmission T5 via the communication network 3 using a specific network address 44 of each specific augmented reality interaction server 5. Depending on the embodiment and / or scenario, the communication device 1 sends an interaction request to each augmented reality interaction server 5. For simplicity, the following steps describe the relationship between a single interaction request and a single augmented reality server 5. In step S501, the augmented reality interaction server 5 receives an interaction request in transmission T5. Access permissions are determined for all augmented reality objects 51 in the virtual real estate object 41. Furthermore, access permissions for an augmented reality interaction between the first user 2 and other users, in particular the second user 2B, are determined. The access permissions for the augmented reality interaction ensure that the first user 2 is permitted to interact in augmented reality with other users, in particular the second user 2B, in the virtual real estate object. In particular, the access permissions for the augmented reality interaction include permissions for the first user 2 to transmit first user data to the second user 2B and for the first user 2 to receive second user data from the second user 2B. The user data (e.g., the first user data and / or the second user data) include data files stored in the communication device 1 or acquired by the communication device 1. In one embodiment, the user data further includes the user's personal data, as well as the user's visual and / or audio data, as described in more detail below. In one embodiment, different augmented reality objects 53 have different access permission levels, such that the user 2 is granted access permission for some but not necessarily all augmented reality objects 53. Depending on the embodiment, the augmented reality interaction server 5 can receive a separate interaction request from the communication device 1 for each augmented reality object 53.

[0051] In step S502, the augmented reality interaction server 5 uses the interaction request to determine the access permission of the communication device 1 and / or the first user 2. In particular, the augmented reality interaction server 5 checks whether the hardware identifier of the communication device 1 included in the interaction request is authorized. Furthermore, the credentials of the first user 2 included in the interaction request, such as a user name and a user password, are verified to determine the access permission. In step S503, the augmented reality interaction server 5 sends an access permission confirmation in transmission T6 to the communication device 1 of the first user 2. In step S6, the communication device 1 receives the access permission confirmation. In one embodiment, the above steps related to access authorization are performed over an encrypted communication channel, for example using SSL encryption. Furthermore, certain augmented reality interaction servers 5 may require two-factor authentication. In one example, the augmented reality interaction server 5 sends a verification code to another user device of the first user 2, who is asked to enter the verification code into the communication device 1, and then the verification code is sent back to the augmented reality interaction server 5, which then determines whether the verification codes match. In one embodiment, the communication device 1 is configured to receive from the augmented reality interaction server 5 as part of the authorization of step S6 a user identifier that is dynamically generated by the augmented reality interaction server 5. The user identifier is transmitted by the communication device 1 in a subsequent data transmission to the augmented reality interaction server 5. Depending on the embodiment, the communication device 1 transmits its public name and / or the user identifier to the augmented reality interaction server 5.

[0052] In one embodiment, the communication device 1 is configured to receive a digital certificate from the augmented reality interaction server 5, the digital certificate including one or more digital credentials of the augmented reality interaction server 5. The communication device 1 is configured to verify the validity of the digital certificate for further communication with the augmented reality interaction server 5. After receiving the access permission, the first user 2 exchanges data with other users located in the virtual real estate object 41 via the augmented reality interaction server 5. In particular, as will be explained in more detail below, the communication device 1 of the first user 2 exchanges first user data with the communication device 1B of a second user 2B, also located in the virtual real estate object 41, via the augmented reality interaction server 5. The communication device 1 thereby enables data exchange between the first user 2 and other users physically or virtually located at the virtual real estate object location 42 (i.e. at a corresponding emulated location). This allows a group of users located at or near the virtual real estate object 41 to interact in augmented reality. In one embodiment, after receiving the access permission, the communication device 1 of the first user 2 receives from the augmented reality interaction server 5 a room list of one or more augmented reality rooms, but may also receive a user list of all users currently present in a particular augmented reality room together with the room name of the particular augmented reality room. The communication device 1 is configured to allow the first user 2 to select a particular augmented reality room. Upon selection of the particular augmented reality room, the first user 2 performs an exchange of credentials with the augmented reality room, e.g. a public key exchange or establishment of a common session key, as described above, in order to increase the security of the data exchange with the augmented reality room, in particular with a second user 2B logged in to the same augmented reality room.

[0053] In one embodiment, after being granted access to the augmented reality room, the communication device 1 of the first user 2 receives one or more of the following: a location, a bounding volume of all other users of the augmented reality room, a list of unoccupied locations of the augmented reality room, or common data of the augmented reality room. The location is a three-dimensional location in the augmented reality room of a visual representation of another user, in particular of the visual representation 54 of the second user 2B. The bounding volume is a three-dimensional volume that encompasses the visual representation of another user, in particular of the visual representation 54 of the second user 2B. The unoccupied locations list comprises one or more locations in the three-dimensional space in the augmented reality room that are not occupied by other users. The common data of the augmented reality room is data provided by the augmented reality room to all participants of the augmented reality room or data provided by other users of the augmented reality room to all participants of the augmented reality room. In one embodiment, the first user 2 occupies a specific position in the augmented reality room. More specifically, a visual representation of the first user 2 occupies a specific position in the augmented reality room, and the rendering system 11 of the communication device 1 is configured to render the augmented reality room with the visual perspective of the first user 2. The communication device 1 of the first user 2 is configured to ensure that other users, in particular the visual representations of the other users, do not occupy the same space as the first user 2. In particular, that the bounding volumes of the other users do not overlap with the bounding volumes of the first user 1. The communication device 1 is configured not to render the visual representations of the other users in case of such an overlap of bounding volumes.

[0054] In step S7, the first user 2 transmits the first user data to the augmented reality interaction server 5 in transmission T7. In step S71, the augmented reality interaction server 5 receives the first user data and routes the first user data to the communication device 1B of the second user 2B in transmission T7B. Furthermore, the augmented reality interaction server 5 can route the first user data to additional users who are also physically or virtually located in the virtual real estate object 41. The communication device 1B of the second user 2B receives the first user data in step S72. The communication device 1B transmits the second user data of the second user 2B to the augmented reality interaction server 5 in transmission T8B. In step S74, the augmented reality interaction server 5 then routes the second user data to the first user 2 in transmission T8. In step S8, the communication device 1 of the first user 2 receives the second user data. In one embodiment, the second user data is processed using a content filter stored in the communication device 1. Furthermore, the content filtering can also be performed in the augmented reality interaction server 5. In particular, the augmented reality interaction server 5 processes the received data (i.e. the first user data and / or the second user data) using a content filter and rejects and does not transmit the received data if the received data matches one of the filter criteria of the content filter. The content filtering in the augmented reality interaction server 5 ensures that only authorized content is routed through the augmented reality interaction server 5. The content filtering in the communication device 1 ensures that only authorized content is stored in the communication device 1 and rendered and / or played by the communication device 1.

[0055] In one embodiment, the communication device 1 further processes the received second user data according to user settings, which include, for example, language settings. For example, if the second user data includes voice data related to the conversation data, the electronic circuit 13 of the communication device 1 processes the conversation data according to the language settings, automatically translates the conversation data from a first language of the conversation data to a second language according to the language settings, or generates text from the conversation data of the voice data. Thus, a user physically located at the virtual real estate object location 42 can interact (e.g., exchange data) with a user virtually located at the virtual real estate object location 42, and vice versa. In particular, the location data 21 of the first user 2 corresponds to an emulated or physical location, and thus the first user 2 can interact with the second user 2B.

[0056] FIG. 9 is a flow diagram illustrating an exemplary sequence of steps for an augmented reality interaction. In step S9, the communication device 1 of the first user 2, in particular the one or more cameras 12, captures one or more images of the first user 2. The images may be still or video images in two or three dimensions of the first user 2 and are transmitted from the one or more cameras 12 to the electronic circuitry 13 of the communication device 1. In one embodiment, voice data of the first user 2 is recorded using one or more microphones 15 and transmitted to the electronic circuitry of the communication device 1. In step S10, the electronic circuitry uses the one or more images to determine visual representation data of the first user 2. The visual representation data is data used to render a visual representation of the first user 2. The visual representation data is used in one embodiment to generate a lifelike two- or three-dimensional rendering of the first user 2. The visual representation data is used in one embodiment to generate an avatar for the first user 2, which is a graphical two- or three-dimensional representation of a user that is not intended to be lifelike. In some embodiments, the visual representation data includes user motion data relating to the movement of the first user 2, pose data relating to the position of the body, particularly the face, of the first user 2, and / or facial expression data, e.g. consisting of facial expressions, eye movements or emotions of the first user 2. In step S11, the communication device 1 transmits the visual representation data to the augmented reality interaction server 5, which routes the data in step S111 to the communication device 1B of the second user 2B. The second user 2B receives the visual representation data of the first user 2 in step S112. The communication device 1B of the second user 2B then uses the visual representation data to render a visual representation of the first user 2. In one embodiment, rendering the visual representation of the first user 2 is using a pre-stored model of the first user 2 stored in the communication device 1B of the second user 2B. In one embodiment, the voice data of the first user 2 is transmitted to the augmented reality interaction server 5 .

[0057] In step S113, the communication device 1B of the second user 2B takes one or more images of the second user 2B and determines visual representation data of the second user 2B. In step S114, the communication device 1B of the second user 2B transmits the visual representation data of the second user 2B to the augmented reality interaction server 5 which routes the data to the communication device 1 of the first user 2 in step S115. The communication device 1 of the first user 2 receives the visual representation data of the second user 2B in step S12. In step S13, the communication device 1 uses the visual representation data and the rendering system 11 to render a visual representation 54 of the second user 2B. In one embodiment, the communication device 1B also records the second user's voice data, which is sent to the augmented reality server 5 and routed by the augmented reality server 5 to the communication device 1 of the first user 2. The communication device 1 plays the second user's voice data using one or more speakers 14. Preferably, the speakers 14 play the second user's voice data as if it were emanating from a visual representation 54 of the second user 2B to enhance the sense of realism. As explained above in relation to FIG. 8, the communication device 1 may further process the received data, including the visual representation data and / or the voice data of the second user, in accordance with user settings.

[0058] In one embodiment, rendering the visual representation 54 of the second user 2B consists in using a pre-stored model of the second user 2B stored in the communication device 1 in conjunction with visual representation data, in particular user motion data. Using the user motion data in conjunction with the visual representation data has the advantage that less data needs to be transmitted, since the user motion data is a compact representation of only the movements of the second user 2B and does not include other details of the visual appearance of the second user 2B. In one embodiment, the above steps are performed continuously such that visual representation data is continuously streamed from the communication device 1 of the first user 2 to the communication device 2 of the second user 2B and vice versa. In one embodiment, the streaming occurs in real time such that a natural interaction in augmented reality is achieved between the first user 2 and the second user 2B. In one embodiment, this is achieved by positioning the augmented reality interaction server 5, which is placed at the virtual real estate object location 42, such that the transmission delay between the users and the augmented reality interaction server 5 is reduced. In one embodiment, in addition to visual representation data being exchanged, audio data is also exchanged between the first user 2 and the second user 2B via the augmented reality interaction server 5. In particular, audio data recorded by one or more microphones 15 is transmitted by the communication device 1 of the first user 1 via the augmented reality interaction server 5 to the communication device 1B of the second user 1B.

[0059] 10 is a flow diagram illustrating an exemplary sequence of steps for interacting with an augmented reality object, which allows a first user 2 and a second user 2B to interact with a common augmented reality object 53. In particular, the communication device 1 of the first user 2 and the communication device 1B of the second user 2B continuously exchange data with the augmented reality interaction server 5 such that the augmented reality object 53 is continuously updated in both the communication device 1 of the first user 2 and the communication device 1B of the second user 2B. In step S14, the augmented reality interaction server 5 transmits the augmented reality object 53 to the communication device 1 of the first user 2 and to the communication device 2 of the second user 2B. The augmented reality object 53 is a representation of an object that can be rendered in augmented reality, in particular rendered by the communication device 1. The augmented reality object 53 can encompass one or more sensory modalities, for example visual, tactile and / or auditory modalities. The augmented reality object 53 comprises, for example, two-dimensional or three-dimensional images (still and / or moving images), sound recordings and / or scents, depending on the embodiment. The two-dimensional images represent, for example, graphic images, photographs or video clips. The three-dimensional images represent, for example, CAD objects, 3D models or topographical maps. The augmented reality object 53 comprises, for example, a document, which consists of textual and / or image content.

[0060] In steps S15 and S15', the communication device 1 of the first user 2 and the communication device 1B of the second user 2B, respectively, receive the augmented reality object 53. In one embodiment, in addition to the augmented reality object 53, an identification 43 of each particular virtual real estate object 41 to which the augmented reality object 53 is assigned is received along with the augmented reality object 53. The received augmented reality object 53 is then processed in the communication device 1 by associating the augmented reality object 53 with the particular virtual real estate object 41, in particular by matching the received identification 43 with the identification 43 of the particular virtual real estate object 41 stored in the communication device 1 (as part of receiving the query response, as described above in relation to step S3 of FIG. 8). In one embodiment, the communication device 1 processes the augmented reality object 53 using a set of content filters stored in the communication device 1. The content filters are used to determine whether the augmented reality object 53 is accepted or rejected. In steps S16 and S16', the communication device 1 of the first user 2 and the communication device 1B of the second user 2B, respectively, render the augmented reality object 53. In one embodiment, the hardware configuration of the communication device 1 may be used to determine which aspects of the augmented reality object 53 are rendered. In particular, whether visual, haptic and / or auditory aspects of the augmented reality object 53 are rendered depends on whether the hardware configuration indicates that the communication device 1 has a hardware module capable of rendering or playing a given content type. In one embodiment, different content types are encrypted with different encryption keys. In particular, the augmented reality room implemented in the augmented reality interaction server 5 generates a content session key for each content type and securely distributes the content session key to the communication device 1 of the first user 2 and to other participants of the augmented reality room (i.e. the communication devices of the other participants of the augmented reality room). The communication device 1 is configured to encrypt data it sends to the augmented reality room with the content session key according to the content type of the data and to decrypt data it receives from the augmented reality room with the content session key according to the content type of the data.

[0061] In one embodiment, the communication device 1 of the first user 2 renders the augmented reality object 53 according to user settings that determine for example the size and the position of the augmented reality object 53. Thereby, the first user 2 can customize the appearance and / or functionality of the augmented reality object 53 according to his requirements, in particular according to the limitations of the communication device 1 or the space in which the augmented reality content 51 is rendered. The augmented reality object 53, and more generally all augmented reality content 51, are rendered such that the augmented reality object 53 is located (mapped and positioned) within the virtual real estate object 41. For example, if the augmented reality object 53 includes visual content, this visual content is made to appear on the rendering system 11 within the virtual real estate object 41 at the virtual real estate object location 42. In particular, the visual content is made to be rendered inside the bounding volume V of the virtual real estate object 41. Furthermore, if the augmented reality object 53 includes auditory content, the auditory content is made to be played as if originating from a virtual sound source within the virtual real estate object 41 at the virtual real estate object location 42, more particularly from a virtual sound source of the augmented reality object 53 or a virtual sound source within it. Furthermore, the communication device 1 is configured to play the auditory content to the first user 2 using the one or more speakers 14 only if the position information 21 of the first user 2 is within the virtual real estate object 41 or its bounding volume V.

[0062] In one embodiment where the communication device 1 is a smartphone, the electronic circuitry 13 is configured to instruct the one or more cameras 12 to capture an image of the real world and to instruct the rendering system 11 to display the real world image. The electronic circuitry 13 then renders on the rendering system 11 an image (visual content) of the augmented reality object 53, with the visual content overlaid on the image of the real world, such that both the real world and the augmented reality object 53 are displayed simultaneously. In some embodiments, the augmented reality object 53 is rendered in step S16 using a template augmented reality object stored in the communication device 1 of the first user 1. The template augmented reality object is configured to provide the communication device 1, and in particular the rendering system 11, with a framework for rendering the augmented reality object 53. In one embodiment, the template augmented reality object is a member of a particular type of augmented reality object, and the communication device 1 is configured to identify the type of augmented reality object 53 and render the augmented reality object 53 using the template augmented reality object assigned to that type of augmented reality object. An advantage of using a template augmented reality object is that less data has to be sent from the augmented reality interaction server 5 to the communication device 1 in order for the communication device 1 to render the augmented reality object 1.

[0063] In step S17, the communication device 1 of the first user 2 determines user interaction data of the first user 2 with the augmented reality object 53. In particular, user inputs of the first user 2 are recorded by the electronic circuit 13. The user inputs include user interface inputs, such as those entered using a keyboard, gesture inputs registered using the motion tracker 16 and / or one or more cameras 12, and voice inputs registered using one or more speakers 14. The user inputs relate to the addition and / or modification of the augmented reality object 53, for example, if the augmented reality object 53 represents a text document, the user inputs can relate to the addition, deletion or modification of the text document. The user interaction data is transmitted by the communication device 1 to the augmented reality interaction server 5 in step S18. In step S181, the augmented reality interaction server 5 receives user interaction data. In one embodiment, the augmented reality interaction server 5 further receives user interaction data from other users via their communication devices, in particular from the communication device 1B of the second user 2B. In step S182, the augmented reality interaction server 5 uses the user interaction data to determine an updated augmented reality object 53. In one embodiment, the augmented reality interaction server 5 determines object update data for the augmented reality object 53. The object update data is configured to enable the communication devices of those users that received the augmented reality object 53 to generate the updated augmented reality object 53. Determining the object update data is advantageous since it only concerns the differences between the augmented reality object 53 and the updated augmented reality object 53, rather than the entire updated augmented reality object 53, and generally requires less data to store.

[0064] In step S191, the augmented reality interaction server 5 transmits the updated augmented reality object to the communication devices, in particular the communication device 1 of the first user 2 and the communication device 1B of the second user 2B. In one embodiment, instead of the updated augmented reality object, object update data is transmitted. In steps S19 and S19', the communication device 1 of the first user 2 and the communication device 1B of the second user 2B, respectively, receive the updated augmented reality object. In steps S20 and S20', the communication device 1 of the first user 2 and the communication device 1B of the second user 2B, respectively, render the updated augmented reality object. In an embodiment in which the object update data has been transmitted, the communication device 1 of the first user 2 and the communication device 1B of the second user 2B, respectively, receive the object update data and render the updated augmented reality object 53 using the object update data, respectively. The above steps allow both the first user 2 and the second user 2B to render a copy of the same augmented reality object 53. User inputs on the augmented reality object 53 of either user are registered as an updated augmented reality object or as user interaction data that is used by the augmented reality interaction server 5 to determine its object update data. The updated augmented reality object or the object update data for the augmented reality object 53 is sent to the communication devices of both users. Very frequent updates of the augmented reality object 53 are achieved, in particular by using the object update data, resulting in a real-time, fluid interaction. This is because less data is needed for the object update data, which requires smaller data messages, reducing bandwidth requirements and also reducing the time between updates. This allows for a consistent and fluid interaction between the two users and the augmented reality object.

[0065] FIG. 11 is a flow diagram showing an exemplary sequence of steps for processing an augmented reality content 51 according to an embodiment of the present invention. In step S21, the communication device 1 receives the augmented reality content 51 consisting of user data (i.e., second user data), representation data of the second user including visual representation data and audio representation data, and the augmented reality object 53. In step S21, the electronic circuit 13 of the communication device 1 verifies whether the augmented reality content 51 fits within the bounding volume V of the virtual real estate object. In particular, the electronic circuit 13 verifies that the visual content of the augmented reality content 51 does not extend beyond the bounding volume V. If the visual content, such as a 3D model or image, does not extend beyond the bounding volume V of the virtual real estate object 51, a positive verification is generated in the electronic circuit 13 and the electronic circuit 13 executes step S23. In step S23, the communication device renders the augmented reality content 51 associated with the aspects of the rendering of the visual representation 54 of the second user 2B and the rendering of the augmented reality object 53, respectively, as described, for example, in step S13 of FIG. 9 and step S16 of FIG. 10. If the visual content extends beyond the bounding volume V of the virtual real estate object 51, a negative verification is generated in the electronic circuit 13, which proceeds to either step S222 or step S232, depending on the configuration of the communication device 1. In step S323, the electronic circuit 13 does not render the augmented reality content 51. In step S222, the augmented reality content 51 is modified to fit inside the bounding volume V of the augmented reality content 51, i.e., not to extend beyond it. This modification of the augmented reality content 51 enables the communication device 1 to render the augmented reality content 51 in step S231 without the augmented reality content 51 extending beyond the virtual real estate object 41 and without extending into adjacent virtual real estate objects 41.

[0066] 12 is a block diagram illustrating modification of the augmented reality content 51 according to an embodiment of the present invention. The augmented reality content 51, in particular the visual content of the augmented reality content 51, e.g. the augmented reality object 53, is resized, transformed, truncated, and / or otherwise modified in step S222 to generate a modified augmented reality content 52, in particular a modified augmented reality object 55, that fits within the bounding volume of the virtual real estate object 41.

[0067] It should be noted that although the order of steps has been presented in a particular order herein, one of ordinary skill in the art would understand that the order of at least some of the steps may be changed without departing from the scope of the present invention.

Claims

1. A communication device (1) comprising an electronic circuit (13), The electronic circuit (13) Obtain (S1) location information (21) of a first user (2) of the communication device (1); Sending (S2) a query request to a computerized virtual real estate directory system (4) via a communications network (3); The inquiry request includes at least the location information (21) of the first user (2); The electronic circuit (13) receiving (S3) a query response from the virtual real estate directory system (4) via the communication network (3); the query response includes one or more virtual real estate objects (41) assigned to virtual real estate object locations (42) within a predetermined region (R) defined by the location information of the first user (2); Each of the virtual real estate objects (41) includes an identification (43) and a network address (44) of an augmented reality interaction server (5); The electronic circuit (13) sending (S5) an interaction request via said communication network (3) to one of said augmented reality interaction servers (5) using said network address (44) of said augmented reality interaction server (5) respectively for at least one particular virtual real estate object (41); each interaction request includes the identification (43) of the particular virtual real estate object (41); The electronic circuit (13) receiving (S6) an access permission for the particular virtual real estate object (41) from the augmented reality interaction server via the communication network (3); transmitting (S7) first user data associated with said particular virtual real estate object (41) to a communication device (1B) of a second user (2B) via said augmented reality interaction server (5); receiving (S8) second user data from the communication device (1B) of the second user (2B) via the augmented reality interaction server (5); A communication device configured to:

2. A communication device (1) according to claim 1, The electronic circuit (13) sending a public user identifier of the first user (2) to the augmented reality interaction server (5), the public user identifier including one or more of the public name of the first user (2) or an encryption key of the first user (2); receiving from the augmented reality interaction server (5) a public user identifier of the second user (2B), the public user identifier comprising one or more of a public name of the second user (2B) or a cryptographic key of the second user (2B); A communication device configured to:

3. A communication device (1) according to claim 1 or 2, The communication device (1) includes one or more cameras (12) and a rendering system (11); The electronic circuit (13) determining (S10) visual representation data of the first user (2) using one or more images of the first user (2) captured by the one or more cameras (12); Transmitting (S11) the visual representation data of the first user (2) included in the first user data to the communication device (1B) of the second user via the augmented reality interaction server (5); receiving (S12) visual representation data of the second user (2B) included in the second user data from the communication device (1B) of the second user via the augmented reality interaction server (5); using said rendering system (11) and said visual representation data of said second user (2B) to render (S13) augmented reality content (51) comprising a visual representation (54) of said second user (2B); A communication device configured to:

4. A communication device (1) according to claim 3, The electronic circuit (13) determining a field of view (F) of the first user (2) using the one or more cameras (12); determining from said field of view (F) coordinates (C) of real-world features within said field of view (F); Rendering (S13) the augmented reality content (51) using the rendering system (11) such that the augmented reality content (51) has a fixed position relative to the coordinates (C) of the real-world feature. The communication device further configured as follows.

5. A communication device (1) according to claim 3 or claim 4, The electronic circuit (13) Verification (S221) of whether the augmented reality content (51) is completely contained within a bounding volume (V) of the virtual real estate object (41); Rendering (S23) the augmented reality content (51) in response to positive verification; in response to a negative verification, either not rendering (S232) the augmented reality content (51) or modifying (S222) the augmented reality content (51) to render modified augmented reality content (S231) that fits (52) within the bounding volume (V) of the virtual real estate object (41); The communication device further configured as follows.

6. A communication device (1) according to any one of claims 1 to 5, The communication device (1) further includes a motion tracker (16); The electronic circuit (13) using the motion tracker (16) to record user motion data relating to bodily movements of the first user (2); transmitting the user motion data included in the first user data to the augmented reality interaction server (5); The communication device further configured as follows.

7. A communication device (1) according to any one of claims 1 to 6, The communication device (1) further includes one or more microphones (15) and one or more speakers (14); The electronic circuit (13) recording voice data of the first user (2) using the one or more microphones (15); Transmitting the voice data of the first user (2) included in the first user data to the augmented reality interaction server (5); receiving voice data of the second user (2B) included in the second user data from the communication device (1B) of the second user; reproducing the audio data of the second user (2B) on the one or more speakers (14); A communication device configured to:

8. A communication device (1) according to any one of claims 3 to 5, The electronic circuit (13) receiving (S15) an augmented reality object (52) from the augmented reality interaction server (5); Rendering (S16) an augmented reality content including the augmented reality object (52) using the rendering system (11); determining (S17) user interaction data of the first user (2) interacting with the augmented reality object (52); Sending (S18) the user interaction data to the augmented reality interaction server (5); receiving (S19) an updated augmented reality object (52) from the augmented reality interaction server (5); Rendering (S20) the augmented reality content including the updated augmented reality object (52) using the rendering system (11); The communication device further configured as follows.

9. A method of interaction in augmented reality for a first user (2) of a communication device (1) having an electronic circuit, comprising: The method comprises: Obtaining (S1) location information (21) of the first user (2) of the communication device (1); sending (S2) a query request from said communication device (1) via a communication network (3) to a computerized virtual real estate directory system (4); Including, The inquiry request includes at least the location information (21) of the first user (2); The method comprises: receiving (S3) a query response from the virtual real estate directory system (4) via the communication network (3) in the communication device (1); the query response includes one or more virtual real estate objects (41) assigned to virtual real estate object locations (42) within a predetermined region (R) defined by the location information of the first user (2); Each of the virtual real estate objects (41) includes an identification (43) and a network address (44) of an augmented reality interaction server (5); The method comprises: sending (S5) an interaction request from said communication device (1) via said communication network (3) to one of said augmented reality interaction servers (5) using said network address (44) of said augmented reality interaction server (5) for at least one particular virtual real estate object (41), each interaction request includes the identification (43) of the particular virtual real estate object (41); The method comprises: receiving (S6) in said communication device (1) an access permission for said particular virtual real estate object (41) from said augmented reality interaction server via said communication network (3); transmitting (S7) first user data associated with the particular virtual real estate object (41) from said communication device (1) via said augmented reality interaction server (5) to a communication device (1B) of a second user (2B); receiving (S8) second user data from the communication device (1B) of the second user (2B) via the augmented reality interaction server (5) in the communication device (1); The method of claim 1, further comprising:

10. 10. The method of claim 9, The method comprises: transmitting from the communication device (1) to the augmented reality interaction server (5) a public user identifier of the first user (2), the public user identifier including one or more of the public name of the first user (2) or an encryption key of the first user (2); receiving, in the communication device (1), from the augmented reality interaction server (5) a public user identifier of the second user (2B), the public user identifier including one or more of a public name of the second user (2B) or an encryption key of the second user (2B); The method according to claim 1, further comprising:

11. 11. The method of claim 9 or claim 10, The communication device (1) includes one or more cameras (12) and a rendering system (11); The method comprises: determining (S10) visual representation data of the first user (2) in the communication device (1) using one or more images of the first user (2) captured by the one or more cameras (12); transmitting (S11) the visual representation data of the first user (2) included in the first user data from the communication device (1) of the first user (2) to the communication device (1B) of the second user (2B) via the augmented reality interaction server (5); receiving (S12) visual representation data of the second user (2B) included in the second user data from the communication device (1B) of the second user (2B) via the augmented reality interaction server (5) in the communication device (1) of the first user (2); using said rendering system (11) and said visual representation data of said second user (2B) to render (S13) augmented reality content (51) comprising a visual representation (54) of said second user (2B); The method according to claim 1, further comprising:

12. 12. The method of claim 11, The method comprises: determining a field of view (F) of the first user (2) in the communication device (1) using the one or more cameras (12); determining, in said communication device (1), from said field of view (F), coordinates (C) of real-world features within said field of view (F) relative to a position (21); rendering the augmented reality content (51) using the rendering system (11) such that the augmented reality content (51) has a fixed position relative to the coordinates (C) of the real-world feature; The method according to claim 1, further comprising:

13. 13. The method of claim 11 or 12, comprising: The method comprises: Verification (S221) in the communication device (1) whether the augmented reality content (51) is completely contained within a bounding volume (V) of the virtual real estate object (41); Rendering (S23) the augmented reality content (51) in response to positive verification; in response to a negative verification, either not rendering (S232) the augmented reality content (51) or modifying (S222) the augmented reality content (51) in the communication device (1) to render (S231) modified augmented reality content that fits (52) within the bounding volume (V) of the virtual real estate object (41); The method according to claim 1, further comprising:

14. The method according to any one of claims 11 to 13, The communication device (1) further includes a motion tracker (16); The method comprises: recording user motion data relating to body movements of the first user (2) using the motion tracker (16) in the communication device (1); transmitting (S8) the user motion data included in the first user data from the communication device (1) to the augmented reality interaction server (5); The method according to claim 1, further comprising:

15. The method according to any one of claims 11 to 14, The communication device (1) further includes one or more microphones (15) and one or more speakers (14); The method comprises: recording voice data of the first user (2) using the one or more microphones (15) in the communication device (1); transmitting (S8) the voice data of the first user (2) included in the first user data from the communication device (1) to the augmented reality interaction server (5); receiving (S8) voice data of the second user (2B) included in the second user data from the communication device (1B) of the second user in the communication device (1); playing the voice data of the second user (2B) on the one or more speakers (14); The method of claim 1, further comprising:

16. The method according to any one of claims 11 to 15, The method comprises: receiving (S15) an augmented reality object (52) from the augmented reality interaction server (5) in the communication device (1); Rendering (S16) an augmented reality content comprising the augmented reality object using the rendering system (11); determining (S17) user interaction data of the first user (2) interacting with the augmented reality object (52) in the communication device (1); transmitting (S18) the user interaction data from the communication device (1) to the augmented reality interaction server (5); receiving (S19) an updated augmented reality object (52) from the augmented reality interaction server (5) in the communication device (1); Rendering (S20) the augmented reality content including the updated augmented reality object (52) using the rendering system (11); The method according to claim 1, further comprising:

17. 1. A computer program product including a non-transitory computer-readable medium, comprising: the non-transitory computer readable medium having computer program code stored thereon; the computer program code being adapted to control an electronic circuit (13) of a communication device (1) such that the communication device (1) executes the steps of the method according to one of claims 9 to 16, 23. A computer program product comprising: The end.

Images