Mixed reality teleconference across multiple locations

Abstract

The present disclosure includes superimposing a graphical representation of a first user on a real-world image of a first location, where the first user is physically located at a second location different from the first location. An interaction between the first user and a second user is detected, where the second user is physically located at the first location. A current position of the second user within the first location is determined. The graphical representation of the first user is caused to face in a direction of the current position of the second user.

Description

Background Technology

[0001] This invention relates generally to computer processing systems, and more specifically to a computing processing system for enabling mixed reality teleconferencing across multiple locations.

[0002] Computer systems can generate mixed reality user experiences by overlaying digitally created objects onto a user's computer screen's real-world environment. Mixed reality experiences allow users to interact with both digitally created and real-world objects. The main difference between mixed reality, virtual reality, and augmented reality lies in the level of interaction available between the user and the displayed environment. Virtual reality is a display technology where a computing system creates a fully simulated environment. Users can interact with objects in the simulated environment but not with real-world objects. Augmented reality is a display technology where a computer system creates an enhanced environment by overlaying computer-generated perceptual information onto the real-world environment. Users can interact with real-world objects but not with computer-generated perceptual information. Summary of the Invention

[0003] Embodiments of the present invention relate to mixed reality teleconferencing across multiple locations. One non-limiting example computer-implemented method includes overlaying a graphical representation of a first user onto a real-world image of a first location, wherein the first user is physically located at a second location different from the first location. An interaction between the first user and a second user, wherein the second user is physically located at the first location, is detected. The current location of the second user within the first location is determined. The graphical representation of the first user is oriented towards the direction of the second user's current location.

[0004] Other embodiments of the present invention implement the features of the above-described method in computer systems and computer program products.

[0005] Additional technical features and advantages are achieved through the technology of this invention. Embodiments and aspects of the invention are described in detail herein and are considered part of the claimed subject matter. For a better understanding, refer to the detailed description and accompanying drawings. Attached Figure Description

[0006] The details of the exclusive rights described herein are specifically pointed out and explicitly claimed in the claims at the end of the specification. The foregoing and other features and advantages of embodiments of the invention will become apparent from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0007] Figure 1 A block diagram illustrating components of a system for generating mixed reality teleconferences according to one or more embodiments of the present invention is shown;

[0008] Figure 2A block diagram of components of a system for generating mixed reality teleconferences according to one or more embodiments of the present invention is shown;

[0009] Figure 3 A flowchart illustrating a process for initiating a mixed reality teleconference according to one or more embodiments of the present invention is shown;

[0010] Figure 4 A flowchart illustrating a process for initiating a mixed reality teleconference according to one or more embodiments of the present invention is shown;

[0011] Figure 5 A flowchart illustrating a process for generating a mixed reality teleconference according to one or more embodiments of the present invention is shown;

[0012] Figure 6 A flowchart illustrating a mixed reality teleconference process according to one or more embodiments of the present invention is shown;

[0013] Figure 7 A cloud computing environment according to one or more embodiments of the present invention is described;

[0014] Figure 8 An abstract model layer is described according to one or more embodiments of the present invention; and

[0015] Figure 9 A block diagram of a computer system for implementing one or more embodiments of the present invention is shown.

[0016] The figures depicted herein are illustrative. Many variations may be made to the figures or operations described herein without departing from the scope of the invention. For example, actions may be performed in a different order, or actions may be added, deleted, or modified. Moreover, the term "coupled" and its variations describe a communication path between two elements and do not imply a direct connection between the elements without any intervening elements / connections between them. All such variations are considered part of the specification. Detailed Implementation

[0017] One or more embodiments of the present invention provide a computing system for generating mixed reality teleconferences for participants at multiple locations. Participants can use computing devices to view three-dimensional graphical representations (e.g., avatars) of other participants who are not physically present. The system takes into account the location of each participant at each location, and therefore, if a participant at one location interacts with a participant at another location, the computer system controls their corresponding avatars to face each other, regardless of the participants' actual physical locations.

[0018] Conventional computer-based teleconference systems generate a teleconference space where three-dimensional representations (e.g., avatars) of remote participants who are not physically present are overlaid on an image of the real-world teleconference location. Participants can view this mixed reality image through a mixed reality display on a mobile computing device (e.g., a head-mounted display (HMD), tablet computer display, or smartphone display). Unfortunately, conventional mixed reality systems simply make the avatars directly mimic the actual movement of the participants, without considering the positioning of other participants. Conventional mixed reality systems do not consider whether the avatar's movement conforms to the natural interaction between two teleconference participants.

[0019] For example, a first participant at a first location can interact with a second participant at a second location. Each participant is represented by an avatar at their physically present location. In this case, the first participant can turn their body to the right to rotate their device and look at the second participant's avatar. In a conventional system, if the first participant turns to the right to see the second participant's avatar's display, the second participant, using their mobile computing device, will see the first participant's avatar turn to the right. Unfortunately, if, at the second location, the second participant is actually to the left of the first participant's avatar, the first participant's avatar will be facing the wrong direction, and the interaction will not look natural.

[0020] One or more embodiments of the present invention address one or more of the aforementioned drawbacks by providing a computer-based teleconferencing system that maps the locations of remote participants without limiting the physical layout of the mixed reality space at each location. The computer-based system transforms the interaction between virtual and real participants into an interaction that simulates as if each participant were physically present at each location, regardless of their location relationships. The computer-based teleconferencing system allows for efficient use of space at each location while maintaining natural interaction.

[0021] Now go to Figure 1A system 100 for generating mixed reality teleconferences according to one or more embodiments of the present invention is generally illustrated. Generally, system 100 operates to create mixed reality teleconferences for participants at remote locations. System 100 includes a first location spatial unit 102 and a second location spatial unit 104. Each of the first location spatial unit 102 and the second location spatial unit 104 is operable to receive topological, geometric, or geographic features of the respective location to identify dimensions and spatial relationships between objects at each location. Each of the first location spatial unit 102 and the second location spatial unit 104 is further operable to overlay computer-generated graphical representations (avatars) of participants onto a visual display at each real-world location. Real-world locations include spaces filled with solid, liquid, and gaseous objects. For example, a conference room in an office building where participants are physically present. System 100 also includes an interaction converter 106 for receiving and analyzing data to determine which participant(s) interact with which participant(s). System 100 also includes an avatar model updater 108 for receiving data from the interaction converter 106 and generating input data to make the avatar's movement conform to natural movement during interaction. (This is in contrast to the following description.) Figure 1 A system 100 is depicted that is operatively communicable with first and second participant devices 110, 112 at a first location and third and fourth participant devices 114, 116 at a second location. However, system 100 can accommodate as many remote locations as are participating in a teleconference. The following references... Figure 2 The system 100 and the first participant device 110 are described in more detail. The description of the first participant device 110 applies to any of the second, third, and fourth participant devices 112, 114, and 1116.

[0022] Reference Figure 2System 100 includes a first location spatial unit 102 operable to receive and aggregate data from multiple participant devices 110, 112. The first location spatial unit 102 is further operable to apply computer vision techniques to the aggregated data for object detection. Object detection includes both image classification and object localization. Image classification includes predicting the category of one or more objects in an image. To perform image classification, the first location spatial unit 102 receives an image as input and outputs category labels as one or more integer values ​​mapped to category values. Object localization includes identifying the location of one or more identified objects in an image. To perform object localization, the first location spatial unit 102 can process the received image and output one or more bounding boxes defining the spatial relationships of objects in the image. Through object detection, the first location spatial unit 102 builds a three-dimensional spatial model to identify objects at the teleconference location. For example, the first location spatial unit 102 can use the model to identify participants, furniture, and visual aids.

[0023] In some embodiments of the invention, the first spatial location unit 102 may apply machine learning techniques to perform object detection. In an exemplary embodiment, the first spatial location unit 102 employs a trained artificial neural network, such as a region-based convolutional neural network (R-CNN). R-CNN typically operates in three stages. First, R-CNN analyzes the image and extracts individual regions from the image and depicts the regions as candidate bounding boxes. Second, R-CNN extracts features from each region, for example, using a deep convolutional neural network. Third, a classifier (e.g., a support vector machine (SVM)) is used to analyze the features and predict the category of one or more objects in the region. In other embodiments, the first spatial location unit 102 is another form of neural network besides R-CNN.

[0024] As used herein, “machine learning” broadly describes the function of an electronic system that learns from data. Machine learning systems, engines, or modules may include machine learning algorithms that may be trained, for example, in an external cloud environment (e.g., cloud computing environment 50), to learn a functional relationship between currently unknown inputs and outputs. In one or more embodiments, machine learning functions may be implemented using artificial neural networks (ANNs) capable of being trained to perform currently unknown functions. In machine learning and cognitive science, ANNs are a family of statistical learning models inspired by the biological neural networks of animals (particularly the brain). ANNs can be used to estimate or approximate systems and functions that depend on a large number of inputs.

[0025] An ANN can be embodied as a so-called "neuromorphic" system of interconnected processor elements that act as simulated "neurons" and exchange "messages" with each other in the form of electronic signals. Similar to the so-called "plasticity" of synaptic neurotransmitter connections that transmit messages between biological neurons, the connections in an ANN that transmit electronic messages between simulated neurons are provided with numerical weights corresponding to the strength or weakness of a given connection. These weights can be adjusted and tuned empirically, allowing the ANN to adapt to inputs and learn. For example, an ANN for handwriting recognition is defined by a set of input neurons that can be activated by pixels of an input image. After being weighted and transformed by a function determined by the network's designer, the activation of these input neurons is then passed to other downstream neurons, often referred to as "hidden" neurons. This process is repeated until an output neuron is activated. The activated output neuron determines which character has been read.

[0026] The first location spatial unit 102 is also operable to receive an avatar model for each participant and merge that model with a local spatial model. The first location spatial unit 102 inputs location features into the avatar model. Features are extracted from images from each participant's device, as well as the device's location and viewpoint. The first location spatial unit 102 inputs this data to synthesize the avatar with objects detected in the real-world image to overlay the avatar onto the real-world image. The appearance of each avatar is at least partially based on the location and viewpoint of each participant's device. For example, based on the participant's device's viewpoint, a top view, a bottom view, or a side view of the avatar can be displayed.

[0027] The first location space unit 102 also determines the positioning of each avatar based at least in part on the physical layout of the real-world locations where the avatars will be displayed. For example, the first and second participants at the first location are positioned across each other at the first square table. The third and fourth participants are at the second location, also at the second square table, but seated on adjacent sides of the table. At the first location, the first location unit superimposes the avatars of the third and fourth participants onto the empty sides of the square table relative to each other. At the second location, the second location space unit 104 superimposes the avatars of the first and second participants onto the empty adjacent sides of the table. Thus, regardless of the participants' actual positioning, the avatar positioning is converted to natural positioning during the conference call.

[0028] Interaction converter 106 is operable to receive interaction data from a first participant device 110. The interaction data is used to determine which participant is interacting with which participant. The interaction data may include data describing the avatar of the first participant visible on the display of a second participant device. The interaction data may further include data describing the avatar of the second participant visible on the display of the first participant device. Based on a determination that the time interval for each avatar to remain on the other participant's display exceeds a threshold time, interaction converter 106 can infer that the two participants are interacting. This determination may also be based on the time interval for a single participant's avatar to be displayed on the other participant's device exceeding a threshold time.

[0029] The interaction data may include the location and orientation of the first participant device. The location and orientation data may include data describing the x, y, and z directions of the first participant device. Interaction converter 106 analyzes the location and orientation data, as well as data from other participant devices, to determine which participant the user of the first participant device 110 is interacting with and the user's location. For example, the first participant device 110 may include a magnetic and gravity field sensor that can calculate the magnetic north direction and downward direction at the location of the first participant device. Interaction converter 106 can use a global map of the GPS coordinates of the first participant device 110's location and the angle between true north and magnetic north to calculate the required correction angle for true north. Interaction converter 106 uses this calculation to determine the location of the first participant device 110, and the result is the direction the first participant device 110 is facing. Based on the direction the first participant device is facing, interaction converter 106 applies a three-dimensional spatial model to determine which participant's avatar will be located in the direction of the first participant device 110.

[0030] Avatar model updater 108 is operable to transform the determination of an interactive participant and generate input for an avatar model of the interactive participant. Avatar model updater 108 transmits the input to first and second position space units 102, 104, which update the avatar model for the interactive participant, respectively. In response, the avatar model provides an output to transform the input into motion of the corresponding avatar.

[0031] The first participant device 110 is a mobile computing device and can be physically guided by the first participant. The first participant device 110 may include a head-mounted display (HMD). An HMD is a display device that can be mounted on the participant's head. The first participant device 110 may also include a smartphone, tablet, or other computing device. The first participant device 110 includes a local spatial unit 200. The local spatial unit 200 is operable to receive captured data from multiple sensors 206 and to perform object detection using computer vision techniques. The local spatial unit 200 is similar to the first location spatial unit 102. However, the local spatial unit 200 receives data from multiple sensors 206 instead of multiple participant devices 110, 112. Furthermore, the local spatial unit 200 can aggregate data from the sensors 206 to generate a three-dimensional spatial model of the teleconference location. The first location spatial unit 102 can aggregate data from multiple participant devices 110, 112 to generate a shared three-dimensional spatial model of the teleconference location.

[0032] The first participant device 110 includes a display 202. The display 202 is a miniaturized display device operable to display computer-generated images combined with real-world imagery, such as a teleconference location. The image displayed on the display 202 is based on the field of view of the image capture sensor 206 of the first participant device 110. The field of view is the extent of the real world observable through the display of the first participant device 110. During operation, the display 202 shows physical locations, including any participants physically present at that location and avatars of participants at other locations. The displayed image changes as the user manipulates the positioning of the first participant device 110. Teleconference participants view the mixed reality teleconference through the display 202.

[0033] The first participant device 110 includes a detection unit 204. The detection unit 204 is operable to receive location and motion data to determine whether the location or motion data suggests interaction between the participant and another participant. The detection unit 204 is further operable to determine that a motion does not indicate interaction with another participant. For example, a participant's sneezing or stretching is a temporary movement that generates motion data, but it does not indicate interaction with another participant. The detection unit 204 can be trained to recognize movements associated with these temporary movements. The detection unit 204 is further operable to receive audio data to determine whether the participant is interacting with another participant. The detection unit 204 may employ natural language processing techniques to analyze the audio to determine whether the participant is having a conversation with another participant. For example, the detection unit 204 may compare the name spoken by the participant with the name provided by the participant when entering a conference call. By comparing the names, the detection unit 204 can determine which participant is speaking to which participant. The audio data can be coupled with motion and location data to determine whether the participant is interacting with another participant. If the detection unit 204 determines that an interaction has occurred between the participants, the detection unit 204 transmits the data to the interaction converter 106.

[0034] Sensor 206 includes image, motion, depth, inertial, magnetic, gravitational field, position, and audio capture sensors. Sensor 206 includes, but is not limited to, cameras, gyroscopes, accelerometers, and position-based circuitry for interfacing with a Global Positioning System (GPS). Sensor 206 is operable to collect data from the environment for 3D analysis, including determining the location and size of objects. To enable 3D analysis, the sensor is operable to capture the shadows, illumination, and reflectivity of the object's surface. Sensor 206 continuously collects data during a teleconference to determine the participant's field of view via display 202 and the participant's positioning.

[0035] refer to Figures 3-6 This document describes methods for creating various stages of a mixed reality teleconference according to one or more embodiments of the present invention. For illustrative purposes, these methods are described with reference to participants A and B at a first location and participants C and D at a second, dissimilar location. References Figure 3 A method 300 for initiating a mixed reality teleconference session from the perspectives of both participants and the server at the same location. In box 302, participant A uses their device to connect to the server. Participant A can, for example, use their device to access a teleconference website to connect to the server. In box 304, participant A requests a mixed reality teleconference S. Once participant A accesses the website, they can enter their identification information and password to request the server to initiate a mixed reality teleconference.

[0036] At box 306, participant A uploads an avatar to the server or causes a participant to upload an avatar to the server. An avatar is a 3D graphical representation of participant A. An avatar can be a 3D humanoid representation and resemble participant A. Avatars can be pre-constructed or selected from a suite of available avatar models.

[0037] In box 308, participant A's device registers the first location where participant A is located. Registration includes using a local computing device to generate a 3D spatial model of the first location. The server collects sensor-based data to map not only the structure of the teleconference location (e.g., room dimensions), but also furniture, visual aids, and any participants. Sensor-based data can be collected via an HMD worn by the participant or a remote device such as sensors arranged around the teleconference room. In box 310, participant A's device uploads the 3D spatial model of the first location to the server.

[0038] At box 312, participant B uses a computing device to connect to the server. Participant B may, for example, use their device to access the same teleconference website as participant A to connect to the server. Participant B may enter identification information and request access to the server. The server may authenticate participant B's credentials and grant or deny access to the server.

[0039] At box 314, participant B joins conference call S, and at box 316, participant B uploads an avatar model to the server. At this point, the system can compare the avatar models uploaded by participant A and participant B and determine if they are the same. If, in this instance, the two avatar models are identical, the server can issue an alert to participants A and B with the proposed modifications (such as color changes, virtual name tags, etc.). Participants A and B can then modify their avatars so that participant A is not confused with participant B.

[0040] In box 318, participant B joins the conference call S, and in box 320, participant B downloads a 3D spatial model of the initial location to their computing device. Participant B's computing device can now transmit sensor-based data to the server. For example, a chair is moved outside the image captured by participant A's device; however, participant B's device captures the moving chair. This data can be transmitted to the server to update the model. From the server's perspective, in response to a request from participant A, the server initiates a mixed reality conference call at box 322.

[0041] In box 324, the server receives a 3D spatial model from participant A and initializes a shared 3D spatial model for a first location. In some embodiments of the invention, the participant's device can generate the spatial model. For example, the HMD can use image sensors and depth sensors to collect data and use that data to generate a local spatial model of the teleconference location where participant A is located.

[0042] In box 326, the server imports an avatar model of participant A into the shared 3D space model of the first location. The avatar model is a mathematical description of a 3D representation of participant A. The avatar generated from the model may include cartoon characters, realistic depictions, animals, or other depictions. The avatar's motion is based on sensor data describing participant A's motion being input into the model and outputting the avatar's motion direction. For example, an HMD can use head-tracking sensors to detect the user's position and orientation, hand-tracking sensors to detect hands, eye-tracking sensors to detect eye movement, facial expressions using face-tracking sensors, and additional sensors for legs and body. These detected motions can be mapped onto the avatar model and input into it. The avatar model can then output the corresponding motion regarding the avatar's position. In box 328, in response to participant B joining the conference call, the server imports an avatar model of participant B into the shared 3D space model of the first location.

[0043] Reference Figure 4 The diagram illustrates a method 400 for initiating a mixed reality teleconference session from the perspectives of both participants and the server, both located at the same point in time. In box 402, participant C uses a computing device to connect to the server, and in box 404, participant C joins the mixed reality teleconference. Participant C may, for example, use their device to access the same teleconference website as participants A and B to connect to the server. Participant C can enter identification credentials through the teleconference website, and the server can grant or deny permission to join the mixed reality teleconference.

[0044] In box 406, participant C uploads an avatar model to the server. The avatar generated from the model can be a 3D representation of participant C. The avatar can be pre-constructed or selected from a suite of available avatar models. The avatar can be stored on participant C's computing device, a remote device, or a server.

[0045] In box 408, participant C registers a second location, which involves using a local computing device to generate a 3D spatial model of the second location. When participant C joins the conference call, participant C's avatar is visible on the devices of participants A and B. The avatar is overlaid on the same real-world location on the displays of participants A and B. For example, both participants A and B can see the avatar displayed on their respective devices while they are sitting in the same chair. However, based on the angle of the participant's device and its distance from the chair, the avatar will appear different for each participant.

[0046] In box 410, the 3D spatial model of the second location is uploaded to the server. In box 412, participant D uses a computing device to connect to the server. Participant D can, for example, use their device to access the same teleconference website as participants A, B, and C to connect to the server. Participant D can similarly verify their identity and join a mixed reality teleconference through the teleconference website.

[0047] In box 414, participant D joins the conference call S, and in box 416, participant D uploads an avatar model to the server. The avatar generated from participant D's model can be pre-constructed or selected from a suite of available avatar models. The avatar can be stored on participant D's computing device, remote device, or server.

[0048] In box 418, participant D joins the teleconference S, and in box 420, participant D downloads a 3D spatial model of the first location. Participant D's avatar is visible to participants A and B through their respective devices. When participants A and B look towards the first location in the real world, they see participant D's avatar being applied at that location.

[0049] From the server's perspective, in response to a request from participant A, at box 422, the server creates a shared location model for the second location. In some embodiments of the invention, the participant's device can generate a spatial model. For example, an HMD can generate a local spatial model by using an image sensor and a depth sensor and using the data to generate a local spatial model of the participant's location in the teleconference.

[0050] In box 424, the server sets the avatar model of participant A onto the 3D spatial model of the second location. Participant A's avatar model can now receive input relating to the physical layout of the second location. In this sense, participant A's avatar responds to the physical layout of the second location. For example, through the devices of participants C and D, participant A's avatar will be treated differently from the physical object at the second location.

[0051] In box 426, the server identifies the avatar of participant C as the interaction target. By setting participant C as the interaction target, the avatar model is not updated based on the interaction between the participant and inanimate objects. Instead, each avatar model is updated based on the interaction with another participant.

[0052] In box 428, the server sets the avatar model of participant B into the shared 3D spatial model of the second location. Participant B's avatar model can now receive input related to the physical layout of the second location. Participant B's avatar responds to the physical layout of the second location. For example, through the devices of participants C and D, participant B's avatar will be treated as a different physical object than the one at the second location.

[0053] In box 430, the server identifies the avatar of participant D as the interaction target. This is done by encoding the avatar output from participant D's avatar model to reflect that the avatar is the interaction target. Setting participant D as the interaction target provides focus to guide the orientation of the face of another participant's avatar.

[0054] refer to Figure 5 The diagram illustrates a method 500 for generating a shared mixed reality teleconference session. At box 502, the server adds the corresponding avatar models of participant C and participant D to a shared first location model. The avatar models of participants C and D can now receive input relating to the physical layout of the first location. In this sense, the avatar of participant A responds to the physical layout of the second location.

[0055] At box 504, the server adjusts the avatars' arrangement to match the arrangement of the first position. For example, in the instance, participants C and D are sitting side-by-side in the second position, but there are only opposite seats in the first position, so the server determines that the avatars of participants C and D will be displayed opposite each other in the first position.

[0056] At box 506, the server sets up avatars for participants C and D and fixes the positions of avatars C and D in the 3D space model used for the first position. Using the example above, the server sets the avatars of participants C and D to be displayed relative to each other at the first position.

[0057] In box 508, the server adds the corresponding avatar models of participant A and participant B to the 3D spatial model of the second location. The avatar models of participant A and B can now receive input related to the physical layout of the second location. In box 510, the server adjusts the arrangement of the avatars of participant A and B to conform to the arrangement of the second location. Similar to the above, the server determines the positioning of the avatars of participant A and B to be displayed at the second location. In box 512, the server fixes the positions of the avatars of participant A and B in the 3D spatial model of the second location.

[0058] refer to Figure 6 The diagram illustrates a method 600 for generating interactions during a mixed reality teleconference. At box 602, a motion detector detects the movement of participant A. For example, participant A might turn their body 30 degrees to the left, and the system can detect this movement. At box 604, a local spatial model stored on participant A's computing device is updated to reflect this movement.

[0059] At box 606, the server updates the shared location model to reflect participant A's movement. For example, a sensor placed on participant A, a sensor placed on participant B, or a sensor placed at the teleconference location detects that participant A has rotated their head 15 degrees in the direction of participant C. The server can then input this data into participant A's avatar model so that the avatar rotates towards participant C by the appropriate number of degrees.

[0060] At box 608, participant A's computing device determines, based on participant A's movement, that participant A is looking at the interaction target—partner C. Continuing from the example above, as participant A rotates, the avatar of participant C becomes visible in participant A's device, and consequently, the avatar of participant A is displayed turning towards participant C in the displays of participants C and D's devices.

[0061] At box 610, participant A's computing device transmits data to the server indicating that participant A is looking at participant C. Participant C has been identified as the interaction target. Therefore, even if participant C is placed near an inanimate object (e.g., a cabinet), the server will make participant A's avatar appear to be looking at participant C.

[0062] At box 614, the server updates the shared 3D spatial model to reflect that participant A is looking at participant C. Therefore, when participant C looks at participant A through participant C's device, it will appear that participant A's avatar is looking at participant C. At box 616, the location model stored on participant B's computing device is synchronized with the updated 3D spatial model stored on the server. Synchronization inputs data into the location model to reflect that participant A is looking at participant C.

[0063] In boxes 618 and 620, the local spatial models stored on the devices of participants C and D, respectively, are synchronized to display the avatar of participant A facing participant C. In other words, on each of the devices of participants C and D, the avatar of participant A will appear to be facing participant C.

[0064] In some embodiments, the avatar may further appear to respond to environmental stimuli at the teleconference location. Each participant's device may be equipped with audio, temperature, and other appropriate sensors. The sensors are operable to sense environmental stimuli at the teleconference location, and the avatar visible at that location may respond to the stimuli. For example, participant A may be at a first location, and participant A's avatar may be visible on participant B's device at a second location. Participant B's device may be configured to sense environmental stimuli, such as a sudden drop in temperature, a sudden noise, or a sudden increase in sunlight. Environmental stimulus data from one location may be transmitted to a server, which may update the avatar model when a participant is at another location and unaware of the environmental stimuli. For example, the avatar of a participant at the first location may squint in response to increased sunlight at the second location. The avatar may turn toward loud sounds, even if the actual participant cannot hear them.

[0065] It should be understood that while this disclosure includes a detailed description of cloud computing, the implementation of the teachings cited herein is not limited to cloud computing environments. Rather, embodiments of the invention can be implemented in conjunction with any other type of computing environment now known or developed hereafter.

[0066] Cloud computing is a service delivery model that enables convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services), which can be rapidly provisioned and released with minimal management effort or interaction with the service provider. This cloud model may include at least five features, at least three service models, and at least four deployment models.

[0067] The features are as follows:

[0068] On-demand self-service: Cloud consumers can unilaterally and automatically provide computing power, such as server time and network storage, as needed, without requiring human interaction with the service provider.

[0069] Extensive network access: Capabilities are available through networks and accessed via standard mechanisms that facilitate the use of heterogeneous thin client or thick client platforms (e.g., mobile phones, laptops, and PDAs).

[0070] Resource pooling: A provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, where different physical and virtual resources are dynamically assigned and reassigned as needed. There is a sense of location independence because consumers typically do not have control or knowledge of the exact location of the resources provided, but may be able to specify the location at a higher level of abstraction (e.g., country, state, or data center).

[0071] Rapid flexibility: The ability to provide capacity quickly and flexibly, automatically scaling down and up rapidly in some situations to scale up rapidly. For consumers, the available supply capacity often appears unlimited and can be purchased in any quantity at any time.

[0072] Measuring services: Cloud systems automatically control and optimize resource usage by leveraging metering capabilities at a level of abstraction appropriate to the service type (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency to both service providers and consumers.

[0073] The service model is as follows:

[0074] Software as a Service (SaaS): This provides consumers with the ability to use the provider's applications running on cloud infrastructure. Applications can be accessed from different client devices via thin client interfaces such as web browsers (e.g., web-based email). Consumers do not manage or control the underlying cloud infrastructure, including the network, servers, operating system, storage, or even individual application capabilities, with possible exceptions such as limited user-specific application configuration settings.

[0075] Platform as a Service (PaaS): This provides consumers with the ability to deploy applications created or acquired by the consumer using programming languages ​​and tools supported by the provider onto cloud infrastructure. Consumers do not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but they have control over the deployed applications and the configuration of any application hosting environment.

[0076] Infrastructure as a Service (IaaS): This provides consumers with the capability to offer processing, storage, networking, and other basic computing resources that enable consumers to deploy and run arbitrary software, which may include operating systems and applications. Consumers do not manage or control the underlying cloud infrastructure, but rather have control over the operating system, storage, deployed applications, and potentially limited control over selected networking components (e.g., host firewalls).

[0077] The deployment model is as follows:

[0078] Private cloud: A cloud infrastructure that operates solely for an organization. It can be managed by the organization or a third party and can exist on-site or off-site.

[0079] Community cloud: A cloud infrastructure shared by several organizations and supporting a specific community with shared concerns (e.g., tasks, security requirements, policies, and compliance considerations). It can be managed by an organization or a third party and can exist on-site or off-site.

[0080] Public cloud: Makes cloud infrastructure available to the public or large industry groups and is owned by an organization that sells cloud services.

[0081] Hybrid cloud: A cloud infrastructure is a combination of two or more clouds (private, community, or public) that remain a single entity but are bound together by standardized or proprietary technologies that enable data and applications to be ported (e.g., cloud bursting for load balancing between clouds).

[0082] Cloud computing environments are service-oriented, focusing on statelessness, loose coupling, modularity, and semantic interoperability. At the heart of cloud computing is the infrastructure comprising a network of interconnected nodes.

[0083] Now for reference Figure 7 This describes an illustrative cloud computing environment 50. As shown, the cloud computing environment 50 includes one or more cloud computing nodes 10 to which local computing devices used by cloud consumers can communicate. These local computing devices include, for example, personal digital assistants (PDAs) or cellular phones 54A, desktop computers 54B, laptop computers 54C, and / or automotive computer systems 54N. The nodes 10 can communicate with each other. They can be physically or virtually grouped (not shown) in one or more networks, such as private clouds, community clouds, public clouds, or hybrid clouds, or combinations thereof, as described above. This allows the cloud computing environment 50 to provide infrastructure, platforms, and / or software as services that cloud consumers do not need to maintain on their local computing devices. It should be understood that... Figure 7 The types of computing devices 54A-54N shown are intended to be illustrative only, and computing node 10 and cloud computing environment 50 can communicate with any type of computerized device via any type of network and / or network-addressable connection (e.g., using a web browser).

[0084] See now Figure 8 This demonstrates a cloud computing environment of 50 ( Figure 7 This provides a set of functional abstractions. It should be understood beforehand. Figure 7 The components, layers, and functions shown are intended to be illustrative only, and embodiments of the invention are not limited thereto. As described, the following layers and corresponding functions are provided:

[0085] The hardware and software layer 60 includes hardware and software components. Examples of hardware components include: a mainframe 61; a RISC (Reduced Instruction Set Computer) based server 62; a server 63; a blade server 64; a storage device 65; and networking and interconnection components 66. In some embodiments, software components include network application server software 67 and database software 68.

[0086] The virtualization layer 70 provides an abstraction layer from which the following examples of virtual entities can be provided: virtual server 71; virtual storage 72; virtual network 73, including virtual private network; virtual application and operating system 74; and virtual client 75.

[0087] In one example, management layer 80 may provide the following functionalities: Resource Provisioning 81 provides dynamic procurement of computing resources and other resources used to perform tasks within the cloud computing environment. Metering and Pricing 82 provides cost tracking as resources are utilized within the cloud computing environment and bills or invoices for the consumption of these resources. In one example, these resources may include application software licenses. Security provides authentication for cloud consumers and tasks, as well as protection for data and other resources. User Portal 83 provides access to the cloud computing environment for consumers and system administrators. Service Level Management 84 provides cloud resource allocation and management to ensure that required service levels are met. Service Level Agreement (SLA) Planning and Fulfillment 85 provides pre-scheduling and procurement of cloud resources based on anticipated future needs according to the SLA.

[0088] Workload layer 90 provides examples of functionalities that can be leveraged in a cloud computing environment. Examples of workloads and functionalities that can be provided from this layer include: mapping and navigation 91; software development and lifecycle management 92; virtual classroom education delivery 93; data analytics and processing 94; transaction processing 95; and creating mixed reality teleconference environments 96.

[0089] It should be understood that this disclosure can be implemented in conjunction with any other type of computing environment now known or developed in the future. For example, Figure 9 A block diagram of a processing system 900 for implementing the techniques described herein is depicted. In an example, the processing system 900 has one or more central processing units (processors) 921a, 921b, 921c, etc. (collectively or generally referred to as (one or more) processors 921 and / or (one or more) processing devices). In aspects of this disclosure, each processor 921 may include a Reduced Instruction Set Computer (RISC) microprocessor. The processor 921 is coupled to system memory (e.g., random access memory (RAM) 924) and various other components via a system bus 933. Read-only memory (ROM) 922 is coupled to the system bus 933 and may include a Basic Input / Output System (BIOS) that controls certain basic functions of the processing system 900.

[0090] Further depicted are input / output (I / O) adapter 927 and network adapter 926 coupled to system bus 933. I / O adapter 927 may be a Small Computer System Interface (SCSI) adapter that communicates with hard disk 923 and / or storage device 925 or any other similar component. I / O adapter 927, hard disk 923, and storage device 925 are collectively referred to herein as mass storage device 934. Operating system 940 for execution on processing system 900 may be stored in mass storage device 934. Network adapter 926 interconnects system bus 933 with external network 936, thereby enabling processing system 900 to communicate with other such systems.

[0091] A display (e.g., a display monitor) 935 is connected to the system bus 933 via a display adapter 932, which may include a graphics adapter and a video controller for improving performance in graphics-intensive applications. In one aspect of this disclosure, adapters 926, 927, and / or 932 may be connected to one or more I / O buses connected to the system bus 933 via an intermediate bus bridge (not shown). Suitable I / O buses for connecting peripheral devices such as hard disk controllers, network adapters, and graphics adapters typically include common protocols such as Peripheral Component Interconnect (PCI). Additional input / output devices are shown connected to the system bus 933 via a user interface adapter 928 and the display adapter 932. Input devices 929 (e.g., keyboard, microphone, touchscreen, etc.), input pointers 930 (e.g., mouse, touchpad, touchscreen, etc.), and / or speakers 931 may be interconnected to the system bus 933 via the user interface adapter 928, which may include, for example, a super I / O chip integrating multiple device adapters into a single integrated circuit.

[0092] In some aspects of this disclosure, the processing system 900 includes a graphics processing unit 937. The graphics processing unit 937 is a dedicated electronic circuit designed to manipulate and modify memory to accelerate the creation of images in a frame buffer intended for output to a display. Generally, the graphics processing unit 937 is highly efficient in manipulating computer graphics and image processing, and has a highly parallel architecture, making it more efficient than a general-purpose CPU for algorithms that process large blocks of data in parallel.

[0093] Thus, as configured herein, the processing system 900 includes processing power in the form of a processor 921, storage capacity including system memory (e.g., RAM 924) and mass storage device 934, input devices such as a keyboard 929 and a mouse 930, and output capacity including a speaker 931 and a display 935. In some aspects of this disclosure, a portion of the system memory (e.g., RAM 924) and mass storage device 934 jointly stores an operating system 940 to coordinate the functions of the different components shown in the processing system 900.

[0094] Various embodiments of the invention are described herein with reference to the accompanying drawings. Alternative embodiments of the invention may be devised without departing from its scope. In the following description and drawings, various connections and positional relationships (e.g., above, below, adjacent, etc.) are illustrated between elements. Unless otherwise specified, these connections and / or positional relationships may be direct or indirect, and the invention is limited in this respect by not illustrating the figures. Therefore, the connection of entities may refer to direct or indirect connections, and the positional relationship between entities may be direct or indirect positional relationships. Furthermore, the various tasks and process steps described herein may be incorporated into a more comprehensive procedure or process with additional steps or functions not described in detail herein.

[0095] One or more methods described herein can be implemented using any of the following techniques or combinations thereof, each of which is well known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having appropriately combined logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0096] For the sake of brevity, conventional techniques relating to the manufacture and use of the present invention may or may not be described in detail herein. Specifically, various aspects of the computing systems and specific computer programs used to implement the different technical features described herein are well known. Consequently, for the sake of brevity, many conventional implementation details are only briefly mentioned or omitted entirely herein, without providing well-known system and / or process details.

[0097] In some embodiments, various functions or actions may occur at a given location and / or in conjunction with the operation of one or more devices or systems. In some embodiments, a portion of a given function or action may be performed at a first device or location, and the remainder of the function or action may be performed at one or more additional devices or locations.

[0098] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used herein, unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well. It should also be understood that when the terms “comprises” and / or “comprising” are used in this specification, they specify the presence of the stated features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or combinations thereof.

[0099] All means or steps in the following claims, plus corresponding structures, materials, actions, and equivalents of the functional elements, are intended to include any structure, material, or action for performing the function in conjunction with other claimed elements as specifically claimed. This disclosure has been presented for purposes of illustration and description, but is not intended to be exhaustive or limited to the forms disclosed. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of this disclosure. These embodiments were chosen and described in order to best explain the principles and practical application of this disclosure, and to enable others skilled in the art to understand this disclosure with respect to different embodiments having different modifications suitable for the particular intended use.

[0100] The diagrams depicted herein are illustrative. Many variations may be made to the diagrams or steps (or operations) described herein without departing from the scope of this disclosure. For example, actions may be performed in a different order, or actions may be added, deleted, or modified. Furthermore, the term "coupled" describes a signal path between two elements and does not imply a direct connection between elements without intermediate elements / connections. All such variations are considered part of this disclosure.

[0101] The following definitions and abbreviations will be used to interpret the claims and description. As used herein, the terms “comprises,” “comprising,” “includes,” “including,” “has,” “having,” “contains,” or “containing,” or any other variations thereof, are intended to cover non-exclusive inclusion. For example, a composition, mixture, process, method, article, or apparatus that comprises a list of elements is not necessarily limited to those elements, but may include other elements not expressly listed or inherent to such composition, mixture, process, method, article, or apparatus.

[0102] Furthermore, the term "exemplary" is used herein to mean "used as an example, illustration, or illustration." Any implementation or design described herein as "exemplary" is not necessarily to be construed as superior to or better than other implementations or designs. The terms "at least one" and "one or more" should be understood to include any integer greater than or equal to one, i.e., one, two, three, four, etc. The term "multiple" should be understood to include any integer greater than or equal to two, i.e., two, three, four, five, etc. The term "connection" can include both indirect "connection" and direct "connection."

[0103] The terms “about,” “substantially,” “roughly,” and their variations are intended to include the degree of error associated with a measurement based on a specific quantity of equipment available at the time of application submission. For example, “about” could include a range of ±8%, 5%, or 2% of a given value.

[0104] This invention can be a system, method, and / or computer program product with any possible level of technical detail integration. The computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions thereon for causing a processor to execute aspects of the invention.

[0105] Computer-readable storage media can be tangible means for retaining and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital universal disk (DVD), memory sticks, floppy disks, mechanical encoding devices such as punch cards or protrusions in slots having instructions recorded thereon, and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses passing through fiber optic cables), or electrical signals transmitted through wires.

[0106] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a suitable computing / processing device via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network), or to an external computer or external storage device. The network may include copper cables, optical fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to a computer-readable storage medium within the suitable computing / processing device.

[0107] Computer-readable program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, integrated circuit configuration data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​(such as Smalltalk, C++, etc.) and procedural programming languages ​​(such as the "C" programming language or similar programming languages). The computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as a standalone software package, partially on a user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)) or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs) may execute computer-readable program instructions by utilizing state information from the computer-readable program instructions to personalize the electronic circuitry in order to perform aspects of this invention.

[0108] The present invention will now be described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0109] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / actions specified in one or more blocks of a flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner, such that the computer-readable storage medium storing the instructions includes an article of manufacture containing instructions that implement aspects of the functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0110] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other device to produce computer-implemented processing, such that the instructions executed on the computer, other programmable apparatus, or other device perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0111] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. Each block in a flowchart or block diagram may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than indicated in the figures. For example, depending on the functions involved, two consecutively shown blocks may actually be executed substantially simultaneously, or these blocks may sometimes be executed in reverse order. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action or executes a combination of dedicated hardware and computer instructions.

[0112] Various embodiments of the invention have been described for illustrative purposes, but are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the described embodiments. The terminology used herein has been chosen to best explain the principles of the embodiments, their practical application, or technical improvements over those found in the market, or to enable those skilled in the art to understand the embodiments described herein.

Claims

1. A computer-implemented method, comprising: The processor detects the location of the first user at the first table in the second position; The processor detects the location of the second user at the second table in the first position; The processor overlays a graphical representation of the first user onto a real-world image of a second table at a first location, wherein the graphical representation of the first user is overlaid onto the second table based on the location of the second user at the second table. The processor overlays a graphical representation of the second user onto a real-world image of the first table at the second location, wherein the graphical representation of the second user is overlaid onto the first table based on the positioning of the first user at the first table. The processor designates at least one of the graphical representations of the first user and the second user as the interaction target; The processor detects the interaction between the first user and the second user, wherein the second user is physically located at the first location, which is different from the second location; The processor determines the current location of the second user at the first location; as well as The processor, based on the specified interaction target, causes the first user's graphical representation to face the direction of the second user's current location.

2. The computer-implemented method according to claim 1, further comprising: Determine the current location of the first user within the second location; as well as The graphic representation of the second user is oriented towards the current location of the first user.

3. The computer-implemented method according to claim 1, wherein: The graphical representation of the first user is positioned on the real-world image of the first location, at least in part based on the physical layout of the first location.

4. The computer-implemented method according to claim 3, wherein: The graphical representation of the second user is positioned on the real-world image of the second location based at least in part on the physical layout of the second location, wherein the physical layout of the first location is different from the physical layout of the second location.

5. The computer-implemented method according to claim 1, wherein detecting the interaction between the first user and the second user comprises: Calculate the positioning and field of view of the first user's computing device; Detect the location of the second user's computing device; as well as The positioning and field of view of the first user's computing device are compared with the positioning of the second user's computing device.

6. The computer-implemented method according to claim 1, wherein detecting the interaction between the first user and the second user comprises: The graphical representation of the first user is detected on the display of the second user's computing device; Determine whether the time interval of the graphic representation of the first user exceeds a threshold time.

7. The computer-implemented method according to claim 1, further comprising: The model of the second location is shared with the computing device of a third user, wherein the third user is physically located at the second location.

8. A computer system, comprising: Memory, containing computer-readable instructions; as well as One or more processors for executing the computer-readable instructions, the computer-readable instructions controlling the one or more processors to perform operations, the operations including: Detect the location of the first user at the first table in the second position; Detect the location of the second user at the second table in the first position; The graphic representation of the first user is superimposed onto a real-world image of the second table at the first location, wherein the graphic representation of the first user is superimposed onto the second table based on the location of the second user at the second table. The graphic representation of the second user is superimposed onto a real-world image of the first table at the second location, wherein the graphic representation of the second user is superimposed onto the first table based on the positioning of the first user at the first table. Designate at least one of the graphical representations of the first user and the second user as the interaction target; Detect the interaction between the first user and the second user, wherein the second user is physically located at the first location, which is different from the second location; Determine the current location of the second user at the first location; and Based on the specified interaction target, the first user's graphical representation is directed toward the second user's current location.

9. The computer system according to claim 8, wherein the operation further comprises: Determine the current location of the first user within the second location; as well as The graphic representation of the second user is oriented towards the current location of the first user.

10. The computer system according to claim 8, wherein: The graphical representation of the first user is positioned on the real-world image of the first location, at least in part based on the physical layout of the first location.

11. The computer system according to claim 10, wherein: The graphical representation of the second user is positioned on the real-world image of the second location based at least in part on the physical layout of the second location, wherein the physical layout of the first location is different from the physical layout of the second location.

12. The computer system of claim 8, wherein detecting the interaction between the first user and the second user comprises: Calculate the positioning and field of view of the first user's computing device; Detect the location of the second user's computing device; as well as The positioning and field of view of the first user's computing device are compared with the positioning of the second user's computing device.

13. The computer system of claim 8, wherein detecting the interaction between the first user and the second user comprises: The graphical representation of the first user is detected on the display of the second user's computing device; Determine whether the time interval of the graphic representation of the first user exceeds a threshold time.

14. The computer system of claim 8, wherein the operation further comprises: The model of the second location is shared with the computing device of a third user, wherein the third user is physically located at the second location.

15. A computer program product having program instructions executable by a processor to cause the processor to perform operations including: Detect the location of the first user at the first table in the second position; Detect the location of the second user located next to the second table in the first position; The graphic representation of the first user is superimposed onto a real-world image of the second table at the first location, wherein the graphic representation of the first user is superimposed onto the second table based on the location of the second user at the second table. The graphic representation of the second user is superimposed onto a real-world image of the first table at the second location, wherein the graphic representation of the second user is superimposed onto the first table based on the positioning of the first user at the first table. Designate at least one of the graphical representations of the first user and the second user as the interaction target; Detect the interaction between the first user and the second user, wherein the second user is physically located at the first location, which is different from the second location; Determine the current location of the second user at the first location; as well as Based on the specified interaction target, the first user's graphical representation is directed toward the second user's current location.

16. The computer program product of claim 15, wherein the operation further comprises: Determine the current location of the first user at the second location; as well as The graphic representation of the second user is oriented towards the current location of the first user.

17. The computer program product according to claim 15, wherein: The graphical representation of the first user is positioned on the real-world image of the first location, at least in part based on the physical layout of the first location.

18. The computer program product according to claim 17, wherein: The graphical representation of the second user is positioned on the real-world image of the second location based at least in part on the physical layout of the second location, wherein the physical layout of the first location is different from the physical layout of the second location.

19. The computer program product of claim 15, wherein detecting the interaction between the first user and the second user comprises: Calculate the positioning and field of view of the first user's computing device; Detect the location of the second user's computing device; as well as The positioning and field of view of the first user's computing device are compared with the positioning of the second user's computing device.

20. The computer program product of claim 15, wherein detecting the interaction between the first user and the second user comprises: The graphical representation of the first user is detected on the display of the second user's computing device; Determine whether the time interval of the graphic representation of the first user exceeds a threshold time.

Images