Placing virtual content in environments with multiple physical participants

Abstract

A device may be configured to determine display attributes for virtual content in an environment having multiple physical participants by capturing an image of the environment, analyzing the captured image to identify at least one object in the environment, determining parameters for the identified object, and determining display attributes for a digital representation of the virtual content based on the determined parameters. Embodiments may include negotiating the display attributes with other devices to generate coordinated display attributes, and rendering the digital representation of the virtual content so that remote participants appear to be in the same fixed position to all co-located participants and are sized consistently with the co-located participants.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This patent application claims priority to U.S. Non-Provisional Application No. 16 / 572,030, filed on September 16, 2019, entitled “Placement of Virtual Contents in Environments with a Plurality of Physical Participants,” which is assigned to the assignee and is expressly incorporated herein by reference. Background Art

[0003] In recent years, software applications that render images and virtual content within an environment have become increasingly popular and used in a wide range of applications, such as electronic games, virtual training systems, collaborative work environments, advanced web-based conferencing and screen sharing applications. In addition, advances in mobile device technology now allow mobile device users to execute powerful software applications on their mobile devices, such as augmented reality software applications that combine real-world images from the user's physical environment with computer-generated images or virtual content. For example, augmented reality applications can add graphics, sounds, and / or tactile feedback to the natural world around the application user. Images, video streams, and information about people and / or objects can be rendered to the user on a wearable electronic display (e.g., smart glasses, augmented reality glasses, etc.), so that the user can view and / or interact with representations of people or objects in the real world.

[0004] As a result of these trends, it is expected that augmented reality applications will allow users to collaborate and interact with others in remote locations as if all participants were located in the same environment (e.g., in the same room). For example, augmented reality applications may render remote participants as three-dimensional representations or two-dimensional videos or avatars so that local and remote participants can interact and communicate in the same manner as if all participants were in the same environment. Summary of the invention

[0005] Various aspects of the present disclosure provide methods for determining display attributes of virtual content in an environment with multiple participants, which may include capturing, by a processor, an image of the environment, analyzing the captured image to identify an object in the environment, determining parameters for the identified object, and determining display attributes of a digital representation of the virtual content based on the determined parameters. In some aspects, the processor may be a processor of a smartphone of one of the multiple participants.

[0006] In some aspects, this may include assigning priorities to the determined parameters.In some aspects, determining the display attributes of the digital representation of the virtual content based on the determined parameters may include determining the display attributes of the digital representation of the virtual content based on the priorities assigned to the determined parameters.

[0007] In some aspects, determining parameters for the identified object may include determining at least one or more of a fixed position parameter, a size parameter, or a line of sight parameter, and determining at least one or more of a field of view parameter, a spacing parameter, a human distance parameter, a height parameter, and a visibility parameter. In some aspects, assigning priorities to the determined parameters may include assigning a first priority to the fixed position parameter, the size parameter, or the line of sight parameter, and assigning a second priority to the field of view parameter, the spacing parameter, the human distance parameter, the height parameter, or the visibility parameter, wherein the first priority is a higher priority than the second priority.

[0008] In some aspects, the method may include identifying one or more other devices in the environment and sending the determined parameters to the identified one or more other devices. In some aspects, the method may include receiving additional parameters from at least one other device in the environment, wherein determining the display attributes of the digital representation of the virtual content based on the determined parameters may include determining one or more display attributes of the digital representation of the virtual content based on the determined parameters and the received additional parameters.

[0009] In some aspects, determining display attributes of the digital representation of the remote participant based on the determined parameters may include negotiating one or more display attributes for rendering the digital representation of the remote participant with at least one other device in the environment to generate one or more coordinated display attributes. In some aspects, this may include rendering the digital representation of the remote participant using the one or more coordinated display attributes so that all co-located participants physically present in the environment perceive the remote participant to be in the same fixed position in the environment. In some aspects, this may include using the one or more coordinated display attributes to adjust the size of the digital representation of the remote participant based on the size of the participant physically present in the environment. In some aspects, this may include rendering the digital representation of the remote participant using the one or more coordinated display attributes so that the virtual content appears to be located between a first participant physically present in the environment and a second participant physically present in the environment.

[0010] In some aspects, rendering the digital representation of the remote participant using one or more coordinated display attributes so that the remote participant appears to be located between a first participant physically present in the environment and a second participant physically present in the environment may include rendering the digital representation of the remote participant using one or more coordinated display attributes so that the remote participant appears to be approximately equidistant from the first participant physically present in the environment and the second participant physically present in the environment.

[0011] In some aspects, this may include rendering the digital representation of the remote participant using one or more coordinated display attributes so that the remote participant appears to be anchored to the empty seat. In some aspects, this may include capturing an updated image of the environment, analyzing the captured updated image to determine whether the number, position, posture, or size of the identified objects has changed, determining updated parameters for the identified objects in response to the determined position, posture, or size of the identified objects having changed, determining whether a difference between the determined parameters and the determined updated parameters exceeds a threshold, and determining updated display attributes for the digital representation of the remote participant in response to determining that the difference between the determined parameters and the determined updated parameters exceeds the threshold.

[0012] In some aspects, determining display attributes of the digital representation of the remote participant based on the determined parameters may include determining at least one display attribute for each digital representation of the plurality of remote participants. In some aspects, capturing an image of the environment may include capturing an image of the environment by a processor associated with a device (e.g., a computing device, a mobile device, a head mounted device, etc.). In some aspects, determining parameters for the identified object may include determining at least one or more of a fixed position parameter, a size parameter, a line of sight parameter, a field of view parameter, a spacing parameter, a person distance parameter, a height parameter, or a visibility parameter. In some aspects, the virtual content may be a remote participant. In some aspects, the virtual content is related to a game. In some aspects, the environment may be an augmented reality game environment.

[0013] Additional aspects may include a computing device equipped with a processor configured with processor-executable instructions to perform the operations of any of the methods outlined above, such as capturing an image of an environment, analyzing the captured image to identify objects in the environment, determining parameters for the identified objects, and determining display properties of a digital representation of virtual content based on the determined parameters. Additional aspects may include a device including an image sensor, a projector configured to project an image onto an optical lens, and a processor coupled to the image sensor and the projector and configured with processor-executable instructions to perform the operations of any of the methods outlined above, such as capturing an image of an environment, analyzing the captured image to identify objects in the environment, determining parameters for the identified objects, and determining display properties of a digital representation of virtual content based on the determined parameters.

[0014] Another aspect may include a non-transitory processor-readable storage medium having processor-executable instructions stored thereon, the processor-executable instructions being configured to cause a processor in a device and / or an associated computing device to perform the operations of any of the above methods. Another aspect may include a device and / or an associated computing device having various components for implementing the functions of the above methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The accompanying drawings, which are incorporated herein and constitute a part of this specification, illustrate example embodiments of the various embodiments and, together with the general description given above and the detailed description given below, serve to explain features of the claims.

[0016] Figure 1A is an illustration of a head-mounted device (e.g., augmented reality glasses) that can be configured to implement various embodiments.

[0017] Figure 1B is a block diagram of components of a smart phone suitable for implementing some embodiments.

[0018] Figure 1C is a system block diagram illustrating computer architecture and sensors that may be included in a device configured to render digital representations of remote participants in a virtual meeting, according to an embodiment.

[0019] Figure 2A is an illustration of a meeting participant wearing a device that may be configured to render a digital representation of a remote participant in a virtual meeting, according to an embodiment.

[0020] Figure 2B is a block diagram illustrating that a digital representation of a remote participant may be rendered such that the remote participant is within the field of view of a co-located participant and does not block the view of either remote or co-located participant.

[0021] Figure 3A and 3B is a block diagram illustrating a device configured to render digital representations of remote participants in a virtual meeting including multiple co-located participants physically present in the same environment and seated around a gaming table in positions suitable for viewing a presentation.

[0022] Figure 4A is a block diagram illustrating that a device may render digital representations of remote conference participants at a natural position relative to a gaming table and enable each of the co-located participants to have a direct line of sight to the digital representations of the remote conference participants.

[0023] Figure 4B is a block diagram illustrating a device configured in accordance with an embodiment that can exchange data with other devices present in an environment to reposition digital representations of remote conference participants to different locations or postures relative to a gaming table such that all co-located participants have direct line of sight with each other and with the digital representations of the remote conference participants.

[0024] Figures 5A-5C is a block diagram illustrating a device configured in accordance with an embodiment that can intelligently determine where digital representations of remote conference participants should be rendered relative to the positions or postures of co-located participants physically present in an environment.

[0025] Figure 6-9 is a process flow diagram illustrating a method of determining display attributes for a digital representation of a remote participant in accordance with some embodiments.

[0026] Fig.10 is a process flow diagram illustrating a method of updating display properties of a digital representation of a remote participant according to an embodiment.

[0027] Fig.11 is a component diagram of an example computing device suitable for use with various embodiments. DETAILED DESCRIPTION

[0028] Various embodiments will be described in detail with reference to the accompanying drawings. Wherever possible, the same reference numerals will be used throughout the drawings to refer to the same or similar parts. References to specific examples and implementations are for illustrative purposes and are not intended to limit the scope of the claims.

[0029] In summary, various embodiments include methods for determining display properties of virtual content in an environment with multiple physical participants and devices (e.g., computing devices, mobile devices, head-mounted devices, etc.) configured to implement these methods. For example, the mobile device or head-mounted device can be configured to capture an image of an environment (e.g., a game room, a meeting area, a sports field, etc.), analyze the captured image to identify at least one object in the environment, determine parameters for the identified object, and determine display properties of a digital representation of a remote participant based on the determined parameters.

[0030] The term "mobile device" as used herein generally refers to any or all of a smartphone, a cellular phone, an Internet of Things (IOT) device, a personal or mobile multimedia player, a laptop computer, a tablet computer, an ultrabook, a handheld computer, a wireless email receiver, a multimedia Internet-enabled cellular phone, a wireless game controller, a head-mounted device, and similar electronic devices including a programmable processor, a memory, and circuitry for sending and / or receiving wireless communication signals to / from a wireless communication network. Although various embodiments are particularly useful in mobile devices such as smartphones and tablets, these embodiments are generally useful in any electronic device including communication circuitry for accessing a cellular or wireless communication network.

[0031] The phrase "head mounted device" and its acronym "HMD" may be used interchangeably herein to refer to any electronic display system that presents a combination of computer generated images and real world images from the user's physical environment (i.e., what the user would see without the glasses) to a user and / or enables the wearer / user to view the generated images in the context of a real world scene. As an example, a head mounted device may be, may include, or may be included in a helmet, glasses, virtual reality glasses, augmented reality glasses, electronic goggles, and other similar technologies / devices. In some embodiments, a head mounted device may include a processor, memory, a display, a camera, and a wireless interface for connecting to the Internet, a network, or other computing devices.

[0032] In some embodiments, the head mounted device may be an accessory for receiving video and / or from another computing device (e.g., a desktop, laptop, smartphone, tablet, etc.), wherein all or part of the processing is performed on the other computing device (e.g., Figure 1B and 11, etc.). Thus, in various embodiments, the head mounted device may be configured to perform all processing locally on a processor in the head mounted device, offload all main processing to a processor in another computing device (e.g., a laptop presented in a gaming room), etc., or split the main processing operations between the processor in the head mounted device and the processor in the other computing device. In addition, where there are multiple head mounted devices (e.g., presented in a conference with multiple participants) and / or there are multiple computing devices (e.g., multiple smartphones, a mix of smartphones, tablets, and laptops), the main processing operations may be performed in a distributed manner by the head mounted device and / or the computing devices may exchange data and computational results via wireless communications (e.g., Bluetooth, WiFi, LTE-Direct, etc.).

[0033] The term "environment" is used herein to generally refer to a visible context in which participants may appear in reality and in virtual form. The environment may include the physical location of local participants. Non-limiting examples of physical environments include game rooms, control rooms, classrooms, training facilities, sports fields (e.g., for games), etc. In some embodiments, the environment may include a virtual location, such as an electronic game field, in which all participants appear in virtual form. For example, the environment may be a virtual space in which participants interact in an augmented reality board game.

[0034] The phrases "remote participants" and virtual content are used herein to refer to people or digital representations of people who are projected to appear in the same virtual environment and interact with others but are not physically present in the same physical environment as other participants (e.g., the same sports field, conference room, etc.). The phrase "co-located participants" is used herein to refer to people (or devices held or worn by the person) who are physically present in the same environment as at least one other participant. In the following description and the appended claims, reference may be made to "virtual content," "remote participants," and "the remote participants," but such references are for ease of description, and the various embodiments are not limited to a single participant. Instead, the methods and operations of the various embodiments may be used to place and render any number of remote participants. Similarly, the various embodiments may work with any number of co-located participants. In addition, the various embodiments may be implemented in applications where all participants are remote participants, such as in virtual game applications. In addition, a "remote participant" may be an image of a virtual participant, such as an avatar (a computer-generated image of a person playing a role in a virtual reality game). In order to cover various types of images of individuals that do not actually exist, the term "virtual content" may be used to refer to virtual participants in the description and claims.

[0035] People are increasingly using their devices (e.g., smartphones, mobile devices, head mounted devices, etc.) to conduct or interact with others in augmented reality applications (such as augmented reality games, augmented reality training, virtual meetings, etc.). A popular use of augmented reality games, which can be played on a participant's smartphone, displays images of individuals who are not actually present, but are rendered in the image on the smartphone as either real appearance (e.g., as seen via a camera on the smartphone) or virtual appearance (e.g., avatar). In addition, augmented reality systems are revolutionizing the way operators of complex equipment such as complex control rooms, operations rooms, aircraft cockpits, robotic surgical environments are trained, and enabling local and remote individuals to participate in the same training sessions can significantly improve team training efficiency and cost effectiveness. In addition, all participants in a meeting no longer need to be physically present to interact with others, as if everyone were in the same game room, because the augmented reality system can be used to present images of all participants as if they were in the same environment.

[0036] Smartphones, modern virtual reality displays, and image capture devices can effectively improve and facilitate interactions between local and remote participants in various types of group activities (e.g., augmented reality games, group chats, meetings, group training, lectures, etc.). By rendering digital representations of remote participants in a virtual reality display, the remote participants appear to be in the same environment as the user / wearer of the user device. However, for virtual group activities (e.g., augmented reality games, meetings, etc.), where two or more co-located participants are actually present in the same environment (e.g., sitting in front of the same physical game board, in a game field, in the same room, area, control center, etc.), if the user device does not intelligently select the position and / or posture in which to render the digital representation of virtual content such as the remote participants, any improvements in productivity or user interaction provided by the user device will be ineffective. For example, if the placement of the digital representation of the remote participant in each co-located participant's user device (e.g., smart phone) is inconsistent, each co-located participant may face / look in a different direction when speaking or listening to virtual content such as the remote participant. This may cause confusion, distraction, disruption, and / or otherwise hinder natural human-to-human communication between participants.

[0037] In general, to facilitate natural person-to-person communication, it may be beneficial for a user device to place virtual content (e.g., digital representations of remote participants, etc.) at a fixed location that appears the same from the perspective of all participants, including co-located participants when some participants are physically present in the same environment. For example, if a digital representation of virtual content (e.g., a remote participant) is placed at a certain location on a gaming table by one user device, all other user devices of co-located participants may render the digital representation of the remote participant at the same location. In addition, it may be beneficial for a user device to intelligently adjust the size of the digital representation of the remote participant so that the remote participant does not appear too large, too small, or unnatural to the co-located participant. For example, the size of the digital representation of virtual content (e.g., a remote participant) may be adjusted to appear approximately the same size as the co-located participant in a smartphone display. Additionally, it may be beneficial for the user device to render digital representations of remote participants such that the remote participants are placed at an appropriate height relative to the co-located participant, within the field of view of the co-located participant, do not block the view of another participant, are appropriately spaced (e.g., all participants appear to be approximately equidistant, etc.), take into account distances between people (e.g., not too close to another participant so that it appears as if that person is violating personal space boundaries, etc.), and are otherwise easily visible (e.g., not placed in front of a bright or complex background, etc.).

[0038] Various embodiments include an augmented reality software application configured to run in a user device (e.g., a smart phone) and / or a user device configured to facilitate natural person-to-person communication (e.g., for eye contact, normal turn taking, etc.) between local and remote participants and virtual content by intelligently determining the position, pose, orientation, size, and other display characteristics of virtual content, such as digital representations of remote participants, to facilitate natural person-to-person communication (e.g., for eye contact, normal turn taking, etc.).

[0039] A device processor (e.g., a processor in a smartphone, in a head-mounted device worn by a co-located participant, in a mobile device communicatively coupled to a head-mounted device, etc.) can be configured to communicate with other device processors (e.g., in smartphones, in head-mounted devices worn by other co-located participants, etc.) to intelligently determine display properties of digital representations of virtual content, such as for remote participants. For example, the device processor can be configured to determine an environment, such as by rendering a virtual environment (e.g., an augmented reality game board, a virtual gaming environment, etc.) or by capturing an actual environment (e.g., a gaming table, a sports field, a conference room, a control room, a training environment, etc.) via a camera of a user device. For an actual environment, the device processor can analyze captured images to identify people and objects within the environment (e.g., tables, desks, chairs, walls, etc.), capture or compute data (e.g., via a sensor array or processor). For a virtual environment, the device processor can determine the shape and position of various virtual objects in the environment. The device processor can calculate various parameters for each identified object based on the captured / computed data, and send the determined parameters to other device processors (e.g., sending the data to the smartphones of other participants via a wireless data link).

[0040] In some embodiments, the device processor may also be configured to create a grid map of the scene using positioning and mapping techniques, such as simultaneous localization and mapping (SLAM), visual simultaneous localization and mapping (VSLAM), and / or other techniques known in the art. The device processor may use the grid map to determine the location of virtual content such as remote participants, and / or share the grid map with other devices in the environment so that all device processors know where virtual content should be placed in the environment.

[0041] The parameters calculated and communicated by the device processor may include fixed position parameters, size parameters, line of sight parameters, field of view parameters, spacing parameters, human distance parameters, height parameters, and / or visibility parameters. Each of these parameters may be associated with one or more weights indicating the importance of the parameter. For example, to accommodate a multi-party game with virtual content and many remote participants, the device processor may increase the weight associated with the fixed position parameter and / or decrease the weight associated with the size parameter. The device processor may also prioritize the parameters, giving fixed position, size, and line of sight parameters a higher priority than field of view, spacing, human distance, height, and visibility parameters.

[0042] Each device processor may receive parameters calculated by the other device processors and use the received parameters in conjunction with parameters calculated on the device to coordinate or negotiate display properties for rendering digital representations of virtual content, such as images of remote participants.

[0043] Each device processor can use coordinated display properties to render digital representations of virtual content, such as images of remote participants, such that all co-located participants (i.e., people physically present in the same physical environment) perceive the virtual content as being located in the same location in the environment (e.g., game board, game room, game scene, conference room, control room, war room, equipment, etc.), anchored to the same physical object (e.g., game piece, empty chair, console, etc.), and / or otherwise in a fixed position relative to the image of the environment.

[0044] The device processor can also be configured to use coordinated display properties to intelligently adjust the size of the digital representation so that virtual content, such as images of remote participants, does not appear unnatural when viewed by a participant through a device (e.g., a smartphone), so that the virtual content is placed at an appropriate height relative to the co-located participant, within the field of view of the co-located participant, does not block the view of another participant, is appropriately spaced (e.g., all participants appear approximately equidistant, etc.), takes into account distances between people (e.g., not rendered so close to another participant that it feels like the person is violating personal space boundaries, etc.) and is otherwise easily visible (e.g., not placed in front of a bright or complex background, etc.).

[0045] In some embodiments, the device processor may be configured to replicate the real world scene, such as by anchoring the digital representation of the virtual content to an empty chair within the game scene, behind a physical or virtual console, etc. In one embodiment, the device processor may be configured to select an empty chair (or location relative to a gaming table, device, etc.) that an individual (when the virtual content is an image of a remote participant) might have selected if he / she were physically present in the environment (e.g., a chair not far from other participants in the game, an empty console in a control room, etc.).

[0046] In some embodiments, the device processor may be configured to detect changes in the number, position, posture, size, etc. of objects, virtual content, or people in the environment, determine whether the detected changes exceed a threshold, and dynamically recalculate parameters or weights to account for the detected changes in response to determining that the detected changes exceed the threshold. For example, a device process (e.g., a processor in a participant's smart phone) may detect that a new participant (virtual, remote, or co-located) has joined a game, meeting, training course, etc. after it has started, and recalculate parameters and / or weights to account for the new participant, and update the rendering of virtual content (e.g., remote participants) based on the recalculated parameters and / or weights. As another example, a device process may detect that a participant (virtual, remote, or co-located) has moved position (e.g., by changing seats, moving laterally, by standing or sitting, etc.) after the game, meeting, training course, etc. has started, determine whether the distance between the participant's current position and the previous position exceeds a threshold, and in response to determining that the distance between the participant's current position and the previous position exceeds the threshold, recalculate parameters and / or update the rendering of the remote participant.

[0047] In some embodiments, the device processor may be configured to calculate additional values ​​and perform additional operations to account for special considerations associated with updating the rendering of remote participants or repositioning the placement of remote participants after a game, meeting, or training session has begun.

[0048] In some embodiments, the device processor may be configured to adjust the rendering of virtual content, such as images of remote participants (e.g., by resizing and re-spacing the virtual content) to support social engagement and promote communication between people. For example, the device processor may be configured to determine the placement of virtual content (such as images of virtual or remote participants) similar to how people typically join co-located games, meetings, training sessions, etc. In fully co-located games and meetings, latecomers typically find seats or standing positions close to others. It is also common for latecomers to choose seats or standing positions that are approximately equal distances from each other participant (e.g., people typically do not sit next to each other if there are other open seats that are more equally spaced). Part of the reason people naturally do this is to improve eye contact with the most participants. Latecomers may also squeeze into small group configurations, and other participants may slightly adjust their positions to make room for the latecomers. To replicate these human behavioral tendencies, the device processor may be configured to automatically adjust the position / rendering of other virtual content to maintain good spacing and / or take into account other considerations discussed above (e.g., line of sight, etc.).

[0049] In some embodiments, the device processor (e.g., the processor of a smartphone) can be configured to set the threshold high enough so that the rendering is not adjusted when certain changes are detected, such as when a single participant is standing. The device processor can set the threshold low enough so that the rendering is adjusted when other changes are detected, such as when most of the co-located people are standing or sitting down.

[0050] In some embodiments, the device processor may be configured to communicate user changes to other device processors for consistent rendering. For example, if a user manually adjusts the position at which a digital representation of virtual content, such as a remote participant, is rendered on his or her smartphone, the device processor may send the adjusted rendering position to other device processors, such as processors in smartphones of other people engaged in the same game, meeting, training, etc. In response, the other device processors may also adjust the digital representation of the same virtual content, thereby keeping the configuration of the virtual content consistent for all users relative to the environment.

[0051] Figure 1A A user device in the form of a head mounted device 100 is shown which may be configured to intelligently determine display characteristics of a virtual object (e.g., a digital representation of a remote participant) in accordance with various embodiments. Figure 1A In the example shown, the head mounted device 100 includes a frame 102, two optical lenses 106, and a processor 114, which is communicatively coupled to an image projector 108, a monocular image sensor or camera 110, a sensor array 112, a memory 116, and a communication circuit 118. The projector 108 can be embedded in the arm portion 104 of the frame 102 and is configured to project an image onto the optical lens 106. The projector 108 may include a light emitting diode (LED) module, an optical channel, a homogenizing lens, an optical display, a folding mirror, and other components well known in prior art projectors or head mounted displays. In some embodiments, the two optical lenses 106 may include imaging elements configured to superimpose an image on the view through the lens, in which case the projector 108 may not be implemented.

[0052] In some embodiments, the head mounted device 100 and / or the processor 114 may be configured to employ SLAM techniques to build and update a map of an unknown environment while keeping track of the location of the head mounted device 100 within the environment. For example, the head mounted device 100 may be equipped with a monocular image sensor that captures images or frames from the environment. The head mounted device 100 may identify salient objects (e.g., co-located participants) or features (e.g., facial and / or body features of co-located participants) within the captured image, estimate the size and proportion of the features in the image, compare the identified features to each other and / or to features in a test image with known size and proportion, and identify correspondences based on the comparison. Each correspondence may be a set of values ​​or an information structure that identifies a feature (or feature point) in one image as having a high probability of having the same feature in another image. In other words, a correspondence may be a set of corresponding image points (e.g., a first point in a first image and a second point in a second image, etc.). The head mounted device 100 may generate a homography matrix information structure based on the identified correspondence, and use the homography matrix to determine its posture (eg, position, orientation, etc.) in the environment.

[0053] Various embodiments may be implemented in a smartphone, an example of which is shown in Figure 1B 1. Smartphone 120 may include a processor 121 coupled to internal memory 122, a display 123, and a speaker 124. In addition, smart phone 120 may include an antenna 125 for sending and receiving electromagnetic radiation, which may be connected to a wireless data link and / or a cellular telephone transceiver 126 coupled to processor 121. Smartphone 120 also typically includes menu selection buttons or rocker switches 127 for receiving user input.

[0054] The typical smart phone 120 also includes a sound coding / decoding (CODEC) circuit 128, which digitizes the sound received from the microphone into a data packet suitable for wireless transmission, and decodes the received sound data packet to generate an analog signal provided to the speaker to produce sound. In addition, one or more of the processor 121, the wireless transceiver 126 and the CODEC 128 may include a digital signal processor (DSP) circuit (not shown separately).

[0055] Figure 1C 1 shows a computer architecture and various sensors that may be included in a head mounted device 100 or a smartphone 120 configured to employ SLAM or other techniques to render digital representations of remote participants in a virtual meeting according to various embodiments. Figure 1CIn the example shown, the head mounted device 100 includes a mainboard assembly 152 , an image sensor 154 , a microcontroller unit (MCU) 156 , an infrared (IR) sensor 158 , an inertial measurement unit (IMU) 160 , a laser distance sensor (LDS) 162 , and an optical flow sensor 164 .

[0056] The sensors 154, 158-164 can collect information useful for employing SLAM technology in the head mounted device 100 or the smart phone 120. For example, the optical flow sensor 164 can measure optical flow or visual motion and output measurements based on the optical flow / visual motion. Optical flow can identify or define patterns of apparent motion of objects, surfaces, and edges in a visual scene caused by relative motion between an observer (e.g., head mounted device 100, smart phone 120, wearer / user, etc.) and the scene. The MCU 156 can use the optical flow information to measure visual motion or relative motion between the head mounted device 100, smart phone 120 and other objects near the head mounted device 100, smart phone 120, etc. In addition, the head mounted device 100 or smart phone 120 can use the optical flow information for motion detection, object segmentation, contact time information, focus of expansion calculation, motion compensated encoding, stereo disparity measurement, and other similar information. In some embodiments, the optical flow sensor 164 may be an image sensor coupled to an MCU 156 (or processor 114, 121) programmed to run an optical flow algorithm. In some embodiments, the optical flow sensor 164 may be a vision chip that includes an image sensor and a processor on the same chip or die.

[0057] The head mounted device 100 or the smart phone 120 may be equipped with a variety of additional sensors including gyroscopes, accelerometers, magnetometers, magnetic compasses, altimeters, cameras, optical readers, orientation sensors, monocular image sensors, and / or similar sensors for monitoring physical conditions (e.g., position, motion, acceleration, orientation, altitude, etc.) or collecting information useful for employing SLAM techniques.

[0058] In the following description of various embodiments, the drawings and descriptions of the drawings refer to example implementations of various embodiments in which local and remote participants meet around a gaming table, such as on which real and virtual table game boards are presented in an augmented reality game. This example is for illustrative purposes and is not intended to be limiting. Various embodiments are equally applicable to other applications in which virtual reality is used to enable local and remote participants and virtual content to interact as if they were physically in the same environment, such as involving a common gaming environment (e.g., a game board, a gaming scene, etc.), a conference table for meetings, a device control panel for a training session involving equipment that may be physically presented or virtually rendered, a training session between a team of decision makers, a meeting in a control room or war room, and the like.

[0059] Figure 2A Head mounted devices 100a, 100b configured in accordance with various embodiments are shown for use in playing a game with virtual content, such as a remote participant 202. A device processor (e.g., processor 121 in a participant's smartphone 200a, processor 114 in head mounted device 100a, etc.) may be configured to cooperate with other device processors (e.g., processor 121 in a participant's smartphone 200b, processor 114 in head mounted device 100b) to intelligently determine display properties of a digital representation of virtual content, such as a remote participant 202. For example, the device processor can be configured to capture images of an environment (e.g., a gaming room) via a camera of a smartphone or head mounted device, analyze the captured images to identify people and objects within the environment (e.g., gaming table 206, gaming board 207, co-located participants 204a and 204b, etc.), capture or compute data (e.g., via the camera and sensors of smartphone 200a, via sensor array 112 or processor 114 of head mounted device 100a, etc.), compute various parameters for each identified object based on the captured / computed data, and send the computed / determined parameters to other device processors (e.g., processor 121 within another smartphone 200b). The device processor can use these parameters to determine various display properties and render a digital representation of virtual content such as remote participant 202, such that the virtual content is placed on gaming table 206 and appears to be within fixed area 208 for each co-located participant 204a, 204b, such as relative to a real or virtual gaming board 207.

[0060] Figure 2A It is also shown that the smartphones 200a, 200b and / or the head mounted devices 100a, 100b can be configured to adjust the size of the digital representation of virtual content, such as the remote participant 202, so that the remote participant does not appear too large, too small, or otherwise unnatural to the co-located participants 204a, 204b. The smartphones 200a, 200b and / or the head mounted devices 100a, 100b can also render the digital representation of virtual content, such as the remote participant 202, so that the virtual content appears to be placed at an appropriate height 210 relative to the co-located participants 204a, 204b, is appropriately spaced (e.g., all participants appear to be approximately equidistant, etc.), takes into account interpersonal distance (e.g., is not rendered so close to another participant that it feels as if the person is encroaching on personal space boundaries, etc.), and is otherwise easily visible (e.g., is not placed in front of a bright or complex background, etc.).

[0061] Figure 2BIt is shown that a digital representation of virtual content, such as a remote participant 202, can be rendered so that the virtual content is within the field of view of co-located participants 204a, 204b viewing through their smartphone displays or head mounted devices and does not block the view of any remote or co-located participant.

[0062] Figure 3A and 3B Multiple conference participants are shown co-located in the same environment (eg, a gaming room), seated around a gaming table 206 and wearing head mounted devices 100a - 100d .

[0063] Reference Figure 3A , the co-located participants 304a-304d have selected their seats and / or have arranged themselves on three sides of the gaming table 206 so that they can interact with each other and view the game board, whiteboard, television, monitor, or presentation 302. Each of the co-located participants 304a-304d is in a slightly different physical location relative to the environment and the gaming table 206, and there is a person-sized separation 306 between the two co-located participants 304b and the co-located participant 304c.

[0064] refer to Figure 3B , the smartphones 200a, 200b and / or the head mounted devices 100a-100d may be configured to communicate with each other to collaboratively determine that a digital representation of virtual content, such as a remote participant 202, should be rendered at a human-sized interval 306. As part of these operations, the smartphones 200a, 200b and / or the head mounted devices 100a-100d may analyze, calculate, or interpret various parameters (and their respective weights), including fixed position parameters, size parameters, line of sight parameters, field of view parameters, interval parameters, interpersonal distance parameters, height parameters, and visibility parameters.

[0065] The fixed position parameter may indicate that a digital representation of virtual content, such as remote participant 202, should be placed at the same fixed position relative to all co-located participants 304a-304d. For example, if a digital representation of virtual content, such as remote participant 202, is placed in front of a certain location on gaming table 206, all co-located participants may see the virtual content area at the same location or at the fixed location. The weight value associated with the fixed position parameter may indicate the weight or importance of rendering the digital representation of virtual content in the fixed location relative to other parameters.

[0066] The size parameters may indicate that the size of the rendering of the digital representation of virtual content, such as the remote participant 202, should be consistent for all co-located participants 304a-304d and should be appropriate (e.g., a natural and expected size) for the location in which it is rendered. For example, the size parameters may indicate that the virtual content, which is the body of the remote participant, should be rendered to be approximately the same size as the body of the co-located participant 304a-304d. To achieve this, one or more of the smart phones 200a, 200b and / or the head mounted displays 100a-100d may be configured to measure the head size of the co-located participants 304a-304d and / or the distance between them, generate an average, and use this information to determine the size of the rendering of the digital representation of virtual content, such as the remote participant 202. The weight value associated with each size parameter may indicate the weight or importance of rendering the digital representation of virtual content, such as the remote participant 202, to be consistent for all co-located participants and / or to be approximately the same size as the co-located participants 304a-304d.

[0067] The line of sight parameters may indicate that a digital representation of virtual content, such as remote participant 202, should not be rendered in a position that blocks the line of sight of any co-located participant 304a-304d, which may be determined by smartphone 200a, 200b and / or head mounted device 100a-100d identifying an area that blocks part or all of another participant's field of view. In some embodiments, one or more of smartphones 200a, 200b and / or head mounted devices 100a-100d may be configured to measure area and / or line of sight broadly so that a digital representation of virtual content, such as remote participant 202, is not placed too close to a neighboring participant (e.g., co-located participant 304b or co-located participant 304c). The weight value associated with each line of sight parameter may indicate the weight or importance of rendering a digital representation of virtual content, such as remote participant 202, to not block the line of sight of any co-located participant 304a-304d.

[0068] The field of view parameters may indicate that a digital representation of virtual content, such as a remote participant 202, should be rendered in a manner that fits within the fields of view of as many co-located participants 304a-304d as possible. The weight values ​​associated with each field of view parameter may indicate the weight or importance of rendering a digital representation of virtual content, such as a remote participant 202, to fit within the field of view of another co-located participant 304a-304d. Because the smartphones 200a, 200b and / or head mounted displays 100a-d may have a narrow field of view, it may be difficult to render virtual content, such as a remote person, within the field of view of all co-located participants 304a-304d. In such cases, the weight of one or more of the field of view parameters may be reduced to account for the characteristics of the individual smartphones 200a, 200b and / or head mounted devices 100a-100d and / or any difficulties associated with rendering a digital representation of virtual content, such as a remote participant 202, within the field of view of all co-located participants 304a-304d.

[0069] The spacing parameter may indicate that a digital representation of virtual content, such as remote participant 202, should be placed equidistant from co-located participants 304a-304d, similar to how people typically organize themselves in a face-to-face gaming setting. For example, one person typically does not sit too close to another person, nor does one sit too far from others. To address this issue, the smartphone 200a, 200b, and / or head mounted display 100a-100d may measure the distance between each other participant, taking into account obstacles and objects in the environment (e.g., presentation 302), and render the digital representation of the remote participant 202 in a position on the acceptable side (e.g., the side that does not include presentation 302) with the most available space. The smartphone 200a, 200b, and / or head mounted display 100a-100d may take the measurements, calculate an average score, use the average score to determine the longest side, and place the digital representation of the remote participant 202 on the longest side. The process may be repeated for subsequent remote participants, taking into account previously placed virtual content, such as remote participants. The weight value associated with each spacing parameter may indicate the weight or importance of rendering a digital representation of virtual content, such as remote participant 202, equidistant from co-located participants 304a-304d.

[0070] The interpersonal distance parameter may indicate that a digital representation of virtual content, such as remote participant 202, should not be rendered so close to another participant (e.g., co-located participant 304b) that it appears as if the remote participant is violating personal space boundaries. Because such interpersonal distance may vary from culture to culture, smartphones 200a, 200b and / or head mounted displays 100a-d may be configured to determine distances based on pre-configured values, behaviors and distances between co-located participants, artificial intelligence, or other similar parameters. A weight value associated with the interpersonal distance parameter may indicate a weight or importance of rendering a digital representation of virtual content, such as remote participant 202, so that the remote participant does not appear to violate the personal space boundaries of co-located participants 304a-304d.

[0071] The height parameter may indicate that a digital representation of virtual content, such as remote participant 202, should be placed at an appropriate height, given the position, posture, and height of the co-located participants 304a-304d. Depending on the position, posture, and height of the co-located participants 304a-304d, this may require a sitting or standing height. In some embodiments, the smartphones 200a, 200b, and / or head mounted devices 100a-d may be configured to determine the height as the average head or eye height of each co-located participant. In some embodiments, the smartphones 200a, 200b, and / or head mounted devices 100a-d may be configured to determine whether most of the co-located participants 304a-304d are sitting or standing, and take the average of the majority. For example, if three co-located participants are sitting and two co-located participants are standing, the smartphones 200a, 200b, and / or head mounted devices 100a-d may take the average eye height of the three seated participants and place a digital representation of virtual content, such as remote participant 202, at that level. The weight value associated with each height parameter may indicate the weight or importance of rendering an appropriate height for a digital representation of virtual content, such as remote participant 202 , relative to the other parameters.

[0072] The visibility parameters may indicate that the digital representation of virtual content, such as remote participant 202, should be placed to optimize visibility, such as by avoiding bright or complex backgrounds or anchoring objects that may obstruct visibility of the digital representation of virtual content, such as remote participant 202, or interfere with the view of co-located participants 304a-304d. The weight value associated with each visibility parameter may indicate the weight or importance of rendering the digital representation to optimize visibility.

[0073] Return to Figure 3BIn the example shown, the smartphones 200a, 200b and / or the head mounted devices 100a-d can identify people and objects in the environment, including game participants, the game table 206 and / or the presentation 302, by capturing and analyzing images of the environment to determine that a digital representation of virtual content, such as the remote participant 202, should be rendered at a person-sized interval 306. The head mounted devices 100a-d can calculate various parameters (e.g., depth, distance, direction, etc.) for the identified objects / people in the environment. The smartphones 200a, 200b and / or the head mounted devices 100a-100d can use the calculated / analyzed parameters to determine that the digital representation of the remote participant 202 should be rendered at a person-sized interval 306. That is, the smart phones 200a, 200b and / or head mounted devices 100a-100d can determine, based on an analysis of one or more of the above parameters and their respective weights, that placing a digital representation of virtual content, such as a remote participant 202, within the human-sized interval 306 will allow co-located participants 304a-304d to continue viewing the presentation 302 and communicating with each other.

[0074] In response to determining that the digital representation of virtual content, such as the remote participant 202, should be rendered at the person-sized interval 306, the smartphone 200a, 200b and / or the head mounted device 100a-100d may search the captured image of the environment to identify a surface or object (e.g., an empty chair) that is within or immediately adjacent to the person-sized interval 306, select the identified surface / object for use as an anchor, and determine display properties (e.g., size, etc.) for the digital representation of virtual content, such as the remote participant 202, based on the calculated / analyzed parameters and the relative distance between the selected surface / object and the co-located participants 304a-304d. The smartphone 200a, 200b and / or the head mounted device 100a-100d may render the digital representation of virtual content, such as the remote participant 202, such that the virtual content appears in a fixed location for all co-located participants (e.g., anchored to an anchor surface / object) and / or such that the co-located participants can easily and naturally communicate and interact with each other and with the virtual content.

[0075] Figure 4A An example is shown where a digital representation of virtual content, such as remote participant 202, is placed in a natural position relative to gaming table 206, and each of co-located participants 204a-204c has a direct line of sight (indicated by dashed arrows) to the digital representation of virtual content. However, the position / pose of the digital representation of virtual content, such as remote participant 202, blocks the line of sight of co-located participants 204b and 204c. This may occur, for example, if co-located participant 204c joins the game after the game is in progress.

[0076] Figure 4B Participant smartphones and / or head mounted devices configured in accordance with an embodiment are shown collaborating to determine that a line of sight between co-located participants 204b and 204c is blocked, and to reposition a digital representation of virtual content, such as remote participant 202, to a different position or posture relative to a gaming table 206 such that all co-located participants 204a-204c have direct line of sight to each other (indicated by solid arrows) and to a digital representation of virtual content, such as remote participant 202 (indicated by dashed arrows).

[0077] Figures 5A-5C It is shown that participant smartphones and / or head mounted devices configured in accordance with embodiments can intelligently determine the positions in which digital representations of virtual content, such as remote participants 202a, 202b, should be rendered relative to the positions or postures of co-located participants 204a-204c. Figure 5A It is shown that the participant smartphones and / or head mounted devices may be configured to calculate distances between co-located participants 204a-204c and determine that the maximum distance (L) between participants is between co-located participant 204a and co-located participant 204c.

[0078] refer to Figure 5B , the participant smartphones and / or head mounted devices may render a digital representation of virtual content, such as first remote participant 202a, between co-located participant 204a and co-located participant 204c. The smartphones and / or head mounted devices may recalculate the distances between the co-located participants 204a-204c and the digital representations of virtual content, such as remote participant 202a, and determine that the maximum distance (L) between the participants is now between co-located participant 204a and co-located participant 204b.

[0079] Figure 5C It is shown that the participant smartphone and / or head mounted device can render a digital representation of virtual content such as a second remote participant 202b between the co-located participant 204a and the co-located participant 204b. The participant smartphone and / or head mounted device can recalculate the distance between the co-located participants 204a-204c and the digital representation of virtual content such as the remote participants 202a, 202b, and determine that the maximum distance (L) between the participants is now between the co-located participant 204b and the co-located participant 204c. The smartphone and / or head mounted device can repeat these operations until all remote participants are rendered in the environment.

[0080] Figure 6-9Methods 600, 700, 800, 900 of determining display attributes of a digital representation of virtual content for a remote participant, such as a remote participant, are shown in accordance with various embodiments (e.g., including any or all of the embodiments discussed above with reference to FIGS. 1-5 ). All or part of the methods 600, 700, 800, 900 may be performed by a device processor, such as a processor in a head mounted device worn by a co-located participant (e.g., Figure 1A ), a processor in a smartphone (e.g., Figure 1B ), independently or communicatively coupled to a head-mounted device worn by a co-located participant, a processor in a server computing device communicatively coupled to a smart phone or head-mounted device, or a processor associated with a cloud network or cloud-based computing system.

[0081] refer to Figure 6 In block 602 of method 600, a device processor may capture, generate, or receive an image of an environment. For example, a camera on a smartphone or head mounted device may capture a video of the environment and provide image frames to the device processor. As another example, images from the environment may be captured by one or more cameras located within the environment and in communication with the device processor.

[0082] In block 604, the device processor may analyze the captured image to identify at least one object in the environment (e.g., a person, a gaming table, an empty seat, etc.). In some embodiments, as part of the operations in block 604, the device processor may use localization and mapping techniques such as machine vision, VSLAM, or other techniques and technologies known in the art to create a grid or map of the environment, identify objects, determine the position, orientation, and / or pose of objects in the environment, and / or perform other similar operations.

[0083] In block 606, the device processor may determine parameters for the identified object. For example, the device processor may determine any or all of a fixed position parameter, a size parameter, a line of sight parameter, a field of view parameter, a spacing parameter, a human distance parameter, a height parameter, and / or a visibility parameter. In some embodiments, each of the determined parameters may be associated with one or more weights or priorities indicating a corresponding importance.

[0084] In some embodiments, the device processor may be configured to adjust the priority or weight associated with the parameter based on user input and / or taking into account various conditions, events, or preferences. For example, the device processor may determine that rendering digital representations of virtual content such as remote participants at different locations for participants with different locations is more dispersed or disruptive to communication and interaction between people than rendering digital representations of virtual content such as remote participants much smaller than the size of co-located participants physically present in the environment. Therefore, in order to accommodate games or meetings with many remote participants, the device processor may increase the weight associated with the fixed position parameter or reduce the weight associated with the size parameter. As another example, the device processor may determine that rendering digital representations of virtual content such as remote participants at different locations for different co-located participants is more dispersed or disruptive to communication and interaction between people than rendering digital representations of virtual content such as remote participants too close to co-located participants physically present in the environment. In response, the device processor may assign a higher priority or weight to the fixed position parameter than the priority / weight assigned to the human distance parameter.

[0085] In some embodiments, the device processor may be configured to adjust the priority or weight associated with the parameter based on user input. In such embodiments, the device processor may allow the user / wearer to manually control the importance of various parameters to that user or to one or more users (e.g., game participants, etc.). In some embodiments, the device processor may receive user input in the form of user selections made in a settings menu of a user interface of a smartphone, head mounted display, and / or related computing device.

[0086] In some embodiments, the device processor may assign a higher priority or weight to a fixed position parameter, size parameter, or line of sight parameter than to a field of view parameter, a spacing parameter, an interpersonal distance parameter, an altitude parameter, and a visibility parameter.

[0087] return Figure 6 , the device processor may determine display properties of the digital representation of virtual content, such as the remote participant, based on the parameters determined in block 608 (and their respective weights / priorities). For example, based on the parameters determined in block 608, the device process may determine that the digital representation of virtual content, such as the remote participant, should be rendered in an area surrounded by empty space (e.g., empty chairs, etc.) presented in the environment. As another example, the device process may determine, based on the parameters determined in block 608, that the digital representation of virtual content, such as the remote participant, should be sized so that the virtual content appears to the user / wearer to be approximately 6 feet tall, with a head width of 6 to 7 inches, a length of 8 to 9 inches, a chest circumference of 44 to 46 inches, and a shoulder width of 16 inches.

[0088] refer to Figure 7 In blocks 602-608 of method 700, the device processor may perform operations of similarly numbered blocks of method 600, as described with reference to Figure 6 as described.

[0089] In block 710, the device processor may identify other devices (e.g., other smartphones, computing devices, mobile devices, head mounted devices, etc.) that are in close physical proximity (e.g., in the same environment, area, gaming room, etc.) to the device processor. This may be done by, for example, broadcasting a ping message or sending a beacon to request that all devices receiving the message / beacon respond with an identifier (e.g., a media access control (MAC) address, etc.), communication parameters, location information, and / or other similar information.

[0090] In block 712, the device processor may send the parameters determined in block 606 and / or the display properties determined in block 608 to the identified device (ie, other devices in the same environment, etc.).

[0091] refer to Figure 8 In blocks 602-606 of method 800, the device processor may perform operations of similarly numbered blocks of method 600, as described with reference to Figure 6 as described.

[0092] In block 808, the device processor may receive additional parameters from one or more other devices that are in close physical proximity (e.g., in the same environment) to the device processor. For example, as each device processor determines parameters, those parameters may be sent to all other device processors within the environment. As another example, if a single device processor is performing the operations of the various embodiments for all devices in the environment, each smartphone or head mounted device may send images, locations, orientations, and other information to that one processing device.

[0093] In block 810 , the device processor may determine display properties of the digital representation of the virtual content for, for example, the remote participant based on the determined parameters and additional parameters received from other devices.

[0094] In block 812, the device processor may send the display properties determined in block 810 to the identified device (ie, other devices in the same environment, etc.).

[0095] refer to Fig. 9 In blocks 602-606 of method 900, the device processor may perform operations of similarly numbered blocks of method 600, as described with reference to Figure 6 as described.

[0096] In box 908, the device processor can negotiate with other devices in the environment (or in close physical proximity to participant smartphones, head mounted devices, head mounted devices, etc.) for display properties for rendering digital representations of virtual content such as remote participants to generate coordinated display properties. For example, the device processor can send suggested display properties to other devices and receive their own suggested display properties from each device. One, some, or all of the device processors in the device can adopt various decision methods to obtain a single set of display properties. For example, each device processor can test each received set of display properties and send an indication to the other device whether any of the various criteria for determining display properties is violated. As another example, one device processor within the environment can be designated as a controller for determining display properties, and can do so based on input (e.g., position and orientation information and images captured by other head mounted displays) and use the collected information to determine the display properties. Other methods can be used to determine coordinated display properties.

[0097] In block 910, the device processor may render a digital representation of virtual content, such as a remote participant, based on the coordinated display properties. For example, the device processor may render the digital representation of virtual content, such as a remote participant, at the same fixed position relative to all co-located participants using the coordinated display properties. That is, because the display properties are negotiated and coordinated, if the digital representation of virtual content, such as a remote participant, is placed at a certain location on a gaming table of another smartphone or head mounted device in the environment, the device processor will render the digital representation of virtual content, such as a remote participant, at the same location.

[0098] Fig.10 A method 1000 for updating display properties of a digital representation of virtual content for a remote participant, such as a remote participant, is shown according to an embodiment. The method 1000 may be performed after the display properties of the digital representation are determined, the digital representation is rendered, or a virtual game has begun. All or part of the method 1000 may be performed by a device processor, such as a processor (e.g., processor 121) in a head mounted device or smartphone worn by a co-located participant. Figure 1A 114).

[0099] In block 1002, the device processor may capture, generate, or receive an updated image of the environment. For example, a camera on one or all devices may capture video of the environment and periodically provide it to a device processor (e.g., a smartphone processor, a processor of a head mounted device, a processor of a remote computing device, etc.). As another example, images of the environment may be periodically captured by one or more cameras located within the environment and in communication with the device processor.

[0100] In block 1004, the device processor may analyze the updated image to determine whether the number, position, posture, or size of people or objects in the environment has changed. Such operations may include determining the boundaries of various participants and objects and comparing the determined boundaries with the boundaries determined and stored in memory the last time display parameters were determined.

[0101] In block 1006 , the device processor may determine updated parameters of objects identified in the environment in response to determining that a position, posture, or size of a person or object in the environment has changed.

[0102] In block 1008 , the device processor may determine the difference between the original parameters and the updated parameters.

[0103] In determination block 1010, the device process may determine whether the difference between the original parameters and the updated parameters exceeds a threshold. The threshold may be set high enough so that the rendering is not adjusted when certain changes are detected, such as when a single participant is standing. Alternatively or additionally, the threshold may be set low enough so that the rendering is adjusted when certain changes are detected, such as when a majority of co-located persons are standing or sitting down.

[0104] In box 1012, in response to determining that the difference between the original parameters and the updated parameters does not exceed the threshold (i.e., determination box 1010 = "No"), the device processor may continue to render a digital representation of virtual content such as a remote participant based on the original parameters (e.g., the parameters calculated in box 606, etc.) and / or the original display properties (e.g., the display properties calculated in box 608, etc.).

[0105] In response to determining that the difference between the original parameters and the updated parameters exceeds the threshold (i.e., determination block 1010 = "Yes"), the device processor may determine updated display properties for the digital representation based on the updated parameters in block 1014. For example, the device processor may perform one or more of the operations of any of methods 600, 700, 800, or 900 to determine the updated parameters in block 1014.

[0106] In block 1016, the device processor may render a digital representation of the virtual content, such as the remote participant, based on the updated display properties.

[0107] Some embodiments may include methods for determining display properties of images for displaying one or more remote participants in a game, augmented reality game, training course, conference, etc., such as in a virtual reality system by capturing (by a processor associated with a smartphone and / or head mounted device) images of a game room / area, analyzing the captured images to identify people and objects in the game room / area, determining parameters of the identified people and objects, and determining display properties of digital representations of virtual content such as the remote participants based on the determined parameters.

[0108] In some embodiments, determining parameters of the identified people and objects may include determining parameters based on one or more factors, wherein the one or more factors include at least one or more of a fixed position factor, a size factor, a line of sight factor, a field of view factor, a spacing factor, a human distance factor, a height factor, or a visibility factor.

[0109] Some embodiments may include assigning a priority to each of one or more factors, wherein determining the display attributes of the digital representation of the remote participant based on the determined parameters may include determining the display attributes of the digital representation of the remote participant based on the priority assigned to each of the one or more factors, and assigning a priority to each of the one or more factors may include assigning a higher priority to a fixed position factor, a size factor, or a line of sight factor than to a field of view factor, a spacing factor, an interpersonal distance factor, a height factor, and a visibility factor.

[0110] Some embodiments may include identifying other head mounted devices in the game room / area and sending the determined parameters to other smartphones and / or head mounted devices in the game room / area. Some embodiments may include receiving additional parameters from other smartphones and / or head mounted devices in the game room / area, wherein display properties of digital representations of virtual content such as remote participants are determined based on the determined parameters, which may include determining display properties of digital representations of virtual content such as remote participants based on the determined parameters and additional parameters received from other smartphones and / or head mounted devices in the game room / area. In some embodiments, determining display properties of digital representations of virtual content such as remote participants based on the determined parameters may include negotiating with other smartphones and / or head mounted devices in the game room / area for rendering digital representations of virtual content to generate coordinated display properties.

[0111] Some embodiments may include using coordinated display properties to render digital representations of virtual content, such as remote participants, such that all co-located participants physically present in the game room / area perceive the virtual content to be in the same fixed location in the game room / area. Some embodiments may include using coordinated display properties to intelligently adjust the size of digital representations of remote participants such that the remote participants appear to be approximately the same size as the co-located participants physically present in the game room / area. Some embodiments may include using coordinated display properties to render digital representations of virtual content, such as remote participants, such that the virtual content appears to be approximately the same average height as the co-located participants physically present in the game room / area.

[0112] Some embodiments may include rendering digital representations of virtual content, such as remote participants, using coordinated display properties so that the virtual content appears to be approximately equidistant from co-located participants physically present in the game room / area. Some embodiments may include rendering digital representations of virtual content, such as remote participants, using coordinated display properties so that the virtual content appears to be anchored to empty chairs at a game table. Some embodiments may include capturing updated images of the game room / area, analyzing the captured updated images to determine whether the number, position, posture, or size of people or objects in the game room / area has changed, identifying people and objects in the game room / area in response to determining that the position, posture, or size of people or objects in the game room / area has changed, determining whether a difference between the determined parameters and the determined updated parameters exceeds a threshold, and determining updated display properties of digital representations of virtual content, such as remote participants, in response to determining that the difference between the determined parameters and the determined updated parameters exceeds the threshold.

[0113] Various embodiment methods may be implemented in various personal computing devices, such as Fig.11 The laptop computer 1100 is shown. The laptop computer 1100 generally includes a processor 1101 coupled to a volatile memory 1102 and a disk drive 1104 of a large capacity non-volatile memory, such as flash memory. The laptop computer 1100 may also include a floppy disk drive 1105 coupled to the processor 1106. The computer receiver device 1100 may also include a plurality of connector ports or other network interfaces coupled to the processor 1101 for establishing a data connection or receiving an external memory receiver device, such as a universal serial bus (USB) or A connector socket, or other network connection circuitry for coupling the processor 1101 to a network (e.g., a communications network). In a notebook configuration, the computer housing includes a touchpad 1110, a keyboard 1112, and a display 1114, all of which are coupled to the processor 1101. As is well known, other configurations of computing devices may include a computer mouse or trackball coupled to the processor (e.g., via USB input), which may also be used in conjunction with various embodiments.

[0114] The processor may be any programmable microprocessor, microcomputer or multi-processor chip or multiple processor chips that can be configured by software instructions (applications) to perform various functions, including the functions of the various embodiments described in this application. In some mobile devices, multiple processors may be provided, such as one processor dedicated to wireless communication functions and one processor dedicated to running other applications. Typically, software applications may be stored in internal memory before they are accessed and loaded into the processor. The processor may include internal memory sufficient to store application software instructions.

[0115] The various embodiments shown and described are provided as examples only to illustrate the various features of the claims. However, the features shown and described with respect to any given embodiment are not necessarily limited to the associated embodiment and may be used or combined with other embodiments shown and described. In addition, the claims are not intended to be limited to any one example embodiment. For example, one or more operations of the method may replace one or more operations of the method or be combined with one or more operations of the method.

[0116] The foregoing method descriptions and process flow charts are provided only as illustrative examples and are not intended to require or imply that the operations of the various embodiments may be performed in the order presented. As will be appreciated by those skilled in the art, the order of operations in the foregoing embodiments may be performed in any order. Words such as "thereafter," "then," "next," etc. are not intended to limit the order of operations; these words are used to guide the reader through the description of the method. In addition, any reference to a claim element in the singular, for example, the use of the article "a," "an," or "the" should not be construed as limiting the element to the singular.

[0117] The various illustrative logic blocks, functional components, functional components, circuits, and algorithmic operations described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software, or a combination of the two. In order to clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, functional components, circuits, and operations have been described above generally according to their functions. Whether such functions are implemented as hardware or software depends on the specific application and the design constraints imposed on the entire system. A skilled person can implement the described functions in different ways for each specific application, but such embodiment decisions should not be interpreted as causing a departure from the scope of the claims.

[0118] The hardware used to implement the various illustrative logics, logic blocks, functional components and circuits described in conjunction with the embodiments disclosed herein may be implemented or performed with a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination designed to perform the functions described herein. A general purpose processor may be a microprocessor, but in an alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in combination with a DSP core, or any other such configuration. Alternatively, some operations or methods may be performed by circuits specific to a given function.

[0119] In one or more embodiments, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored as one or more instructions or codes on a non-transitory computer-readable medium or a non-transitory processor-readable medium. The operation of the method or algorithm disclosed herein may be embodied in a processor-executable software module, which may reside on a non-transitory computer-readable or processor-readable storage medium. A non-transitory computer-readable or processor-readable storage medium may be any storage medium accessible by a computer or processor. As an example and not limitation, such non-transitory computer-readable or processor-readable media may include RAM, ROM, EEPROM, flash memory, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the required program code in the form of instructions or data structures and can be accessed by a computer. As used herein, discs and disks include compact discs (CDs), laser discs, optical discs, digital versatile discs (DVDs), floppy discs, and blue discs, where discs typically reproduce data magnetically, while discs reproduce data optically with lasers. The above combinations are also included within the scope of non-transitory computer-readable and processor-readable media. Furthermore, the operations of a method or algorithm may reside as one or any combination or set of codes and / or instructions on a non-transitory processor-readable medium and / or computer-readable medium, which may be incorporated into a computer program product.

[0120] The foregoing description of the disclosed embodiments is provided to enable any person skilled in the art to formulate or use the claims. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments and implementations without departing from the scope of the claims. Therefore, the present disclosure is not intended to be limited to the embodiments and implementations described herein, but is to be consistent with the broadest scope consistent with the following claims and the principles and novel features disclosed herein.

Claims

1. A method for determining a display location of virtual content for an environment having a plurality of co-located participants, each co-located participant having a respective user device, the method comprising: capturing an image of the environment by an image sensor controlled by a processor of a user device, the user device being a user device of one of the plurality of co-located participants; analyzing, by the processor, the captured images to identify objects in the environment; determining, by the processor, parameters for the identified object; receiving, by the processor, additional parameters from at least one other user device in the environment; determining, by the processor, a position of the digital representation of the virtual content relative to the identified object based on the determined parameter and the received additional parameter, the determining comprising negotiating with the at least one other user device to generate a coordinated display position of the virtual content; sending, by the processor, the coordinated display position to the at least one other user device; as well as A digital representation of the virtual content is rendered using the coordinated display position such that the plurality of co-located participants perceive the virtual content to be at a same fixed location in the environment.

2. The method according to claim 1 also includes assigning priorities to the determined parameters, wherein determining the position of the digital representation of the virtual content relative to the identified object based on the determined parameters includes determining the position of the digital representation of the virtual content relative to the identified object based on the priorities assigned to the determined parameters.

3. The method according to claim 2, wherein: Determining the parameters for the identified object comprises: determining at least one or more of a fixed position parameter, a size parameter, or a line of sight parameter; and At least one or more of a field of view parameter, a separation parameter, an interpersonal distance parameter, an altitude parameter, and a visibility parameter are determined.

4. The method according to claim 3, wherein: The parameters identified are assigned priorities including: assigning a first priority to the fixed position parameter, the size parameter, or the line of sight parameter; and assigning a second priority to the field of view parameter, the separation parameter, the interpersonal distance parameter, the height parameter or the visibility parameter, and The first priority is a higher priority than the second priority. 5 . The method of claim 1 , further comprising using one or more coordinated display properties to adjust a size of the digital representation of the virtual content based on a size of a participant physically present in the environment. 6 . The method of claim 1 , further comprising rendering the digital representation of the virtual content using one or more coordinated display attributes such that the virtual content appears to be located between a first co-located participant and a second co-located participant.

7. The method according to claim 6, wherein: Rendering the digital representation of the virtual content using the one or more coordinated display attributes so that the virtual content appears to be located between the first co-located participant and the second co-located participant includes rendering the digital representation of the virtual content using the one or more coordinated display attributes so that the virtual content appears to be approximately equidistant from the first co-located participant and the second co-located participant.

8. The method according to claim 6, wherein: The virtual content includes remote participants.

9. The method of claim 8, further comprising rendering the digital representation of the remote participant using the one or more coordinated display attributes so that the remote participant appears to be anchored to an empty seat.

10. The method according to claim 1, further comprising: capturing an updated image of the environment; analyzing the captured updated image to determine whether the number, position, pose, or size of the identified objects has changed; In response to determining that the position, posture, or size of the identified object has changed, determining updated parameters for the identified object; determining whether a difference between the determined parameter and the determined updated parameter exceeds a threshold; as well as In response to determining that a difference between the determined parameter and the determined updated parameter exceeds a threshold, an updated display property of the digital representation of the virtual content is determined.

11. The method according to claim 1, wherein: Determining the position of the digital representation of the virtual content relative to the identified object based on the determined parameters includes determining at least one position of each digital representation of each of a plurality of remote participants.

12. The method according to claim 1, wherein: Determining the parameters for the identified object includes determining at least one or more of the following: Fixed position parameters; Size parameters; Line of sight parameters; Field of view parameters; Interval parameters; Human distance parameters; Height parameter; or Visibility parameter.

13. The method according to claim 1, wherein: The virtual content is related to the game.

14. A method for determining a display position of virtual content for an environment having a plurality of co-located participants, each co-located participant having a respective user device, the user device comprising: Memory; Image sensor; as well as a processor coupled to the memory and the image sensor, the processor being configured to: capturing an image of the environment using the image sensor; analyzing the captured images to identify objects in the environment; determining parameters for the identified object; receiving additional parameters from at least one other user device in the environment; determining a position of a digital representation of the virtual content relative to the identified object based on the determined parameters and the received additional parameters, the determining comprising negotiating with the at least one other user device to generate a coordinated display position of the virtual content; sending the coordinated display position to the at least one other user device; as well as A digital representation of the virtual content is rendered using the coordinated display position such that the plurality of co-located participants perceive the virtual content to be at a same fixed location in the environment.

15. The user equipment according to claim 14, wherein: The processor is configured to: assigning priorities to the determined parameters; and The position of the digital representation of the virtual content relative to the identified object is determined based on the determined parameters by determining the position of the digital representation of the virtual content relative to the identified object based on the priorities assigned to the determined parameters.

16. The user equipment according to claim 15, wherein: The processor is configured to: The parameters for the identified object are determined by: determining at least one or more of a fixed position parameter, a size parameter, or a line of sight parameter; and At least one or more of a field of view parameter, a separation parameter, an interpersonal distance parameter, an altitude parameter, and a visibility parameter are determined.

17. The user equipment according to claim 16, wherein: The processor is configured to: The priorities are assigned to the determined parameters in the following manner: assigning a first priority to the fixed position parameter, the size parameter, or the line of sight parameter; and A second priority is assigned to the field of view parameter, the interval parameter, the interpersonal distance parameter, the height parameter, or the visibility parameter, wherein the first priority is a higher priority than the second priority.

18. The user equipment according to claim 14, wherein: The processor is configured to use one or more coordinated display properties to adjust a size of the digital representation of the virtual content based on a size of a participant physically present in the environment.

19. The user equipment according to claim 14, wherein: The processor is configured to render the digital representation of the virtual content using one or more coordinated display attributes such that the virtual content appears to be located between a first co-located participant and a second co-located participant.

20. The user equipment according to claim 14, wherein: The processor is configured such that the virtual content is a remote participant.

21. The user equipment according to claim 20, wherein: The processor is configured to render the digital representation of the remote participant using the one or more coordinated display attributes such that the remote participant appears to be anchored to the empty seat.

22. The user equipment according to claim 14, wherein: The processor is configured to: capturing an updated image of the environment; analyzing the captured updated image to determine whether the number, position, pose, or size of the identified objects has changed; In response to determining that the position, posture, or size of the identified object has changed, determining updated parameters for the identified object; determining whether a difference between the determined parameter and the determined updated parameter exceeds a threshold; as well as In response to determining that a difference between the determined parameter and the determined updated parameter exceeds a threshold, an updated display property of the digital representation of the virtual content is determined.

23. The user equipment according to claim 14, wherein: The processor is configured to determine the position of the digital representation of the virtual content based on the determined parameters by determining at least one position of each digital representation of each of a plurality of remote participants.

24. The user equipment according to claim 14, further comprising: a projector configured to project an image onto the optical lens, Wherein, the processor is coupled to the projector.

25. The user equipment according to claim 14, wherein: The user device is a smartphone.

26. The user equipment according to claim 14, wherein: The virtual content is related to the game.

27. A non-transitory computer-readable storage medium having processor-executable instructions stored thereon, the processor-executable instructions being configured to cause a processor in a device to perform the operations of any one of the methods of claims 1 to 13.

28. An apparatus for determining a display position of virtual content for an environment having a plurality of co-located participants, each co-located participant having a respective user device, the apparatus comprising means for performing the operations of any of the methods of claims 1 to 13.

Images