Placement of virtual content in environments with multiple physical participants

Abstract

The device may be configured to determine display properties for virtual content in an environment having multiple physical participants by capturing an image of the environment, analyzing the captured image to identify at least one object in the environment, determining parameters for the identified object, and determining display properties of a digital representation of virtual content based on the determined parameters. Embodiments may include negotiating display properties with other devices to generate coordinated display properties, and rendering a digital representation of virtual content so that a remote participant appears to be in the same fixed position for all juxtaposed participants and appears to be sized to match the juxtaposed participants.

Description

Technology Field

[0001] Claim of priority

[0002] This patent application claims priority to U.S. Regular Application No. 16 / 572,030, filed on September 16, 2019, under the title "Placement of Virtual Content in Environments with a Plurality of Physical Participants," which is assigned to the assignee of this application and is expressly incorporated herein by reference. Background Technology

[0003] Recently, software applications that render images and virtual content within an environment are growing in popularity and use across various applications, such as electronic gaming, virtual training systems, collaborative work environments, and advanced web-based conferencing and screen sharing applications. Additionally, advancements in mobile device technologies now enable mobile device users to run powerful software applications on their mobile devices, such as augmented reality software applications that combine real-world images from the user's physical environment with computer-generated images or virtual content. For example, augmented reality applications may add graphics, sounds, and / or haptic feedback to the natural world surrounding the user. Images, video streams, and information regarding people and / or objects may also be presented to the user on wearable electronic displays (e.g., smart glasses, augmented reality glasses, etc.), allowing the user to see and / or interact with representations of real-world people or objects.

[0004] Due to these trends, augmented reality applications are expected to allow users to cooperate and interact with other users in remote locations as if all participants were located in the same environment (e.g., the same room). For example, augmented reality applications may render remote participants as 3D representations or 2D videos or avatars, so that local and remote participants may interact and communicate in the same way as if all participants were in the same environment. means of solving the problem

[0005] Various embodiments of the present disclosure provide methods for determining display properties for virtual content in an environment having a plurality of participants, the methods may include, by a processor, capturing an image of the environment; analyzing the captured image to identify an object in the environment; determining parameters for the identified object; and determining display properties of a digital representation of virtual content based on the determined parameters. In some embodiments, the processor may be a processor of a smartphone among the plurality of participants.

[0006] Some embodiments may include assigning priorities to determined parameters. In some embodiments, determining display properties of a digital representation of virtual content based on determined parameters may include determining display properties of a digital representation of virtual content based on priorities assigned to determined parameters.

[0007] In some embodiments, determining parameters for an identified object may include determining at least one of a fixed position parameter, a size parameter, or a line-of-sight parameter, and determining at least one of a field-of-view parameter, a spacing parameter, a person-to-person distance parameter, a height parameter, and a visibility parameter. In some embodiments, assigning priorities to the determined parameters may include assigning a first priority to a fixed position parameter, a size parameter, or a line-of-sight parameter, and assigning a second priority to a field-of-view parameter, a spacing parameter, a person-to-person distance parameter, a height parameter, or a visibility parameter, wherein the first priority is a higher priority than the second priority.

[0008] Some embodiments may include identifying one or more other devices in the environment and transmitting determined parameters to one or more identified other devices. Some embodiments may include receiving additional parameters from at least one other device in the environment, wherein determining display properties of a digital representation of virtual content based on the determined parameters may include determining one or more display properties of a digital representation of virtual content based on the determined parameters and the received additional parameters.

[0009] In some embodiments, determining display properties of a digital representation of a remote participant based on determined parameters may include negotiating one or more display properties for rendering the digital representation of the remote participant with at least one other device in the environment to generate one or more coordinated display properties. Some embodiments may include using one or more coordinated display properties to render the digital representation of the remote participant such that all co-located participants physically present in the environment perceive the remote participant as being in the same fixed position in the environment. Some embodiments may include using one or more coordinated display properties to size the digital representation of the remote participant based on the size of the participants physically present in the environment. Some embodiments may include using one or more coordinated display properties to render the digital representation of the remote participant such that the remote participant appears to be positioned between a first participant physically present in the environment and a second participant physically present in the environment.

[0010] In some embodiments, using one or more adjusted display properties to render a digital representation of a remote participant so that the remote participant appears to be positioned between a first participant physically present in the environment and a second participant physically present in the environment may include using adjusted display properties to render a digital representation of a remote participant so that the remote participant appears to be approximately equidistant from the first participant physically present in the environment and the second participant physically present in the environment.

[0011] Some embodiments may include using one or more adjusted display properties to render a digital representation of a remote participant so that the remote participant appears to be anchored to a blank sheet. Some embodiments may include capturing an updated image of the environment; analyzing the captured updated image to determine whether the number, location, position, or size of an identified object has changed; determining an updated parameter for an identified object in response to determining that the location, position, or size of the identified object has changed; determining whether the difference between the determined parameter and the determined updated parameter exceeds a threshold; and determining an updated display property for the digital representation of the remote participant in response to determining that the difference between the determined parameter and the determined updated parameter exceeds a threshold.

[0012] In some embodiments, determining display properties of a digital representation of a remote participant based on determined parameters may include determining at least one display property for each of the respective digital representations of multiple remote participants. In some embodiments, capturing an image of the environment may include capturing an image of the environment by a processor associated with a device (e.g., a computing device, a mobile device, a head-mounted device, etc.). In some embodiments, determining parameters for an identified object may include determining at least one of a fixed position parameter, a size parameter, a line of sight parameter, a field of view parameter, a spacing parameter, a person-to-person distance parameter, a height parameter, or a visibility parameter. In some embodiments, the virtual content may be a remote participant. In some embodiments, the virtual content relates to a game. In some embodiments, the environment may be an augmented reality game environment.

[0013] Additional embodiments may include a computing device equipped with a processor comprising processor-executable instructions for performing operations of any of the methods summarized above, such as capturing an image of an environment, analyzing the captured image to identify at least one object in the environment, determining parameters for the identified object, and determining display properties of a digital representation of a remote participant based on the determined parameters. Additional embodiments may include a device comprising an image sensor, a projector configured to project images onto optical lenses, and a processor coupled to the image sensor and the projector, wherein the processor comprises processor-executable instructions for performing operations of any of the methods summarized above, such as capturing an image of an environment, analyzing the captured image to identify at least one object in the environment, determining parameters for the identified object, and determining display properties of a digital representation of a remote participant based on the determined parameters.

[0014] Further embodiments may include a non-transient processor-readable storage medium storing processor-executable instructions configured to enable a processor in the device and / or in an associated computing device to perform operations of any of the methods summarized above. Further embodiments may include a device and / or an associated computing device having various means for achieving the functions of the methods summarized above. Brief explanation of the drawing

[0015] The accompanying drawings, incorporated into and constituting a part of this specification, illustrate exemplary embodiments of various embodiments and provide for explaining the features of the claims together with the general description given above and the detailed description given below. FIG. 1a is an example of a head-mounted device (e.g., augmented reality glasses) that may be configured to implement various embodiments. FIG. 1b is a component block diagram of a smartphone suitable for implementing some embodiments. FIG. 1c is a system block diagram illustrating computer architecture and sensors that may be included in a device configured to render a digital representation of a remote participant in a virtual meeting according to embodiments. FIG. 2a is an example of meeting participants wearing devices that can be configured to render a digital representation of a remote participant in a virtual meeting according to embodiments. FIG. 2b is a block diagram illustrating that a digital representation of a remote participant may be rendered so that the remote participant is within the line of sight of the juxtaposed participants and does not block the line of sight of any of the remote or juxtaposed participants. FIGS. 3a and 3b are block diagrams illustrating devices configured to render a digital representation of a remote participant in a virtual meeting that includes multiple juxtaposed participants physically present in the same environment and seated around a game table in positions suitable for viewing a presentation. FIG. 4a is a block diagram illustrating that the device may render a digital representation of a remote meeting participant in a natural position relative to the game table, and thus each of the juxtaposed participants has a direct line of sight to the digital representation of the remote meeting participant. FIG. 4b is a block diagram illustrating that a device configured according to embodiments may exchange data with other devices present in the environment to relocate the digital representation of a remote meeting participant to a different location or position on a game table so that all juxtaposed participants have a direct line of sight to the digital representation of the remote meeting participant and to each other. FIGS. 5a to 5c are block diagrams illustrating that devices configured according to embodiments may intelligently determine the locations where digital representations of remote meeting participants are to be rendered for the locations or positions of juxtaposed participants physically existing in the environment. FIGS. 6 through 9 are process flow diagrams illustrating methods for determining display attributes for a digital representation of a remote participant according to some embodiments. FIG. 10 is a process flow diagram illustrating a method for updating display attributes for a digital representation of a remote participant according to one embodiment. FIG. 11 is a component diagram of an exemplary computing device suitable for use with various embodiments. Specific details for implementing the invention

[0016] Various embodiments will be described in detail with reference to the accompanying drawings. Wherever possible, the same reference numerals will be used throughout the drawings to refer to identical or similar parts. References made to specific examples and implementations are for illustrative purposes only and are not intended to limit the scope of the claims.

[0017] Generally, various embodiments include methods for determining display properties for virtual content in environments having multiple physical participants, and devices configured to implement the methods (e.g., computing devices, mobile devices, head-mounted devices, etc.). For example, a mobile device or a head-mounted device may be configured to capture an image of an environment (e.g., a game room, a meeting area, a play field, etc.), analyze the captured image to identify at least one object in the environment, determine parameters for the identified object, and determine display properties of a digital representation of a remote participant based on the determined parameters.

[0018] The term “mobile device” is used herein to generally refer to any one or all of smartphones, cellular phones, Internet of Things (IoT) devices, personal or mobile multimedia players, laptop computers, tablet computers, ultrabooks, palm-top computers, wireless email receivers, multimedia internet-enabled cellular phones, wireless gaming controllers, head-mounted devices, and similar electronic devices comprising a programmable processor, memory, and circuitry for transmitting and / or receiving wireless communication signals to and from wireless communication networks. While various embodiments are particularly useful in mobile devices such as smartphones and tablets, such embodiments are generally useful in any electronic device comprising communication circuitry for accessing cellular or wireless communication networks.

[0019] The terms “head-mounted device” and its abbreviation “HMD” may be used interchangeably herein to refer to any electronic display system that presents to users a combination of computer-generated images and real-world images from a user’s physical environment (i.e., what the user can see without glasses) and / or enables a wearer / user to view images generated in the context of a real-world scene. Examples include a helmet, glasses, virtual reality glasses, augmented reality glasses, electronic goggles, and other similar technologies / devices, may include or be included therein. In some embodiments, the head-mounted device may include a processor, memory, a display, a camera, and a wireless interface for connecting to the Internet, a network, or other computing devices.

[0020] In some embodiments, the head-mounted device may be an accessory to another computing device (e.g., a desktop, laptop, smartphone, tablet computer, etc.) or may receive video from another computing device, and all or part of the processing is performed on the processor of the other computing device (e.g., computing devices exemplified in FIG. 1b and FIG. 11, etc.). As such, in various embodiments, the head-mounted device may be configured to perform all processing locally on the processor in the head-mounted device, offload all main processing to a processor in another computing device (e.g., a laptop in a game room, etc.), or split the main processing operations between the processor in the head-mounted device and the processor in the other computing device. Additionally, in situations where there are multiple head-mounted devices and / or multiple computing devices (e.g., a mix of multiple smartphones, tablets, and laptop computers) (e.g., in a meeting with multiple participants), the main processing operations may be performed in a distributed manner in which the head-mounted devices and / or computing devices may exchange data and output results via wireless communications (e.g., Bluetooth, WiFi, LTE-Direct, etc.).

[0021] The term “environment” is used herein to generally refer to a visible context that may be seen by participants both in reality and virtually. The environment may include the physical location of local participants. Non-limiting examples of physical environments include game rooms, control rooms, classrooms, training facilities, play fields (e.g., for games), etc. In some embodiments, the environment may include a virtual location, such as an electronic game domain, where all participants are seen virtually. For example, the environment may be a virtual space where participants interact in an augmented reality board game.

[0022] The terms “remote participant” and “virtual content” are used herein to refer to a person or a digital representation of a person who is projected to be visible in the same virtual environment and interacts with other participants, but is not physically present in the same physical environment as the other participants (e.g., the same play field, meeting room, etc.). The term “juxtaposed participant” is used herein to refer to a person (or a device held or worn by that person) who is physically present in the same environment as at least one other participant. In the following descriptions and claims, “virtual content,” “remote participant,” and “the remote participant” may be referred to, but such references are for ease of description and the various embodiments are not limited to rendering a virtual presence for a single participant. Rather, the methods and operations of the various embodiments may be used to position and render any number of remote participants. Similarly, the various embodiments may operate with any number of juxtaposed participants. Additionally, the various embodiments may be implemented in applications where all participants are remote participants, such as in virtual gaming applications. Additionally, "remote participant" may be an image of a virtual participant, such as an avatar, or a computer-generated image of a person performing a role in a virtual reality game. To encompass images of various types of individuals that do not physically exist, the term "virtual content" may be used to refer to virtual participants in the descriptions and claims.

[0023] People are increasingly using their devices (e.g., smartphones, mobile devices, head-mounted devices, etc.) to perform or interact with others in augmented reality applications, such as augmented reality games, augmented reality training, and virtual meetings. There is a popular use of augmented reality games that can be played on participants' smartphones, representing personal images that appear to be presented in images on the smartphone—whether in a real appearance or a virtual appearance (e.g., an avatar)—even though they are not physically present (e.g., as an image through a smartphone camera). Furthermore, augmented reality systems are revolutionizing the training of operators of sophisticated equipment in complex control rooms, war rooms, aircraft cockpits, and robotic surgical environments, and enabling local and remote individuals to participate in the same training sessions can dramatically improve team training efficiency and cost-effectiveness. Additionally, because augmented reality systems can be used to present images of all participants as if they were in the same environment, all participants in a meeting no longer need to be physically present to interact with other participants, as if everyone were in the same game room.

[0024] Smartphones, modern virtual reality displays, and image capture devices can productively improve and facilitate interactions between local and remote participants in various types of group engagements (e.g., augmented reality games, group chats, meetings, group training, lectures, etc.) by rendering digital representations of remote participants on virtual reality displays so that the remote participants appear to be in the same environment as the user / wearer of the user device. However, for virtual group engagements (e.g., augmented reality games, meetings, etc.) in which two or more juxtaposed participants are physically present in the same environment (e.g., sitting in front of the same physical game board, on a play field, in the same room, area, control center, etc.), any improvements in productivity or user-interaction provided by the user device (e.g., a smartphone) may be nullified unless the user device intelligently selects the location and / or position where digital representations of virtual content, such as those of the remote participants, are rendered. For example, if the placement of a remote participant's digital representation does not match on the respective user devices (e.g., smartphones) of the juxtaposed participants, each juxtaposed participant may face or look in different directions when talking to or listening to virtual content such as the remote participant. This can disrupt, distract, impede, and / or otherwise interfere with natural human-to-human communication between the participants.

[0025] Generally, to facilitate natural human-to-human communication, it may be advantageous for user devices to place virtual content (e.g., digital representations of remote participants) at the same fixed position from the perspective of all participants, including juxtaposed participants, when some participants are physically present in the same environment. For example, if a digital representation of virtual content (e.g., a remote participant) is placed at a specific position on a game table by one user device, all other user devices of the juxtaposed participants may render the digital representation of the remote participant at that same position. Additionally, it may be advantageous for user devices to intelligently size the digital representation of the remote participant so that the remote participant does not appear too large, too small, or otherwise unnatural to the juxtaposed participants. For example, the digital representation of virtual content (e.g., a remote participant) may be sized on smartphone displays to appear approximately the same size as the juxtaposed participants. Additionally, it may be advantageous for user devices to render the digital representation of a remote participant such that the remote participant is positioned at an appropriate height relative to the juxtaposed participants, is within the field of view of the juxtaposed participants, does not block the line of sight of other participants, is appropriately spaced (e.g., all participants appearing to be roughly equidistant), explains the distance between people (e.g., is not rendered so close to other participants that it feels as though the person is encroaching on personal space boundaries), and otherwise is easily visible (e.g., is not placed in front of a bright or complex background).

[0026] Various embodiments include user devices configured to facilitate natural human-to-human communication between local and remote participants and virtual content (e.g., eye contact, normal exchange, etc.) by intelligently determining locations, positions, orientations, sizes, and other display characteristics of virtual content, such as digital representations of remote participants, to facilitate natural human-to-human communication (e.g., eye contact, normal exchange, etc.), and / or augmented reality software applications configured to function on user devices (e.g., smartphones).

[0027] A device processor (e.g., a processor in a smartphone, a head-mounted device worn by a juxtaposed participant, a mobile device communically coupled to a head-mounted device, etc.) may be configured to cooperate with other device processors (e.g., smartphones, head-mounted devices worn by other juxtaposed participants, etc.) to intelligently determine display attributes of a digital representation of virtual content, such as a remote participant. For example, the device processor may be configured to determine the environment by rendering, for instance, a virtual environment (e.g., an augmented reality game board, a virtual game environment, etc.) or by capturing images of a real environment (e.g., a game table, a play field, a conference room, a control room, a training environment, etc.) through a camera of a user device. For real environments, the device processor may analyze the captured images to identify people and objects within the environment (e.g., a desk, a table, a chair, a wall, etc.) and may capture or produce data (e.g., through a sensor array or processor). For virtual environments, the device processor may determine the shapes and locations of various virtual objects within the environment. Based on captured / calculated data, the device processor may calculate various parameters for each of the identified objects and transmit the determined parameters to other device processors (e.g., transmitting data to other participants' smartphones via wireless data links).

[0028] In some embodiments, the device processor may also be configured to use localization and mapping techniques, such as simultaneous localization and mapping (SLAM), visual simultaneous localization and mapping (VSLAM), and / or other techniques known in the art, to generate a grid map of the scene. The device processor may use the grid map to determine the position of virtual content, such as a remote participant, and / or may share the grid map with other devices in the environment so that all device processors know where the virtual content should be placed in the environment.

[0029] Parameters calculated and communicated by the device processor may include fixed position parameters, size parameters, line of sight parameters, field of view parameters, spacing parameters, distance between people parameters, height parameters, and / or visibility parameters. Each of these parameters may be associated with one or more weights indicating the importance of the parameter. For example, to accommodate virtual content and multiplayer games with multiple remote participants, the device processor may increase and / or decrease the weights associated with fixed position parameters or decrease the weights associated with size parameters. The device processor may also prioritize parameters, giving higher priority to fixed position, size, and line of sight parameters than to field of view, spacing, distance between people, height, and visibility parameters.

[0030] Each device processor receives parameters calculated by other device processors and may use the received parameters along with the parameters calculated on the device to adjust or negotiate display properties for rendering digital representations of virtual content, such as images of remote participants.

[0031] Each device processor may use adjusted display properties to render digital representations of virtual content, such as images of remote participants, so that all juxtaposed participants (i.e., people physically present in the same physical environment) perceive the virtual content as being in the same location in the environment (e.g., game board, game room, game scene, conference room, control room, war room, equipment, etc.), as being anchored to the same physical object (e.g., game piece, empty chair, console, etc.), and / or otherwise as being in a fixed position relative to images of the environment.

[0032] Device processors may be configured to use display properties adjusted to intelligently size digital representations, such as images of remote participants, so that virtual content is placed at an appropriate height for the juxtaposed participants, is within the field of view of the juxtaposed participants, does not block the line of sight of other participants, is appropriately spaced (e.g., all participants appear to be roughly equidistant), accounts for the distance between people (e.g., is not rendered so close to other participants that it feels like a person is encroaching on the boundaries of personal space), and otherwise can be easily seen (e.g., is not placed in front of a bright or complex background), and so that the virtual content does not appear unnatural to the participants when viewed through the participants' devices (e.g., smartphones).

[0033] In some embodiments, device processors may be configured to replicate real-world scenarios by anchoring a digital representation of virtual content to an empty chair, for example, within a game scene, behind a physical or virtual console, etc. In one embodiment, the device processor may be configured to select an empty chair (or a position for a game table, equipment, etc.) that the individual might have selected if they were physically present in the environment (e.g., a chair not far from other participants in the game, an empty console in a control room, etc.) (when the virtual content is an image of a remote participant).

[0034] In some embodiments, device processors may be configured to detect changes in the number, location, position, size, etc. of objects, virtual content, or people in an environment, determine whether the detected changes exceed a threshold, and dynamically recalculate parameters or weights to describe the detected changes in response to the determination that the detected changes exceed the threshold. For example, a device process (e.g., a processor in a participant's smartphone) may detect that a new participant (virtual, remote, or juxtaposed) has joined a game, meeting, training session, etc. after it has started, recalculate parameters and / or weights to describe the new participant, and update the renderings of virtual content (e.g., remote participants) based on the recalculated parameters and / or weights. As another example, the device process may detect that a participant (virtual, remote, or juxtaposed) has moved positions (e.g., moving laterally by changing seats, moving vertically by standing up or sitting, etc.) after a game, meeting, or training session has started, determine whether the distance between the participant's current position and previous position exceeds a threshold, recalculate parameters in response to the determination that the distance between the participant's current position and previous position exceeds the threshold, and / or update the renderings of the remote participants.

[0035] In some embodiments, device processors may be configured to calculate additional values ​​and perform additional actions to account for special considerations related to updating renderings of remote participants or repositioning the placement of remote participants after a game, meeting, or training session has started.

[0036] In some embodiments, the device processor may be configured to adjust renderings of virtual content, such as images of remote participants, to support social conventions and facilitate human-to-human communication (e.g., by resizing and respacing images of virtual content). For example, the device processor may be configured to determine the positioning of virtual content, such as images of virtual or remote participants, similar to how people typically join juxtaposed games, meetings, training sessions, etc. In fully juxtaposed games and meetings, it is common for a latecomer to find a seat or standing position near the rest of the group. It is also common for a latecomer to choose a seat or standing position at approximately the same distance from each of the other participants (e.g., people generally would not sit right next to each other if there were other open seats spaced more equally apart). People do this partly naturally to improve eye contact with the maximum number of participants. A latecomer may also join the group composition, and other participants may slightly adjust their positions to make space for the latecomer. To replicate these human behavioral tendencies, the device processor may be configured to automatically adjust the positions / renderings of other virtual content to maintain good spacing and / or to account for other considerations discussed above (e.g., line of sight, etc.).

[0037] In some embodiments, the device processor (e.g., the processor of a smartphone) may be configured to set thresholds high enough so that renderings are not adjusted when certain changes are detected, such as when a single participant is standing. The device processor may also set thresholds low enough so that renderings are adjusted when other changes are detected, such as when a majority of juxtaposed people are standing or sitting.

[0038] In some embodiments, the device processor may be configured to communicate user changes to other device processors for consistent rendering. For example, if a user manually adjusts the position where a digital representation of virtual content, such as a remote participant, is rendered on their smartphone, the device processor may transmit the adjusted position of the rendering to other device processors, such as processors on the smartphones of other people involved in the same game, meeting, training, etc. In response, the other device processors may also adjust the digital representation of the same virtual content so that the configuration of the virtual content for the environment remains consistent for all users.

[0039] FIG. 1a illustrates a user device in the form of a head-mounted device (100) that may be configured to intelligently determine display characteristics of a virtual object (e.g., a digital representation of a remote participant) according to various embodiments. In the example illustrated in FIG. 1a, the head-mounted device (100) comprises a frame (102), two optical lenses (106), and an image projector (108), a monocular image sensor or cameras (110), a sensor array (112), a memory (116), and a processor (114) communicably coupled to a communication circuit (118). The projector (108) may be embedded in the arm portions (104) of the frame (102) and may be configured to project images onto the optical lenses (106). The projector (108) may include a light-emitting diode (LED) module, an optical tunnel, a homogenizing lens, an optical display, a fold mirror, and other components widely known in projectors or head-mounted displays of the prior art. In some embodiments, two optical lenses (106) may include image forming elements configured to superimpose an image onto a view through the lenses, in which case the projector (108) may not be implemented.

[0040] In some embodiments, the head-mounted device (100) and / or processor (114) may be configured to employ SLAM techniques to build and update a map of an unknown environment while simultaneously maintaining tracking of the position of the head-mounted device (100) within that environment. For example, the head-mounted device (100) may be equipped with a monocular image sensor that captures images or frames from that environment. The head-mounted device (100) may identify protruding objects (e.g., juxtaposed participants) or features (e.g., faces and / or body features of juxtaposed participants) within the captured image, estimate the dimensions and scale of the features in the image, compare the identified features with each other and / or features in test images having known dimensions and scales, and identify correspondences based on the comparisons. Each correspondence may be a set of values ​​or an information structure that identifies a feature (or feature point) in one image as having a high probability of being the same feature in another image. In other words, a correspondence may be a set of corresponding image points (e.g., a first point in a first image and a second point in a second image, etc.). The head-mounted device (100) may generate a homography matrix information structure based on the identified correspondences and may use the homography matrix to determine its pose (e.g., position, orientation, etc.) within the environment.

[0041] Various embodiments may be implemented as smartphones, one example of which is illustrated in FIG. 1b. A smartphone (120) may include a processor (121) coupled to an internal memory (122), a display (123), and a speaker (124). Additionally, the smartphone (120) may include a cellular phone transceiver (126) coupled to the processor (121) and / or an antenna (125) for transmitting and receiving electromagnetic radiation that may be connected to a wireless data link. Smartphones (120) typically also include menu selection buttons or rocker switches (127) for receiving user inputs.

[0042] A conventional smartphone (120) also includes a sound encoding / decoding (codec) circuit (128) that digitizes sound received from a microphone into data packets suitable for wireless transmission and decodes the received sound data packets to generate analog signals provided to a speaker to produce sound. Additionally, one or more of the processor (121), wireless transceiver (126), and codec (128) may include a digital signal processor (DSP) circuit (not separately shown).

[0043] FIG. 1c illustrates a computer architecture and various sensors that may be included in a head-mounted device (100) or smartphone (120) configured to employ SLAM or other techniques to render a digital representation of a remote participant in a virtual meeting according to various embodiments. In the example illustrated in FIG. 1c, the head-mounted device (100) includes a main board component (152), an image sensor (154), a microcontroller unit (MCU) (156), an infrared (IR) sensor (158), an inertial measurement unit (IMU) (160), a laser distance sensor (LDS) (162), and an optical flow sensor (164).

[0044] Sensors (154, 158-164) may collect information useful for employing SLAM techniques in a head-mounted device (100) or a smartphone (120). For example, an optical flow sensor (164) may measure optical flow or visual motion and output measurements based on optical flow / visual motion. Optical flow may identify or define patterns of apparent motion of objects, surfaces, and edges in a visual scene caused by relative motion between an observer (e.g., head-mounted device (100), smartphone (120), wearer / user, etc.) and the scene. Optical flow information may be used by an MCU (156) to measure visual motion or relative motion between the head-mounted device (100), smartphone (120), and other objects in the vicinity of the head-mounted device (100), smartphone (120), etc. Additionally, the head-mounted device (100) or smartphone (120) may use optical flow information for motion detection, object segmentation, time-to-contact information, focus of extended calculations, motion-compensated encoding, stereo disparity measurements, and other similar information. In some embodiments, the optical flow sensor (164) may be an image sensor coupled to an MCU (156) (or processor (114, 121)) programmed to execute an optical flow algorithm. In some embodiments, the optical flow sensor (164) may be a vision chip comprising an image sensor and a processor on the same chip or die.

[0045] A head-mounted device (100) or a smartphone (120) may be equipped with various additional sensors, including a gyroscope, accelerometers, magnetometer, magnetic compass, altimeter, camera, optical reader, orientation sensor, monocular image sensor, and / or similar sensors for monitoring physical conditions (e.g., position, motion, acceleration, orientation, altitude, etc.) or for collecting information useful for employing SLAM techniques.

[0046] In the following descriptions of various embodiments, the drawings and descriptions thereof refer to exemplary implementations of various embodiments in which local and remote participants are meeting around a game table, such as a table presented in an augmented reality game, with actual and virtual game boards. These examples are for illustrative purposes only and are not intended to be limiting. Various embodiments are equally applicable to other applications in which virtual reality is used to enable local and remote participants, as well as virtual content, to interact as if they were physically located in the same environment, such as games involving a common game environment (e.g., game board, game scene, etc.), conference tables for meetings, equipment control panels for training sessions involving equipment that may be physically present or virtually rendered, training sessions between teams of decision-makers, meetings in control rooms or war rooms, etc.

[0047] FIG. 2a illustrates that head-mounted devices (100a, 100b) configured according to various embodiments may be used to play games with virtual content such as a remote participant (202). A device processor (e.g., a processor (121) in the participant's smartphone (200a), a processor (114) in the head-mounted device (100a), etc.) may be configured to cooperate with other device processors (e.g., a processor (121) in the participant's smartphone (200b), a processor (114) in the head-mounted device (100b)) to intelligently determine display attributes of a digital representation of virtual content such as a remote participant (202). For example, the device processor may be configured to capture an image of an environment (e.g., a game room) through a camera of a smartphone or head-mounted device, analyze the captured image to identify people and objects within the environment (e.g., a game table (206), a game board (207), juxtaposed participants (204a and 204b), etc.), capture or produce data (e.g., through cameras and other sensors of the smartphone (200a), through the sensor array (112) or processor (114) of the head-mounted device (100a), etc.), produce various parameters for each of the identified objects based on the captured / produced data, and transmit the produced / determined parameters to other device processors (e.g., a processor (121) in another smartphone (200b)). Device processors may determine various display properties so that virtual content is placed on a game table (206) and appears to each juxtaposed participant (204a, 204b) as being within a fixed area (208) for, for example, a real or virtual game board (207), and may use parameters to render a digital representation of virtual content such as a remote participant (202).

[0048] FIG. 2a also illustrates that smartphones (200a, 200b) and / or head-mounted devices (100a, 100b) may be configured to size a digital representation of virtual content, such as a remote participant (202), so that the remote participant does not appear too large, too small, or otherwise unnatural to the juxtaposed participants (204a, 204b). Smartphones (200a, 200b) and / or head-mounted devices (100a, 100b) may also render a digital representation of virtual content, such as a remote participant (202), so that the virtual content appears to be placed at an appropriate height (210) relative to the juxtaposed participants (204a, 204b), is appropriately spaced (e.g., all participants appear to be roughly equidistant), explains the distance between people (e.g., is not rendered so close to other participants that it feels like a person is encroaching on the boundaries of personal space), and otherwise is easily visible (e.g., is not placed in front of a bright or complex background).

[0049] FIG. 2b illustrates that a digital representation of virtual content, such as that of a remote participant (202), may be rendered so that the virtual content is within the field of view of the juxtaposed participants (204a, 204b) through their smartphone displays or head-mounted devices, and does not block the line of sight of any of the remote or juxtaposed participants.

[0050] FIGS. 3a and 3b illustrate multiple meeting participants placed in the same environment (e.g., a game room) and sitting around a game table (206) and wearing head-mounted devices (100a-100d).

[0051] Referring to FIG. 3a, the juxtaposed participants (304a-304d) have selected their sheets and / or arranged themselves around three sides of the game table (206) so that they can communicate with each other and view the game board, whiteboard, TV, monitor, or presentation (302). Each of the juxtaposed participants (304a-304d) is in a slightly different physical position relative to the environment and the game table (206), and there is a person-sized gap (306) between two juxtaposed participants (304b) and juxtaposed participant (304c).

[0052] Referring to FIG. 3b, smartphones (200a, 200b) and / or head-mounted devices (100a-100d) may be configured to communicate with each other to cooperatively determine whether a digital representation of virtual content, such as a remote participant (202), should be rendered in a person-sized gap (306). As part of these operations, the smartphones (200a, 200b) and / or head-mounted devices (100a-100d) may analyze, calculate, or explain various parameters (and their individual weights), including fixed position parameters, size parameters, line of sight parameters, field of view parameters, spacing parameters, person-to-person distance parameters, height parameters, and visibility parameters.

[0053] The fixed position parameter may indicate that a digital representation of virtual content, such as a remote participant (202), must be placed in the same fixed position for all juxtaposed participants (304a-304d). For example, if a digital representation of virtual content, such as a remote participant (202), is placed in front of a specific position on the game table (206), all juxtaposed participants may view the virtual content within that same position or fixed area. The weight value associated with the fixed position parameter may indicate the weight or importance of rendering the digital representation of virtual content in the fixed position relative to other parameters.

[0054] Size parameter(s) may indicate that the rendered size of the digital representation of virtual content, such as the remote participant (202), must be consistent for all juxtaposed participants (304a-304d) and appropriate for the position in which it is rendered (e.g., natural and expected size). For example, the size parameter may indicate that the virtual content, which is the body of the remote participant, must be rendered in approximately the same size as the bodies of the juxtaposed participants (304a-304d). To achieve this, one or more of the smartphones (200a, 200b) and / or head-mounted displays (100a-100d) may be configured to measure the sizes of the heads of the juxtaposed participants (304a-304d) and / or the distances between them, generate averages, and use this information to determine the render size of the digital representation of virtual content, such as the remote participant (202). The weight value associated with each size parameter may indicate the weight or importance of rendering a digital representation of virtual content, such as a remote participant (202), consistent for all juxtaposed participants and / or approximately the same size as the juxtaposed participants (304a-304d).

[0055] The line of sight parameter(s) may indicate that a digital representation of virtual content, such as a remote participant (202), should not be rendered in a position that blocks the line of sight of any of the juxtaposed participants (304a-304d), which may be determined by smartphones (200a, 200b) and / or head-mounted devices (100a-100d) that identify areas that block part or all of the other participant's view. In some embodiments, one or more of the smartphones (200a, 200b) and / or head-mounted devices (100a-100d) may be configured to measure the areas and / or lines of sight extensively so that a digital representation of virtual content, such as a remote participant (202), is not placed too close to an adjacent participant (e.g., juxtaposed participant (304b) or juxtaposed participant (304c)). The weight value associated with each line of sight parameter may indicate the weight or importance of rendering a digital representation of virtual content, such as a remote participant (202), so as not to block the line of sight of any of the juxtaposed participants (304a-304d).

[0056] The field of view parameter(s) may indicate that the digital representation of virtual content, such as a remote participant (202), should be rendered in such a way that it fits within the field of view of as many parallel participants (304a-304d) as possible. The weight value associated with each field of view parameter may indicate the weight or importance of rendering the digital representation of virtual content, such as a remote participant (202), so that it fits within the field of view of other parallel participants (304a-304d). Since smartphones (200a, 200b) and / or head-mounted displays (100a-d) may have a narrow field of view, it may be difficult to render virtual content, such as a remote person, within the field of view of all parallel participants (304a-304d). In this case, one or more of the field of view parameter(s) may be reduced to account for any difficulties associated with rendering digital representations of virtual content, such as the characteristics of individual smartphones (200a, 200b) and / or head-mounted devices (100a-100d) and / or remote participants (202), within the field of view of all juxtaposed participants (304a-304d).

[0057] Spacing parameter(s) may indicate that a digital representation of virtual content, such as a remote participant (202), should be placed at an equidistant distance from the juxtaposed participants (304a-304d), similar to how people typically organize themselves in a face-to-face game setting. For example, a person would not typically sit too close to another person, nor would they sit too far from the rest of the group. To illustrate this, smartphones (200a, 200b) and / or head-mounted displays (100a-100d) may measure the distances between each of the other participants, describe obstacles and objects in the environment (e.g., presentation (302)), and render a digital representation of the remote participant (202) on the acceptable side (e.g., the side not containing the presentation (302)) that has the most available space. Smartphones (200a, 200b) and / or head-mounted displays (100a-100d) may take measurements, calculate an average score, use the average score to determine the longest side, and place a digital representation of the remote participant (202) on the longest side. This process may be repeated for subsequent remote participants, taking into account previously positioned virtual content such as the remote participants. Weight values ​​associated with each spacing parameter may indicate the weight or importance of rendering a digital representation of virtual content such as the remote participant (202) at an equidistant distance from the juxtaposed participants (304a-304d).

[0058] The person-to-person distance parameter(s) may indicate that a digital representation of virtual content, such as a remote participant (202), should not be rendered so close to another participant (e.g., a juxtaposed participant (304b)) that it feels as though the remote participant is encroaching on the boundaries of their personal space. Since such person-to-person distance may vary by culture, smartphones (200a, 200b) and / or head-mounted displays (100a-d) may be configured to determine the distance based on pre-configured values, distances and behaviors between juxtaposed participants, artificial intelligence, or other similar parameters. A weight value associated with the person-to-person distance parameter may indicate the weight or importance of rendering a digital representation of virtual content, such as the remote participant (202), so that it does not appear as though the remote participant is encroaching on the boundaries of the personal space of the juxtaposed participants (304a-304d).

[0059] Height parameter(s) may indicate that a digital representation of virtual content, such as a remote participant (202), should be placed at an appropriate height given the locations, positions, and heights of the juxtaposed participants (304a-304d). This may involve a sitting or standing height depending on the locations, positions, and heights of the juxtaposed participants (304a-304d). In some embodiments, the smartphones (200a, 200b) and / or head-mounted devices (100a-d) may be configured to determine the height as the average head or eye height of each juxtaposed participant. In some embodiments, the smartphones (200a, 200b) and / or head-mounted devices (100a-d) may be configured to determine whether the majority of the juxtaposed participants (304a-304d) are sitting or standing, and to take the average of either. For example, if three parallel participants are sitting and two parallel participants are standing, smartphones (200a, 200b) and / or head-mounted devices (100a-d) may take the average eye level of the three sitting participants and place a digital representation of virtual content, such as a remote participant (202), at that level. The weight value associated with each height parameter may indicate the weight or importance of rendering a digital representation of virtual content, such as a remote participant (202), at an appropriate height for other parameters.

[0060] Visibility parameter(s) may indicate that the digital representation of virtual content, such as a remote participant (202), should be positioned to be optimized for visibility by avoiding bright or complex backgrounds or anchor objects that might obstruct the visibility of the digital representation of virtual content, such as a remote participant (202), or interfere with the view of the juxtaposed participants (304a-304d). The weight value associated with each visibility parameter may indicate the weight or importance of rendering the digital representation to be optimized for visibility.

[0061] Returning to the example illustrated in FIG. 3b, smartphones (200a, 200b) and / or head-mounted devices (100a-d) may determine that a digital representation of virtual content, such as a remote participant (202), should be rendered in a person-sized gap (306) by capturing and analyzing images of the environment to identify people and objects in the environment, including game participants, a game table (206), and / or a presentation (302). The head-mounted devices (100a-d) may also calculate various parameters (e.g., depth, distance, orientation, etc.) for the identified objects / people in the environment. Smartphones (200a, 200b) and / or head-mounted devices (100a-100d) may use calculated / analyzed parameters to determine whether a digital representation of a remote participant (202) should be rendered in a human-sized gap (306). That is, smartphones (200a, 200b) and / or head-mounted devices (100a-100d) may, based on an analysis of one or more of the parameters and their individual weights, determine that positioning a digital representation of virtual content such as a remote participant (202) in a human-sized gap (306) will cause the juxtaposed participants (304a-304d) to continue watching the presentation (302) and communicate with each other.

[0062] In response to determining that a digital representation of virtual content, such as a remote participant (202), should be rendered in a human-sized gap (306), smartphones (200a, 200b) and / or head-mounted devices (100a-100d) may search captured images of the environment to identify a surface or object (e.g., an empty chair) inside or near the human-sized gap (306), select the identified surface / object for use as an anchor, and determine display attributes (e.g., size, etc.) for the digital representation of virtual content, such as the remote participant (202), based on the relative distances between the selected surface / object and the participants (304a-304d) juxtaposed with it and the calculated / analyzed parameters. Smartphones (200a, 200b) and / or head-mounted devices (100a-100d) may render a digital representation of virtual content, such as a remote participant (202), so that the virtual content is visible to all juxtaposed participants in a fixed position (e.g., anchored to an anchor surface / object) and / or so that the juxtaposed participants can easily and naturally communicate and interact with each other and with the virtual content.

[0063] FIG. 4a illustrates an example in which a digital representation of virtual content, such as a remote participant (202), is placed in a natural position relative to the game table (206), and each of the juxtaposed participants (204a-204c) has a direct line of sight (indicated by dotted arrows) to the digital representation of virtual content. However, the position of the digital representation of virtual content, such as the remote participant (202), blocks the line of sight of the juxtaposed participants (204b and 204c). This may occur, for example, when a juxtaposed participant (204c) joins the game after it is in progress.

[0064] FIG. 4b illustrates that participant smartphones and / or head-mounted devices configured according to an embodiment may cooperate to determine that the line of sight between the parallel participants (204b and 204c) is blocked, and to reposition the digital representation of the virtual content, such as the remote participant (202), to a different location or position relative to the game table (206) so that all the parallel participants (204a-204c) have a direct line of sight to each other (indicated by solid arrows) and a direct line of sight to the digital representation of the virtual content, such as the remote participant (202) (indicated by dotted arrows).

[0065] FIGS. 5a through 5c illustrate that participant smartphones and / or head-mounted devices configured according to embodiments may intelligently determine locations where digital representations of virtual content, such as remote participants (202a, 202b), are to be rendered for locations or positions of juxtaposed participants (204a-204c). FIG. 5a illustrates that participant smartphones and / or head-mounted devices may be configured to calculate distances between juxtaposed participants (204a-204c) and determine that the greatest distance (L) between participants is between juxtaposed participant (204a) and juxtaposed participant (204c).

[0066] Referring to FIG. 5b, the participant smartphones and / or head-mounted devices may render a digital representation of virtual content, such as a first remote participant (202a), between the juxtaposed participant (204a) and the juxtaposed participant (204c). The smartphones and / or head-mounted devices may recalculate the distances between the juxtaposed participants (204a-204c) and the digital representation of virtual content, such as the remote participant (202a), and determine that the greatest distance (L) between the participants is now between the juxtaposed participant (204a) and the juxtaposed participant (204b).

[0067] FIG. 5c illustrates that participant smartphones and / or head-mounted devices may render a digital representation of virtual content, such as a second remote participant (202b), between a parallel participant (204a) and a parallel participant (204b). The participant smartphones and / or head-mounted devices may recalculate the distances between digital representations of virtual content, such as the parallel participants (204a-204c) and remote participants (202a, 202b), and determine that the greatest distance (L) between the participants is now between the parallel participant (204b) and the parallel participant (204c). The smartphones and / or head-mounted devices may repeat these operations until all remote participants are rendered in the environment.

[0068] FIGS. 6 through 9 illustrate methods (600, 700, 800, 900) for determining display properties for a digital representation of virtual content, such as a remote participant, according to various embodiments (e.g., any or all of the embodiments discussed above with reference to FIGS. 1 through 5). All or part of the methods (600, 700, 800, 900) may be performed by a device processor, such as a processor in a head-mounted device worn by the parallel participant (e.g., the processor (114) exemplified in FIG. 1a), a processor in a smartphone independently or communically coupled to the head-mounted device worn by the parallel participant (e.g., the processor (121) exemplified in FIG. 1b), a processor in a server computing device communically coupled to the smartphone or head-mounted device, or a processor associated with a cloud network or a cloud-based computing system.

[0069] Referring to FIG. 6, in block 602 of method (600), the device processor may capture, generate, or receive images of the environment. For example, a camera on a smartphone or head-mounted device may capture video of the environment and provide image frames to the device processor. As another example, images from the environment may be captured by one or more cameras positioned within the environment and communicating with the device processor.

[0070] In Block 604, the device processor may analyze the captured image to identify at least one object (e.g., a person, a game table, an empty sheet, etc.) in the environment. In some embodiments, as part of the operations in Block 604, the device processor may use localization and mapping techniques such as machine vision, VSLAM, or other techniques and methods known in the art, generate a grid or map of the environment, identify objects, determine the location, orientation, and / or position of objects in the environment, and / or perform other similar operations.

[0071] In block 606, the device processor may determine parameters for the identified object. For example, in block 606, the device processor may determine any or all of a fixed position parameter, a size parameter, a line of sight parameter, a field of view parameter, a spacing parameter, a person-to-person distance parameter, a height parameter, and / or a visibility parameter. In some embodiments, each of the determined parameters may be associated with one or more weights or priorities indicating individual importance.

[0072] In some embodiments, the device processor may be configured to adjust priorities or weights associated with parameters based on user inputs and / or to account for various conditions, events, or preferences. For example, the device processor may determine that rendering a digital representation of virtual content, such as a remote participant, at different positions for different juxtaposed participants is more distracting or disruptive to human-to-human communications and interactions than rendering a digital representation of virtual content, such as a remote participant, that is much smaller than the sizes of the juxtaposed participants physically present in the environment. Thus, to accommodate a game or meeting with multiple remote participants, the device processor may increase weights associated with fixed position parameters or decrease weights associated with size parameters. As another example, the device processor may determine that rendering a digital representation of virtual content, such as a remote participant, at different positions relative to different juxtaposed participants distracts or hinders human-to-human communications and interactions more than rendering a digital representation of virtual content, such as a remote participant, that is too close to juxtaposed participants physically present in the environment. In response, the device processor may assign a higher priority or weight to a fixed position parameter than to the priority / weight assigned to the distance parameter between people.

[0073] In some embodiments, the device processor may be configured to adjust priorities or weights associated with parameters based on user inputs. In such embodiments, the device processor may allow the user / wearer to manually control the degree to which various parameters are important to that user or group or users (e.g., game participants, etc.). In some embodiments, the device processor may receive user inputs in the form of user selections made in a settings menu of a user interface of a smartphone, a head-mounted display, and / or an associated computing device.

[0074] In some embodiments, the device processor may assign a higher priority or weight to a fixed position parameter, size parameter, or line of sight parameter than to the priorities / weights assigned to the field of view parameter, spacing parameter, distance between people parameter, height parameter, and visibility parameter.

[0075] Returning to Fig. 6, the device processor may determine display properties of a digital representation of virtual content, such as a remote participant, based on parameters (and their individual weights / priorities) determined in block 608. For example, the device processor may determine, based on parameters determined in block 608, that the digital representation of virtual content, such as a remote participant, should be rendered in an area surrounded by empty space present in the environment (e.g., an empty chair). As another example, the device processor may determine, based on parameters determined in block 608, that the digital representation of virtual content, such as a remote participant, should be sized so that the virtual content appears to the user / wearer to be approximately 6 feet tall, with a head size between 6 to 7 inches wide and 8 to 9 inches long, a chest circumference between 44 and 46 inches, and a shoulder width of 16 inches.

[0076] Referring to FIG. 7, in blocks (602-608) of method (700), the device processor may perform operations of the identically numbered blocks of method (600) as described with reference to FIG. 6.

[0077] In block 710, the device processor may identify other devices (e.g., other smartphones, computing devices, mobile devices, head-mounted devices, etc.) that are physically close to the device processor (e.g., in the same environment, area, game room, etc.). This may be achieved, for example, by broadcasting a ping message, or by sending a beacon requesting that all devices receiving the message / beacon respond with an identifier (e.g., a Media Access Control (MAC) address, etc.), communication parameters, location information, and / or other similar information.

[0078] In block 712, the device processor may transmit the parameters determined in block 606 and / or the display attributes determined in block 608 to identified devices (i.e., other devices in the same environment, etc.).

[0079] Referring to FIG. 8, in blocks (602-606) of method (800), the device processor may perform operations of the identically numbered blocks of method (600) as described with reference to FIG. 6.

[0080] In block 808, the device processor may receive additional parameters from one or more other devices that are physically close to the device processor (e.g., in the same environment). For example, as each device processor determines the parameters, such parameters may be transmitted to all other device processors in the environment. As another example, if a single device processor performs operations of various embodiments for all devices in the environment, each smartphone or head-mounted device may transmit images, positions, orientations, and other information to that single processing device.

[0081] In block 810, the device processor may determine display attributes for a digital representation of virtual content, such as a remote participant, based on the determined parameters and additional parameters received from other devices.

[0082] In block 812, the device processor may transmit the display attributes determined in block 810 to identified devices (i.e., other devices in the same environment, etc.).

[0083] Referring to FIG. 9, in blocks (602-606) of method (900), the device processor may perform operations of the identically numbered blocks of method (600) as described with reference to FIG. 6.

[0084] In Block 908, the device processor may negotiate display properties for rendering a digital representation of virtual content, such as a remote participant, with other devices in the environment (or physically close to participant smartphones, head-mounted devices, etc.) to generate coordinated display properties. For example, the device processor may transmit proposed display properties to other devices and receive their own proposed display properties from each device. Various determination methods may be employed by one, some, or all device processors among the devices to arrive at a single set of display properties. For example, each device processor may test each received set of display properties and transmit to other devices an indication of whether any of the various criteria for determining display properties are violated. As another example, one device processor in the environment may be designated as a controller for determining display properties, and may do so based on inputs (e.g., position and orientation information, and images captured by other head-mounted displays), and may use collected information to determine coordinated display properties. Other methods may be used to determine the adjusted display properties.

[0085] In block 910, the device processor may render a digital representation of virtual content, such as a remote participant, based on coordinated display properties. For example, the device processor may use coordinated display properties to render a digital representation of virtual content, such as a remote participant, at the same fixed position for all juxtaposed participants. That is, because the display properties are negotiated and coordinated, if a digital representation of virtual content, such as a remote participant, is placed at a specific position on the game table on another smartphone or head-mounted device in the environment, the device processor will render the digital representation of virtual content, such as a remote participant, at that same position.

[0086] FIG. 10 illustrates a method (1000) for updating display properties for a digital representation of virtual content, such as a remote participant, according to one embodiment. The method (1000) may be performed after the display properties of the digital representation are determined, the digital representation is rendered, or the virtual game has started. All or part of the method (1000) may be performed by a device processor, such as a smartphone (e.g., the processor (121) illustrated in FIG. 11) or a processor in a head-mounted device worn by a parallel participant (e.g., the processor (114) illustrated in FIG. 1a).

[0087] In block 1002, the device processor may capture, generate, or receive updated images of the environment. For example, a camera on one or all of the devices may capture video of the environment and periodically provide image frames to the device processor (e.g., a smartphone processor, a processor of a head-mounted device, a processor of a remote computing device, etc.). As another example, images of the environment may be periodically captured by one or more cameras positioned within the environment and communicating with the device processor.

[0088] In block 1004, the device processor may analyze the updated image to determine whether the number, location, position, or size of people or objects in the environment has changed. Such operations may include determining the boundaries of various participants and objects, and comparing the determined boundaries with the boundaries determined at the last time the display parameters were determined and stored in memory.

[0089] In block 1006, the device processor may determine updated parameters for an identified object in the environment in response to determining that the location, position, or size of people or objects in the environment has changed.

[0090] In block 1008, the device processor may determine the difference between the original parameters and the updated parameters.

[0091] In decision block 1010, the device process may determine whether the difference between the original parameters and the updated parameters exceeds a threshold. The threshold may be set high enough so that renderings are not adjusted when certain changes are detected, such as when a single participant is standing. Alternatively or additionally, the threshold may be set low enough so that renderings are adjusted when certain changes are detected, such as when a majority of juxtaposed people are standing or sitting.

[0092] In response to determining that the difference between the original parameters and the updated parameters does not exceed a threshold value (i.e., determination block 1010 = "No"), the device processor may, in block 1012, continue to render a digital representation of virtual content, such as a remote participant, based on the original parameters (e.g., parameters calculated in block 606, etc.) and / or the original display properties (e.g., display properties calculated in block 608, etc.).

[0093] In response to determining that the difference between the original parameters and the updated parameters exceeds a threshold value (i.e., determination block 1010 = "Yes"), the device processor may, in block 1014, determine updated display properties for a digital representation based on the updated parameters. For example, the device processor may, in block 1014, perform one or more operations of any of the methods (600, 700, 800, or 900) to determine the updated parameters.

[0094] In block 1016, the device processor may render a digital representation of virtual content, such as a remote participant, based on updated display properties.

[0095] Some embodiments may include methods for determining display properties for displaying images of one or more remote participants in a game, augmented reality game, training session, meeting, etc., as in a virtual reality system, by capturing an image of a game room / area (by a processor associated with a smartphone and / or head-mounted device), analyzing the captured image to identify people and objects in the game room / area, determining parameters for the identified people and objects, and determining display properties of a digital representation of virtual content, such as a remote participant, based on the determined parameters.

[0096] In some embodiments, determining parameters for identified people and objects may include determining parameters based on one or more factors, wherein the one or more factors include at least one of a fixed position factor, a size factor, a line of sight factor, a field of view factor, a spacing factor, a distance between people factor, a height factor, or a visibility factor.

[0097] Some embodiments may include assigning priorities to each of one or more factors, wherein determining display properties of a remote participant's digital representation based on determined parameters may include determining display properties of a remote participant's digital representation based on priorities assigned to each of one or more factors, and assigning priorities to each of one or more factors may include assigning a higher priority to a fixed position factor, size factor, or line of sight factor than to priorities assigned to a field of view factor, spacing factor, person-to-person distance factor, height factor, and visibility factor.

[0098] Some embodiments may include identifying other head-mounted devices in a game room / area and transmitting determined parameters to other smartphones and / or head-mounted devices in the game room / area. Some embodiments may include receiving additional parameters from other smartphones and / or head-mounted devices in the game room / area, wherein determining display properties of a digital representation of virtual content, such as a remote participant, based on the determined parameters may include determining display properties of a digital representation of virtual content, such as a remote participant, based on the determined parameters and additional parameters received from other smartphones and / or head-mounted devices in the game room / area. In some embodiments, determining display properties of a digital representation of virtual content, such as a remote participant, based on the determined parameters may include negotiating display properties for rendering the digital representation of virtual content to generate adjusted display properties with other smartphones and / or head-mounted devices in the game room / area.

[0099] Some embodiments may include using adjusted display properties to render a digital representation of virtual content, such as a remote participant, so that all juxtaposed participants physically present in the game room / area perceive the virtual content as being in the same fixed position in the game room / area. Some embodiments may include using adjusted display properties to intelligently sizing a digital representation of a remote participant so that the remote participant appears to be approximately the same size as the juxtaposed participants physically present in the game room / area. Some embodiments may include using adjusted display properties to render a digital representation of virtual content, such as a remote participant, so that the virtual content appears to be approximately the same height as the average height of the juxtaposed participants physically present in the game room / area.

[0100] Some embodiments may include using adjusted display properties to render a digital representation of virtual content, such as a remote participant, so that the virtual content appears to be approximately equidistant from the juxtaposed participants physically present in the game room / area. Some embodiments may include using adjusted display properties to render a digital representation of virtual content, such as a remote participant, so that the virtual content appears to be anchored to an empty chair on a game table. Some embodiments may include capturing an updated image of the game room / area; analyzing the captured updated image to determine whether the number, location, position, or size of people or objects in the game room / area has changed; determining updated parameters for identified people and objects in the game room / area in response to determining that the location, position, or size of people or objects in the game room / area has changed; determining whether the difference between the determined parameters and the determined updated parameters exceeds a threshold; and determining updated display properties for a digital representation of virtual content, such as a remote participant, in response to determining that the difference between the determined parameters and the determined updated parameters exceeds a threshold.

[0101] Various embodiments and methods may be implemented in various personal computing devices, such as a laptop computer (1100) as illustrated in FIG. 11. The laptop computer (1100) will typically include a processor (1101) coupled to a large amount of non-volatile memory, such as volatile memory (1102) and a disk drive (1104) of flash memory. The laptop computer (1100) may also include a floppy disk drive (1105) coupled to the processor (1106). The computer receiver device (1100) may also include a number of connector ports or other network interfaces coupled to the processor (1101) to establish data connections or to accommodate external memory receiver devices, such as Universal Serial Bus (USB) or FireWire® connector sockets for coupling the processor (1101) to a network (e.g., a communication network), or other network connection circuits. In a notebook configuration, the computer housing includes a touchpad (1110), a keyboard (1112), and a display (1114), all of which are coupled to a processor (1101). Other configurations of computing devices may include a computer mouse or trackball coupled to the processor (e.g., via a USB input), as is widely known, and may also be used with various embodiments.

[0102] Processors may be any programmable microprocessor, microcomputer, or multiprocessor chip or chips configured by software instructions (applications) to perform various functions, including the functions of the various embodiments described in this application. In some mobile devices, multiple processors may be provided, such as one processor dedicated to wireless communication functions and one processor dedicated to running other applications. Typically, software applications may be stored in internal memory before being accessed and loaded into the processor. The processor may include sufficient internal memory to store application software instructions.

[0103] The various embodiments illustrated and described are provided merely as examples to illustrate the various features of the claims. However, the features illustrated and described for any given embodiment are not necessarily limited to the associated embodiments and may be used or combined with other embodiments illustrated and described. Additionally, the claims are not intended to be limited by any one exemplary embodiment. For example, one or more of the operations of the methods may be replaced or combined with one or more of the operations of the methods.

[0104] The foregoing method descriptions and process flow diagrams are provided merely as exemplary examples and are not intended to require or imply that the operations of the various embodiments may be performed in the order presented. As will be recognized by those skilled in the art, the order of operations in the foregoing embodiments may be performed in any order. Words such as “thereafter,” “next,” “next,” etc., are not intended to limit the order of operations; these words are used to guide the reader through the description of the methods. Additionally, any reference to claim elements in the singular using, for example, articles (“a,” “an,” or “the”) is not to be interpreted as limiting the element to the singular.

[0105] The various exemplary logic blocks, functional components, functional components, circuits, and algorithmic operations described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or a combination of both. To clearly exemplify such interchangeability between hardware and software, the various exemplary components, blocks, functional components, circuits, and operations have generally been described in terms of their functionality. Whether such functionality is implemented as hardware or as software depends on the design constraints imposed on the overall system and the specific application. Those skilled in the art may implement the described functionality in various ways for each specific application, but such determinations of embodiments should not be interpreted as causing a deviation from the scope of the claims.

[0106] Hardware used to implement the various exemplary logics, logic blocks, functional components, and circuits described in connection with the embodiments disclosed herein may be implemented or performed as a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general-purpose processor may be a microprocessor, but alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other configuration. Alternatively, some operations or methods may be performed by circuits specific to a given function.

[0107] In one or more embodiments, the described functions may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored as one or more instructions or code on a non-transient computer-readable medium or a non-transient processor-readable medium. Operations of the method or algorithm disclosed herein may be implemented in a processor-executable software module that may reside on a non-transient computer-readable or processor-readable storage medium. Non-transient computer-readable or processor-readable storage media may be any storage medium that may be accessed by a computer or processor. As an example, but not a limitation, such non-transient computer-readable or processor-readable media may include RAM, ROM, EEPROM, flash memory, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that may be used to store desired program code in the form of instructions or data structures and may be accessed by a computer. As used herein, disks and discs include compact discs (CDs), laser discs, optical discs, digital multifunction discs (DVDs), floppy discs, and Blu-ray discs, wherein disks typically reproduce data magnetically, while discs reproduce data optically using a laser. The above combinations are also included within the scope of non-transient computer-readable and processor-readable media. Additionally, operations of a method or algorithm may reside on non-transient processor-readable and / or computer-readable media as one or any combination or set of codes and / or instructions, and may be incorporated into a computer program product.

[0108] The foregoing description of the disclosed embodiments is provided to enable those skilled in the art to manufacture or use the claims. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments and implementations without departing from the scope of the claims. Accordingly, the present disclosure is not intended to be limited to the embodiments and implementations described herein, but should be given the broadest scope consistent with the principles, novel features, and the following claims disclosed herein.

Claims

Claim 1 A method (600; 700; 800) for determining a display position for virtual content in an environment having multiple participants, wherein each participant has a user device (100; 200), and the method comprises: a step (602) of capturing an image of the environment by an image sensor controlled by a processor of a user device associated with one of the multiple participants; a step (606) of determining parameters for at least one virtual object to be rendered as a digital representation for the environment by the processor; a step (808) of receiving additional parameters for the at least one virtual object from at least one other user device of another participant among the multiple participants by the processor; a step (810) of determining the position of the digital representation of the at least one virtual object in the environment based on the determined parameters and the received additional parameters by the processor, wherein the determining step includes generating an adjusted display position for the at least one virtual object; and a step (712; 812) of transmitting the adjusted display position to the at least one other user device by the processor. A method for determining a display position for virtual content for an environment having a plurality of participants, comprising the step of rendering the digital representation of the at least one virtual object at the adjusted display position. Claim 2 A method for determining a display location for virtual content in an environment having a plurality of participants, wherein, in claim 1, the step of assigning priority to the determined parameter further comprises the step of determining the location of the digital representation of the at least one virtual object in the environment based on the determined parameter and the received additional parameter (810), wherein the step of determining the location of the digital representation of the at least one virtual object in the environment based on the priority assigned to the determined parameter. Claim 3 A method for determining a display position for virtual content in an environment having a plurality of participants, wherein the step of determining parameters for at least one virtual object comprises: determining at least one of a fixed position parameter, a size parameter, or a line-of-sight parameter; and determining at least one of a field of view parameter, a spacing parameter, a person-to-person distance parameter, a height parameter, and a visibility parameter, and the step of assigning a priority to the determined parameters comprises: assigning a first priority to the fixed position parameter, the size parameter, or the line-of-sight parameter; and assigning a second priority to the field of view parameter, the spacing parameter, the person-to-person distance parameter, the height parameter, or the visibility parameter, wherein the first priority is a higher priority than the second priority. Claim 4 A method for determining a display position for virtual content in an environment having multiple participants, further comprising the step of using one or more adjusted display attributes to size the digital representation of the at least one virtual object based on the size of a participant physically present in the environment. Claim 5 A method for determining a display location for virtual content in an environment having multiple participants, wherein, in claim 1, at least one virtual object comprises a digital representation of a remote participant, and the method further comprises the step of using one or more adjusted display properties to render the digital representation of the remote participant so that the remote participant appears to be approximately equidistant from a first participant and a second participant. Claim 6 A method for determining a display location for virtual content for an environment having multiple participants, further comprising the step of using one or more adjusted display properties to render the digital representation of the remote participant so that the remote participant appears to be anchored to the environment. Claim 7 A method for determining a display position for virtual content in an environment having a plurality of participants, further comprising: receiving a user input that adjusts the display position in which the digital representation of the at least one virtual object is rendered; rendering the at least one virtual object at the adjusted display position; and transmitting the adjusted display position to the at least one other user device. Claim 8 A method for determining a display location for virtual content in an environment having multiple participants, wherein the step of determining the location of the digital representation of the at least one virtual object based on the determined parameter and the received additional parameter includes the step of determining at least one location for each of the digital representations of the multiple participants. Claim 9 A method for determining a display position for virtual content in an environment having a plurality of participants, wherein the step of determining a parameter for at least one virtual object comprises determining at least one of a fixed position parameter; a size parameter; a line of sight parameter; a field of view parameter; a spacing parameter; a distance between people parameter; a height parameter; or a visibility parameter. Claim 10 In claim 1, the method for determining the display location of virtual content for an environment having a plurality of participants, wherein at least one virtual object is associated with a game. Claim 11 A method for determining a display location for virtual content for an environment having multiple participants, wherein at least one virtual object is a digital representation of a remote participant, in claim 1. Claim 12 A user device (100; 120) configured to determine a display location for virtual content for an environment having multiple participants, wherein the user device is associated with one of the multiple participants and includes a memory (116; 122); an image sensor (110); and a processor (114; 121) coupled to the memory and the image sensor, wherein the processor is configured to capture an image of the environment using the image sensor (602); determine parameters for at least one virtual object to be rendered as a digital representation for the environment (606); receive additional parameters for the at least one virtual object from at least one other user device of another participant among the multiple participants (808); determine a location for the digital representation of the at least one virtual object for the environment based on the determined parameters and the received additional parameters (810); transmit a coordinated display location to the at least one other user device (712; 812); and render the digital representation of the at least one virtual object at the coordinated display location. Claim 13 In claim 12, the processor is further configured to perform the method of any one of claims 2 to 11, a user device. Claim 14 delete Claim 15 delete Claim 16 delete Claim 17 delete Claim 18 delete Claim 19 delete Claim 20 delete Claim 21 delete Claim 22 delete Claim 23 delete Claim 24 delete Claim 25 delete Claim 26 delete Claim 27 delete Claim 28 delete Claim 29 delete Claim 30 delete Claim 31 delete Claim 32 delete Claim 33 delete Claim 34 delete

Images