2D and 3D conversion for rendering of users participating in communication session

Abstract

A system for converting a user interface arrangement from display of a two-dimensional image of a user to rendering of a three-dimensional representation of the user is provided. The system may start from a UI that includes a rendering of the user based on the 2D image file. The system may receive input configured to cause the system to convert display of rendering of a 2D image of a selected user to rendering of a three-dimensional representation of the selected user. To display a rendering of a 3D representation of a selected user, the system displays the 3D representation of the user using permission data and a three-dimensional model defining a location and orientation. The system allows a user to switch between viewing modes to allow the user to interact with content using the most efficient hardware type.

Description

Background Art

[0001] The use of meta-verse environments in online meeting applications is becoming ubiquitous. Participants in online meetings now meet in three-dimensional virtual environments and share content within those virtual environments. Despite many benefits compared to other forms of collaboration, using 3D environments to share content can come with many disadvantages.

[0002] One of the main issues with using a metaverse environment for online meeting applications is that there may be scenarios where not all participants of a meeting have the same type of device. For example, some participants may be operating a PC while others may be operating a VR headset. This scenario may be due to the availability of hardware, while in other cases, users may have different types of devices based on personal preference. Personal preferences for hardware types may be based on the capabilities of each device. Some types of computers, such as desktop devices, are better suited for certain roles and meeting functions, such as editing content. In this case, it may be better for some people who want to participate in a metaverse environment meeting to use a desktop computer while other users who consume content can use a head-mounted display device.

[0003] In some cases, desktop device users are at a disadvantage because they cannot navigate or interact with all users in the virtual environment. When a computer provides a 2D view of a 3D environment, the computer is limited in the input gestures it can receive from the user to navigate or interact with the 3D environment. Given these issues involving different types of devices and different types of environments, current technology cannot provide the same experience for VR headset users and PC users. In addition, even if a user wants to transition from a VR headset to a desktop device, or vice versa, some existing systems may not always provide a seamless transition during events such as parties or corporate meetings. Summary of the invention

[0004] The technology disclosed herein enables the system to convert the user interface layout from the display of the two-dimensional image of the user to the rendering of the three-dimensional representation of the user while the user is participating in a communication session (e.g., a meeting). In some configurations, the selected user can start in a gallery mode, where the rendering of the user is based on data received from the user's image file or real-time video. In some configurations, the selected user can be displayed in the user interface of other users who are participating in the meeting. Other users can be displayed as 2D renderings from images or real-time video streams. Other users can also be displayed as 3D representations located in a 3D environment, such as avatars. The system can receive input, which is configured to cause the system to convert the display of the rendering of the two-dimensional image of the selected user to the rendering of the three-dimensional representation of the selected user. In order to display the rendering of the 3D representation of the selected user, the system accesses a three-dimensional model, which defines the position and orientation of the three-dimensional representation of the selected user in the 3D environment. The 3D environment can be a model of a conference room, where the 3D representation of the user is located. The input can include voice commands, key inputs, or another type of input gesture. In some embodiments, the input can indicate that the user is operating with a specific device type (e.g., a desktop PC), and based on the device type, the system can cause the conversion. In response to input for causing a transition from display of a rendering of a two-dimensional image of the user to rendering of a three-dimensional representation of the user, the system can modify the user interface arrangement to remove the rendering of the two-dimensional image of the user and add display of the three-dimensional representation of the user in the rendering of the 3D environment. Such a transition can cause the system to place the user's avatar in the 3D environment using the position and orientation defined in the three-dimensional model.

[0005] The present technical disclosure provides many technical advantages. For example, when the system detects that a user is operating a desktop computer, some systems may display a 2D image of the user. This may be due to the fact that desktop computers and desktop cameras may be more suitable for displaying 2D images of the user. However, if the user wishes to participate in a meeting using a 3D representation (e.g., an avatar in a 3D environment), the present technical disclosure allows the user to use a desktop computer to make the transition without switching to a headset. The technology disclosed herein also changes the operating mode of the computer to allow the user to navigate within the 3D environment using keyboard controls. This allows the user to interact with the computer in a 3D environment while using certain types of computers (e.g., desktop devices) to access specialized editing tools optimized for certain 3D models.

[0006] By reading the following detailed description and viewing the associated drawings, features and technical advantages other than those explicitly described above will become apparent. This summary is intended to introduce a set of concepts in a simplified form, which will be further described in the detailed description below. This summary is not intended to identify the key features or essential features of the subject matter protected, nor is it intended to be used to help determine the scope of the subject matter claimed. For example, as allowed by the above context and throughout the document, the term "technology" may refer to systems, methods, computer-readable instructions, modules, algorithms, hardware logic and / or operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The detailed description is described with reference to the accompanying drawings. In the drawings, the leftmost digit of a reference number identifies the figure in which the reference number first appears. The same reference numbers in different figures indicate similar or identical items. References to individual items in a plurality of items may use reference numbers with letters of the alphabetical sequence to refer to each individual item. General references to items may use the specific reference number without the alphabetical sequence.

[0008] Figure 1A A first stage of a transformation of a user interface of a desktop display device is shown, where the transformation begins with display of a two-dimensional image of a user and changes to rendering of a three-dimensional representation of the user while the user is participating in a communication session.

[0009] Figure 1B A second stage of transition of the user interface from display of a two-dimensional image of the user to rendering of a three-dimensional representation of the user while the user is participating in a communication session is shown.

[0010] Figure 2A A first stage of a transformation of a user interface of a head mounted display device is shown, where the transformation begins with display of a two-dimensional image of a user and changes to rendering of a three-dimensional representation of the user while the user is participating in a communication session.

[0011] Figure 2B A second stage of transition of the user interface from display of a two-dimensional image of the user to rendering of a three-dimensional representation of the user while the user is participating in a communication session is shown.

[0012] Figure 3A A first user interface arrangement is shown for a particular user identified in the user interface.

[0013] Figure 3B A second user interface arrangement is shown for a particular user identified in the user interface.

[0014] Figure 4A Additional features of UI transitions are shown.

[0015] Figure 4BTwo modes of operation of the system are shown, along with how each mode of operation changes the permissions displayed for individuals participating in a communication session.

[0016] Figure 5A A scene is shown in which avatars of a first user and a second user are oriented such that the users are viewing a display of virtual content within a virtual environment.

[0017] Figure 5B A scenario is shown in which an avatar of a user newly added to a communication session is positioned and oriented based on avatars of existing users who are viewing a virtual display of content within a virtual environment.

[0018] Fig. 6A A scene is shown in which avatars of a first user and a second user are oriented such that the users are viewing each other within a virtual environment.

[0019] Figure 6B A scenario is shown in which an avatar of a user newly added to a communication session is positioned and oriented based on the avatars of existing users who are viewing each other within a virtual environment.

[0020] Figure 7 is a flow chart illustrating aspects of a routine that enables a system to transition a user interface arrangement from display of a two-dimensional image of a user to rendering of a three-dimensional representation of the user while the user is participating in a communication session.

[0021] Figure 8 is a computer architecture diagram showing an illustrative computer hardware and software architecture for a computing system capable of implementing aspects of the techniques and technologies presented herein.

[0022] Fig. 9 is a computer architecture diagram illustrating a computing device architecture of a computing device capable of implementing aspects of the techniques and technologies presented herein. DETAILED DESCRIPTION

[0023] Figure 1A and Figure 1BAn example of a transition of a user interface arrangement from a display of a two-dimensional image of a user to a rendering of a three-dimensional representation of the user while the user is participating in a communication session is shown. The communication session may be managed by a system 100, which includes a plurality of computers 11, each corresponding to a plurality of users 10. The computer may be in the form of a desktop computer, a head mounted display unit, a tablet device, a mobile phone, etc. The system may generate a user interface that shows aspects of the communication session to each user. In this example, the user interface 101 may include a plurality of renderings of each user 10. The rendering may include a rendering of a two-dimensional (2D) image, which may include a picture or a real-time video feed of the user. The rendering may also include a rendering of a three-dimensional (3D) representation, which may include an avatar located within a 3D virtual environment 200. In this particular example, the user interface 101 includes a 2D rendering 151A of a first user 10A, a 2D rendering 151B of a second user 10B, and a 2D rendering 151A of a third user 10C. The user interface 101 also includes a 3D rendering of a representation 251D of a fourth user 10D and another 3D rendering of a representation 251E of a fifth user 10E.

[0024] In this example, the rendering of the third user 10C will undergo a transition from a 2D mode in which they are shown as a rendered 2D image to a 3D mode in which they are represented as a 3D object in a 3D environment. To illustrate aspects of this transition, Figure 1A The user interface 101 shown may be a user interface displayed on a computing device of another user, such as the first computer 11A associated with the first user 10A, or any other user other than the third user 10C. This shows others what happens when the third user 10C switches from 2D mode to 3D mode. This example is provided for illustrative purposes only and should not be considered limiting. It is understood that this exemplary UI may be displayed on any computing device of any user participating in a communication session.

[0025] In order to start the conversion, the system can receive an input for causing a conversion of the rendered display of the two-dimensional image of a specific user. In this example, the input identifies the third user 10C. The input can also provide permission for the system to access the 3D model of the position and orientation of the three-dimensional representation of the third user 10C. The position and orientation can include vectors and coordinates of the representation within the 3D environment 200, which is also referred to as the virtual environment 200 in this article.

[0026] like Figure 1A and Figure 1B As shown, in response to receiving the input, one or more computers of system 100 may cause modification of user interface 101 to remove the Figure 1A The rendering of the image 151C of the user 10C shown in FIG. 10A and adding the image 151C of the user 10C shown in FIG. 10B and adding the image 151C of the user 10C shown in FIG. 10C Figure 1BA rendering of a 3D representation 251C of a user 10C is shown. The rendering of the 3D representation 251C of the user 10C may be positioned and oriented in the 3D environment according to coordinates and / or vectors defined in the 3D model.

[0027] In this example, the rendering of the 2D image of user 10C is removed and can be replaced with other renderings. For example, Figure 1A The 2D image rendering of the third user 10C is shown in the UI by another user ( Figure 1B In one embodiment, the present invention provides a method for converting a display of a user's video content to another 2D image of a fourth user 10D shown in the communication session. This conversion (also referred to as teleportation herein) enables a specific user (e.g., a third user 10C) to convert the display they present to other users of the communication session. In addition, this conversion enables the user to interact with the computing device using different modes. For example, in this example, if the selected user 10C wishes to convert from a real-time video stream in the communication session to another operating mode that allows the user to interact with other users in a 3D environment, the system will convert the user from a mode that allows them to interact with general content, documents, spreadsheets, and slide groups to another mode that allows them to interact with 3D objects. This conversion during the communication session allows the selected user to use editing tools in each environment suitable for different content types. For example, if someone in the video stream wants to leave the 2D mode that uses 2D images to show the user, and enter a 3D environment to show other users how to move objects in a specific position or shape a specific 3D object, once they are able to convert within the communication session, the user can do this more easily. The user (e.g., third user 10C, Charlotte Smith) can use a desktop PC to perform this conversion without using any type of virtual reality or augmented reality headset, such as an HMD. As described herein, this conversion using a desktop enables the user (e.g., third user) to use a desktop computer to enter a 3D mode for interacting with a 3D computing environment, which may be more suitable for editing or viewing certain types of content.

[0028] The technology disclosed herein also provides tools that are counter-intuitive to other systems. For example, even if the user is working on a desktop computer, they can transition to a 3D environment without using a VR headset. This is counter-intuitive to embodiments that require the user to wear a headset to enter a 3D environment. This enables the user to benefit from the accuracy of a PC (e.g., a mouse and keyboard) while also working in a 3D environment interacting with 3D objects. The user can benefit from the gestures of the mouse to move objects while navigating in a 3D environment using the arrow keys.

[0029] A specific type of input (which may be computer-generated or caused by user input) may prompt the conversion described herein. For example, user input such as voice commands, key inputs, or other gestures may be used to invoke the conversion described herein. In addition, input indicating that a remote user wants to share content with a selected user may also invoke the conversion described herein. In a specific example, consider a scenario in which a second user 10B shares a 3D file (e.g., an AutoCAD file) with a third user 10C. If the input is received while the second user is interacting with the computer via a 3D environment, and the third user is not interacting with the computer via a 3D environment, such input received in the scenario may invoke the conversion described herein. In yet another embodiment, if the input is received from the second user while the second user is interacting with the computer via a 3D environment, and the third user is on a desktop computer and is not interacting with the computer using a rendering of the 3D environment, such input received in the scenario may invoke the conversion described herein. Relative to other systems, this may be counter-intuitive because even if the third user is operating on a desktop computer and is not using a head mounted display (HMD) unit, the person may also be converted to a 3D environment.

[0030] One of the technical advantages is that the system can allow users to switch between 3D and 2D modes of a communication session, regardless of what hardware they are interacting with. In this way, a person who uses software that is best suited for a desktop computer using a mouse and keyboard to interact with a specific type of content (such as a 3D model or AutoCAD file), can use the computer while also interacting with the content in a 3D environment. This is counterintuitive to existing systems because those systems require a headset to interact with a 3D environment. When a user uses a desktop computer or a device that communicates with a keyboard and pointing device (such as a mouse), the system can assign specific gestures that can be captured by a camera or special keys to allow the user to navigate in a 3D environment. In this way, a traditional computer (such as a desktop device) can allow the user to access a pointing device for accurate movement to interact with an AutoCAD file, while also providing special keys or then placing a device (such as a camera) to allow the user to navigate in a 3D environment.

[0031] The disclosed techniques provide a technical advantage in that a user of a personal computer can change their representation (e.g., how they appear to others) and also change how the user sees others from a given perspective (e.g., how others appear to the user) while maintaining operation of a single device (e.g., a desktop computer). In such a scenario, the user can maintain use of only one computing device, such as a desktop computer, while transitioning from a 2D computing environment to a 3D computing environment. Thus, the user can begin in a 2D computing environment and be represented by a rendering of a 2D image 151C, such as Figure 1AThen, in response to one or more inputs (e.g., a user begins editing content having a particular file type), or based on an input indicating an instruction to perform a UI transition, the system can transition the UI to remove the rendering of the 2D image 151C, such as Figure 1A As shown, and a rendering of a 3D representation 251C of the user is generated, as Figure 1B This allows users to transition into a 3D environment without actually using an augmented reality or virtual reality (AR / VR) device.

[0032] The technology is also applicable to other types of devices, such as head mounted display devices. In such an embodiment, the user can switch the interaction model from the 3D computing environment to the 2D computing environment while maintaining only one computing device, such as an HMD. Thus, the user can start in a 3D computing environment and be represented by a rendering of the 3D representation 251C, such as Figure 1B Then, in response to one or more inputs, such as a user starting to edit content having a particular file type or based on input indicating an intent to perform a UI transition, the system can transition the UI to remove the rendering of the 3D representation 251C, such as Figure 1B As shown, and a rendering of a 2D image 151C of the user is generated, for example Figure 1A This allows the user to transition to a 2D environment without actually using a desktop device that utilizes a flat screen display and keyboard.

[0033] Figure 2A and Figure 2B Another example of a conversion of a user interface from a display of a two-dimensional image of a user to a rendering of a three-dimensional representation of a user while the user is participating in a communication session is shown. In this example, the user interface 201 is a rendering of a 3D environment based on a 3D model. In this example, the user interface 201 starts with a 3D rendering of a representation 251A of a first user 10A and a 3D rendering of a representation 251B of a second user 10B. The 3D rendering of each representation has a position and direction determined by virtual object properties stored in the 3D model. In this example, the 3D environment also includes a virtual object 275 in the form of a virtual flat screen television mounted on a wall of the virtual environment. The virtual object 275 has a display surface that displays a virtual user interface that displays a 2D rendering 151C of a third user 10C and a 2D rendering 151D of a fourth user 10D. In this example, the system receives input identifying a user, which, for illustration purposes, is the third user 10C associated with the rendering 151C of the 2D image.

[0034] like Figure 2A and Figure 2BAs shown, in response to input causing a transition from display of a rendering of a 2D image of user 10C to rendering of a three-dimensional representation of user 10C, the system modifies user interface arrangement 201 to remove rendering 151D of the 2D image of user 10C and add display of a three-dimensional representation 251C of the user in the rendering of 3D environment 200. This transition may cause the system to place the user's avatar in the 3D environment using position and orientation data defined in the 3D model.

[0035] Figure 3A and Figure 3B Another aspect of the transition for a third user 10C, Charlotte Smith, is shown. These figures show the user interface 301 from the perspective of Charlotte's computer. In particular, these figures show the user experience from Charlotte's perspective, such as how the transition is presented to a third user who is transitioning from a 2D mode to a 3D mode. In this example, the rendering of the third user 10C undergoes a transition from a 2D mode in which they are displayed to others as a rendering of a 2D image to a 3D mode in which they are represented to others as a 3D object in a 3D environment. To illustrate aspects of this transition from the third user's perspective, Figure 3A The user interface 301 shown is a user interface displayed on the computing device 11C of a third user 10C Charlotte Smith. This shows the third user a perspective of what happens when the third user 10C switches from 2D mode to 3D mode.

[0036] like Figure 3A As shown, user interface 301 begins with a display of 2D images of Jazmine, Lawrence, and Mike, each of which is displayed as a rendering of image 151A, 151B, and 151D, respectively. The user interface also includes a rendering of a 3D environment 200 having two 3D representations 251A and 251B of other users. In response to input data as described herein, such as a third user editing some type of data or one or more users indicating in a voice or text chat that the third user wants to switch to 3D mode, the system performs the switch. In the switch of the third user, the third computer 11C of the third user 10C switches from 3D mode to 3D mode. Figure 3A The user interface shown switches to Figure 3B The user interface shown.

[0037] like Figure 3BAs shown, after the transition, Charlotte's computer, computer 11C, displays a modified user interface 301 having an enlarged rendering of the 3D environment 200 including two 3D representations 251A and 251B of two other users. The system maintains the state of each user, e.g. Figure 3A In the figure, two users are shown as 3D representations 251A and 251B, Figure 3B The 3D representations 251A and 251B are also maintained. Figure 3B It is also shown that the modified user interface 301 also includes a virtual object 275, which is a virtual display device in this example, and the virtual object 275 shows a 2D rendering of other users that were originally displayed as 2D images, such as Figure 3A The modified user interface 301 now shows Charlotte's perspective as if she had been teleported from the 2D environment to the 3D environment. Similar to the other examples, in this teleportation, the system can also determine the position and orientation of Charlotte's avatar based on one or more factors. In this example, the orientation and position of Charlotte's avatar is such that she is facing the shared content (e.g., a virtual display monitor) and is located on a virtual chair.

[0038] In such an example, Charlotte may be operating a single device, such as a PC, and she starts by viewing the conference in 2D mode, such as her perspective is not in a 3D environment. Then, in response to one or more inputs described herein, the system may Figure 3A The user interface is converted to Figure 3B The conversion in this example can occur even without using a computer traditionally used to view 3D renderings, such as an HMD.

[0039] In another example, the conversion may also involve Charlotte from Figure 3B The user interface starts and transitions to Figure 3A In such an example, Charlotte might be operating a single device, such as a head-mounted display, from which she Figure 3B Then, in response to one or more inputs described herein, the system may start from Figure 3B The user interface is converted to Figure 3A The conversion in this example can occur even without using a computer traditionally used to view 2D images, such as a desktop computer.

[0040] As may be applied to other examples described herein, in some embodiments, the transformations that occur on Charlotte's image are performed while maintaining the rendering type of other users. Figure 3A In the example, a user named Jazmine is in Figure 3A It was first shown as a 2D rendering, and then after a conversion by a third user, Charlotte, Jazmine Figure 3B This also applies to other users that were initially shown a 3D rendering, such as Figure 3A After Charlotte's conversion, these users remain as 3D renderings, as shown in Figure 3B The system controls the displays of other users to minimize disruptions that may be caused by multiple users switching simultaneously.

[0041] Figure 4A Additional features of UI transitions are shown. In some embodiments, when input is received that causes a UI transition from rendering of a 2D image of user 10C to rendering of a 3D representation of user 10C, the system can determine the position and orientation of the 3D representation of user 10C. For example, if the model of the virtual environment 200 starts with only two virtual objects 351A and 351B representing users, the system can determine the position and orientation of the newly added virtual object 351C representing the user. In this example, when the input indicates a specific user (e.g., a third user 10C), the system can determine the position and orientation of the virtual object 351C representing the third user 10C based on the positions of other users and / or the position of shared content within the virtual environment 200.

[0042] In one illustrative example, if the system determines that a virtual object 351C representing the third user 10C is to be added to the virtual environment 200, the system may position the virtual object 351C in a manner such that the virtual object 351C appears to be the user's avatar viewing the content shared with the user 10C. In another example, if the system determines that a virtual object 351C representing the third user 10C is to be added to the virtual environment 200, the system may position the virtual object 351C in a manner such that the virtual object 351C appears to be the user's avatar viewing the avatar of the user with whom the user 10C is conversing. As described herein, the system positions the person's avatar so that the avatar does not block other users from viewing the content and / or the system positions the person's avatar toward shared salient information within the virtual environment.

[0043] In some embodiments, the placement of each virtual object 351 can be based on the distribution of team members, user groups, and / or policies established by individual users or user groups. For example, if a person is part of a team within a company, when one of those individuals is identified in the input for converting the user interface, their corresponding avatar will be located within a threshold distance of other team members. The orientation of the user's avatar can be based on an analysis of the orientation of the avatars of its team members. For example, if a threshold number of users within the team are viewing content, the avatar of the user entering the 3D environment can also be oriented to view the content. The system can also configure permissions. For example, when a specific user's avatar enters a 3D environment and a threshold number of teammates are viewing content, the system can also provide access rights to allow the user to access the content. When the user leaves the 3D environment, the access rights can be revoked. A person's permissions can also reflect the permissions of others in the 3D environment. For example, if other users in the team are able to edit content, users in the team can also obtain editing rights when they have avatars in the 3D environment.

[0044] Figure 4A Also shown are aspects of a system configured to implement the technology disclosed herein. For illustrative purposes, a rendering of a 2D image file or a rendering of a user's 2D image can be generated by a 2D rendering engine 551 that receives 2D image data 310 (e.g., an image file). The rendering of a 2D image file can include a 2D environment (e.g., the background of an image) and a 2D object (e.g., an image or avatar of a person). An image file (e.g., image data 310) can have pixels arranged in two dimensions, such as pixels arranged in a two-dimensional coordinate system (x, y). The data may also be referred to herein as a two-dimensional model based on a two-dimensional coordinate system. Each part of the image may be a pixel or any other geometric shape, such as a triangle. For example, a group of pixels or triangles may be used to generate a rendering of a user's two-dimensional avatar or a real-time video image of a person.

[0045] A two-dimensional environment with multiple 2D images of participants of a communication session is also referred to herein as a "grid environment". The image data or the communication data stream may define a two-dimensional environment or a two-dimensional object, and the two-dimensional environment may be rendered on a display screen. The rendering may be referred to herein as a two-dimensional rendering of a two-dimensional environment or a two-dimensional rendering of a two-dimensional object. This is also referred to herein as "rendering of a two-dimensional image".

[0046] For illustrative purposes, a rendering of a 3D model or a rendering of a 3D representation of a user may be generated by a 3D rendering engine 552 that accesses 3D model data 320 (e.g., a 3D model). The 3D model may include parameters defining a 3D environment 200 (e.g., a room model), as well as parameters defining 3D objects, such as the size, shape, and position data of a representation 351 of a user or other virtual object. A three-dimensional environment is a computing environment model based on a three-dimensional coordinate system. The properties of a three-dimensional environment and a three-dimensional object in the three-dimensional environment are based on components located within a three-dimensional coordinate system (x, y, z). Each component may be a triangle or any other geometric shape. Each component may have a position (e.g., a location in a three-dimensional coordinate system) and an orientation (e.g., the direction in which the triangle points). For example, a set of triangles may be used to generate a rendering of a three-dimensional avatar of a user or a three-dimensional rendering of a three-dimensional object.

[0047] The three-dimensional environment is also referred to as an "immersive environment" in this article. Model data or a three-dimensional model can be included in the communication data stream, and the model data can define the three-dimensional environment. The three-dimensional environment can be based on a three-dimensional coordinate system. When the rendering engine 552 generates a 3D rendering from the 3D model, the rendering is generated from a reference point in the environment (e.g., a perspective with a position relative to the virtual environment). For illustrative purposes, the reference point is also referred to as a virtual camera 350 in this article. The camera can have a field of view that is used to generate a rendering of a 3D environment or a 3D object based on the position of the virtual camera 350. The rendering of a three-dimensional object in the three-dimensional environment is based on the position and orientation of the three-dimensional object and the position of the virtual camera 350.

[0048] In some embodiments, a two-dimensional image may be displayed within a three-dimensional environment. This may occur, for example, when a communication system receives a two-dimensional video stream of a user, but the participant receiving the video stream is viewing a 3D environment using an HMD. This may cause the system to display the image of the user as if they were appearing on a virtual television on a wall of the virtual environment. This is referred to herein as a two-dimensional rendering of the user within a three-dimensional environment. This may include Figure 2A A third user 10C is shown as rendering 151C.

[0049] In some embodiments, a three-dimensional environment and a three-dimensional object defined by a three-dimensional model can be displayed as a two-dimensional rendering. For example, when a communication session involves a user interface that displays a two-dimensional image, such as when Teams is in grid mode (GridMode), this can happen. While in this mode, the system may need to display images of users interacting in a 3D environment. In this case, a 2D image of the 3D environment is displayed from a specific position (e.g., a virtual camera position), and the 2D image is displayed within one of the grids. This rendering may be referred to as a two-dimensional rendering of a three-dimensional environment in this article. In order to achieve two-dimensional rendering of a three-dimensional environment, a transformation can be used to project the model data defining the three-dimensional environment. The transformation can generate a rendering so that the width, height, and depth of the three-dimensional object can be expressed on a flat screen using a vector projection from the model of the three-dimensional object to the viewpoint (e.g., a virtual camera position).

[0050] Figure 4B Two modes of operation of the system are shown, along with the manner in which each mode of operation can change the permissions of individuals participating in a communication session. Figure 4B ), the permissions may allow the system to use an image file to display a 2D image of the user when the user's representation is not included in the 3D model. In this case, the 3D model data is in a first state 320A, where the selected user has no virtual objects representing the user within the 3D environment 200. When the 3D model is in this state, where the selected user has no virtual objects representing the user within the 3D environment, the permissions data 315 associated with the user is configured to allow the system and other users to access the image data 310 of the user. This means that the system and each remote user's client can use the image data 310 to generate a rendering of the user, or the system can edit the image data 310.

[0051] When the system detects that the 3D model data is in the second state, for example, the model data 320B includes a virtual object 351C representing a selected user, the system modifies the permissions to limit the use of the image data by the specific user. As shown, the permission data 315 is modified to limit the system to read the image data 310 to display the 2D image of the specific user. In this operating mode, the permissions are configured to restrict all users from accessing the image data, thereby preventing all clients from accessing or displaying the 2D image file.

[0052] Figure 5A and Figure 5B Features of a system configured to position a representation of a user relative to shared content within a 3D environment 200 are shown. The figures show a top view of the 3D environment 200. Figure 5AA scene is shown in which the avatars of the first user 351A and the second user 351B are oriented to view shared content within a 3D environment. The shared content can be displayed on a virtual object, such as a virtual display screen. When the system detects that a threshold number of users are viewing the shared content, the system can generate an orientation for a third user to have the avatar enter the 3D environment. An example of this feature is shown in FIG. Figure 5B In this example, the avatar of the third user 351C is added to the virtual environment. In response to the system detecting that other users have shared content within their field of view, the avatar of the third user 351C is oriented toward the shared content. The system can also determine the geometry of each person's field of view and determine the position of the third user's avatar so that the third user's avatar does not block the field of view of other users.

[0053] Fig. 6A and Figure 6B Features of a system configured to position a representation of a user relative to other users within 3D environment 200 are shown. Fig. 6A A scenario is shown in which the avatars of a first user and a second user are oriented so that the users are viewing each other within a virtual environment. In a particular team or predetermined group, when the system determines that a threshold number of people are viewing each other, the system can orient the avatar of a third user entering the environment so that the avatar is positioned to view the other users. Fig. 6A A plurality of avatars with at least three users are shown, and these users have other group members within the field of view. When the system determines that a threshold number of avatars have other group members within the field of view, such as Figure 6B As shown, the system may allow a new group member to join a virtual environment having a position and orientation that allows the user to view other group members.

[0054] Figure 7 7 is a diagram illustrating aspects of a routine 700 for providing a transition of a user interface arrangement from a display of a two-dimensional image of a user to a rendering of a three-dimensional representation of the user while the user is engaging in a communication session. It should be understood by those of ordinary skill in the art that the operations of the methods disclosed herein need not be presented in any particular order, and that performing some or all of the operations in an alternative order is possible and contemplated. For ease of description and illustration, the operations have been presented in the order in which they are presented. The operations may be added, omitted, performed together, and / or performed simultaneously without departing from the scope of the appended claims.

[0055] It should also be understood that the method shown can be started or ended at any time without being executed in full. Some or all of the operations of the method and / or substantially equivalent operations can be performed by executing computer-readable instructions contained on a computer storage medium defined herein. The term "computer-readable instructions" and its variations used in the specification and claims are widely used herein to include routines, applications, application modules, program modules, programs, components, data structures, algorithms, etc. Computer-readable instructions can be implemented on various system configurations, including single-processor or multi-processor systems, minicomputers, mainframe computers, personal computers, handheld computing devices, microprocessor-based programmable consumer electronics, combinations thereof, and the like. Although the exemplary routines described below are run on a system (e.g., one or more computing devices), it is understood that the routines can be executed on any computing system, which can include any number of computers working together to perform the operations disclosed herein.

[0056] It should be understood, therefore, that the logical operations described herein are implemented as a series of computer-implemented actions or program modules running on a computing system (such as the computing system described herein) and / or interconnected machine logic circuits or circuit modules within the computing system. The implementation is a matter of choice depending on the performance and other requirements of the computing system. Therefore, the logical operations may be implemented in software, firmware, dedicated digital logic, and any combination thereof.

[0057] in addition, Figure 7 The operations shown in the figures and other figures can be implemented in association with the exemplary user interfaces and systems described herein. For example, the various devices and / or modules described herein can generate, send, receive and / or display data associated with the content of a communication session (e.g., real-time content, broadcast events, recorded content, etc.) and / or a rendering UI including one or more participants of a remote computing device, avatars, channels, chat sessions, video streams, images, virtual objects and / or applications associated with the communication session.

[0058] Routine 700 includes operation 701, in which system 100 displays a user interface that includes a rendering of a 2D image of a particular user (also referred to herein as a "selected user"). The user interface may include multiple other users displayed using 2D images, which may include real-time video or static images, or the other users may be represented by avatars, such as renderings of 3D representations such as avatars.

[0059] At operation 703 , the system may receive input causing the UI to transition to remove the 2D rendering of the user and add the 3D rendering of the user. Figure 1A and Figure 1BAn example of this transition is shown in , where a third user 10C is first displayed using a 2D image or video. In response to the input, the 2D image is removed from the user interface and a 3-dimensional representation of the third user is displayed.

[0060] At operation 705, the system updates the 3D model to include a representation of the selected user with a determined position and orientation. In some configurations, the position and orientation may be set to a landing area within the 3D model that is predetermined by a default value or by one or more policies. In some configurations, the position and orientation of the selected user's representation may be determined based on the positions of other people associated with the selected user. For example, if the selected user is associated with multiple people, such as teammates or colleagues, the person may be positioned to maintain any predetermined distance from the avatars of those teammates. The position and orientation of the selected user's representation may also be based on the gaze direction of other representations in the three-dimensional environment or the location of content shared with the selected user. For example, if the selected user's avatar is entering the 3D environment and other users are viewing shared content in the 3D environment, the selected user's representation is positioned and oriented so that it faces the shared content.

[0061] At operation 707, the system may change permissions in response to receiving input for transitioning the user interface. In response to the input, the system may analyze the three-dimensional model and determine whether access rights to the corresponding image of the selected user should be restricted or allowed. If the selected user has an associated representation within the 3D environment, the system will respect permissions that restrict all users from accessing any 2D image files or 2D image streams of the selected user. However, if the selected user is not associated with a representation within the 3D environment, the system will respect permissions that allow all users to access the 2D image files or 2D image streams of the selected user.

[0062] At operation 709, the system can modify the user interface to remove the rendering of the 2D image of the selected user based on the permissions. In addition, the system can access the 3D model to display a rendering of a 3D representation of the selected user on the user interface. The rendering of the 3D representation can be based on the position and orientation information in the 3D model.

[0063] At operation 711, the system may receive input for converting the user interface to include a 2D image and removing the 3D representation of the selected user. In this particular example, the input is configured to cause the user interface to change from Figure 1B The user interface shown switches back to Figure 1A The user interface shown.

[0064] At operation 713 , the system changes the permissions to allow access to the 2D image file or 2D image stream, and in response to the input, converts the user interface to include the 2D image and removes the 3D representation of the selected user.

[0065] At operation 715, the system may cause a transition of the user interface to include the 2D image and remove the 3D representation of the selected user. In response to the input, the system causes the user interface to change from Figure 1B The user interface shown switches back to Figure 1A The user interface shown.

[0066] Figure 8 600 is a diagram illustrating an exemplary environment 600 in which a system 602 can implement the techniques disclosed herein. It should be understood that the above-described subject matter can be implemented as a computer-controlled device, a computer process, a computing system, or an article of manufacture such as a computer-readable storage medium. The operations of the exemplary methods are represented by individual blocks and summarized with reference to these blocks. The methods are shown as a logical flow of blocks, each of which can represent one or more operations, which can be implemented in hardware, software, or a combination thereof. In the context of software, an operation represents a computer-executable instruction stored on one or more computer-readable media, which, when executed by one or more processors, enables the one or more processors to perform the operation.

[0067] Typically, computer executable instructions include routines, programs, objects, modules, components, data structures, etc. that perform specific functions or implement specific abstract data types. The order in which the operations are described should not be considered limiting, and any number of the operations may be performed in any order, combined in any order, subdivided into multiple sub-operations, and / or performed in parallel to implement the process. The process may be performed by resources associated with one or more devices, such as one or more internal or external CPUs or GPUs, and / or one or more hardware logic units, such as a field programmable gate array ("FPGA"), a digital signal processor ("DSP"), or other type of accelerator.

[0068] All of the above methods and processes may be embodied in, and fully automated via, software code modules executed by one or more general-purpose computers or processors. The code modules may be stored in any type of computer-readable storage medium or other computer storage device, such as the computer-readable storage medium or computer storage device described below. Some or all of the methods may alternatively be embodied in special-purpose computer hardware, such as the special-purpose computer hardware described below.

[0069] Any routine description, element or block in the flowcharts described herein and / or depicted in the accompanying drawings should be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing a specific logical function or element in the routine. Alternative implementations are included within the scope of the examples described herein, in which elements or functions may be deleted, or executed in a different order than that shown or discussed, including substantially simultaneously or in reverse order, depending on the functions involved as understood by those skilled in the art.

[0070] In some implementations, system 602 can be used to collect, analyze, and share data displayed to users of communication session 604. As shown, communication session 603 can be implemented between multiple client computing devices 606(1)-606(N) (where N is a number having a value of 2 or greater) associated with or as part of system 602. Client computing devices 606(1)-606(N) enable users (also referred to as individuals) to participate in communication session 603.

[0071] In this example, communication session 603 is hosted by system 602 over one or more networks 608. That is, system 602 can provide services that enable users of client computing devices 606(1) to 606(N) to participate in communication session 603 (e.g., via live viewing and / or recorded viewing). Thus, "participants" of communication session 603 can include users and / or client computing devices (e.g., multiple users can participate in a communication session in a room using a single client computing device), each of which can communicate with other participants. Alternatively, communication session 603 can be hosted by one of client computing devices 606(1) to 606(N) utilizing peer-to-peer technology. System 602 can also host chat conversations and other team collaboration functionality (e.g., as part of an application suite).

[0072] In some embodiments, such chat conversations and other team collaboration functions are considered external communication sessions that are distinct from the communication session 603. The computing system 602 that collects participant data in the communication session 603 may be able to link to such external communication sessions. Thus, the system may receive information, such as date, time, session details, etc., that enables connection to such external communication sessions. In one example, a chat conversation may be conducted in accordance with the communication session 603. Additionally, the system 602 may host a communication session 603 that includes at least a plurality of participants that are co-located at a meeting location (e.g., a conference room or auditorium) or located at different locations.

[0073] In the examples described herein, client computing devices 606(1) to 606(N) participating in a communication session 603 are configured to receive and render communication data for display on a user interface of a display screen. The communication data may include a collection of various instances or streams of real-time content and / or recorded content. The collection of various instances or streams of real-time content and / or recorded content may be provided by one or more cameras (e.g., video cameras). For example, individual streams of real-time content or recorded content may include media data associated with a video feed provided by a camera (e.g., audio and visual data capturing the appearance and voice of users participating in the communication session). In some embodiments, the video feed may include such audio and visual data, one or more still images, and / or one or more avatars. The one or more still images may also include one or more avatars.

[0074] Another example of an individual stream of real-time content or recorded content may include media data including avatars of users participating in a communication session and audio data capturing the users' voices. Yet another example of an individual stream of real-time content or recorded content may include media data including a file displayed on a display screen and audio data capturing the users' voices. Thus, individual streams of real-time content or recorded content within the communication data enable remote conferencing between a group of people and content sharing within the group of people. In some embodiments, individual streams of real-time content or recorded content within the communication data may originate from multiple co-located cameras located in a space such as a room for recording or live streaming a presentation including one or more individuals making a presentation and one or more individuals consuming the presented content.

[0075] Participants or attendees can view the content of communication session 603 in real time as the activity occurs, or alternatively, view the content of communication session 603 via recording at a later time after the activity occurs. In the examples described herein, client computing devices 606(1) to 606(N) participating in communication session 603 are configured to receive and render communication data for display on a user interface of a display screen. The communication data may include a collection of various instances or streams of real-time content and / or recorded content. For example, an individual stream of content may include media data associated with a video feed (e.g., audio and visual data that captures the appearance and voice of users participating in the communication session). Another example of an individual stream of content may include media data that includes an avatar of a user participating in the conference session and audio data that captures the user's voice. Yet another example of an individual stream of content may include media data that includes content items displayed on a display screen and / or audio data that captures the user's voice. Thus, the individual streams of content within the communication data enable a group of people dispersed in remote locations to conveniently hold a meeting or broadcast a presentation.

[0076] A participant or attendee of a communication session is a person who is within range of a camera or other image and / or audio capture device, so that the person's movements and / or sounds generated while the person is viewing and / or listening to content shared via the communication session can be captured (e.g., recorded). For example, a participant may be sitting in a crowd, viewing shared content in real time at a broadcast location where a stage presentation is taking place. Alternatively, a participant may be sitting in an office conference room, viewing the shared content of a communication session with other colleagues via a display screen. Even more, a participant may be alone in his or her office or home, sitting or standing in front of a personal device (e.g., a tablet device, a smartphone, a computer, etc.) viewing the shared content of a communication session.

[0077] Figure 8 System 602 includes device 610. Device 610 and / or other components of system 602 may include distributed computing resources that communicate with each other and / or with client computing devices 606(1) to 606(N) via one or more networks 608. In some examples, system 602 may be a stand-alone system that is tasked with managing aspects of one or more communication sessions (e.g., communication session 603). As an example, system 602 may be managed by an entity such as Slack, WEBEX, GOTOMEETING, GOOGLEHANGOUTS, etc.

[0078] The network 608 may include, for example, a public network (e.g., the Internet), a private network (e.g., an institutional and / or personal intranet), or some combination of a private network and a public network. The network 608 may also include any type of wired and / or wireless network, including but not limited to a local area network ("LAN"), a wide area network ("WAN"), a satellite network, a wired network, a Wi-Fi network, a WiMax network, a mobile communication network (e.g., 3G, 4G, etc.), or any combination thereof. The network 608 may use communication protocols, including packet-based and / or datagram-based protocols, such as the Internet Protocol ("IP"), the Transmission Control Protocol ("TCP"), the User Datagram Protocol ("UDP"), or other types of protocols. In addition, the network 608 may also include multiple devices that facilitate network communications and / or form the basis of network hardware, such as switches, routers, gateways, access points, firewalls, base stations, repeaters, backbone devices, etc.

[0079] In some examples, network 608 may also include devices that enable connection to a wireless network, such as a wireless access point ("WAP"). Examples support connection via a WAP that sends and receives data via various electromagnetic frequencies (e.g., wireless frequencies), including WAPs that support the Institute of Electrical and Electronics Engineers ("IEEE") 802.11 standards (e.g., 802.11g, 802.11n, 802.11ac, etc.) and other standards.

[0080] In various examples, device 610 may include one or more computing devices that operate in a cluster or other grouping configuration to share resources, balance loads, improve performance, provide failover support or redundancy, or for other purposes. For example, device 610 may belong to various types of devices, such as traditional server type devices, desktop computer type devices, and / or mobile type devices. Therefore, although shown as a single type of device or server type device, device 610 may include a variety of device types and is not limited to a particular type of device. Device 610 may represent, but is not limited to, a server computer, a desktop computer, a web server computer, a personal computer, a mobile computer, a laptop computer, a tablet computer, or any other type of computing device.

[0081] The client computing device (e.g., one of client computing devices 606(1)-606(N)) (each client computing device also referred to herein as a "data processing system") can be of various types of devices, which can be the same as or different from device 610, such as a traditional client type device, a desktop type device, a mobile type device, a dedicated type device, an embedded type device, and / or a wearable type device. Thus, the client computing device can include, but is not limited to, a desktop computer, a gaming console and / or gaming device, a tablet computer, a personal data assistant ("PDA"), a mobile phone / tablet hybrid device, a laptop computer, a telecommunications device, a computer navigation type client computing device (e.g., a satellite-based navigation system, including a global positioning system ("GPS") device), a wearable device, a virtual reality ("VR") device, an augmented reality ("AR") device, an implanted computing device, an automotive computer, a network television, a thin client, a terminal, an Internet of Things ("IoT") device, a workstation, a media player, a personal video recorder ("PVR"), a set-top box, a camera, an integrated component (e.g., a peripheral device) for inclusion in a computing device, an appliance, or any other type of computing device. Furthermore, the client computing device may include a combination of the previously listed client computing device examples, such as, for example, a desktop computer-type device or a mobile-type device combined with a wearable device, etc.

[0082] Client computing devices 606(1) to 606(N) of various classes and device types may represent any type of computing device having one or more data processing units 692 operably connected to a computer-readable medium 694, for example, via a bus 616, which in some cases may include a system bus, a data bus, an address bus, a PCI bus, a Mini-PCI bus, and one or more of any variety of local, peripheral, and / or independent buses.

[0083] The executable instructions stored on the computer-readable medium 694 may include, for example, the operating system 619 , the client module 620 , the profile module 622 , and other modules, programs, or applications that may be loaded and executed by the data processing unit 692 .

[0084] The client computing devices 606(1)-606(N) may also include one or more interfaces 624 to enable communication between the client computing devices 606(1)-606(N) and other networked devices (e.g., device 610) over the network 608. Such network interfaces 624 may include one or more network interface controllers (NICs) or other types of transceiver devices for sending and receiving communications and / or data over the network. In addition, the client computing devices 606(1)-606(N) may include input / output ("I / O") interfaces (devices) 626 that enable communication with input / output devices, such as user input devices, including peripheral input devices (e.g., game controllers, keyboards, mice, pens, voice input devices (e.g., microphones), cameras for acquiring and providing video feeds and / or still images, touch input devices, gesture input devices, etc.) and / or output devices, including peripheral output devices (e.g., displays, printers, audio speakers, tactile output devices, etc.). Figure 8 Client computing device 606 ( 1 ) is shown connected in some manner to a display device (eg, display screen 629 (N)) that can display a UI according to the techniques described herein.

[0085] exist Figure 8 In the exemplary environment 600 of FIG. 6 , client computing devices 606 ( 1 ) to 606 (N) can use their respective client modules 620 to connect to each other and / or other external devices to participate in a communication session 603 or to contribute activities to a collaborative environment. For example, a first user can use client computing device 606 ( 1 ) to communicate with a second user of another client computing device 606 ( 2 ). When executing client module 620, the users can share data, which can cause client computing device 606 ( 1 ) to connect to system 602 and / or other client computing devices 606 ( 2 ) to 606 (N) via network 608 .

[0086] The client computing devices 606(1) to 606(N) may use their respective profile modules 622 to generate participant profiles ( Figure 8602) and provide the participant profile to other client computing devices and / or devices 610 of system 602. The participant profile may include one or more of an identity of a user or group of users (e.g., name, unique identifier (“ID”), etc.), user data such as personal data (e.g., machine data such as location (e.g., IP address, room in a building, etc.), and technical capabilities, etc. The participant profile may be used to register participants for a communication session.

[0087] like Figure 8 As shown, device 610 of system 602 includes server module 630 and output module 632. In this example, server module 630 is configured to receive media streams 634(1) to 634(N) from individual client computing devices (e.g., client computing devices 606(1) to 606(N)). As described above, the media streams may include video feeds (e.g., audio and visual data associated with a user), audio data to be output along with a presentation of an avatar of the user (e.g., an audio-only experience that does not transmit the user's video data), text data (e.g., a text message), file data, and / or screen sharing data (e.g., a document, a slideshow, an image, a video displayed on a display screen, etc.), etc. Thus, server module 630 is configured to receive a collection of various media streams 634(1) to 634(N) (the collection referred to herein as "media data 634") during a real-time viewing communication session 603. In some scenarios, not all client computing devices participating in communication session 603 provide media streams. For example, the client computing device may be a consuming device or a “listening” device only, such that it only receives content associated with the communication session 603 but does not provide any content to the communication session 603 .

[0088] In various examples, the server module 630 can select aspects of the media stream 634 to be shared with individual client computing devices of the participating client computing devices 606(1) to 606(N). Thus, the server module 630 can be configured to generate session data 636 based on the stream 634 and / or pass the session data 636 to the output module 632. The output module 632 can then transmit communication data 639 to the client computing devices (e.g., the client computing devices 606(1) to 606(3) participating in the real-time viewing communication session). The communication data 639 can include video, audio, and / or other content data provided by the output module 632 based on content 650 associated with the output module 632 and based on the received session data 636. The content 650 can include the stream 634 or other shared data, such as an image file, a spreadsheet file, a slide show, a document, etc. The stream 634 can include a video component that depicts an image captured by the I / O device 626 on each client computer.

[0089] As shown, output module 632 sends communication data 639(1) to client computing device 606(1), sends communication data 639(2) to client computing device 606(2), sends communication data 639(3) to client computing device 606(3), etc. The communication data 639 sent to the client computing devices may be the same or may be different (e.g., the positioning of the flow of content within the user interface may vary from device to device).

[0090] In various implementations, the device 610 and / or the client module 620 may include a GUI presentation module 640. The GUI presentation module 640 may be configured to analyze the communication data 639 to be transmitted to one or more client computing devices 606. Specifically, the UI presentation module 640 at the device 610 and / or the client computing device 606 may analyze the communication data 639 to determine an appropriate manner to display the video, image, and / or content on the display screen 629 of the associated client computing device 606. In some implementations, the GUI presentation module 640 may provide the video, image, and / or content to a presentation GUI 646 rendered on the display screen 629 of the associated client computing device 606. The GUI presentation module 640 may cause the presentation GUI 646 to be rendered on the display screen 629. The presentation GUI 646 may include the video, image, and / or content analyzed by the GUI presentation module 640.

[0091] In some implementations, the presentation GUI 646 may include multiple sections or grids that may render or include video, images, and / or content for display on the display screen 629. For example, a first section of the presentation GUI 646 may include a video feed of a presenter or individual, and a second section of the presentation GUI 646 may include a video feed of an individual consuming conference information provided by the presenter or individual. The GUI presentation module 640 may populate the first section and the second section of the presentation GUI 646 in a manner that appropriately mimics an environmental experience that the presenter and individual may share.

[0092] In some embodiments, the GUI presentation module 640 can zoom in or provide a zoomed view of the individual represented by the video feed to highlight the individual's reaction to the presenter, such as facial features. In some embodiments, the presentation GUI 646 can include video feeds of multiple participants associated with a meeting (e.g., a general communication session). In other embodiments, the presentation GUI 646 can be associated with a channel, such as a chat channel, an enterprise team channel, etc. Thus, the presentation GUI 646 can be associated with an external communication session that is different from a general communication session.

[0093] Fig. 9A diagram showing exemplary components of an exemplary device 700 (also referred to herein as a "computing device") that is configured to generate data for some user interfaces disclosed herein. The device 700 can generate data that can include one or more sections that can render or include video, images, virtual objects, and / or content for display on a display screen 629. The device 700 can represent one of the devices described herein. Additionally or alternatively, the device 700 can represent one of the client computing devices 606.

[0094] As shown, device 700 includes one or more data processing units 702, computer-readable media 704, and communication interface 706. The components of device 700 may be operatively connected, for example, via bus 709, which may include one or more of a system bus, a data bus, an address bus, a PCI bus, a Mini-PCI bus, and any kind of local, peripheral, and / or independent bus.

[0095] As utilized herein, a data processing unit (e.g., data processing unit 702 and / or data processing unit 692) may represent, for example, a CPU-type data processing unit, a GPU-type data processing unit, a field programmable gate array ("FPGA"), another type of DSP, or other hardware logic components that may be driven by a CPU in some cases. For example, but not limited to, illustrative types of hardware logic components that may be utilized include application specific integrated circuits ("ASICs"), application specific standard products ("ASSPs"), systems on chips ("SOCs"), complex programmable logic devices ("CPLDs"), and the like.

[0096] As used herein, computer-readable media (e.g., computer-readable media 704 and computer-readable media 694) can store instructions that can be executed by a data processing unit. The computer-readable medium can also store instructions that can be executed by an external data processing unit (e.g., an external CPU, an external GPU) and / or instructions that can be executed by an external accelerator (e.g., an FPGA-type accelerator, a DSP-type accelerator, or any other internal or external accelerator). In various examples, at least one CPU, GPU, and / or accelerator is incorporated into the computing device, while in some examples, one or more of the CPU, GPU, and / or accelerator is located outside the computing device.

[0097] Computer readable media (also referred to herein as computer readable media) may include computer storage media and / or communication media. Computer storage media may include one or more of volatile memory, non-volatile memory and / or other persistent and / or secondary computer storage media, removable and non-removable computer storage media, implemented in any method or technology for storing information such as computer readable instructions, data structures, program modules or other data. Thus, computer storage media includes media in tangible and / or physical form contained in a device and / or in a hardware component that is part of the device or external to the device, including, but not limited to, random access memory ("RAM"), static random access memory ("SRAM"), dynamic random access memory ("DRAM"), phase change memory ("PCM"), read-only memory ("ROM"), erasable programmable read-only memory ("EPROM"), electrically erasable programmable read-only memory ("EEPROM"), flash memory, compact disc read-only memory ("CD-ROM"), digital versatile disc ("DVD"), optical cards or other optical storage media, magnetic cassettes, magnetic tape, magnetic disk storage devices, magnetic cards or other magnetic storage devices or media, solid-state memory devices, storage arrays, network attached storage devices, storage area networks, hosted computer storage devices, or any other storage memory, storage device, and / or storage medium that can be used to store and maintain information for access by a computing device. Computer storage media may also be referred to herein as computer-readable storage media, non-transitory computer-readable storage media, non-transitory computer-readable media, or computer storage media.

[0098] In contrast to computer storage media, communication media may embody computer-readable instructions, data structures, program modules, or other data in a modulated data signal (e.g., carrier wave) or other transport mechanism. As defined herein, computer storage media does not include communication media. That is, computer storage media does not include communication media that consists solely of a modulated data signal, carrier wave, or propagation signal itself.

[0099] The communication interface 706 may represent, for example, a network interface controller ("NIC") or other type of transceiver device for sending and receiving communications over a network. In addition, the communication interface 706 may include one or more cameras and / or audio devices 722 to generate video feeds and / or still images, etc.

[0100] In the illustrated example, computer-readable medium 704 includes a data repository 708. In some examples, data repository 708 includes a data storage device, such as a database, a data warehouse, or other type of structured or unstructured data storage device. In some examples, data repository 708 includes a corpus and / or a relational database having one or more tables, indexes, stored procedures, etc. to enable data access, including, for example, one or more of a Hypertext Markup Language ("HTML") table, a Resource Description Framework ("RDF") table, a Web Ontology Language ("OWL") table, and / or an Extensible Markup Language ("XML") table.

[0101] The data repository 708 may store data for the operation of processes, applications, components, and / or modules stored in the computer-readable medium 704 and / or executed by the data processing unit 702 and / or accelerator. For example, in some examples, the data repository 708 may store session data 710 (e.g., Figure 8 The data repository 708 may also include context data 714, such as content including video, audio, or other content to be rendered and displayed on one or more display screens 629. The hardware data 711 may define aspects of any device, such as multiple display screens of a computer. The context data 714 may define any type of activity or state associated with individual users 10A-10L, each of which is associated with an individual video stream in the multiple video streams 634. For example, the context data may define a person's level in an organization, the relationship between each person's level and the levels of other people, a person's performance level, or any other activity or state information that may be used to determine the location of a person's rendering within the virtual environment. This contextual information can also be fed into any model to help emphasize key words spoken by a person of a certain level, highlight the UI when background voices of a person of a certain level are detected, or change the emotion display in a specific way when a person of a certain level is detected to have a certain emotion.

[0102] Alternatively, some or all of the above data may be stored in a separate memory 716 on one or more data processing units 702, such as a CPU-type processor, a GPU-type processor, an accelerator of an FPGA type, an accelerator of a DSP type, and / or a memory on another accelerator. In this example, the computer-readable medium 704 also includes an operating system 718 and an application programming interface 710 (API) configured to expose the functions and data of the device 700 to other devices. In addition, the computer-readable medium 704 includes one or more modules, such as a server module 730, an output module 732, and a GUI rendering module 740, but the number of modules shown is only an example, and the number may vary. That is, the functions described in association with the modules shown herein may be performed by a smaller number of modules or a larger number of modules on a single device, or distributed on multiple devices.

[0103] The following exemplary clauses supplement this disclosure.

[0104] Exemplary clause A. A method for converting a user interface (101) from a rendering of a two-dimensional image (151C) of a user (10C) to a rendering of a three-dimensional representation (251C) of the user (10C) while the user (10C) is participating in a communication session (604), such as Figure 1A to Figure 1B The method is configured to be executed on a system (100), the method comprising: causing display of a user interface (101) including a rendering of a two-dimensional image (151C) of a user (10C), wherein the rendering of the two-dimensional image (151C) is from a two-dimensional image file (310), wherein the user interface (101) simultaneously displays the rendering of the two-dimensional image (151C) of the user (10C) and renderings of other users (10A-10B) participating in a communication session (604) with the user (10C); this is included in Figure 1A or Figure 2AIn , users start in gallery mode, which is defined as a "rendering of a two-dimensional image." Other users can be in gallery mode or immersive mode. The gallery mode may be on a wall of a VR room or in a standard grid mode; receiving input for causing a transition from a rendered display of a two-dimensional image (151C) of a user (10C) to a rendered three-dimensional representation (251C) of the user (10C), wherein the input causes the system to access a three-dimensional model (320) defining a position and orientation of the three-dimensional representation (251C) of the user (10C) within a three-dimensional environment (200), the input being intended to cause a transition from the gallery mode to the immersive mode, the input causing the system to access the 3D model; and in response to the input for causing a transition from a rendered display of a two-dimensional image of the user to a rendered three-dimensional representation of the user, causing a modification to the user interface arrangement to remove the rendering (151) of the two-dimensional image of the user and add the display of the three-dimensional representation (251C) of the user in the rendering of the three-dimensional environment (200) using the position and orientation defined in the three-dimensional model (320), which will remove the 2D image and display the 3D model of the user in the 3D environment (e.g., room). The 3D representation uses the 3D model data to determine the user's position and orientation in the 3D environment.

[0105] Exemplary Clause B. This embodiment is a conversion from 3D to 2D using a single device, such as Figure 1B to Figure 1A The conversion shown is a method for converting a user interface (101) from a rendering of a three-dimensional representation (251C) of a user (10C) to a two-dimensional image (151C) of the user (10C) while the user (10C) is participating in a communication session (604), the method being configured to be executed on a system (100), the method comprising: causing display of a user interface (101) including a rendering of a three-dimensional representation (251C) of the user (10C), wherein the user interface (101) simultaneously displays the three-dimensional representation (251C) of the user (10C) and renderings of other users (10A-10B) participating in the communication session (604) with the user (10C), the renderings of the other users being either 2D images or 3D representations; the feature being included Figure 1A or Figure 2AIn the embodiment of the present invention, a user starts in an immersive mode, which is defined as a rendering of a 3D rendering, and other users may be in a gallery mode or an immersive mode; receiving an input for causing a transition from a display of a rendering of a three-dimensional representation (251C) of a user (10C) to a two-dimensional image (151C) of the user (10C), the input being intended to cause a transition from the immersive mode to the gallery mode, the input causing the system to transition from accessing a 3D model to accessing a 2D image file; and in response to the input for causing a transition from a display of a rendering of a three-dimensional representation (251C) of the user (10C) to a two-dimensional image (151C) of the user (10C), causing a modification to a user interface arrangement to remove the rendering of the three-dimensional representation (251C) of the user (10C) and add the display of the two-dimensional image (151C) of the user (10C).

[0106] Exemplary clause C. A method for converting a user interface (101), such as from FIG. 3A to FIG. 3B The method is configured to be executed on a system (100), in coordination with a change in the system operating mode from a first operating mode, in which a rendering of a user (10C) as a 2D image (151C) is displayed to other users (10A-10B, 10D-10E) of a communication session, to a second operating mode, in which a rendering of a user (10C) as a 3D representation (251C) of the user (10C) is displayed to other users of the communication session while the user (10C) is participating in the communication session (604), the method comprising:

[0107] In response to being in a first operating mode, causing display of a user interface (101) on a display device of a computer (11C) associated with a user (10C), the user interface (101) comprising rendering of 2D images (151A-151B) of a first group of users (10A-10B) of other users participating in a communication session (604) with the user (10C), the user interface (101) further comprising rendering of 3D representations (251A-251B) of a second group of users (10D-10E) of other users participating in the communication session (604) with the user (10C), wherein a first viewing perspective of the rendering of the 3D representations (251A-251B) is based on a position (200) of a virtual object (351A-351B) of the 3D representations (251A-251B) within the 3D environment relative to a position of a virtual camera (350); Figure 3A, a third user starts in a gallery mode in which the 3D environment is a smaller window showing the front face of the avatar, receives input for causing the system operating mode to change from a first operating mode in which the user (10C) is displayed to other users (10A-10B, 10D-10E) of the communication session as a rendering of a 2D image (151C) to a second operating mode in which the user (10C) is displayed to other users of the communication session as a rendering of a 3D representation (251C) of the user (10C) while the user (10C) is participating in the communication session (604); Figure 1A and Figure 1B As shown, and in response to an input for causing the system operating mode to change from a first operating mode to a second operating mode, causing a modification of a user interface (101) to enlarge a rendering of a 3D environment (200) relative to a first size of the rendering of the 3D environment (200) in the first operating mode, wherein a viewing perspective of a user (10C) of the 3D environment (200) is based on a position and orientation of a virtual object (351C) associated with a 3D representation (251C) of the user (10C), and wherein a second viewing perspective of the 3D representations (251A-251B) is based on a first size of the rendering of the 3D environment (200) in the first operating mode. For the position of the virtual objects (351A-351B) of the 3D representations (251A-251B) within the 3D environment (200), the position and orientation of the virtual objects (351C) associated with the 3D representations (251C) of the users (10C), in this example the 3D environment (200) showing the back of the avatar, wherein the second operating mode causes the user interface to include the 2D images (151A-151B) of the first group of users (10A-10B) in the virtual objects (275) formed as a virtual display screen positioned with the virtual environment (200).

[0108] Exemplary clause D. A method for converting a user interface (101), such as from Figure 3B to Figure 3A , coordinated with a change in the system operating mode from a second operating mode to a first operating mode, in which, while the user (10C) is participating in a communication session (604), a rendering of the user as a 3D representation (251C) of the user (10C) is displayed to other users of the communication session, as shown in FIG. Figure 1B As shown, in a first mode of operation, a user (10C) is displayed to other users (10A-10B, 10D-10E) of a communication session as a rendering of a 2D image (151C), such as Figure 1A As shown, the method is configured to be executed on the system (100), the method comprising: in response to being in the second operation mode, as Figure 3BAs shown, causing display of a user interface (101) on a display device of a computer (11C) associated with a user (10C), the user interface (101) including a first size of a rendering of a 3D environment (200), wherein the user's (10C) first viewing perspective of the 3D environment (200) is based on a position and orientation of a virtual object (351C) associated with a 3D representation (251C) of the user (10C), for example, wherein other avatars are positioned relative to the user's avatar, wherein a second operating mode causes the user interface to include 2D images (151A-151B) of the first group of users (10A-10B) in a virtual object (275) formed as a virtual display screen positioned with the virtual environment (200). Receiving input for causing the system operating mode to change from a second operating mode to a first operating mode in which, while the user (10C) is participating in a communication session (604), a rendering of the user as a 3D representation (251C) of the user (10C) is displayed to other users of the communication session, such as Figure 1B As shown, in a first mode of operation, a user (10C) is displayed to other users (10A-10B, 10D-10E) of a communication session as a rendering of a 2D image (151C), such as Figure 1A as shown; and in response to an input for causing the system operating mode to change from a second operating mode to a first operating mode, causing modification to the user interface (101) to reduce the rendering of the 3D environment (200) from a first size, wherein in the first operating mode, the user interface (101) includes rendering of 3D representations (251A-251B) of a second group of users (10D-10E) of other users participating in a communication session (604) with the user (10C), wherein a second viewing perspective of the rendering of the 3D representations (251A-251B) is based on the position of virtual objects (351A-351B) of the 3D representations (251A-251B) within the 3D environment (200) relative to the position of the virtual camera (350), wherein the first operating mode causes the 2D images (151A-151B) of the first group of users (10A-10B) to be displayed without displaying the virtual objects (275).

[0109] Exemplary clause E, a method as described in any exemplary clause, wherein the input for causing the transition includes input data indicating that the user is interacting with a computing device having a keyboard and a mouse, wherein based on detecting the keyboard and mouse communicating with a client computing device associated with the user (10C), the permission data is modified to allow the transition, and wherein the system is configured to allow the transition only in response to determining that the permission data allows the transition of the user interface based on detecting the keyboard and mouse communicating with the client computing device.

[0110] Exemplary clause F, a method as described in any exemplary clause, wherein the input for causing the transition includes input data indicating that the user is interacting with a desktop computing device with a keyboard, wherein, based on detecting a keyboard communicating with a desktop computing device associated with the user (10C), the permission data is modified to allow the transition, wherein the system is configured to allow the transition only in response to determining that the permission data allows the transition of the user interface based on detecting a keyboard communicating with a desktop computing device associated with the user, wherein the transition of the user interface is allowed only in response to detecting that the user remains using the desktop computing device with a keyboard.

[0111] Exemplary clause G, a method as described in any exemplary clause, wherein the input for causing the transition includes input data indicating that the user is interacting with a computer in the form of a head-mounted display device, the computer being used to display an augmented reality or virtual reality computing environment, and wherein the system is configured to allow the transition only in response to determining that permission data is based on detecting that the user remains using the computer in the form of a head-mounted display device to allow the transition of the user interface.

[0112] Finally, although various configurations have been described using language specific to structural features and / or methodological acts, it should be understood that the subject matter defined in the appended expressions is not necessarily limited to the specific features or acts described. Rather, the specific features and acts are disclosed as exemplary forms of implementing the claimed subject matter.

Claims

1. A method for converting a user interface from a rendering of a two-dimensional image of a user to a rendering of a three-dimensional representation of the user while the user is participating in a communication session, the method being configured to be executed on a system, the method comprising: causing display of the user interface including a rendering of the two-dimensional image of the user, wherein the rendering of the two-dimensional image is from a two-dimensional image file, wherein the user interface simultaneously displays the rendering of the two-dimensional image of the user and renderings of other users participating in the communication session with the user; receiving an input for causing a transition from display of a rendering of the two-dimensional image of the user to a rendering of the three-dimensional representation of the user, wherein the input causes the system to access a three-dimensional model defining a position and orientation of the three-dimensional representation of the user within a three-dimensional environment; and In response to the input for causing the transition from display of a rendering of the two-dimensional image of the user to rendering of the three-dimensional representation of the user, causing modification of the user interface arrangement to remove rendering of the two-dimensional image of the user and add display of the three-dimensional representation of the user in the rendering of the three-dimensional environment using a position and orientation defined in the three-dimensional model.

2. The method according to claim 1, wherein: The input for causing the transition includes input data indicating that the user is interacting with a computing device having a keyboard and a mouse, wherein based on detecting the keyboard and the mouse in communication with a client computing device associated with the user, the permission data is modified to allow the transition, wherein the system is configured to allow the transition only in response to determining that the permission data allows the transition of the user interface based on detecting the keyboard and the mouse in communication with the client computing device.

3. The method according to claim 1, wherein: The input for causing the transition includes input data indicating that the user is interacting with a desktop computing device having a keyboard, wherein, based on detecting the keyboard in communication with the desktop computing device associated with the user, the permission data is modified to allow the transition, wherein the system is configured to allow the transition only in response to determining that the permission data allows the transition of the user interface based on detecting the keyboard in communication with the desktop computing device associated with the user.

4. The method according to claim 1, wherein: The position and orientation of the three-dimensional representation are configured to provide display of the representation such that, responsive to determining that a threshold number of representations of other users participating in the communication session are positioned as if viewing a rendering of the shared content, the representations appear to be viewing a rendering of the shared content.

5. The method according to claim 1, wherein: The positions and orientations of the three-dimensional representations are configured to provide display of the representations such that, in response to determining a threshold number of representations of other users participating in the communication session are positioned and oriented as if viewing each other, the representations appear to be viewing representations of the other users participating in the communication session.

6. The method according to claim 1, wherein: The input for causing the transition includes input data indicating that the user has put on a head-mounted display device for the communication session, wherein the input data is generated from a motion sensor mounted to the head-mounted display device, the sensor detecting movement of the user indicating that the user has put on the head-mounted display device.

7. The method according to claim 1, wherein: The input for causing the transition includes input data indicating a predetermined device type, wherein, based on detecting that the predetermined device type is a desktop computing device, permission data is modified to allow the transition, and wherein the system is configured to allow the transition only in response to determining that the permission data allows the transition of the user interface based on detecting the predetermined device type.

8. A system for converting a user interface from a rendering of a two-dimensional image of a user to a rendering of a three-dimensional representation of the user while the user is participating in a communication session, the system comprising: one or more processing units; as well as A computer-readable storage medium having computer-executable instructions encoded thereon to cause the one or more processing units to perform a method comprising: causing display of the user interface including a rendering of the two-dimensional image of the user, wherein the rendering of the two-dimensional image is from a two-dimensional image file, wherein the user interface simultaneously displays the rendering of the two-dimensional image of the user and renderings of other users participating in the communication session with the user; receiving an input for causing a transition from display of a rendering of the two-dimensional image of the user to a rendering of the three-dimensional representation of the user, wherein the input causes the system to access a three-dimensional model defining a position and orientation of the three-dimensional representation of the user within a three-dimensional environment; and In response to the input for causing the transition from display of a rendering of the two-dimensional image of the user to rendering of the three-dimensional representation of the user, causing modification of the user interface arrangement to remove rendering of the two-dimensional image of the user and add display of the three-dimensional representation of the user in the rendering of the three-dimensional environment using a position and orientation defined in the three-dimensional model.

9. The system according to claim 8, wherein: The input for causing the transition includes input data indicating that the user is interacting with a computing device having a keyboard and a mouse, wherein based on detecting the keyboard and the mouse in communication with a client computing device associated with the user, the permission data is modified to allow the transition, wherein the system is configured to allow the transition only in response to determining that the permission data allows the transition of the user interface based on detecting the keyboard and the mouse in communication with the client computing device.

10. The system according to claim 8, wherein: The input for causing the transition includes input data indicating that the user is interacting with a computing device having a keyboard, wherein based on detecting the keyboard in communication with a client computing device associated with the user, permission data is modified to allow the transition, wherein the system is configured to allow the transition only in response to determining that the permission data allows the transition of the user interface based on detecting the keyboard in communication with the client computing device associated with the user.

11. The system according to claim 8, wherein: The position and orientation of the three-dimensional representation are configured to provide display of the representation such that, responsive to determining that a threshold number of representations of other users participating in the communication session are positioned as if viewing a rendering of the shared content, the representations appear to be viewing a rendering of the shared content.

12. The system according to claim 8, wherein: The positions and orientations of the three-dimensional representations are configured to provide display of the representations such that, in response to determining that a threshold number of representations of other users participating in the communication session are positioned and oriented as if viewing each other, the representations appear to be viewing representations of the other users participating in the communication session.

13. The system according to claim 8, wherein: The input for causing the transition includes input data indicating that the user has put on a head-mounted display device for the communication session, wherein the input data is generated from a motion sensor mounted to the head-mounted display device, the sensor detecting movement of the user indicating that the user has put on the head-mounted display device.

14. The system according to claim 8, wherein: The input for causing the transition includes input data indicating a predetermined device type, wherein, based on detecting that the predetermined device type is a desktop computing device, permission data is modified to allow the transition, and wherein the system is configured to allow the transition only in response to determining that the permission data allows the transition of the user interface based on detecting the predetermined device type.

Images