Adaptive adjustment of perspective views to improve detail awareness of user associated with target entity of virtual environment

Abstract

A system provides adaptive adjustments to perspective views for improved perception of details for a user associated with a target entity of a virtual environment. The system can generate a customized three-dimensional (3D) view for each individual user participating in the communication session. The system is capable of generating a customized three-dimensional view for each individual user without modifying the 3D model of the virtual environment, thus being capable of maintaining the 3D environment while each participant may have an adjusted angle and position for various virtual objects. The system is capable of adaptively adjusting the angle or position for an entity in a viewing angle of view, or changing the scale of a perspective view for a target entity. The adjustment can maximize detail awareness for each participant of the communication session according to the viewpoint of each viewer. These adjustments can be made simultaneously while maintaining participants arranged in specific spatial relationships without changing.

Description

Background Art

[0001] The use of metaverse environments for online meeting applications is becoming ubiquitous. Participants in online meetings now meet in three-dimensional virtual environments and share content within these virtual environments. Although there are many benefits compared to other forms of collaboration, using 3D environments to share content comes with many disadvantages.

[0002] When conducting a meeting in a 3D immersive virtual environment, depending on the user's position and the virtual camera position, it may be difficult to see the presented content, parts of virtual objects, user activities, user gestures, or facial expressions. Important information and social signals conveyed in a meeting (which can come from facial gestures or body language) may be obscured or blocked depending on the position of the person's avatar. In one example, a virtual object in the foreground may obscure important visual details of a virtual object in the background.

[0003] In another example, when a group of users have avatars positioned around a virtual computer screen, not all users may be able to share the same experience because some user avatars may be positioned in front of the virtual computer screen, while other user avatars may be positioned near the sides of the virtual computer screen. Users with avatars located near the sides of the virtual computer screen may miss important information because the rendering on the virtual computer screen may appear compressed. Furthermore, when conferencing across 2D and 3D virtual environments across devices, users with avatars positioned near the sides of the virtual computer screen may miss details of the desktop user's 2D display.

[0004] These shortcomings can lead to inefficient interaction between computing devices and users. In addition, the above-mentioned shortcomings of existing systems can lead to loss of user engagement. Computing devices that do not promote user engagement or worse contribute to user engagement loss can lead to productivity loss and inefficiency relative to multiple computing resources. For example, when a user becomes tired or absent-minded, when shared content is missed or ignored, the user may need to refer to other resources, such as documents or participate in other forms of communication. When a viewer misses important points or clues during a live conference, the missed content may need to be resent. Such activities can lead to inefficient or repeated use of networks, processors, memory or other computing resources. Therefore, it is necessary to develop improved solutions to help make the user experience more like a face-to-face meeting and more attractive. In addition, when information in a metaverse environment is not properly displayed, when such content is displayed on a virtual object such as a virtual screen, it is assumed that some content hidden to some users will be exposed unintentionally. Summary of the invention

[0005] The technology disclosed herein provides adaptive adjustment of perspective views for improving the perception of details for users associated with target entities of a virtual environment. A system is capable of generating a customized three-dimensional 3D view for each individual user participating in a communication session. The system is capable of generating a customized three-dimensional view for each individual user without modifying the 3D model of the virtual environment. This allows the system to maintain a 3D environment, while each participant can have adjusted angles and positions for various virtual objects. For the purpose of illustration, virtual objects are also referred to as "entities" in this article, and each entity can be an avatar, virtual furniture, a virtual screen for sharing content, or a virtual camera for generating a perspective view of a virtual environment. Some entities can be associated with perspective views and viewpoints, such as when the entity is an avatar representing an end user, or some entities may not be associated with perspective views and viewpoints, such as when the entity is in the form of a virtual display screen, virtual furniture, boundary walls, etc. In some configurations, the system is capable of adaptively adjusting the angle or position of an entity positioned within the perspective view of a target entity. Therefore, virtual objects in the avatar's view can appear in an optimized position to help display content to the user controlling the avatar. The adjustments can include changing the viewing angle of a virtual object such as a virtual screen, for example, a virtual screen in the avatar's view can appear to be rotated for a particular user. The adjustments can also include changing the position of a virtual object, for example, a virtual screen in the avatar's view can appear to be in a more centered position for a particular user. The adjustments can also include changing the zoom level for a particular user. The adjustments can be based on each user's point of view to maximize the perception of detail for each participant of the communication session. These adjustments can be made simultaneously while keeping participants and other entities arranged in a particular spatial relationship unchanged.

[0006] Adaptive adjustment of perspective views can provide many technical benefits to computing systems. For example, by providing adaptive adjustment of perspective views, each user of a communication session can see more equal views of the same shared content. This benefit can promote user engagement and help the system reduce user fatigue. By reducing user fatigue, especially in communication systems, users can exchange information more effectively. This helps to alleviate the occurrence of missing or ignoring shared content. This can reduce the occurrence of users needing to resend information. More effective communication of shared content can also help avoid the need for external systems (such as mobile phones for text messaging and other messaging platforms). The systems and features described in this article can also help reduce the reuse of networks, processors, memory, or other computing resources.

[0007] The features disclosed herein also provide many improvements to the security of the system. When system permissions are set to restrict a person from seeing certain content, the content can be further obscured by turning the content away from the user instead of turning the content toward the user. In addition, permissions are provided for controlling access to viewing data to prevent other users from seeing a person's selection of entities, content of interest, viewing preferences, etc.

[0008] Features and technical benefits in addition to those explicitly described above will be apparent by reading the following detailed description and reviewing the associated drawings. This summary is provided to introduce a selection of concepts in simplified form, which will be further described in the following detailed description. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter. For example, the term "technology" may refer to systems, methods, computer-readable instructions, modules, algorithms, hardware logic, and / or operations allowed by the above context and the entire document. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The detailed description is described with reference to the accompanying drawings. In the drawings, the leftmost digit(s) of a reference number identifies the drawing in which the reference number first appears. The same reference numbers in different drawings indicate similar or identical items. References to individual items in a plurality of items can use reference numerals with letters of the alphabetical sequence to refer to each individual item. Generic references to items can use specific reference numerals without the alphabetical sequence.

[0010] Figure 1A The state of the virtual environment before an entity is selected is shown.

[0011] Figure 1B The state of the virtual environment after selection of an entity is shown.

[0012] Figure 2 An example of rendering a perspective view from a virtual camera's viewpoint using the original model data is shown.

[0013] Figure 3 An example of rendering a perspective view from the viewpoint of a selected entity using the original model data is shown.

[0014] Figure 4 An example of rendering a perspective view from a viewpoint of a selected entity is shown, wherein the perspective view includes adjustments based on supplemental model data.

[0015] Figure 5 The first stage of adjustment is shown, where the transparency level of the entities is modified to give a sense of priority to the entities.

[0016] Figure 6 The second stage of adjustment is shown, where the transparency level of the entities is modified to give perception to the priority entities.

[0017] Figure 7 The third stage of adjustment is shown, in which the transparency level of the entities is modified to give perception to the priority entities.

[0018] Figure 8 A fourth stage of adjustment is shown, in which the transparency levels of entities are modified to give perception to priority entities.

[0019] Fig. 9 The transparency level of a portion of an adjustment entity is shown to bring a perceived adjustment to the priority entity.

[0020] Fig.10 is a flow chart illustrating aspects of a routine that enables adjustment of features of a viewing perspective of a 3D environment in response to selection of a target entity from a plurality of entities located within the 3D environment.

[0021] Fig.11 is a computer architecture diagram illustrating an exemplary computer hardware and software architecture of a computing system capable of implementing various aspects of the techniques and technologies presented herein.

[0022] Fig.12 is a computer architecture diagram illustrating a computing device architecture of a computing device capable of implementing various aspects of the techniques and technologies presented herein. DETAILED DESCRIPTION

[0023] Figure 1A and Figure 1B Aspects of a process for adjusting a viewing perspective of a 3D environment in response to selection of a target entity from among a plurality of entities located within the 3D environment are illustrated. Figure 1AAspects of a virtual environment 200 including a plurality of entities are shown. In this example, there are three entities 131: a first entity 131A is in the form of an avatar representing a first participant in a communication session, a second entity 131B is in the form of an avatar representing a second participant in the communication session, and a third entity 131C is in the form of a virtual display screen. As shown, each entity has a position in the 3D environment 200, which can be based on coordinates, and each entity has an orientation, which is represented as a vector originating from a point on the entity, such as a viewpoint at the location of the avatar's eyes. The viewpoint can also be associated with a position in the 3D environment. In some configurations, an orientation vector 113 can indicate which direction the entity is facing. In this example, a first orientation vector 113A shows the direction the first entity 131A is facing, a second orientation vector 113B shows the direction the second entity 131B is facing, and a third orientation vector 113C shows the direction the third entity 131C is facing. The orientation vector 113 for an entity can also be referred to as the "original orientation" of the entity based on the original model data 190.

[0024] As also shown, model data 190 (also referred to herein as raw model data 190 or data structure 190) can define the positions and orientations of multiple entities 131. An entity can be a virtual object, such as an avatar representing a user, or an entity can be another type of virtual object, such as a virtual television screen, virtual furniture, a virtual camera, etc. A virtual camera is similar to an avatar in that it can have a position, orientation, and viewing angle, but it may not have a displayable entity, for example, it may be an invisible virtual camera. Model data 190 can also define viewing angles 121 originating from viewpoints 111 for individual entities. The viewing angles 121 extend in a direction having an orientation vector, and each viewing angle 121 can have a scale that increases with increasing distance from the viewpoint, such as width or height. The model data can also define physical properties of each entity, such as size, shape, surface pattern, etc.

[0025] In one example, the entity can be an avatar or representation of a user. Viewpoint 111 can be at a particular location relative to the entity, such as the point where the avatar's eyes are located. A viewing perspective 121 of the entity can be projected from viewpoint 111, and viewing perspective 121 can be used to generate a rendering that is displayed to a user controlling the avatar. The system can display the entity of 3D environment 200 based on the position and orientation of the avatar and the position of the entity positioned within viewing perspective 121. Figure 1AIn the example of FIG. 1 , the first entity 131A has a viewing perspective 121A that follows the direction of the orientation vector 113A. The rendering of the viewing perspective 121A can include any entity within the viewing perspective 121A, such as the second entity 131B and the third entity 131C. The rendering of the viewing perspective 121A can be a 2D image generated using any suitable projection technology that captures images from the first viewpoint 111A.

[0026] exist Figure 1A In the example of , the rendering generated based on the model data 190 enables the system to generate a 2D image in which the top of the second entity is only partially displayed because the position of the second entity only places a portion of the entity within the first viewing perspective, for example, only the top third of the avatar's head is displayed because the top third is within the viewing perspective 121A. In addition, given the position of the first viewpoint 111A and the position and orientation of the second entity 131B, the rendering generated for the viewing perspective 121A will only show the right side of the avatar's face. Therefore, a selection surface of the second entity, such as the front of the avatar's face, may be obscured or limited in the display because the left half of the face will not be displayed. A selection surface of a third entity, such as the front screen of the virtual display screen, may also be limited because the third entity is oriented away from the first entity and the third entity is positioned such that a portion of the third entity is positioned outside the first viewing perspective 121A. Therefore, given that a portion of the third entity is located outside the first viewing perspective 121A, the rendering of the viewing perspective may crop a portion of the entity. A more detailed example of a rendering of a viewing perspective is shown in FIG. Figure 2-Figure 4 is shown in .

[0027] As described herein, selecting a target entity 131A from the plurality of entities 131 causes the system 100 to modify the viewing perspective 121A originating from the viewpoint 111A associated with the target entity 131 A. In this example, the input results in a selection of a first entity. Figure 1A Shows the state of the model data before an entity is selected. Figure 1B The status of the model data and the additional supplementary model data after selecting an entity is shown.

[0028] In some configurations, prior to selecting a particular entity, the original model data 190 can be used to generate an unadjusted rendering, i.e., rendering the position and orientation of each entity based on the original model data, which may cause one or more surfaces of the particular entity to be obscured or cropped. However, after selecting a particular entity (e.g., a target entity), the system can generate additional model data 191 to improve the display of the entity of interest. The entity of interest can be an entity that has at least a portion of the entity within a viewing perspective according to the original model data. If the first entity is selected as the target entity, examples of entities of interest (also referred to herein as selected entities) include the second entity because the second entity is within the viewing perspective of the first entity.

[0029] As in Figure 1B As shown in , after selecting a first entity or selecting a first viewing perspective, the system can generate supplementary model data 191, which has a new position and / or new orientation for the selected entity to improve the display of the entity in the viewing perspective. In some configurations, the supplementary model data can give the effect that the entity has moved or turned. The new position and / or new orientation can enable the system to generate a rendering of the viewing perspective so that the selected surface, such as the display screen surface or the face of the avatar, is more prominently shown in the rendering of the perspective view. The new position and / or new orientation for a specific entity can increase the display of one or more selected surfaces. The generation of the supplementary data can be completed in a manner that keeps the original model data intact. This allows the view of one user to be modified while maintaining the position of the entity and the view of other users, while modifying the first user controlling the first avatar. These features also allow the system to maintain the orientation and position of each entity to maintain the integrity of the 3D model. Therefore, the viewing perspective 121B for the second entity 131B does not change for the user controlling the second entity.

[0030] In some configurations, the selected entity can also include an entity identified in the user preferences. For example, the user preferences of the first user controlling the first entity can identify a fourth entity that is not in the user's viewing perspective based on the original model data. In such a scenario, the system can generate supplemental model data 191, which has a new position and / or a new orientation for the selected entity to improve the display of the entity, even if it is not based on the viewing perspective of the original model data. The new position and / or new orientation can enable the system to generate a rendering of the viewing perspective so that the selected surface of the entity, such as the display screen surface or the face of the avatar, is more prominently shown in the rendering of the perspective view. This can be done without causing modification to the original data, so that other user perspective views can be generated while maintaining the orientation and position of each entity to maintain the integrity of the 3D model of the environment.

[0031] To achieve the results described herein, the system may access a data structure 190 defining a 3D environment 200, wherein the data structure 190 defines a position 132 and an orientation 113 for each of a plurality of entities 131 of the 3D environment 200, the data structure 190 defining at least one viewing perspective 121, the at least one viewing perspective 121 being based on one or more viewpoints 111 for the individual entities 131, the one or more viewpoints 111 being selectable for use in generating a customized rendering 300 of the at least one viewing perspective 121 for the computing device 11A of the communication session 604. The orientation of each entity can be defined by one or more matrices or any other data set capable of defining a vector from a point in space. The point in space can be at or based on the entity location.

[0032] The system can then receive input indicating a selection of a target entity 131A from the plurality of entities 131. The input causes the system 100 to modify the viewing perspective 121A originating from the viewpoint 111A associated with the target entity 131A. The selection can include selection of an entity (e.g., an avatar or a virtual room camera) for generating a modified view based on the supplemental model data. The input selecting an entity can also include computer-generated input, where the computer can determine that the perspective of the meeting should be from a particular avatar, or the input can be provided through user input such as a voice command, a touch screen command, etc. The computer-controlled input can be triggered by one or more actions. For example, a meeting can have an agenda, and the system can select different entities or viewing perspectives or viewpoints based on the agenda. For example, in the introductory portion of the agenda, the system can generate input to select a first camera view, such as in Figure 2 . Then, during the presentation portion of the session, the system can generate different inputs to select different viewing perspectives from various avatars. These examples are provided for illustrative purposes and should not be construed as limiting. It will be appreciated that any type of computer-generated input or user-generated input can be utilized. In some implementations, the input can select a perspective view, a viewpoint, or an associated entity, and selection of any of these items can mean selecting the associated entity as a target entity and its associated perspective view. An entity such as a target entity can be associated with a user and their computer, with the entity having permission to control the orientation, position, and / or posture of the entity.

[0033] In response to the input indicating selection of a target entity, such as first entity 131A, as in Figure 1BAs shown in the supplementary data of , the system can generate supplementary model data 191, which defines at least one of a new position 133B, 133C or a new orientation 134B, 134C for at least one entity 131B, 131C of the plurality of entities 131. The new position and the new orientation can be applied to any selected entity. The selection of the target entity can also mean the selection of a specific viewpoint or the selection of a viewing perspective, such as a first viewing perspective. The new position and the new orientation can be associated with a user with permission to control the target entity (e.g., the selected avatar).

[0034] For illustrative purposes, the target entity is the entity selected by the input, and an "entity of interest" is identified by the system in response to the selection of the target entity or the selection of a perspective view or viewpoint. Once the target entity is selected based on the input, the system can analyze the entities within the virtual environment to identify entities of interest. If one or more entities of interest are identified, the system can generate a new position and / or a new orientation for each of the entities of interest. In some configurations, an entity of interest can be identified when any portion of the entity is within the viewing perspective based on the original model data. For example, in Figure 1A In the example of , the second entity 131B has at least a portion of the entity 131B in the viewing perspective 121A of the first entity 131A, and therefore, the second entity is selected as the entity of interest for the new position or new orientation to be applied to the supplemental model data 191. If the second entity does not have any portion in the viewing perspective 121A of the first entity 131A, the second entity is not selected as the entity of interest and a new orientation and a new position are not generated for the entity.

[0035] In some configurations, the selection of the entity of interest can be based on one or more criteria. For example, if a particular entity has more than a threshold portion within a viewing angle, where another portion of the particular entity is outside the viewing angle, the system can generate a new orientation and / or a new position for the particular entity. For example, if the threshold is 50% and 51% of the entity within the viewing angle, the system can select the entity as an entity of interest and generate a new position or orientation for the entity of interest. However, in this example, if the entity is only 49% within the viewing view, the system may not select the particular entity as an entity of interest, and the system will not generate a new orientation or position for the entity. This allows the system to reduce the use of computing power required to generate new positions and orientations. Therefore, if the virtual environment has thousands of entities within the viewing angle, the system can only change the position or orientation of those objects that the user is interested in.

[0036] This example is provided for illustrative purposes and is to be construed as non-limiting. It can be appreciated that other criteria can be utilized to select an entity as an entity of interest. For example, if a particular entity has a portion less than a threshold within a viewing angle, the system can select the entity as an entity of interest. This type of criteria can bring more objects into more prominent viewing angles and positions when the original model data does not allow the user to see enough of a particular object. Therefore, the system may not generate supplemental model data for objects that have a portion exceeding a threshold within a viewing angle. The portions measured for these determinations can be based on a percentage of the total volume of the entity, a percentage of a selected surface of the entity, or a measure of any other unit.

[0037] The system can also determine whether a new orientation is needed. Figure 1A In the example of , if the orientation vector of the second entity is pointing toward the first entity and is only turned away by 15%, the system may not generate a new orientation for the entity. However, if the orientation vector of the second entity is directed away from the first entity, or if the orientation vector of the second entity is turned away from the first entity by more than a threshold degree, for example, more than 25%, the system may generate a new orientation for the entity.

[0038] refer to Figure 1B , the supplementary data can be configured such that for a new position of the second entity 131B and the third entity 131C, such as a new position 133B, 133C or a new orientation 134B, 134C, at least one selection surface of at least one entity 131B, 131C positioned within the viewing perspective 121A associated with the target entity 131A is increased in display. Thus, the supplementary model data can be generated to create a new view for only the selected avatar, e.g., the user controls the target entity, wherein increasing the display of the selection surface means that it shows more parts of the entity or more parts of the selected part of the entity, e.g., the selection surface is the face of the avatar, or the display screen of a virtual display screen, etc.

[0039] The examples of selected surfaces are provided for illustrative purposes and should not be interpreted as limiting. It will be appreciated that other surfaces of entities can be selected, such as an avatar's hand or any other portion of an area of ​​interest. In some configurations, user preferences can define one or more surfaces of interest to the user, and the orientation of other entities can be rotated or repositioned with respect to the supplemental model data so that the selected surface indicated in the user preferences is more prominently displayed and the perspective view is rendered. For example, if the first user's preferences indicate an interest in viewing the engine of a virtual motorcycle, and the original model data shows the motorcycle from the perspective of the first user, the system can generate supplemental model data to include the orientation and position of the motorcycle, which allows the selected portion (e.g., the engine) to be more prominently displayed to the first user's specific perspective view.

[0040] Figures 2 to 4 An example is shown of how original model data 190 can be used to display a rendering 250 of a perspective view and how supplemental model data 191 can be used to display a customized rendering 300 of a perspective view for a particular user without affecting the original model data or affecting the perspective views of other users. For purposes of illustration, the perspective views disclosed herein can also be referred to as "views."

[0041] Figure 2 An example of a rendering 250 based on a perspective view of a virtual room camera with a viewpoint behind a group of avatars representing users of a communication session is shown. The rendering 250 shows the positions and orientations of entities within the virtual environment 200 based on the original model data 190, an example of which is shown in FIG. Figure 1A . In this example, the first entity 131A is a first avatar, the second entity 131B is a second avatar, the third entity 131C is a third avatar, the fourth entity 131D is a virtual screen, and the fifth entity 131E is a model of a product (e.g., a car charger) generated based on the shared content of one of the users. The virtual screen also includes multiple 2D images 211 of other conference participants. The 2D images can include 2D still images of the users or 2D live video feeds.

[0042] Aspects of system 100 are also Figure 2 The system can be configured to manage communication sessions, and the system can include multiple computers 11, each computer corresponding to multiple users 10 or participants 10. The computers can be in the form of desktop computers, head mounted display units, tablet computers, mobile phones, etc. As described herein, the system 100 may also include servers and other devices to facilitate communication sessions.

[0043] The system can generate a user interface 101 showing aspects of a communication session with each of the users. The user interface 101 can include a rendering of a three-dimensional (3D) representation of each user, which can include an avatar positioned within a 3D virtual environment 200. In this particular example, the user interface includes a 3D rendering 131A of a representation of a first user 10A, a 3D rendering 131B of a representation of a second user 10B, and a 3D rendering 131C of a representation of a third user 10C. The user interface 101 also includes a 2D rendering 211A of a live streaming video of a fourth user 10D, a 2D rendering 211B of a live streaming video of a fifth user 10E, and a 2D rendering 211C of a live streaming video of a sixth user 10F.

[0044] Figure 3 An example of a rendering 250 of a perspective view based on a first perspective view 121A from a viewpoint 111A at the location of a first entity 131A (e.g., an avatar representing a user of a communication session) is shown. The rendering 250 shows the position and orientation of entities within a virtual environment 200 based on original model data 190, for example, based on inputs controlling the activities and shared content of each avatar. In this example, a second avatar 131B, a third avatar 131C, and a virtual display screen 131D are displayed along with a car charger 131E. The virtual screen also includes multiple 2D images 211 of other conference participants. Given the position of each entity, the angle of each avatar and the virtual screen and car charger is not optimal for the first user. For example, the rendering 250 based on the original model data only shows half of the face of the second avatar 131B and the third avatar 131C. Virtual screen 131D is also oriented in a way that may make viewing of the content more difficult because the rendering 250 shows virtual screen 131D from the side of the virtual display screen and it is partially cropped from the view. The car charger is also shown from the side and it is partially cropped from the view. If the item is not at the best angle and the rendering of the item is cropped, this can result in missed content in the meeting.

[0045] To account for positions and angles that are not optimal for a particular view, the system can generate adaptive adjustments to the perspective view to improve the user's perception of details associated with a target entity of the virtual environment. An example of adaptive adjustments to the perspective view is provided in Figure 4 In this example, the rendering 300 is a rendering of a perspective view based on a first perspective view 121A from a viewpoint 111A at a location of a first entity 131A.

[0046] The rendering 300 shows the position and orientation of entities within the virtual environment 200 based on the supplemental model data 191, for example, the supplemental model data generated with new attributes based on the selection of the target entity. In this example, the second avatar 131B, the third avatar 131C, the virtual display screen 131D, and the car charger 131E are displayed using new positions and new orientations that increase the display of at least one selected surface of each entity. As shown, the front of the display screen is directed so that it faces the user's viewpoint more directly, and the car charger is also oriented and positioned so that the entire side of the car charger is displayed. The faces of the avatars are also directed so that they face the viewpoint of the first user more directly. This rendering 300 based on the supplemental model data more significantly increases the display of the surface of each face of each avatar relative to the rendering 250 based on the original model data.

[0047] The adjusted display (e.g., customized display) of the rendering 350 can use the supplemental model data 191 that defines at least one of a new position 133B, 133C or a new orientation 134B, 134C for at least one entity 131B, 131C, wherein the customized rendering 300 is based on a viewing perspective 121A originating from a viewpoint 111A associated with the target entity 131A. The use of the supplemental model data 191 that defines at least one of a new position 133B, 133C or a new orientation 134B, 134C for at least one entity 131B, 131C is used to generate the customized rendering 300 for the computing device 11A without causing modifications to other perspective views 121B associated with other entities 131B. This means that the system can create the customized rendering 300 without causing any additional modifications to the original model data. This means that the original model data can actually follow the movement of the entity and maintain the integrity of the model data for the 3D environment under human control, rather than having modifications of the perspective view causing any changes to the original model.

[0048] The adjusted or customized display of the rendering 350 can use the supplemental model data 191 that defines at least one of a new position 133B, 133C or a new orientation 134B, 134C for at least one entity 131B, 131C, wherein the customized rendering 300 is based on a viewing perspective 121A originating from a viewpoint 111A associated with the target entity 131A. The customized rendering 300 for the computing device 11A is generated using the supplemental model data 191 that defines at least one of a new position 133B, 133C or a new orientation 134B, 134C for at least one entity 131B, 131C without causing modifications to other perspective views 121B associated with other entities 131B. This means that the system can create the customized rendering 300 without causing any additional modifications to the original model data. This means that the original model data can actually follow the movement of the entity and maintain the integrity of the model data for the 3D environment according to human control, rather than having modifications to the perspective view that cause any changes to the original model.

[0049] The supplemental data can also include the difference or offset between the new orientation and the new position. This allows the system to make real-time adjustments to the supplemental model data as the original model data is modified. For example, if a user moves in a 3D environment according to an input control (e.g., turn right 10 degrees), the original model data will change the position or orientation of the avatar for that user. Using the data defining the differences or offsets in the supplemental data, the system can update the supplemental model data using these differences or offsets. Thus, if the second user provides an input that turns the second entity 10 degrees, the system can use an offset to change the orientation of the supplemental model data by 10 degrees or a predetermined fraction of the rotation. Thus, the adjusted view can show movement, but the movement can be damped or biased toward the viewpoint of the target entity.

[0050] Also as in Figure 4 As shown in , customization of the view can also include adjusting the zoom level for a particular perspective view. In this example, the viewing angle is widened to provide a wider display of the virtual environment. In response to determining that a threshold number of entities have at least a portion of a surface that was cropped in the original view based on the original model data, the zoom level can be widened. For example, if the system has a threshold of four entities and there are five entities that have at least a portion of a surface that was cropped from the original view, the system can widen the perspective view rather than generating supplemental model data for each object. In addition to determining new values ​​for position, orientation, or zoom level, the system can also increase the size of an entity. As shown in Figure 4As shown in , assuming that the two avatars are entities of interest, the system can enlarge the rendering of the entity to depict the user focus and perceive the two element activities. The increase in the size of the entity can be in response to any of the trigger actions described herein, such as the selection of the target entity.

[0051] Reference now Figures 5 to 9 , some embodiments can provide customized views that involve adjusting the transparency level of a selected entity to bring awareness of other entities that are visually obscured by the selected entity. Figure 4 As shown in , the system is able to determine whether a particular entity is overwriting another entity with a threshold priority. In this example, as shown in Figure 5 As shown in , a rendering 211C of a conference participant's 2D video stream is overlaid by a 3D entity 131E positioned in front of the virtual display screen. In response to determining that the foreground entity 131E is covering or obscuring the user's view of a background entity 211C having a priority exceeding a threshold priority, the system is able to change the transparency level for the foreground entity 131E, or change the transparency level for a portion of the foreground entity 131E.

[0052] The system can determine whether a particular entity has a priority threshold. The priority threshold can be based on an activity or data type associated with the entity. For example, any shared content that can be rendered as a virtual object in a perspective view, such as a live video stream or a shared file, can be considered an entity with a priority threshold. The priority threshold can also be based on the file type of the data that defines the entity. For example, if an entity such as a car charger is shared using a 3D AutoCAD file, the system can determine that the charger is a priority entity because it is based on a predetermined file type, such as an AutoCAD file.

[0053] In response to determining that the identity is a priority entity, the system can determine whether the priority entity is obscured by display of a second entity. If the priority entity is obscured or covered by the second entity, the system can change the transparency of at least a portion of the second entity to allow display of the priority entity. Figures 5 to 8 Such modification to the user interface is shown in the progress of FIG. In response to determining that an entity (e.g., a rendering of a participant's video stream) is prioritized, and in response to determining that the entity is obscured by a foreground entity 131E (e.g., a charger), the system changes the transparency of the foreground entity 131E to allow display of the priority object, e.g., the rendering of the video stream. The user can return display of the object by providing input. The system can also change only a portion of an entity (such as in FIG. 1 ). Fig. 9The size and shape of the portion of the entity with a higher transparency level can be set to allow a threshold portion of the background entity to be shown in the rendering of the perspective view. For example, the size and position of the square is set to show the 2D rendering of the participant's video stream.

[0054] In addition to changing the transparency of the entity object, the system can also move the foreground object to allow display of the background object that is obscured by the foreground object. The movement can be in response to determining that the background entity has a threshold priority and that the foreground entity is obscuring the background entity. Figure 5 In the example shown in , the charger can be moved higher to allow the video stream of the participant to be displayed.

[0055] In some configurations, data defining the transparency of objects causing visual occlusion can be stored in the supplemental model data. This allows the system to generate customized visual perspective views for individual users while maintaining the integrity of the original model 190 and enabling the system to generate visual perspectives for other users following the model.

[0056] In an embodiment disclosed herein, the system can assign permissions to the supplemental model data for each user, so that users associated with a selected perspective view or associated target entity can access the supplemental model data generated for their associated target entity. These permissions can also restrict other users from accessing the supplemental model data. For example, the first user can be granted permission to control a first avatar, and the first avatar can be selected by the user as a target entity. In response to the selection, the system can generate supplemental model data for the first user to access to generate their view. The system can also configure permission data to restrict others from accessing the generated supplemental model data. This ensures that the system does not disclose the preferences of the first user, such as permissions for selecting entities of interest, other view preferences, etc.

[0057] Fig.10 1 is a diagram illustrating aspects of a routine 500 for adjusting a viewing perspective 121A associated with a target entity 131A selected from a plurality of entities 131 located within a 3D environment 200. It should be understood by one of ordinary skill in the art that the operations of the methods disclosed herein are not necessarily presented in any particular order, and that it is contemplated that some or all of the operations may be performed in an alternative order. For ease of description and illustration, the operations are presented in the order presented. Operations may be added, omitted, performed, and / or performed simultaneously without departing from the scope of the appended claims.

[0058] It should also be understood that the illustrated method can start or end at any time and does not need to be performed as a whole. Some or all of the operations of the method and / or substantially equivalent operations can be performed by executing computer-readable instructions included on a computer storage medium, as defined herein. As used in the specification and claims, the term "computer-readable instructions" and its variants are widely used herein to include routines, applications, application modules, program modules, programs, components, data structures, algorithms, etc. Computer-readable instructions can be implemented on various system configurations, including single-processor or multi-processor systems, minicomputers, mainframe computers, personal computers, handheld computing devices, microprocessor-based programmable consumer electronics, combinations thereof, etc. Although the example routines described below operate on a system (e.g., one or more computing devices), it is appreciated that the routines can be executed on any computing system, which may include any number of computers working together to perform the operations disclosed herein.

[0059] Therefore, it should be appreciated that the logical operations described herein are implemented as a sequence of computer-implemented actions or program modules running on a computing system such as described herein and / or as interconnected machine logic circuits or circuit modules within the computing system. The implementation is a matter of choice depending on the performance and other requirements of the computing system. Therefore, the logical operations may be implemented in software, firmware, dedicated digital logic, and any combination thereof.

[0060] In addition, Figure 7 The operations illustrated in the other figures can be implemented in association with the example user interfaces and systems described herein. For example, the various devices and / or modules described herein can generate, send, receive, and / or display data associated with the content of a communication session (e.g., live content, broadcasted events, recorded content, etc.) and / or a rendering UI including a rendering of one or more participants of a remote computing device, avatar, channel, chat session, video stream, image, virtual object, and / or application associated with the communication session.

[0061] Routine 500 includes operation 501, wherein system 100 accesses a data structure 190 defining a 3D environment 200, data structure 190 defining a position 132 and an orientation 113 for each of a plurality of entities 131 for the 3D environment 200, data structure 190 defining at least one viewing perspective 121, viewing perspective 121 being based on one or more perspective views 111 for the individual entities 131, one or more perspective views 111 being selectable for generating a customized rendering 300 for at least one viewing perspective 121 for a computing device 11A for a communication session 604.

[0062] At operation 503, the system receives input indicating a selection of a target entity 131A from the plurality of entities 131, wherein the input causes the system 100 to modify the viewing perspective 121A originating from the viewpoint 111A associated with the target entity 131 A. This is a selection of an entity, such as an avatar or a virtual room camera, for generating a view for a user associated with or controlling the entity.

[0063] At operation 505, the system selects an entity of interest. In some configurations, the selection of the entity of interest can be based on one or more criteria. For example, if a particular entity has more than a threshold portion within a viewing angle, wherein another portion of the particular entity is outside the viewing angle, the system can generate a new orientation and / or a new position for the particular entity. For example, if the threshold is 50% and 51% of the entity within the viewing angle, the system can select the entity as an entity of interest and generate a new position or orientation for the entity of interest. However, in this example, if the entity is only 49% within the viewing angle, the system may not select the particular entity as an entity of interest, and the system will not generate a new orientation or a new position for the entity. This allows the system to reduce the use of computing power required to generate new positions and new orientations. Therefore, if the virtual environment has thousands of entities within the viewing angle, the system can only change the position or orientation of those objects that the user is interested in. For illustrative purposes, at least one entity 131B in the claims is an example of an entity of interest.

[0064] At operation 507, the system generates supplemental model data 191 that defines at least one of a new position 133B, 133C or a new orientation 134B, 134C for at least one entity 131B, 131C (e.g., an entity of interest) for the plurality of entities 131. The generation can be responsive to input indicating a selection of a target entity 131A. The new position 133B, 133C or the new orientation 134B, 134C for the at least one entity 131B, 131C adds display of at least one selected surface of the at least one entity 131B, 131C that is within the viewing perspective 121A associated with the target entity 131A. Examples of the supplemental model data are shown in FIG. Figure 1B. Supplemental model data 191 is generated to create a new view only for the selected avatar (e.g., target entity) for increasing the display of the selection surface means showing more parts of the object or more parts of the face of the entity, for example, the selection surface is the face of the avatar. The generation of the supplemental model data does not cause modification of the data structure 190 defining the 3D environment 200. In some embodiments, during the generation or modification of the supplemental model data, the position 132 and orientation 113 of each entity in the plurality of entities 131 of the 3D environment 200 defined in the data structure 190 are not modified. System permissions can also restrict modification of the UI data structure 190 during modification or generation of the supplemental model data to retain other perspective views 121B associated with another viewpoint 111B of other entities 131B.

[0065] At operation 509, the system 100 causes display of a customized rendering 300 for the computing device 11A using the supplemental model data 191 defining at least one of a new position 133B, 133C or a new orientation 134B, 134C for the at least one entity 131B, 131C. The customized rendering 300 is based on a viewing perspective 121A originating from a viewpoint 111A associated with the target entity 131A, wherein the use of the supplemental model data 191 defining at least one of a new position 133B, 133C or a new orientation 134B, 134C for the at least one entity 131B, 131C is used to generate the customized rendering 300 for the computing device 11A without causing modifications to other perspective views 121B associated with other entities 131B. Figure 1B and Figure 2 New model data 191 is shown, and the rendering is based on the new model data, while other model data is maintained to keep the original views of other avatars the same. Although this example suggests that the new perspective view is for a computing device 11A associated with a user controlling the target entity 131A, for example, a computer controlling an avatar, it can be appreciated that the customized rendered display can be transmitted to any computing device of any user with appropriate permissions. This includes computers controlled by users with host or administrative privileges. This allows a host or administrator to select a specific perspective view for selected participants of a meeting. This allows a host or administrator to switch between different perspective views, and also allows the system to automatically correct the position and / or orientation of entities of interest within each perspective view.

[0066] Figure 8600 is a diagram illustrating an example environment 600 in which a system 602 can implement the technology disclosed herein. It should be appreciated that the subject matter described above can be implemented as a computer-controlled device, a computer process, a computing system, or an article such as a computer-readable storage medium. The operation of the example method is illustrated in individual blocks and is summarized with reference to these blocks. The method is illustrated as a logical flow of blocks, each of which can represent one or more operations that can be implemented in hardware, software, or a combination thereof. In the context of software, the operation represents a computer-executable instruction stored on one or more computer-readable media, which, when executed by one or more processors, enables the one or more processors to perform the described operation.

[0067] Typically, computer executable instructions include routines, programs, objects, modules, components, data structures, etc. that perform specific functions or implement specific abstract data types. The order in which the operations are described is not intended to be interpreted as a limitation, and any number of the described operations can be performed in any order, combined in any order, subdivided into multiple sub-operations, and / or performed in parallel to implement the described processes. The described processes can be performed by resources associated with one or more devices such as one or more internal or external CPUs or GPUs, and / or one or more hardware logic such as a field programmable gate array ("FPGA"), a digital signal processor ("DSP"), or other types of accelerators.

[0068] All methods and processes described above can be embodied in software code modules executed by one or more general-purpose computers or processors and fully automated. The code modules can be stored in any type of computer-readable storage medium or other computer storage device, such as those described below. Some or all of the methods described can alternatively be embodied in dedicated computer hardware, such as the computer hardware described below.

[0069] Any routine description, element or block in the flowcharts described herein and / or depicted in the accompanying drawings should be understood to potentially represent a module, segment or portion of code, which includes one or more executable instructions for implementing a specific logical function or element in the routine. Alternative implementations are included within the scope of the examples described herein, where, depending on the functionality as will be understood by those skilled in the art, elements or functions may be deleted or performed from the order shown or discussed, including substantially synchronously or in reverse order.

[0070] In some implementations, system 602 can be used to collect, analyze, and share data displayed to users of communication session 604. As illustrated, communication session 603 can be implemented between a plurality of client computing devices 606(1)-606(N) (where N is a number having two or greater values) that are associated with or part of system 602. Client computing devices 606(1)-606(N) enable users, also referred to as individuals, to participate in communication session 603.

[0071] In this example, communication session 603 is hosted by system 602 over one or more networks 608. That is, system 602 can provide services that enable users of client computing devices 606(1)-606(N) to participate in communication session 603 (e.g., via live viewing and / or recorded viewing). Thus, "participants" of communication session 603 can include users and / or client computing devices (e.g., multiple users can be in a room participating in a communication session via use of a single client computing device), each of which can communicate with other participants. Alternatively, communication session 603 can be hosted by one of client computing devices 606(1)-606(N) using peer-to-peer technology. System 602 can also host chat conversations and other team collaboration functionality (e.g., as part of an application suite).

[0072] In some implementations, such chat conversations and other team collaboration functions are considered external communication sessions distinct from the communication session 603. The computing system 602 that collects participant data in the communication session 603 may be able to link to such an external communication session. Thus, the system may receive information that enables connection to such an external communication session, such as date, time, session details, etc. In an example, a chat conversation can be conducted in accordance with the communication session 603. Additionally, the system 602 may host a communication session 603 that includes at least a plurality of participants co-located at a meeting location (such as a conference room or auditorium) or located at different locations.

[0073] In the example described herein, client computing devices 606(1) to 606(N) participating in a communication session 603 are configured to receive and render communication data for display on a user interface of a display screen. The communication data can include a collection of various instances or streams of live content and / or recorded content. The collection of various instances or streams of live content and / or recorded content can be provided by one or more cameras (such as a video camera). For example, an individual stream of live or recorded content can include media data associated with a video feed provided by a video camera (e.g., audio and visual data capturing the appearance and voice of a user participating in the communication session). In some embodiments, the video feed can include such audio and visual data, one or more still images, and / or one or more avatars. The one or more still images can also include one or more avatars.

[0074] Another example of an individual stream of live or recorded content can include media data including avatars of users participating in the communication session and audio data capturing the users' voices. Yet another example of an individual stream of live or recorded content can include media data including a file displayed on a display screen and audio data capturing the users' voices. Thus, the various streams of live or recorded content within the communication data enable remote conferencing to be facilitated between a group of people and content to be shared within a group of people. In some embodiments, the various streams of live or recorded content within the communication data can originate from a plurality of co-located video cameras positioned in a space, such as a room, to record or stream a presentation including one or more individuals presenting and one or more individuals consuming the presented content.

[0075] Participants or attendees can view the content of the communication session 603 as the activity occurs or alternatively via a recording at a later time after the activity occurs. In the example described herein, the client computing devices 606(1) to 606(N) participating in the communication session 603 are configured to receive and render communication data for display on a user interface of a display screen. The communication data can include a collection of various instances or streams of live and / or recorded content. For example, an individual stream of content can include media data associated with a video feed (e.g., audio and visual data that captures the appearance and voice of users participating in the communication session). Another example of an individual stream of content can include media data that includes an avatar of a user participating in the conference session and audio data that captures the user's voice. Yet another example of an individual stream of content can include media data that includes content items displayed on a display screen and / or audio data that captures the user's voice. Thus, the various streams of content within the communication data enable facilitating a conference or broadcast presentation between a group of people spread across remote locations.

[0076] A participant or attendee of a communication session is a person within range of a camera or other image and / or audio capture device, so that the person's movements and / or sounds generated when the person is viewing and / or listening to the content shared via the communication session can be captured (e.g., recorded). For example, a participant may be sitting in a crowd viewing the shared content live at a broadcast location where a stage presentation is taking place. Or, a participant may be sitting in an office conference room viewing the shared content of a communication session with other colleagues via a display screen. Further, a participant may be sitting or standing in front of a personal device (e.g., a tablet, smart phone, computer, etc.) that is viewing the shared content of the communication session individually in their office or home.

[0077] Figure 8 System 602 includes device(s) 610. Device(s) 610 and / or other components of system 602 can include distributed computing resources that communicate with each other and / or with client computing devices 606(1)-606(N) via one or more networks 608. In some examples, system 602 can be a stand-alone system responsible for managing various aspects of one or more communication sessions, such as communication session 603. As an example, system 602 can be managed by an entity such as SLACK, WEBEX, GOTOMEETING, GOOGLE HANGOUTS, etc.

[0078] The network(s) 608 may include, for example, a public network such as the Internet, a private network such as an institutional and / or personal intranet, or some combination of private and public networks. The network(s) 608 may also include any type of wired and / or wireless network, including, but not limited to, a local area network ("LAN"), a wide area network ("WAN"), a satellite network, a cable network, a Wi-Fi network, a WiMax network, a mobile communication network (e.g., 3G, 4G, etc.), or any combination thereof. The network(s) 608 may utilize communication protocols, including packet-based and / or datagram-based protocols, such as the Internet Protocol ("IP"), the Transmission Control Protocol ("TCP"), the User Datagram Protocol ("UDP"), or other types of protocols. In addition, the network(s) 608 may also include a plurality of devices that facilitate network communications and / or form a hardware foundation for the network, such as switches, routers, gateways, access points, firewalls, base stations, repeaters, backbone devices, etc.

[0079] In some examples, network(s) 608 may also include devices that enable connection to a wireless network, such as a wireless access point (“WAP”). Connection is supported by examples of WAPs that send and receive data over various electromagnetic frequencies (e.g., radio frequencies), including WAPs that support the Institute of Electrical and Electronics Engineers (“IEEE”) 802.11 standards (e.g., 802.11g, 802.11n, 802.11ac, etc.), as well as other standards.

[0080] In various examples, (one or more) devices 610 may include one or more computing devices that operate in a cluster or other grouping configuration to share resources, balance loads, increase performance, provide failover support or redundancy, or for other purposes. For example, (one or more) devices 610 may belong to various types of devices, such as conventional server-type devices, desktop computer-type devices, and / or mobile devices. Therefore, although illustrated as a single type of device or server-type device, (one or more) devices 610 may include a variety of device types and are not limited to a particular type of device. (One or more) devices 610 may represent, but are not limited to: a server computer, a desktop computer, a web server computer, a personal computer, a mobile computer, a laptop computer, a tablet computer, or any other type of computing device.

[0081] A client computing device (e.g., one of client computing device(s) 606(1)-606(N)) (each of which is also referred to herein as a "data processing system") may be of various types of devices, which may be the same as or different from device(s) 610, such as a conventional client-type device, a desktop-type device, a mobile-type device, a dedicated-type device, an embedded device, and / or a wearable-type device. Thus, the client computing device can include, but is not limited to, a desktop computer, a game console and / or gaming device, a tablet computer, a personal data assistant ("PDA"), a mobile phone / tablet hybrid device, a laptop computer, a telecommunications device, a computer navigation type client computing device such as a satellite-based navigation system, a satellite-based navigation system including a global positioning system ("GPS") device, a wearable device, a virtual reality ("VR") device, an augmented reality ("AR") device, an implanted computing device, an automotive computer, a network-enabled television, a thin client, a terminal, an Internet of Things ("IoT") device, a workstation, a media player, a personal video recorder ("PVR"), a set-top box, a camera, an integrated component (e.g., a peripheral device) for inclusion in a computing device, an appliance, or any other type of computing device. In addition, the client computing device may include a combination of the previously listed examples of client computing devices, such as, for example, a desktop computer type device or a mobile type device combined with a wearable device, etc.

[0082] The various classes and device types of client computing devices (one or more) 606 (1) to 606 (N) can represent any type of computing device having one or more data processing units 692, which are operably connected to a computer-readable medium 694, such as via a bus 616, which in some cases can include one or more of a system bus, a data bus, an address bus, a PCI bus, a Mini-PCI bus, and any of a variety of local, peripheral and / or independent buses.

[0083] The executable instructions stored on the computer-readable medium 694 may include, for example, an operating system 619 , a client module 620 , a profile module 622 , and other modules, programs, or applications loadable and executable by the data processing unit(s) 692 .

[0084] The client computing device(s) 606(1)-606(N) may also include one or more interfaces 624 to enable communication between the client computing device(s) 606(1)-606(N) and other networked devices (such as device(s) 610) via the network(s) 608. Such network interface(s) 624 may include one or more network interface controllers (NICs) or other types of transceiver devices to send and receive communications and / or data via the network. In addition, the client computing device(s) 606(1)-606(N) may include input / output ("I / O") interfaces (devices) 626 to enable communication with user input devices such as peripheral input devices (e.g., game controllers, keyboards, mice, pens, voice input devices such as microphones, video cameras for obtaining and providing video feeds and / or still images, touch input devices, gesture input devices, etc.) and / or output devices such as peripheral output devices (e.g., displays, printers, audio speakers, tactile output devices, etc.). Figure 8 The client computing device 606 ( 1 ) is illustrated as being connected in some manner to a display device (eg, display screen 629 (N)) that is capable of displaying a UI according to the techniques described herein.

[0085] exist Figure 8 In the example environment 600 of FIG. 600 , client computing devices 606 ( 1 ) through 606 (N) can use their respective client modules 620 to connect to each other and / or to (one or more) other external devices in order to participate in a communication session 603 or to contribute activities to the collaborative environment. For example, a first user can utilize client computing device 606 ( 1 ) to communicate with a second user of another client computing device 606 ( 2 ). When executing client module 620 , the users can share data, which can cause client computing device 606 ( 1 ) to connect to system 602 and / or other client computing devices 606 ( 2 ) through 606 (N) via network (s) 608 .

[0086] The client computing device(s) 606(1)-606(N) may use their respective profile modules 622 to generate participant profiles (in Figure 8 602) and provide participant profiles to other client computing devices and / or device(s) 610 of system 602. A participant profile may include one or more of the following: an identity of a user or a group of users (e.g., name, unique identifier (“ID”), etc.), user data such as personal data, machine data such as location (e.g., IP address, room in a building, etc.) and technical capabilities, etc. The participant profile may be used to register participants for a communication session.

[0087] As in Figure 8 , device(s) 610 of system 602 include a server module 630 and an output module 632. In this example, server module 630 is configured to receive media streams 634(1) to 634(N) from individual client computing devices, such as client computing devices 606(1) to 606(N). As described above, the media streams can include video feeds (e.g., audio and visual data associated with a user), audio data to be output with a presentation of an avatar of the user (e.g., an audio-only experience that does not send the user's video data), text data (e.g., a text message), file data, and / or screen sharing data (e.g., a document, a slide deck, an image, a video displayed on a display screen, etc.), etc. Thus, server module 630 is configured to receive a collection of various media streams 634(1) to 634(N) (the collection is referred to herein as "media data 634") during a live view of communication session 603. In some scenarios, not all client computing devices participating in communication session 603 provide media streams. For example, the client computing device may be a consuming device or a “listening” device only, such that it only receives content associated with the communication session 603 , but does not provide any content to the communication session 603 .

[0088] In various examples, the server module 630 can select aspects of the media stream 634 to be shared with individual client computing devices of the participating client computing devices 606(1) to 606(N). Thus, the server module 630 can be configured to generate session data 636 based on the stream 634 and / or pass the session data 636 to the output module 632. The output module 632 can then transmit communication data 639 to the client computing devices (e.g., the client computing devices 606(1) to 606(3) participating in the real-time viewing of the communication session). The communication data 639 can include video, audio, and / or other content data provided by the output module 632 based on content 650 associated with the output module 632 and based on the received session data 636. The content 650 can include the stream 634 or other shared data, such as an image file, a spreadsheet file, a slide deck, a document, etc. The stream 634 can include a video component depicting an image captured by the I / O device 626 on each client computer.

[0089] As shown, output module 632 transmits communication data 639(1) to client computing device 606(1), and communication data 639(2) to client computing device 606(2), and communication data 639(3) to client computing device 606(3), etc. The communication data 639 transmitted to the client computing devices can be the same or can be different (e.g., the positioning of the content flow within the user interface can vary from one device to the next).

[0090] In various embodiments, the device(s) 610 and / or the client module 620 can include a GUI presentation module 640. The GUI presentation module 640 can be configured to analyze the communication data 639 for delivery to one or more of the client computing devices 606. Specifically, at the device(s) 610 and / or the client computing device 606, the UI presentation module 640 can analyze the communication data 639 to determine an appropriate manner for displaying the video, image, and / or content on the display screen 629 of the associated client computing device 606. In some embodiments, the GUI presentation module 640 can provide the video, image, and / or content to a presentation GUI 646 that is presented on the display screen 629 of the associated client computing device 606. The presentation GUI 646 can be rendered on the display screen 629 by the GUI presentation module 640. The presentation GUI 646 can include the video, image, and / or content analyzed by the GUI presentation module 640.

[0091] In some implementations, the presentation GUI 646 may include multiple sections or grids that may render or include videos, images, and / or content for display on the display screen 629. For example, a first section of the presentation GUI 646 may include a video feed of a presenter or individual, and a second section of the presentation GUI 646 may include a video feed of the individual consuming conference information provided by the presenter or individual. The GUI presentation module 640 may populate the first section and the second section of the presentation GUI 646 in a manner that appropriately mimics an ambient experience that the presenter and individual may share.

[0092] In some embodiments, the GUI presentation module 640 can magnify or provide a zoomed view of the individual represented by the video feed to highlight the individual's reaction to the presenter, such as facial features. In some embodiments, the presentation GUI 646 can include video feeds of multiple participants associated with a meeting, such as a general communication session. In other embodiments, the presentation GUI 646 can be associated with a channel, such as a chat channel, an enterprise team channel, etc. Therefore, the presentation GUI 646 can be associated with an external communication session that is different from a general communication session.

[0093] Fig. 9 A diagram showing example components of an example device 700 (also referred to herein as a "computing device") configured to generate data for some of the user interfaces disclosed herein is illustrated. Device 700 can generate data that can include one or more portions of a video, image, virtual object, and / or content that can be rendered or included for display on display screen 629. Device 700 can represent one of the device(s) described herein. Additionally or alternatively, device 700 can represent one of client computing devices 606.

[0094] As illustrated, device 700 includes one or more data processing units 702, computer readable media 704, and communication interface(s) 706. The components of device 700 are operably connected, for example, via bus 709, which may include one or more of a system bus, a data bus, an address bus, a PCI bus, a Mini-PCI bus, and any of a variety of local, peripheral, and / or independent buses.

[0095] As used herein, data processing unit(s) (such as data processing unit(s) 702 and / or data processing unit(s) 692) may represent, for example, a CPU-type data processing unit, a GPU-type data processing unit, a field programmable gate array (“FPGA”), another type of DSP, or other hardware logic components that may be driven by a CPU in some cases. By way of example and not limitation, illustrative types of hardware logic components that may be utilized include application specific integrated circuits (“ASICs”), application specific standard products (“ASSPs”), systems on chips (“SOCs”), complex programmable logic devices (“CPLDs”), and the like.

[0096] As used herein, computer-readable media (such as computer-readable media 704 and computer-readable media 694) can store instructions that can be executed by (one or more) data processing units. Computer-readable media can also store instructions that can be executed by external data processing units, such as by an external CPU, an external GPU, and / or can be executed by an external accelerator, such as an FPGA type accelerator, a DSP type accelerator, or any other internal or external accelerator. In various examples, at least one CPU, GPU, and / or accelerator is incorporated into the computing device, and in some examples, one or more of the CPU, GPU, and / or accelerator is external to the computing device.

[0097] Computer readable media (also referred to herein as computer readable media) may include computer storage media and / or communication media. Computer storage media may include one or more of volatile memory, non-volatile memory and / or other persistent and / or secondary computer storage media, removable and non-removable computer storage media implemented in any method or technology for storing information such as computer readable instructions, data structures, program modules or other data. Thus, computer storage media includes media in tangible and / or physical form included in a device and / or hardware component that is part of a device or a hardware component external to a device, including, but not limited to: random access memory ("RAM"), static random access memory ("SRAM"), dynamic random access memory ("DRAM"), phase change memory ("PCM"), read-only memory ("ROM"), erasable programmable read-only memory ("EPROM"), electrically erasable programmable read-only memory ("EEPROM"), flash memory, compact disk read-only memory ("CD-ROM"), digital versatile disk ("DVD"), optical card or other optical storage medium, magnetic cassette, magnetic tape, magnetic disk storage, magnetic card or other magnetic storage device or medium, solid-state memory device, storage array, network attached storage device, storage area network, hosted computer storage device, or any other storage memory, storage device, and / or storage medium that can be used to store and maintain information for access by a computing device. Computer storage media may also be referred to herein as computer-readable storage media, non-transitory computer-readable storage media, non-transitory computer-readable media, or computer storage media.

[0098] In contrast to computer storage media, communication media may embody computer-readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave or other transport mechanism. As defined herein, computer storage media does not include communication media. That is, computer storage media itself does not include communication media consisting solely of a modulated data signal, a carrier wave, or a propagated signal.

[0099] The communication interface 706 may represent, for example, a network interface controller ("NIC") or other type of transceiver device for sending and receiving communications over a network. In addition, the communication interface 706 may include one or more video cameras and / or audio devices 722 to enable the generation of video feeds and / or still images, etc.

[0100] In the illustrated example, computer-readable medium 704 includes data storage 708. In some examples, data storage 708 includes a data storage, such as a database, a data warehouse, or other types of structured or unstructured data storage. In some examples, data storage repository 708 includes a corpus and / or a relational database having one or more tables, indexes, stored procedures, etc. to enable data access, including, for example, one or more of a Hypertext Markup Language ("HTML") table, a Resource Description Framework ("RDF") table, a Web Ontology Language ("OWL") table, and / or an Extensible Markup Language ("XML") table.

[0101] The data store 708 may store data for operations of processes, applications, components, and / or modules stored in the computer-readable medium 704 and / or executed by (one or more) data processing units 702 and / or (one or more) accelerators. For example, in some examples, the data store 708 may store data structures 190, supplemental data 191, and session data 636. The session data 636 may include the total number of participants (e.g., users and / or client computing devices) in the communication session, the activities occurring in the communication session, the list of invitees to the communication session, and / or other data related to when and how the communication session is conducted or hosted. The session data may also include contextual data, such as content including video, audio, or other content for presentation and display on one or more display screens. The hardware data 711 may define aspects of any device, such as multiple display screens of a computer. The contextual data may define any type of activity or state associated with individual users 10A-10L associated with each of the multiple video streams 634. For example, contextual data may define the levels of people in an organization, how each person's level relates to the levels of other people, the performance level of people, or any other activity or state information that may be used to determine the location of a rendering of a person within a virtual environment. This contextual information may also be fed into any model to help emphasize key words spoken by a person of a certain level, highlight the UI when background voices of a person of a certain level are detected, or change the emotion display in a specific way when a person of a certain level is detected to have a certain emotion.

[0102] Alternatively, some or all of the above data may be stored on a separate memory 716 on one or more data processing units 702, such as a memory on a single board, a CPU-type processor, a GPU-type processor, an FPGA-type accelerator, a DSP-type accelerator, and / or another accelerator. In this example, the computer-readable medium 704 also includes an operating system 718 and an application programming interface 710 (API) configured to disclose functions and data of the device 700 to other devices. In addition, the computer-readable medium 704 includes one or more modules, such as a server module 730, an output module 732, and a GUI presentation module 740, although the number of modules shown is only an example and the number may vary. That is, the functionality described in conjunction with the modules described herein may be performed by a smaller number of modules or a larger number of modules on one device or spread across multiple devices.

[0103] In summary, although various configurations have been described in language specific to structural features and / or methodological actions, it should be understood that the subject matter defined in the accompanying representations is not necessarily limited to the specific features or actions described. Instead, the specific features and actions are disclosed as example forms of implementing the claimed subject matter.

Claims

1. A computer-implemented method for adjusting a viewing perspective associated with a target entity, the target entity being one of a plurality of entities located within a 3D environment, the method being for execution on a system, the method comprising: accessing a data structure defining the 3D environment, the data structure defining a position and orientation for each of the plurality of entities of the 3D environment, the data structure defining at least one viewing perspective, the at least one viewing perspective being based on at least one viewpoint for an individual entity, the individual entity selectable for use by a computing device for a communication session to generate a customized rendering of the at least one viewing perspective; receiving input indicating a selection of the target entity from the plurality of entities, wherein the input causes the system to modify a viewing perspective from a viewpoint associated with the target entity; and generating, in response to the input indicating the selection of the target entity, supplemental model data defining at least one of a new position or a new orientation for at least one of the plurality of entities, wherein display of at least one selection surface of the at least one entity positioned within the viewing perspective associated with the target entity is augmented for the new position or the new orientation of the at least one entity; and Using the supplemental model data defining at least one of a new position or a new orientation for the at least one entity to cause display of the customized rendering for the computing device, wherein the customized rendering is based on the viewing perspective originating from the viewpoint associated with the target entity, wherein the supplemental model data defining at least one of the new position or the new orientation for the at least one entity is used to generate the customized rendering for the computing device without causing modification to other perspective views associated with another viewpoint of other entities.

2. The computer-implemented method of claim 1 , wherein: The supplemental model data defining at least one of the new position or the new orientation increases the display of at least one selected surface of the at least one entity in the custom rendering, wherein the custom rendering is based on a viewing perspective relative to the display of at least one selected surface of the at least one entity when using the position and the orientation of the at least one entity defined in the data structure, wherein the new position and the new orientation are generated without modifying the position and the orientation of the at least one entity defined in the data structure.

3. The computer-implemented method of claim 1 , wherein: The computer-implemented method also includes: selecting the at least one entity from the plurality of entities, wherein, based on the data structure defining the 3D environment, the at least one entity is selected if a portion of the at least one entity is within the viewing perspective associated with the target entity, wherein the at least one entity is selected for determining the new position or the new orientation, and wherein other entities that are outside the viewing perspective associated with the target entity are not selected for determining the new position or the new orientation.

4. The computer-implemented method of claim 1 , wherein: The computer-implemented method also includes: selecting the at least one entity from the plurality of entities, wherein, based on the data structure defining the 3D environment, the at least one entity is selected if a threshold portion of the at least one entity is within the viewing perspective associated with the target entity, wherein the at least one entity is selected for determining the new position or the new orientation, and wherein other entities having less than the threshold portion within the viewing perspective associated with the target entity are not selected for determining the new position or the new orientation.

5. The computer-implemented method of claim 1 , wherein: The computer-implemented method also includes: selecting the at least one entity from the plurality of entities, wherein, based on the data structure defining the 3D environment, the at least one entity is selected if a threshold portion of the at least one entity is outside the viewing perspective associated with the target entity, wherein the at least one entity is selected for determining the new position or the new orientation, and wherein other entities having less than the threshold portion outside the viewing perspective associated with the target entity are not selected for determining the new position or the new orientation.

6. The computer-implemented method of claim 1 , wherein: The method further comprises: determining that the at least one entity has at least a threshold portion outside of the viewing perspective; and In response to determining that the at least one entity has at least the threshold portion outside the viewing angle, a width or a height of the viewing angle is modified to increase the display of the at least one surface of the at least one entity.

7. The computer-implemented method of claim 1 , wherein: The permission data allows a user associated with the computing device to access the supplemental model data, while the permission data restricts other users of other computing devices from accessing the supplemental model data.

8. The computer-implemented method of claim 1 , wherein: The generation of the supplementary model data does not cause modification of the data structure defining the 3D environment, and during the generation or the modification of the supplementary model data, the position and the orientation of each entity of the multiple entities of the 3D environment defined in the data structure are not modified, wherein the system permits limiting the modification of the data structure during the modification or the generation of the supplementary model data to preserve the other perspective view associated with the other viewpoint of the other entity.

9. A computing device for adjusting a viewing perspective associated with a target entity selected from a plurality of entities positioned within a 3D environment, comprising: one or more processing units; as well as A computer-readable storage medium having computer-executable instructions encoded thereon to cause the one or more processing units to: accessing a data structure defining the 3D environment, the data structure defining a position and orientation for each of the plurality of entities of the 3D environment, the data structure defining at least one viewing perspective, the at least one viewing perspective being based on at least one viewpoint for an individual entity, the individual entity selectable for use by a computing device for a communication session to generate a customized rendering of the at least one viewing perspective; receiving input indicating a selection of the target entity from a plurality of entities, wherein the input causes the system to modify a viewing perspective from a viewpoint associated with the target entity; and generating, in response to the input indicating the selection of the target entity, supplemental model data defining at least one of a new position or a new orientation for at least one of the plurality of entities, wherein display of at least one selection surface of the at least one entity positioned within the viewing perspective associated with the target entity is augmented for the new position or the new orientation of the at least one entity; and Using the supplemental model data defining at least one of a new position or a new orientation for the at least one entity to cause display of the customized rendering for the computing device, wherein the customized rendering is based on the viewing perspective originating from the viewpoint associated with the target entity, wherein the supplemental model data defining at least one of the new position or the new orientation for the at least one entity is used to generate the customized rendering for the computing device without causing modification to other perspective views associated with another viewpoint of other entities.

10. The computing device of claim 9, wherein: The supplemental model data defining at least one of the new position or the new orientation increases the display of at least one selected surface of the at least one entity in the custom rendering, wherein the custom rendering is based on a viewing perspective relative to the display of at least one selected surface of the at least one entity when using the position and the orientation of the at least one entity defined in the data structure, wherein the new position and the new orientation are generated without modifying the position and the orientation of the at least one entity defined in the data structure.

11. The computing device of claim 9, wherein: The computer-executable instructions also cause the one or more processing units to select the at least one entity from the plurality of entities, wherein, based on the data structure defining the 3D environment, the at least one entity is selected if a portion of the at least one entity is within the viewing perspective associated with the target entity, wherein the at least one entity is selected for determining the new position or the new orientation, and wherein other entities that are outside the viewing perspective associated with the target entity are not selected for determining the new position or the new orientation.

12. The computing device of claim 9, wherein: The computer-executable instructions also cause the one or more processing units to select the at least one entity from the plurality of entities, wherein, based on the data structure defining the 3D environment, the at least one entity is selected if a threshold portion of the at least one entity is outside the viewing perspective associated with the target entity, wherein the at least one entity is selected for determining the new position or the new orientation, and wherein other entities having less than the threshold portion outside the viewing perspective associated with the target entity are not selected for determining the new position or the new orientation.

13. The computing device of claim 9, wherein: The computer executable instructions further cause the one or more processing units to: determining that the at least one entity has at least a threshold portion outside of the viewing perspective; as well as In response to determining that the at least one entity has at least the threshold portion outside the viewing angle, a width or a height of the viewing angle is modified to increase the display of the at least one surface of the at least one entity.

14. The computing device of claim 9, wherein: The permission data allows a user associated with the computing device to access the supplemental model data, while the permission data restricts other users of other computing devices from accessing the supplemental model data.

Images