Intelligent management of content related to objects displayed within a communication session

Abstract

The technology disclosed in this article improves the efficiency of the system by providing intelligent management of content associated with objects displayed within a communication session. Participants can generate content objects associated with 3D objects. The content objects can be in the form of 3D virtual objects, such as arrows pointing to a table, text boxes for annotations, etc. The content objects can also include functional features for collecting and displaying information voting agents. The system can generate a data structure that associates the objects with the content objects. When various operations are applied to any object, the data structure enables the system to maintain the association between the object and the content object. Therefore, if a remote computer sends a request for the content object, the associated object is delivered together with the content object.

Description

Technical Field

[0001] The present invention relates to intelligent management of content displayed within a communication session. Background Art

[0002] There are many platforms that provide virtual reality ("VR") and mixed reality ("MR") environments. Users are able to create, edit, and share two-dimensional (2D) and three-dimensional (3D) objects in a collaborative environment facilitated by a communication session. Although some existing applications provide users with many features for creating and sharing multi-dimensional objects, some existing systems have many shortcomings. For example, when a group of users creates objects within a particular communication session (e.g., a private chat, conference, or broadcast), it may be difficult for the users to share those objects with users participating in other communication sessions.

[0003] For example, when a participant in one communication session wishes to share a 3D object and content associated with the 3D object with a user of another communication session, the participant is often required to save a specific version of the object and the associated content and take some manual steps to share the object and the associated content with other users. This manual process of managing objects and associated content can be time consuming and inefficient when computing resources (e.g., memory resources, processing resources, network resources, etc.) are involved.

[0004] For purposes of illustration, consider a scenario where a participant in one communication session (such as a group editing session) wishes to share a 3D object and annotations associated with the 3D object with a user of a private chat session. To facilitate such a transfer, the user may be required to create a file for the object and a file for the associated annotations. The user may then be required to share these two files with another communication session. This process results in inefficiencies with respect to computing resources because the user needs to retrieve, recreate, and transfer multiple files each time data is transferred between different types of communication sessions.

[0005] These shortcomings of existing systems become more complex as users increase the number of content objects associated with a 2D or 3D object. For example, if a user adds annotations and virtual drawing objects to a 3D object, the user must perform multiple manual steps to share the 3D object and associated content with users of different communication sessions, which may operate on different types of platforms. When there are a large number of content objects, the management of those content objects can become a challenge from a logistical perspective. The user must even perform many manual steps to manage the files that define the content objects and associated 3D objects. Summary of the invention

[0006] The technology disclosed herein improves the efficiency of the system by providing intelligent management of content associated with objects displayed within a communication session. For example, consider a scenario in which a participant of a communication session is viewing a three-dimensional (3D) object (such as a table in a room). The participant can generate a content object associated with the 3D object. The content object can be in the form of a 3D virtual object, such as an arrow pointing to the table, a text box containing annotations related to the table, etc. The content object can also include functional features that collect and display information, such as a question and answer (Q&A) object, a voting poll object, etc. In order to simplify the operation of managing the content object, the system generates a data structure that associates an object (e.g., a 3D table in a 3D environment) with a content object (e.g., a virtual arrow pointing to the table). The data structure enables the system to maintain a graphical association between the object and the content object in response to a user interaction with the object or content object. The data structure also enables the system to automatically share objects and associated content objects with a computing device of a remote communication session in response to one or more predetermined events. The permissions defined in the data structure can control the user's ability to move, edit, or transmit the object or content object.

[0007] In an illustrative example, the system can provide a view of a virtual object or a real-world object in a mixed reality environment. A user can position a content object in association with a virtual object or a real-world object (both of which are collectively referred to herein as "objects"). The content object can be in the form of a note, a message, a data collection agent, an annotation, an arrow, or any other data or operation that can be rendered and utilized in association with a selected object. The system can generate a data structure defining the association between an object and a content object. In some embodiments, an object is defined as a "parent object" and a content object is defined as a "child object." The data structure can define three-dimensional coordinates for a parent object and a child object. The data structure can also define the relative position between a parent object and a child object, for example, a parent object is below, above, or beside a child object. The data structure can also define functions that can be used to display and collect information, such as Q&A or voting polls.

[0008] The system can detect one or more predetermined events. For example, when a user provides a gesture indicating that a parent object is desired to be moved within a display environment, the system can move the parent object and the child object in a coordinated manner to maintain a graphical association between the two items. In addition, the system can transmit the parent object and the child object to a remote communication session based on one or more events. In an illustrative example, the system monitors multiple remote communication sessions and determines when a specific communication session references a parent object or a child object. For example, a participant participating in a private chat may reference a specific parent object, such as a table that is the subject of an MS team meeting. In response to the reference, the system can automatically retrieve the data structure to share and display the parent object and the child object with the computing device of the private chat session without the manual task of retrieving the parent object or the child object from the originating communication session. The data structure defining the parent object, the child object, and the related permissions can be used to control the transmission of the data structure and whether a specific user of the control remote communication session can display, edit, or access the parent object or the child object.

[0009] The technology disclosed in this article provides many features that improve existing computers. For example, as users switch between different sessions, computing resources such as processor cycles, memory, network bandwidth and power are used more efficiently. Content can be shared and displayed between sessions without recreating data or content for users. In addition, by providing recommended automatically generated visualizations or automatically generated resource recommendations for individual objects, recommendations enable users to make more effective decisions. The technology disclosed in this article also improves user interaction with various types of computing devices. Improvements to user interaction or reductions in the need for user input can alleviate unintentional input, redundant input, and other types of user interactions that utilize computing resources. Other technical advantages not specifically mentioned in this article can also be achieved by implementing the disclosed subject matter.

[0010] Those skilled in the art will also appreciate that aspects of the subject matter described herein can be practiced on or in conjunction with other computer system configurations than those specifically described herein, including multi-processor systems, microprocessor-based or programmable consumer electronic devices, AR, VR and MR devices, video gaming devices, handheld computers, smartphones, smart televisions, self-driving cars, smart watches, e-readers, tablet computing devices, dedicated hardware devices, networked devices, and other devices.

[0011] Features and technical advantages other than those explicitly described above will be apparent by reading the following detailed description and reviewing the associated drawings. This Summary is provided to introduce a selection of concepts that are further described in the detailed description below in a simplified form. This Summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used to assist in determining the scope of the claimed subject matter. For example, the term "technology" may refer to (one or more) systems, (one or more) methods, computer-readable instructions, (one or more) modules, algorithms, hardware logic, and / or (one or more) operations as allowed by the context described above and throughout this document. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The specific embodiments are described with reference to the accompanying drawings. In the drawings, the leftmost (one or more) digits of a reference number identify the figure in which the reference number first appears. The same reference numbers in different figures indicate similar or identical items. References to individual items in a plurality of items can use reference numbers with letters of the alphabetical sequence to refer to each individual item. General references to items can use specific reference numbers without the alphabetical sequence.

[0013] Figure 1 An exemplary scenario involving a system 100 that can be used to implement the techniques disclosed herein is illustrated.

[0014] Figure 2A A process involving input for moving an object is illustrated.

[0015] Figure 2B Illustrated is a process involving moving a content object in response to input applied to the object.

[0016] Figure 2C The process involving movement of content objects and rendering of the objects is illustrated.

[0017] Figure 2D The result of the movement of the content object and the rendering of the object is illustrated.

[0018] Figure 3A An example of a parent object associated with multiple child objects is illustrated.

[0019] Figure 3B The results of a move operation applied to a parent object associated with multiple child objects are illustrated.

[0020] Figure 4A The starting positions of real-world objects and content objects are illustrated.

[0021] Figure 4B The results of input applied to a content object associated with a real-world object are illustrated.

[0022] Figure 4C Illustrated is a rendering of a real-world object displayed in conjunction with a content object.

[0023] Figure 4D Movement of a rendering of a real-world object in response to a moving person applied to an associated content object is illustrated.

[0024] Figure 5A are block diagrams illustrating two embodiments of data structures for defining associations between objects and content objects in the form of Q&A objects.

[0025] Figure 5B is a block diagram illustrating another embodiment of a data structure defining a content object in the form of a voting poll object.

[0026] Figure 6 is a block diagram illustrating the transmission of data structures and content updates between communication sessions.

[0027] Fig. 7A The diagram illustrates a scenario in which two computers in different communication sessions can simultaneously display objects of content objects of a single data structure.

[0028] Figure 7B A scenario is illustrated in which two computers in different communication sessions are able to display and receive updates to a Q&A object.

[0029] Fig. 8A is a block diagram illustrating the communication of data structure and content updates in response to user commands applied to content objects.

[0030] Figure 8B is a block diagram illustrating the transmission of a data structure to a server in response to a user command applied to a content object.

[0031] Fig. 9 is a block diagram illustrating the transmission of one or more data structures in response to events detected at different communication sessions.

[0032] Fig.10 is a flow chart illustrating aspects of a routine for computationally efficient management of content.

[0033] Fig.11 is a computing system diagram showing aspects of an illustrative operating environment for the techniques disclosed herein.

[0034] Fig.12 is a computing architecture diagram illustrating various aspects of the configuration and operation of a computing device capable of implementing various aspects of the techniques disclosed herein.

[0035] Fig.13is a computing device diagram illustrating aspects of the configuration and operation of an MR device capable of implementing aspects of the disclosed technology, according to one embodiment disclosed herein. DETAILED DESCRIPTION

[0036] Figure 1 An exemplary scenario involving a system 100 that can be used to implement the technology disclosed herein is illustrated. The system 100 can be configured to provide a collaborative environment that facilitates communication between two or more computing devices. The system that provides the collaborative environment can allow participants to exchange live video, live audio, and other forms of data within a communication session. The collaborative environment can be any suitable communication session format, including but not limited to: a private chat session, a multi-user editing session, a group meeting, a broadcast, etc.

[0037] The system 100 can facilitate communication sessions between any suitable number of computing devices. In this example, the system 100 facilitates communication sessions between a first computing device 101A, a second computing device 101B, a third computing device 101C, and a fourth computing device 101D. The computing devices can be in any form, such as a laptop, a desktop computer, a tablet computer, a phone, a virtual reality headset, or a mixed reality device.

[0038] exist Figure 1 In the example shown in , the first computing device 101A is a mixed reality device that displays and provides an augmented view 120. The augmented view 120 includes a view of real-world objects 110 (also referred to herein as "physical objects") in a real-world environment 112. The augmented view 120 can also display computer-generated objects that are displayed simultaneously with the view of the real-world environment 112 to augment one or more real-world objects 110.

[0039] In some embodiments, the computer-generated objects can be superimposed on the real-world view by using a prism that provides the user with a direct line-of-sight view of the real-world objects 110 and the real-world environment 112. Thus, the user can physically see the real-world objects 110 and the real-world environment 112 through the prism. The prism allows the user to see natural light reflected from the real-world objects 110 and the real-world environment 112, while also allowing the user to see light generated from a display device used to render the computer-generated objects. By directing light from the real-world objects 110 and light from the device used to render the computer-generated objects toward the user's eyes, the prism allows the system to enhance aspects of the real-world view by providing a coordinated display of the computer-generated objects. Although a prism is utilized in this example, it can be appreciated that other optical devices can be utilized to generate the augmented view 120. For example, in an alternative embodiment, the mixed reality device can capture an image of the real-world objects 110 and the real-world environment 112 and display the image on a display screen, wherein the computer-generated objects can enhance the image of the real-world objects 110.

[0040] In some embodiments, the first computing device 101A utilizes an imaging device such as a camera to capture images of the real-world objects 110 and the real-world environment 112. The first computing device 101A can also include a sensor for generating model data defining a three-dimensional (3D) model of the real-world objects and the real-world environment 112. The model data and the images can be shared with other computing devices to generate a 3D rendering or a 2D rendering of the real world 110 and the real-world environment 112.

[0041] The second computing device 101B is a virtual reality device capable of displaying a 3D virtual environment 121 to a user. In this example, the virtual reality device displays a 3D rendering of an object 110' within a rendering of a real-world environment 112. The model data and image data communicated from the first computing device 101A can define aspects of the real-world object 110 and the real-world environment 112 to generate a rendering of the object 110' (also described herein as "rendering 110'").

[0042] The third computing device 101C is in the form of a desktop computer capable of displaying a 2D perspective of the collaborative environment. In this example, the 2D perspective takes the form of a virtual conference user interface 122. The desktop computer displays a 2D rendering of the object 110' and the real world environment 112 within the virtual user interface 122 using the model data and image data transmitted from the first computing device 101A.

[0043] The fourth computing device 101D is in the form of a mobile device capable of displaying a 2D perspective of the collaborative environment. In this example, the 2D perspective takes the form of a private chat user interface 123. The mobile device displays a 2D rendering of the object 110' within the private chat user interface 123 using the model data and image data transmitted from the first computing device 101A.

[0044] To illustrate aspects of the present disclosure, consider a scenario in which a user provides input at first computing device 101A that results in the generation of content object 111 to be placed within real-world environment 112. In some embodiments, content object 111 is positioned in association with real-world object 110 ("object 110"). For example, content object 111 can be positioned above, beside, or below object 110. In this particular example, content object 111 is positioned above object 110, which in this example is a table. Content object 111 is specifically associated with the right corner of the top surface of the table.

[0045] In some embodiments, the positions of object 110 and content object 111 can be defined by a coordinate system. Coordinate data for object 110 can be derived from depth map data captured by a sensor of first computing device 101A. Coordinate data for content object 111 can be derived from an input gesture by a user to place content object 111 within a rendered environment view. "Relative position" can be defined by an offset, which is generated based on the difference in the coordinate position of one object from the coordinate position of another object (such as content object 111). The offset can be defined by 3D coordinates, for example, a content object is offset (X=-5.0, Y=10, Z=-30) from an object located at position (X=-500, Y=103, Z=100).

[0046] The content object 111 can be in any form. For example, the content object 111 can be in the form of a 3D virtual object, such as an arrow pointing to a table, a text box including notes related to the table, etc. The content object 111 can include any type of data, including video data, audio data, animated graphics, images, etc. The content object can also include functional features for displaying and collecting information, such as a question and answer (Q&A) object, a voting poll object, etc. Figure 1In the example of , content object 111 is in the form of a question and answer ("Q&A") object, which is positioned in association with object 110 (e.g., a table). Content object 111 can also include animate objects, inanimate objects, such as a live video feed of a person or a live video feed of a computer-generated avatar. Similarly, object 110 can also be of any form. Object 110 can include a virtual object with functions, or object 110 can be a passive object with no functions. Therefore, both object 110 and content object 111 can have similar forms, for example, both can be Q&A objects, where one is considered a parent object and the other is considered a child object. Object 110 and content object 111 can be rendered in 2D or 3D format.

[0047] As will be described in more detail below, the Q&A object is configured to display one or more queries and allow participants to provide responses to the queries. The Q&A object is configured to display the queries and collect responses from users. The Q&A object can also be configured to selectively display the responses to participants of the communication session who have appropriate permissions defined in the data structure.

[0048] When the content object 111 is placed in association with the object, the system generates a data structure defining the association between the object and the content object. In some embodiments, the object can be defined as a "parent object" and the content object can be defined as a "child object". In some embodiments, the system can be allowed to have multiple child objects associated with a single parent object. The parent object and the child object can be associated with an identifier to assist the computing device in tracking each associated item. The data structure can define coordinates for the parent object and the child object and other associated data defined in the relative position between the parent object and the associated child object. The data structure is configured to enable the computing device to maintain a graphical association between the parent object and the child object. Therefore, when an object (e.g., a table) is moved or deleted, the associated content object (e.g., Q&A) is relocated or deleted to maintain a consistent visual association with the moved or deleted object. When a content object is moved or deleted, the associated object is relocated or deleted to maintain a consistent visual association with the moved or deleted content object.

[0049] The data structure can cause other computing devices to display renderings of content object 111. For example, input provided at first computing device 101A can result in creation of a data structure defining content object 111. The data structure can cause second computing device 101B to generate and display a rendering of content object 111 within a virtual environment. The data structure can cause third computing device 101C to generate and display a rendering of content object 111 within a two-dimensional conference environment (e.g., 122), and the data structure can cause fourth computing device 101D to generate and display a rendering of content object 111 within other user interface formats such as chat user interface 123.

[0050] The data structure enables the system to maintain the relationship between parent objects and child objects during multiple actions taken for each object. For example, the system can analyze input indicating an operation to be applied to a content object or objects. In response to detecting that the operation is to be applied to the content object, the system can apply the operation to the content object based on the association defined in the data structure and copy the operation to the object. For example, if the user input indicates that the user wishes to transfer the content object (e.g., Q&A) to a remote communication session, the system transfers the content object to the remote communication session. In addition, based on the permissions defined in the data structure, the system will selectively transfer the object to the remote communication session.

[0051] Similar operations may also occur when a user applies an operation to an object (e.g., a table). The system can analyze input indicating an operation to be applied to the content object or objects. In response to detecting that an operation is to be applied to an object, the system can apply the operation to the object based on the association defined in the data structure and copy the operation to the content object. For example, if the user input indicates that the user wishes to transfer an object (e.g., a table) to a remote communication session, the system transfers the object to the remote communication session. In addition, based on the permissions defined in the data structure, the system will selectively transfer the content object to the remote communication session.

[0052] The data structure can be used to affect a number of operations applied to a parent object or a child object. In some embodiments, the operations that can be applied to an object or a content object include, but are not limited to, graphical movement of an object and communication of an object. FIG. 2 to FIG. 4 are illustrative examples of "graphic movement" of an object, and FIG. Figures 6 to 9 An example of communication of objects is shown.

[0053] Figures 2A to 2D An example of how a content object may be moved in response to movement of an object in the form of a real-world object is illustrated. FIG. 3A to FIG. 3BAn example of how an image of a real-world object may be moved in response to movement of a content object is illustrated.

[0054] Figure 2A and Figure 2B A perspective view from the first computing device 101A is shown, which displays a view of the real world environment 112. Figure 2A and Figure 2B , a real-world object 110 (e.g., a table) is physically moved from a first position to a second position in the direction of an arrow. In this example, the table is physically moved within a room 112, and the sensor of the first computing device 101A tracks the movement.

[0055] As in Figure 2B As shown in , in response to detecting the movement of the table, the first computing device 101A moves the content object 111 to maintain the relative position between the content object 111 and the real-world object 110. When the real-world object 110 is in motion, the content object 111 is moved to track the movement of the real-world object 110, and the movement maintains the graphical association defined in the data structure. As described below, sensor data from the first computing device 101A or other sensors can be used to detect the movement of a physical object (such as a table). Although this example involves the use of a head-mounted display, any computing device that receives sensor data can control the movement of the content object 111.

[0056] A computing device utilizing the data structure can maintain a graphical association between content object 111 and real-world object 110 in a similar manner. For example, Figure 2C and Figure 2D The movement of the real-world object 110 and the content object 111 is shown from the perspective of the third computing device 101C displaying the virtual meeting user interface 122 . Figure 2C Renderings of real-world objects 110 and content objects 111 are shown in their original positions. Figure 2D Rendering of the real-world object 110 and the content object 111 is shown during and after movement of the real-world object 110. The position of the content object 111 tracks the position of the real-world object 110 as the real-world object 110 is moved.

[0057] Although the examples described above relate to examples where the parent object is an actual real-world object, it can be appreciated that the parent object can also relate to a virtual object. Figure 2A and Figure 2BIn the example shown in , object 110 can actually be a virtual object rendered within view 121, and the association between content object 111 and the virtual object can be stored within a data structure. Thus, in this example, object 110 can be moved using user input such as an input gesture, and in response to such input, content object 111 can be moved to maintain the graphical association between the two objects.

[0058] Figure 3A and Figure 3B An example of an object 110 associated with multiple content objects 111 is illustrated. In this example, object 110 (shown as a rendering of object 110') is associated with a first content object 111A (Q&A), a second content object 111B (notes), and a third content object 111C (arrows). In such a scenario, a data structure will be generated that associates each content object (111A-111C) with object 110. This example is illustrated from the perspective of a third computing device 101C displaying a virtual meeting user interface 122.

[0059] In this example, the user input indicates a selection of the first content object 111A. The user input also provides an indication of a new location for the first content object 111A. In response to receiving the user input indicating a new location for the first content object 111A, the rendering of the object 110' is moved to maintain the graphical association between the first content object 111A and the rendering of the object 110'. In addition, given the associations with the other content objects (111B-111C), the system also moves the other content objects (111B-111C) to maintain the graphical associations defined in the data structure. As shown in Figure 3B As shown in , the rendering of the object 110' is repositioned to maintain a relative position with respect to the first content object 111A. In addition, the second content object 111B and the third content object 111C are repositioned to maintain the same relative position with respect to the rendering of the object 110'.

[0060] FIG. 4A to FIG. 4D Techniques for resolving visual conflicts that may occur from a direct view of an object 110 provided by a mixed reality device are illustrated. Specifically, Figure 4A and Figure 4B1 illustrates a visual conflict that may occur when a mixed reality device provides a direct view of an object 110 in a real world environment 112. In this example, a user input is provided to move a content object 111. As described above, such input enables the computer to move the rendering of the object 110 associated with the content object 111. Such a feature works well with a 3D virtual environment 121, a virtual meeting user interface 122, and a private chat user interface 123, where the computer displays a rendering of the real world object 110. However, as in Figure 4A and Figure 4B As shown in the transition between , when the mixed reality device provides a direct view of the object 110, the mixed reality device cannot move the real-world object 110 viewed within the real-world environment 112.

[0061] In order to give users of mixed reality devices the same experience as users of other types of computing devices, some embodiments disclosed herein can generate a rendering of an object 110' (also referred to herein as rendering 110') that serves as an overlay on the real-world object 110. Figure 4C In the example shown in , a rendering of object 110' is generated and placed in direct view of the actual real-world object 110. The position of the rendering 110' can also be coordinated with the position of the content object 111. Thus, when input is received that causes the real-world object 110 to be moved, the mixed reality device can move the rendering 110' to maintain association with the content object 111. In some embodiments, such as in Figure 4D , the mixed reality device can also render a graphical element 401 to obstruct the direct view of the object 110. The graphical element 401 can be optional, as the system may need to provide a view of the real-world object 110 simultaneously with rendering the real-world object 110'. In such an embodiment, one or more graphical effects can be applied to the view of the real-world object 110 or the rendering of the real-world object 110', such as a dimming effect to help the user distinguish the physical object from the rendered object.

[0062] Reference now Figure 5A , additional aspects of data structure 501 are shown and described below. In some embodiments, data structure 501 can include association data 503, model data 505, and content data 507. In general, association data 503 defines the relationship between a parent object and a child object. Association data 503 can provide any information that enables a computing device to maintain a graphical association between a parent object and a child object. In one embodiment, association data 503 can define a relative position between a parent object and a child object. The relative position can be defined by an offset generated from a difference in the coordinate position of object 110 and the coordinate position of content object 111.

[0063] Model data 505 defines aspects of a parent object. Content data 507 can define aspects of a child object. Model data 505 and content data 507 can define images, textures, dimensions, colors, etc. Model data 505 and content data 507 can also define functional characteristics of parent and child objects. Model data 505 and content data 507 can also include keywords or phrases that can be used to provide context to parent and child objects.

[0064] Figure 5A Also illustrated is an example of a data structure 501' defining a relationship between an object (real world object or virtual object) and a content object in the form of a Q&A. In this example, associated data 503' defines a position for a parent object and a position for a child object. The position can be in any suitable format, including a point in a 3D or 2D coordinate system. In addition, associated data 503' defines a relative position between a real world object and a content object. The relative position between two objects can be defined by an offset. The parent object may be located at a certain position in the coordinate system, and the offset can define the position of the child object. The offset can be in any suitable format, such as a 3D (X, Y, Z) offset or a 2D (X, Y) offset that defines an increment between the position of the parent object and the position of the child object.

[0065] Model data 505' can generally define the physical properties of the parent object. Figure 5A In the example of , if the parent object is a real-world object, the model data 505' can define the dimensions, texture, color, and other aspects of the real-world object. The model data 505' can also include an image of the real-world object. If the parent object is a virtual object, the model data can define the coordinates of each surface, the texture of each surface, the color of each surface, etc.

[0066] Content data 507' can generally define any physical or functional characteristics of content object 111. For example, if content object 111 is a real-world object, content data 507' can define the dimensions, texture, color, and other aspects of the real-world object. Content data 507' can also include an image of content object 111. If content object 111 is a virtual object, the content data can define the coordinates of each surface, the texture of each surface, the color of each surface, and the like.

[0067] exist Figure 5AIn the example of , content object 111 is in the form of a Q&A. Therefore, content data 507' defines the user interface layout, the text format for the questions, and the text format for the answers. Content data 507' also includes functional features, such as user permissions and / or data filters. The permissions can define individual users or group users who can view the Q&A, and define the permissions for individual users or group users who can provide responses to the Q&A. In addition, the permissions can control the read and write permissions for individual answers to the Q&A. The filters can be used to display or hide selected answers that meet a threshold priority. Priority can be based on the level of relevance that the answer may have to the question.

[0068] Figure 5B Another example of a data structure 501" is illustrated that defines relationships between objects and content objects in the form of a poll. In this example, the associated data 503" and the model data 505" are similar to the examples described above. However, in this example, the associated data 503" includes user permissions. In this example, the permissions provide a first user (User 1) with read and write permissions for object 110 and content object 111. In addition, the permissions provide a second user (User 2) with read-only permissions for object 110 and content object 111, and the permissions restrict all other users from read and write access to object 110 and content object 111. Although this example provides permissions for individual users, it can be appreciated that the data structure can define permissions for a group of users for an object or content object. In some embodiments, the permissions can include definitions such as a null value that enables open access for all users to apply operations to an object or content object.

[0069] By using permissions defined in data structure 501", the computing system can control read and write access to objects or content objects relative to specific users or specific roles. Therefore, when objects and content objects are transferred to a remote computer, only certain people can build to modify parent objects or child objects. In the example above, the first user can move parent objects and child objects, but the second user can only view the objects.

[0070] It is also possible to assign more granular permissions to each user. For example, a particular user or group of users may be able to make changes to a content object, such as providing information updates (e.g., providing a vote or answer) or changing the position of a rendering relative to a parent object, but the permissions may restrict the user from making modifications to the parent object. In this example, the user may be able to move the content object around the parent object, such as moving the object from the side of the parent object to the top of the parent object, but the user may not be able to change the position of the parent object.

[0071] Return to Figure 5B, content data 507" illustrates exemplary properties of a content object in the form of a voting poll. In this example, content data 507" defines a format for a poll description, data defining a chart format, and functional features such as features for user participation and user permissions. Thus, based on permissions defining the roles of various participants, certain voters are able to cast weighted votes over other voters. Similarly, based on permissions defining the roles of various participants, certain users or groups of users may have access to certain aspects of the results, as well as permissions to vote. These examples are provided for illustrative purposes and should not be construed as limiting. It will be appreciated that the permissions can provide fine-grained control over time and content. For example, a particular identity may have the ability to access and edit an object or content object at a predetermined time or predetermined time slot.

[0072] By using the permissions of data structure 501, the system can control whether certain users or groups of users can access or edit specific iterations of a file. For example, consider a scenario where an employee is editing a file and makes multiple iterations of the file. The employee can set permissions for certain users to access specific iterations (e.g., certain versions of a file), and the access level can be based on time. For example, an employee can allow a specific team to have read-only access for a certain part of the workday, and then provide edit access for another part of the workday.

[0073] As outlined above, data structure 501 can be generated by one or more computing devices participating in a communication session. Data structure 501 is configured and arranged to allow parent objects and child objects to be transmitted to other computing devices of other communication sessions. Data structure 501 can be sent to multiple different types of communication sessions based on one or more events or conditions. In one illustrative example, data structure 501 can be transmitted to a remote communication session in response to user input indicating that an object or content object is shared with the remote communication session. In another illustrative example, data structure 501 can be transmitted to the remote communication session if user activity of the remote communication session indicates user activity related to a context defined within the data structure. Figure 6 Examples of these features are illustrated in .

[0074] exist Figure 6In the example of , data structure 501 is generated by one or more computing devices (101A-101D) of the first communication session 1104A. Data structure 501 can include definitions of two or more objects with a parent-child association. In response to one or more events, data structure 501 can be transmitted to any computing device (101E-101K) of the second communication session 1104B or any computing device (101L-110R) of the third communication session 1104C. In an illustrative embodiment, data structure 501 can be transmitted to the second communication session or the third communication session based on a request received from any computing device (101E-101R) of the remote communication session. In another illustrative embodiment, data structure 501 can be pushed to the second communication session or the third communication session from a command issued by the computing device (101A-101D) of the first communication session 1104A. In yet another embodiment, user activity detected at any computing device (101E-101R) indicating interest in an object defined in data structure 501 can invoke a request for data structure 501. Thus, any user activity (such as a conversation, chat, or gesture) that includes a keyword or other identifier with a threshold level similar to the context defined in data structure 501 results in delivery of data structure 501 to the appropriate communication session. When data structure 501 includes functional capabilities to collect information, such as Q&A or polling, data structure 501 enables a remote communication session to collect information from participants of the remote communication session and send content updates 602 back to the originating communication session. Thus, a Q&A or voting poll generated by a first communication session 1104A can easily share information with other communication sessions to collect votes or feedback from users who are not participating in the communication session that originated data structure 501.

[0075] Fig. 7AAn example is illustrated in which a third computing device 101C of a first communication session 1104A initiates a Q&A (content object 111) associated with a rendering of an object 110'. In this example, the chat is managed by a computing device 101M of a third communication session 1104C. The user participating in the chat indicates interest in the table referenced in the first communication session. In response to detection of such user activity, a data structure defining the table and the associated Q&A is transmitted to the third communication session 1104C. In this example, a second computing device 101M of the third communication session 1104C receives a data structure 501 and displays various aspects of the Q&A associated with the image of the object (e.g., a table). In this example, content objects and objects are displayed based on the context of the data structure 501 and the user interface 123. In this example, the layout display of the objects and content objects is modified to accommodate a specific category of user interfaces. To implement this feature, the computing device can analyze the user interface of the application. If the user interface is within a specific category, such as a chat user interface, an MS team user interface, a virtual reality user interface, or a user interface of any other category, the computing device can adjust the layout of the content object. As in Fig. 7A As shown in the lower right user interface of , assuming that the object is displayed on the private chat user interface 123, the content object 111 is displayed in a different orientation relative to the object 110. However, on the team user interface 122, the content object 111 is displayed in a different orientation relative to the rendering of the object 110'.

[0076] exist Fig. 7A In another feature shown, a user of the second computing device 101M of the third communication session 1104C can automatically connect to the first communication session 1104A by interacting with the user interface 123. For example, the user can select a rendering of the object 110' and in response to such input, the second computing device 101M of the third communication session 1104C can automatically connect to the first communication session 1104A. The concept referred to herein as "teleporting" allows a user to switch to a different communication session of the originating object 110 or content object 111. This feature not only allows the user to discover the existence of related communication sessions, but the feature gives the user an easy way to move to a different communication session without the manual steps of disconnecting and reconnecting the device from the communication session.

[0077] Continue in Figure 7B In this example, when the user provides an additional answer to the Q&A (A3), one or more computers of the third communication session 1104C provide content updates to one or more computers of the first communication session 1104A ( Figure 6 In response, such as Figure 7BAs shown in the lower right UI of , the computer (such as computing device 101C) participating in the first communication session can easily display additional answers provided by the computer of the third session.

[0078] Content update 602 can include a variety of different types of information. Figure 7B , if the user of computing device 101M moves the rendering of object 110' to the left or right, computing device 101C will receive content update 602 indicating the input and move the rendering of object 110' and the associated content object 111. In such an example, computing device 101C will only move the rendering of object 110' and the associated content object 111 if data structure 501 provides appropriate permissions for allowing the user to invoke graphical movement of the object.

[0079] Fig. 8A Another example of a mechanism capable of causing the transmission of a data structure 501 is illustrated. In this example, a user of the computing device 101C of the first communication session provides a gesture to move a content object 111 to a remote communication session (a second communication session 1104B). In this example, the content object 111 relates to a voting poll. In response to the input, the data structure 501 is transmitted to the computing device of the second communication session. Once the computer of the second communication session 1104B receives the data structure 501, the child object and the parent object are displayed in association with each other. In this example, the data structure 501 causes the third computing device 101G of the second communication session 1104B to display various aspects of the voting poll. In this example, the data structure 501 instructs the computing device to display a description of the voting poll, results with a specific chart type (e.g., a bar chart, a line chart, a scatter chart, etc.), and one or more input elements for receiving input from one or more users. In this example, the user of the computing device 101G has selected a second graphical element indicating a vote for the selected candidate. Such user input can cause the computing device 101G to generate content updates, and can send such content updates back to the originating communication session, ie, the first communication session 1104A.

[0080] Figure 8B Another example of a mechanism that enables a computing device to share a data structure 501 with a central service (e.g., server 801) to provide persistent storage of the data structure 501 is illustrated. In this example, a user of the computing device 101C of the first communication session 1104A provides a gesture to save a content object, which in this case is in the form of a Q & A. The gesture to save the content object causes the computing device to save the entire data structure that defines the content object and its associated objects.

[0081] The act of receiving data structure 501 in a remote device such as server 801 can be based on a number of actions. For example, a computing device managing a communication session (such as computing device 101C) can periodically save data structure 501 to any storage medium that allows any computer of the communication session to access data structure 501. Data structure 501 can also be saved in response to an event, such as an update to a content object or objects, or based on other events.

[0082] In an illustrative embodiment, data structure 501 can also include historical data. The historical data can provide a history of all edits and all position moves applied to a content object or object. The historical data can also provide a specific description of the identity of each individual and the operations applied to the content object or object by each individual. The historical data can also provide a description of the communication session associated with each individual applied to the content object or object in the operation. The historical data can be used by any computing device for a variety of purposes. For example, if a particular individual has applied multiple edits to a particular object, and the particular individual is mentioned in a remote communication session, such as a private chat session between other users, the data structure defining the particular object can be automatically transferred to the private chat session.

[0083] In this example, data structure 501 is stored in memory device 803 of server 801. Server 801 can be configured to persistently store any number of data structures 501A to 501N, and has the ability to transmit any of the data structures to a communication session or individual computing device, indicating a need for a content object or computing process of an object described in data structure 501. Fig. 9 As shown in , any communication session, such as the second communication session 1104B and the third communication session 1104C, can retrieve the data structure from the server 801. In addition, the server 801 can update the content of each data structure by receiving content updates 602 from the computing devices of each communication session.

[0084] These examples are provided for illustrative purposes and should not be construed as limiting. Although these examples perform one or more operations based on user input, it can be appreciated that one or more operations can be invoked by detecting a predetermined event. For example, a specific user scenario such as the presence of a specific user in a communication session or the presence of specific content may result in invoking one or more operations. In other examples, an intelligent agent can invoke one or more operations. For example, an object or content object can be an autonomous agent. In such an embodiment, the autonomous agent can decide to migrate from one communication session to another based on a threshold level of relevance to the shared content of the communication session.

[0085] Although these examples illustrate user input to a specific type of content object (e.g., a Q&A or voting poll), it can be appreciated that a data structure defining an object of a child object can be transmitted in response to a gesture to other objects (such as a parent object). By indicating a command for a selection of a parent or child using the techniques disclosed herein, the computing device is configured to apply an operation to the selected parent or child, and also copy the same operation to the associated parent or child based on the properties defined in data structure 501.

[0086] Fig.10 1 is a flow chart illustrating aspects of a routine 1000 for computationally efficient management of content associated with an object displayed within a communication session. It should be understood by those of ordinary skill in the art that the operations of the methods disclosed herein are not necessarily presented in any particular order, and that it is possible and contemplated to perform some or all of the operations in (one or more) alternative orders. For ease of description and illustration, the operations have been presented in the order presented. Operations may be added, omitted, performed together, and / or performed simultaneously without departing from the scope of the appended claims.

[0087] It should also be understood that the illustrated method can be terminated at any time and need not be fully executed. Some or all of the operations of the method and / or substantially equivalent operations can be performed by executing computer-readable instructions contained on a computer storage medium, as defined herein. The term "computer-readable instructions" and its variants used in the specification and claims are used broadly herein to include routines, applications, application modules, program modules, programs, components, data structures, algorithms, etc. Computer-readable instructions can be implemented on a variety of system configurations, including single-processor or multi-processor systems, minicomputers, mainframe computers, personal computers, handheld computing devices, microprocessor-based programmable consumer electronics, combinations thereof, etc.

[0088] It should therefore be appreciated that the logical operations described herein are implemented as: (1) a sequence of computer-implemented actions or program modules running on a computing system such as those described herein, and / or (2) interconnected machine logic circuits or circuit modules within the computing system. The implementation is a matter of choice depending on the performance and other requirements of the computing system. Thus, the logical operations may be implemented in software, firmware, dedicated digital logic, and any combination thereof.

[0089] In addition, Fig.10The operations illustrated in the other figures can be implemented in association with the exemplary presentation UI described above. For example, the various devices and / or modules described herein (one or more) can generate, send, receive and / or display data associated with the content of a communication session (e.g., live content, broadcast events, recorded content, etc.) and / or present a UI that includes a rendering of one or more participants of a remote computing device, avatar, channel, chat session, video stream, image, virtual object and / or application associated with the communication session.

[0090] Routine 1000 begins at operation 1002, where computing device 101 receives sensor data defining a 3D representation of a real-world environment. The sensor data can be captured by a depth map sensor (e.g., a depth map camera). Additionally, the sensor data can be captured by an image sensor (e.g., a camera), where the depth map sensor and the image sensor can be part of the same component or in separate components. The sensor data includes depth map data defining a three-dimensional model of the real-world environment and an image of the real-world environment. For example, the real-world environment may include the walls of a room and certain objects within the room, such as in a room. Figure 1 . The sensor data can define the physical characteristics of an object in a real-world environment. The sensor data also indicates the location of one or more objects within the environment. For purposes of illustration, the real-world object is also referred to herein as "object 110."

[0091] The routine 1000 also includes an operation 1004, in which the computing device 101 receives model data defining one or more virtual objects to be displayed within the view of the collaborative environment. The model data can define a specific location where the virtual object will be placed within the user interface of the communication session. For the purpose of illustration, the virtual object positioned as a result of any received model data is referred to herein as a "content object 111". In some configurations, the operation 1004 includes receiving input from a user, resulting in the generation of model data defining one or more content objects 111. Examples of content objects 111 can include, but are not limited to, arrows and annotations, Q&A objects, voting poll objects, etc. The input can define the location of the content object 111 in the input and can also identify the associated object 110. For example, if the location of the new content object 111 is within a predetermined distance of the object 110, the new content object 111 can be associated with the object 110.

[0092] Routine 1000 also includes operation 1006, wherein computing device 101 generates a data structure defining an association between content object 111 and associated object 110. In some configurations, the data structure defines a position of content object 111 and a position for associated object 110. The data structure may also define a position offset between the position of content object 111 relative to the position of object 110. In an illustrative example, object 110 is classified as a parent object, and content object 111 is classified as a child object. The data structure may also include permissions for individual users or group users. The permissions allow individual users or group users to apply operations to the object. In addition, the permissions may prevent one or more users from applying one or more operations to the object.

[0093] The routine 1000 continues at operation 1008, where the computing device 101 displays the content object 111 and the associated object 110. The display of the content object 111 and the associated object 110 is arranged to illustrate a graphical association between the two objects. In some configurations, the graphical association can be identified by a proximity or distance between the two objects. In other embodiments, a graphical element such as a line or arrow can graphically connect the two objects to illustrate the graphical association.

[0094] Next, the routine 1000 continues at operation 1010, where the computing device 101 receives input to apply the selected operation to the object 110 or the associated content object 111. The input can be user input or the input can be a computer-generated instruction. The user input can be in the form of voice commands, gestures, and interactions with a display screen or any other type of input through an input device. The input can be applied to the object or content object. For example, Figure 3B The example of FIG. 1 shows user input applied to content object 111A, a Q&A content object. Figure 3B , an example of user input applied to an object would include user input directed to the rendering of object 110 ′ (eg, a table).

[0095] Next, at operation 1012, computing device 101 applies the selected operation to the selected object or the selected content object. For example, in the above example where the user input points to a content object (e.g., a Q&A content object), the computing device applies the selected operation to the content object. An example of the selected operation may include moving the selected content object within a display screen. In the current example, such an operation will move the rendering of the Q&A content object within the display screen. Another example of the selected operation may include sending the selected content object to the computing device of the remote communication session. In the current example, such an operation will cause the Q&A content object to be sent to the computing device of the remote communication session.

[0096] Next, at operation 1014, the computing device 101 applies the selected operation to the associated object or content object based on the data structure. In the current example, in response to determining that the input command is applied to the content object 111, the computing device applies the selected operation to the associated object, such as a table, based on the attributes of the data structure. Operation 1014 may include analyzing the data structure to determine all related objects, related content objects, and also determining one or more permissions for applying the selected operation. If the permissions do not indicate that the selected operation can access the associated object or the associated content object, the computing device may prevent the application of the selected operation.

[0097] Reference Figure 3B In the example shown in , in the case where there are multiple content objects 111 associated with a single object 110, operation 1014 will also include applying the selected operation to other content objects, such as the second content object 111B and the third content object 111C, based on the association defined in the data structure. Figure 3B In the example shown in , the second content object 111B and the third content object 111C are also moved based on the move command applied to the associated content object, the first content object 111A.

[0098] Operation 1014 can also involve applying the selected operation to the associated content object. For example, if the input is directed to object 110, operation 1014 will apply the selected operation to the associated content object according to the data structure.

[0099] It should be appreciated that the above subject matter can be implemented as a computer-controlled device, a computer process, a computing system, or an article of manufacture such as a computer-readable storage medium. An exemplary method is shown in individual blocks and summarized with reference to those blocks. The method is illustrated as a logical flow of blocks, each of which can represent one or more operations that can be implemented in hardware, software, or a combination thereof. In the context of software, the operations represent computer-executable instructions stored on one or more computer-readable media that, when executed by one or more processors, enable one or more processors to perform the operations.

[0100] Typically, computer executable instructions include routines, programs, objects, modules, components, data structures, etc. that perform specific functions or implement specific abstract data types. The order in which the operations are described is not intended to be interpreted as a limitation, and any number of the described operations can be performed in any order, combined in any order, subdivided into multiple sub-operations, and / or performed in parallel to implement the described processes. The described processes can be performed by resources associated with one or more devices (such as one or more internal or external CPUs or GPUs) and / or one or more hardware logic (such as a field programmable gate array ("FPGA"), a digital signal processor ("DSP"), or other types of accelerometers).

[0101] All of the above methods and processes can be embodied in a software code module executed by one or more general-purpose computers or processors, and fully automated via the software code module. The code module can be stored in any type of computer-readable storage medium or other computer storage device, such as those described below. Some or all of the methods can be alternatively embodied in special-purpose computer hardware, such as described below.

[0102] Any routine description, element or block in the flowcharts described herein and / or depicted in the accompanying drawings should be understood to potentially represent a module, segment or portion of code, which includes one or more executable instructions for implementing a specific logical function or element in a routine. Alternative implementations are included within the scope of the examples described herein, wherein elements or functions may be deleted or performed in a different order than shown or discussed, including substantially simultaneously or in reverse order, depending on the functions involved, as will be understood by those skilled in the art.

[0103] Fig.11 1100 is a diagram illustrating an exemplary environment 1100 in which a system 1102 can implement the techniques disclosed herein. In some implementations, the system 1102 can be used to collect, analyze, and share data defining one or more objects that are displayed to users of a communication session 1004.

[0104] As illustrated, a communication session 1104 may be implemented between a plurality of client computing devices 1106(1)-1106(N) (where N is a number of two or greater) associated with or as part of the system 1102. The client computing devices 1106(1)-1106(N) enable users (also referred to as individuals) to participate in the communication session 1104. For example, a first client computing device 1106(1) may be Figure 1 The computing device 101 or Fig.13 A computing device 1300 is provided.

[0105] In this example, communication session 1104 is hosted by system 1102 on one or more networks 1108. That is, system 1102 can provide services that enable users of client computing devices 1106(1) to 1106(N) to participate in communication session 1104 (e.g., via live viewing and / or recorded viewing). Thus, "participants" of communication session 1104 can include users and / or client computing devices (e.g., multiple users can participate in a communication session in a room via the use of a single client computing device), each participant being able to communicate with other participants. Alternatively, communication session 1104 can be hosted by one of client computing devices 1106(1) to 1106(N) utilizing peer-to-peer technology. System 1102 can also host chat conversations and other team collaboration functionality (e.g., as part of an application suite).

[0106] In some implementations, such chat conversations and other team collaboration functions are treated as external communication sessions distinct from the communication session 1104. A computerized agent that collects participant data in the communication session 1104 can be linked to such an external communication session. Thus, the computerized agent can receive information, such as date, time, session details, etc., enabling connection to such an external communication session. In one example, a chat conversation can be conducted in accordance with the communication session 1104. Additionally, the system 1102 can host a communication session 1104 that includes at least a plurality of participants co-located at a meeting location, such as a conference room or auditorium, or located at different locations.

[0107] In the examples described herein, client computing devices 1106(1) to 1106(N) participating in a communication session 1104 are configured to receive communication data and render it on a user interface of a display screen for display. The communication data can include a collection of various instances or streams of live content and / or recorded content. The collection of various instances or streams of live content and / or recorded content can be provided by one or more cameras (such as a video camera). For example, an individual stream of live or recorded content can include media data associated with a video feed provided by a video camera (e.g., audio and video data capturing the appearance and voice of a user participating in the communication session). In some implementations, the video feed can include such audio and visual data, one or more still images, and / or one or more avatars. The one or more still images can also include one or more avatars.

[0108] Another example of an individual stream of live or recorded content can include media data having an avatar of a user participating in a communication session and audio data capturing the user's voice. Yet another example of an individual stream of live or recorded content can include media data having a file displayed on a display screen and audio data capturing the user's voice. Thus, various streams of various live or recorded content within the communication data enable remote conferencing between a group of people and content sharing within the group of people to be facilitated. In some implementations, the various streams of live or recorded content within the communication data can originate from multiple co-located video cameras located in a space (such as a room) to record or live stream a presentation of one or more individuals presenting and consuming the presented content.

[0109] Participants or attendees can view the content of the communication session 1104 live when the activity occurs, or alternatively, view it via recording at a later time after the activity occurs. In the example described herein, the client computing devices 1106 (1) to 1106 (N) participating in the communication session 1104 are configured to receive and render communication data for display on a user interface of a display screen. The communication data can include a collection of various instances or streams of live and / or recorded content. For example, an individual content stream can include media data associated with a video feed (e.g., audio and visual data that captures the appearance and voice of a user participating in the communication session). Another example of an individual content stream can include media data with an avatar of a user participating in a conference session and audio data that captures the user's voice. Yet another example of an individual content stream can include media data with content items displayed on a display screen and / or audio data that captures the user's voice. Therefore, the various content streams within the communication data enable a conference or broadcast presentation to be facilitated between a group of people dispersed in a remote location.

[0110] A participant or attendee of a communication session is a person who is within range of a camera or other image and / or audio capture device, thereby being able to capture (e.g., record) the person's movements and / or sounds produced while the person is viewing and / or listening to content shared via the communication session. For example, a participant may be sitting in a crowd viewing content shared live at a broadcast location where a stage presentation is occurring. Alternatively, a participant may be sitting in an office conference room viewing shared content of a communication session with other colleagues via a display screen. Even further, a participant may be sitting or standing in front of a personal device (e.g., a tablet computer, a smartphone, a computer, etc.) viewing shared content of a communication session alone in his or her office or home.

[0111] System 1102 includes device(s) 1110. Device(s) 1110 and / or other components of system 1102 can include distributed computing resources that communicate with each other and / or with client computing devices 1106(1)-1106(N) via one or more networks 1108. In some examples, system 1102 may be a stand-alone system tasked with managing aspects of one or more communication sessions, such as communication session 1104. As an example, system 1102 may be managed by an entity such as Slack, WEBEX, GOTOMEETING, GOOGLE HANGOUTS, etc.

[0112] The network(s) 1108 may include, for example, a public network such as the Internet, a private network such as an institutional and / or personal intranet, or some combination of private and public networks. The network(s) 1108 may also include any type of wired and / or wireless network, including, but not limited to, a local area network ("LAN"), a wide area network ("WAN"), a satellite network, a cable network, a Wi-Fi network, a WiMax network, a mobile communication network (e.g., 3G, 4G, etc.), or any combination thereof. The network(s) 1108 may utilize communication protocols, including packet-based and / or datagram-based protocols, such as the Internet Protocol ("IP"), the Transmission Control Protocol ("TCP"), the User Datagram Protocol ("UDP"), or other types of protocols. In addition, the network(s) 1108 may also include a plurality of devices that facilitate network communications and / or form a hardware foundation for the network, such as switches, routers, gateways, access points, firewalls, base stations, repeaters, backbone devices, etc.

[0113] In some examples, network(s) 1108 may also include devices that enable connection to wireless networks, such as wireless access points (“WAPs”). Examples support connection via WAPs that send and receive data over various electromagnetic frequencies (e.g., radio frequencies), including WAPs that support the Institute of Electrical and Electronics Engineers (“IEEE”) 802.11 standards (e.g., 802.11g, 802.11n, 802.11ac, etc.) and other standards.

[0114] In various examples, (one or more) devices 1110 may include one or more computing devices operating in a cluster or other grouping configuration to share resources, balance loads, improve performance, provide failover support or redundancy, or for other purposes. For example, (one or more) devices 1110 may belong to various categories of devices, such as conventional server-type devices, desktop-type devices, and / or mobile devices. Therefore, although illustrated as a single type of device or server-type device, (one or more) devices 1110 may include multiple device types and are not limited to a specific type of device. (One or more) devices 1110 may represent, but are not limited to: a server computer, a desktop computer, a network server computer, a personal computer, a mobile computer, a laptop computer, a tablet computer, or any other type of computing device.

[0115] The client computing device (e.g., one of client computing devices 1106(1)-1106(N)) can belong to a variety of categories of devices, which can be the same as or different from device(s) 1110, such as conventional client-type devices, desktop computer-type devices, mobile-type devices, dedicated-type devices, embedded devices, and / or wearable-type devices. Thus, the client computing device can include, but is not limited to, a desktop computer, a game console and / or gaming device, a tablet computer, a personal data assistant ("PDA"), a mobile phone / tablet hybrid device, a laptop computer, a telecommunications device, a computer navigation-type client computing device (such as a satellite-based navigation system including a Global Positioning System ("GPS") device), a wearable device, a virtual reality ("VR") device, an augmented reality ("AR") device, an implanted computing device, an automotive computer, a network-enabled television, a thin client, a terminal, an Internet of Things ("IoT") device, a workstation, a media player, a personal video recorder ("PVR"), a set-top box, a camera, an integrated component (e.g., a peripheral device) for inclusion in a computing device, an appliance, or any other type of computing device. Furthermore, the client computing device may include a combination of the earlier listed examples of client computing devices, such as, for example, a desktop computer-type device or a mobile-type device combined with a wearable device, or the like.

[0116] Client computing devices 1106(1) to 1106(N) of various classes and device types can represent any type of computing device having one or more data processing units 1112 such as operably connected to a computer-readable medium 1184 via a bus 1116, which in some cases can include one or more of: a system bus, a data bus, an address bus, a PCI bus, a Mini-PCI bus, and any kind of local, peripheral and / or independent bus.

[0117] The executable instructions stored on the computer-readable medium 1194 can include, for example, an operating system 1119 , a client module 1120 , a profile module 1122 , and other modules, programs, or applications that can be loaded and executed by the data processing unit 1192 .

[0118] The client computing devices 1106(1)-1106(N) may also include one or more interfaces 1124 to enable communication between the client computing devices 1106(1)-1106(N) and other network devices such as device(s) 1110 over the network 1108. Such network interface(s) 1124 may include one or more network interface controllers (NICs) or other types of transceiver devices to send and receive communications and / or data over the network. In addition, the client computing devices 1106(1)-1106(N) may include input / output ("I / O") interfaces (devices) 1126 that enable communication with input / output devices, such as user input devices, including peripheral input devices (e.g., game controllers, keyboards, mice, pens, voice input devices (such as microphones), video cameras for obtaining and providing video feeds and / or still images, touch input devices, gesture input devices, etc.), and / or output devices, including peripheral output devices (e.g., displays, printers, audio speakers, tactile output devices, etc.). Fig.11 The client computing device 1106 ( 1 ) is illustrated as being connected in some manner to a display device (eg, display screen 1129 ( 1 )) that is capable of displaying a UI according to the techniques described herein.

[0119] exist Fig.11 In the exemplary environment 1100 of FIG. 1 , client computing devices 1106(1) to 1106(N) can use their respective client modules 1120 to connect to each other and / or to other external devices (one or more) to participate in a communication session 1104 or to contribute to a collaborative environment. For example, a first user can utilize client computing device 1106(1) to communicate with a second user of another client computing device 1106(2). When executing client modules 1120, the users can share data, which can cause client computing device 1106(1) to connect to system 1102 and / or other client computing devices 1106(2) to 1106(N) via network(s) 1108.

[0120] The client computing devices 1106(1) to 1106(N) may use their respective profile modules 1122 to generate participant profiles (in Fig.11), and provide the participant profile to other client computing devices and / or to the device(s) 1110 of the system 1102. The participant profile may include one or more identities of a user or group of users (e.g., name, unique identifier (“ID”), etc.), user data (such as personal data), machine data (such as location (e.g., IP address, room in a building, etc.), and technical capabilities, etc. The participant profile may be used to register a participant for a communication session.

[0121] As in Fig.11 , device(s) 1110 of system 1102 include server module 1130 and output module 1132. In this example, server module 1130 is configured to receive media streams 1134(1) to 1134(N) from individual client computing devices, such as client computing devices 1106(1) to 1106(N). As described above, the media streams can include a video feed (e.g., audio and visual data associated with a user), audio data to be output with a presentation of an avatar of the user (e.g., an audio-only experience in which the user's video data is not sent), text data (e.g., a text message), file data, and / or screen sharing data (e.g., a document, a slideshow, an image, a video displayed on a display screen, etc.), etc. Thus, server module 1130 is configured to receive a collection of various media streams 1134(1) to 1134(N) (the collection is referred to herein as "media data 1134") during a live view of communication session 1104. In some scenarios, not all client computing devices participating in the communication session 1104 provide media streams. For example, a client computing device may only be a consuming or "listening" device, such that it only receives content associated with the communication session 1104 but does not provide any content to the communication session 1104.

[0122] In various examples, server module 1130 can select aspects of media stream 1134 to be shared with individual client computing devices of participating client computing devices 1106(1)-1106(N). Thus, server module 1130 can be configured to generate session data 1136 based on stream 1134 and / or pass session data 1136 to output module 1132. Output module 1132 can then transmit communication data 1138 to client computing devices (e.g., client computing devices 1106(1)-1106(3) participating in the live viewing communication session). Communication data 1138 can include video, audio, and / or other content data provided by output module 1132 based on content 1150 associated with output module 1132 and based on received session data 1136.

[0123] As shown, output module 1132 sends communication data 1139(1) to client computing device 1106(1), and sends communication data 1139(2) to client computing device 1106(2), and sends communication data 1139(3) to client computing device 1106(3), etc. The communication data 1139 sent to the client computing devices can be the same or can be different (e.g., the positioning of the content stream within the user interface can be different from one device to the next).

[0124] In various implementations, the device(s) 1110 and / or the client module 1120 can include a UI presentation module 1140. The UI presentation module 1140 can be configured to analyze the communication data 1139 for delivery to the one or more client computing devices 1106. Specifically, the UI presentation module 1140 at the device(s) 1110 and / or the client computing device 1106 can analyze the communication data 1139 to determine an appropriate manner for displaying the video, image, and / or content on the display screen 1129 of the associated client computing device 1106. In some implementations, the UI presentation module 1140 can provide the video, image, and / or content to a presentation UI 1146 rendered on the display screen 1129 of the associated client computing device 1106. The UI presentation module 1140 can cause the presentation UI to be rendered on the display screen 1129. The presentation UI 1146 can include the video, image, and / or content analyzed by the UI presentation module 1140.

[0125] In some implementations, the presentation UI 1146 may include multiple sections or grids that may render or include video, images, and / or content for display on the display screen 1129. For example, a first section 1146 of the presentation UI 1146 may include a video feed of a presenter or individual, and a second section of the presentation UI 1146 may include a video feed of an individual consuming the conference information provided by the presenter or individual. The UI presentation module 1140 may populate the first and second sections of the presentation UI 1146 in a manner that appropriately mimics an ambient experience that the presenter and individual may be sharing.

[0126] In some implementations, the UI presentation module 1140 can amplify or provide a zoomed view of the individual represented by the video feed to highlight the individual's reaction to the presenter, such as facial features. In some implementations, the presentation UI 1146 can include video feeds of multiple participants associated with a meeting (such as a general communication session). In other implementations, the presentation UI 1146 can be associated with a channel (such as a chat channel, an enterprise team channel, etc.). Therefore, the presentation UI 1146 can be associated with an external communication session that is different from a general communication session.

[0127] Fig.12 A diagram showing exemplary components of an exemplary device 1200 (also referred to herein as a "computing device") is illustrated, which is configured to generate data for some user interfaces disclosed herein. The device 1200 can generate data that can include one or more parts, which can render or include video, images, virtual objects 116, and / or content for display on a display screen 1129. The device 1200 can represent one of the (one or more) devices described herein. Additionally or alternatively, the device 1200 can represent one of the client computing devices 1106.

[0128] As illustrated, device 1200 includes one or more data processing units 1202, computer-readable media 1204, and communication interface(s) 1206. The components of device 1200 are operably connected, for example, via bus 1208, which may include one or more of a system bus, a data bus, an address bus, a PCI bus, a Mini-PCI bus, and any kind of local, peripheral, and / or independent bus.

[0129] As used herein, data processing unit(s) (such as data processing unit 1202 and / or data processing unit(s) 1192) may represent, for example, a CPU-type data processing unit, a GPU-type data processing unit, a field programmable gate array ("FPGA"), another type of DSP, or other hardware logic components that may be driven by a CPU in some cases. For example, but not limited to, illustrative types of hardware logic components that may be utilized include application specific integrated circuits ("ASICs"), application specific standard products ("ASSPs"), systems on chips ("SOCs"), complex programmable logic devices ("CPLDs"), and the like.

[0130] As used herein, a computer-readable medium (such as computer-readable medium 1204 and computer-readable medium 1194) can store instructions that can be executed by (one or more) data processing units. The computer-readable medium can also store instructions that can be executed by an external data processing unit (such as an external CPU, an external GPU) and / or can be executed by an external accelerometer (such as an FPGA-type accelerometer, a DSP-type accelerometer, or any other internal or external accelerometer). In various examples, at least one CPU, GPU, and / or accelerometer is incorporated into the computing device, while in some examples, one or more of the CPU, GPU, and / or accelerometer is external to the computing device.

[0131] Computer readable media (also referred to herein as computer readable media) may include computer storage media and / or communication media. Computer storage media may include one or more of volatile memory, non-volatile memory and / or other persistent and / or secondary computer storage media, removable and non-removable computer storage media implemented in any method or technology for storing information such as computer readable instructions, data structures, program modules or other data. Thus, computer storage media includes media in tangible and / or physical form contained in a device and / or as a hardware component either as part of the device or external to the device, including, but not limited to: random access memory (“RAM”), static random access memory (“SRAM”), dynamic random access memory (“DRAM”), phase change memory (“PCM”), read only memory (“ROM”), erasable programmable read-only memory (“EPROM”), electrically erasable programmable read-only memory (“EEPROM”), flash memory, compact disk read-only memory (“CD-ROM”), digital versatile disk (“DVD”), optical cards or other optical storage media, cassettes, magnetic tape, magnetic disk storage, magnetic cards or other magnetic storage devices or media, solid-state memory devices, storage arrays, network attached storage, storage area network, hosted computer storage, or any other storage memory, storage device, and / or storage medium that can be used to store and maintain information for access by a computing device.

[0132] In contrast to computer storage media, communication media may embody computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism. As defined herein, computer storage media does not include communication media. That is, computer storage media does not include communication media consisting only of a modulated data signal, a carrier wave, or a propagation signal itself.

[0133] The communication interface(s) 1206 may represent, for example, a network interface controller ("NIC") or other type of transceiver device to send and receive communications over a network. Additionally, the communication interface(s) 1206 may include one or more video cameras and / or audio devices 1222 to enable the generation of video feeds and / or still images, etc.

[0134] In the illustrated example, computer-readable medium 1204 includes a data repository 1208. In some examples, data repository 1208 includes a data store such as a database, a data warehouse, or other type of structured or unstructured data store. In some examples, data repository 1208 includes a corpus and / or a relational database having one or more tables, indexes, storage processes, etc. to enable data access, including, for example, one or more Hypertext Markup Language ("HTML") tables, Resource Description Framework ("RDF") tables, Web Ontology Language ("OWL") tables, and / or Extensible Markup Language ("XML") tables.

[0135] The data repository 1208 can store data for operations of processes, applications, components, and / or modules stored in the computer-readable medium 1204 and / or executed by the (one or more) data processing units 1202 and / or (one or more) accelerometers. For example, in some examples, the data repository 1208 can store session data 1210 (e.g., session data 1136), profile data 1212 (e.g., associated with participant profiles), and / or other data. The session data 1210 can include the total number of participants (e.g., users and / or client computing devices) in the communication session, the activities occurring in the communication session, the list of invitees for the communication session, and / or other data related to when and how the communication session is conducted or hosted. The data repository 1208 can also include content data 1214, such as video, audio, or other content for rendering and display on one or more display screens 1129.

[0136] Alternatively, some or all of the above data can be stored on a separate memory 1216 on one or more data processing units 1202, such as a CPU-type processor, a GPU-type processor, an FPGA-type accelerometer, a DSP-type accelerometer, and / or a memory on another accelerometer. In this example, the computer-readable medium 1204 also includes an operating system 1218 and (one or more) application programming interfaces 1211 (APIs), which are configured to expose the functions and data of the device 1200 to other devices. In addition, the computer-readable medium 1204 includes one or more modules, such as a server module 1230, an output module 1232, and a GUI presentation module 1240, but the number of modules illustrated is only an example, and the number can be higher or lower. That is, the functions associated with the illustrated modules described herein can be performed by a smaller number of modules or a larger number of modules distributed across multiple devices.

[0137] Fig.131 is a diagram of a computing device that illustrates various aspects of the configuration and operation of a computing device 1300 capable of implementing various aspects of the system disclosed herein. Figure 1 . The computing device 1300 is capable of providing an augmented reality ("AR") environment or a virtual reality ("VR") environment. Generally speaking, an AR environment superimposes a computer-generated ("CG") image on a user's view of a real-world environment. For example, the computing device 1300 is capable of generating a synthetic view so that a user can visually perceive a computer-generated image superimposed on a rendering of a real-world environment 112, wherein the rendering of the real-world environment 112 is created by a camera 105 pointing to the real-world environment (such as a room). In some embodiments, the computing device 1300 is capable of generating a synthetic view so that a user can visually perceive a computer-generated image superimposed on a direct view of the real-world environment 112. Therefore, the computing device 1300 may have a prism or other optical device that allows a user to see a direct view of a real-world object or a real-world environment through the optical device, and at the same time, the computer-generated image is superimposed on the view of the real-world object. The AR environment can also be referred to as a mixed reality ("MR") environment in this article. MR devices can provide AR and VR environments. The VR environment includes a virtual environment and a computer-generated image of a virtual object. MR and AR environments can utilize depth map sensors to determine the distance between the device and real-world objects. This allows the computer to scale and position computer-generated graphics on real-world objects in a realistic manner.

[0138] exist Fig.13 In the example shown, the optical system 1302 includes an illumination engine 1304 to generate electromagnetic ("EM") radiation that includes both a first bandwidth for generating CG images and a second bandwidth for tracking physical objects. The first bandwidth may include some or all of the visible light portion of the EM spectrum, while the second bandwidth may include any portion of the EM spectrum suitable for deploying a desired tracking protocol. In this example, the optical system 1302 also includes an optical assembly 1306 that is positioned to receive the EM radiation from the illumination engine 1304 and direct the EM radiation (or individual bandwidths thereof) along one or more predetermined optical paths.

[0139] For example, illumination engine 1304 may emit EM radiation along a common optical path shared by both the first bandwidth and the second bandwidth into optical assembly 1306. Optical assembly 1306 may also include one or more optical components configured to separate the first bandwidth from the second bandwidth (e.g., by causing the first and second bandwidths to propagate along different image generation and object tracking optical paths, respectively).

[0140] In some cases, the user experience depends on the computing device 1300 accurately identifying the characteristics of physical objects 103 ("real-world objects") or planes (such as real-world floors), and then generating CG images based on these identified characteristics. For example, assume that the computing device 1300 is programmed to generate a user perception that a virtual game character is running towards and ultimately jumping over a real-world structure. To achieve this user perception, the computing device 1300 can obtain detailed data defining the characteristics of the real-world environment 112 surrounding the computing device 1300. To provide this functionality, the optical system 1302 of the computing device 1300 may, in some embodiments, include a laser line projector and a differential imaging camera (neither of which is in the Fig.13 ).

[0141] In some examples, computing device 1300 utilizes optical system 1302 to generate a synthetic view (e.g., from the perspective of a user wearing computing device 1300) that includes both one or more CG images and a view of at least a portion of real-world environment 112. For example, optical system 1302 may utilize various techniques (such as, for example, AR techniques) to generate a synthetic view that includes a CG image superimposed on a real-world view. As such, optical system 1302 may be configured to generate a CG image via optical assembly 1306 that includes display panel 1314.

[0142] In the illustrated example, the display panel includes separate right-eye and left-eye transparent display panels, labeled 1314R and 1314L, respectively. In some examples, the display panel 1314 includes a single transparent display panel that can be viewed with both eyes or a single transparent display panel that can only be viewed with one eye. Therefore, it can be appreciated that the technology described herein can be deployed in a monocular device (e.g., a GOOGLE GLASS AR device) and a binocular device (e.g., a MICROSOFT HOLOLENS AR device).

[0143] Light received from the real-world environment 112 passes through the see-through display panel 1314 to one or both eyes of the user. Graphical content calculated by the image generation engine 1326 executed on the processing unit 1320 and displayed by the right-eye and left-eye display panels (if configured as see-through display panels) can be used to visually enhance or otherwise modify the real-world environment 112 viewed by the user through the see-through display panel 1314. In this configuration, the user is able to view virtual objects 104 that are not present in the real-world environment 112 while the user is viewing physical objects 103 within the real-world environment 112. This creates an illusion or appearance based on light effects that the virtual objects 104 are physical objects 103 or physically present within the real-world environment 112.

[0144] In some examples, display panel 1314 is a waveguide display that includes one or more diffractive optical elements ("DOEs") for coupling incident light input into the waveguide, expanding the incident light in one or more directions to exit the pupil dilation, and / or coupling incident light output out of the waveguide (e.g., toward the user's eyes). In some examples, computing device 1300 also includes additional see-through optical components, such as in Fig.13 , in the form of a transparent mask 1316 located between the real world environment 112 and the display panel 1314. It can be appreciated that the transparent mask 1316 can be included in the computing device 1300 for purely aesthetic and / or protective purposes.

[0145] The computing device 1300 may also include various other components (in Fig.13 1300 ), such as a front-facing camera (e.g., a red / green / blue (“RGB”), black and white (“B&W”), or infrared (“IR”) camera), a speaker, a microphone, an accelerometer, a gyroscope, a magnetometer, a temperature sensor, a touch sensor, a biometric sensor, other image sensors, an energy storage component (e.g., a battery), a communication facility, a global positioning system (“GPS”) receiver, a laser line projector, a differential imaging camera, and other types of sensors. Data obtained from one or more sensors 1308 (some of which are identified above) can be used to determine the orientation, position, and movement of the computing device 1300. As discussed above, data obtained from the differential imaging camera and laser line projector or other types of sensors can also be used to generate a 3D depth map of the surrounding real-world environment 112.

[0146] In the illustrated example, the computing device 1300 includes one or more logic devices and one or more computer memory devices storing instructions executable by the (one or more) logic devices to implement the functionality disclosed herein. In particular, the controller 1318 can include one or more processing units 1320, one or more computer readable media 1322 for storing an operating system 1324, and an image generation engine 1326 and a terrain mapping engine 1328, as well as other programs (e.g., a 3D depth map generation module configured to generate depth map data ("mesh data") and data in the manner disclosed herein.

[0147] In some implementations, computing device 1300 is configured to analyze data obtained by sensor 1308 to perform feature-based tracking of the orientation of computing device 1300. For example, in a scene where object data includes an indication of a stationary physical object 103 within a real-world environment 112 (e.g., an engine), computing device 1300 may monitor the position of the stationary object within a terrain mapping field of view (“FOV”). Then, based on changes in the position of the stationary object within the terrain mapping field of view FOV and the depth of the stationary object from computing device 1300, a terrain mapping engine executing on processing unit 1320AR may calculate changes in the orientation of computing device 1300.

[0148] It can be appreciated that these feature-based tracking techniques can be used to monitor changes in the orientation of computing device 1300 for monitoring the orientation of the user's head (e.g., assuming computing device 1300 is properly worn by user 102). The calculated orientation of computing device 1300 can be utilized in a variety of ways, some of which have been described above.

[0149] The processing unit(s) 1320 can represent, for example, a central processing unit ("CPU") type processor, a graphics processing unit ("GPU") type processing unit, an FPGA, one or more digital signal processors ("DSP"), or other hardware logic components that may be driven by a CPU in some cases. For example, but not limited to, illustrative types of hardware logic components that can be used include ASICs, application specific standard products ("ASSPs"), systems on chips ("SOCs"), complex programmable logic devices ("CPLDs"), etc. The controller 1318 may also include one or more computer-readable media 1322, such as the computer-readable media described above.

[0150] It should be appreciated that, unless otherwise specifically stated, conditional language used herein, such as "may," "can," "might," or "could," is understood in context to present certain features, elements, and / or steps as being included in certain examples but not included in other examples. Thus, such conditional language is generally not intended to imply that certain features, elements, and / or steps are in any way necessary for one or more examples, or that one or more examples necessarily include logic for determining, with or without user input or prompting, whether certain features, elements, and / or steps are included or will be performed in any particular example. Unless otherwise specifically stated, conjunctions such as phrases "at least one of X, Y, or Z" should be understood to indicate that an item, term, or the like can be X, Y, or Z, or a combination thereof.

[0151] It should also be appreciated that many variations and modifications may be made to the above examples, and that elements thereof should be understood as being in other acceptable examples. All such modifications and variations are intended to be included within the scope of this disclosure and protected by the following claims.

[0152] Finally, although various configurations have been described in language specific to structural features and / or methodological acts, it should be understood that the subject matter defined in the appended representations is not necessarily limited to the specific features or acts described. Rather, the specific features and acts are disclosed as exemplary forms of implementing the claimed subject matter.

Claims

1. A system comprising: one or more data processing units; as well as A computer readable medium having encoded thereon computer executable instructions for causing the one or more data processing units to: receiving input data indicating a location for a content object to be displayed in association with an object rendered on one or more computing devices of a communication session; generating a data structure defining an association between the content object and the object, a location of the content object and a location of the object, characteristics of the content object and characteristics of the object, and one or more permissions for applying an operation to the content object or the object; detecting an input indicating an operation for graphically moving the content object or the object; in response to detecting that the operation is to be applied to the content object, applying the operation to the content object and copying the operation to the object based on the one or more permissions defined in the data structure that allow the object to be graphically moved with the content object, wherein the operation applied to the content object is also applied to at least one associated object based on the one or more permissions defined in the data structure that allow the object to be graphically moved with the content object; and In response to detecting that the operation is to be applied to the object, the operation is applied to the object and the operation is copied to the content object based on the one or more permissions defined in the data structure that allow the content object to be graphically moved with the object, wherein the operation applied to the object is also applied to at least one associated content object based on the one or more permissions defined in the data structure that allow the content object to be graphically moved with the object.

2. The system according to claim 1, wherein: The input indicates that the operation includes movement of the object to a new location within the three-dimensional environment, and wherein applying the operation to the object and copying the operation to the content object comprises: moving a rendering of the object to the new location in response to the input; and The rendering of the content object is repositioned to track the movement of the object, wherein the repositioning of the content object maintains a graphical association between the content object and the object defined in the data structure.

3. The system according to claim 2, wherein: The one or more permissions are configured to allow or deny individual users to move the content object, wherein repositioning the rendering of the content object is denied or allowed based on the permissions.

4. The system according to claim 1, wherein: The input indicates that the operation includes movement of the content object to a new location within the three-dimensional environment, wherein applying the operation to the content object and copying the operation to the object includes: moving a rendering of the content object to the new location in response to the input; The rendering of the object is repositioned to track the movement of the content object, wherein the repositioning of the object maintains a graphical association between the content object and the object defined in the data structure.

5. The system according to claim 4, wherein: in, The one or more permissions are configured to allow or deny individual users to move the object, wherein repositioning the rendering of the object is denied or allowed based on the permissions.

6. The system according to claim 1, wherein: The operations also include instructions for sending the object to a remote computing device of a second communication session, wherein applying the operation to the object and copying the operation to the content object comprises: transmitting model data defining the object to the remote computing device; and Model data defining the content object is transmitted to the remote computing device in response to verifying the association defined in the data structure, wherein the data structure provides access to the object and the content object to a computing device of the second communication session.

7. The system according to claim 6, wherein: The data structure defines a plurality of content objects and permissions for individual content objects of the plurality of objects, wherein the communicating of the model data defining the content objects is based on the permissions associated with the content objects.

8. The system according to claim 1, wherein: The operations also include instructions for sending the content object to a remote computing device of a second communication session, wherein applying the operations to the content object and copying the operations to the object comprises: transmitting model data defining the content object to the remote computing device; and Model data defining the object is transmitted to the remote computing device in response to verifying the association defined in the data structure, wherein the data structure provides access to the object and the content object to a computing device of the second communication session.

9. The system according to claim 8, wherein: The data structure defines permissions for the object, wherein the transmitting of the model data defining the object is responsive to determining that the permissions indicate that a user associated with the input has permissions to access the model data defining the object.

10. The system according to claim 1, wherein: Detecting the input indicative of the operation comprises: analyzing flows between the plurality of computers of the second communication session; Determining that at least one stream includes at least one reference to the object or the content object, wherein the operations include instructions for sending the content object and the object to a remote computing device in response to determining that the at least one stream includes the at least one reference to the object or the content object.

11. A method comprising: receiving input data indicating a location for a content object to be displayed in association with an object rendered on one or more computing devices of a communication session; generating a data structure defining an association between the content object and the object, a location of the content object and a location of the object, characteristics of the content object and characteristics of the object, and one or more permissions for applying an operation to the content object or the object; receiving input invoking an operation that causes the content object or the object to be transmitted to a remote computing device participating in a second communication session; in response to detecting that the operation is to be applied to the content object, applying the operation to the content object and copying the operation to the object based on one or more permissions defined in the data structure for applying the operation to the content object or the object and allowing the object to be transmitted to the remote computing device participating in the second communication session, wherein the operation applied to the content object is also applied to at least one associated object based on one or more permissions defined in the data structure allowing the object to be graphically moved with the content object; and In response to detecting that the operation is to be applied to the object, the operation is applied to the object and the operation is copied to the content object based on one or more permissions defined in the data structure for applying the operation to the content object or the object and allowing the content object to be transmitted to the remote computing device participating in the second communication session, wherein the operation applied to the object is also applied to at least one associated content object based on one or more permissions defined in the data structure that allow the content object to be graphically moved with the object.

12. The method according to claim 11, wherein: The input indicates that the operation also includes movement of the object to a new location within the three-dimensional environment, and wherein applying the operation to the object and copying the operation to the content object comprises: moving a rendering of the object to the new location in response to the input; and The rendering of the content object is repositioned to track the movement of the object, wherein the repositioning of the content object maintains a graphical association between the content object and the object defined in the data structure.

13. The method according to claim 11, wherein: The data structure defines one or more permissions configured to allow or deny an individual user to move the content object, wherein repositioning the rendering of the content object is allowed or denied based on the one or more permissions.

14. The method according to claim 11, wherein: The data structure defines permissions for the object, wherein the transmitting comprises: in response to determining that the permissions indicate that a user associated with the input has permission to access model data defining the object, causing the model data defining the object to be transmitted to the remote computing device.

Images