Intelligent agent for managing data associated with 3D objects

Abstract

The technology disclosed in this article improves the efficiency of the system by providing an intelligent agent for managing data associated with objects displayed in a mixed reality and virtual reality collaborative environment. Individual agents are configured to collect, analyze and store data associated with individual objects in a shared view. The agent can identify real-world objects and virtual objects discussed in a meeting, collect information about each object, and generate recommendations for each object based on the collected information. The recommendations can suggest modifications to the object, provide resources for obtaining or modifying the object, and provide actionable information to allow users to reach a consensus on the object. Data can be shared between different communication sessions without the user manually storing and presenting a collection of content for each object. The intelligent agent can also be retained through different communication sessions to enhance user engagement and improve productivity.

Description

Background Art

[0001] Many productivity applications provide dedicated tools for displaying and manipulating the content of a file. Some productivity applications also provide a shared workspace in which multiple users can view and edit the content of a file simultaneously from separate locations. Some systems also allow multiple users to collaboratively edit content using devices that provide virtual reality ("VR") and mixed reality ("MR") environments.

[0002] Although current technologies can provide dedicated functions for sharing and manipulating content, some existing applications cannot provide a satisfactory user experience when the workflow requires a group of users to reach a consensus on shared content. For example, in a multi-user collaborative workspace, when group consensus is required, users may have to manually prepare dedicated content to allow other users to visualize each idea. A visual preview of each proposed idea can help participants gain understanding or awareness of each idea. However, the manual process of preparing each preview can be time-consuming and inefficient when computing resources (e.g., memory resources, processing resources, network resources, etc.) are involved.

[0003] Additionally, when users collaborate using one forum (such as a group editing session using a Skype session) and then switch to another forum (such as a private chat session), not all edited content can be transferred between the different types of sessions. This shortcoming can lead to other inefficiencies in computing resources because users may be required to retrieve, transfer, or even recreate content each time they switch between different types of communication sessions. Summary of the invention

[0004] The technology disclosed in this article improves the efficiency of the system by providing an intelligent agent for managing data associated with virtual objects and real-world objects displayed in a collaborative environment. Individual agents are configured to collect, analyze and store data associated with individual objects in a shared view. The agent can: identify real-world objects and virtual objects discussed in a meeting, collect information about each object, and generate recommendations for each object based on the collected information. The recommendation can suggest modifications to the object, provide resources for obtaining or modifying the object, and provide actionable information to allow users to reach a consensus on the object. Data can be shared between different communication sessions without the user manually storing and presenting a collection of content for each object. The intelligent agent can also continue through different communication sessions to enhance user engagement and improve productivity.

[0005] For example, if a first group of users is conducting a Skype conference about car engine design, an intelligent agent can be instantiated for each part of the engine. The intelligent agent can monitor all types of user activities during the multi-user communication session, such as polling, editing, text chat, and voice conversations. Data derived from the monitored activities can be displayed and stored in association with each object (e.g., each engine component). The agent is configured so that when the Skype conference ends, the agent and the stored data are retained after the communication session. Therefore, when a new communication channel is created (e.g., a private chat session or a new meeting), the agent allows the user to access the agent and the stored data.

[0006] Each agent can generate and display recommendations for individual objects. The recommendations can include recommendations for modifying an object. The recommendations can be in the form of: (1) a modification to a virtual object; (2) a new virtual object positioned over a real-world object; or (3) an annotation recommending a modification to a real-world object. Data defining the recommendations associated with each object can also be stored persistently across communication sessions. Thus, when an object (such as an engine component) is moved or deleted during a session, the recommendations will be modified and retained across other communication sessions to indicate such changes.

[0007] Intelligent agents provide many features that improve existing computers. For example, as users switch between different sessions, computing resources such as processor cycles, memory, network bandwidth, and power are used more efficiently. Users can share and display content between sessions without recreating data or content. In addition, the recommendations enable users to make more effective decisions by providing automatically generated recommendation visualizations or automatically generated resource recommendations for individual objects. The technology disclosed in this article also improves user interaction with various types of computing devices. Improvements in user interaction or reductions in the need for user input can mitigate unintentional input, redundant input, and other types of user interactions that utilize computing resources. Other technical benefits not specifically mentioned in this article can also be achieved by implementing the disclosed subject matter.

[0008] Those skilled in the art will also recognize that aspects of the subject matter described herein can be practiced on or with other computer system configurations than those specifically described herein, including multi-processor systems, microprocessor-based or programmable consumer electronics, AR, VR and MR devices, video gaming devices, handheld computers, smartphones, smart televisions, self-driving cars, smart watches, e-readers, tablet computing devices, dedicated hardware devices, networked devices, and other devices.

[0009] Features and technical benefits other than those explicitly described above will be apparent from reading the detailed description below and viewing the associated drawings. This disclosure is provided to introduce some concepts in a simplified form, which will be further described in the detailed description below. This disclosure is neither intended to identify the key features or essential features of the claimed subject matter, nor is it intended to be used to assist in determining the scope of the claimed subject matter. For example, the term "technology" may refer to the context described above and to (one or more) systems, (one or more) methods, computer-readable instructions, (one or more) modules, algorithms, hardware logic and / or operations allowed throughout the document. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The detailed description is described with reference to the accompanying drawings. In the drawings, the leftmost digit(s) of a reference number identifies the drawing in which the reference number first appears. The same reference numbers in different drawings indicate similar or identical items. References to individual items of a plurality of items can use reference numbers with letters in the alphabetical sequence to refer to each individual item. Generic references to items can use specific reference numbers without the alphabetical sequence.

[0011] Figure 1 Illustrated is a display of a collaborative environment generated by a computing device that captures images of real-world objects.

[0012] Figure 2 is a block diagram illustrating several intelligent agents generated in association with virtual objects and real-world objects.

[0013] Figure 3 Illustrated are various forms of communication that may be performed by users interacting in a collaborative environment.

[0014] Figure 4 is a block diagram illustrating several intelligent agents for collecting communication data for a collaborative environment by using an application programming interface.

[0015] Figure 5 is a block diagram illustrating multiple external resources that may be used by an intelligent agent to collect supplemental data associated with virtual objects and real-world objects.

[0016] Figure 6 A display of a collaborative environment showing a plurality of recommendations associated with virtual objects and real-world objects generated by an intelligent agent is illustrated.

[0017] Figure 7 A block diagram of a system with a management agent for managing proxy data between communication sessions is illustrated.

[0018] Figure 8Illustrated is a display of a subsequent collaborative environment involving a private chat utilizing an intelligent agent created in a first collaborative environment.

[0019] Fig. 9 Illustrated is a display of a subsequent collaboration environment involving a team session utilizing an intelligent agent created in a first collaboration environment.

[0020] Fig.10 is a flow chart illustrating aspects of a routine for computing an efficient intelligent agent for managing data associated with objects displayed within a mixed reality and virtual reality collaborative environment.

[0021] Fig.11 is a computing system diagram showing aspects of an illustrative operating environment for the techniques disclosed herein.

[0022] Fig.12 is a computing architecture diagram illustrating various aspects of the configuration and operation of a computing device that may be capable of various aspects of the techniques disclosed herein.

[0023] Fig.13 is a computing device diagram illustrating aspects of the configuration and operation of an MR device capable of implementing aspects of the disclosed technology, according to one embodiment disclosed herein. DETAILED DESCRIPTION

[0024] Figure 1 An exemplary user interface 100 is illustrated that displays aspects of a collaborative environment that may be used to facilitate aspects of the present disclosure. In this example, an image sensor 105 (e.g., a camera) of a computing device 101 is used to capture an image of a real-world object 103 within a real-world environment 112. In this illustrative example, the real-world object 103 is an engine. The computing device 101 is able to share the image of the real-world object 103 with multiple individual users 102A-102D ("users 102") communicating within the collaborative environment. The computing device 101 is also able to coordinate with other computing devices to generate a shared display of a virtual object 104, which is a visual representation of the real-world object 103. Figure 1 Rendered.

[0025] The user 102 can create and edit virtual objects 104 by using voice commands, moving gestures or other types of interactions with various input devices. In this illustrative example, the first virtual object 104A (fan) and the second virtual object 104B (fixture) are located at a specific position relative to the real world object 103. The user 102 can move and resize the virtual object 104. In addition, the user 102 can use one or more gestures (including voice commands) to associate the virtual object 104 with one or more real world objects. In this illustrative example, the first virtual object 104A and the second virtual object 104B are associated with the real world object 103 (engine). The association between each object can be based on the relative position between each object, a graphical element showing the association, a text description, or any other suitable graphical representation. In this example, the first virtual object 104A and the second virtual object 104B are aligned with the axis of the engine to illustrate the association between the objects, for example, how to assemble the parts.

[0026] Users 102 can also communicate with each other in the collaborative environment using a variety of different media, including but not limited to shared audio, video and text communications. In this example, a first user 102A, a second user 102B, a third user 102C and a fourth user 102D all communicate with each other, and the video stream of each user 102 is presented in the user interface 100.

[0027] The computing device 101 can identify both real-world objects and virtual objects within the collaborative environment. For the purpose of illustration, a general reference to "one or more objects" refers to both virtual objects and real-world objects. The technology disclosed herein can use any number of object recognition techniques to identify objects, including but not limited to edge detection, pattern detection, or shape detection algorithms. In this example, the computing device 101 identifies the real-world object 103 by its shape, size, and other physical characteristics. It is also possible to identify one or more real-world objects by using the sound emitted by the object detected by the microphone. Color, shape, pattern, surface texture or other characteristics can be used to identify models, manufacturers, parts, etc. It is also possible to identify virtual objects by performing analysis on the model data that defines each virtual object. The shape, size or other characteristics can be interpreted directly based on the analysis of the model data.

[0028] In response to identifying the real-world objects and virtual objects displayed within the collaborative environment, the computing device 101 generates individual agents associated with each individual object. Figure 2) can be an autonomous computing unit that is configured to collect and manage data associated with each object. Each agent is, for example, permanently stored in a data repository that maintains all data associated with each agent 201 even after the communication session associated with the collaborative environment is terminated. In some embodiments, each agent can be in the form of a virtual machine that can be stored on a single computing device and can communicate between different computing devices.

[0029] In some embodiments, computing device 101 is capable of generating proxies for particular objects of interest. Objects of interest can be selected based on physical properties of the object or user activity. For example, a particular object can be considered an object of interest if it contains moving parts, exceeds a threshold size, has a predetermined shape, etc. In another example, a particular object can be considered an object of interest based on analysis of user discussion or preference data. User 102 who makes a comment about a particular object (such as an engine) can cause computing device 101 to select the object based on keywords or gestures associated with the object. In one specific example, if a user communication includes a threshold number of keywords or gestures associated with an engine, the object can be selected in the context of the user communication. Figure 1 The engine shown considers objects of interest. Other objects in the direct view of the camera 105 of the computing device 101 (such as the walls of a room, a light switch near the user) may not be considered objects of interest if the user has little interaction with those objects.

[0030] In some embodiments, computing device 101 can generate a score to determine whether a particular object is an object of interest. When the associated score exceeds a threshold, the object can be selected as an object of interest. The score can be generated based on multiple factors. For example, the score can be generated based on the number of times or frequency that a user interacts with a particular object. The score can also be based on the movement or other characteristics (such as size, shape, etc.) of the object. The score can be used to rank the object. The ranking can be used to select a group of objects with a predetermined size. For example, a collaborative environment can have data defining the maximum number of objects for a particular communication session or a particular computing device. In such an embodiment, the highest ranked object, up to the maximum number of objects, can be selected as an object of interest.

[0031] Figure 2 An exemplary implementation is illustrated that includes multiple agents 201 generated in association with the identified objects. In this example, a first agent 201A is generated in association with the real-world object 103, a second agent 201B is generated in association with the first virtual object 104A, and a third agent 201C is generated in association with the second virtual object 104B.

[0032] Each agent 201 manages a database 202 for storing data records that define keywords, descriptions, parameters, or other data about individual objects. In some embodiments, each database 202 can be configured to operate autonomously, for example, each database 202 is stored in a data structure that can be independently communicated from one computing device to another computing device without affecting other databases 202. In some embodiments, each database 202 is stored in a data structure that also defines the associated agent 201, referred to herein as "agent data." As shown in Figure 2 As shown in , this example includes a first database 202A in communication with a first agent 201A, a second database 202B in communication with a second agent 201B, and a third database 202C in communication with a third agent 201C.

[0033] Also as in Figure 2 As shown in , computing device 101 can generate and modify data records associated with each object. Data records can be generated based on multiple factors. In some embodiments, data records describing various aspects of each object can be generated based on the shape, size, or other physical characteristics of a real-world object or a virtual object. For example, in this example, a data record 203A describing the diameter of an engine drive shaft can be generated based on an analysis of an image of the engine, wherein the image can be used to measure parameters of a specific component such as a drive shaft. Such data can be generated by analyzing depth map data and image data captured by the computing device. By using the measurements obtained from the depth map data and image data, the computing device can determine the geometry and shape of the real-world object. Such measurements can be recorded in the data record.

[0034] The computing device 101 can also analyze text or other marks to generate or modify the data record associated with each object. For example, the computing device can determine and record the model, product brand or other relevant characteristics of the object. In this example, the data record 203B describing the horsepower of the engine can be generated based on the text engraved on the engine or the size or shape of the engine. These examples are provided for illustrative purposes and should not be interpreted as restrictive. It can be appreciated that any type of keyword, description or parameter of the object can be generated by the analysis of a specific object.

[0035] The computing device 101 can also analyze model data defining virtual objects, and can also be used to generate or modify data records describing various aspects of each object. For example, in this example, the model data defining the first virtual object 104A can be analyzed to determine that the object is made of a specific material such as steel. One or more records (such as record 204A) can be generated by an associated agent (such as the second agent 201B) to store such information. Figure 2 As shown in , another record 205A is also generated describing various aspects (eg, size) of the second virtual object 104B.

[0036] Agent 201 can also monitor a variety of different types of user activities during a multi-user communication session to generate or modify data records. The monitored user activities can include any type of user interaction with a computer or any type of communication, such as polling, editing, text chat, and voice conversation. The technology disclosed herein can monitor any type of interaction data that defines user input from an input device, user shared content, user sent or received streams or received content through a communication session. Figure 3 As shown in , agent 201 is capable of monitoring various communication media, including but not limited to: emails, phone calls, @ mentions, video calls, text messages, audio data of communication sessions, etc. For example, as shown in Figure 3 As shown in , each of the agents can analyze the email string sent between the first user 102A and the second user 102B to determine that the email is related to the second virtual object 104B. Each other communication form (e.g., video broadcast, @ mention or private call) can also be analyzed and parsed to identify parameters, preferences or other information related to each object.

[0037] Each agent 201 can collect and store activity data defining monitored activities in an individual database 202 associated with its respective object. Figure 4 As shown in , based on the monitored communications between users, each agent 201 stores the collected information related to each object. In some configurations, the collected information is received through an application programming interface (API) in communication with each agent 201. Each instance of communication (such as data from a phone call, email, or @mention) can be parsed and stored in a specific database 202. The data parsed from each communication instance can be stored in a record.

[0038] If a particular instance of user activity references an object, computing device 101 can associate activity data defining the particular instance of user activity with the object. For example, a keyword, phrase, image, audio data, or any other information having a defined threshold level of relevance to a particular object can cause computing device 101 to associate the particular instance of user activity with the object. Once an association with an object is established, the activity data defining the particular instance of user activity can be stored in an associated database.

[0039] For example, the broadcasted data can be analyzed to determine that the fan has a specification requirement, for example, the fan needs a performance of 200 cubic feet per minute (CFM). The analysis of such communication can be stored in a database record, such as record 204B. The email can be analyzed to determine various aspects of the fastening device, for example, the nut needs to have a specific strength, and the reply email can be analyzed to determine the parameters of the required strength, for example, the engine can produce 60 foot pounds of torque. The analysis of such communication can be stored in the database record of the associated object. For example, a record 203C of the first database 202A can be generated to indicate the engine torque specification, and a record 205B of the third database 202C can be generated to indicate the torque requirement of the fan. In addition, @ mentions can be analyzed to determine various aspects of the first virtual object 104A (e.g., fan). In response, a record such as record 204C can be generated to indicate that the fan needs to be made of carbon fiber.

[0040] Other forms of communication can be analyzed and parsed in a similar manner, and information related to a particular object is stored in an associated database. In this example, based on the Skype call, data record 203D indicates a need to increase the horsepower of the engine, and based on the text, data record 203E indicates a recommended brand name for the engine.

[0041] Now go to Figure 5 In some configurations, each agent 201 can retrieve supplemental data 501 from external resources 502. Supplemental data 501 can include any information related to the data records associated with each object. For example, queries can be generated based on existing data records, and each query can be sent to various resources 502, such as, but not limited to: inventory database 502A, company directory 502B, and search engine 502C. Each resource 502 can return supplemental data 501 in response to the query. In addition, each resource 502 can also push relevant supplemental data 501 to each agent.

[0042] In one illustrative example, first agent 201A can generate a query based on records (e.g., records 203A-203E) stored in first database 202A. In one illustrative example, first agent 201A can generate a query that defines parameters of a real-world object (e.g., shaft diameter, horsepower, brand name, etc.). In response to the query, a resource such as inventory database 502A can return supplemental data 501 that defines a new engine (e.g., Acme Model 1) that meets the performance requirements indicated in data record 203. Such supplemental data 501 can be stored in a new data record 203F in first database 202A.

[0043] In another illustrative example, the second agent 201B can generate a query defining various aspects based on the record of the first virtual object 104A, e.g., a 200 CFM fan is required and the model data indicates steel construction. In response to the query, a resource such as a company directory 502B can return supplemental data 501 identifying individuals with expertise in such an object. Such supplemental data 501 can be stored in a new data record 204D in the second database 202B.

[0044] In yet another illustrative example, the third agent 201C can generate a query based on the record associated with the second virtual object 104B, for example, the fastening device is a specific size and requires a specific strength. In response to the query, a resource such as a search engine 502C can return supplemental data 501 including a recommendation for the use of a torque wrench. Such supplemental data 501 can be stored in a new data record 205C in the third database 202C. It can be appreciated that the supplemental data 501 can include data in any format, including three-dimensional model data, performance statistics, images, audio data, etc.

[0045] As in Figure 6 As shown in , computing device 101 can generate recommendations for individual objects based on the stored information. The recommendations can suggest modifications to the object, provide resources for obtaining or modifying the object, and provide actionable information that allows a user to agree on an object. The recommendations can be in the form of: (1) modifications to an existing virtual object; (2) new virtual objects positioned above real-world objects; or (3) annotations recommending modifications to real-world objects. Data defining the recommendations associated with each object can also be permanently stored in the data record.

[0046] Figure 6 A number of exemplary recommendations are illustrated. For example, a recommendation can take the form of a new virtual object 601 that is displayed as a virtual partial overlay over an image of a real-world object. In this example, the new virtual object 601 takes the form of another engine (e.g., Acme Model 1) that meets the performance requirements indicated in data record 203. Such recommendations can be automatically presented as a three-dimensional or two-dimensional virtual object that is provided as an overlay on a real-world object or other virtual objects. By providing an overlay on a real-world object, a user can easily visualize the offer without the user having to manually manipulate the content.

[0047] The recommendations can also include computer-generated modifications to the design based on agent analysis of the communication data and supplemental data. Figure 6As shown in , a new virtual component 602 (eg, a gasket) can be added to the schematic layout. The position and size of the new virtual component 602 can be based on the information provided in the supplementary data 501.

[0048] The agent recommendation (also referred to herein as a "recommendation") can also include a graphical element that provides other contextual information 603 or instructions 604 related to the object. Figure 6 As shown in , a graphical element can be displayed in association with a particular object. Context information 603 or instructions 604 can be generated based on supplementary data 501 or any other data record associated with the object. Figure 6 In the example shown in , some recommendations indicate that the steel fan should be replaced with a carbon fiber fan, and identify the supplier, etc.

[0049] The computing device 101 can prioritize and rank various recommendations based on the content of the data record. In some configurations, the computing device 101 can store data defining the maximum number of recommendations that can be displayed. The computing device 101 can display any number of recommendations, up to the maximum number of recommendations. In some embodiments, the display of the recommendations can be arranged according to the defined priority of each recommendation, for example, the recommendations can be sorted from the highest priority to the lowest priority. Therefore, the recommendations can also be prioritized based on any type of contextual information. For example, the recommendations can be prioritized and ranked based on each user's votes or preferences. In other examples, the priority for each recommendation can be based on the number of comments made by various users on a particular object, the number of interactions that reference the recommendation, or a combination of other types of user activities, such as the number of times a user views a particular object or recommendation.

[0050] The user can interact with the proxy recommendations by selecting or otherwise interacting with the displayed recommendations. The user can select a particular recommendation by providing a voice gesture or a touch gesture. In some configurations, user interactions with a particular recommendation can be interpreted as a vote or preference for the particular recommendation. Communication data can also be monitored to identify votes or preferences for a particular recommendation. Supporting comments or votes can be calculated, and when a particular defined threshold of comments or votes is reached, the computing device 101 can determine consensus for a particular group of users.

[0051] When consensus is reached, computing device 101 can take a number of different actions. For example, when a threshold number of votes are received for a particular recommendation, the modification can be permanently written to database 202 associated with the particular object. In another example, when a threshold number of votes are received for a particular recommendation, computing device 101 can generate a subsequent query for additional supplemental information based on the recommendation, which in turn can result in the generation of additional recommendations.

[0052] As outlined above, intelligent agent 201 can be retained in different communication sessions to enhance user participation and improve productivity. For example, if a first group of users is conducting a Skype conference about automobile engine design, intelligent agents can be instantiated for each component of the engine. Intelligent agents can monitor all types of user activities during a multi-user communication session, such as polling, editing, text chat, and voice dialogue. Data derived from monitored activities can be displayed and stored in association with each object (e.g., each engine component). Agents and associated databases are configured to make when the Skype conference ends, agents and stored data are retained after the communication session. Therefore, when creating a new communication channel such as a private chat session or a new meeting, agents allow users to access agents and stored data.

[0053] Reference now Figure 7 , various aspects of a computing device that enables permanent storage of agents and stored data are shown and described below. In this illustrative example, server 701 includes memory 703 that stores agent data 704 that defines various aspects of individual agents 201 and associated databases 202. In continuing with the example described above, Figure 7 Shown are first agent data 704A defining a first agent 201A and a first database 202A, second agent data 704B defining a second agent 201B and a second database 202B, and third agent data 704C defining a third agent 201C and a third database 202C.

[0054] In this example, the management agent 705 can receive agent data 704 from any communication session, such as a Skype conference, broadcast, or chat session, as well as the stored agent data 704 and memory 703. The memory 703 can be configured to maintain the agent data 704 independently of the life cycle of each communication session 1004. Thus, the management agent 705 can deliver the agent data 704 to other communication sessions.

[0055] exist Figure 7 In the example shown in FIG. 1 , a first communication session 1004A (such as in Figure 1 ) can generate agent data 704 defining individual agents 201 and associated databases 202. During or at the end of a communication session, the agent data 704 can be transmitted to a server 701 executing a management agent 705. The management agent 705 can store the agent data permanently in a memory 703, which can be in the form of a database, a persistent memory, or any other memory device that allows the agent data to persist regardless of the execution and termination of any communication session.

[0056] Next, when creating other communication sessions (such as the second communication session 1004B and the third communication session 1004C), the management agent 705 can provide relevant agent data 704 for each session. The agent data 704 that defines a particular agent 201 can be selected based on one or more factors. For example, if the new communication session includes content with keywords that describe a particular object, the agent data 704 related to the object can be transmitted to the computing device managing the communication session. In this example, the second agent data 704B and the third agent data 704C are delivered to the second communication session 1004B, and the third agent data 704C is delivered to the third communication session 1004C.

[0057] Reference now Figure 8 , an exemplary user interface 800 for a second communication session 1004B is shown and described below. In this example, the second communication session 1004B takes the form of a private chat session between two individuals, Sarah and Steve. In this example, it is assumed that their conversation includes a discussion about fans and fastening devices. In response to the agent detecting the context of the conversation, the objects defined in the relevant agent data 704 are displayed in the user interface 800 of its communication session. This functionality eliminates the need for the user to collect information and interact with the computer to display the collected information within the session.

[0058] It can be appreciated that the second communication session 1004B can be processed as described above. Thus, the user activity of the private chat session can be used to obtain additional supplemental data, as well as cause the generation of additional data records for each object. Likewise, additional recommendations can be generated and stored in new data records associated with each object. Additional agents 201 and associated databases 202 can be generated when a user creates a new virtual object or views a new real-world object.

[0059] Reference now Fig. 9, an exemplary user interface 900 for a third communication session 1004C is shown and described below. In this example, the third communication session 1004C takes the form of a team session between a large number of participants. In the team session, multiple users communicate via a chat window and a shared video session 902. In this example, the participants begin chatting about a specific object (fastening device). In response to detecting keywords or other information related to an object defined in at least one database 202 (such as a fastening device), the computing device managing the third communication session 1004C is able to retrieve proxy data 704C associated with the object. Based on the content of the proxy data 704C, the computer managing the third communication session 1004C is able to present the relevant object and other information stored in the associated data record. As shown in Fig. 9 As shown in , the second virtual object 104B is presented in the user interface 900 along with other information (such as relevant contextual information 603). The agent can automatically retrieve and automatically present such information based on the context of the conversation. The second virtual object 104B and the relevant contextual information 603 can be retrieved and displayed in response to a conversation object reference (e.g., a question or mention of a virtual object).

[0060] The retrieval and display of contextually relevant information in a usable format can improve the interaction between a user and a computing device by reducing the need for manual operations required to perform those operations. This feature can reduce unintentional input and increase productivity for individuals while also reducing the computing resources required to perform those manual operations.

[0061] It can be appreciated that the second communication session 1004B can also be processed as described above. Thus, the user activity of the private chat session can be used to obtain additional supplemental data, as well as cause the generation of additional data records for each object. In addition, additional recommendations can be generated and stored in new data records associated with each object. Additional agents 201 and associated databases 202 can be generated when a user creates a new virtual object or views a new real-world object.

[0062] In some configurations, machine learning techniques may be used to examine data records to generate recommendations. The term "machine learning" may refer to one or more programs that learn from data that it receives and analyzes. For example, a machine learning mechanism may build, modify, or otherwise utilize a model created from exemplary inputs and use the model to make predictions or decisions. The machine learning mechanism may be used to improve the identification or generation of the recommendations based on the requirements of an object or user preferences. The model may be trained using supervised and / or unsupervised learning. For example, over time, as the machine learning mechanism receives more data, the recommendations displayed within the collaborative environment may change over time based on data defining user activity.

[0063] Different machine learning mechanisms may be utilized. For example, a classification mechanism may be used to determine proxy recommendations based on requirements associated with an object and the availability of other objects or information that meet those requirements. In another example, different classifications can indicate whether a user likes or dislikes a particular object or proxy recommendation. The classification mechanism may classify display elements into different categories to provide an indication of whether the display element should be displayed.

[0064] In other examples, a statistical mechanism may be used to determine whether a particular proxy recommendation should be displayed or whether a particular object should be selected as an object of interest. For example, a linear regression mechanism may be used to generate a score indicating the likelihood that an object is an object of interest. Linear regression may refer to a process for modeling the relationship between a variable and one or more other variables. Different linear regression models may be used to calculate the score. For example, the least squares method may be used, the maximum likelihood estimation may be used, or another method may be used.

[0065] Fig.10 is a diagram illustrating aspects of a routine 1000 for computationally efficient management of data associated with objects displayed within a mixed reality and virtual reality collaborative environment. It should be understood by those of ordinary skill in the art that the operations of the methods disclosed herein are not necessarily presented in any particular order, and that performance of some or all operations in (one or more) alternative orders is possible and contemplated. For ease of description and illustration, the operations have been presented in the order demonstrated. Operations may be added, omitted, performed together, and / or performed simultaneously without departing from the scope of the appended claims.

[0066] It should also be understood that the illustrated method can be terminated at any time and does not need to be performed in its entirety. As defined herein, some or all of the operations and / or substantially equivalent operations of the method can be performed by executing computer-readable instructions included on a computer storage medium. The term "computer-readable instructions" used in the specification and claims and variations thereof are widely used herein to include routines, applications, application modules, program modules, programs, components, data structures, algorithms, etc. Computer-readable instructions can be implemented on various system configurations, including single-processor or multi-processor systems, minicomputers, mainframe computers, personal computers, handheld computing devices, microprocessor-based programmable consumer electronics, combinations thereof, etc.

[0067] Therefore, it should be appreciated that the logical operations described herein are implemented as: (1) a sequence of computer-implemented actions or program modules running on a computing system such as those described herein; and / or (2) interconnected machine logic circuits or circuit modules within a computing system. The implementation is a matter of choice depending on the performance and other requirements of the computing system. Therefore, the logical operations may be implemented in software, firmware, dedicated digital logic, and any combination thereof.

[0068] In addition, it is possible to implement in association with the exemplary presentation UI described above. Fig.10 For example, the various device(s) and / or module(s) described herein can generate, send, receive, and / or display data associated with the content of a communication session (e.g., live content, broadcast events, recorded content, etc.) and / or present a UI that includes a rendering of one or more participants of a remote computing device, avatar, channel, chat session, video stream, image, virtual object, and / or application associated with the communication session.

[0069] Routine 1000 begins at operation 1002, where the computing device 101 receives sensor data defining a 3D representation of a real-world environment. The sensor data can be captured by a depth map sensor (e.g., a depth map camera). Additionally, the sensor data can be captured by an image sensor (e.g., a camera), where the depth map sensor and the image sensor can be part of the same component or in separate components. The sensor data includes depth map data defining a three-dimensional model of the real-world environment and an image of the real-world environment. For example, the real-world environment may include the walls of a room and certain objects within the room, such as in a room. Figure 1The sensor data can define physical properties of an object or multiple real-world objects within the real-world environment. The sensor data also indicates the geographic location of one or more objects within the environment. Therefore, measurements of objects or measurements of the environment can be performed by analyzing the sensor data. One or more objects defined in the sensor data are shared with multiple users participating in the collaborative environment. The collaborative environment can include a communication session that allows users to send, receive and view aspects of the sensor data presented on the display device.

[0070] The routine then proceeds to operation 1004, where the computing device 101 receives model data defining one or more virtual objects to be displayed within the view of the collaborative environment. The model data can define specific locations where the virtual objects are placed within the user interface of the collaborative environment.

[0071] At operation 1006, the computing device can identify virtual objects and real-world objects of interest. As described herein, objects can be deemed objects of interest based on a number of factors, including, but not limited to, a threshold level of user interaction with a particular object, a threshold level of communication about a particular object, a threshold level of movement of a particular object, and the like.

[0072] In some embodiments, an object can be considered an object of interest by determining the level of interaction of an input signal received from an input device relative to at least one of a plurality of real-world objects. This can include the number of inputs used to edit a particular object, the number of comments about a particular object, people's attention to a particular item, such as the time someone spends looking at a particular item, etc. This can be determined by tracking the eye movements of a particular user and recording how much time a person spends looking at an object over a period of time. The system can determine when the level of interaction exceeds a defined threshold. In response to determining that the level of interaction exceeds the threshold, the system can determine that at least one object is an object of interest. Such technology can also be applied to various aspects of an object, such as the size of the object, the amount of movement of the object, the color of the object, the temperature of the object, the texture of the object, etc. The level of movement can represent speed, distance, rotation speed, or any other type of movement. When the movement exceeds the threshold, the system can regard the particular object as an object of interest.

[0073] In response to detecting a threshold level change in at least one of temperature, color, or a physical property of a particular object, the system can deem the particular object as an object of interest. The physical property change can be, for example, shape, e.g., the object is melting, changing from liquid to solid, from solid to liquid, etc.

[0074] In response to determining that the physical properties of a particular object match one or more predetermined properties, the system may deem the particular object an object of interest. For example, the system may have a preference file indicating predetermined properties, such as a visual outline of an object, such as an engine or a computer. If the image data or depth map data indicates that a particular real-world object has physical properties, such as dimensions, color, or size, that match the physical properties described in the preference file at least within a threshold difference, then the particular real-world object may be selected as an object of interest. For example, this feature may allow the system to select all computers in a room or all engines in a view. This can avoid identifying unwanted objects, such as walls, furniture, or people. The technology can also allow computing devices to be more efficient by creating proxies and databases only for specific objects in a meeting. Therefore, proxies will only be generated for items that are considered important in a meeting.

[0075] Next, at operation 1008, the computing device can generate one or more agents and an associated database associated with the virtual object of interest. For illustrative purposes, the generation of an agent is referred to herein as the generation of data defining an agent instance, wherein the agent instance is associated with at least one object. The data defining the agent instance is permanently stored for access by multiple communication sessions. As described herein, individual agents are instantiated for the identified objects. The individual agents are configured to analyze various aspects of the object to generate descriptions, keywords, or other information about the object.

[0076] At operation 1010, the computing device monitors user activity to identify information associated with each object. Keyword descriptions or other parameters associated with the objects are collected from various sources such as input devices, search engines, inventory databases, committee directories, etc.

[0077] At operation 1012, the computing device can update the individual database with the identified information.As described herein, any information collected in association with an object can be stored by an agent and stored within a database associated with the object.

[0078] At operation 1014, the computing device can generate one or more recommendations associated with the individual object. The recommendations can include recommendations for modifying the object. The recommendations can be in the form of: (1) modifications to an existing virtual object; (2) new virtual objects positioned over real-world objects; or (3) annotations recommending modifications to real-world objects. The recommendations can also include display statistics, performance parameters, requirements, and other contextual data.

[0079] The data defining the recommendations associated with each object can also be persistently stored across communication sessions. Thus, when an object (such as an engine component) is moved or deleted during a session, the recommendations are modified in the other communication sessions to indicate such changes. For example, if the first communication session described above is running concurrently with the second communication session, a modification to a particular object within the first communication session will update the record data and subsequently update the display of the particular object or the recommendations for the object within the second communication session.

[0080] In some configurations, operation 1014 can include generating a virtual display of one or more recommendations associated with renderings of individual virtual objects and views of at least one real-world object. The recommendations can be in at least one of the following formats, as shown below. In one example, the recommendation can include a modification to at least one virtual object including at least one new parameter. For example, the recommendation can actually modify a second virtual object (such as a fixture) to have a different size, a different color, a different shape, a different texture, etc. In another example, the recommendation can include a new virtual object positioned above the real-world object. This is illustrated in the example above, where a virtual model of the new engine is displayed over the real-world engine. In yet another example, the recommendation can include an annotation that recommends a modification to at least one real-world object or virtual object. As shown in Figure 6 As shown in the example of , many annotations can provide any information for the associated object, such as a new size, a new color, a different shape, or any other type of modification. In addition, annotations can provide supplier names, contact names, or any other contextual data useful for obtaining a specific project. Other information can be displayed in association with a specific object, such as voting results, consensus decisions, or any other decision-making data.

[0081] At operation 1016, the computing device can store and process the recommendations for machine learning purposes. As described herein, any collected information (such as communication data or supplemental data) or any generated information (such as recommendations) can be provided as input to a machine learning algorithm to improve the generation of future recommendations.

[0082] It should be appreciated that the above subject matter can be implemented as a computer-controlled device, a computer process, a computing system, or an article such as a computer-readable storage medium. The operations of the exemplary method are illustrated in separate boxes and are summarized with reference to those boxes. The method is illustrated as a logical flow of boxes, each of which can represent one or more operations that can be implemented in hardware, software, or a combination thereof. In the context of software, the operations represent computer-executable instructions stored on one or more computer-readable media that, when executed by one or more processors, enable one or more processors to perform the operations.

[0083] Typically, computer executable instructions include routines, programs, objects, modules, components, data structures, etc. that perform specific functions or implement specific abstract data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations can be performed in any order, combined in any order, subdivided into multiple sub-operations, and / or performed in parallel to implement the described processes. The described processes can be performed by resources associated with one or more devices (such as one or more internal or external CPUs or GPUs) and / or one or more hardware logic (such as a field programmable gate array ("FPGA"), a digital signal processor ("DSP"), or other types of accelerators).

[0084] All methods and processes described above may be embodied in and fully automated by software code modules executed by one or more general-purpose computers or processors. The code modules may be stored in any type of computer-readable storage medium or other computer storage device, such as those described below. Some or all of the methods described may alternatively be embodied in dedicated computer hardware, such as described below.

[0085] Any conventional description, element or box in the flowcharts described herein and / or depicted in the accompanying drawings should be understood to potentially represent a module, segment or part of a code, which includes one or more executable instructions for implementing a specific logical function or element in a routine. Alternative implementations are included within the scope of the examples described herein, wherein, as will be appreciated by those skilled in the art, depending on the functionality involved, elements or functions may be deleted or executed in a different order than shown or discussed, including substantially simultaneously or in a reverse order.

[0086] Fig.11 1104 . 1104 . 1105 . 1106 . 1107 . 1108 . 1109 . 1110A . 1111B . 1112C . 1114 . 1115 . 1116 . 1117 . 1118 . 1119 . 1120 . 1121 . 1122 . 1123 . 1124 . 1125 . 1126 . 1127 . 1128 . 1129 . 1130 . 1131

[0087] As illustrated, a communication session 1104 may be implemented between a plurality of client computing devices 1106(1) through 1106(N) associated with or as part of the system 1102 (where N is a number equal to or greater than 2). The client computing devices 1106(1) through 1106(N) enable users (also referred to as individuals) to participate in the communication session 1104. For example, the first client computing device 1106(1) may be Figure 1 The computing device 101 or Fig.13 A computing device 1300 is provided.

[0088] In this example, communication session 1104 is hosted by system 1102 on one or more networks 1108. That is, system 1102 can provide services that enable users of client computing devices 1106(1)-1106(N) to participate in communication session 1104 (e.g., via real-time viewing and / or recorded viewing). Thus, "participants" of communication session 1104 can include users and / or client computing devices (e.g., multiple users can participate in a communication session in a room using a single client computing device), each of which can communicate with other participants. Alternatively, communication session 1104 can be hosted by one of client computing devices 1106(1)-1106(N) using peer-to-peer technology. System 1102 can also host chat conversations and other team collaboration functionality (e.g., as part of an application suite).

[0089] In some implementations, such chat conversations and other team collaboration functions are considered to be external communication sessions that are distinct from the communication session 1104. A computerized agent that collects participant data in the communication session 1104 can be linked to such an external communication session. Thus, the computerized agent can receive information such as date, time, session participants, etc. that enables connection to such an external communication session. In one example, a chat session can be conducted based on the communication session 1104. In addition, the system 1102 can host the communication session 1104, which includes at least a plurality of participants co-located at a meeting location (such as a conference room or auditorium) or located at different locations.

[0090] In the examples described herein, client computing devices 1106(1) to 1106(N) participating in a communication session 1104 are configured to receive and render communication data for display on a user interface of a display screen. The communication data can include a collection of various instances or streams of real-time content and / or recorded content. The collection of various instances or streams of real-time content and / or recorded content can be provided by one or more cameras (e.g., a video camera). For example, a single stream of real-time or recorded content can include media data associated with a video feed provided by a video camera (e.g., audio and visual data capturing the appearance and voice of a user participating in the communication session). In some implementations, the video feed can include such audio and visual data, one or more still images, and / or one or more avatars. The one or more still images can also include one or more avatars.

[0091] Another example of a real-time or recorded individual content stream can include media data having avatars of users participating in a communication session and audio data capturing the user's voice. Another example of a real-time or recorded individual content stream can include media data having a file displayed on a display screen and audio data capturing the user's voice. Thus, various streams of real-time or recorded content in the communication data enable remote conferencing to be facilitated between a group of people and sharing content within a group of people. In some implementations, various streams of real-time or recorded content within the communication data can originate from multiple co-located video cameras located in a space such as a room to record or stream a presentation including one or more individuals presenting the content and one or more individuals consuming the presented content.

[0092] Participants or attendees can view the content of the communication session 1104 in real time as the activity occurs, or alternatively view it via recording at a later time after the activity occurs. In the example described herein, the client computing devices 1106 (1) to 1106 (N) participating in the communication session 1104 are configured to receive and render to display communication data on a user interface of a display screen. The communication data can include a collection of various instances or streams of real-time and / or recorded content. For example, an individual content stream can include media data associated with a video feed (e.g., audio and visual data that captures the appearance and voice of a user participating in the communication session). Another example of an individual content stream can include media data with an avatar of a user participating in a conference session and audio data that captures the user's voice. Yet another example of an individual content stream can include media data with content items displayed on a display screen and / or audio data that captures the user's voice. Therefore, the various content streams within the communication data enable a meeting or broadcast presentation to be facilitated between a group of people dispersed in a remote location.

[0093] A participant or attendee of a communication session is a person who is within range of a camera or other image and / or audio capture device, thereby being able to capture (e.g., record) the person's movements and / or sounds generated while the person is viewing and / or listening to content shared via the communication session. For example, a participant may be sitting in a crowd, viewing real-time content shared at a broadcast location where a stage presentation is taking place. Or, a participant may be sitting in an office conference room, viewing shared content of a communication session with other colleagues via a display screen. Even further, a participant may be sitting or standing in front of a personal device (e.g., a tablet computer, a smartphone, a computer, etc.), viewing shared content of a communication session alone in his or her office or home.

[0094] System 1102 includes device(s) 1110. Device(s) 1110 and / or other components of system 1102 can include distributed computing resources that communicate with each other and / or with client computing devices 1106(1)-1106(N) via one or more networks 1108. In some examples, system 1102 may be a stand-alone system tasked with managing aspects of one or more communication sessions, such as communication session 1104. For example, system 1102 may be managed by an entity such as Slack, WEBEX, GOTOMEETING, GOOGLE HANGOUTS, etc.

[0095] The network(s) 1108 may include, for example, a public network (such as the Internet), a private network (such as an institutional and / or personal intranet), or some combination of private and public networks. The network(s) 1108 may also include any type of wired and / or wireless network, including, but not limited to, a local area network ("LAN"), a wide area network ("WAN"), a satellite network, a cable network, a Wi-Fi network, a WiMax network, a mobile communication network (e.g., 3G, 4G, etc.), or any combination thereof. The network(s) 1108 may utilize communication protocols, including packet-based and / or datagram-based protocols, such as the Internet Protocol ("IP"), the Transmission Control Protocol ("TCP"), the User Datagram Protocol ("UDP"), or other types of protocols. In addition, the network(s) 1108 may also include devices that facilitate network communications and / or form the hardware foundation of the network, such as switches, routers, gateways, access points, firewalls, base stations, repeaters, backbone devices, and the like.

[0096] In some examples, network(s) 1108 may also include devices that enable connection to wireless networks, such as wireless access points (“WAPs”). Examples support connectivity via WAPs that send and receive data over various electromagnetic frequencies (e.g., radio frequencies), including WAPs that support the Institute of Electrical and Electronics Engineers (IEEE) 802.11 standards (e.g., 802.11g, 802.11n, 802.11ac, etc.), as well as other standards.

[0097] In various examples, (one or more) devices 1110 may include one or more computing devices operating in a cluster or other grouping configuration to share resources, balance loads, improve performance, provide failover support or redundancy, or for other purposes. For example, (one or more) devices 1110 may belong to various categories of devices, such as conventional server-type devices, desktop computer-type devices, and / or mobile devices. Therefore, although illustrated as a single type of device or a server-type device, (one or more) devices 1110 may include various device types and are not limited to a particular type of device. Device 1110 may represent, but is not limited to, a server computer, a desktop computer, a network server computer, a personal computer, a mobile computer, a laptop computer, a tablet computer, or any other type of computing device.

[0098] The client computing device (e.g., one of client computing devices 1106(1)-1106(N)) can belong to a variety of categories of devices, which can be the same as or different from device(s) 1110, such as conventional client-type devices, desktop-type devices, mobile-type devices, dedicated-type devices, embedded devices, and / or wearable-type devices. Thus, the client computing device can include, but is not limited to, a desktop computer, a game console and / or gaming device, a tablet computer, a personal data assistant ("PDA"), a mobile phone / tablet hybrid device, a laptop computer, a telecommunications device, a computer navigation-type client computing device such as a satellite-based navigation system including a global positioning system ("GPS") device, a wearable device, a virtual reality ("VR") device, an augmented reality ("AR") device, an implanted computing device, an automotive computer, a web-enabled television, a thin client, a terminal, an Internet of Things ("IoT") device, a workstation, a media player, a personal video recorder ("PVR"), a set-top box, a camera, an integrated component (e.g., a peripheral device) included in a computing device, an appliance, or any other kind of computing device. Furthermore, the client computing device may include a combination of the earlier listed examples of client computing devices, such as, for example, a desktop-type device or a combination of a mobile-type device and a wearable device, and the like.

[0099] Client computing devices 1106(1) to 1106(N) of various classes and device types can represent any type of computing device having one or more data processing units 1112 operably connected to a computer-readable medium 1194 such as via a bus 1116, which in some instances can include one or more of: a system bus, a data bus, an address bus, a PCI bus, a Mini-PCI bus, and any of a variety of local, peripheral and / or independent buses.

[0100] The executable instructions stored on the computer-readable medium 1194 may include, for example, an operating system 1119 , a client module 1120 , a profile module 1122 , and other modules, programs, or applications that can be loaded and executed by the data processing unit 1192 .

[0101] The client computing devices 1106(1)-1106(N) may also include one or more interfaces 1124 to enable communications between the client computing devices 1106(1)-1106(N) and other networked devices (such as device(s) 1110) over the network(s) 1108. Such network interface(s) 1124 may include one or more network interface controllers (NICs) or other types of transceiver devices to send and receive communications and / or data over the network. In addition, the client computing devices 1106(1)-1106(N) may include input / output ("I / O") interfaces (devices) 1126 to enable communications with input / output devices, such as user input devices including peripheral input devices (e.g., game controllers, keyboards, mice, pens, voice input devices (such as microphones), video cameras for obtaining and providing video feeds and / or still images, touch input devices, gesture input devices, etc.), and / or output devices including peripheral output devices (e.g., displays, printers, audio speakers, tactile output devices, etc.). Fig.11 The client computing device 1106 ( 1 ) is illustrated as being connected in some manner to a display device (eg, display screen 1129 ( 1 )) that is capable of displaying a UI in accordance with the techniques described herein.

[0102] exist Fig.11In the exemplary environment 1100 of FIG. 1 , client computing devices 1106(1) to 1106(N) can use their respective client modules 1120 to connect to each other and / or (one or more) other external devices to participate in a communication session 1104 or to contribute activities to the collaborative environment. For example, a first user can utilize client computing device 1106(1) to communicate with a second user of another client computing device 1106(2). When executing client modules 1120, the users can share data, which may result in client computing device 1106(1) connecting to system 1102 and / or other client computing devices 1106(2) to 1106(N) via network(s) 1108.

[0103] The client computing devices 1106(1) to 1106(N) may use their respective profile modules 1122 to generate participant profiles (in Fig.11 1102) and provide the participant profile to other client computing devices and / or device(s) 1110 of system 1102. A participant profile may include one or more of the following: an identity of a user or a group of users (e.g., name, unique identifier (“ID”), etc.), user data (such as personal data), machine data (such as location (e.g., IP address, room in a building, etc.), and technical capabilities, etc. The participant profile may be used to register a participant to a communication session.

[0104] As in Fig.11 , the device(s) 1110 of the system 1102 include a server module 1130 and an output module 1132. In this example, the server module 1130 is configured to receive media streams 1134(1) to 1134(N) from individual client computing devices (such as client computing devices 1106(1) to 1106(N)). As described above, the media streams can include a video feed (e.g., audio and visual data associated with a user), audio data to be output with a presentation of an avatar of the user (e.g., an audio-only experience in which the user's audio data is not sent), text data (e.g., a text message), file data, and / or screen sharing data (e.g., a document, slide, image, video, etc. displayed on a display screen), etc. Thus, the server module 1130 is configured to receive a collection of various media streams 1134(1) to 1134(N) (the collection is referred to herein as "media data 1134") during the real-time viewing communication session 1104. In some scenarios, not all client computing devices participating in the communication session 1104 provide media streams. For example, a client computing device may be a consuming device or a "listening" device only, such that it only receives content associated with the communication session 1104, but does not provide any content to the communication session 1104.

[0105] In various examples, the server module 1130 can select aspects of the media stream 1134 to be shared with individual client computing devices of the participating client computing devices 1106(1)-1106(N). Accordingly, the server module 1130 can be configured to generate session data 1136 based on the media stream 1134 and / or pass the session data 1136 to the output module 1132. The output module 1132 can then communicate communication data 1138 to the client computing devices (e.g., the client computing devices 1106(1)-1106(3) participating in the real-time viewing communication session). The communication data 1138 can include video, audio, and / or other content data, which is provided by the output module 1132 based on the content 1150 associated with the output module 1132 and based on the received session data 1136.

[0106] As shown, output module 1132 sends communication data 1139(1) to client computing device 1106(1), and sends communication data 1139(2) to client computing device 1106(2), and sends communication data 1139(3) to client computing device 1106(3), etc. The communication data 1139 sent to the client computing devices can be the same or different (e.g., the location of the content stream within the user interface can vary between devices).

[0107] In various implementations, the device(s) 1110 and / or the client module 1120 can include a UI presentation module 1140. The UI presentation module 1140 can be configured to analyze the communication data 1139 that is transmitted to the one or more client computing devices 1106. Specifically, at the device(s) 1110 and / or the client computing device 1106, the UI presentation module 1140 can analyze the communication data 1139 to determine an appropriate manner for displaying the video, image, and / or content on the display screen 1129 of the associated client computing device 1106. In some implementations, the UI presentation module 1140 can provide the video, image, and / or content to a presentation UI 1146 that is presented on the display screen 1129 of the associated client computing device 1106. The UI presentation module 1140 can cause the presentation UI 1146 to be presented on the display screen 1129. The presentation UI 1146 can include the video, image, and / or content analyzed by the UI presentation module 1140.

[0108] In some implementations, the presentation UI 1146 may include multiple sections or grids that may render or include video, images, and / or content for display on the display screen 1129. For example, a first section of the presentation UI 1146 may include a video feed of a presenter or individual, and a second section of the presentation UI 1146 may include a video feed of an individual consuming conference information provided by the presenter or individual. The UI presentation module 1140 may populate the first section and the second section of the presentation UI 1146 in a manner that appropriately mimics an ambient experience that the presenter and individual may share.

[0109] In some implementations, the UI presentation module 1140 can zoom in or provide an enlarged view of the individual represented by the video feed to highlight the individual's reaction to the presenter, such as facial features. In some implementations, the presentation UI 1146 can include video feeds of multiple participants associated with a conference (such as a general communication session). In other implementations, the presentation UI 1146 can be associated with a channel (such as a chat channel, an enterprise team channel, etc.). Therefore, the presentation UI 1146 can be associated with an external communication session that is different from a general communication session.

[0110] Fig.12 A diagram showing exemplary components of an exemplary device 1200 (also referred to herein as a "computing device") is illustrated that is configured to generate data for some user interfaces disclosed herein. The device 1200 can generate data that can include one or more parts that can render or include video, images, virtual objects 116, and / or content for display on a display screen 1129. The device 1200 can represent one of the (one or more) devices disclosed herein. Additionally or alternatively, the device 1200 can represent one of the client computing devices 1106.

[0111] As illustrated, device 1200 includes one or more data processing units 1202, computer-readable media 1204, and communication interface(s) 1206. The components of device 1200 are operably connected, for example, via bus 1209, which may include one or more of a system bus, a data bus, an address bus, a PCI bus, a Mini-PCI bus, and any of a variety of local, peripheral, and / or independent buses.

[0112] As utilized herein, data processor unit(s) (such as data processing unit(s) 1202 and / or data processing unit(s) 1192) may represent, for example, a CPU-type data processing unit, a GPU-type data processing unit, a field programmable gate array (“FPGA”), another type of DSP, or other hardware logic components that may be driven by a CPU in some cases. For example, but not limited to, exemplary types of hardware logic components that may be utilized include application specific integrated circuits (“ASICs”), application specific standard products (“ASSPs”), systems on chips (“SOCs”), complex programmable logic devices (“CPLDs”), and the like.

[0113] As used herein, computer-readable media (such as computer-readable media 1204 and computer-readable media 1194) can store instructions that can be executed by (one or more) data processing units. The computer-readable medium can also store instructions that are executed by an external data processing unit (such as by an external CPU, an external GPU), and / or instructions that can be executed by an external accelerator (such as an FPGA-type accelerator, a DSP-type accelerator, or any other internal or external accelerator). In various examples, at least one CPU, GPU, and / or accelerator is incorporated into the computing device, and in some examples, one or more of the CPU, GPU, and / or accelerator is external to the computing device.

[0114] Computer readable media (media) may also be referred to herein as computer readable media (medium), which may include computer storage media and / or communication media. Computer storage media may include one or more of the following: volatile memory, non-volatile memory, and / or other permanent and / or auxiliary computer storage media, removable and non-removable computer storage media, which are implemented in any method or technology for storing information such as computer readable instructions, data structures, program modules or other data. Thus, computer storage media includes media in tangible and / or physical form contained in a device and / or hardware component that is part of a device or external to a device, including, but not limited to: random access memory (“RAM”), static random access memory (“SRAM”), dynamic random access memory (“DRAM”), phase change memory (“PCM”), read only memory (“ROM”), erasable programmable read-only memory (“EPROM”), electrically erasable programmable read-only memory (“EEPROM”), flash memory, compact disk read-only memory (“CD-ROM”), digital versatile disk (“DVD”), optical cards or other optical storage media, cassettes, magnetic tape, magnetic disk storage, magnetic cards or other magnetic storage devices or media, solid-state memory devices, storage arrays, network attached storage, storage area network, hosted computer storage, or any other storage memory, storage device, and / or storage medium that can be used to store and maintain information accessed by a computing device.

[0115] In contrast to computer storage media, communication media may embody computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transmission mechanism. As defined herein, computer storage media does not include communication media. That is, computer storage media itself does not include communication media consisting only of a modulated data signal, carrier wave, or propagation signal.

[0116] The communication interface 1206 may represent, for example, a network interface controller ("NIC") or other type of transceiver device to send and receive communications over a network. In addition, the communication interface(s) 1206 may include one or more video cameras and / or audio devices 1222 to enable the generation of video feeds and / or still images, etc.

[0117] In the illustrated example, computer-readable medium 1204 includes a data repository 1208. In some examples, data repository 1208 includes a data store such as a database, a data warehouse, or other type of structured or unstructured data store. In some examples, data repository 1208 includes a corpus and / or a relational database having one or more tables, indexes, storage processes, etc., to enable data access, for example, including one or more hypertext markup language ("HTML") tables, resource description framework ("RDF") tables, web ontology language ("OWL") tables, and / or extensible markup language ("XML") tables.

[0118] The data repository 1208 can store data for the operation of processes, applications, components, and / or modules stored in the computer-readable medium 1204 and / or executed by the (one or more) data processing units 1202 and / or (one or more) accelerators. For example, in some examples, the data repository 1208 can store session data 1210 (e.g., session data 1136), profile data 1212 (e.g., associated with participant profiles), and / or other data. The session data 1210 can include the total number of participants (e.g., users and / or client computing devices) in the communication session, the activities occurring in the communication session, the list of invitees for the communication session, and / or other data related to when and how the communication session is conducted or hosted. The data repository 1208 can also include content data 1214, such as content including video, audio, or other content for rendering and display on one or more display screens 1129.

[0119] Alternatively, some or all of the above reference data can be stored on a separate memory 1216 on one or more data processing units 1202, such as a memory on a CPU-type processor, a GPU-type processor, an FPGA-type accelerator, a DSP-type accelerator, and / or other accelerator. In this example, the computer-readable medium 1204 also includes an operating system 1218 and an application programming interface 1220 (API) configured to disclose functions and data of the device 1200 to other devices. In addition, the computer-readable medium 1204 includes one or more modules, such as a server module 1230, an output module 1232, and a GUI presentation module 1240, but the number of modules illustrated is only an example, and the number may be higher or lower. That is, the functions associated with the illustrated modules described herein may be performed by a smaller number of modules or a larger number of modules distributed on one device or across multiple devices.

[0120] Fig.131 is a diagram of a computing device that illustrates various aspects of the configuration and operation of a computing device 1300 capable of implementing various aspects of the system disclosed herein. Figure 1 . The computing device 1300 is capable of providing an augmented reality ("AR") environment or a virtual reality ("VR") environment. Generally speaking, an AR environment superimposes a computer-generated image ("CG") on a user's view of a real-world environment. For example, the computing device 1300 is capable of generating a synthetic view so that a user can visually perceive a computer-generated image superimposed on a rendering of a real-world environment 112, wherein the rendering of the real-world environment 112 is created by a camera 105 pointing to the real-world environment (such as a room). In some embodiments, the computing device 1300 is capable of generating a synthetic view so that a user can visually perceive a computer-generated image superimposed on a direct view of the real-world environment 112. Therefore, the computing device 1300 may have a prism or other optical device that allows a user to see a direct view of a real-world object or a real-world environment through the optical device, while the computer-generated image is superimposed on the view of the real-world object. The AR environment can also be referred to herein as a mixed reality ("MR") environment. MR devices can provide both AR and VR environments. The VR environment includes computer-generated virtual environments and images of virtual objects. MR and AR environments can utilize depth map sensors to determine the distance between the device and real-world objects. This allows the computer to scale and position computer-generated graphics on real-world objects in a realistic manner.

[0121] exist Fig.13 In the example shown, the optical system 1302 includes an illumination engine 1304 for generating electromagnetic ("EM") radiation including both a first bandwidth for generating CG images and a second bandwidth for tracking physical objects. The first bandwidth may include some or all of the visible light portion of the EM spectrum, while the second bandwidth may include any portion of the EM spectrum suitable for deploying a desired tracking protocol. In this example, the optical system 1302 also includes an optical assembly 1306 positioned to receive the EM radiation from the illumination engine 1304 and direct the EM radiation (or a single bandwidth thereof) along one or more predetermined optical paths.

[0122] For example, illumination engine 1304 can emit EM radiation along a common optical path shared by both the first bandwidth and the second bandwidth into optical assembly 1306. Optical assembly 1306 can also include one or more optical components configured to separate the first bandwidth from the second bandwidth (e.g., by causing the first bandwidth and the second bandwidth to propagate along different image generation and object tracking optical paths, respectively).

[0123] In some instances, the user experience depends on the computing device 1300 accurately identifying the characteristics of the physical object 103 ("real-world object") or plane (such as a real-world floor), and then generating a CG image based on these identified characteristics. For example, assume that the computing device 1300 is programmed to generate a user perception that the virtual game character is running towards and ultimately jumping over a real-world structure. To achieve this user perception, the computing device 1300 can obtain detailed data defining the characteristics of the real-world environment 112 surrounding the computing device 1300. To provide this functionality, in some embodiments, the optical system 1302 of the computing device 1300 may include a laser line projector and a differential imaging camera (neither of which is shown in FIG. 1 ). Fig.13 ).

[0124] In some examples, computing device 1300 utilizes optical system 1302 to generate a synthetic view that includes one or more CG images and a view of at least a portion of real-world environment 112 (e.g., from the perspective of a user wearing computing device 1300). For example, optical system 1302 may utilize various techniques (such as AR techniques) to generate a synthetic view that includes a CG image superimposed on a real-world view. As such, optical system 1302 may be configured to generate a CG image via optical assembly 1306 that includes display panel 1314.

[0125] In the illustrated example, the display panel includes separate right-eye and left-eye transparent display panels, labeled 1314R and 1314L, respectively. In some examples, the display panel 1314 includes a single transparent display panel that can be viewed with both eyes or a single transparent display panel that can only be viewed by a single eye. Therefore, it can be appreciated that the technology described herein can be deployed in a monocular device (e.g., a GOOGLE GLASS AR device) and in a binocular device (e.g., a MICROSOFT HOLOLENS AR device).

[0126] Light received from the real-world environment 112 passes through the see-through display panel 1314 to one or both eyes of the user. Graphics content is calculated by the image generation engine 1326 executing on the processing unit 1320 and displayed by the right-eye and left-eye display panels (when configured as transparent display panels), and can be used to visually enhance or otherwise modify the real-world environment 112 viewed by the user through the transparent display panel 1314. In this configuration, the user is able to view virtual objects 104 that are not present in the real-world environment 112 while the user is viewing physical objects 103 within the real-world environment 112. This creates an illusion or appearance based on light effects that the virtual objects 104 are physical objects 103 or physically present within the real-world environment 112.

[0127] In some examples, display panel 1314 is a waveguide display that includes one or more diffractive optical elements ("DOEs") for coupling incident light input into the waveguide, expanding the incident light in one or more directions for exiting the through-aperture expansion, and / or coupling incident light output out of the waveguide (e.g., toward the user's eyes). In some examples, computing device 1300 also includes additional transparent optical components, such as Fig.13 13 is shown in the form of a transparent veil 1316, which is located between the real world environment 112 and the display panel 1314. It can be appreciated that the transparent veil 1316 can be included in the computing device 1300 for purely aesthetic and / or protective purposes.

[0128] The computing device 1300 may also include various other components (in Fig.13 1300 ), for example, a front-facing camera (e.g., a red / green / blue (“RGB”), black and white (“B&W”), or infrared (“IR”) camera), a speaker, a microphone, an accelerometer, a gyroscope, a magnetometer, a temperature sensor, a touch sensor, a biometric sensor, other image sensors, an energy storage component (e.g., a battery), a communication facility, a global positioning system (“GPS”), a receiver, a laser line projector, a differential imaging camera, and possibly other types of sensors. Data obtained from one or more sensors 1308 (some of which are identified above) can be used to determine the orientation, position, and motion of the computing device 1300. As discussed above, data obtained from the differential imaging camera and the laser line projector or other types of sensors can also be used to generate a 3D depth map of the surrounding real-world environment 112.

[0129] In the illustrated example, the computing device 1300 includes one or more logic devices and one or more computer memory devices that store instructions that can be executed by the (one or more) logic devices to implement the functionality disclosed herein. Specifically, the controller 1318 can include one or more processing units 1320, one or more computer readable media 1322 for storing an operating system 1324, and an image generation engine 1326 and a terrain mapping engine 1328, as well as other programs (such as a 3D depth map generation module, which is configured to generate depth map data ("mesh data") and data in the manner disclosed herein.

[0130] In some implementations, computing device 1300 is configured to analyze data obtained by sensor 1308 to perform feature-based tracking of the orientation of computing device 1300. For example, where the object data includes an indication of a stationary physical object 103 (e.g., an engine) within real-world environment 112, computing device 1300 may monitor the position of the stationary object within a terrain mapping field of view (“FOV”). Then, based on the change in position of the stationary object within the terrain mapping FOV and the depth of the stationary object from computing device 1300, a terrain mapping engine executing on processing unit 1320AR may calculate a change in the orientation of computing device 1300.

[0131] It can be appreciated that these feature-based tracking techniques can be used to monitor changes in the orientation of computing device 1300 for monitoring the orientation of the user's head (e.g., assuming computing device 1300 is properly worn by user 102). The calculated orientation of computing device 1300 can be used in a variety of ways, some of which are described above.

[0132] Processing unit(s) 1320 can represent, for example, a central processing unit ("CPU") type processor, a graphics processing unit ("GPU") type processing unit, an FPGA, one or more digital signal processors ("DSP"), or other hardware logic components that can be driven by a CPU in some cases. For example, but not limited to, illustrative types of hardware logic components that can be used include: ASIC, application specific standard product ("ASSP"), system on chip (SOC), complex programmable logic device ("CPLD"), etc. Controller 1318 can also include one or more computer readable media 1322, such as the computer readable media described above.

[0133] It should be appreciated that, unless otherwise specifically stated, conditional language used herein, such as "can," "may," "might," or "could," is understood in context to mean that certain examples include and other examples do not include certain features, elements, and / or steps. Thus, such conditional language is generally not meant to imply that one or more examples require certain features, elements, and / or steps in any way, or that one or more examples must include a method for determining whether certain features, elements, and / or steps are included or to be performed in any particular example, with or without user input or prompting. Unless otherwise expressly stated, conjunctions such as phrases "at least one of X, Y, or Z" should be understood to mean that an item, term, or the like, may be X, Y, or Z, or a combination thereof.

[0134] It should also be understood that many changes and modifications may be made to the above examples, and that the elements of these examples are understood to be in other acceptable examples. All such modifications and variations are intended to be included within the scope of this disclosure and protected by the accompanying claims.

[0135] Finally, although various configurations have been described using language specific to structural features and / or methodological acts, it should be understood that the subject matter defined in the appended representations is not necessarily limited to the specific features or acts described. Rather, the specific features and acts are disclosed as exemplary forms of implementing the claimed subject matter.

Claims

1. A method for managing data associated with real-world objects and virtual objects displayed in a communication session, comprising: receiving, at a computing device, sensor data generated by a depth map sensor and an image sensor, the sensor data comprising image data of a real-world object and depth map data, the depth map data defining a three-dimensional model of the real-world object located within a real-world environment; receiving model data defining a virtual object for display within a view of the real-world object and the real-world environment; generating data defining a plurality of agent instances, individual agent instances having associated databases for individual virtual objects and individual real-world objects of interest identified by analysis of the model data, the depth map data, and the image data; monitoring communication data of a plurality of users of the communication session for identifying keywords, descriptions or parameters to be stored in an individual database of the individual agent instance associated with the individual virtual object or the individual real-world object of interest; Based on the keywords, descriptions, or parameters, a display of one or more recommendations associated with a rendering of the individual virtual object and a view of at least one real-world object is generated, wherein the one or more recommendations include at least one of: a modification to the at least one virtual object including at least one new parameter, a new virtual object positioned above the at least one real-world object, or an annotation recommending a modification to the at least one real-world object, wherein data defining the one or more recommendations and the keywords, descriptions, or parameters are permanently stored in an individual database of the individual agent instance associated with the individual virtual object or the individual real-world object of interest for access by multiple communication sessions.

2. The method according to claim 1, wherein: The individual real-world object of interest is identified by: determining an interactivity level of an input signal received from an input device relative to the at least one of the real-world objects; determining that the level of interaction exceeds a threshold; as well as The at least one real-world object is determined to be one of the individual real-world objects of interest in response to determining that the level of interaction exceeds the threshold.

3. The method according to claim 1, wherein: The individual real-world object of interest is selected by: determining that a level of movement of the at least one of the real-world objects exceeds a threshold; The at least one real-world object is determined to be one of the individual real-world objects of interest in response to determining that the level of movement of the at least one real-world object exceeds the threshold.

4. The method according to claim 1, wherein: The individual real-world object of interest is selected by the following operations: determining that a level of change in at least one of temperature, color, or shape relative to the at least one real world exceeds a threshold; The at least one real-world object is determined to be one of the individual real-world objects of interest in response to determining that the level of change with respect to at least one of temperature, color, or shape exceeds the threshold.

5. The method according to claim 1, wherein: The individual real-world object of interest is selected by: determining that a physical property of the at least one real-world object matches one or more predetermined physical properties; The at least one real-world object is determined to be one of the individual real-world objects of interest in response to determining that the physical attribute of the at least one real-world object matches the one or more predetermined physical attributes.

6. The method according to claim 1, further comprising: generating a query using the keywords, descriptions or parameters associated with the individual virtual object or the individual real-world object of interest; sending the query to a computing resource configured to receive the query and configured to return supplemental data based on the query; obtaining the supplemental data based on the query from the computing resource; as well as The supplemental data is provided within the one or more recommendations associated with the rendering of the individual virtual object and the view of at least one real-world object.

7. The method according to claim 1, further comprising: generating a query using specifications stored in a database associated with the at least one real-world object; sending the query to a computing resource configured to receive the query and configured to return supplemental data based on the query; obtaining the supplemental data based on the query from the computing resource, wherein the supplemental data defines a model of a new object that satisfies the specification requirements; and The supplemental data is provided within the one or more recommendations, wherein the supplemental data causes display of a rendering of the new object, wherein the new object is located over at least a portion of the at least one real-world object.

8. The method according to claim 1, further comprising: Associating priorities with individual recommendations, wherein the priorities associated with the individual recommendations increase as the number of user interactions referencing the recommendations increases, and wherein the display of the one or more recommendations includes a ranked list of the individual recommendations based on the individual priorities assigned to each of the individual recommendations.

9. A system for managing data associated with real-world objects and virtual objects displayed in a communication session, comprising: one or more data processing units; as well as A computer-readable medium having computer-executable instructions encoded thereon to cause the one or more data processing units to: receiving, at the system, sensor data generated by a depth map sensor, the sensor data comprising depth map data defining a three-dimensional model of a real-world object within a real-world environment; receiving model data defining a virtual object for display within a view of the real-world environment; generating data defining a plurality of agent instances, individual agent instances having a database associated with at least one real-world object identified by analysis of the depth map data; monitoring user interaction data of a plurality of users of the communication session from one or more input devices for identifying parameters to be stored within individual databases of the individual agent instances associated with individual real-world objects; Based on the parameters, a display of one or more recommendations associated with a rendering of the individual virtual object and a view of at least one real-world object is generated, wherein the one or more recommendations include displaying a modified image for confirming at least a portion of the at least one real-world object having the parameters.

10. The system according to claim 9, wherein: Data defining the one or more recommendations is persistently stored for access across multiple communication sessions.

11. The system according to claim 9, wherein: The proxy instance and the database are generated during a first communication session associated with the plurality of users, wherein the instructions further cause the one or more data processing units to: storing data defining the one or more recommendations within the database associated with the at least one real-world object; and determining that a second communication session has begun; In response to determining that a second communication session has begun, proxy data defining the proxy instance and the database is transmitted to one or more computing devices associated with the second communication session.

12. The system according to claim 9, wherein: The proxy instance and the database are generated during a first communication session associated with the plurality of users, wherein the instructions further cause the one or more data processing units to: determining that at least one user of the second communication session referenced the at least one real-world object; In response to determining that at least one user of the second communication session referenced the at least one real-world object, proxy data defining the proxy instance and the database is transmitted to one or more computing devices associated with the second communication session.

13. The system according to claim 9, wherein: The instructions further cause the one or more data processing units to: determining that a level of change in at least one of temperature, color, or shape relative to the at least one real world exceeds a threshold; and In response to determining that the level of change with respect to at least one of temperature, color, or shape exceeds the threshold, data defining the agent instance and the database associated with the at least one real-world object is generated.

14. The system according to claim 9, wherein: The instructions further cause the one or more data processing units to: determining that a physical property of the at least one real-world object matches one or more predetermined physical properties; and In response to determining that the physical properties of the at least one real-world object match the one or more predetermined physical properties, data defining the proxy instance and the database associated with the at least one real-world object is generated.

15. The system according to claim 9, wherein: The instructions further cause the one or more data processing units to: determining that a level of movement of the at least one of the real-world objects exceeds a threshold; and In response to determining that the level of movement of the at least one real-world object exceeds the threshold, data defining the agent instance and the database associated with the at least one real-world object is generated.

Images