Augmented technology for tracking the movement of real-world objects to improve virtual object positioning
By tracking the position of real-world objects through remote device sensors and maintaining the association between virtual and real objects across devices, the problem of inaccurate positioning of virtual objects when real objects move out of view in existing systems is solved, achieving more efficient resource utilization and accurate display of mixed reality environments.
Patent Information
- Application Number
- CN201980079950.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2018-12-06
- Filing Date
- 2019-11-29
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2039-11-29
AI Technical Summary
Existing mixed reality systems struggle to accurately track the relationship between virtual and real objects when real-world objects move out of the device's field of view. This leads to the separation of virtual and real objects, disrupting the precise placement of objects in the mixed reality environment, especially in real-time conferencing or multi-user communication sessions.
By tracking the location of real-world objects through sensors on remote devices, identifying real-world objects using data shared by the first device, and maintaining the association between virtual and real objects on the second device, cross-device location tracking and rendering are achieved, and shared data is stored to maintain the association in different communication sessions.
It expands the device coverage, reduces user input, improves the efficiency of computing resource utilization, reduces the possibility of unintentional input and errors, and ensures the accurate positioning of virtual objects during the movement of real objects.
Smart Images

Figure CN113168231B_ABST
Abstract
Description
Background Art
[0001] Mixed reality displays present computer-generated virtual objects that can "augment" the user's view of the real-world environment around them. For example, a user can place a three-dimensional virtual object (such as a cup) to appear as if the virtual object is sitting on a "physical" object in the real world (such as a table).
[0002] Although some existing computing devices allow users to place virtual objects on or near physical objects in a real-world environment, the functionality that enables users to interact with such virtual objects may be limited. For example, in the above example involving a virtual cup being positioned to appear on a table in the real world, not all systems can manage the relationship between the cup and the table when the table moves. When using a specific device (such as a head-mounted device) to track the movement of the table, many errors may occur when the table moves out of the field of view of the head-mounted device. In this case, the virtual object may become separated from the real-world object, and the computer may lose the ability to display the virtual object in the correct orientation or in the correct location. Such a result may completely undermine the purpose of the mixed reality environment, which is to place items in a precise manner to enhance the view of the real world. This problem is even more serious when the mixed reality environment is displayed during a communication session such as a real-time meeting or a communication broadcast to many users. Summary of the invention
[0003] The present disclosure provides an enhanced technology for tracking the movement of real-world objects to improve the positioning of virtual objects shared within a collaborative environment. The present disclosure provides at least two improvements to existing systems. In the first aspect, the present disclosure enables a device to use a sensor of a remote device to track the position of a real-world object that has moved out of the viewing area of the device. When the real-world object moves out of the viewing area of the first device, when the object enters the viewing area of the second device, the second device can use the data shared by the first device to identify the real-world object. The shared data can describe the physical properties of the real-world object and can also define any associated virtual objects. Then, even when the real-world object has moved out of the viewing area of the first device, the second device can maintain the association between the real-world object and the associated virtual object. Such technology can expand the coverage of the device that is tracking the movement of the real-world object.
[0004] In another aspect, by storing shared data in a persistent manner, the system can maintain associations between real-world objects and virtual objects between different communication sessions, such as online meetings or broadcasts. For example, in a group communication session using Skype, a first device can determine physical properties of a real-world object. Data defining the physical properties and other data defining the virtual object can be stored during or at the end of the communication session. When a new communication session is instantiated, such as a private chat, the device can use the stored data to identify a real-world object that matches the physical properties and then display the associated virtual object created in the previous communication session.
[0005] The technology disclosed herein provides many features that improve existing computers. For example, since the computer can track the location of one or more objects without user input, computing resources such as processor cycles, memory, network bandwidth, and power can be used more efficiently. The reduction of user input can reduce the possibility of unintentional input and other related errors (which can lead to low user productivity), as well as reduce the use of computing resources for such entries and corrections to unintentional entries. Other technical benefits not specifically mentioned herein can also be achieved through the embodiments of the disclosed subject matter.
[0006] Those skilled in the art will also appreciate that aspects of the subject matter described herein may be practiced on or in conjunction with other computer system configurations than those specifically described herein, including: multi-processor systems, microprocessor-based or programmable consumer electronics, AR, VR and MR devices, video gaming devices, handheld computers, smart phones, smart TVs, self-driving cars, smart watches, e-readers, tablet computing devices, dedicated hardware devices, networked appliances, and the like.
[0007] By reading the following detailed description and browsing the associated drawings, in addition to the functions and technical advantages explicitly described above, other functions and technical advantages will also be apparent. This disclosure is provided to introduce the selection of design concepts further described below in the detailed description in a simplified form. This disclosure is not intended to identify the key or important features of the claimed subject matter, nor is it intended to assist in determining the scope of the claimed subject matter. For example, the term "technology" may refer to systems, methods, computer-readable instructions, modules, algorithms, hardware logic units, and / or operations as allowed by the above context and the entire document. BRIEF DESCRIPTION OF THE DRAWINGS
[0008] The detailed description is described with reference to the accompanying drawings. In the drawings, the leftmost digit of a reference number identifies the drawing in which the reference number first appears. The same reference number indicates similar or identical items in different drawings. References to individual items of a plurality of items may use reference numbers with letters in the alphabetical sequence to refer to each individual item. General references to items may use specific reference numbers without the alphabetical sequence.
[0009] Figure 1 An example scenario involving a computing system that can be used to implement the techniques disclosed herein is shown.
[0010] Figure 2 A view displayed to a user using a computing device for detecting characteristics of a real-world object in a real-world environment is shown.
[0011] Figure 3 A view displayed to a user using a computing device for tracking movement of a real-world object in a real-world environment from a first location is shown.
[0012] Figure 4 A view displayed to a user using a computing device for detecting movement of a real-world object in a real-world environment to a second room location is shown.
[0013] Figure 5 A view displayed to a user using a computing device for tracking movement of a real-world object in a real-world environment from a first room to a second room is shown.
[0014] Figure 6 A view displayed to a user using a computing device for tracking movement of a real-world object is shown, the view showing a scene in which a wall blocks a direct view of the real-world object.
[0015] Figure 7 A view displayed to a user using a remote computing device for tracking movement of a real-world object is shown, where tracking involves detection of one or more characteristics of the real-world object.
[0016] Figure 8 A view displayed to a user using a remote computing device for tracking movement of a real-world object is shown, where the remote computing device shares metadata with another computing device to coordinate tracking of the real-world object.
[0017] Fig. 9 A view displayed to a user using a computing device to track the movement of a real-world object is shown, wherein the computing device displays a rendering based on metadata shared from a remote computer that tracks the position of the real-world object.
[0018] Fig.10is a flow chart illustrating aspects of a routine for a computationally efficient process for tracking the movement of real-world objects displayed in a mixed reality and virtual reality collaborative environment.
[0019] Fig.11 is a computing system diagram showing aspects of an illustrative operating environment for the techniques disclosed herein.
[0020] Fig.12 is a computing architecture diagram illustrating various aspects of the configuration and operation of a computing device that may implement various aspects of the techniques disclosed herein.
[0021] Fig.13 is a computing device diagram illustrating various aspects of the configuration and operation of an AR device that may implement various aspects of the disclosed technology, according to one embodiment disclosed herein. DETAILED DESCRIPTION
[0022] Figure 1 An example scenario involving a computing system 100 that can be used to implement the techniques disclosed herein is shown. The computing system 100 may include a first computer 104A and a second computer 101B. In this case, the first computing device 101 records aspects of a real-world object 110 within a viewing area 121A of the first computing device 101A. In this illustrative example, the real-world object 110 is a table. One or more sensors may be used to measure and record metadata 105 that defines characteristics of the table, such as table size, color, texture, position, etc.
[0023] The first computing device 101A may also associate the virtual object 111 with the real-world object 110 and generate association data 106 defining a plurality of associations between the virtual object 111 and the real-world object 110. For example, the association data 106 may define the positions of the virtual object 111 and the real-world object 110. The data defining the positions may be used to define relative positions, for example, the difference between the positions of each item. Other metadata 105 may also define the associations between the virtual object 111 and the real-world object 110. For example, image data 108 (also referred to herein as “image 108”) may be generated by an imaging device (e.g., a camera), and the image data 108 may be utilized by one or more computing devices 101 to identify and generate data defining a graphical association between two items, for example, one object is located on top of another item, next to another item, and so on. For purposes of illustration, the terms “object” and “item” may refer to either a real-world object or a virtual object.
[0024] The first computing device 101A may also render a virtual object 111 by using the model data 109, which defines parameters of the virtual object 111, such as size, shape, texture, or color. The virtual object 111 may be rendered in a specific position relative to the real-world object 110. As shown, the virtual object 111 (a cup in this example) is positioned to have the following appearance: the virtual object 111 is placed on the top surface of the real-world object 110 (the table). This example is provided for illustrative purposes and should not be construed as limiting. Any virtual object 111 and any real-world object 110 may be utilized by the techniques disclosed herein and displayed in any relative position relative to each other.
[0025] One or more sensors of computing device 101A may be used to track the movement of real-world object 110 and modify the position of virtual object 111 based on the new position of real-world object 110. Thus, if the table moves in a particular direction, the position of the cup follows the movement of the table to maintain a graphical association between the two items, e.g., the cup is on top of the table.
[0026] The techniques disclosed herein enable multiple computing devices to track the movement of a real-world object 110 even when the particular real-world object 110 is outside of the viewing area of the particular computer, such as no longer within the viewing area of the particular computer. Figure 1 Consider an example scenario in which a real-world object table moves from a first room ("Room 1") to a second room ("Room 2"). In this case, as real-world object 110 moves from viewing area 121A of first computing device 101A to a new location outside of viewing area 121A of first computing device 101A, first computing device 101A may coordinate with second computing device 101B to continue tracking the movement of real-world object 110. To facilitate coordination, metadata 105 including association data 106, one or more images 108 of the object, and / or model data 109 are transmitted from first computing device 101A to second computing device 101B. In some embodiments, metadata 105 may define one or more characteristics of the real-world object detected by first computing device 101A. As will be described in more detail below, association data 106 may define an association between a virtual object and a real-world object, as well as the locations of the virtual object and the real-world object. Model data 109 may define a virtual object to be displayed simultaneously with the display of the real-world object.
[0027] Second computing device 101B may then utilize metadata 105 to identify real-world object 110, such as a table, within viewing area 121B of second computing device 101B. In one illustrative example, second computing device 101B may measure physical characteristics of real-world object 110, such as size, one or more dimensions, texture, color, and / or shape. Any measurements detected from sensors directed toward repositioned real-world object 110 may be used to determine that the real-world object within viewing area 121B of second computing device 101B is the same real-world object viewed by first computing device 101A.
[0028] If the measurements generated by each computer are determined to be within a threshold difference, then it can be determined that a particular real-world object viewed by the second computing device 101B matches the real-world object viewed by the first computing device 101A. In some configurations, a confidence score can be determined based on the difference between the measurements collected by the two computing devices. For example, the first computing device 101A can determine that the table in its corresponding viewing area 121A is 15.1 inches high, while the second computing device 101B can determine that the table in its corresponding viewing area 121B is 14 inches high. If the two measurements are within a threshold difference, then one or more computing devices can determine that the perspectives of the two measurements match.
[0029] In some embodiments, a confidence score can be generated based on the two measurements of each device combined. The confidence score can be combined with other confidence scores (e.g., confidence scores for differences in color, shape, and / or other physical properties), and the combination of confidence scores can be compared to a threshold. If the combined confidence score exceeds the threshold, a match can be determined.
[0030] The confidence score may also be based on the location of the object detected by each computing device. In one illustrative example, consider a scenario in which a first computing device 101A generates data indicating the last known location of a real-world object. Then, after the object has moved, a second computing device 101B detects the presence of the real-world object and generates data indicating the first known location of the object. The confidence score may be generated based on the distance between the last known location of the first computing device and the first known location of the second computing device. If the confidence score exceeds a threshold, a match may be determined.
[0031] Once the second computing device 101B identifies the real-world object, e.g., determines a match, the metadata 105 may be used to render the virtual object 111 at a graphical location consistent with any association detected by the first computing device 101A. Figure 1As shown, virtual object 111 may be displayed by second computing device 101B, wherein the position of virtual object 111 relative to real-world object 110 is similar to the arrangement detected by first computing device 101A.
[0032] The first perspective view 131A displayed to the user of the first computing device 101A shows the view at a first time ("Time 1") before the table is moved from Room 1 to Room 2. The second perspective view 131B displayed to the user of the second computing device 101B shows the view at a second time ("Time 2") after the table is moved from Room 1 to Room 2. Figure 1 As shown, in this example, the second computing device 101B renders the cup on the right hand side of the table top near the right front leg, which is similar to the display shown at the first computer 101.
[0033] Figure 2 A view of a user 201 utilizing a first computing device 101A is shown. As shown, the first computing device 101 has a viewing area 121A for a real-world environment 112, such as a room having one or more real-world objects 110. One or more sensors may be used to generate data defining an image of the real-world environment 112. Using one or more display devices of the first computing device 101A, the user 201 may view a first perspective image 131A.
[0034] Depth map data (also referred to herein as "mesh data") may be generated by one or more sensors of the first computing device 101A. For example, a depth map sensor and / or a camera may be used to determine coordinates of various points on the surface of a real-world object. Based on these coordinates, measurements of the real-world object 110 (e.g., a table), such as height (H), width (W), and length (L), may be generated. If the real-world object moves to a location outside of the viewing area 121A of the first computing device 101A and enters the viewing area of other devices, these measurements, as well as other measurements defining one or more characteristics, may be shared with one or more devices to allow the other devices to recognize the real-world object 110. The depth map data may be stored in the associated data 106, or in a separate package of the metadata 105.
[0035] Reference now Figure 3-Figure 9 , the following description illustrates several stages of the tracking process performed by the viewing angle 131 of each computing device 101. Figure 3As shown, a first computing device 101A may detect a position of a real-world object 110 (e.g., a table) at a first location in a real-world environment (e.g., Room 1). In this example, the first computing device receives model data defining a virtual object 111 (e.g., a cup) that is positioned so that it appears to be on top of the table.
[0036] The virtual object is associated with the table so that when the table moves, the cup moves in a corresponding manner. Figure 4 As shown, when the table physically moves to the right, the cup moves with the table, so that the cup remains in the same position relative to the table, such as at the top. Figure 5 As shown, as the table moves out of the viewing area of the first computing device 101A, the sensor of the first computing device 101A may not be able to determine the position of the real-world object 110. In this case, the rendering of the virtual object 111 may not accurately depict the graphical association between the two items.
[0037] like Figure 6 As shown, when the table is outside the viewing area 121A of the first computing device 101A, the first computing device 101A cannot accurately determine the position of the table and therefore cannot accurately position the rendering of the virtual object 111. As the real-world object moves outside the viewing area 121A of the first computing device 101A to a threshold level, the first computing device 101A may generate event data indicating when the real-world object 110 is outside the viewing area 121A of the first computing device 101A.
[0038] The generation of event data may be based on a variety of factors. For example, when the image of real-world object 110 becomes distorted or blurred to a threshold level, first computing device 101A may determine that real-world object 110 is outside viewing area 121A. Thus, if real-world object 110 is partially covered by another object to a threshold level, first computing device 101A may determine that real-world object 110 is outside viewing area 121A. In this example, as the wall of the room begins to cover the real-world object until a threshold level is reached (e.g., one-quarter of a table, one-half of a table, or three-quarters of a table), first computing device 101A may determine that real-world object 110 is outside viewing area 121A of first computing device 101A. Such a determination may also be made when the view of the real-world object becomes too blurry or too small.
[0039] In response to determining that the real-world object 110 is outside the viewing area 121A of the first computing device 101A, the first computing device 101A may initiate communication of at least a portion of the metadata 105 to one or more remote computing devices (e.g., the second computing device 101B). The communication may include at least one of the associated data 106, the image 108, or the model data 109.
[0040] In some embodiments, predetermined actions within the communication session may cause the transfer of metadata from the first computing device 101A to one or more remote computers (e.g., the second computing device 101B). For example, if a user within the collaborative environment posts a comment referencing the virtual object 111 or the real-world object 110, the first computing device 101A may send the metadata to the one or more remote computers, such as the second computing device 101B. Thus, a user may say "I see the table," and in response, the system may send metadata 105 to the user's device. The metadata may be used to verify the identity of the real-world object.
[0041] The metadata 105 may be sent to a particular computing device based on other factors. In one illustrative example, the first computing device 101A may transmit the metadata 105 to any computing device within a predetermined distance or any computing device within a predetermined area. In such an embodiment, the first computing device 101A may use a location device such as a GPS device or a network device to determine the location of the first computing device 101A. The first computing device 101A may also receive location information from other computing devices such as the second computing device 101B. If a particular computer is determined to be within a predetermined distance and / or if a particular computer is determined to be within a predetermined area, which may include a geographic area or a specific area in a building, the metadata 105 may be sent to those computers that meet the criteria.
[0042] In other illustrative examples, first computing device 101A may use at least one of preference data, compatibility data, or performance data to select a remote computing device for receiving metadata 105. Preference data may include, for example, a list of computing devices established by a user or a computer. Figure 1 In the example shown, second computing device 101B may be on the preferences file of the first computing device. In response to determining that the preferences file includes second computing device 101B, first computing device 101A may transmit metadata 105 to the second computing device.
[0043] The compatibility data may indicate specifications of one or more remote devices. For example, a file on the first computing device may determine that metadata 105 may be transferred to any remote computer that includes a particular camera and a particular depth map sensor. In other illustrative examples, the first computing device may determine that metadata may be transferred to any remote computer that has a camera of a particular type, brand name, etc.
[0044] The performance data may indicate the performance of one or more remote computing devices. For example, a file on a first computing device may determine that metadata may be transmitted to any remote computer having a microprocessor that can manage a threshold level of operation for a period of time. In other examples, the first computing device may determine that metadata may be transmitted to any remote computing device having a camera capable of capturing images with a threshold level of resolution.
[0045] Once a remote computer, such as the second computing device 101B, receives the metadata 105 , the remote computer may utilize the metadata 105 and other contextual data to determine when the real-world object 110 enters within the viewing area 121B of the second computing device 101B.
[0046] like Figure 7 As shown, when real-world object 110 enters viewing area 121B of second computing device 101B, second computing device 101B may utilize sensor data to identify physical characteristics of real-world object 110 within viewing area 121B. Any suitable physical characteristics may be measured, such as the height (H), width (W), and length (L) of real-world object 110. The characteristics obtained by the second computing device may be used to determine whether the real-world object in viewing area 121B is the same, for example, whether it matches the real-world object observed in viewing area 121A of first computing device 101A.
[0047] As described above, one or more techniques for determining a match may be utilized. A value or score may be assigned to a physical characteristic of a real-world object. A match may be determined when a value of a physical characteristic of a real-world object within viewing area 121B of second computing device 101B is within a threshold difference of a value of at least one characteristic of the real-world object detected by a sensor of a remote computing device, e.g., a real-world object within viewing area 121A of first computing device 101A.
[0048] Once a match is detected, such as Figure 8As shown, virtual object 111 is rendered within viewing perspective 131B for display to a user of second computing device 101B. Second computing device can utilize association data 106 to determine a location for rendering virtual object 111, which can be at a specific location relative to the location of real-world object 110.
[0049] In one illustrative embodiment, the associated data 106 may define a specific point within the surface of the real-world object 110. For example, the associated data 106 may include coordinates defining a specific point on the top of a table (e.g., the center of the table). Additionally, the associated data 106 may indicate a point or surface of the virtual object 111. For example, the associated data may include coordinates defining the bottom of a cup. Such data may be utilized by the second computing device 101B to render the virtual object 111 and position it at a specific location relative to the real-world object 110.
[0050] This graphical association (also referred to herein as a “visual association”) between real-world object 110 and virtual object 111 may be maintained when real-world object 110 moves to different locations. For example, a table may be moved from a first location within room 2 to a second location within room 2. While real-world object 110 remains in the viewing area of the second computing device, the second computing device may move virtual object 111 to maintain the same graphical association between the objects regardless of the location of real-world object 110.
[0051] Also in Figure 8 , once the second computing device 101B identifies the real-world object 110, the second computing device 101B may send metadata 105 (e.g., updated metadata 105) back to the first computing device 101A. The updated metadata 105 may include updates to any of the associated data 106, the one or more images 108, and / or the model data 109. In one illustrative example, the model data 109 may include positioning data indicating the location of the virtual object 111 and / or the location of the real-world object 110. The positioning data may be in any suitable format, which may include using global positioning data (GPS) or other coordinate-based data formats to indicate the three-dimensional location of the virtual object 111 and / or the location of the real-world object 110. Updates may be sent to the first computing device 101A or other computing devices periodically. Updates may also be sent in response to specific actions, for example, the virtual object 111 has been moved or modified, or the real-world object 110 has been moved.
[0052] In a collaborative environment, if multiple computers are communicating with the second computing device 101B, updates to the metadata 105 may be sent to each computer in response to a particular action, such as a modification or move of the virtual object 111. Updates to the metadata 105 may also be sent to other computers in response to other user actions. For example, a computing device may monitor communications of a collaborative environment, such as a communication session in which multiple people are sharing a video stream, an audio stream, a text communication, etc. The communications may be analyzed to determine one or more keywords, such as the name of an object, a reference to an object, etc.
[0053] If a user within the collaborative environment posts a comment that references a virtual object 111 or a real-world object 110, updates to the metadata 105 may also be sent to other computers, including computers that send or receive communication data containing comments that reference the virtual object or the real-world object. In some embodiments, updates to the metadata 105 may also be shared with a computing device that manages the communication session 1104. In such an embodiment, the updated metadata 105 may be sent to any number of computing devices participating in the communication session. Updates to the metadata 105 may include text descriptions summarizing any relevant communications. Computers that receive the text descriptions may display these descriptions as annotations to the virtual object 111 or the real-world object 110.
[0054] Any computing device (e.g., first computing device 101A) that receives an update to metadata 105 may then render virtual object 111 according to the updated metadata. Fig. 9 As shown, a virtual object 111 (cup) is displayed within a first perspective view 131A of a first computing device 101A. In this example, the first computing device 101A also displays a graphical element 900 that includes aspects of the updated metadata 105. Specifically, in this example, the updated metadata 105 causes the first computing device 101A to display a graphical element 900 that indicates that the virtual cup is in the next room and that the virtual cup is a specific distance from the first computing device 101A. This example is provided for illustrative purposes and should not be construed as limiting. It will be appreciated that any information related to a virtual object or a real-world object may be transmitted to the first computing device 101A or any other computing device, and any annotations related to the object may be transmitted and displayed.
[0055] In some embodiments, the computing device receiving the updated metadata may also modify the display of the virtual object 111 to indicate a particular state, such as that the virtual object 111 is behind a wall or at a particular distance. Fig. 9As shown, the display of the virtual object 111 is slightly modified (e.g., rendered with a thinner outline) to indicate that the virtual object 111 is located at a certain distance or behind a real-world object (e.g., the wall of a building).
[0056] Also like Fig. 9 As shown, a computing device (e.g., first computing device 101A) that receives updated metadata 105 may also display a rendering 110' of real-world object 110. The rendering allows a user of a computing device (e.g., first computing device 101A) to see the location of real-world object 110 that has been moved out of the viewing area of first computing device 101A. Rendering 110' of the real-world object may be at a position and scale that represents the location of real-world object 110 detected by second computing device 101B. Rendering 110' may be modified (e.g., rendered with dashed lines) to indicate that the real-world object is blocked or otherwise obscured in some manner.
[0057] In another aspect, the system can maintain associations between real-world objects and virtual objects between different communication sessions, such as online meetings or broadcasts. For example, in a group communication session using Skype, a first device can determine physical properties of a real-world object. Data defining the physical properties and other metadata defining the virtual object can be stored during or at the end of the communication session. The metadata can be stored in a remote computer, such as a server that manages multiple communication sessions. When a new communication session is instantiated, such as a private chat, the device can use the stored data to identify a real-world object that matches the physical properties and then display the associated virtual object created in the previous communication session.
[0058] Fig.10 1 is a diagram illustrating aspects of a routine 1000 for computationally efficiently managing data associated with objects displayed within a mixed reality and virtual reality collaborative environment. It should be understood by those of ordinary skill in the art that the operations of the methods disclosed herein need not be presented in any particular order, and that it is possible and contemplated to perform some or all of the operations in an alternative order. For ease of description and illustration, the operations have been presented in the order in which they are presented. These operations may be added, omitted, performed together, and / or performed simultaneously without departing from the scope of the appended claims.
[0059] It should also be understood that the method shown can be terminated at any time and does not need to be performed as a whole. Some or all of the operations of the method and / or substantially equivalent operations can be performed by executing computer-readable instructions included on a computer storage medium, as defined herein. The term "computer-readable instructions" and its variants used in the specification and claims are widely used herein to include routines, applications, application modules, program modules, programs, components, data structures, algorithms, etc. Computer-readable instructions can be implemented on various system configurations, including single-processor or multi-processor systems, minicomputers, mainframe computers, personal computers, handheld computing devices, microprocessor-based programmable consumer electronics, combinations thereof, etc. Although the example routines described below operate on a computing device 101, which may refer to a first computing device 101A or a second computing device 101B, it is understood that the routines can be performed on any computing system that can include multiple computers that work together to perform the operations disclosed herein.
[0060] It should therefore be understood that the logical operations described herein are implemented as (1) a sequence of computer-implemented actions or program modules running on a computing system (such as those described herein) and / or (2) interconnected machine logic circuits or circuit modules within the computing system. The implementation is a matter of choice depending on the performance and other requirements of the computing system. Thus, the logical operations may be implemented in software, firmware, dedicated digital logic units, and any combination thereof.
[0061] In addition, the example user interface (UI) can be implemented in association with the above example. Fig.10 For example, the various devices and / or modules described herein can generate, send, receive, and / or display data associated with the content of a communication session (e.g., live content, broadcast events, recorded content, etc.) and / or present a UI that includes a rendering of one or more participants of a remote computing device, avatar, channel, chat session, video stream, image, virtual object, and / or application associated with the communication session.
[0062] The routine 1000 begins at operation 1002, where the computing device 101 receives sensor data that defines a 3D representation of a real-world environment 112. The sensor data may be captured by a depth map sensor, such as a depth map camera. Additionally, the sensor data may be captured by an image sensor, such as a camera, where the depth map sensor and the image sensor may be part of the same component or may be part of separate components. The sensor data includes depth map data that defines a three-dimensional model of the real-world environment and an image of the real-world environment. For example, the real-world environment may include the walls of a room and certain objects within the room, such as Figure 1The sensor data may define physical properties of a plurality of real-world objects within a real-world environment. The sensor data may also indicate the location of one or more objects in the environment. Thus, measurements of objects or measurements of the environment may be made by analysis of the sensor data. One or more objects defined in the sensor data may be shared with a plurality of users participating in the collaborative environment. The collaborative environment may include a communication session that allows a user to send, receive, and view various aspects of sensor data rendered on a display device.
[0063] The routine 1000 then proceeds to operation 1002, where the computing device 101 receives model data defining one or more virtual objects to be displayed within a view of the collaborative environment. The model data may define specific locations where the virtual objects are placed within the user interface of the collaborative environment.
[0064] At operation 1006, computing device 101 may identify at least one characteristic of the real-world object. In some embodiments, operation 1006 may include identifying a physical feature of the real-world object or measuring the real-world object. The characteristic (e.g., shape, size, texture, color) or any other identifiable characteristic may be captured by one or more sensors of the computing device. The characteristic may also include other markings, such as handwritten text, symbols, bar codes, etc.
[0065] Next, computing device 101 may transmit metadata 105 defining the characteristics to one or more remote computers at operation 1008. The characteristics of the real-world object may be stored in associated data 106. Associated data 106 may enable remote computers to easily identify the real-world object by using the characteristics.
[0066] In some configurations, metadata 105 may be sent to a remote computer in response to a particular event. For example, when a real-world object is moved out of viewing area 121A of computing device 101, or if the view of the real-world object is at least partially obstructed, metadata 105 may be transmitted to other computing devices.
[0067] In some configurations, metadata 105 may be sent to the remote computer in response to determining that the remote computer meets one or more criteria regarding hardware compatibility. In such an embodiment, the first computing device 101A may receive context data indicating specifications of one or more components (i.e., sensors) of the remote computing device. For example, the second computing device 101B and other computers may send context data indicating that the second computing device 101B has a sensor with a particular resolution, or a camera with a particular light sensitivity, or a particular model of depth map sensor.
[0068] When the specifications (hardware specifications or performance specifications) of one or more components of the remote computing device meet one or more criteria, the first computing device 101A can select the remote computing device from the plurality of computing devices. Then, the first computing device 101A can transmit metadata defining at least one characteristic of the real-world object to the specific remote computing device having one or more components that meet the criteria. This embodiment enables efficient communication of metadata and only sends metadata to computing devices having sensors or other components that are capable of performing the techniques disclosed herein.
[0069] Next, at operation 1010, computing device 101 (e.g., second computing device 101B) may receive metadata 105 and identify a real-world object. As described herein, second computing device 101B may identify the real-world object as a match to a real-world object observed by first computing device 101A if one or more characteristics match within a threshold level.
[0070] In some embodiments, computing device 101 may determine that a first value of a physical characteristic of a real-world object within a viewing area is within a threshold difference of a second value of at least one physical characteristic of the real-world object observed by a sensor of a remote computing device. In such embodiments, first computing device 101A may associate a first value (e.g., a first score) with one or more characteristics (e.g., color, size, measurement, texture, etc.). The first value may be based on data generated by one or more sensors (of the first computing device) that captured an image or depth map image of the object before it was moved from its original position, such as Figure 1 shown.
[0071] After moving the object into the viewing area of the second computing device 101B (e.g., the viewing area of the sensor of the second computing device 101B), the second computing device may associate a second value (e.g., a second score) with one or more characteristics (e.g., color, size, measurement, etc.). The second value may be based on data generated by one or more sensors (of the second computing device) that capture an image or depth map image of the object after it has been moved from its original position, such as Figure 1 shown.
[0072] Any suitable value or score may be associated with a characteristic. For example, each color in a color spectrum may be associated with a score. A light red may be associated with a score of 1, a dark red may be associated with a score of 2, a light green may be associated with a score of 5, and a dark green may be associated with a score of 6. Thus, as each computing device interprets a view of a real-world object, they may use such values to determine whether a characteristic matches within a threshold. Thus, even if a second computing device identifies a color as light green and a first computing device identifies a color as dark green, the two computing devices may identify a match according to the techniques disclosed herein. Other characteristics, such as length, texture, and other measurements, may be scored in a similar manner and other corresponding thresholds may be used.
[0073] The computing device may compare the first value to the second value. In response to determining that the first value and the second value are within a threshold difference, the second computing device may take additional actions. For example, the second computing device may render the virtual object at a location on the display device relative to the display of the real-world object within the viewing area of the second computing device 101B. As another example, the second computing device 101B may begin tracking the position or movement of the real-world object.
[0074] At operation 1012, a computing device (e.g., second computing device 101B) may track the movement of a real-world object. As shown in the figure and described above, second computing device 101B may render a virtual object at a specific position relative to the real-world object based on the metadata. Second computing device 101B may track the movement of the real-world object and move the position of the virtual object accordingly to maintain a graphical association between the two items.
[0075] At operation 1014, the computing device (e.g., second computing device 101B) and updated metadata 105 may be transmitted to other computers including first computing device 101A. The updated metadata 105 may allow other computers to display renderings of virtual objects using updated locations and / or augmentations made to the virtual objects. The updated metadata 105 may also enable the computing device to display annotations or comments made about the virtual objects. At operation 1016, the updated metadata may also be persistently stored to enable one or more computing devices to utilize the updated metadata and other communication sessions, with the routine 1000 terminating thereafter.
[0076] It should be understood that the above subject matter can be implemented as a computer-controlled device, a computer process, a computing system, or an article such as a computer-readable storage medium. The operations of the example methods are shown in separate boxes and are summarized with reference to those boxes. These methods are shown as a logical flow of boxes, each of which can represent one or more operations that can be implemented with hardware, software, or a combination thereof. In the context of software, an operation represents a computer-executable instruction stored on one or more computer-readable media, which, when executed by one or more processors, enables the one or more processors to perform the operations described.
[0077] Typically, computer executable instructions include routines, programs, objects, modules, components, data structures, etc. that perform specific functions or implement specific abstract data types. The order in which the operations are described should not be construed as a limitation, and any number of the described operations may be performed in any order, combined in any order, subdivided into multiple sub-operations and / or performed in parallel to implement the described process. It may be performed by resources associated with one or more devices (e.g., one or more internal or external CPUs or GPUs) and / or one or more hardware logic units (e.g., field programmable gate arrays ("FPGAs"), digital signal processors ("DSPs"), or other types of accelerators).
[0078] All methods and processes described above may be embodied and fully automated by software code modules executed by one or more general purpose computers or processors. The code modules may be stored in any type of computer readable storage medium or other computer storage device, such as those described below. Some or all of these methods may alternatively be embodied in dedicated computer hardware, as described below.
[0079] Any conventional description, element or box in the flowcharts described herein and / or depicted in the accompanying drawings should be understood to potentially represent a module, segment or portion of code, which includes one or more executable instructions for implementing specific logical functions or elements in a routine. Alternative implementations are included within the scope of the examples described herein, in which elements or functions can be deleted or executed out of order from the order shown or discussed, including execution in substantially synchronous or reverse order, depending on the functions involved as can be understood by those skilled in the art.
[0080] Fig.11 1104 . 1104 . 1105 . 1106 . 1107 . 1108 . 1109 . 1110A . 1111B . 1112C . 1114 . 1115 . 1116 . 1117 . 1118 . 1119 . 1120 . 1121 . 1122 . 1123 . 1124 . 1125 . 1126 . 1127 . 1128 . 1129 . 1130
[0081] As shown, a communication session 1104 may be implemented between a plurality of client computing devices 1106(1) through 1106(N) associated with or part of the system 1102 (where N is a number having a value of 2 or greater). The client computing devices 1106(1) through 1106(N) enable users (also referred to as individuals) to participate in the communication session 1104. For example, the first client computing device 1106(1) may be Figure 1 The computing device 101 or Fig.13 One of the computing devices 1300.
[0082] In this example, communication session 1104 is hosted by system 1102 over one or more networks 1108. That is, system 1102 may provide services that enable users of client computing devices 1106(1)-1106(N) to participate in communication session 1104 (e.g., via real-time viewing and / or recorded viewing). Thus, "participants" of communication session 1104 may include users and / or client computing devices (e.g., multiple users may be in a room participating in a communication session via the use of a single client computing device), and each participant may communicate with other participants. Alternatively, communication session 1104 may be hosted by one of client computing devices 1106(1)-1106(N) using peer-to-peer technology. System 1102 may also host chat conversations and other team collaboration functionality (e.g., as part of an application suite).
[0083] In some embodiments, such chat conversations and other team collaboration functions are considered to be external communication sessions that are distinct from the communication session 1104. A computerized agent configured to collect participant data in the communication session 1104 can be linked to such an external communication session. Thus, the computerized agent can receive information such as date, time, session-specific information, etc. that enables connection to such an external communication session. In an example, a chat conversation can be conducted in accordance with the communication session 1104. In addition, the system 1102 can host a communication session 1104 that includes at least a plurality of participants co-located at a meeting location (e.g., a conference room or auditorium) or located at different locations.
[0084] In the examples described herein, client computing devices 1106(1) to 1106(N) participating in a communication session 1104 are configured to receive and render communication data for display on a user interface of a display screen. The communication data may include a collection of various instances or streams of real-time content and / or recorded content. The collection of various instances or streams of real-time content and / or recorded content may be provided by one or more cameras (e.g., video cameras). For example, a single stream of real-time content or recorded content may include media data associated with a video source provided by a video camera (e.g., audio and visual data capturing the appearance and voice of a user participating in the communication session). In some embodiments, the video source may include such audio and visual data, one or more still images, and / or one or more avatars. The one or more still images may also include one or more avatars.
[0085] Another example of a single stream of real-time and / or recorded content may include media data including avatars of users participating in a communication session together with audio data capturing the user's voice. Yet another example of a single stream of real-time or recorded content may include media data including a file displayed on a display screen together with audio data capturing the user's voice. Thus, various streams of real-time and / or recorded content in communication data enable remote conferencing between a group of people and sharing content within a group of people. In some embodiments, various streams of real-time and / or recorded content within communication data may originate from multiple co-located cameras located in a space (e.g., a room) for recording or live streaming a presentation, the presentation including one or more personal presentations and one or more individuals using the presented content.
[0086] Participants or attendees can view the content of the communication session 1104 in real time as the activity occurs, or alternatively, at a later time after the activity occurs by recording. In the example described herein, the client computing devices 1106(1) to 1106(N) participating in the communication session 1104 are configured to receive and render communication data for display on a user interface of a display screen. The communication data may include a collection of various instances or streams of real-time and / or recorded content. For example, a single content stream may include media data associated with a video source (e.g., audio and visual data capturing the appearance and voice of users participating in the communication session). Another example of a single content stream may include media data including an avatar of a user participating in the conference session together with audio data capturing the user's voice. Yet another example of a single content stream may include media data including content items displayed on a display screen and / or audio data capturing the user's voice. Thus, the various content streams within the communication data enable facilitating a conference or broadcast presentation between a group of people dispersed in remote locations.
[0087] Participants or attendees of a communication session are people within range of a camera or other image and / or audio capture device so that the motion and / or sounds of the person as they view and / or listen to content shared through the communication session can be captured (e.g., recorded). For example, a participant may be sitting in a crowd, viewing shared live content from a broadcast location occurring at a stage show. Alternatively, a participant may be sitting in an office conference room, viewing shared content of a communication session with other colleagues through a display screen. Even further, a participant may be sitting or standing in front of a personal device (e.g., a tablet, smartphone, computer, etc.), viewing shared content of a communication session alone in his office or at home.
[0088] System 1102 includes device 1110. Device 1110 and / or other components of system 1102 may include distributed computing resources that communicate with each other and / or with client computing devices 1106(1) to 1106(N) via one or more networks 1108. In some examples, system 1102 may be a stand-alone system that is responsible for the task of managing various aspects of one or more communication sessions (e.g., communication session 1104). As an example, system 1102 may be managed by an entity such as Slack, WEBEX, GOTOMEETING, GOOGLE HANGOUTS, etc.
[0089] The network 1108 may include, for example, a public network such as the Internet, a private network such as an institutional and / or personal intranet, or some combination of private and public networks. The network 1108 may also include any type of wired and / or wireless network, including but not limited to a local area network ("LAN"), a wide area network ("WAN"), a satellite network, a cable network, a Wi-Fi network, a WiMax network, a mobile communication network (e.g., 3G, 4G, etc.), or any combination thereof. The network 1108 may utilize communication protocols, including packet-based and / or datagram-based protocols, such as the Internet Protocol ("IP"), the Transmission Control Protocol ("TCP"), the User Datagram Protocol ("UDP"), or other types of protocols. In addition, the network 1108 may also include a plurality of devices that facilitate network communications and / or form the hardware foundation of the network, such as switches, routers, gateways, access points, firewalls, base stations, repeaters, backbone equipment, and the like.
[0090] In some examples, network 1108 may also include devices capable of connecting to a wireless network, such as a wireless access point (“WAP”). Examples support connectivity via WAPs that send and receive data over various electromagnetic frequencies (e.g., radio frequencies), including WAPs that support the Institute of Electrical and Electronics Engineers (“IEEE”) 802.11 standards (e.g., 802.11g, 802.11n, 802.11ac, etc.) and other standards.
[0091] In various examples, device 1110 may include one or more computing devices that operate in a cluster or other grouped configuration to share resources, balance loads, improve performance, provide failover support or redundancy, or for other purposes. For example, device 1110 may belong to various types of devices, such as traditional server type devices, desktop computer type devices, and / or mobile type devices. Therefore, although shown as a single type of device or server type device, device 1110 may include a variety of device types and is not limited to a particular type of device. Device 1110 may represent but is not limited to a server computer, a desktop computer, a network server computer, a personal computer, a mobile computer, a laptop computer, a tablet computer, or any other type of computing device.
[0092] The client computing device (e.g., one of client computing devices 1106(1)-1106(N)) can belong to various categories of devices, which can be the same as or different from device 1110, such as traditional client-type devices, desktop computer-type devices, mobile-type devices, dedicated-type devices, embedded-type devices, and / or wearable-type devices. Thus, the client computing device can include, but is not limited to, a desktop computer, a game console and / or gaming device, a tablet computer, a personal data assistant ("PDA"), a mobile phone / tablet hybrid device, a laptop computer, a telecommunications device, a computer navigation-type client computing device (e.g., a satellite-based navigation system including a Global Positioning System ("GPS") device), a wearable device, a virtual reality ("VR") device, an augmented reality ("AR") device, an implanted computing device, an automotive computer, a network-enabled television, a thin client, a terminal, an Internet of Things ("IoT") device, a workstation, a media player, a personal video recorder ("PVR"), a set-top box, a camera, for inclusion as an integrated component (e.g., a peripheral device) in a computing device, an appliance device, or any other category of computing device. Furthermore, the client computing device may include a combination of the earlier listed examples of client computing devices, such as, for example, a desktop computer type device or a mobile type device in combination with a wearable device, and so forth.
[0093] The various classes and device types of client computing devices 1106(1) to 1106(N) may represent any type of computing device having one or more data processing units 1192 operably connected to a computer-readable medium 1194 (e.g., via a bus 1116), which in some instances may include a system bus, a data bus, an address bus, a PCI bus, a Mini-PCI bus, and one or more of any of a variety of local, peripheral, and / or independent buses.
[0094] The executable instructions stored on the computer-readable medium 1194 may include, for example, an operating system 1119 , a client module 1120 , a profile module 1122 , and other modules, programs, or applications that may be loaded and executed by the data processing unit 1192 .
[0095] The client computing devices 1106(1)-1106(N) may also include one or more interfaces 1124 to enable communication between the client computing devices 1106(1)-1106(N) and other networked devices (e.g., device 1110) via the network 1108. Such network interfaces 1124 may include one or more network interface controllers (NICs) or other types of transceiver devices to send and receive communications and / or data via the network. In addition, the client computing devices 1106(1)-1106(N) may include input / output ("I / O") interfaces (devices) 1126 that enable communication with input / output devices, such as user input devices including peripheral input devices (e.g., game controllers, keyboards, mice, pens, voice input devices such as microphones, cameras for obtaining and providing video feeds and / or still images, touch input devices, gesture input devices, etc.) and / or output devices including peripheral output devices (e.g., displays, printers, audio speakers, tactile output devices, etc.). Fig.11 The client computing device 1106 ( 1 ) is shown connected in some manner to a display device (eg, display screen 1129 ( 1 )) that can display a UI in accordance with the techniques described herein.
[0096] exist Fig.11 In the example environment 1100 of FIG. 1 , client computing devices 1106(1) to 1106(N) can use their respective client modules 1120 to connect to each other and / or other external devices to participate in a communication session 1104 or to contribute activities to a collaborative environment. For example, a first user can utilize client computing device 1106(1) to communicate with a second user of another client computing device 1106(2). When executing client module 1120, the users can share data, which can cause client computing device 1106(1) to connect to system 1102 and / or other client computing devices 1106(2) to 1106(N) via network 1108.
[0097] The client computing devices 1106(1) to 1106(N) may use their respective profile modules 1122 to generate participant profiles ( Fig.111100 of the system 1102). The participant profile may include one or more of the following: an identity of a user or a group of users (e.g., name, unique identifier (“ID”), etc.), user data (e.g., personal data), machine data (e.g., location) (e.g., IP address, room in a building, etc.), and technical capabilities, etc. The participant profile may be used to register participants for a communication session.
[0098] like Fig.11 As shown, the device 1110 of the system 1102 includes a server module 1130 and an output module 1132. In this example, the server module 1130 is configured to receive media streams 1134(1) to 1134(N) from various client computing devices (e.g., client computing devices 1106(1) to 1106(N)). As described above, the media streams may include video information sources (e.g., audio and visual data associated with a user), audio data to be output with the presentation of the user's avatar (e.g., a pure audio experience without sending the user's video data), text data (e.g., text messages), file data and / or screen sharing data (e.g., documents, slides, images, videos, etc. displayed on a display screen), etc. Therefore, the server module 1130 is configured to receive a collection of various media streams 1134(1) to 1134(N) (the collection is referred to herein as "media data") during the real-time viewing communication session 1104. In some cases, not all client computing devices participating in the communication session 1104 provide media streams. For example, the client computing device may be a consuming device or a “listening” device only, such that it only receives content associated with the communication session 1104 but does not provide any content to the communication session 1104 .
[0099] In various examples, the server module 1130 can select various aspects of the media stream 1134 to be shared with various ones of the participating client computing devices 1106(1)-1106(N). Accordingly, the server module 1130 can be configured to generate session data 1136 based on the stream 1134 and / or pass the session data 1136 to the output module 1132. The output module 1132 can then transmit communication data 1139 to the client computing devices (e.g., the client computing devices 1106(1)-1106(N) participating in the real-time viewing communication session). The communication data 1139 can include video, audio, and / or other content data, provided by the output module 1132 based on the content 1150 associated with the output module 1132 and based on the received session data 1136.
[0100] As shown, output module 1132 sends communication data 1139(1) to client computing device 1106(1), and sends communication data 1139(2) to client computing device 1106(2), and sends communication data 1139(3) to client computing device 1106(3), etc. The communication data 1139 sent to the client computing devices may be the same or may be different (e.g., the positioning of the content stream within the user interface may vary between devices).
[0101] In various implementations, the device 1110 and / or the client module 1120 may include a GUI presentation module 1140. The GUI presentation module 1140 may be configured to analyze the communication data 1139 for delivery to one or more client computing devices 1106. Specifically, the GUI presentation module 1140 at the device 1110 and / or the client computing device 1106 may analyze the communication data 1139 to determine an appropriate manner for displaying the video, image, and / or content on the display screen 1129 of the associated client computing device 1106. In some implementations, the GUI presentation module 1140 may provide the video, image, and / or content to a presentation GUI 1146 rendered on the display screen 1129 of the associated client computing device 1106. The GUI presentation module 1140 may cause the presentation GUI 1146 to be rendered on the display screen 1129. The presentation GUI 1146 may include the video, image, and / or content analyzed by the GUI presentation module 1140.
[0102] In some implementations, the presentation GUI 1146 may include multiple sections or grids that may render or include video, images, and / or content for display on the display screen 1129. For example, a first section of the presentation GUI 1146 may include a video feed of a presenter or individual, and a second section of the presentation GUI 1146 may include a video feed of an individual consuming conference information provided by the presenter or individual. The GUI presentation module 1140 may populate the first and second sections of the presentation GUI 1146 in a manner that appropriately mimics an ambient experience that the presenter and individual may share.
[0103] In some embodiments, the GUI presentation module 1140 can zoom in or provide a zoomed view of an individual represented by a video feed to highlight the individual's reactions to the presenter, such as facial features. In some embodiments, the presentation GUI 1146 can include video feeds of multiple participants associated with a conference (e.g., a general communication session). In other embodiments, the presentation GUI 1146 can be associated with a channel such as a chat channel, an enterprise team channel, etc. Thus, the presentation GUI 1146 can be associated with an external communication session that is different from a general communication session.
[0104] Fig.12 A diagram showing example components of an example device 1200 (also referred to herein as a "computing device") configured to generate data for some user interfaces disclosed herein is illustrated. The device 1200 may generate data that may include one or more portions that may render or include video, images, virtual objects 116, and / or content for display on a display screen 1129. The device 1200 may represent one of the devices described herein. Additionally or alternatively, the device 1200 may represent one of the client computing devices 1106.
[0105] As shown, device 1200 includes one or more data processing units 1202, computer-readable media 1204, and communication interface 1206. The components of device 1200 are operably connected, for example, via a bus, which may include one or more of a system bus, a data bus, an address bus, a PCI bus, a Mini-PCI bus, and any kind of local, peripheral, and / or independent bus.
[0106] As used herein, a data processing unit (e.g., data processing unit 1202 and / or data processing unit 1192) may represent, for example, a CPU-type data processing unit, a GPU-type data processing unit, a field programmable gate array ("FPGA"), another type of digital signal processor ("DSP"), or other hardware logic components that may be driven by a CPU in some cases. By way of example and not limitation, illustrative types of hardware logic components that may be utilized include application specific integrated circuits ("ASICs"), application specific standard products ("ASSPs"), systems on chips ("SOCs"), complex programmable logic devices ("CPLDs"), and the like.
[0107] As used herein, computer-readable media (e.g., computer-readable media 1204 and computer-readable media 1194) can store instructions that can be executed by a data processing unit. The computer-readable medium can also store instructions that can be executed by an external data processing unit (e.g., an external CPU, an external GPU) and / or by an external accelerator (e.g., an FPGA-type accelerator, a DSP-type accelerator, or any other internal or external accelerator). In various examples, at least one CPU, GPU, and / or accelerator is incorporated into the computing device, while in some examples, one or more of the CPU, GPU, and / or accelerator is external to the computing device.
[0108] Computer readable media (also referred to herein as computer readable media) may include computer storage media and / or communication media. Computer storage media may include one or more of the following: volatile memory, non-volatile memory and / or other persistent and / or secondary computer storage media, removable and non-removable computer storage media implemented in any method or technology for storing information such as computer readable instructions, data structures, program modules or other data. Thus, computer storage media includes media in tangible and / or physical form included in devices and / or hardware components that are part of a device or external to a device, including, but not limited to, random access memory (“RAM”), static random access memory (“SRAM”), dynamic random access memory (“DRAM”), phase change memory (“PCM”), read only memory (“ROM”), erasable programmable read-only memory (“EPROM”), electrically erasable programmable read-only memory (“EEPROM”), flash memory, compact disk read-only memory (“CD-ROM”), digital versatile disk (“DVD”), optical cards or other optical storage media, magnetic cassettes, magnetic strips, magnetic disk storage, magnetic cards or other magnetic storage devices or media, solid-state memory devices, storage arrays, network attached storage, storage area networks, hosted computer storage, or any other storage memory, storage devices, and / or storage media that can be used to store and maintain information for access by a computing device.
[0109] In contrast to computer storage media, communication media may embody computer-readable instructions, data structures, program modules, or other data in a modulated data signal (e.g., a carrier waveform) or other transport mechanism. As defined herein, computer storage media does not include communication media. That is, computer storage media itself does not include communication media consisting solely of a modulated data signal, a carrier wave, or a propagating signal.
[0110] The communication interface 1206 may represent, for example, a network interface controller ("NIC") or other type of transceiver device to send and receive communications over a network. In addition, the communication interface 1206 may include one or more cameras and / or audio devices 1222 to enable the generation of video feeds and / or still images, etc.
[0111] In the illustrated example, computer-readable medium 1204 includes a data storage unit 1208. In some examples, data storage unit 1208 includes a data repository such as a database, a data warehouse, or other type of structured or unstructured data storage. In some examples, data storage unit 1208 includes a corpus and / or a relational database having one or more tables, indexes, stored procedures, etc. to enable data access including, for example, one or more of the following: Hypertext Markup Language ("HTML") tables, Resource Description Framework ("RDF") tables, Web Ontology Language ("OWL") tables, and / or Extensible Markup Language ("XML") tables.
[0112] The data storage unit 1208 may store data for the operation of processes, applications, components, and / or modules stored in the computer-readable medium 1204 and / or executed by the data processing unit 1202 and / or accelerator. For example, in some examples, the data storage unit 1208 may store session data 1210, profile data 1212 (e.g., associated with participant profiles), and / or other data. The session data 1210 may include the total number of participants (e.g., users and / or client computing devices) in the communication session, the activities occurring in the communication session, the list of invitees for the communication session, and / or other data related to when and how the communication session is conducted or hosted. The data storage unit 1208 may also include content data 1214, such as content including video, audio, or other content for rendering and display on one or more display screens 1129.
[0113] Optionally, some or all of the above-referenced data may be stored on separate memories 1216 on one or more data processing units 1202, such as memories on CPU-type processors, GPU-type processors, FPGA-type accelerators, DSP-type accelerators, and / or other accelerators. In this example, the computer-readable medium 1204 also includes an operating system 1218 and an application programming interface 1210 (API) configured to expose functions and data of the device 1200 to other devices. In addition, although the number of modules shown is only an example and the number may be higher or lower, the computer-readable medium 1204 includes one or more modules, such as a server module 1230, an output module 1232, and a GUI presentation module 1240. That is, the functions described herein and associated with the modules shown may be performed by a smaller number of modules or a larger number of modules on one device or distributed across multiple devices.
[0114] Fig.13 1 is a diagram of a computing device that illustrates various aspects of the configuration and operation of a computing device 1300 that may implement various aspects of the system disclosed herein. Figure 1Detail of computing device 101 shown. Computing device 1300 may provide an augmented reality ("AR") environment or a virtual reality ("VR") environment. In general, an AR environment superimposes a computer-generated ("CG") image on a user's view of a real-world environment. For example, computing device 1300 may generate a synthetic view to enable a user to visually perceive a computer-generated image superimposed on a rendering of a real-world environment 112, where the rendering of the real-world environment 112 may be displayed using image data of a sensor 1399 (e.g., a depth map sensor or a camera) directed toward the real-world environment 112 (e.g., a room). In some embodiments, computing device 1300 may generate a synthetic view to enable a user to visually perceive a computer-generated image superimposed on a direct view of the real-world environment 112. Thus, computing device 1300 may have a prism or other optical device that allows a user to see through the optical device to see a real-world object or a direct view of the real-world environment, and simultaneously see a computer-generated image superimposed on a view of the real-world object. An AR environment may also be referred to herein as a mixed reality ("MR") environment. VR environments include computer-generated images of virtual environments and virtual objects. MR and AR environments can utilize depth map sensors to determine the distance between the device and real-world objects. This allows the computer to scale and position computer-generated graphics on images of real-world objects in a realistic manner.
[0115] exist Fig.13 In the example shown, the optical system 1302 includes an illumination engine 1304 to generate electromagnetic ("EM") radiation including a first bandwidth for generating CG images and a second bandwidth for tracking physical objects. The first bandwidth may include some or all of the visible light portion of the EM spectrum, while the second bandwidth may include any portion of the EM spectrum suitable for deploying a desired tracking protocol. In this example, the optical system 1302 also includes an optical assembly 1306 positioned to receive the EM radiation from the illumination engine 1304 and direct the EM radiation (or its individual bandwidths) along one or more predetermined optical paths.
[0116] For example, illumination engine 1304 can emit EM radiation along a common optical path shared by both the first bandwidth and the second bandwidth into optical assembly 1306. Optical assembly 1306 can also include one or more optical components configured to separate the first bandwidth from the second bandwidth (e.g., by causing the first bandwidth and the second bandwidth to propagate along different image generation and object tracking optical paths, respectively).
[0117] In some cases, the user experience depends on the computing device 1300 accurately identifying the characteristics of a physical object or plane (e.g., a real-world floor) and then generating a CG image based on these identified characteristics. For example, assume that the computing device 1300 is programmed to generate the following user perception: a virtual game character is running towards and ultimately jumping over a real-world structure. To obtain this user perception, the computing device 1300 can obtain detailed data that defines the characteristics of the real-world environment 112 surrounding the computing device 1300. To provide this functionality, in some embodiments, the optical system 1302 of the computing device 1300 may include a laser line projector and a differential imaging camera (neither of which is in the Fig.13 ).
[0118] In some examples, computing device 1300 utilizes optical system 1302 to generate a synthetic view that includes one or more CG images and a view of at least a portion of real-world environment 112 (e.g., from the perspective of a user wearing computing device 1300). For example, optical system 1302 may utilize various techniques such as AR techniques to generate a synthetic view that includes a CG image superimposed on a real-world view. As such, optical system 1302 may be configured to generate a CG image via optical assembly 1306 that includes display panel 1314.
[0119] In the example shown, the display panel includes separate right-eye and left-eye transparent display panels, respectively labeled 1314R and 1314L. In some examples, the display panel 1314 includes a single transparent display panel that can be viewed with both eyes or a single transparent display panel that can only be viewed with a single eye. Therefore, it can be understood that the technology described herein can be deployed in a monocular device (e.g., a GOOGLE GLASS AR device) and a binocular device (e.g., a MICROSOFT HOLOLENS AR device).
[0120] Light received from the real-world environment 112 passes through the see-through display panel 1314 to one or both eyes of the user. Graphical content calculated by the image generation engine 1326 executed on the processing unit 1320 and displayed by the right-eye and left-eye display panels (if configured as see-through display panels) can be used to visually enhance or otherwise modify the real-world environment 112 viewed by the user through the see-through display panel 1314. In this configuration, the user is able to view virtual objects that are not present in the real-world environment 112 while viewing physical objects (also referred to herein as "real-world objects") in the real-world environment 112. This creates an illusion or appearance that the virtual objects are physical objects or physically present light-based effects in the real-world environment 112.
[0121] In some examples, display panel 1314 is a waveguide display that includes one or more diffractive optical elements ("DOEs") for coupling incident light into the waveguide, expanding the incident light in one or more directions for exit pupil expansion, and / or coupling the incident light out of the waveguide (e.g., toward the user's eyes). In some examples, computing device 1300 also includes an additional see-through optical component in the form of a transparent tulle 1316 disposed on the Fig.13 13, the transparent tulle is located between the real world environment 112 and the display panel 1314. It will be appreciated that the transparent tulle 1316 may be included in the computing device 1300 for purely aesthetic and / or protective purposes.
[0122] The computing device 1300 may also include various other components (in Fig.13 1300 ), for example, a front-facing camera (e.g., a red / green / blue (“RGB”), black and white (“B&W”), or infrared (“IR”) camera), a speaker, a microphone, an accelerometer, a gyroscope, a magnetometer, a temperature sensor, a touch sensor, a biometric sensor, other image sensors, energy storage components (e.g., a battery), communication facilities, a global positioning system (“GPS”), a receiver, a laser line projector, a differential imaging camera, and possibly other types of sensors. Data obtained from one or more sensors 1308 (some of which have been identified above) can be used to determine the orientation, position, and movement of the computing device 1300. As described above, data obtained from the differential imaging camera and laser line projector or other types of sensors can also be used to generate a 3D depth map of the surrounding real-world environment 112.
[0123] In the example shown, the computing device 1300 includes one or more logic devices and one or more computer storage devices that store instructions that can be executed by the logic devices to implement the functions disclosed herein. In particular, the controller 1318 may include one or more processing units 1320, one or more computer readable media 1322 for storing an operating system 1324, an image generation engine 1326, and a terrain mapping engine 1328, as well as other programs (e.g., a 3D depth map generation module configured to generate mesh data in the manner disclosed herein) and data.
[0124] In some implementations, computing device 1300 is configured to analyze data obtained by sensor 1308 to perform feature-based tracking of the orientation of computing device 1300. For example, in a scenario where object data includes an indication of a stationary physical object (also referred to herein as a "real-world object") within real-world environment 112 (e.g., a table), computing device 1300 may monitor the position of the stationary object within a terrain mapping field of view ("FOV"). Then, based on changes in the position of the stationary object within the terrain mapping FOV and the depth of the stationary object from computing device 1300, a terrain mapping engine executed on processing unit 1320 may calculate a change in the orientation of computing device 1300.
[0125] It will be appreciated that these feature-based tracking techniques may be used to monitor changes in the orientation of computing device 1300 for the purpose of monitoring the orientation of the user's head (e.g., assuming the user is properly wearing computing device 1300). The calculated orientation of computing device 1300 may be utilized in a variety of ways, some of which have been described above.
[0126] The processing unit 1320 may represent, for example, a central processing unit ("CPU") type processor, a graphics processing unit ("GPU") type processing unit, an FPGA, one or more digital signal processors ("DSP"), or other hardware logic components that may be driven by a CPU in some cases. Illustrative types of hardware logic components that may be used include, by way of example and not limitation, an ASIC, an application specific standard product ("ASSP"), a system on a chip ("SOC"), a complex programmable logic device ("CPLD"), etc. The controller 1318 may also include one or more computer readable media 1322, such as the computer readable media described above.
[0127] Unless otherwise specifically stated, it should be understood that conditional language used herein, such as "can," "could," "might," or "may," is to be understood in context to express that certain examples include and other examples do not include certain features, elements, and / or steps, in addition to other items. Therefore, such conditional language is generally not intended to imply that one or more examples require certain features, elements, and / or steps in any way, or that one or more examples must include logic for determining (with or without user input or prompting) whether certain features, elements, and / or steps are included or to be performed in any particular example. Unless otherwise explicitly stated, conjunction language such as the phrase "at least one of X, Y, or Z" should be understood to mean that an item, term, etc. can be X, Y, or Z, or a combination thereof.
[0128] It should also be understood that many variations and modifications may be made to the above examples, and the elements thereof should be understood to be among other acceptable examples. All such modifications and variations are intended to be included in the scope of this disclosure and protected by the accompanying claims.
[0129] Finally, although various configurations have been described in language specific to structural features and / or methodological acts, it should be understood that the subject matter defined in the appended representations is not necessarily limited to the specific features or acts described. Rather, the specific features and acts disclosed serve as example forms of implementing the claimed subject matter.
Claims
1. A system for tracking the movement of a real-world object, comprising: one or more data processing units; as well as A computer readable medium having computer executable instructions encoded thereon to cause the one or more data processing units to perform the following operations: receiving sensor data generated by a depth map sensor, the sensor data comprising depth map data defining a three-dimensional model of a real-world object within a viewing area of the system; receiving model data defining a virtual object, the model data causing the system to display a rendering of the virtual object in parallel with display of the real-world object; identifying at least one characteristic of the real-world object based on an analysis of the sensor data; transmitting metadata defining the at least one characteristic of the real-world object to a remote computing device, the metadata causing the remote computing device to identify the real-world object using the at least one characteristic to track movement of the real-world object within a viewing area of the remote computing device, wherein the remote computing device generates updated metadata indicating the movement of the real-world object within the viewing area of the remote computing device; and receiving the updated metadata from the remote computing device, the updated metadata indicating the movement of the real-world object within the viewing area of the remote computing device; and Based on the metadata indicative of the movement of the real-world object, an updated rendering of the virtual object is displayed at a new location.
2. The system according to claim 1, wherein: The instructions further cause the one or more data processing units to: determining that the real-world object is outside the viewing area of the system; as well as In response to determining that the real-world object is outside of the viewing area of the system, the metadata defining the at least one characteristic of the real-world object is transmitted to the remote computing device.
3. The system according to claim 2, wherein: The real-world object is determined to be outside the viewing area of the system when image data generated by an imaging camera directed at the real-world object indicates that the image of the real-world object is distorted or blurred at a threshold level.
4. The system according to claim 1, wherein: The instructions further cause the one or more data processing units to: determining a distance between the system and each computing device of a plurality of computing devices; selecting the remote computing device from a plurality of computing devices when the distance between the system and the remote computing device is within a threshold distance; as well as The metadata defining the at least one property of the real-world object is transmitted to the remote computing device.
5. The system according to claim 1, wherein: The instructions further cause the one or more data processing units to: receiving location data indicating a location of the remote computing device; determining that the remote computing device is within a predetermined area; as well as In response to determining that the remote computing device is within a predetermined area, the metadata defining the at least one characteristic of the real-world object is transmitted to the remote computing device.
6. The system according to claim 1, wherein: The instructions further cause the one or more data processing units to: analyzing one or more streams of the communication session to identify references to the real-world object; selecting the remote computing device from a plurality of computing devices when the remote computing device transmits one or more streams indicative of a keyword referencing the real-world object; In response to determining that the one or more streams from the remote computing device indicate a keyword referencing the real-world object, the metadata defining the at least one property of the real-world object is transmitted to the remote computing device.
7. The system according to claim 1, wherein: The instructions further cause the one or more data processing units to: receiving context data indicating specifications of one or more sensors of the remote computing device; selecting the remote computing device from a plurality of computing devices when the specifications of the one or more components of the remote computing device meet one or more criteria; In response to determining that the specifications of the one or more components of the remote computing device satisfy the one or more criteria, the metadata defining the at least one property of the real-world object is transmitted to the remote computing device.
8. The system according to claim 1, wherein: The instructions further cause the one or more data processing units to: receiving context data indicative of performance specifications of one or more sensors of the remote computing device; selecting the remote computing device from a plurality of computing devices when the performance specifications of the one or more sensors of the remote computing device meet one or more criteria; In response to determining that the performance specification of the one or more sensors of the remote computing device satisfies the one or more criteria, the metadata defining the at least one characteristic of the real-world object is transmitted to the remote computing device.
9. A method for tracking the movement of a real-world object, comprising: receiving, at a computing device, metadata defining at least one characteristic of a real-world object, wherein the at least one characteristic is detected by a sensor of a remote computing device, the metadata further comprising model data defining a virtual object and associated data defining a position of the virtual object relative to the real-world object; receiving, at the computing device, sensor data generated by a depth map sensor of the computing device, the sensor data comprising depth map data defining a three-dimensional model of the real-world object within a viewing area of the depth map sensor of the computing device; determining a first value of a physical property of the real-world object within the viewing area of the depth map sensor of the computing device, the first value of the physical property determined by analyzing the sensor data; determining that the first value of the physical property of the real-world object within the viewing area is within a threshold difference of a second value of the at least one property of the real-world object detected by the sensor of the remote computing device; and In response to determining that the first value and the second value are within the threshold difference, tracking the position of the real-world object using the depth map sensor and rendering the virtual object on a display device at a position relative to a display of the real-world object based on the position.
10. The method according to claim 9, further comprising: tracking movement of the real-world object within the viewing area using the depth map sensor to determine a new position of the real-world object within the viewing area; as well as The updated metadata is transmitted to the remote computing device, the updated metadata causing the remote computing device to display a rendering of the real-world object, wherein the rendering indicates the new position.
11. The method according to claim 9, further comprising: determining a new position of the real-world object within the viewing area by tracking movement of the real-world object within the viewing area using the depth map sensor; as well as The updated metadata is transmitted to the remote computing device, the updated metadata causing the remote computing device to display a rendering of the virtual object at a location based on the new location of the real-world object.
12. The method according to claim 9, wherein: The first value of the physical property indicates a first measurement result of at least one dimension of the real-world object detected by the depth map sensor of the computing device, and wherein the second value of the at least one property indicates a second measurement result of the at least one dimension of the real-world object detected by the sensor of the remote computing device.
13. The method according to claim 9, wherein: The first value of the physical property indicates a first measurement of at least one color of the real-world object detected by a camera of the computing device, and wherein the second value of the at least one property indicates a second measurement of the at least one color of the real-world object detected by a camera of a remote computing device.
14. The method according to claim 9, wherein: The first value of the physical property indicates a first measurement of a shape of the real-world object detected by a camera of the computing device, and wherein the second value of the at least one property indicates a second measurement of the shape of the real-world object detected by a camera of a remote computing device.
15. The method according to claim 9, wherein: The first value of the physical property indicates a first measurement of a surface texture of the real-world object detected by a camera of the computing device, and wherein the second value of the at least one property indicates a second measurement of the surface texture of the real-world object detected by a camera of a remote computing device.
