Mixed reality annotation on a physical display screen

Abstract

An illustrative method may be performed during a mixed reality session presented by a mixed reality device within an environment. The method may include identifying a physical screen of a physical device present in the environment, receiving data representing a virtual annotation generated within the environment, and transmitting the data representing the virtual annotation to the physical device. Corresponding devices, methods, systems, and computer readable media are also disclosed.

Description

MIXED REALITY ANNOTATION ON A PHYSICAL DISPLAY SCREENBACKGROUND

[0001] Mixed reality is an umbrella term referring to various technologies that serve to augment, virtualize, or otherwise extend a user’s experience of reality in a variety of ways. For example, virtual reality, augmented reality, and other types of mixed reality have been developed and deployed for use with entertainment, educational, vocational, and other types of applications. In certain cases, mixed reality experiences may be presented on handheld devices such as smartphones or laptop computers viewed from a few feet away. In other cases, mixed reality experiences may be presented by way of head-mounted display devices that immerse users more fully in virtual or augmented worlds by presenting images in front of the user’s eyes.SUMMARY

[0002] Methods and systems described herein relate to annotations produced during mixed reality experiences such as augmented reality experiences, virtual reality experiences, and so forth. Certain mixed reality controllers may be configured to facilitate users not only in controlling various aspects of a mixed reality experience (e.g., moving a cursor, selecting an item, performing a virtual action, etc.) but also in generating a virtual annotation within the environment being experienced during the mixed reality session. For example, a controller may be used by a user in a similar manner as a pen or marker to allow the user to sketch on a virtual whiteboard or to make virtual markings on real -world objects in the environment. In other examples, annotations may be made using hand gestures (e.g., drawing with a finger, etc.) or may otherwise be produced without physical mixed reality controllers.

[0003] Conventionally, virtual annotations may be visible to the user making them and perhaps to other users sharing in the virtual reality experience (e.g., by way of their own devices). As such, virtual annotations have conventionally been contained within virtual realms so as to be visible and usable only by those with suitable access and mixed reality equipment. Methods and systems described herein for mixed reality annotation on a physical display screen, however, are not contained or limited in this way. Rather, as will be described in detail below, virtual annotations in accordance with principles described herein may impact not only the virtual or augmented reality world of a mixed reality session, but alsoimpact real-world devices. For example, virtual annotations may be associated with and transmitted to, then displayed, stored, and / or otherwise used by, physical devices with physical screens in the user’s environment (e.g., laptops, televisions, phones, tablets, etc.).

[0004] In one implementation (of the proposed solution), a method includes operations such as: 1) identifying, during a mixed reality session presented by a mixed reality device within an environment, a physical screen of a physical device present in the environment; 2) receiving data representing a virtual annotation generated within the environment; and 3) transmitting the data representing the virtual annotation to the physical device.

[0005] For example, the receiving and the transmitting of the data representing the virtual annotation may be performed to allow the physical device to present the virtual annotation in real time for viewing within the environment and without aid of the mixed reality device. Accordingly, the proposed solution may include presenting, by the physical device, the virtual annotation in real time (i.e., without undue delay and thus immediately, or at least substantially immediately, such as with a delay not recognizable for a human user) for viewing within the environment and without aid of the mixed reality device.

[0006] In another implementation, a non-transitory computer-readable medium may store instructions that, when executed, cause a processor of a computing device to perform a process including: 1) identifying, during a mixed reality session presented by a mixed reality device within an environment, a physical screen of a physical device present in the environment; 2) receiving data representing a virtual annotation generated within the environment; and 3) transmitting the data representing the virtual annotation to the physical device.

[0007] In yet another implementation, a mixed reality device may include: 1) a memory storing instructions; and 2) one or more processors communicatively coupled to the memory and configured to execute the instructions to perform a process. The process may include: 1) identifying, during a mixed reality session presented by the mixed reality device within an environment, a physical screen of a physical device present in the environment; 2) receiving data representing a virtual annotation generated within the environment; and 3) transmitting the data representing the virtual annotation to the physical device.

[0008] The details of these and other implementations are set forth in the accompanying drawings and the description below. Other features will also be made apparent from the following description, drawings, and claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0009] FIG. 1 shows an illustrative mixed reality device operating in accordance with principles described herein to perform mixed reality annotation on a physical display screen within an environment.

[0010] FIG. 2 shows an illustrative method for mixed reality annotation on a physical display screen in accordance with principles described herein.

[0011] FIG. 3 A shows illustrative aspects of an environment in which a mixed reality device may operate in accordance with principles described herein.

[0012] FIG. 3B shows an illustrative mixed reality device operating to produce example virtual annotations within the environment of FIG. 3 A in accordance with principles described herein.

[0013] FIG. 4 shows illustrative software applications executing on a physical device and to which example virtual annotations can be directed in accordance with principles described herein.

[0014] FIG. 5 shows illustrative aspects of an environment that includes a plurality of screens as the environment is viewed during a mixed reality session in accordance with principles described herein.

[0015] FIGS. 6A-6C show illustrative factors that may be considered as a determination is made that a virtual annotation is targeted to one screen in the environment instead of another in accordance with principles described herein.

[0016] FIG. 7 shows an illustrative computing system that may be used to implement various devices and / or systems described herein.DETAILED DESCRIPTION

[0017] Methods and systems for mixed reality annotation on a physical display screen are described herein. Mixed reality technologies may be used to engage in mixed reality sessions of various types. As used herein, a mixed reality session refers to a period of time during which a mixed-reality device is operated to present, to at least one user of the computing device, mixed reality including virtual and real -world objects. Certain mixed reality sessions may be relatively non-interactive. For example, just as a person may wish to use a computer or mobile device to watch a media program or listen to music, a mixed reality device may be used to engage in a largely non-interactive session in which the user views preprogrammed content or participates in another low-engagement session. Other mixedreality sessions, however, may be associated with higher levels of engagement and interaction. For example, just as a person could also use a computer or mobile device to use productivity applications to create work product or perform useful tasks (e.g., drafting text, creating artwork, editing photos or video, etc.), certain mixed reality sessions may be more interactive and objective-driven to achieve similar goals. Indeed, in some cases, a mixed reality device may be used as an alternative to a personal computer for a user performing various computing tasks (e.g., at home, in the office, etc.).

[0018] In any of these examples, and particularly for productivity -related tasks, it may be desirable for annotations to be made during mixed reality sessions. As one example, a user reading a document in a web browser may desire to highlight a passage or make a note in the margin. As another example, a user creating a slide presentation may desire to position a picture and write text next to the picture on a slide. In still another example, a user may desire to sketch an idea on a virtual white board (e.g., in the form of a block diagram, etc.) so that the idea can be shared, further developed, and stored for later.

[0019] These and myriad other potential tasks for work and personal use may benefit from users being able to generate virtual annotations during the mixed reality sessions in which they are engaged. However, certain technical problems arise when such virtual annotations are created using conventional mixed reality tools. For example, at least one technical problem with existing annotation technologies is that they either function in the real-world or in a virtual world but fail to bridge the gap between these in a manner that would be intuitive for a user engaging in a mixed reality session. Using existing technologies, for instance, the user could use a physical marker to draw on a physical white board that could be seen by others in the room and perhaps scanned and digitized after the fact, but which is not inherently integrated with the virtual world. The user could also use a virtual annotation device to create annotations that are native to the virtual world (such that no scanning or digitization after the fact has to be performed), but such virtual annotations would be visible only to those who are sharing in the same mixed reality session (e.g., with their own mixed reality devices, etc.) and not to other people in the room who, for example, lack their own mixed reality device to take part in the session.

[0020] Not only does this technical problem make it arduous for annotations to be fully captured (digitized, stored for later, etc.) and shared with all who might be present (e.g., including some who may be taking part in the mixed reality session and others who may not be), but this segmentation between the real world and the virtual world may further create a frustrating and non-intuitive experience for any user creating the annotations. For example,there may be one or more virtual screens in the environment on which the user can freely annotate, including, for instance, a virtual desktop screen on which one or more annotatable software applications is executing. If there also happens to be one or more physical screens in the environment (e.g., a physical monitor of a desktop or laptop computer, a television, a portable device such as a tablet or smartphone, etc.), it may be confusing or irritating to the user that these physical screens (and any software applications executing thereon) are not annotatable in the same manner as the virtual screens. For example, the user may be distracted or frustrated by having to mentally keep track of which screens can accept and display annotations and which screens cannot.

[0021] Accordingly, systems and methods described herein for mixed reality annotation on physical display screens provide a technical solution to this technical problem by closing the gap between the real and virtual worlds that are experienced during a mixed reality session. Specifically, systems and methods described herein may be configured to automatically move virtual annotations into the real world where they can be presented, stored, received as input, and / or otherwise used by physical devices. Rather than virtual annotations being limited to the virtual world, a framework described herein is configured to determine when virtual annotations are targeted to physical screens and to then transmit data representing such virtual annotations to the physical devices associated with those screens. In this way, the annotations may be presented, stored, and used in the same manner as they are with virtual screens in the environment.

[0022] In certain examples, this process of determining the target for a virtual annotation and providing the annotation to a targeted physical device may be performed in real time (i.e., without undue delay such that the effect is experienced immediately by a human user, or at least substantially immediately so that the delay is not noticeable or inconvenient to the user) such that others in the environment may view or use the annotation regardless of whether they are part of the mixed reality session in which the annotation was produced. For example, an instructor presenting slides displayed on (or projected onto) a large screen could create virtual annotations using a mixed reality device the instructor is holding or wearing, and people in the audience could view those annotations in real time whether or not they also have a mixed reality device with which to engage in the mixed reality session. If desired by the presenter, annotations made in this way would also be easily stored in the physical device executing the presentation software after the presentation is complete and no extra steps would be needed to move the annotations from the virtual world in which they were created to the physical device.

[0023] Accordingly, at least one technical effect arising from frameworks for mixed reality annotation on physical display screens described herein is that the distinction between physical screens and virtual screens that both happen to be present in an environment may be abstracted away, thereby reducing mental energy any user has to exert to keep the distinction in mind for various screens around the room. The user experience associated with this abstraction is that the user may create annotations targeted to any screens he or she sees in the environment and may expect that the device associated with the targeted screen will accept, display, store, or otherwise receive that annotation input in the manner intended, regardless of whether the targeted screen is physical or virtual. Accordingly, another technical effect arises from methods and systems described herein is that transfer of annotation data between real and virtual realms is made to be easier, faster, more efficient and automated, more convenient and intuitive for users, and so forth.

[0024] Various implementations will now be described in more detail with reference to the figures. It will be understood that particular implementations described below are provided as non-limiting examples and may be applied in various situations. Additionally, it will be understood that other implementations not explicitly described herein may also fall within the scope of the claims set forth below. Systems and methods described herein for mixed reality annotation on a physical display screen may result in any or all of the technical effects mentioned above, as well as various additional effects and benefits that will be described and / or made apparent below.

[0025] FIG. 1 shows an illustrative mixed reality device 100 operating in accordance with principles described herein to perform mixed reality annotation on a physical display screen within an environment. As used herein, virtual annotation may refer to various ways in which a user may virtually write, sketch, underline, highlight, or otherwise annotate an object. As these types of virtual annotations are sometimes related to other forms of user input or computer control (e.g., moving a cursor, hand or finger gestures, selecting items, dragging and dropping, etc.), virtual annotation may also refer to these types of actions as well in certain implementations. Certain virtual annotations may be generated using an instrument especially configured for this purpose (e.g., a mixed reality controller, a virtual annotation pen, etc.) or using another more general -purpose tool (e.g., a finger or hand, a stylus or wand, etc.). Unlike physical annotation, however, which generally involves a writing instrument (e.g., a pen, pencil, marker, etc.) that can produce writing on a physical object (e.g., a sheet of paper, a whiteboard, etc.), virtual annotation has generally not produced an impact on real -world objects and has only been visible using a mixed realitydevice. As will be described in more detail below, mixed reality device 100 is therefore configured to help bridge the gap for virtual annotation to produce an effect with respect to real -world objects such as real -world screens (e.g., computer monitors, phone or tablet screens, television screens, projected images, etc.). To this end, various aspects of mixed reality device 100 and its operation within an environment will now be described.

[0026] As shown, mixed reality device 100 may include a processor 102 and a memory 104 storing instructions associated with a process 106. For example, processor 102 may represent one or more of any type of computer processor or processing resources configured to execute the instructions stored in memory 104 to thereby perform process 106. In some examples, process 106 may be loaded into memory 104 from a non -transitory computer-readable medium (not shown) storing instructions that, when executed, cause processor 102 to perform process 106.

[0027] As further shown in FIG. 1, mixed reality device 100 may be communicatively coupled with a controller 108 and may be located so as to have a view of an environment 110 (e.g., a physical environment in which a mixed reality session may take place) that includes various real-world objects 112. For example, mixed reality device 100 and controller 108 may be worn or carried by a user in a room that implements environment 110. While real- world objects 112 are represented in FIG. 1 as simple circles, it will be understood that these circles represent any real -world objects as may be present in a given type of environment in which mixed reality device 100 may operate. For example, if environment 110 were an office environment, real-world objects 112 may include objects like desks, desk chairs, walls or partitions between workspaces, objects on the desks, and so forth. Conversely, if environment 110 were a living room environment, real -world objects 112 may include objects like furniture (sofas, coffee tables, etc.), decorations, and so forth.

[0028] Whatever type of environment 110 mixed reality device 100 happens to be operating in, FIG. 1 shows that one of the real -world objects 112 may be a physical screen 114 that is associated with a physical device 116. For example, physical screen 114 could be the screen of a laptop computer, a tablet or phone device, or another device having a built-in screen. As another example, physical screen 114 may be associated with another physical device that is at least somewhat separate from the screen such as a computer monitor (implementing physical screen 114) that sits on a desk and is associated with a computer workstation device (implementing physical device 116) that is under the desk or otherwise out of sight. In still other examples, the physical device may be present and visible within environment 110 but may still be distinct from the physical screen 114. For instance, aprojector device implementing physical device 116 may project light onto a wall or projector screen that implements the physical screen 114 in this example and is separate from the projector device.

[0029] In operation (e.g., as process 106 is being performed), mixed reality device 100 is shown to have a field of view 118 into environment 110. This field of view 118 is shown to include some objects 112 in environment 110 and to exclude others, depending on how field of view 118 is directed (i.e., what part of the room it is facing, etc.). The field of view 118 into environment 110 may be provided by any suitable camera or scene capture device that is associated with mixed reality device 100 in any manner as may serve a particular implementation. If mixed reality device 100 includes or is implemented by a mixed reality head-mounted display device, for instance, field of view 118 may represent a field of view of the user wearing the head-mounted display device (as that field of view is captured and detected by one or more front-facing cameras and / or other sensors featured on the device). If mixed reality device 100 is implemented by another type of non-head-mounted device (e.g., a smartphone held at arm’s length, etc.), field of view 118 may represent a field of view captured by the device (e.g., by a back-facing camera of a smartphone) as the device is held and pointed in different directions within environment 110.

[0030] At the moment represented in FIG. 1, field of view 118 is shown to be directed toward various objects 112 within environment 110, one of which happens to be physical screen 114. As will be described and illustrated in more detail below, mixed reality device 100 may be configured, as part of a mixed reality session being presented by mixed reality device 100 within environment 110, to identify physical screen 114 of physical device 116 within the environment. For example, based on object recognition models, previous knowledge stored in memory 104 (if physical screen 114 has previously been identified), or other suitable data and / or techniques, mixed reality device 100 may detect that among the other real-world objects 112 in the field of view 118, one of the objects is a physical screen that is associated with a physical device to which mixed reality device 100 may communicate.

[0031] Mixed reality device 100 may further receive data 120 representing a virtual annotation generated within environment 110. For instance, as shown in this example, the data 120 representing the virtual annotation may be received by mixed reality device 100 from a controller 108 that is associated with mixed reality device 100. Controller 108 may be implemented as any suitable mixed reality control device, including, for example, a 6-degree- of-freedom (6DOF) pen controller specially configured to facilitate virtual annotation input ina comfortable and familiar manner similar to writing with a pen. In certain examples, data 120 representing the virtual annotation may be received or generated in other ways not involving any particular controller device. For instance, cameras or other sensors of mixed reality device 100 may be configured to monitor a particular non-controller object (e.g., a fingertip of the user, a wand or stylus held by the user, etc.) and generate data 120 representing a virtual annotation being made by the user using this object.

[0032] Upon identifying the presence of physical screen 114 within environment 110 and receiving (e.g., inputting, generating, etc.) the data 120 representing the virtual annotation, mixed reality device 100 be configured to determine that the virtual annotation is targeted to the physical screen 114. This determination may be made in a variety of ways and / or using a variety of different factors, as will be described and illustrated in more detail below. In some cases, this determination be based on or associated with one or more other determinations, such as that the annotation is targeted to a person and that the person is using or otherwise associated with the device (e.g., by holding the device, looking at the screen, pointing at the screen, having security permissions to operate the device, etc.). Additionally, this determination may include determining not only that a virtual annotation is targeted to physical screen 114 but also that the virtual annotation is not targeted to another virtual or physical screen within environment 110 (not explicitly shown in FIG. 1). For example, physical screen 114 may be one of several screens (physical and / or virtual screens) present, and mixed reality device 100 may determine, given data 120 of a particular virtual annotation, which of the various screens the annotation is targeted to.

[0033] In the event that mixed reality device 100 determines that the illustrated data 120 for this particular virtual annotation is targeted to physical screen 114, FIG. 1 further shows that mixed reality device 100 may transmit (e.g., based on this determining that the virtual annotation is targeted to the physical screen) the data 120 representing the virtual annotation to the physical device 116 to which physical screen 114 belongs. In this way, physical device 116 may direct physical screen 114 to display (e.g., in real time) the virtual annotation as it is being created and / or modified. Using the data 120 provided by mixed reality device 100, physical device 116 may also now store or otherwise manipulate the virtual annotation in any manner as may serve a particular implementation.

[0034] FIG. 2 shows an illustrative method 200 for mixed reality annotation on a physical display screen in accordance with principles described herein. For example, method 200 shows one sequence of operations that may be performed by mixed reality device 100 to implement process 106 described above. While FIG. 2 shows illustrative operations 202-206according to one implementation, other implementations of method 200 may omit, add to, reorder, and / or modify any of the operations 202-206 shown in FIG. 2. In some examples, multiple operations shown in FIG. 2 or described in relation to FIG. 2 may be performed concurrently (e.g., in parallel) with one another, rather than being performed sequentially as illustrated and / or described. Each of operations 202-206 of method 200 will now be described in more detail as the operations may be performed by an implementation of mixed reality device 100.

[0035] At operation 202, mixed reality device 100 may identify a physical screen of a physical device present in an environment during a mixed reality session. For example, while mixed reality device 100 is presenting the mixed reality session, a field of view associated with mixed reality device 100 may come to include a physical screen of a physical device (such as described above in relation to FIG. 1, where field of view 118 is shown to include physical screen 114 of physical device 116). The identifying of the physical screen at operation 202 may be performed using any suitable sensor data (e.g., data being received in real time from sensors of mixed reality device 100, stored sensor data that was captured previously, etc.), device data (e.g., data indicative of properties of physical devices known or anticipated to be within the environment, etc.), algorithms, machine learning models, techniques, and / or other resources as may serve a particular implementation. For instance, as mentioned above, an object recognition model trained to identify certain physical screens associated with certain physical devices could be employed in one example. Once a physical screen has been identified in one video frame, any suitable data and / or techniques may also be used thereafter to keep track of the identified device even as the field of view changes with respect to the identified screen. For example, certain object tracking techniques may be used to track the identified screen from frame to frame as the field of view changes.

[0036] At operation 204, mixed reality device 100 may receive data representing a virtual annotation generated within the environment (e.g., such as data 120 described above). The virtual annotation may be made by a user of mixed reality device 100 and data representing that annotation may be detected, generated, input, accessed, or otherwise received by mixed reality device 100. For example, the user may use a controller device (e.g., implementing a 6DOF pen or other controller device configured to facilitate creation of virtual annotations) and the data representing the virtual annotation may be received at operation 204 by inputting data from the controller device. In another example, no controller device may be employed for the detection of a particular virtual annotation; rather the annotation may be observed by mixed reality device 100 (e.g., by watching a fingertip of theuser drawing the annotation, producing an annotation associated with a detected gesture, etc.) and the data representing the virtual annotation may be received at operation 204 by mixed reality device 100 generating the data based on these observations.

[0037] At operation 206, mixed reality device 100 may transmit the data representing the virtual annotation to the physical device associated with the physical screen to which the annotation is targeted. For example, this transmission of the data representing the virtual annotation to the physical device may be performed based on a determination that the virtual annotation is targeted to the physical screen, or targeted to a person associated with the physical screen in any of the ways described herein. For example, as an additional operation not explicitly shown in method 200, mixed reality device 100 may determine that the virtual annotation for which the data was received at operation 204 is targeted to the physical screen identified at operation 202. While this determination could be relatively straightforward under certain circumstances, such as if only a singular screen is present in the environment and the virtual notation is created in a location or manner that shows clear intent to target that screen, it will be understood that the determination may be far less straightforward under other circumstances. For instance, mixed reality device 100 may encounter an environment in which more than one screen has been identified as a potential target of virtual annotations and the determination may hence involve designating one screen instead of any of the others to be the target of the virtual annotation. In some cases, the location or manner in which the user creates the virtual annotation may not clearly indicate the intended target device. Accordingly, as will be described in more detail below, successful performance of this determination operation may involve consideration and weighting of data associated with a variety of factors (e.g., rules, heuristics, etc.) that will be described in more detail below.

[0038] The data transmission of operation 206 may be made by way of any suitable networking or data transmission protocol and / or in accordance with any new or established data transmission standards. In some examples, for instance, mixed reality device 100 may establish, with the physical device, a wireless communicative connection (or link) such as a Bluetooth connection (e.g., using Bluetooth classic, Bluetooth Low Energy (BLE), etc.), a Wi-Fi connection (e.g., using Wi-Fi Direct, etc.), an ultra-wideband (UWB) connection, a Zigbee connection, a cellular connection, or the like. The mixed reality device may then transmit the data representing the virtual annotation to the physical device over that connection. In some examples, this data input to the physical device may be performed in a similar manner as may be used by an external peripheral device (e.g., a wireless mouse, trackpad, pen device, etc.) that could attach to the physical device to record similar types ofannotations and input from a user of the physical device.

[0039] FIG. 3 A shows illustrative aspects of an environment in which a mixed reality device may operate in accordance with principles described herein. More particularly, a view is shown of an example office-type environment that may implement the environment 110 described above (labeled environment 110 for this purpose). A few particular real -world objects 112 are shown in this example of environment 110. For example, among these objects are a table 112-1, a portion of a white-board 112-2 (including markers and an eraser on a tray at the bottom of the whiteboard), and a display monitor located on the table 112-1 and labeled as an implementation of physical screen 114. It will be understood that physical screen 114 may constitute an example of a physical screen 114 that is integrated into a physical device 116, or physical screen 114 may represent an example of a display monitor for a physical device that does not happen to be visible given the current field of view (e.g., a workstation or desktop tower hidden under the desk or otherwise not explicitly shown). While FIG. 3 A shows an office-type environment that includes office-type objects for purposes of this example, it will be understood that principles described herein may apply to a variety of use cases in a variety of different environments each featuring different types of objects including at least one physical screen. As all of the objects shown in FIG. 3 A are real -world objects, it will be understood that no mixed reality device need be used to view these objects in the state in which they are shown.

[0040] FIG. 3B, in contrast, shows certain aspects of how this same environment 110 may appear when viewed through an implementation of mixed reality device 100 after certain virtual annotations have been created. Specifically, as shown, an implementation of mixed reality device 100 is represented by a head-mounted display device that may be used in connection with an implementation of controller 108 to produce certain virtual annotations within environment 110. More particularly, controller 108 may be used by a user to create certain annotations 302 and 304 in the virtual world displayed by mixed reality device 100 during a particular mixed reality session. Data representing virtual annotations 302 and 304 may each be received by mixed reality device 100 as input from controller 108 or in other suitable ways as have been described.

[0041] Annotation 304 will be understood to represent an annotation that is limited to the virtual world (as is typical of conventional virtual annotations). For this reason, as shown, annotation 304 (displayed as handwritten text reading: “Annotation 2”) is visible on the whiteboard when viewed through mixed reality device 100 but is absent when viewed without the aid of a mixed reality device. This absence is highlighted in FIG. 3B by an emptydashed-line box 306 in the location on white-board 112-2 where annotation 304 is visible through the head-mounted display. While content of annotation 304 may be viewed, stored, and / or otherwise used by way of mixed reality device 100, annotation 304 does not have any actual connection to white-board 112-2 or to any other real -world object 112 and thus cannot be experienced or used outside of the virtual world in which it was created.

[0042] In contrast, virtual annotation 302 will be understood to represent an annotation that is not constrained to the virtual world in this same way, but, rather, can be experienced and used in both the virtual and real worlds associated with environment 110. In other words, mixed reality device 100 may receive and transmit data representing a virtual annotation (e.g., data 120 described above) in a manner that allows a physical device controlling physical screen 114 to present the virtual annotation in substantially real time for viewing within environment 110 without aid of any mixed reality device. To illustrate, annotation 302 (displayed as handwritten text reading: “Annotation 1” in this example) is shown to be visible on the physical screen 114 of the display monitor not only when viewed within a virtual world presented by mixed reality device 100, but also when physical screen 114 is viewed without the aid of mixed reality device 100 (as shown by the depiction outside the view of mixed reality device 100). The methods and systems described herein therefore may provide a way for a virtual annotation generated using controller 108 to be automatically and immediately transmitted to a real-world device so that the annotation can be presented for mixed reality and real-world (non-mixed-reality-assisted) viewing in substantially real time (e.g., as the annotation is being created, even if after a short delay of a few seconds or less).

[0043] As has been mentioned, one way that data representing a virtual annotation (e.g., either of virtual annotations 302 or 304) may be received is by way of input from a physical controller such as controller 108. More particularly, the data representing the virtual annotation may be generated by a physical controller communicatively coupled to mixed reality device 100. In other examples, as has also been mentioned, virtual annotations such as virtual annotations 302 and / or 304 may be received in other ways. For instance, instead of using a physical controller, a user may create a virtual annotation using hand gestures, drawing with a finger, using another instrument (e.g., a wand or stylus that is visually monitored by mixed reality device 100 but is not a controller in communication with mixed reality device 100), or in other suitable ways. In any of these examples, the data representing the virtual annotation may be received by mixed reality device 100 by being generated, input, accessed, or otherwise obtained in any manner as may serve a particular implementation.

[0044] In some examples, a physical controller may be specially adapted to provideefficient assistance with producing virtual annotations. For example, the physical controller could function as a standard, non-annotation-related controller when in a controller mode, and then may be configured to generate the data representing the virtual annotation while in an annotation mode triggered by the physical controller being held in an annotation grip. As long as the user is holding controller 108 with both hands or with a thumb near the buttons, etc., controller 108 may be considered to be in a controller grip and may be configured to serve as a standard mixed reality controller. When the user holds controller 108 in an annotation grip (e.g., the way one would hold a pen, etc.), however, the physical controller may be configured to automatically switch to the annotation mode to facilitate the entering of a virtual annotation. As used herein, the annotation grip refers to a particular way that the user may hold controller 108 that is electronically detectable by the controller (e.g., by way of one or more sensors included in the controller to detect hand placement, controller orientation, etc.). In some examples, in the annotation mode, controller 108 may operate as a 6DOF pen that can be used to virtually write on a two-dimensional (2D) surface (e.g., a real 2D surface such as a tabletop or screen, a virtual 2D surface presented in the air in front of the user, etc.).

[0045] While a particular field of view of environment 110 is illustrated to be presented by mixed reality device 100 in FIG. 3B (i.e., the field of view showing table 112-1 with physical screen 114 and showing part of white-board 112-2), it will be understood that this field of view may be in constant flux as a user of mixed reality device 100 freely moves mixed reality device 100 and directs it toward different objects in the environment. For example, if mixed reality device 100 is implemented as a head-mounted display device worn by the user, the field of view may change responsive to the turning of the user’s head within the environment. Accordingly, mixed reality device 100 may be configured not only to identify objects such as physical screen 114 once, but also to track them from moment to moment during a mixed reality session (i.e., from frame to frame of content being presented in the mixed reality session). To this end, 100 may be configured to track physical screen 114, once the physical screen is identified, by determining and tracking boundaries of the physical screen with respect to a coordinate space (e.g., a coordinate space for the environment that is used by the mixed reality device to add annotation and / or other augmented content to the world). Once the boundaries of physical screen 114 are detected using suitable feature detection and / or object detection algorithms and models, suitable correspondence algorithms and / or models may be employed to track the corners and boundaries of the physical screen 114, such as by way of image data captured by a front-facing camera of mixed reality device 100.

[0046] Tracking a physical screen such as physical screen 114 within an environment may allow for operations described herein to be performed in a gradual and / or continuous manner. For example, along with the possibility of receiving a virtual annotation in its entirety and then transmitting the entirety of the virtual annotation data to a targeted device, mixed reality device 100 may also be configured to continuously receive and transmit data representing a virtual annotation to a targeted device even as the data is being generated (i.e., as the user is in the process of writing, drawing, gesturing, and / or otherwise creating the annotation). In this way, physical screen 114 may present the annotation as it is in the process of being created (e.g., stroke by stroke as the user writes or draws, etc.) and the user may perceive that the user is annotating the physical screen using a virtual annotation tool in real time similarly as the user might annotate a physical object with a physical writing tool (e.g., using a pen on paper, a marker on a whiteboard, etc.). This intuitive annotation effect may be produced and facilitated by a smooth and continual tracking of the bounds of the physical screen throughout the mixed reality session, or at least while the physical screen is within the field of view of the mixed reality device and / or while the virtual annotation is in the process of being created and for as long as the annotation is expected to be visible (e.g., prior to being stored away and cleared to make room for another annotation, etc.).

[0047] Another consequence of tracking a physical screen within the environment in the ways described above is that virtual annotations may be displayed on the physical screen in a natural and convenient way (e.g., at a suitable projection angle, with a suitable size, etc.) that may not necessarily line up with the way that the screen is perceived in the field of view throughout the session. For example, if a user’s vantage point on a physical screen is such that the screen is posed at an angle relative to the user as the user creates an annotation targeted to that screen, the tracking of the comers of the screen may enable mixed reality device 100 to reproject the virtual annotation so as to look natural when the screen is viewed straight on (e.g., rather than being angled in the way that the user may actually see the screen from the user’s present vantage point). Similarly, mixed reality device 100 may determine a natural size for the virtual annotation on the physical screen that may be, but need not be, the actual size produced by the user. For example, the size at which the annotation is presented may be determined based on the distance of the physical screen from the user, the size of the physical screen and / or the software application executing thereon, and so forth. In some examples, the pose of the screen with respect to the coordinate system may be determined (e.g., based on the tracking of the screen’s corners and other features, etc.) and the annotationmay be recorded with respect to a 2D plane that is mapped onto the plane of the screen. For example, when a controller is held in an annotation grip, the controller may enter an annotation mode in which annotations are made with respect to a 2D plane that can be projected onto the plane of the physical screen without regard for how the screen may be posed relative to the field of view of mixed reality device 100.

[0048] In some applications or use cases, it might be desirable for virtual annotations not only to be presented and / or stored by a particular physical device (i.e., for presentation on a particular physical screen such as the physical screen 114 shown in FIGS. 3 A-3B), but further for the virtual annotations to be presented and / or stored in connection with particular software applications executing on the physical device. To this end, the transmitting of data representing the virtual annotation performed by a mixed reality device such as mixed reality device 100 may include directing the data to a software application executing on the physical device, where the software application is configured to perform at least one of storing or presenting the virtual annotation.

[0049] To illustrate, FIG. 4 shows a close-up view of physical screen 114 to show several example software applications 400 that may be executing on the physical device and to which example virtual annotations may be directed. Specifically, as shown, a first software application 400-1 (labeled “Software Application 1”) will be understood to be executing on the physical device such that a user interface for the application is presented on physical screen 114. User interfaces for other software applications 400-2 (labeled “Software Application 2”) and 400-3 (labeled “Application 3”) are similarly depicted on the display of physical screen 114 in this example. As shown, when the user creates certain annotations 402 (handwritten text reading “Annotation 1”), 404 (handwritten text reading “Annotation 2”), and 406 (handwritten text reading “Ann. 3”), these annotations may not only be targeted for the device of physical screen 114 but also may further be targeted to particular software applications 400. For example, as shown, virtual annotation 402 may be targeted specifically to software application 400-1, virtual annotation 404 may be targeted specifically to software application 400-2, and virtual annotation 406 may be targeted specifically to software application 400-3.

[0050] In one example mixed reality session, for instance, software application 400-1 may be implemented by an art application, software application 400-2 may be implemented by a document editor application, and software application 400-3 may be implemented by a slide presentation application. In this example, each of the annotations 402, 404, and 406 may thus be appropriate for the type of work being done with the application to which it istargeted. For instance, annotation 402 may include a sketch or a set of strokes associated with a painting or drawing being created in the art application; annotation 404 may include highlights, underlining, a signature, margin notes, etc., as may be produced by the user as he or she reviews a document in the document editor application; and annotation 406 may include notes and revisions for a slide presentation being drafted using the slide presentation application.

[0051] Each software application 400 may be configured to receive input by way of one or more application programming interfaces (APIs), such as APIs for mouse, keyboard, touchscreen, or other input provided by an operating system of the physical device. One way to differentiate which software application 400 is to receive a given virtual annotation being generated by the user is to use these APIs and to provide the data representing the virtual annotation to whichever software application 400 is presently active (in a similar way as a software application might receive a mouse click or a pen stroke on an external touch screen device). In this manner, virtual annotations provided by way of mixed reality device 100 may utilize established input / output pathways already being used by the physical device and mixed reality device 100 may be configured to automatically set itself up to serve as an external peripheral (similar to accessories such as physical keyboards, writing pads, trackpads, mice, etc.) to the physical device. For example, the physical device executing software applications 400 may link to mixed reality device 100 using a Bluetooth or other suitable link and may process the data representing the virtual annotations in the same way as data representing peripheral input (e.g., trackpad movements, mouse movements, button clicks, etc.) would be processed.

[0052] Consistent with this input / output framework, it will be understood that data described herein to represent virtual annotations may not be limited merely to annotations in the relatively narrow sense of writing, sketches, and other marks made using a pen-like object. While certain implementations may relate only to annotations in this narrower sense, other implementations may involve annotations implemented by gestures intended to perform control tasks beyond strictly writing or sketching. For example, virtual annotations could be implemented by control gestures intended to move a cursor, select one or more items displayed on the physical screen 114, drag or otherwise manipulate the selected items, and / or otherwise perform user interfacing tasks in a similar manner as may be performed using a mouse, trackpad, digital pen, or the like.

[0053] Example scenarios illustrated above, such as the scenario illustrated in FIGS. 3 A and 3B, involved a singular physical screen 114 that may receive virtual annotations inaccordance with mixed reality devices and methods described herein. In such examples, the determination of whether a received virtual annotation is targeted to the physical screen is a determination of whether the virtual annotation is targeted to the screen or is not (e.g., since the annotation could be targeted to another object in the environment or to no object). Additional complexity may be introduced, however, when there are a plurality of candidate physical and / or virtual screens that could potentially be targeted for a particular virtual annotation. For example, along with identifying the physical screen present in the environment during the mixed reality session (as described at operation 202 of method 200), mixed reality device 100 may be further configured to identify, during the mixed reality session, an additional screen present in the environment. For example, the additional screen may be an additional physical screen of an additional physical device present in the environment or may be a virtual screen being presented within the environment by way of the mixed reality device. In some cases, multiple additional screens (e.g., including one or more of each screen type, physical and virtual) may be identified. In these cases, the determining of where the virtual annotation is targeted (see operation 206 of method 200) may yield a determination (sometimes referred to herein as a target determination or stroke assignment) that the virtual annotation is targeted to the physical screen and not to (instead of to) the additional screen. In other words, the determination may include determining not only a screen to which the virtual annotation is targeted but also one or more screens to which the virtual annotation is not targeted, since this example includes a plurality of possible target screens.

[0054] To illustrate this type of scenario, FIG. 5 shows illustrative aspects of environment 110 in a scenario where the environment includes a plurality of screens as the environment is viewed during a mixed reality session. Specifically, as shown in FIG. 5, environment 110 in this example still includes table 112-1 and the physical screen 114 illustrated and described above in relation to FIGS. 3A and 3B. However, as further shown, environment 110 in this example also includes two additional screens that were not present in the previous example. As a first addition, a physical screen 502 implemented as a smaller device (e.g., a smartphone device, a tablet device, etc.) is shown to be present in the environment, laying on table 112-1. As a second addition, a virtual screen 504 implemented as a large screen behind table 112-1 and physical screen 114 is also shown to be present in this scenario. For example, virtual screen 504 may present a larger projection of the same content being displayed on physical screen 114, may serve as a second virtual monitor to display content being presented by the physical device associated with physical screen 114(physical device 116, not explicitly shown), may serve as a virtual monitor to display content generated by mixed reality device 100, or may serve another suitable role during the mixed reality session.

[0055] It will be understood that the view of environment 110 shown in FIG. 5 is the mixed reality view that may be provided by a mixed reality device such as mixed reality device 100. As such, certain real-world objects such as table 112-1 and physical screens 114 and 502 are visible in the mixed reality view along with certain virtual augmentations such as virtual screen 504. As shown, certain real -world objects may also be obscured, blocked, deemphasized, etc., as a result of the virtual objects. For instance, the white-board 112-2 shown to be present in environment 110 in FIGS. 3A and 3B is not visible in this view since the whiteboard is covered by virtual screen 504.

[0056] Before received annotations can be correlated and transmitted to certain targets, mixed reality device 100 may first determine the set of candidate screen targets in the environment by identifying physical screen 114 and each of the additional screens (i.e., additional screens 502 and 504 in this example). The identification of any virtual screens (e.g., such as virtual screen 504) may be trivial for mixed reality device 100 since the virtual screens are virtual constructions of mixed reality device 100 itself. However, there may still be a question about whether a given virtual screen is a potential target of virtual annotations, since mixed reality device 100 could generate certain virtual screens that are not intended to be annotated. Accordingly, mixed reality device 100 may determine whether a given virtual screen such as virtual screen 504 is selected (e.g., based on stored settings, user preference parameters, etc.) as an annotation target and, if it is, include the virtual screen in the set of candidate targetable screens.

[0057] The identification of physical screens (e.g., such as physical screen 114 and physical screen 502) within the environment may involve a more careful analysis, since these may or may not have any particular relationship or connection to mixed reality device 100 or otherwise be known to exist. The discovery and identification of physical screens in the environment (e.g., to construct a set of candidate targetable screens for virtual annotations that may be received) may hence be performed in a variety of ways as may serve a particular implementation.

[0058] As a first example, a screen discovery process may be performed by scanning the environment to identify devices that are present and capable of connecting to the mixed reality device to receive virtual annotations during the mixed reality session. In this type of implementation, the identifying of each physical screen may be based on scan data generatedby a device scan configured to discover, within the environment, candidate targetable screens for virtual annotations. For example, mixed reality device 100 may use Bluetooth, Wi-Fi, or other wireless communication technologies to scan the environment for devices that are present and could be connected to mixed reality device 100. For each physical screen discovered in this way, mixed reality device 100 may access (e.g., based on stored settings, based on prompting the user for input, etc.) selection data representing a selection of the physical screen as a targetable screen for virtual annotations. In this way, the user may exert a degree of control over which screens and devices serve as candidate targetable screens for virtual annotations. For example, if the user intends to annotate a physical computer monitor and a physical tablet on a desk the user is working at while not intending to annotate a television screen that also happens to be in the room, all three devices may be discovered but only the monitor and the tablet may be included in the set of candidate targetable screens while the television may be excluded as per the user’s preference. For instance, the user may explicitly direct mixed reality device 100 to connect to the monitor and the tablet while abstaining from directing mixed reality device 100 to make such a connection to the television. In another implementation, mixed reality device 100 may present the user a list of all three discovered devices and the user could make a selection of only the two that the user intends to annotate.

[0059] As a second example, an automatic screen connection process may be employed as either an alternative to the screen discovery process described above or as a convenient default process to recreate the set of candidate targetable screens for subsequent mixed reality sessions after a first session in which the screen discovery process is performed. In this example, the identifying of each physical screen may be based on connection data representing a previous connection of mixed reality device 100 to the physical device, as well as selection data representing a previous selection of the physical screen as a targetable screen for virtual annotations. This type of approach may be convenient for a user who often works in the same environment with the same combination of physical and virtual screens. Each time that the user initiates a mixed reality session, connection data indicative of a previous connection of a physical device to the mixed reality device 100 (e.g., Bluetooth connection data, Wi-Fi connection data, LAN connection data, etc.) may be accessed and used to connect to the desired physical devices based on stored selection data that indicates which devices have been of interest to the user for annotations in previous sessions.

[0060] In some cases, information about the devices that are present in the room (e.g., that have been discovered and / or connected to using the processes described above) mayassist mixed reality device 100 in identifying and tracking the proper screens. For example, if mixed reality device 100 has identified that the room includes a large television, a midsize computer monitor, and a relatively small tablet device, this information may be useful as screens are detected and recognized in the field of view and mixed reality device 100 attempts to correlate identified screens with particular devices that it may or may not be connected to.

[0061] In FIG. 5, each of the screens present in mixed reality device 100 is shown to be presenting a virtual annotation received from mixed reality device 100. Specifically, physical screen 114 is shown to be presenting an annotation 506 (handwritten text reading: “Annotation 1”), virtual screen 504 is shown to be presenting an annotation 508 (handwritten text reading: “Annotation 2”), and physical screen 502 is shown to be presenting an annotation 510 (handwritten text reading: “Ann. 3”). It will be understood that each of these annotations 506, 508, and 510 may have been created by a user during a mixed reality session and then determined to be targeted to the different respective screens and transmitted to the devices accordingly. A determination that a particular annotation is targeted to one screen instead of to another (e.g., to physical screen 114 and not to physical screen 502, to virtual screen 504 and not to physical screen 114, etc.) may be a more difficult and complex determination to make than whether an annotation is targeted to a singular screen in the environment or field of view. Moreover, the complexity of this determination may grow with the number of candidate targetable devices that may be in play for a particular environment 110 and / or a particular mixed reality session. Accordingly, the determination that a virtual annotation is targeted to one screen and not to another may be based on a variety of considerations, rules, heuristics, and / or other factors as may serve a particular implementation. A few examples of illustrative factors that may be considered as a determination is made that a virtual annotation is targeted to one screen in the environment instead of to another will be described and illustrated with reference to FIGS. 6A-6C.

[0062] Additionally, it will be understood that in some cases, an annotation could be targeted to more than one screen (or device). For example, if mixed reality device 100 may identify, during the mixed reality session, an additional screen present in the environment, and may then determine that the virtual annotation is targeted to both the physical screen (e.g., physical screen 114) and to the additional screen (e.g., physical screen 502, virtual screen 504, etc.). As such, mixed reality device 100 may further be configured to transmit the data representing the virtual annotation to a device associated with the additional screen (e.g., to the mobile device of physical screen 502, to a device responsible for virtual screen 504,which may be mixed reality device 100, etc.).

[0063] In some cases, the targeting to two different screens may occur at the same time. For example, a user may create an annotation targeted to both screens and desire that it be shown on both screens at the same time. Accordingly, the transmitting of the data to the physical device and the transmitting of the data to the device associated with the additional screen may be performed to allow the physical screen and the additional screen to present the virtual annotation simultaneously. In other cases, the targeting to two different screens may occur at different times. For example, a user may create an annotation targeted to one screen and then wish to move it to a different screen. Thus, in this case, the determining that the virtual annotation is targeted to the physical screen may occur at a different time (e.g., prior to, subsequent to, etc.) the determining that the virtual annotation is targeted to the additional screen. The transmitting of the data representing the virtual annotation to the device associated with the additional screen may then include transferring the data representing the virtual annotation from the physical device to the device associated with the additional screen (e.g., transferring the annotation from the physical device associated with physical screen 114 to the mobile device with physical screen 502, etc.).

[0064] In each of FIGS. 6A-6C, an implementation of mixed reality device 100 (shown as a head-mounted display device similar to the implementation illustrated in FIG. 3B, though it will be understood that similar principles would apply to other form factors and types of mixed reality devices) is shown together with two candidate targetable screens 602-1 and 602-2. The candidate targetable screens 602-1 and 602-2 may each represent any of the physical or virtual screens that have been illustrated or described in examples herein. For instance, screen 602-1 may represent the physical screen 114 described in examples above, while screen 602-2 may represent physical screen 502, virtual screen 504, or another suitable screen that may be present in the environment. While only two candidate targetable screens 602 are shown in the examples of FIGS. 6A-6C, it will be understood that the principles described in these examples may be used to facilitate target determinations between however many candidate targetable screens may be identified in a given environment (e.g., including three or more screens in some cases).

[0065] Each of figures 6A-6C illustrates a different factor that may be accounted for by mixed reality device 100 as a target determination between the two candidate targetable screens is performed. However, while each of these factors is described and illustrated separately in the following description, it will be understood that a combination of multiple factors (including the factors described below and / or other suitable factors as may serve aparticular implementation) may be considered and used to make the targeting determination in certain implementations. For example, confidence scores and / or weighting techniques may be used to combine determinations made on the basis of each of these factors into a final determination (e.g., a final target determination that is based on a confidence-weighted analysis of multiple factors).

[0066] In FIG. 6A, a first factor for target determination (also referred to as stroke assignment) is illustrated that accounts for relative locations (illustrated as four-pointed stars located approximately at the center of the objects of interest) of the candidate targetable screens and where the virtual annotation is created. More particularly, as shown, a virtual annotation may be generated within the environment (e.g., using by controller 108) at an annotation location 604- A. A determination that the virtual annotation is targeted to screen 602-1 instead of to screen 602-2 may then be based on: 1) a first proximity 606-1 between annotation location 604-A and a first screen location 608-1 where screen 602-1 is located, and 2) a second proximity 606-2 between annotation location 604-A and a second screen location 608-2 where screen 602-2 is located.

[0067] This location-based factor may be useful for distinguishing which screen is intended to be targeted since a user may intuitively create a virtual annotation near (in close proximity to) or at least with a direct line of sight to the screen that the user intends to annotate. For instance, in some cases, the user may write or draw directly on screen 602-1 with controller 108 (e.g., in the annotation grip), such that mixed reality device 100 may determine that it is very likely that the annotation is intended for screen 602-1 (instead of screen 602-2) based on that proximity. In other examples, the annotation may be created at some distance from either screen, but the location or proximity of annotation location 604-A may still be used as an indicator which screen is to be targeted, since the annotation may be made with a line of sight to the intended screen (or may at least be made in closer proximity to the intended screen than to the unintended one).

[0068] In the example of FIG. 6A, first proximity 606-1 is shown to be considerably smaller than second proximity 606-2 since annotation location 604-A is closer to first screen location 608-1 than to second screen location 608-2. Moreover, it is noted that annotation location 604-A is directly on the line of sight between mixed reality device 100 and screen 602-1 (and significantly outside the line of sight between mixed reality device 100 and screen 602-2), providing further location-based evidence that the user likely intends for the virtual annotation to be targeted at screen 602-1 instead of screen 602-2.

[0069] In some implementations, location-based target determination for a particularvirtual annotation may account for the live-tracked boundaries of each of the candidate screens present in the environment. For example, a convex hull operation to the boundaries may be used to determine which convex hull (which of the boundaries of the various screens) contains the largest percentage of a given annotation, and to determine that the screen associated with that convex hull may be the target. In some cases, mixed reality device 100 may subtract out all the convex hulls defined by the screen from a universal set so as to leave only background objects (such as table 112-1). If the bulk or entirety of a virtual annotation is outside of the convex hulls of all the screens, mixed reality device 100 could be configured to revert to a conventional environment annotation mode in which real -world objects 112 are annotated and received virtual annotations are not assigned to any physical or virtual screens.

[0070] In FIG. 6B, a second factor for target determination (or stroke assignment) is illustrated that accounts for the field of view of mixed reality device 100 at the time a virtual annotation is created. More particularly, as shown, mixed reality device 100 may present a field of view 118-T at a time T when the virtual annotation is generated. This field of view 118-T may represent at least approximately where the user was looking during time T as the user produced the annotation. The target determination may then be based on a position of field of view 118-T with respect to both the first screen location 608-1 where screen 602-1 is located and with respect to the second screen location 608-2 where screen 602-2 is located.

[0071] This field of view-based factor may be useful for distinguishing which screen is intended to be targeted since a user may tend to look at the annotation they are creating even if their hand is elsewhere as it produces the annotation. For example, it may be useful to write on a surface (e.g., a desktop or tabletop, a book held on a lap, etc.) that can provide desirable haptic feedback to the writing instrument (e.g., controller 108 in these examples). As such, a user could have multiple screens before them on a desk but always keep their hand in a similar spot on the desk as they make annotations. This mode of operation may be analogous to how a user may use a mouse or trackpad in a stationary location while moving their head to look at different external monitors on a desktop. In this type of set up, the location-based factor illustrated in FIG. 6A may be less useful for the target determination since the annotation location and screen locations may all be relatively static regardless of which screen is targeted for any given annotation. Detecting where the user is looking, on the other hand, may be highly telling in this example of where the user intends for a virtual annotation to be assigned.

[0072] Indeed, as shown in the example of FIG. 6B, field of view 118-T is clearly shown to be directed in the direction of first screen location 608-1, indicating that the user islikely looking at screen 602-1 at time T as the annotation is created. Even though an annotation location 604-B where the annotation is created in this example is neither particularly proximate to, nor in the line of sight of, screen 602-1, field of view 118-T may nevertheless provide evidence that the user likely intends for the virtual annotation to be targeted at screen 602-1 instead of screen 602-2. It will be understood that other gaze-based information related to field of view 118-T may also be accounted for in a similar way. For instance, along with the pose and / or field of view of mixed reality device 100 with respect to the respective screen locations 608-1 and 608-2, mixed reality device 100 may also detect a gaze of the user (where the user’s eyes are looking while producing a virtual annotation) to further attempt to determine the user’ s intent.

[0073] In FIG. 6C, a third factor for target determination (or stroke assignment) is illustrated in which mixed reality device 100 may account for the size (and / or other characteristics) of a virtual annotation with respect to relevant attributes of the candidate targetable devices. More particularly, as shown, controller 108 may be used to create an annotation with a particular size 610. The target determination may then be based on the size 610 of the annotation with respect to a first size 612-1 of screen 602-1 and with respect to a second size 612-2 of screen 602-2. To emphasize the difference in sizes, this example shows screen 602-1 as a relatively small screen (i.e., first size 612-1 is shown to be a small size) while showing screen 602-2 as a relatively large screen (i.e., second size 612-2 is shown to be a large size).

[0074] This size-based factor may be useful for distinguishing which screen is intended to be targeted since a user may tend to produce smaller and more carefully drawn annotations for smaller screens and to produce larger and more freely drawn annotations for larger screens. For example, referring to the devices illustrated in FIG. 5, annotations intended for physical screen 502 may be written with small, careful handwriting so that they will fit on the relatively small screen. Conversely, annotations intended for virtual screen 504 may involve large strokes of the entire hand or arm since so that the annotations will be conspicuous on the relatively large screen. Other characteristics (besides size 610) of a given annotation could also be used in a similar way.

[0075] As shown in the example of FIG. 6C, size 610 of the virtual annotation is shown to be relatively large in comparison to first size 612-1 of screen 602-1, though size 610 would appear to fit comfortably on screen 602-2 given the larger second size 612-2. Accordingly, an analysis of these sizes by mixed reality device 100 may be considered to yield evidence that the virtual annotation of FIG. 6C is intended for screen 602-2 instead ofscreen 602-1.

[0076] While a few specific target determination factors have been described, it will be understood that other context clues not explicitly described in relation to these factors may also be used. As one example, information from previous mixed reality sessions may be accounted for. For instance, if a user has been observed to heavily favor annotations on one screen over other screens, a strong presumption may be given to that screen in future target determinations. As another example, continuity during a session may be accounted for. For instance, if a user has recently been annotating one screen, that screen may be considered more likely than others to be the target of other virtual annotations. This may reduce the risk that annotations will be wrongly attributed to unintended screens during a session and may involve detecting especially deliberate or exaggerated user actions to determine an intent to switch from annotating one screen to another. As yet another example, the determination of which device an annotation is targeted to may be based on a form factor of the physical device. For example, if the mixed reality session is associated with a presentation and the annotation is created in this context, it may be more likely that the annotation is targeted to one device form factor (e.g., a large screen on which the presentation is given) than to another device form factor (e.g., a small mobile device that happens to be sitting next to the large screen). Similarly, if the mixed reality session is associated with gaming and the annotation is created in this context, it may be more likely that the annotation it targeted to a television form factor (where the game is being presented) than to a computer monitor form factor (e.g., a work laptop that does not have games installed on it).

[0077] Additionally, while the examples described herein have largely focused on determining which screen or screens a particular annotation is targeted to, it will also be understood that annotations may be targeted to other things, and / or targeted to screens in indirect ways (e.g., by being directed to something that is itself associated with the screen). As one example, a mixed reality device may be configured to identify, during a mixed reality session, a person present in the environment. The mixed reality device may also determine that the virtual annotation is targeted to the person. In this case, the transmitting of the virtual annotation to the physical screen could then be based on this determining that the virtual annotation is targeted to the person. This is because the person themselves may be associated with the physical screen in some way. For example, after determining who the annotation is meant for, the mixed reality device may select an appropriate device (e.g., the device closest to that person or identified with that person). In some examples, the mixed reality device may present the virtual annotation to a device that is tied to the person making the annotation,such that a second person may present their own virtual annotations to a second device.

[0078] To select the device or physical screen based on a person to whom the virtual annotation is targeted, the mixed reality device may consider various factors such as which device or screen is closest to the person making the annotation, which device or screen the person is looking at or pointing at, which device the person has permissions to use, or other such considerations. More particularly, the transmitting may be further based on at least one of: 1) a proximity between a location of the person and a screen location where the physical screen is located; 2) detecting that the person is using the physical device; 3) detecting that the person has a security permission to use the physical device; 4) detecting a direction in which the person is looking within the environment; 5) detecting a direction in which the person is pointing within the environment; or another suitable consideration.

[0079] As has been mentioned, various methods and processes described herein may be implemented at least in part as instructions embodied in a non-transitory computer- readable medium and executable by one or more computing devices. In general, a processor (e.g., a microprocessor) receives instructions, from a non-transitory computer-readable medium (e.g., a memory, etc.), and executes those instructions, thereby performing one or more operations such as the operations described herein. Such instructions may be stored and / or transmitted using any of a variety of known computer-readable media.

[0080] A computer-readable medium (also referred to as a processor-readable medium) includes any non-transitory medium that participates in providing data (e.g., instructions) that may be read by a computer (e.g., by a processor of a computer). Such a medium may take many forms, including, but not limited to, non-volatile media, and / or volatile media. Non-volatile media may include, for example, optical or magnetic disks and other persistent memory. Volatile media may include, for example, dynamic random-access memory (DRAM), which typically constitutes a main memory. Common forms of computer- readable media include, for example, a disk, hard disk, magnetic tape, any other magnetic medium, a compact disc read-only memory (CD-ROM), a digital video disc (DVD), any other optical medium, random access memory (RAM), programmable read-only memory (PROM), electrically erasable programmable read-only memory (EPROM), FLASH- EEPROM, any other memory chip or cartridge, or any other tangible medium from which a computer can read.

[0081] FIG. 7 shows an illustrative computing system 700 that may be used to implement various devices and / or systems described herein. For example, computing system 700 may include or implement (or partially implement) mixed reality devices such as mixedreality device 100 and / or any components thereof.

[0082] As shown in FIG. 7, computing system 700 may include a communication interface 702, a processor 704, a storage device 706, and an input / output (I / O) module 708 communicatively connected via a communication infrastructure 710. While an illustrative computing system 700 is shown in FIG. 7, the components illustrated in FIG. 7 are not intended to be limiting. Additional or alternative components may be used in other embodiments. Components of computing system 700 shown in FIG. 7 will now be described in additional detail.

[0083] Communication interface 702 may be configured to communicate with one or more computing devices. Examples of communication interface 702 include, without limitation, a wired network interface (such as a network interface card), a wireless network interface (such as a wireless network interface card), a modem, an audio / video connection, and any other suitable interface.

[0084] Processor 704 generally represents any type or form of processing unit capable of processing data or interpreting, executing, and / or directing execution of one or more of the instructions, processes, and / or operations described herein. Processor 704 may direct execution of operations in accordance with one or more applications 712 or other computerexecutable instructions such as may be stored in storage device 706 or another computer- readable medium.

[0085] Storage device 706 may include one or more data storage media, devices, or configurations and may employ any type, form, and combination of data storage media and / or device. For example, storage device 706 may include, but is not limited to, a hard drive, network drive, flash drive, magnetic disc, optical disc, RAM, dynamic RAM, other non-volatile and / or volatile data storage units, or a combination or sub-combination thereof. Electronic data, including data described herein, may be temporarily and / or permanently stored in storage device 706. For example, data representative of one or more executable applications 712 configured to direct processor 704 to perform any of the operations described herein may be stored within storage device 706. In some examples, data may be arranged in one or more databases residing within storage device 706.

[0086] I / O module 708 may include one or more VO modules configured to receive user input and provide user output. One or more VO modules may be used to receive input for a single virtual experience. I / O module 708 may include any hardware, firmware, software, or combination thereof supportive of input and output capabilities. For example, I / O module 708 may include hardware and / or software for capturing user input, including, but not limitedto, a keyboard or keypad, a touchscreen component (e.g., touchscreen display), a receiver (e.g., an RF or infrared receiver), motion sensors, and / or one or more input buttons.

[0087] I / O module 708 may include one or more devices for presenting output to a user, including, but not limited to, a graphics engine, a display (e.g., a display screen), one or more output drivers (e.g., display drivers), one or more audio speakers, and one or more audio drivers. In certain embodiments, I / O module 708 is configured to provide graphical data to a display for presentation to a user. The graphical data may be representative of one or more graphical user interfaces and / or any other graphical content as may serve a particular implementation.

[0088] The following examples describe systems and methods for mixed reality annotation on a physical display screen in accordance with principles described herein:

[0089] 1. A method comprising: identifying, during a mixed reality session presented by a mixed reality device within an environment, a physical screen of a physical device present in the environment; receiving data representing a virtual annotation generated within the environment; and transmitting the data representing the virtual annotation to the physical device.

[0090] 2. The method of any of the preceding examples, wherein the receiving and the transmitting of the data representing the virtual annotation are performed to allow the physical device to present the virtual annotation in real time for viewing within the environment and without aid of the mixed reality device.

[0091] 3. The method of any one of any of the preceding examples, further comprising: identifying, during the mixed reality session, an additional screen present in the environment; and determining that the virtual annotation is targeted to the physical screen and not to the additional screen.

[0092] 4. The method of any of the preceding examples, wherein: the virtual annotation is generated within the environment at an annotation location; and the determining is based on: a first proximity between the annotation location and a first screen location where the physical screen is located, and a second proximity between the annotation location and a second screen location where the additional screen is located.

[0093] 5. The method of any one of any of the preceding examples, wherein: the mixed reality device presents a field of view when the virtual annotation is generated; and the determining is based on a position of the field of view with respect to a first screen location where the physical screen is located and with respect to a second screen location where the additional screen is located.

[0094] 6. The method of any one of any of the preceding examples, wherein the determining is based on a size of the virtual annotation with respect to a first size of the physical screen and with respect to a second size of the additional screen.

[0095] 7. The method of any one of any of the preceding examples, wherein the determining is based on a form factor of the physical device.

[0096] 8. The method of any one of any of the preceding examples, wherein the additional screen is an additional physical screen of an additional physical device present in the environment.

[0097] 9. The method of any one of any of the preceding examples, wherein the additional screen is a virtual screen being presented within the environment by way of the mixed reality device.

[0098] 10. The method of any one of any of the preceding examples, further comprising: identifying, during the mixed reality session, an additional screen present in the environment; determining that the virtual annotation is targeted to the physical screen and to the additional screen; and transmitting the data representing the virtual annotation to a device associated with the additional screen.

[0099] 11. The method of any of the preceding examples, wherein: the transmitting of the data to the physical device and the transmitting of the data to the device are performed to allow the physical screen and the additional screen to present the virtual annotation simultaneously.

[0100] 12. The method of any of the preceding examples, wherein: the determining that the virtual annotation is targeted to the physical screen occurs prior to the determining that the virtual annotation is targeted to the additional screen; and the transmitting the data representing the virtual annotation to the device associated with the additional screen includes transferring the data representing the virtual annotation from the physical device to the device.

[0101] 13. The method of any one of any of the preceding examples, further comprising: identifying, during the mixed reality session, a person present in the environment; and determining that the virtual annotation is targeted to the person; wherein the transmitting is based on the determining that the virtual annotation is targeted to the person.

[0102] 14. The method of any of the preceding examples, wherein the transmitting is further based on at least one of: a proximity between a location of the person and a screen location where the physical screen is located; detecting that the person is using the physicaldevice; detecting that the person has a security permission to use the physical device; detecting a direction in which the person is looking within the environment; or detecting a direction in which the person is pointing within the environment.

[0103] 15. The method of any one of any of the preceding examples, wherein the identifying the physical screen is based on: scan data generated by a device scan configured to discover, within the environment, candidate targetable screens for virtual annotations; and selection data representing a selection of the physical screen as a targetable screen for virtual annotations.

[0104] 16. The method of any one of any of the preceding examples, wherein the identifying the physical screen is based on: connection data representing a previous connection of the mixed reality device to the physical device; and selection data representing a previous selection of the physical screen as a targetable screen for virtual annotations.

[0105] 17. The method of any one of any of the preceding examples, wherein the transmitting the data representing the virtual annotation includes directing the data to a software application executing on the physical device, the software application being configured to perform at least one of storing or presenting the virtual annotation.

[0106] 18. The method of any one of any of the preceding examples, further comprising determining boundaries of the physical screen with respect to a coordinate space once the physical screen is identified.

[0107] 19. The method of any one of any of the preceding examples, wherein the data representing the virtual annotation is generated by a physical controller communicatively coupled to the mixed reality device.

[0108] 20. The method of any of the preceding examples, wherein the physical controller generates the data representing the virtual annotation while in an annotation mode triggered by the physical controller being held in an annotation grip.

[0109] 21. A non-transitory computer-readable medium storing instructions that, when executed, cause a processor of a computing device to perform a process comprising: identifying, during a mixed reality session presented by a mixed reality device within an environment, a physical screen of a physical device present in the environment; receiving data representing a virtual annotation generated within the environment; and transmitting the data representing the virtual annotation to the physical device.

[0110] 22. The non-transitory computer-readable medium of any of the preceding examples, wherein the receiving and the transmitting of the data representing the virtualannotation are performed so as to allow the physical device to present the virtual annotation in real time for viewing within the environment and without aid of the mixed reality device.

[0111] 23. The non-transitory computer-readable medium of any one of any of the preceding examples, wherein: the process further comprises identifying, during the mixed reality session, an additional screen present in the environment; the virtual annotation is generated within the environment at an annotation location; and the process further comprises determining that the virtual annotation is targeted to the physical screen and not to the additional screen, the determining being based on: a first proximity between the annotation location and a first screen location where the physical screen is located, and a second proximity between the annotation location and a second screen location where the additional screen is located.

[0112] 24. A mixed reality device comprising: a memory storing instructions; and one or more processors communicatively coupled to the memory and configured to execute the instructions to perform a process comprising: identifying, during a mixed reality session presented by the mixed reality device within an environment, a physical screen of a physical device present in the environment; receiving data representing a virtual annotation generated within the environment; and transmitting the data representing the virtual annotation to the physical device.

[0113] 25. The mixed reality device of any of the preceding examples, wherein the receiving and the transmitting of the data representing the virtual annotation are performed so as to allow the physical device to present the virtual annotation in real time for viewing within the environment and without aid of the mixed reality device.

[0114] 26. The mixed reality device of any one of any of the preceding examples, wherein: the process further comprises identifying, during the mixed reality session, an additional screen present in the environment; the virtual annotation is generated within the environment at an annotation location; and the process further comprises determining that the virtual annotation is targeted to the physical screen and not to the additional screen, the determining being based on: a first proximity between the annotation location and a first screen location where the physical screen is located, and a second proximity between the annotation location and a second screen location where the additional screen is located.

[0115] 27. A system including a mixed reality device of any one of claims 24-26 and at least one physical device with at least one physical screen.

[0116] Various implementations of the systems and techniques described herein can be realized in digital electronic circuitry, integrated circuitry, specially designed ASICs(application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0117] A number of implementations have been described. Nevertheless, it will be understood that various modifications may be made without departing from the spirit and scope of the description and claims. In addition, the logic flows depicted in the figures do not require the particular order shown, or sequential order, to achieve desirable results. In addition, other steps may be provided, or steps may be eliminated, from the described flows, and other components may be added to, or removed from, the described systems. Accordingly, other implementations are within the scope of the following claims.

[0118] Specific structural and functional details disclosed herein are merely representative for purposes of describing example implementations. Example implementations, however, may be embodied in many alternate forms and should not be construed as limited to only the implementations set forth herein.

[0119] It will be understood that, although the terms first, second, etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. A first element could be termed a second element, and, similarly, a second element could be termed a first element, without departing from the scope of the implementations of the disclosure. As used herein, the term and / or includes any and all combinations of one or more of the associated listed items.

[0120] The terminology used herein is for the purpose of describing particular implementations only and is not intended to be limiting of the implementations. As used herein, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises,” “comprising,” “includes,” and / or “including,” when used in this specification, specify the presence of the stated features, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof.

[0121] It will be understood that when an element is referred to as being “coupled,” “connected,” or “responsive” to, or “on,” another element, it can be directly coupled,connected, or responsive to, or on, the other element, or intervening elements may also be present. In contrast, when an element is referred to as being “directly coupled,” “directly connected,” or “directly responsive” to, or “directly on,” another element, there are no intervening elements present. As used herein the term “and / or” includes any and all combinations of one or more of the associated listed items.

[0122] Spatially relative terms, such as “beneath,” “below,” “lower,” “above,” “upper,” and the like, may be used herein for ease of description to describe one element or feature in relationship to another element(s) or feature(s) as illustrated in the figures. It will be understood that the spatially relative terms are intended to encompass different orientations of the device in use or operation in addition to the orientation depicted in the figures. For example, if the device in the figures is turned over, elements described as “below” or “beneath” other elements or features would then be oriented “above” the other elements or features. Thus, the term “below” can encompass both an orientation of above and below. The device may be otherwise oriented (rotated 130 degrees or at other orientations) and the spatially relative descriptors used herein may be interpreted accordingly.

[0123] Unless otherwise defined, the terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which these concepts belong. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and / or the present specification and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[0124] Further to the descriptions above, a user may be provided with controls allowing the user to make an election as to both if and when systems, programs, or features described herein may enable collection of user information (e.g., information about a user's social network, social actions, or activities, profession, a user's preferences, or a user's current location), and if the user is sent content or communications from a server. In addition, certain data may be treated in one or more ways before it is stored or used, so that personally identifiable information is removed. For example, a user's identity may be treated so that no personally identifiable information can be determined for the user, or a user's geographic location may be generalized or location information is obtained (such as to a city, zip code, or state level), so that a particular location of a user cannot be determined. Thus, the user may have control over what information is collected about the user, how that information is used, and what information is provided to the user.

[0125] While certain features of the described implementations have been illustrated as described herein, many modifications, substitutions, changes, and equivalents may occur to those skilled in the art. It is therefore to be understood that the appended claims are intended to cover such modifications and changes as fall within the scope of the implementations. It will be understood that they have been presented by way of example only, not limitation, and various changes in form and details may be made. Any portion of the apparatus and / or methods described herein may be combined in any combination, except mutually exclusive combinations. The implementations described herein can include various combinations and / or sub-combinations of the functions, components, and / or features of the different implementations described. As such, the scope of the present disclosure is not limited to the particular combinations hereafter claimed, but instead extends to encompass any combination of features or example implementations described herein irrespective of whether or not that particular combination has been specifically enumerated in the accompanying claims at this time.

Claims

WHAT IS CLAIMED IS:

1. A method compri sing : identifying, during a mixed reality session presented by a mixed reality device within an environment, a physical screen of a physical device present in the environment; receiving data representing a virtual annotation generated within the environment; and transmitting the data representing the virtual annotation to the physical device.

2. The method of claim 1, wherein the receiving and the transmitting of the data representing the virtual annotation are performed to allow the physical device to present the virtual annotation in real time for viewing within the environment and without aid of the mixed reality device.

3. The method of any one of claims 1-2, further comprising: identifying, during the mixed reality session, an additional screen present in the environment; and determining that the virtual annotation is targeted to the physical screen and not to the additional screen.

4. The method of claim 3, wherein: the virtual annotation is generated within the environment at an annotation location; and the determining is based on: a first proximity between the annotation location and a first screen location where the physical screen is located, and a second proximity between the annotation location and a second screen location where the additional screen is located.

5. The method of any one of claims 3-4, wherein: the mixed reality device presents a field of view when the virtual annotation is generated; and the determining is based on a position of the field of view with respect to a first screen location where the physical screen is located and with respect to a second screen location where the additional screen is located.

6. The method of any one of claims 3-5, wherein the determining is based on a size of the virtual annotation with respect to a first size of the physical screen and with respect to a second size of the additional screen.

7. The method of any one of claims 3-6, wherein the determining is based on a form factor of the physical device.

8. The method of any one of claims 3-7, wherein the additional screen is an additional physical screen of an additional physical device present in the environment.

9. The method of any one of claims 3-7, wherein the additional screen is a virtual screen being presented within the environment by way of the mixed reality device.

10. The method of any one of claims 1-2, further comprising: identifying, during the mixed reality session, an additional screen present in the environment; determining that the virtual annotation is targeted to the physical screen and to the additional screen; and transmitting the data representing the virtual annotation to a device associated with the additional screen.

11. The method of claim 10, wherein: the transmitting of the data to the physical device and the transmitting of the data to the device are performed to allow the physical screen and the additional screen to present the virtual annotation simultaneously.

12. The method of claim 10, wherein: the determining that the virtual annotation is targeted to the physical screen occurs prior to the determining that the virtual annotation is targeted to the additional screen; and the transmitting the data representing the virtual annotation to the device associated with the additional screen includes transferring the data representing the virtual annotation from the physical device to the device.

13. The method of any one of claims 1-2, further comprising: identifying, during the mixed reality session, a person present in the environment; and determining that the virtual annotation is targeted to the person; wherein the transmitting is based on the determining that the virtual annotation is targeted to the person.

14. The method of claim 13, wherein the transmitting is further based on at least one of: a proximity between a location of the person and a screen location where the physical screen is located; detecting that the person is using the physical device; detecting that the person has a security permission to use the physical device; detecting a direction in which the person is looking within the environment; or detecting a direction in which the person is pointing within the environment.

15. The method of any one of claims 1-14, wherein the identifying the physical screen is based on: scan data generated by a device scan configured to discover, within the environment, candidate targetable screens for virtual annotations; and selection data representing a selection of the physical screen as a targetable screen for virtual annotations.

16. The method of any one of claims 1-15, wherein the identifying the physical screen is based on: connection data representing a previous connection of the mixed reality device to the physical device; and selection data representing a previous selection of the physical screen as a targetable screen for virtual annotations.

17. The method of any one of claims 1-16, wherein the transmitting the data representing the virtual annotation includes directing the data to a software application executing on the physical device, the software application being configured to perform at least one of storing or presenting the virtual annotation.

18. The method of any one of claims 1-17, further comprising determining boundaries of the physical screen with respect to a coordinate space once the physical screen is identified.

19. The method of any one of claims 1-18, wherein the data representing the virtual annotation is generated by a physical controller communicatively coupled to the mixed reality device.

20. The method of claim 19, wherein the physical controller generates the data representing the virtual annotation while in an annotation mode triggered by the physical controller being held in an annotation grip.

21. A non-transitory computer-readable medium storing instructions that, when executed, cause a processor of a computing device to perform a process comprising: identifying, during a mixed reality session presented by a mixed reality device within an environment, a physical screen of a physical device present in the environment; receiving data representing a virtual annotation generated within the environment; and transmitting the data representing the virtual annotation to the physical device.

22. The non-transitory computer-readable medium of claim 21, wherein the receiving and the transmitting of the data representing the virtual annotation are performed so as to allow the physical device to present the virtual annotation in real time for viewing within the environment and without aid of the mixed reality device.

23. The non-transitory computer-readable medium of any one of claims 21-22, wherein: the process further comprises identifying, during the mixed reality session, an additional screen present in the environment; the virtual annotation is generated within the environment at an annotation location; and the process further comprises determining that the virtual annotation is targeted to the physical screen and not to the additional screen, the determining being based on: a first proximity between the annotation location and a first screen location where the physical screen is located, anda second proximity between the annotation location and a second screen location where the additional screen is located.

24. A mixed reality device comprising: a memory storing instructions; and one or more processors communicatively coupled to the memory and configured to execute the instructions to perform a process comprising: identifying, during a mixed reality session presented by the mixed reality device within an environment, a physical screen of a physical device present in the environment; receiving data representing a virtual annotation generated within the environment; and transmitting the data representing the virtual annotation to the physical device.

25. The mixed reality device of claim 24, wherein the receiving and the transmitting of the data representing the virtual annotation are performed so as to allow the physical device to present the virtual annotation in real time for viewing within the environment and without aid of the mixed reality device.

26. The mixed reality device of any one of claims 24-25, wherein: the process further comprises identifying, during the mixed reality session, an additional screen present in the environment; the virtual annotation is generated within the environment at an annotation location; and the process further comprises determining that the virtual annotation is targeted to the physical screen and not to the additional screen, the determining being based on: a first proximity between the annotation location and a first screen location where the physical screen is located, and a second proximity between the annotation location and a second screen location where the additional screen is located.

27. A system including a mixed reality device of any one of claims 24-26 and at least one physical device with at least one physical screen.

Images