Real screen in extended reality

Abstract

To provide a method and a system for displaying a screen in extended reality.SOLUTION: This method includes: receiving video data of a real-world environment of a user of an extended reality device; identifying a screen of a separate computing device in the video data; pairing the extended reality device and the separate computing device; generating a graphical outline around the screen of the separate computing device; and displaying the screen of the separate computing device and the graphical outline on a display of the extended reality device. The method can further include: detecting a user interaction with the graphical outline based on the received video data; and, in response to the detection, performing an action associated with the user interaction.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] This disclosure relates generally to extended reality displays, and more specifically, describes systems and methods for identifying and displaying screens of separate computing devices as virtual components of an extended reality display and enabling user manipulation of the screens within the extended reality display. [Background technology]

[0002] Extended reality (XR) is a general term referring to the convergence of virtual reality (VR), augmented reality (AR), and / or mixed reality (MR) technologies.

[0003] Virtual reality is an immersive experience in which the user is completely immersed in a computer-generated environment. Augmented reality is a technology that overlays digital information or objects onto the real world. Mixed reality is a fusion of virtual and augmented reality, allowing users to interact with digital objects in the real world.

[0004] Extended reality technology allows users to experience increased immersion and interactivity, which can be used in various fields such as entertainment, education, healthcare, and business. By blending the physical and virtual worlds, extended reality has the potential to change the way we interact with and understand our environments.

[0005] Thus, XR technologies can either immerse a user in a virtual environment (virtual reality or "VR"), or add to or extend a user's real-world environment by displaying virtual components superimposed on video data of the real-world environment (augmented reality or "AR"), or both. Summary of the Invention

[0006] Aspects and advantages of embodiments of the disclosure will be set forth in part in the description that follows, or may be learned from the description, or may be learned by practice of the embodiments.

[0007] One exemplary aspect of the disclosure is directed to a computer-implemented method for displaying a screen in extended reality. The method may include receiving, by a processor, video data of a real-world environment of a user of an extended reality device, and identifying, by the processor, a screen of a separate computing device within the video data. The method may further include pairing, by the processor, the extended reality device and the separate computing device, generating, by the processor, a graphical outline around the screen of the separate computing device, and displaying, by the processor, the screen of the separate computing device and the graphical outline around the screen of the separate computing device on a display of the extended reality device, wherein the displayed screen on the display of the extended reality device has at least one display characteristic that is different from the screen displayed on the separate computing device. The method further includes detecting, by the processor, a user interaction with the graphical outline around the screen of the separate computing device based on the received video data, and performing, by the processor, an action associated with the user interaction in response to detecting the user interaction.

[0008] Another exemplary aspect of the present disclosure is directed to a computing system for displaying a screen in extended reality. The computing system may include one or more processors and one or more non-transitory computer-readable media storing instructions that, when executed by the one or more processors, cause the computing system to perform operations. The operations may include receiving video data of a real-world environment of a user of the extended reality device and identifying a screen of the separate computing device within the video data. The operations may further include pairing the extended reality device with the separate computing device, generating a graphical outline around the screen of the separate computing device, and displaying the screen of the separate computing device and the graphical outline around the screen of the separate computing device on a display of the extended reality device, wherein the displayed screen on the display of the extended reality device has at least one display characteristic that is different from the screen displayed on the separate computing device. The operations may further include detecting a user interaction with the graphical outline around the screen of the separate computing device based on the received video data, and performing an action associated with the user interaction in response to detecting the user interaction.

[0009] Further exemplary aspects of the present disclosure are directed to one or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations. The operations may include receiving video data of a real-world environment of a user of an extended reality device and identifying a screen of a separate computing device within the video data. The operations may further include pairing the extended reality device and the separate computing device, generating a graphical outline around the screen of the separate computing device, and displaying the screen of the separate computing device and the graphical outline around the screen of the separate computing device on a display of the extended reality device, wherein the displayed screen on the display of the extended reality device has at least one display characteristic that is different from the screen displayed on the separate computing device. The operations may further include detecting a user interaction with the graphical outline around the screen of the separate computing device based on the received video data, and performing an action associated with the user interaction in response to detecting the user interaction.

[0010] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A detailed description of embodiments directed to those skilled in the art is set forth herein with reference to the accompanying drawings. [Brief description of the drawings]

[0011] [Figure 1A] FIG. 1 illustrates a perspective view of a head-mounted device according to an exemplary embodiment of the present disclosure. [Figure 1B] FIG. 1 illustrates a perspective view of a head-mounted device according to an exemplary embodiment of the present disclosure. [Figure 1C] 1 illustrates a side view of a head-mounted device according to an exemplary embodiment of the present disclosure. [Diagram 2]1 illustrates a block diagram of an extended reality device according to an exemplary embodiment of the present disclosure. [Diagram 3] 1 illustrates a virtual screen being displayed in a real-world environment, according to an exemplary embodiment of the present disclosure. [Figure 4] 1 illustrates a virtual screen being displayed in a real-world environment, according to an exemplary embodiment of the present disclosure. [Diagram 5] 1 illustrates a flowchart of a method for displaying a user device screen on a head-mounted device, according to an exemplary embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Reference numbers repeated among the drawings are intended to identify like features in the various embodiments.

[0013] In general, the present disclosure is directed to an extended reality (XR) device that can be paired with one or more separate computing devices. The XR device can replicate the screens of the one or more separate computing devices in an XR display environment associated with the XR device. This enhances the immersive feel of the XR device by allowing a user to interact with the screens displayed from those computing devices in the display environment of the XR device without the user having to actually interact with the separate devices.

[0014] More specifically, certain XR devices allow users to view various virtual components (e.g., a full virtual environment in virtual reality and / or virtual components overlaid on live video data in augmented reality). However, a user of an XR device may wish to use a separate computing device, such as a smartphone, laptop computer, desktop computer, etc., while still using the XR device. Typically, this involves the user turning on a pass-through mode for the XR device, where the XR device only displays live video data of the user's surrounding environment.

[0015] To solve this problem, the present disclosure allows a user to identify the XR device they are using (e.g., wearing) and pair it with another computing device. Some techniques that can be used to pair an XR device with another computing device include, for example, Bluetooth pairing, Wi-Fi Direct, QR Code scanning, and / or cable connection. For example, QR code scanning can include generating a QR code on the other computing device using a QR code generator or application. A QR code scanner on the XR device can then scan the code. This can allow a connection between the devices to be established.

[0016] In some implementations, the XR device includes technology that allows the XR device to automatically detect or recognize the screens of other computing devices. This means that the XR device can identify the presence and characteristics of a separate screen, such as its size, resolution, and orientation, without requiring input from a user. In one example, the XR device can include sensors, such as a camera or an infrared sensor. These sensors can capture information about the surrounding environment and use algorithms to identify the screens of other computing devices. Alternatively, the XR device may use wireless protocols, such as Bluetooth or Wi-Fi, to communicate with the other computing devices and obtain information about their screens.

[0017] Automatic screen detection or automatic screen recognition may provide many benefits to XR device users. For example, it may streamline the pairing process by eliminating the need for manual input or configuration. Furthermore, it may ensure that the XR display environment is properly calibrated and optimized to the screen of the other computing device, resulting in a more seamless and immersive user experience. Automatic screen detection or automatic screen recognition is an innovative feature that can enhance the functionality and usability of XR devices in a variety of contexts, from gaming and entertainment to education and training.

[0018] In some implementations, the XR device may highlight detected and / or paired screens by placing a virtual border around the screen in the XR display environment. This means that a user can easily identify when a screen has been detected by the XR device and / or when a screen has been paired with the XR device. Highlighting of detected screens may be achieved by a variety of techniques, depending on the particular implementation of the XR technology. For example, the XR device may use sensors or wireless protocols to identify the location and orientation of the paired computing device's screen and then superimpose a virtual border or other visual cue onto the corresponding location in the XR display environment.

[0019] When an XR device is paired with another computing device, the XR device can receive data about the content currently being displayed on the computing device(s) with which the XR device is paired. This means that any data being output on the screen of the paired computing device can also be sent to the display environment of the XR device. The type of data that can be received depends on the specific implementation of the pairing technology. In some cases, the XR device may be able to receive a live video stream of the content being displayed on the screen of the computing device, effectively mirroring the screen in the XR display environment. In other cases, the XR device may receive data about the content, such as metadata or text, and can then use the data to display the content in a different format or context within the XR environment, such as in an expanded format.

[0020] In some implementations, a user may be able to perform intuitive interactions with the detected and / or paired screen(s) on the XR device. For example, a user may perform one or more gestures to "drag and drop" a highlighted screen to create a virtual mirrored screen in the XR display environment. This means that a user can seamlessly integrate content displayed on the paired computing device(s) with the XR environment, allowing them to interact with the content in new and immersive ways.

[0021] The process of creating a virtual mirrored screen may vary depending on the specific implementation of XR technology. In some cases, a user can create a virtual mirrored screen by simply dragging a highlighted screen to a desired location in the XR environment. In other cases, a user may need to initiate a specific command or gesture to trigger the creation of a virtual mirrored screen.

[0022] An XR device can replicate content displayed on other computing devices as a graphical overlay (e.g., a "virtual screen") within the XR device's display environment, allowing a user to interact with the virtual screen, for example, by rearranging the virtual screen to match the user's desired layout of the display, resizing the virtual screen, interacting with elements on the virtual screen (e.g., pressing a next page button), and other actions.

[0023] This allows a user to continue working on multiple screens while remaining within the display environment of the XR device, which improves the user experience of the XR device by allowing the user to better control the virtual content displayed on the XR device without having to remove themselves from the immersive XR environment (e.g., by removing the XR headset or viewing and / or interacting with other computing devices while the XR device is in pass-through mode).

[0024] Therefore, the present disclosure proposes a system that allows an XR device to automatically identify and pair with other computing devices. The system can utilize communication technologies such as Bluetooth or Wi-Fi to establish a connection between the devices. Once connected, a user can access information or applications on a separate computing device through the XR display environment. This allows for a seamless and streamlined user experience, allowing users to switch between devices without having to remove the XR device each time or manually pair the devices.

[0025] Overall, the proposed solution provides a more streamlined user-friendly experience for those using XR devices. By automating the pairing process, users can focus on the task at hand without having to worry about technical challenges or interruptions. This innovation has the potential to improve productivity and efficiency in a variety of contexts, such as business, education, and healthcare, where XR technology is increasingly being used. Furthermore, the ability to receive data about content being displayed on a paired computing device can provide many benefits to XR device users. For example, it may enable users to seamlessly switch between devices and access information or applications from multiple sources within a single display environment. Additionally, it may allow XR applications and experiences to be enhanced or extended by data from external sources, such as social media feeds or real-time environmental data.

[0026] The present disclosure provides several technical effects and advantages. By way of example, by providing seamless and immersive integration of multiple screens within an XR environment, the present technology enhances the user experience by allowing users to simultaneously interact with multiple sources of digital content. This enables more efficient workflows and collaboration, particularly in remote work environments, where users can view and manipulate content from various sources in real time.

[0027] The technology also provides technical advantages with respect to the virtual mirroring process: by detecting and recognizing the screen of another computing device and creating a virtual boundary and mirrored screen, the technology facilitates data transmission between the XR device and the paired computing device, providing a streamlined and efficient approach to integrating digital content within the XR environment.

[0028] In addition to these advantages, the proposed technique provides a technical solution to a technical problem, namely, the problem of integrating multiple sources of digital content within an XR environment. By providing a solution to this problem, the present technique provides a unique technical advantage over existing solutions that often require complex and time-consuming manual processes to integrate multiple content sources within an XR environment. Thus, the proposed technique that enables an XR device to replicate the screen of another computing device in an XR display environment is relevant to the technical field, has a technical effect on physical entities, and provides a technical solution to a technical problem.

[0029] The proposed approach can provide benefits in many applications. As an example, in the field of remote work and collaboration, the ability to seamlessly integrate multiple sources of data and content within an XR environment can improve productivity and facilitate more efficient and effective communication and collaboration between remote teams. For example, a team of designers or engineers can use XR technology to collaborate on a complex 3D model, with each member able to view and manipulate the model from their own device within the XR environment.

[0030] As another example, in the education sector, XR technology can enable immersive and interactive learning experiences by allowing learners to view and interact with educational content from a variety of sources such as laptops, tablets, and smartphones. The ability to create virtual mirrored screens within the XR environment also allows teachers to view and interact with educational content in real time, enhancing student engagement and fostering more effective learning outcomes.

[0031] Referring now to the figures, exemplary embodiments of the present disclosure are described in further detail.

[0032] 1A and 1B are perspective views and FIG. 1C is a side view of an exemplary head mounted device ("HMD") that a user may wear to generate an immersive extended reality experience. The HMD shown in FIGS. 1A-1C is provided only as one example of an XR device. Other XR devices are possible, including, but not limited to, handheld devices (e.g., smartphones or tablets), immersive rooms (e.g., using multiple screens and projectors to create a fully immersive XR experience), and / or wearables (e.g., smart watches or bracelets). The exemplary HMD 100 may include a housing 110 in which optical components may be received. The housing 110 may be coupled, e.g., rotatably coupled, and / or removably attachable to a frame 120 that allows the housing 110 to be attached or worn on the user's head. An audio output device 130 may also be coupled to the frame 120, e.g., attached to headphones and may include a speaker coupled to the frame 120.

[0033] 1B, a front surface 110a of the housing 110 rotates away from or is otherwise removably attached to a base portion 110b of the housing 110. A display 140, such as, for example, a smartphone or other display device, may be attached to the front surface 110a of the housing 110. A lens 150 may be mounted within the housing 110 between a user's eyes and the display 140 when the front surface 110a is in a closed position relative to the base portion 110b of the housing 110. The position of the lens 150 may be adjusted by an adjustment device 158 so that the lens 150 may be aligned with the optical axis of each of the user's eyes to provide a relatively wide field of view and a relatively short focal length.

[0034] The HMD 100 may also include a sensing system 160, including various sensing system devices 162-164, and a control system 170, including various control system devices 171-176, to facilitate manual user control and automatic control of the HMD 100. The control system 170 may also include a processor 190 for controlling the operation of the components of the control system 170, both manually and independently, and in response to conditions detected / sensed by the sensing system 160 in response to commands received by the control system 170.

[0035] In some embodiments, the HMD 100 may also include a camera 180 that may capture still and dynamic images of the real-world environment and display these images to the user on the display 140 in a pass-through mode. In the pass-through mode, the user may be able to leave the virtual world and temporarily return to the real world without removing the HMD 100 or otherwise changing the configuration of the HMD 100, for example by moving the housing 110 out of the user's line of sight. In some embodiments, the display 140 may be a transparent display, allowing the user to see the real-world environment through the display 140 in some modes of operation.

[0036] In some embodiments, the camera 180 can capture still and moving images of the real-world environment and display these images to the user on the display 140 while simultaneously displaying the virtual components to the user on the display 140 in an augmented reality mode. The various sensing system devices 162-164 can sense objects in the real-world environment and the user's movements in the real-world environment (e.g., the user's hand movements), and based on this sensing, the control system 170 can enable the user to interact with the virtual components displayed on the display 140 when the user "interacts" with the virtual components, such as when the user moves their hand to a position or location where the virtual components are displayed on the display overlaid with the real world. When the user's hand is detected at the location of the displayed virtual components, the control system 170 can enable the user to perform one or more actions associated with "interacting" with the virtual components, such as moving the location of the virtual components on the display 140 or initiating another action. Additional details regarding example interactions are provided below in conjunction with FIGS. 3 and 4.

[0037] A block diagram of a system 200 for displaying virtual components on a display of an HMD is shown in Figure 2. The system 200 includes an XR device 205, such as the HMD described with respect to Figures 1A-1C. The XR device 205 can include a control system 210, a sensing system 220, a display system 230, a processor 240, and a memory 250.

[0038] The control system 210 and the sensing system 220 may be similar to the control system 170 and the sensing system 160 shown in Figures 1A-1C, respectively. In the exemplary embodiment shown in Figure 2, the control system 210 may include a power control device 211 for controlling power usage at the XR device 205, a pause control device 212 for controlling the display of virtual components or a pass-through mode of the XR device 205, an audio control device 213 for controlling audio that may be provided to the user via the XR device 205, a manual video control device 214 for controlling the video display (both real world and virtual world) at the XR device 205, a rotation control device 215 for controlling the rotation of the housing relative to the frame, an optical adjustment control device 216 for adjusting the optical components of the XR device 205, a virtual component interaction system 217, and a pairing module 218.

[0039] The virtual component interaction system 217 receives object detection data from the sensing system 220 and displayed virtual component data from the display system 230 to determine when a user is "interacting" with a displayed virtual component. For example, when a user's hand is detected as "touching" a display screen, which is a virtual component displayed at a location within the display of the XR device 205 (e.g., determining that the display screen or a graphical outline around the display screen and the detected hand is at the same coordinate within the display), the virtual component interaction system 217 determines an interaction type to perform with the display screen (such as closing the display screen, moving the display screen to a new coordinate within the display, etc.). In some embodiments, the virtual component interaction system 217 can determine the interaction type based on an identified gesture of the user's hand. For example, the virtual component interaction system 217 can detect that the user's hand is making a pinching gesture, which can indicate that the user wants to reposition the display screen. In another example, the virtual component interaction system 217 can detect two fingers of a user's hand "touching" the display screen, which can indicate that the user wants to stop viewing the display screen. The virtual component interaction system 217 can also generate virtual components to display, such as the display screen, outlines (bounding boxes) of identified objects, etc.

[0040] The pairing module 218 allows the XR device 205 to pair with a separate computing device. For example, the XR device 205 can be paired with a smartphone, a laptop computer, a desktop computer, or any other suitable computing device that includes a display screen for providing information to a user of the separate computing device. The XR device 205 and the separate computing device can be paired using Bluetooth™, Wi-Fi, or other wireless communication means.

[0041] When the XR device 205 and the separate computing device are paired, the XR device 205 and the separate computing device can share information, such as sharing to the XR device 205 what is being displayed by the separate computing device, which can then display virtual components on the display of the XR device 205 that reproduces the current display screen of the separate computing device.

[0042] To pair the XR device 205 with the separate computing device, the pairing module 218 can identify in the video data a pattern or QR code displayed by the screen of the separate computing device. For example, the separate computing device can display a QR code that can be detected by the XR device 205. The QR code uniquely identifies the separate computing device as a device to be paired to the XR device 205. After identifying the QR code, the pairing module 218 can create a communication connection between the XR device 205 and the separate computing device, such that the separate computing device can send data regarding the screen currently displayed on the separate computing device to the XR device 205 for display on the display of the XR device 205.

[0043] In another embodiment, the pairing module 218 can identify a specific pattern, a specific image, or a specific character in the video data that indicates that the separate computing device is the device with which the XR device 205 is paired. These specific patterns, images, or characters uniquely identify the separate computing device as the computing device with which the XR device 205 is paired. After identifying the specific pattern, image, or character, the pairing module 218 can create a communication connection between the XR device 205 and the separate computing device, such that the separate computing device can send data regarding a screen currently displayed on the separate computing device to the XR device 205 for display on the display of the XR device 205.

[0044] The sensing system 220 may include a light sensor 221 for detecting light-emitting objects (such as a display screen from a separate computing device), a distance / proximity sensor 222 for detecting the distance between the XR device 205 and one or more objects in the user's real-world environment, a camera 223 for capturing image and video data of the user's real-world environment, and an object detection system 224. The object detection system 224 may detect various objects in the real-world environment based on video and other data, such as detecting a user's hand using the XR device 205, other computing devices with display screens, and other related objects. Data related to the detected objects may then be returned to the control system 210, such as the identity of the detected object, the location of the detected object in the real-world environment, and the coordinates of the detected object on the display.

[0045] The display system 230 includes a display 231, a rendering module 232, and a pass-through module 233. The display 231 is a display for video data that is either a user's real-world environment, a virtual component (such as a virtual environment and / or one or more other virtual objects), or a combination of the two. The rendering module 232 receives data from the sensing system 220 and / or the control system 210 and renders the virtual component for display on the display 231 based on the received data. The pass-through module 233 allows video data collected from the camera to be displayed on the display 233 instead of or in conjunction with the virtual component / background.

[0046] The processor 240 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be one or more operatively connected processors. The memory 250 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 250 can store data 251 and instructions 252 that are executed by the processor 240 to cause the XR device 205 to perform operations.

[0047] It should be understood that the XR device 205, the control system 210, the sensing system 220, and the display system 230 may include more or less devices and systems depending on the particular implementation. The elements included in the XR device 205, the control system 210, and the sensing system 220 may have different physical arrangements within the XR device 205.

[0048] 3 illustrates a virtual screen 300 displayed in a real-world environment according to an exemplary embodiment of the present disclosure. The virtual screen 300 is displayed as a graphical overlay of the actual screen of the separate computing device 305 displayed on the display of the XR device operating in pass-through mode (e.g., displaying the real-world environment as seen through the XR device). The virtual screen 300 is generated by the XR device because the XR device and the separate computing device 305 are paired with each other. The XR device generates a graphical outline 310 to show the user of the XR device where the virtual screen 300 of the separate computing device 305 is located and to enable user interaction with the virtual screen 300.

[0049] When an XR device replicates the screen of a separate computing device 305 in an XR display environment, the virtual screen 300 may have the same resolution and layout as the physical screen of the device 305, or may have a different resolution, size, or layout.

[0050] If the XR device replicates the screen of a separate computing device 305 with the same resolution and aspect ratio, the virtual screen 300 can have a similar resolution and layout as the physical screen, allowing a seamless transition from the physical screen to the virtual screen 300 and allowing the user to interact with digital content in a familiar way.

[0051] However, if the XR device is replicating the screen of a separate computing device 305 with a different resolution, aspect ratio, or physical size, the virtual screen 300 can be adjusted to match the user's preferences or specific use case. For example, the user can adjust the virtual screen 300 to be larger or smaller than the physical screen, or the user may adjust the virtual screen 300 to have a different aspect ratio that corresponds to their own specific application. In one example, the virtual screen 300 can be made significantly larger to allow the user to see certain items in more detail.

[0052] In such cases, the XR device can use various techniques to adjust the resolution, size, and layout of the virtual screen 300. For example, the XR device may use a scaling algorithm to adjust the resolution of the virtual screen 300 to match the physical screen, or may use cropping techniques to adjust the layout of the virtual screen 300 to suit the user's preferences.

[0053] According to another aspect, the XR device can use the video data to identify when the user's hand 315 interacts with the virtual screen 300 and / or the graphical outline 310, for example, by detecting when the position of the hand 315 is in the same position as the virtual screen 300 and / or the graphical outline 310 in the displayed video.

[0054] The XR device can also determine the gesture 320 that the hand 315 is performing. Based on the identified gesture, various actions can be performed. For example, the gesture 320 is a pinch gesture. When the XR device determines that the user is performing the pinch gesture, the XR device determines that the user wants to move the virtual screen 300 to a new location 325 in the display. The action can continue while the user is still performing the gesture 320 (e.g., the user can continue to move the virtual screen 300 to the new location 325 or a different location). When the user stops performing the gesture 320, the action can be stopped (e.g., the virtual screen 300 is placed at the new location 325 and the user cannot move the virtual screen 300 without performing the gesture 320 again).

[0055] According to another aspect, in some implementations, multiple virtual screens corresponding to multiple devices can be combined into a single visual representation in an XR display environment, allowing a user to view and interact with content from multiple devices simultaneously without having to switch between different physical screens or XR devices.

[0056] The combined virtual representation of multiple screens can take a variety of forms, depending on the particular application or use case. For example, the virtual screens may be arranged side-by-side in a particular order, stacked vertically, or on top of each other. XR devices can also provide a variety of techniques for controlling and interacting with the multiple virtual screens, such as resizing or repositioning the screens, selecting which screen is the active screen, and adjusting the relative brightness or contrast of each screen.

[0057] 4 illustrates a virtual screen 400 displayed in a real-world environment according to an exemplary embodiment of the present disclosure. Similar to the virtual screen 300 described in FIG. 3, the virtual screen 400 is a graphical overlay of an actual screen of a separate computing device 405 (in this case, a desktop computer). A graphical outline 410 is generated by the XR device to indicate where the virtual screen 400 is located. The XR device detects a user's hand 415 making a gesture 420 (e.g., a pinching gesture), and based on the detected gesture 420 and the location of the hand 415, the XR device performs an action such as having the user drag the virtual screen 400 to a new location 425.

[0058] 5 illustrates a flow chart diagram of an exemplary method 500 performed according to an exemplary embodiment of the present disclosure. Although FIG. 5 illustrates operations performed in a particular order for purposes of illustration and description, the method of the present disclosure is not limited to that specifically illustrated order or arrangement. Various operations of method 500 may be omitted, rearranged, combined, and / or adapted in various ways without departing from the scope of the present disclosure.

[0059] At operation 502, a computing system may receive video data of a real-world environment of a user of an extended reality device. This data may be received, for example, from a camera of the extended reality device while the extended reality device is in pass-through mode.

[0060] At operation 504, the computing system may identify a screen of a separate computing device within the video data. For example, the computing system may identify a QR code, a particular pattern, a particular image, or particular characters that uniquely identify the separate computing device. The computing system may detect the QR code, the particular pattern, the particular image, or particular characters and identify the separate computing device for pairing with the extended reality device.

[0061] In another example, a computing system uses a machine learning model to recognize, detect, or identify emissive screens from video data captured by its sensors. This machine learning model can be trained on a dataset of various emissive screens and learns features such as shape, size, and color. Once trained, the model can be used by the computing system to recognize emissive screens in real time and identify distinct computing devices to pair with.

[0062] The use of machine learning in this process can improve the accuracy and reliability of identifying emissive screens, especially in complex or dynamic environments where the screen may be partially covered or in motion. Additionally, the machine learning model can be continually improved and updated over time to enhance its performance and adapt to new types of emissive screens.

[0063] At operation 506, the computing system may pair the extended reality device with the separate computing device by creating a communications connection (e.g., using Bluetooth™, Wi-fi, etc.).

[0064] At operation 508, the computing system may generate a graphical outline or other graphical element around or adjacent to the screen of the separate computing device. This graphical outline may surround a graphical overlay overlay that currently displays a virtual representation of what is actually displayed on the screen of the separate computing device (e.g., a virtual screen).

[0065] At operation 510, the computing system displays the screen of the separate computing device and a graphical outline around the screen of the separate computing device on the display of the extended reality device. For example, the computing system overlays a graphical overlay on the screen of the separate computing device and then displays the graphical outline around the graphical overlay. In some embodiments, the display screen of the extended reality device's display has at least one display characteristic that is different from the screen displayed on the separate computing device. For example, the screen displayed on the separate computing device may have a size, resolution, aspect ratio, brightness, and other display characteristics, some or all of which may be dictated by the separate computing device (e.g., a particular aspect ratio for displaying the screen if the separate computing device is a smartphone, or a different aspect ratio for displaying the screen if the separate computing device is a desktop computer). In order to display the screen on the display of the extended reality device, these display characteristics may need to be modified so that the content of the image is not distorted, blurred, cropped, or otherwise distorted on the display of the extended reality device. Thus, the extended reality device may make one or more modifications to the screen to ensure a clear and accurate display of the screen on the extended reality device's display, such as changing the size, resolution, aspect ratio, brightness, or other display characteristics.

[0066] At operation 512, the computing system detects user interaction with a graphical outline around the screen of the separate computing device based on the received video data. For example, the computing system can detect the location of the user's hand in the video data and determine whether the user's hand is in the same location or overlaps with a location that includes a graphical overlay and / or graphical outline. The computing system can also determine a gesture being made by the user's hand, such as a pinching motion, a grasping motion, an open palm, etc.

[0067] In response to detecting the user interaction at operation 514, the computing system may perform an action associated with the user interaction. For example, based on the determined gesture being a pinch action, the computing system may initiate a drag-and-drop action, where the user may drag the graphical overlay to a new location within the display. In another example, based on the determined gesture being a grab action, the computing system may stop displaying the screen and the graphical outline.

[0068] After a user performs a drag-and-drop operation to reposition the virtual screen within the XR display environment, the computing system can utilize sensors and tracking devices to determine the new position of the virtual screen. These technologies can accurately track the movement and orientation of the XR device, as well as any physical objects or markers placed within the environment.

[0069] Using this information, the computing system can place the virtual screen exactly where it was dropped by the user, ensuring that the user's interactions with the screen are accurately translated into the XR display environment and that the virtual screen behaves in a natural and intuitive way.

[0070] In addition to allowing a user to interact with a virtual screen in an XR display environment, the computing system can also communicate these interactions back to a paired computing device. For example, if a user types on a virtual keyboard displayed in an XR environment, the computing system can send the text input back to the paired computing device, which can then process the input and display corresponding output.

[0071] Similarly, when a user makes a gesture, such as a swipe or pinch, on the virtual screen, the computing system can translate this into a corresponding action on the paired computing device, allowing the user to seamlessly interact with the virtual screen in an XR environment while taking advantage of the computing power and capabilities of the paired device.

[0072] Overall, this two-way communication between the XR device and the paired computing device enables a more integrated and streamlined user experience, where virtual and physical interfaces can be used in parallel to accomplish desired tasks.

[0073] The technology described herein refers to servers, databases, software applications, and other computer-based systems, as well as actions performed on and information sent to and from such systems. The inherent flexibility of computer-based systems allows for a wide variety of possible configurations, combinations, and divisions of tasks and functions among components and among more than two components. For example, the processes described herein can be implemented using a single device or component, or multiple devices or components operating in combination. Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can operate serially or in parallel.

[0074] Although the present subject matter has been described in detail with reference to various specific exemplary embodiments thereof, each example is provided for the purpose of illustration and not as a limitation of the present disclosure. Those skilled in the art, upon understanding the above content, can easily make modifications, variations, and equivalents of such embodiments. Thus, the present disclosure does not exclude the inclusion of such modifications, alterations, and / or additions to the present subject matter as would be readily apparent to a person skilled in the art. For example, features illustrated or described as part of one embodiment can be used with another embodiment to produce yet another embodiment. Thus, the present disclosure is intended to cover such modifications, alterations, and equivalents.

Claims

1. 1. A computer-implemented method for displaying a screen in extended reality, comprising: receiving, by a processor, video data of a real world environment of a user of the extended reality device; identifying, by the processor, screens of distinct computing devices within the video data; pairing, by the processor, the extended reality device and the separate computing device; generating, by the processor, a graphical outline of a periphery of the screen of the separate computing device; displaying, by the processor, the screen of the separate computing device and the graphical outline around the screen of the separate computing device on a display of the extended reality device, the displayed screen on the display of the extended reality device having at least one display characteristic different from the screen displayed on the separate computing device; detecting, by the processor, a user interaction with the graphical outline around the screen of the separate computing device based on the received video data; in response to detecting the user interaction, performing, by the processor, an action associated with the user interaction; 4. A computer-implemented method comprising:

2. The computer-implemented method of claim 1 , wherein the video data of the real world environment of the user is received when the extended reality device is configured to operate in a pass-through mode.

3. Identifying the screen of the separate computing device includes: identifying, by the processor, at least one image or pattern displayed on the screen of the separate computing device; uniquely identifying the separate computing device for pairing with the extended reality device based on the identification of the at least one image or pattern; The computer-implemented method of claim 1 , comprising:

4. Identifying the screen of the separate computing device includes: identifying, by the processor, a QR code displayed on the screen of the separate computing device; uniquely identifying the separate computing device for pairing with the extended reality device based on the identification of the QR code; The computer-implemented method of claim 1 , comprising:

5. Displaying the screen and the graphical outline of the separate computing device includes: determining, by the processor, a position of the screen on the display of the extended reality device; generating, by said processor, a graphical overlay for said screen; displaying, by said processor, said graphical overlay on said screen at said location on said screen; displaying, by the processor, the graphical outline around the graphical overlay on the screen; The computer-implemented method of claim 1 , comprising:

6. Detecting the user interaction with the graphical outline includes: determining, by the processor, a position of the user's hands in the video data; determining, by the processor, that a position of the hand in the video data overlaps with a position of the graphical outline; The computer-implemented method of claim 5 , comprising:

7. Detecting the user interaction with the graphical outline includes: determining, by the processor, a hand gesture of the user; selecting, by the processor, the action to perform based on the determined gesture; The computer-implemented method of claim 6 , further comprising:

8. The computer-implemented method of claim 1 , wherein the action includes moving the displayed screen and the graphical outline to a new location within the display of the extended reality device.

9. 1. A computing system for displaying a screen in extended reality, comprising: one or more processors; and one or more non-transitory computer-readable media storing instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations including: Receiving video data of a real world environment of a user of the extended reality device; identifying a screen of a distinct computing device within the video data; Pairing the extended reality device and the separate computing device; generating a graphical outline of a periphery of the screen of the separate computing device; displaying the screen of the separate computing device and the graphical outline around the screen of the separate computing device on a display of the extended reality device, the displayed screen on the display of the extended reality device having at least one display characteristic different from the screen displayed on the separate computing device; detecting a user interaction with the graphical outline around the screen of the separate computing device based on the received video data; In response to detecting the user interaction, performing an action associated with the user interaction; The computing system comprising:

10. 10. The computing system of claim 9, wherein the video data of the real world environment of the user is received when the extended reality device is configured to operate in a pass-through mode.

11. Identifying the screen of the separate computing device includes: identifying at least one image or pattern displayed on the screen of the separate computing device; uniquely identifying the separate computing device for pairing with the extended reality device based on the identification of the at least one image or pattern; The computing system of claim 9 , comprising:

12. Identifying the screen of the separate computing device includes: identifying a QR code displayed on the screen of the separate computing device; uniquely identifying the separate computing device for pairing with the extended reality device based on the identification of the QR code; The computing system of claim 9 , comprising:

13. Displaying the screen and the graphical outline of the separate computing device includes: determining a position of the screen on the display of the extended reality device; generating a graphical overlay of said screen; displaying the graphical overlay on the screen at the location on the screen; displaying the graphical outline around the graphical overlay on the screen; The computing system of claim 9 , comprising:

14. Detecting the user interaction with the graphical outline includes: determining a position of the user's hands in the video data; and determining that a position of the hand in the video data overlaps with a position of the graphical outline; The computing system of claim 13 , comprising:

15. Detecting the user interaction with the graphical outline includes: determining a hand gesture of the user; selecting the action to perform based on the determined gesture; and The computing system of claim 14 further comprising:

16. One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations, the operations including: Receiving video data of a real world environment of a user of the extended reality device; identifying a screen of a distinct computing device within the video data; Pairing the extended reality device and the separate computing device; generating a graphical outline of a periphery of the screen of the separate computing device; displaying the screen of the separate computing device and the graphical outline around the screen of the separate computing device on a display of the extended reality device, the displayed screen on the display of the extended reality device having at least one display characteristic different from the screen displayed on the separate computing device; detecting a user interaction with the graphical outline around the screen of the separate computing device based on the received video data; In response to detecting the user interaction, performing an action associated with the user interaction; [0023] In one or more non-transitory computer readable media,

17. Identifying the screen of the separate computing device includes: identifying at least one image or pattern displayed on the screen of the separate computing device; uniquely identifying the separate computing device for pairing with the extended reality device based on the identification of the at least one image or pattern; 20. The one or more non-transitory computer-readable media of claim 16, comprising:

18. Identifying the screen of the separate computing device includes: identifying a QR code displayed on the screen of the separate computing device; uniquely identifying the separate computing device for pairing with the extended reality device based on the identification of the QR code; 20. The one or more non-transitory computer-readable media of claim 16, comprising:

19. Displaying the screen and the graphical outline of the separate computing device includes: determining a position of the screen on the display of the extended reality device; generating a graphical overlay of said screen; displaying the graphical overlay on the screen at the location on the screen; displaying the graphical outline around the graphical overlay on the screen; 20. The one or more non-transitory computer-readable media of claim 16, comprising:

20. Detecting the user interaction with the graphical outline includes: determining a position of the user's hands in the video data; and determining that a position of the hand in the video data overlaps with a position of the graphical outline; 20. The one or more non-transitory computer readable media of claim 19, comprising:

Images