Remote Landmark Rendering for Extended Reality Interfaces

Abstract

Systems and techniques are described for providing a virtual interface for an XR device using a display interface device. The display interface device can display a landmark pattern, such as a Quick Response (QR) code, on its display. The display can be within the field of view of the camera of the XR device such that the camera captures an image depicting the displayed landmark pattern. The XR device can generate and display on its display an output image based on the captured image, for example, by overlaying the virtual interface over the landmark pattern. The pose and / or size of the virtual interface in the output image can be based on the pose and / or size of the landmark pattern and / or the display in the captured image. The display interface device can receive input, such as a touch-based input, via its display interface, which the display interface device can identify for the XR device.

Description

[Technical field]

[0001] TECHNICAL FIELD The present disclosure relates generally to image processing. For example, aspects of the present disclosure include systems and techniques for providing a virtual touch-based interface in extended reality (XR). [Background technology]

[0002] An extended reality (XR) device is a device that displays an environment to a user, for example through a head-mounted display (HMD), glasses, a mobile handset, or other device. The environment is at least partially distinct from the real-world environment in which the user is located. A user can generally interactively change the user's view of the environment, for example by tilting or moving the HMD or other device (e.g., by moving the user's head, etc.). Virtual reality (VR), augmented reality (AR), and mixed reality (MR) are examples of XR.

[0003] In some use cases of XR, it is useful to receive input from a user, for example, to control virtual objects or to adjust settings of the XR device itself. For example, in an XR-based video game, a user can use input to control a virtual character. A user can also use input to adjust volume levels, control song or video playback, etc. Traditional video game controllers can be inaccurate for certain types of inputs, such as fine scrolling or sliding adjustments, and can be bulky and inconvenient for users to carry. Air gestures can be inaccurate and inconsistent due to a lack of haptic feedback and a reliance on hand tracking.

[0004] Display interface devices having a display-based interface are commonly used by users, including, for example, mobile handsets, tablet devices, laptop computers, televisions, and smart watches. Display interface devices having a display-based interface can include, for example, touchscreen devices having a touchscreen interface that can display an interface on a touchscreen and receive input via the touchscreen. Display interface devices having a display-based interface can also include devices that interact with the displayed interface using a cursor, trackpad, keypad, controller, remote control, etc. Display interface devices having a display-based interface can receive precise and tactile input, allowing a user to interact with the displayed interface. Summary of the Invention

[0005] In some examples, systems and techniques are described for providing a virtual interface for an XR device using one or more display devices. The XR device can use one or more cameras to capture one or more images of a real-world scene within the field of view of the one or more cameras. The display interface device can be configured to display a recognizable landmark pattern (e.g., a quick response (QR) code or other landmark pattern) on its display. The display interface device can be within the field of view of the one or more cameras of the XR device while the display interface device is displaying the landmark pattern such that the one or more images captured by the cameras of the XR device depict the display interface device displaying the landmark pattern. The XR device can generate one or more output images based on the one or more captured images. The XR device overlays a virtual interface over the landmark pattern in the one or more output images. Overlaying the virtual interface over the landmark pattern and / or replacing the landmark pattern with the virtual interface can ensure that the virtual interface appears clear and sharp when displayed to the user by the XR device and can enable the user to provide accurate input through feedback (e.g., visual feedback, haptic feedback, audible feedback, vibration feedback, or a combination thereof). In some examples, the virtual interface is a touch-based virtual interface and the one or more display interface devices are one or more touch screen devices capable of receiving one or more touch inputs via a touch screen.The display interface device can send a display interface input identifier for the display interface input to the XR device. The display interface input identifier can identify coordinates (e.g., coordinates of a touch input, a mouse click, etc.) on the display and / or display interface of each of the one or more display interface inputs. The XR device and / or display interface device can identify whether the display interface input aligns with and thus interacts with any interface elements of the virtual interface based on the landmark pattern and the touch input identifier. The XR device and / or display interface device can update the virtual interface based on the display interface input aligning with and / or interacting with one or more interface elements of the virtual interface. The XR device can display or update virtual content within a field of view of the XR device in response to the touch input interacting with one or more interface elements of the virtual interface. The XR device can play or update audio content, and / or play or update visual content, and / or output vibrations in response to the display interface input aligning with and / or interacting with one or more interface elements of the virtual interface.

[0006] In one example, an apparatus for image processing is provided. The apparatus includes a memory and one or more processors (e.g., implemented in a circuit) coupled to the memory. The one or more processors are configured and can be: receive an input image of a scene captured by an image sensor, detect in the input image of the scene a landmark pattern displayed on a first display in the scene, determine a pose of the landmark pattern in the input image, and cause a second display to display an output image based on the input image, where a virtual interface is overlaid on the landmark pattern in the output image, and the pose of the virtual interface in the output image is based on the pose of the landmark pattern in the input image.

[0007] In another example, a method of image processing is provided that includes receiving an input image of a scene captured by an image sensor, detecting in the input image of the scene a landmark pattern displayed on a first display in the scene, determining a pose of the landmark pattern in the input image, and causing a second display to display an output image based on the input image, where a virtual interface is overlaid over the landmark pattern in the output image, and the pose of the virtual interface in the output image is based on the pose of the landmark pattern in the input image.

[0008] In another example, a non-transitory computer-readable medium is provided having instructions stored thereon that, when executed by one or more processors, cause the one or more processors to receive an input image of a scene captured by an image sensor, detect in the input image of the scene a landmark pattern displayed on a first display in the scene, determine a pose of the landmark pattern in the input image, and display on a second display an output image based on the input image, where a virtual interface is overlaid on top of the landmark pattern in the output image, and where the pose of the virtual interface in the output image is based on the pose of the landmark pattern in the input image.

[0009] In another example, an apparatus for image processing is provided that includes means for receiving an input image of a scene captured by an image sensor, means for detecting, in the input image of the scene, a landmark pattern displayed on a first display within the scene, means for determining a pose of the landmark pattern in the input image, and means for causing a second display to display an output image based on the input image, where a virtual interface is overlaid over the landmark pattern in the output image, and the pose of the virtual interface in the output image is based on the pose of the landmark pattern in the input image.

[0010] In some aspects, the landmark pattern comprises at least one of a linear glyph, a linear barcode, a barcode, a two-dimensional (2D) glyph, a 2D barcode, a quick response (QR) code, a micro QR code, a barcode, a MaxiCode, an Aztec code, a PDF417 code, an ArUco code, a data matrix, a grid matrix, a Code One code, a stacked barcode, a Schott code, a JAB code, a high capacity color barcode (HCCB), a checkerboard pattern, a three-dimensional (3D) glyph, a 3D barcode, and one or more colors.

[0011] In some aspects, the methods, apparatus, and computer-readable media described above further include identifying that the object occludes an area of ​​the first display that includes at least a portion of the landmark pattern in the input image, and causing the second display to display the output image includes occluding a portion of the virtual interface that corresponds to the area of ​​the first display in the output image.

[0012] In some aspects, the methods, apparatus, and computer-readable media described above further include generating at least a portion of the virtual interface. In some aspects, the methods, apparatus, and computer-readable media described above further include receiving at least a portion of the virtual interface from a display device including the first display.

[0013] In some aspects, the methods, apparatus, and computer-readable media described above further include generating at least a portion of the output image. In some aspects, generating at least a portion of the output image includes modifying the virtual interface using perspective distortion based on the pose of the landmark pattern in the input image.

[0014] In some aspects, the methods, apparatus, and computer-readable media described above further include generating landmark pattern data corresponding to the landmark pattern, and in response to receiving the landmark pattern data, transmitting the landmark pattern data to a display device including the first display, for the display device to display the landmark pattern on the first display.

[0015] In some aspects, the methods, apparatus, and computer-readable media described above further include receiving a display interface input identifier from a display device including the first display, the display interface indicating a portion of the first display receiving the display interface input via a display interface of the display device, the display interface being associated with the first display. In some aspects, the first display is a display layer of a touchscreen display of the display device, the display interface is a touch-sensitive layer of the touchscreen display, and the display interface input is a touch input detected by the touch-sensitive layer of the touchscreen display. In some aspects, the display interface controls a cursor on the first display, and the display interface input is a cursor input based on a position of the cursor on the first display, and the display interface includes at least one of a mouse, a trackpad, a touch-sensitive surface, a touchscreen, a joystick, a keypad, a keyboard, a button, a controller, and a remote control. In some aspects, the display interface performs hand tracking of the hand relative to the first display, and the display interface input indicates a position on the first display corresponding to the position of the hand, the display interface includes at least one of a camera and a distance sensor, and the display interface input is associated with at least one of a hand touching a position on the first display, a hand hovering over a position on the first display, a hand pointing to a position on the first display, and a hand gesturing relative to a position on the first display.

[0016] In some aspects, the above-mentioned methods, devices, and computer-readable media further include identifying that a portion of the first display indicated by the display interface input identifier aligns with a portion of the virtual interface in the output image. In some aspects, the above-mentioned methods, devices, and computer-readable media further include automatically modifying the virtual interface in response to identifying that a portion of the first display indicated by the display interface input identifier aligns with a portion of the virtual interface in the output image. In some aspects, the above-mentioned methods, devices, and computer-readable media further include receiving a second input image of the scene captured by the image sensor after capture of the input image, and causing the second display to display a second output image including virtual content overlaid on the second input image, the virtual content being automatically configured based on identifying that a portion of the first display indicated by the display interface input identifier aligns with a portion of the virtual interface in the output image. In some aspects, the above-mentioned methods, devices, and computer-readable media further include automatically outputting an audio clip in response to identifying that a portion of the first display indicated by the display interface input identifier aligns with a portion of the virtual interface in the output image. In some aspects, the methods, apparatus, and computer-readable media described above further include automatically outputting a vibration in response to identifying that a portion of the first display identified by the display interface input identifier aligns with a portion of the virtual interface in the output image.

[0017] In some aspects, the methods, apparatus, and computer-readable media described above further include determining a size of the first display in the input image, where the size of the virtual interface in the output image is based on the size of the first display in the input image. In some aspects, the methods, apparatus, and computer-readable media described above further include determining a size of a landmark pattern in the input image, where the size of the virtual interface in the output image is based on the size of the landmark pattern in the input image.

[0018] In some aspects, the methods, devices, and computer-readable media described above further comprise an image sensor. In some aspects, the methods, devices, and computer-readable media described above further comprise a second display.

[0019] In some aspects, the device is, is part of, and / or includes a wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a mobile device (e.g., a mobile phone or so-called "smartphone" or other mobile device), a wireless communication device, a camera, a personal computer, a laptop computer, a server computer, a vehicle or a computing device or component of a vehicle, a head mounted display (HMD) device, or other device. In some aspects, the device includes a camera or multiple cameras for capturing one or more images. In some aspects, the device further includes a display for displaying one or more images, notifications, and / or other displayable data. In some aspects, the devices described above can include one or more sensors (e.g., one or more inertial measurement units (IMUs), such as one or more gyrometers, one or more accelerometers, any combination thereof, and / or other sensors).

[0020] This Summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used independently to determine the scope of the claimed subject matter, which subject matter should be understood by reference to the entire specification of this patent, any or all drawings, and appropriate portions of each claim.

[0021] The above, together with other features and embodiments, will become more apparent with reference to the following specification, claims, and accompanying drawings. [Brief description of the drawings]

[0022] Exemplary embodiments of the present application are described in detail below with reference to the following drawings: [Figure 1] FIG. 1 is a block diagram illustrating an example architecture of an image capture and processing system, according to some embodiments. [Diagram 2] FIG. 1 is a block diagram illustrating an example architecture of an extended reality (XR) system having an XR device and a display interface device, according to some embodiments. [Figure 3A] FIG. 1 is a perspective view of a head mounted display (HMD) for use as an extended reality (XR) system, according to some embodiments. [Figure 3B] FIG. 3B is a perspective view illustrating the head mounted display (HMD) of FIG. 3A being worn by a user, according to some embodiments. [Figure 4A] FIG. 1 is a perspective view showing the front side of a mobile handset that includes a front-facing camera and can be used as an extended reality (XR) device or a touchscreen device, according to some embodiments. [Figure 4B] FIG. 1 is a perspective view of the back of a mobile handset that includes a rear-facing camera and can be used as an extended reality (XR) device or a display interface device, according to some embodiments. [Figure 5A]FIG. 1 is a perspective view showing a user wearing an extended reality (XR) device and holding a display interface device displaying a landmark pattern, according to some embodiments. [Figure 5B] FIG. 5B is a perspective view showing a field of view (FOV) of a user wearing the extended reality (XR) device of FIG. 5A and viewing an environment through the XR device of FIG. 5A with a virtual interface overlaid on top of a landmark pattern, according to some embodiments. [Figure 6A] FIG. 1 is a perspective view showing a user wearing an extended reality (XR) device and holding a display interface device displaying two landmark patterns that are partially occluded by the user's hands and fingers, according to some embodiments. [Figure 6B] FIG. 6B is a perspective view showing the field of view (FOV) of a user wearing the extended reality (XR) device of FIG. 6A and viewing an environment through the XR device of FIG. 6A, with a virtual interface overlaid on top of two landmark patterns and an occlusion visible above the virtual interface, according to some embodiments. [Figure 7A] FIG. 1 is a perspective view showing a user wearing an extended reality (XR) device and holding a display interface device displaying a landmark pattern and a displayed interface, according to some embodiments. [Figure 7B] FIG. 7B is a perspective view showing the field of view (FOV) of a user wearing the extended reality (XR) device of FIG. 7A and looking at an environment through the XR device of FIG. 7A, where a virtual interface has been overlaid on top of the landmark pattern but the displayed interface is still visible, according to some embodiments. [Figure 8A] FIG. 1 is a perspective view showing a user wearing an extended reality (XR) device and holding a display interface device displaying two landmark patterns, according to some embodiments. [Figure 8B] FIG. 8B is a perspective view showing the field of view (FOV) of a user wearing the extended reality (XR) device of FIG. 8A and viewing an environment through the XR device of FIG. 8A, with a virtual interface overlaid on top of two landmark patterns, according to some embodiments. [Figure 9] FIG. 1 is a conceptual diagram illustrating a display interface device that switches between displaying multiple different landmark patterns over time, according to some embodiments. [Figure 10] FIG. 2 is a swim lane diagram illustrating operations performed by an extended reality (XR) device and a display interface device to provide a virtual interface, according to some embodiments. [Figure 11] FIG. 4 is a flow diagram illustrating operations for processing image data, according to some embodiments. [Figure 12] FIG. 1 illustrates an example of a computing system for implementing some aspects described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0023] Some aspects and embodiments of the present disclosure are provided below. As will be apparent to those skilled in the art, some of these aspects and embodiments may be applied independently, and some of them may be applied in combination. In the following description, for the purpose of explanation, specific details are set forth to provide a thorough understanding of the embodiments of the present application. However, it will be apparent that various embodiments can be practiced without these specific details. The figures and descriptions are not intended to be limiting.

[0024] The following description provides only exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the present disclosure. Rather, the following description of exemplary embodiments provides those skilled in the art with an enabling description for implementing the exemplary embodiments. It should be understood that various changes may be made in the function and arrangement of elements without departing from the spirit and scope of the present application as set forth in the appended claims.

[0025] A camera is a device that uses an image sensor to receive light and capture image frames, such as still images or video frames. The terms "image," "image frame," and "frame" are used interchangeably herein. A camera can be configured with various image capture and image processing settings. Different settings result in images with different appearances. Some camera settings, such as ISO, exposure time, aperture size, F-stop, shutter speed, focus, and gain, are determined and applied before or during the capture of one or more image frames. For example, settings or parameters may be applied to an image sensor for capturing one or more image frames. Other camera settings may configure post-processing of one or more image frames, such as changing contrast, brightness, saturation, sharpness, levels, curves, or colors. For example, settings or parameters may be applied to a processor (e.g., an image signal processor or ISP) for processing one or more image frames captured by the image sensor.

[0026] An extended reality (XR) device is a device that displays an environment to a user and may include, for example, a head mounted display (HMD), glasses (e.g., augmented reality (AR) glasses), a mobile handset, or other devices. The environment may be at least partially different from the real-world environment in which the user and device are located, and may include, for example, virtual content. In some examples, the environment that the XR device displays to the user may be at least partially virtual. In some cases, the user may interactively change the user's view of the environment that the XR device displays, for example, by tilting the XR device and / or moving the XR device laterally. Tilting the XR device may include tilting or rotating along a pitch axis, a yaw axis, a roll axis, or a combination thereof. Lateral movement of the XR device may include lateral movement along a path drawn in a three-dimensional volume having three perpendicular axes, such as an X-axis, a Y-axis, and a Z-axis. An XR device that tracks only the rotation of the XR device may be referred to as an XR device having three degrees of freedom (3DoF). An XR device that tracks both tilt and lateral movement of the XR device may be referred to as an XR device with six degrees of freedom (6DoF). Extended reality (XR) may include virtual reality (VR), augmented reality (AR), mixed reality (MR), or a combination thereof.

[0027] The XR device may include sensors such as an image sensor (e.g., a camera), an accelerometer, a gyroscope, an inertial measurement unit (IMU), a light detection and ranging (LIDAR) sensor, a radio detection and ranging (RADAR) sensor, a sound detection and ranging (SODAR) sensor, a sound navigation and ranging (SONAR) sensor, one or more time-of-flight (ToF) sensors, one or more structured light sensors, one or more microphones, one or more other sensors described herein, or a combination thereof. The XR device may be an HMD, e.g., two cameras may be positioned approximately at positions on the HMD corresponding to the user's left and right eyes. The XR device may use data captured by these sensors to detect movement of the XR device within the real-world environment, e.g., such that the XR device may interactively update the user's view of the environment based on rotation and / or lateral movement of the XR device. The image sensor of the XR device may be used to capture a visual representation of the real-world environment. Some XR devices can also use data captured by these sensors to detect and / or track features of one or more objects, such as a user's hand(s) or other person(s) in the environment, for example, through the use of feature detection, feature recognition, feature tracking, object detection, object recognition, object tracking, vehicle detection, vehicle recognition, vehicle tracking, face detection, face recognition, face tracking, person detection, person recognition, person tracking, animal detection, animal recognition, animal tracking, or combinations thereof. The XR device can display content based on the sensor data captured by its sensors (e.g., a visual representation of the environment captured by an image sensor) to a user of the XR device via one or more displays of the XR device.

[0028] In some use cases of XR, it is useful to receive input from a user, for example, to control virtual objects, to interact with an interface, to adjust settings on the XR device itself, and / or to turn the XR device itself on or off. For example, in an XR-based video game, a user can use the input to control a virtual character. A user can also use the input to adjust volume levels, control playback of a song or video that the XR device is playing for the user (e.g., via functions such as pause, play, rewind, fast forward, or scrubbing), etc. Traditional video game controllers can be inaccurate for certain types of inputs, such as fine scrolling or sliding adjustments, and can be bulky and inconvenient for users to carry. Air gestures can be inaccurate due to a lack of haptic feedback and a reliance on hand tracking, can have inconsistent reliability, and can require high power consumption in the XR device.

[0029] Display interface devices with a display-based interface are commonly used by users, including, for example, mobile handsets, tablet devices, laptop computers, televisions, and smart watches. Display interface devices with a display-based interface can include, for example, touchscreen devices with a touchscreen interface that can display an interface on a touchscreen and receive input via the touchscreen. Display interface devices with a display-based interface can also include devices that use a cursor, a trackpad, a keypad, a controller, a remote control, and the like to interact with the displayed interface. Display interface devices with a display-based interface can receive precise and tactile input, allowing a user to interact with the displayed interface. For example, a touchscreen can receive tactile touch-based input to a touchscreen device, allowing a user to interact with the displayed interface precisely. However, display interface devices have not traditionally been used with XR devices. In some cases, content displayed on a display screen, such as an interface displayed on a touchscreen, can appear unclear or include one or more visual artifacts in an image captured by an image sensor of the XR device, for example, due to an asynchronous or mismatch between the refresh rate of the display screen and the capture rate of the image sensor of the XR device.

[0030] Described herein are techniques for enabling an XR device to utilize one or more display interface devices. Using such techniques, XR can provide a virtual interface for XR content presented to a user by the XR device. The virtual interface can provide real display interface input from the display interface device to the XR device (e.g., real touch-based input from a touchscreen of the display interface device). The use of real display interface input from the display interface device by the XR device can improve accuracy by providing a haptic feel (e.g., of a touchscreen, of a mouse, of a trackpad, of a keypad or one or more buttons of a controller or remote control, and / or of a joystick) to the user providing the input, and can improve over air gestures that may be imprecise and lack haptic feel or feedback. The use of real display interface input from the display interface device by the XR device can improve accuracy for certain types of input to the XR device, such as fine scroll or slide adjustments that are typically imprecise with air gestures, or other inputs that lack a display interface component.

[0031] Because display interface devices can have a variety of sizes, screen dimensions, and form factors, it may be difficult for an XR device to track the position and orientation of a display interface device. Thus, it may be difficult for an XR device to track a user's interaction with the display interface. It may also be difficult for an XR device to augment a visual representation of an interface displayed by a display interface device that the XR device captures through its camera in order to display an augmented version of the visual representation to a user of the XR device.

[0032] Rather than displaying a display interface for the XR device 202, the display interface device may be configured to display one or more landmark patterns. The one or more landmark patterns may be designed to enable the XR device to effectively recognize a visual representation of the landmark patterns in image data captured by a camera of the XR device. The one or more landmark patterns may be designed to enable the XR device to effectively track the pose and / or size of the visual representation of the landmark patterns in the image data over time. The pose may include position (e.g., two-dimensional coordinates in the image and / or three-dimensional coordinates in the environment), orientation (e.g., pitch, yaw, and / or roll), or a combination thereof. The size may include three-dimensional volume, two-dimensional area, and / or one-dimensional measurements (e.g., height, width). The size may be absolute or relative. By tracking the pose of the landmark patterns, the XR device may also track the pose and / or size of the display of the display interface device.

[0033] In some examples, the one or more landmark patterns can include one or more Quick Response (QR) codes, one or more Micro QR codes, one or more barcodes, one or more MaxiCodes, one or more Aztec codes, one or more PDF417 codes, one or more ArUco codes, one or more Data Matrix, one or more Grid Matrix, one or more Code One codes, one or more Stack Barcodes, one or more Schott Codes, one or more JAB codes, one or more High Capacity Color Barcodes (HCCB), one or more two-dimensional (2D) barcodes, one or more three-dimensional (3D) barcodes, one or more checkerboard patterns, another type of recognizable glyph or pattern, or a combination thereof. In the output image that the XR device displays to its user, the XR device can overlay a virtual interface on top of the landmark pattern, replace the landmark pattern with a virtual interface, or a combination thereof. The pose and / or size of the virtual interface can be based on the pose and / or size of the landmark pattern and / or based on the pose and / or size of the display of the display interface device. For example, the XR device may position, move, resize, resample, rescale, upsample, upscaling, downsampling, downscaling, magnify, shrink, rotate, skew, warp (e.g., perspective warping), and / or distort (e.g., perspective distortion) the virtual interface to simulate the pose and / or size of the landmark pattern and / or the display of the display interface device.

[0034] In an example embodiment, the XR device can capture an image of a scene using an image sensor of the XR device. The scene includes at least a portion of a display of a display interface device displaying a landmark pattern on its display. The display is within a field of view of the image sensor of the XR device, and the image depicts at least the landmark pattern as it is displayed on the display of the display interface device. The XR device can detect and / or identify, within the image of the scene, a visual representation (e.g., a depiction) of the landmark pattern displayed on the display of the display interface device. The XR device can determine a pose and / or size of the landmark pattern and thus a pose and / or size of the display of the display interface device based on the visual representation of the landmark pattern. The XR device can generate an output image based on the image of the scene and based on the pose and / or size of the display of the display interface device. For example, the output image may be a modified version of the captured image in which the XR device overlays a virtual interface over the visual representation of the landmark pattern, and in some cases replaces the visual representation of the landmark pattern with the virtual interface. The XR device can position, orient, resize, rotate, skew, warp, and / or skew the virtual interface to have a virtual and / or simulated pose based on the recognized pose of the landmark pattern and / or the recognized pose of the display of the display interface device. The XR device can position, orient, resize, rotate, skew, warp, and / or skew the virtual interface to have a virtual and / or simulated size based on the recognized size of the landmark pattern and / or the recognized size of the display of the display interface device. The XR device can display the output image to the user via one or more displays of the XR device.Thus, to a user of the XR device, the virtual interface may appear to be displayed on the display of the display interface device 250 over and / or in place of the landmark pattern.

[0035] As described above, by providing the XR device with a display interface of a display interface device that can receive display interface input from a display interface device, the accuracy of the input used by the XR device can be improved, a haptic sensation can be provided to the user providing the input, and an improvement can be made for air gestures. Similarly, by providing the XR device with a display interface that can receive display interface input from a display interface device, the accuracy can be improved for certain types of input to the XR device, such as fine scrolling or sliding adjustments that are typically imprecise with game controllers (without a corresponding display interface), air gestures, or other input actions that lack a corresponding display interface. Overlaying a virtual interface over the landmark pattern and / or replacing the landmark pattern with a virtual interface can ensure that the virtual interface appears clear and sharp when displayed to the user by the XR device, overcoming any visual artifacts (e.g., scan lines or chromatic aberrations) that may otherwise be caused by asynchrony or mismatch between the refresh rate of the display of the display interface device and the capture rate of the image sensor of the XR device, and / or the interaction between light from the display of the display interface device and the lens and / or other optical elements of the image capture hardware of the XR device. Technical improvements therefore include the reduction or elimination of such visual artifacts, as well as improving the clarity and sharpness of the interface that is displayed to a user of the XR device using the display of the XR device.Overlaying a virtual interface over a landmark pattern and / or replacing a landmark pattern with a virtual interface can also allow the virtual interface to appear clearer than would be possible on the display of the display interface device, for example allowing the virtual interface to exceed the limited resolution and / or color gamut and / or color range of the display of the display interface device. Thus, technical improvements include improvements in the resolution, clarity, and / or sharpness of the interface. Overlaying a virtual interface over a landmark pattern and / or replacing a virtual interface with a virtual interface can also allow the virtual interface to appear more colorful than would be possible on the display of the display interface device, for example allowing the virtual interface to be full color even if the display of the display interface device is a monochrome or limited color display (e.g., an e-ink display or e-paper display as used in e-reader devices). Thus, technical improvements include improvements in the color and / or capabilities of the interface. Overlaying a virtual interface over a landmark pattern and / or replacing a virtual interface with a virtual interface can also allow portions of the virtual interface to extend beyond the physical dimensions of the display of the display interface device and / or the display interface device itself. For example, portions of the virtual interface may appear to extend beyond the edges of the display of the display interface device and / or beyond the edges of the display interface device itself. Thus, technical improvements include improvements in the dimensions and / or size of the interface.The display interface device may also provide vibration feedback to the user, for example as haptic feedback for a button press, a touch-based interaction, a "rumble" vibration based on detection of a particular event or condition (e.g., a video game event), or a combination thereof. Thus, technical improvements include improvements to feedback from interface interactions. Knowledge of how the landmark pattern should look on the XR device may also enable the XR device to identify, with improved accuracy, the exact metes and bounds of any occlusion (e.g., the user's fingers and / or hands) that occludes at least a portion of the landmark pattern, allowing the XR device to accurately reproduce the occlusion to occlude a corresponding portion of the virtual interface. Thus, technical improvements include improvements to occlusion representation and accuracy. Different landmark patterns may be displayed on the display of the display interface device over time according to a pre-set schedule, allowing the XR device to improve latency detection based on the time difference between the display interface device changing the landmark pattern and the XR device's detection of the change to the landmark pattern, thus improving time synchronization between the display interface device and the XR device. Technical improvements therefore include improvements to mapping display interface inputs to time, and improved synchronization of display interface inputs to the virtual interface. Overlaying the virtual interface on top of the landmark pattern and / or replacing the landmark pattern with the virtual interface can also enable the virtual interface to remain private to the user of the XR device, since anyone viewing the display of the display interface device who is not the user of the XR device will simply see the landmark pattern, rather than the virtual interface that the XR device overlays on top of the landmark pattern.Thus, technical improvements include improvements in security and privacy regarding the interface.

[0036] 1 is a block diagram illustrating the architecture of an image capture and processing system 100. The image capture and processing system 100 includes various components that are used to capture and process an image of a scene (e.g., an image of a scene 110). The image capture and processing system 100 can capture a standalone image (or photograph) and / or can capture a video that includes multiple images (or video frames) in a particular sequence. A lens 115 of the system 100 faces the scene 110 and receives light from the scene 110. The lens 115 bends the light toward an image sensor 130. The light received by the lens 115 passes through an aperture controlled by one or more control mechanisms 120 and is received by the image sensor 130.

[0037] The one or more controls 120 may control exposure, focus, and / or zoom based on information from image sensor 130 and / or based on information from image processor 150. The one or more controls 120 may include multiple mechanisms and components. For example, the control 120 may include one or more exposure controls 125A, one or more focus controls 125B, and / or one or more zoom controls 125C. The one or more controls 120 may also include additional controls other than those shown, such as controls to control analog gain, flash, HDR, depth of field, and / or other image capture characteristics.

[0038] The focus control mechanism 125B of the control mechanism 120 can obtain the focus setting. In some examples, the focus control mechanism 125B stores the focus setting in a memory register. Based on the focus setting, the focus control mechanism 125B can adjust the position of the lens 115 relative to the position of the image sensor 130. For example, based on the focus setting, the focus control mechanism 125B can move the lens 115 closer to or farther from the image sensor 130 by actuating a motor or servo, thereby adjusting the focus. In some cases, additional lenses, such as one or more microlenses above each photodiode of the image sensor 130, may be included in the system 100, each of which bends light received from the lens 115 toward a corresponding photodiode before the light reaches the photodiode. The focus setting may be determined via contrast detection autofocus (CDAF), phase detection autofocus (PDAF), or some combination thereof. The focus settings may be determined using the control mechanism 120, the image sensor 130, and / or the image processor 150. The focus settings may be referred to as image capture settings and / or image processing settings.

[0039] The exposure control 125A of the control mechanism 120 can obtain the exposure setting. In some cases, the exposure control 125A stores the exposure setting in a memory register. Based on the exposure setting, the exposure control 125A can control the size of the aperture (e.g., aperture size or F-stop), the duration the aperture is open (e.g., exposure time or shutter speed), the sensitivity of the image sensor 130 (e.g., ISO speed or film speed), the analog gain applied by the image sensor 130, or any combination thereof. The exposure setting may be referred to as an image capture setting and / or an image processing setting.

[0040] The zoom control 125C of the control mechanism 120 can obtain the zoom setting. In some examples, the zoom control 125C stores the zoom setting in a memory register. Based on the zoom setting, the zoom control 125C can control the focal length of an assembly of lens elements (lens assembly) including the lens 115 and one or more additional lenses. For example, the zoom control 125C can control the focal length of the lens assembly by actuating one or more motors or servos to move one or more of the lenses relative to each other. The zoom setting may be referred to as an image capture setting and / or an image processing setting. In some examples, the lens assembly may include a parfocal zoom lens or a variable focus zoom lens. In some examples, the lens assembly may include a focusing lens (which may be the lens 115 in some cases) that first receives light from the scene 110, and then the light passes through an afocal zoom system between the focusing lens (e.g., the lens 115) and the image sensor 130 before the light reaches the image sensor 130. In some cases, an afocal zoom system may include two positive (e.g., converging, convex) lenses of equal or similar focal lengths (e.g., within a threshold difference) with a negative (e.g., diverging, concave) lens between them. In some cases, the zoom control 125C moves one or more of the lenses in the afocal zoom system, such as one or both of the negative and positive lenses.

[0041] The image sensor 130 includes one or more arrays of photodiodes or other light-sensitive elements. Each photodiode measures an amount of light that ultimately corresponds to a particular pixel in the image produced by the image sensor 130. In some cases, different photodiodes may be covered by different color filters and may therefore measure light that matches the color of the filter covering the photodiode. For example, a Bayer color filter includes a red color filter, a blue color filter, and a green color filter, and each pixel of the image is generated based on red light data from at least one photodiode covered by a red color filter, blue light data from at least one photodiode covered by a blue color filter, and green light data from at least one photodiode covered by a green color filter. Other types of color filters may use yellow, magenta, and / or cyan (also called "emerald") color filters instead of or in addition to red, blue, and / or green color filters. Some image sensors may be completely devoid of color filters and instead use different photodiodes (possibly stacked vertically) across the entire pixel array. Different photodiodes across the pixel array can have different spectral sensitivity curves and therefore respond to different wavelengths of light. Monochrome image sensors may also lack color filters and therefore no color depth.

[0042] In some cases, image sensor 130 may alternatively or additionally include an opaque and / or reflective mask that blocks light from reaching some photodiodes or portions of some photodiodes at some times and / or from some angles, which may be used for phase detection autofocus (PDAF). Image sensor 130 may also include an analog gain amplifier for amplifying an analog signal output by the photodiode and / or an analog-to-digital converter (ADC) for converting an analog signal output from the photodiode (and / or amplified by the analog gain amplifier) ​​to a digital signal. In some cases, instead or in addition, some components or functions described with respect to one or more of control mechanisms 120 may be included in image sensor 130. The image sensor 130 may be a charge-coupled device (CCD) sensor, an electron-multiplying CCD (EMCCD) sensor, an active pixel sensor (APS), a complementary metal-oxide semiconductor (CMOS), an n-type metal-oxide semiconductor (NMOS), a hybrid CCD / CMOS sensor (e.g., sCMOS), or some other combination thereof.

[0043] Image processor 150 may include one or more processors, such as one or more image signal processors (ISP) (including ISP 154), one or more host processors (including host processor 152), and / or one or more of any other types of processors 1210 discussed with respect to computing device 1200. Host processor 152 may be a digital signal processor (DSP) and / or other types of processors. In some implementations, image processor 150 is a single integrated circuit or chip (called a system-on-chip or SoC) that includes host processor 152 and ISP 154. In some cases, the chip may also include one or more input / output ports (e.g., input / output (I / O) ports 156), a central processing unit (CPU), a graphics processing unit (GPU), a broadband modem (e.g., 3G, 4G, or LTE, 5G, etc.), memory, connectivity components (e.g., Bluetooth™, Global Positioning System (GPS), etc.), any combination thereof, and / or other components. The I / O ports 156 may include any suitable input / output ports or interfaces according to one or more protocols or specifications, such as an Inter Integrated Circuit 2 (I2C) interface, an Inter Integrated Circuit 3 (I3C) interface, a serial peripheral interface (SPI) interface, a serial general purpose input / output (GPIO) interface, a Mobile Industry Processor Interface (MIPI) (such as MIPI CSI-2), a physical (PHY) layer port or interface, an Advanced High Performance Bus (AHB) bus, any combination thereof, and / or other input / output ports. In one illustrative example, the host processor 152 may communicate with the image sensor 130 using an I2C port and the ISP 154 may communicate with the image sensor 130 using a MIPI port.

[0044] Image processor 150 may perform several tasks such as demosaicing, color space conversion, image frame downsampling, pixel interpolation, automatic exposure (AE) control, automatic gain control (AGC), CDAF, PDAF, automatic white balance, merging image frames to form HDR images, image recognition, object recognition, feature recognition, receiving input, managing output, managing memory, or any combination thereof. Image processor 150 may store image frames and / or processed images in random access memory (RAM) 140 and / or 1225, read-only memory (ROM) 145 and / or 1220, a cache, a memory unit, another storage device, or any combination thereof.

[0045] Various input / output (I / O) devices 160 may be connected to image processor 150. I / O devices 160 may include a display screen, a keyboard, a keypad, a touch screen, a track pad, a touch-sensitive surface, a printer, any other output device 1235, any other input device 1245, or some combination thereof. In some cases, captions may be entered into image processing device 105B through a physical keyboard or keypad of I / O device 160 or through a virtual keyboard or keypad of a touch screen of I / O device 160. I / O 160 may include one or more ports, jacks, or other connectors that enable a wired connection between system 100 and one or more peripheral devices, through which system 100 may receive data from and / or transmit data to one or more peripheral devices. I / O 160 may include one or more wireless transceivers that enable a wireless connection between system 100 and one or more peripheral devices, through which system 100 may receive data from and / or transmit data to one or more peripheral devices. The peripheral devices may include any of the types of I / O devices 160 previously described, and may themselves be considered I / O devices 160 when coupled to a port, jack, wireless transceiver, or other wired and / or wireless connector.

[0046] In some cases, image capture and processing system 100 may be a single device. In some cases, image capture and processing system 100 may be two or more separate devices including image capture device 105A (e.g., a camera) and image processing device 105B (e.g., a computing device coupled to a camera). In some implementations, image capture device 105A and image processing device 105B may be coupled together, for example, via one or more wires, cables, or other electrical connectors and / or wirelessly via one or more wireless transceivers. In some implementations, image capture device 105A and image processing device 105B may be separate from one another.

[0047] As shown in Fig. 1, a vertical dashed line divides image capture and processing system 100 of Fig. 1 into two portions representing image capture device 105A and image processing device 105B, respectively. Image capture device 105A includes lens 115, control mechanism 120, and image sensor 130. Image processing device 105B includes image processor 150 (including ISP 154 and host processor 152), RAM 140, ROM 145, and I / O 160. In some cases, some components shown in image capture device 105A, such as ISP 154 and / or host processor 152, may be included within image capture device 105A.

[0048] The image capture and processing system 100 may include an electronic device, such as a mobile or fixed telephone handset (e.g., a smartphone, a mobile phone, etc.), a desktop computer, a laptop or notebook computer, a tablet computer, a set-top box, a television, a camera, a display device, a digital media player, a video gaming console, a video streaming device, an Internet Protocol (IP) camera, or any other suitable electronic device. In some examples, the image capture and processing system 100 may include one or more wireless transceivers for wireless communication, such as cellular network communication, 802.11 wi-fi communication, wireless local area network (WLAN) communication, or any combination thereof. In some implementations, the image capture device 105A and the image processing device 105B may be different devices. For example, the image capture device 105A may include a camera device, and the image processing device 105B may include a computing device, such as a mobile handset, a desktop computer, or other computing device.

[0049] Although image capture and processing system 100 is shown as including several components, one skilled in the art will appreciate that image capture and processing system 100 may include many more components than those shown in FIG. 1. The components of image capture and processing system 100 may include software, hardware, or one or more combinations of software and hardware. For example, in some implementations, the components of image capture and processing system 100 may include and / or be implemented using electronic circuitry or other electronic hardware, which may include one or more programmable electronic circuits (e.g., microprocessors, GPUs, DSPs, CPUs, and / or other suitable electronic circuits), and / or may include and / or be implemented using computer software, firmware, or any combination thereof, to perform various operations described herein. The software and / or firmware may include one or more instructions stored in a computer-readable storage medium and executable by one or more processors of an electronic device implementing image capture and processing system 100.

[0050] Described herein are systems, apparatus, processes, and computer-readable media for providing a virtual interface for an XR device using a display interface device. The display interface device may be a touch screen device, including a touch screen, in some examples. The display interface device may include one or more display interfaces for interacting with an interface displayed on the display, such as a touch screen, a mouse for controlling a cursor or other display element on the display, a track pad for controlling a cursor or other display element on the display, a keypad for controlling text input and / or cursor or other display element on the display, a controller for controlling text input and / or cursor or other display element on the display, a remote control for controlling text input and / or cursor or other display element on the display, a hand tracker for hand tracking interaction with the display using a camera (e.g., of the XR device and / or the display interface device), a hand tracker for hand tracking interaction with the display using an active depth sensor (e.g., RADAR, LIDAR, SONAR, SODAR, structured light, time of flight) (e.g., of the XR device and / or the display interface device), a hand tracker for hand tracking interaction with the display using an ultrasonic sensor (e.g., of the XR device and / or the display interface device), another input device 1245 for providing an input interface associated with the display, or a combination thereof. The XR device may include one or more cameras capable of capturing one or more images. The one or more cameras may each include an image capture and processing system 100, an image capture device 105A, an image processing device 105B, an image sensor 130, or a combination thereof.The XR device can process the image and detect within the image a representation of a landmark pattern displayed on a display of a display interface device that is within the field of view of the image sensor. The XR device 202 can detect within the image a pose and / or size of the landmark pattern in the image and can overlay a virtual interface over the landmark pattern such that the virtual interface has a corresponding pose and / or size in an output image that the XR device displays to its user.

[0051] 2 is a block diagram illustrating an example architecture of an extended reality (XR) system 200 having an XR device 202 and a display interface device 250, according to some embodiments. The display interface device 250 can be referred to as a display device. The XR device 202 of the XR system 200 includes an XR application 204 that can be stored by a memory of the XR device 202 and / or executed by one or more processors of the XR device 202. The XR device 202 includes one or more inertial sensors 216 that can include one or more accelerometers, one or more gyroscopes, one or more positioning receivers, one or more inertial measurement units (IMUs), or combinations thereof. The position receiver can include a global navigation satellite system (GNSS) receiver, such as a global positioning system (GPS) receiver. The positioning receiver can include a beacon-based short-range radio signal receiver that receives short-range radio signals from a beacon device that transmits short-range radio signals. The positioning receiver can include a wireless local area network (WLAN) receiver, such as a Wi-Fi receiver. The positioning receiver can include a cellular network receiver, such as a 3G, 4G, LTE, or 5G network receiver. Using one or more inertial sensors 216, the XR device 202 (e.g., an XR application) can perform inertial tracking 218 of the XR device 202. In some embodiments, the inertial tracking 218 can be performed by the XR application 204 of the XR device 202, by an inertial tracking engine of the XR device 202, by a processor of the XR device 202, or by a combination thereof. The inertial tracking 218 can include tracking the orientation of the XR device 202 within its real-world environment.The attitude of the XR device 202 can include the position (e.g., three-dimensional coordinates in the environment and / or two-dimensional coordinates such as latitude and longitude), orientation (e.g., pitch, yaw, and / or roll) of the XR device 202, or a combination thereof. The inertial tracking 218 can include tracking the attitude of the XR device 202 in three degrees of freedom (3DoF), six degrees of freedom (6DoF), or a combination thereof.

[0052] The XR device 202 may include one or more image sensors 220. Each of the one or more image sensors 220 may be an example of the image sensor 130 of FIG. 1. In some examples, each of the one or more image sensors 220 includes the image capture device 105A, the image processing device 105B, the image capture and processing system 100 of FIG. 1, or a combination thereof. The one or more image sensors 220 may capture one or more images of a scene. In some examples, the scene may include at least a portion of the display interface device 250 such that the one or more images of the scene depict at least a portion of the display interface device 250 (e.g., at least a portion of the display 260 and / or the display interface 261). FIG. 2 shows a shaded triangle extending to the right from a box representing the one or more image sensors 220. The shaded triangle represents a possible field of view (FOV) of the one or more image sensors 220. The shaded triangle extends to a box of display interface device 250 representing display 260 and / or display interface 261 of display interface device 250 to indicate that the FOV of one or more image sensors 220 may include at least a portion of display 260 and / or display interface 261 of display interface device 250. In some embodiments, display 260 may be the display of a touchscreen display and display interface 261 may be the touch-sensitive layer of a touchscreen display. In embodiments where display 260 and / or display interface 261 is a touchscreen display,

[0053] The one or more images captured by the one or more image sensors 220 of the XR device 202 can be used for inertial tracking 218. For example, the one or more images captured by the one or more image sensors 220 of the XR device 202 can include successive video frames that can be compared to one another to identify changes to the FOV of the one or more image sensors 220 of the XR device 202 indicative of movement of the XR device 202 along one or more of the 6 DoF. The one or more images captured by the one or more image sensors 220 of the XR device 202 can be used for hand tracking 222 to track the posture(s) of one or both hands of a user of the XR device 202 and / or one or more hands of other individuals within the FOV of the one or more image sensors 220 of the XR device 202. In some examples, the hand tracking 222 can be performed by the XR application 204 of the XR device 202, by a hand tracking engine of the XR device 202, by a processor of the XR device 202, or by a combination thereof. In some embodiments, the hand tracking 222 utilizes feature detection, feature recognition, feature tracking, object detection, object recognition, object tracking, hand detection, hand recognition, hand tracking, finger detection, finger recognition, finger tracking, person detection, person recognition, person tracking, face detection, face recognition, face tracking, face detection, or a combination thereof. In some embodiments, the hand tracking 222 can also be performed using inertial tracking data from the inertial tracking 218 (e.g., indicative of the pose of the XR device 202) by the XR device 202, for example, to help identify which movements of the hand(s) in the image(s) captured by the image sensor(s) 220 actually represent the movement of the hand(s) in the environment, and which movements of the hand(s) in the image(s) captured by the image sensor(s) are caused by the movement of the XR device 202 in the environment.

[0054] One or more images captured by one or more image sensors 220 of the XR device 202 can be used for display interface device tracking 224 to track the pose of the display interface device 250, for example, by detecting and tracking the pose and / or size of a landmark pattern displayed by the display 260 and / or display interface 261 of the display interface device 250. The display interface device tracking 224 can be performed by the XR application 204 of the XR device 202, by a display interface device tracking engine of the XR device 202, by a processor of the XR device 202, or by a combination thereof. As described above, the pose can include a position (e.g., two-dimensional coordinates in the image and / or three-dimensional coordinates in the environment), an orientation (e.g., pitch, yaw, and / or roll), or a combination thereof. The size can include a three-dimensional volume, a two-dimensional area, and / or a one-dimensional measurement (e.g., height, width). The size can include an absolute size and / or a relative size (relative to others in the image data). The size of the landmark pattern may be identified as an absolute size or as a size relative to another size in the image data (eg, the size of the display interface device 250 and / or its display).

[0055] In some embodiments, the XR device 202 can store a reference copy of the landmark pattern, or landmark pattern data characterizing the landmark pattern, and / or can be used to recreate a reference copy of the landmark pattern (e.g., data encoded by a QR code or barcode or other coding scheme of the landmark pattern). The XR device 202 can compare a representation of the landmark pattern in one or more images captured by the one or more image sensors 220 of the XR device 202 to the stored reference copy of the landmark pattern to identify how the representation of the landmark pattern in the one or more images has been positioned, moved, resized, rescaled, resampled, rotated, skewed, warped (e.g., perspective warping), and / or distorted (e.g., perspective distortion) relative to the stored reference copy of the landmark pattern. In this manner, the XR device 202 can identify a pose of the landmark pattern, which can match a pose of the display 260 and / or display interface 261 of the display interface device 250. Similarly, the XR device 202 can identify a size of the landmark pattern and can determine a size of the display 260 and / or display interface 261 of the display interface device 250 based on the size of the landmark pattern. In some embodiments, the display interface device tracking 224 utilizes feature detection, feature recognition, feature tracking, object detection, object recognition, object tracking, or a combination thereof.

[0056] For example, the pose of the landmark pattern may be the pose of the display 260 of the display interface device 250. The display interface device 250 may be configured and may display the landmark pattern such that the landmark pattern covers a predetermined amount or percentage of the surface of the display 260 (e.g., 10%, 20%, 30%, 40%, 50%, 60%, 70%, 80%, 90%, 100%, or a value between any two of the values ​​previously enumerated). For example, if the display interface device 250 is configured to display the landmark pattern across 100% of the display 260, the XR device 202 may determine that the size of the display 260 is equal to the size of the landmark pattern. If the display interface device 250 is configured to display the landmark pattern across 50% of the display 260, the XR device 202 may determine that the size of the display 260 may be twice the size of the landmark pattern, and so on.

[0057] In some examples, the XR device 202 may also track the pose and / or size of the display 260 and / or the display interface device 250 separately from tracking the landmark patterns, e.g., based on feature extraction, feature detection, and recognition via one or more features of the corners and / or edges of the display 260 and / or the display interface device 250. In some examples, the display interface device 250 may display other content on the display 260 (e.g., highly saturated colors or patterns, or any of those identified herein that may be landmark patterns) at least near one or more of the edges and / or corners of the display 260 to facilitate the XR device 202 in identifying and extracting features based on the edge and / or corner features of the display 260. In some embodiments, the display interface device 250 can transmit information about the display interface device 250 to the XR device 202 (e.g., via the wireless transceiver 270 as part of block 272), and the XR device 202 can receive information about the display interface device 250 from the display interface device 250 (e.g., via the wireless transceiver 230 as part of block 228). The information about the display interface device 250 can, in some cases, identify the size and / or dimensions of the display 260, the display interface 261, and / or the display interface device 250.The information about the display interface device 250 may optionally include an identifier (e.g., model name and / or brand) of the display interface device 250, based on which the XR device 202 may determine the size and / or dimensions of the display 260, the display interface 261, and / or the display interface device 250 (e.g., by querying a database or table or other data structure using the identifier of the display interface device 250). In some examples, the display interface device 250 may send information about the display interface device 250 to the XR device 202 in response to the XR device 202 sending a request for information about the display interface device 250 to the display interface device 250 (e.g., via the wireless transceiver 230 as part of block 226) and the display interface device 250 receiving a request from the XR device 202 (e.g., via the wireless transceiver 270 as part of block 274). The size of the display 260, the landmark pattern(s) displayed thereon, the display interface 261, and / or the display interface device 250 may be important in order to scale the virtual interface and / or virtual content to be larger for larger displays 260 and / or display interface devices 250 and smaller for smaller displays 260 and / or display interface devices 250.

[0058] In some embodiments, inertial tracking data from inertial tracking 218 (e.g., indicative of the pose of the XR device 202) can also be used by the XR device 202 to perform display interface device tracking 224, for example, to help identify which movements of the display interface device 250 in the image(s) captured by the image sensor(s) 220 actually represent movements of the display interface device 250 in the environment, and which movements of the display interface device 250 in the image(s) captured by the image sensor(s) are caused by movements of the XR device 202 in the environment. In some embodiments, hand tracking data from hand tracking 222 (e.g., indicative of the pose of the hand) can also be used by the XR device 202 to perform display interface device tracking 224, for example, to help identify the pose of the display interface device 250 by tracking the pose of the hand holding and / or interacting with the display interface device 250. In some examples, display interface device tracking data from display interface device tracking 224 (e.g., indicative of a pose of the display interface device 250) can be used by the XR device 202 to perform hand tracking 222, e.g., to track the pose of the display interface device 250 and help identify the pose of a hand holding and / or interacting with the display interface device 250. In some examples, hand tracking data from hand tracking 222 (e.g., indicative of a pose of the hand) can also be used by the XR device 202 to perform inertial tracking 218, e.g., to help identify the pose of the XR device 202 relative to one or more hand(s) in the environment.In some embodiments, display interface device tracking data (e.g., indicative of the pose of the display interface device 250) from the display interface device tracking 224 can be used by the XR device 202 to perform inertial tracking 218, for example, to help identify the pose of the XR device 202 relative to the display interface device 250 in the environment.

[0059] The XR application 204 may include a landmark pattern generator 214. In some embodiments, the landmark pattern generator 214 of the XR application 204 may generate a landmark pattern that is displayed on the display 260 of the display interface device 250. The landmark pattern generator 214 of the XR application 204 may also store a reference copy of the landmark pattern in the XR device 202, which the XR device 202 may use for the display interface device tracking 224 by comparing a depiction of the landmark pattern in the image(s) captured by the image sensor(s) 220 to the reference copy of the landmark pattern. In some embodiments, the landmark pattern generator 214 of the XR application 204 may generate initial landmark pattern data that may be used by the XR device 202 or the display interface device 250 (e.g., by the landmark pattern generator 258) to actually generate the landmark pattern. The landmark pattern may be or may include a linear glyph, such as a linear barcode. The landmark pattern may be or include one or more of a two-dimensional (2D) glyph, such as a 2D barcode, a Quick Response (QR) code, a micro QR code, a barcode, a MaxiCode, an Aztec code, a PDF417 code, an ArUco code, a Data Matrix, a Grid Matrix, a Code One code, a stacked barcode, a Schott code, a JAB code, a High Capacity Color Barcode (HCCB), a checkerboard pattern, or any combination thereof. The landmark pattern may be or include a three-dimensional (3D) glyph, such as a 3D barcode. The landmark pattern may be or include one or more predefined colors (e.g., highly saturated colors that can stand out in an image). The landmark pattern may be or include another type of recognizable glyph or pattern.In some examples, the landmark pattern may visually encode initial landmark pattern data. For example, barcodes, QR codes, Aztec codes, MaxiCodes, PDF417 codes, and many of the other examples of landmark patterns identified above are visual encodings of data and may be used to visually encode text and / or other types of data. The initial landmark pattern data may include text and / or other types of data visually encoded in such landmark patterns.

[0060] The XR device 202 may include one or more wireless transceivers 230. The XR device 202 may transmit the landmark pattern data to the display interface device 250 using the one or more wireless transceivers 230 (block 226). The transmitting of the landmark pattern data (block 226) may be performed by the XR application 204 of the XR device 202, the wireless transceiver(s) 230, by a processor of the XR device 202, or a combination thereof. The display interface device 250 may receive the landmark pattern data at one or more wireless transceivers 270 of the display interface device 250 (block 274). The receiving of the landmark pattern data (block 274) may be performed by the display interface application 252 of the display interface device 250, the wireless transceiver(s) 270, by a processor of the display interface device 250, or a combination thereof. The landmark pattern data transmitted from the XR device 202 to the display interface device 250 (block 226) and received at the display interface device 250 (block 274) can include the landmark pattern itself, for example, if the XR device 202 generates the landmark pattern entirely in the landmark pattern generator 214 of the XR device 202. The landmark pattern data transmitted from the XR device 202 to the display interface device 250 (block 226) and received at the display interface device 250 (block 274) can include initial landmark pattern data representing data that the landmark pattern encodes using a visual encoding scheme (e.g., QR code, Aztec code, etc.), for example, if the XR device 202 generates initial landmark pattern data in the landmark pattern generator 214 of the XR device 202.

[0061] The display interface device 250 of the XR system 200 includes a display interface application 252 that may be stored by a memory of the display interface device 250 and / or executed by one or more processors of the display interface device 250. The display interface application 252 may include a landmark pattern generator 258. The landmark pattern generator 258 may generate a landmark pattern that is displayed on a touch screen 260 of the display interface device 250. The display interface device 250 may receive landmark pattern data from the XR device 202 using the wireless transceiver(s) 270 of the display interface device 250 (block 274), as described above. In some embodiments, the landmark pattern generator 258 of the display interface application 252 of the display interface device 250 generates the landmark pattern based on the landmark pattern data received by the display interface device 250 from the XR device 202 (block 274). As described above, the landmark pattern data received by the display interface device 250 from the XR device 202 (block 274) may include the landmark pattern itself. If the received landmark pattern data (block 274) includes the landmark pattern itself, the landmark pattern generator 258 may generate the landmark pattern by using the landmark pattern received in the landmark pattern data received by the display interface device 250 from the XR device 202 (block 274).If the received landmark pattern data (block 274) includes the landmark pattern itself, the landmark pattern generator 258 may generate the landmark pattern by resizing, resampling, rotating, and / or otherwise processing the landmark pattern received in the landmark pattern data received (block 274) by the display interface device 250 from the XR device 202. As described above, the landmark pattern data received (block 274) by the display interface device 250 from the XR device 202 may include initial landmark pattern data that represents data that the landmark pattern encodes using a visual encoding scheme (e.g., a QR code, an Aztec code, etc.). If the received landmark pattern data (block 274) includes initial landmark pattern data, the landmark pattern generator 258 may generate the landmark pattern by encoding the initial landmark pattern data using a visual encoding scheme.

[0062] In some examples, the landmark pattern generator 258 of the display interface application 252 of the display interface device 250 generates the landmark pattern without receiving (at block 274) based on the landmark pattern data at the display interface device 250 from the XR device 202. For example, the landmark pattern generator 258 can generate the initial landmark pattern data and / or the landmark pattern itself entirely by itself. The display interface device 250 can transmit the landmark pattern data to the XR device 202 using its wireless transceiver(s) 270 (block 272). The XR device 202 can receive the landmark pattern data using its wireless transceiver(s) 230 (block 228), and the XR device 202 can use this landmark pattern data to have a reference landmark pattern to compare with the depiction of the landmark pattern in the image(s) captured by the image sensor(s) 220 of the XR device 202. Receiving the landmark pattern data (block 228) may be performed by the XR application 204 of the XR device 202, the wireless transceiver(s) 230, by a processor of the XR device 202, or by a combination thereof. The landmark pattern data transmitted (block 272) from the display interface device 250 to the XR device 202 and received (block 228) by the XR device 202 from the display interface device 250 may include initial landmark pattern data and / or the landmark pattern itself. If the received landmark pattern data (block 228) includes initial landmark pattern data, the landmark pattern generator 214 may generate the landmark pattern by encoding the initial landmark pattern data using a visual encoding scheme.

[0063] The display interface device 250 may display 262 content on its display 260 and / or display interface 261. Displaying 262 content may be performed by a display interface application 252 of the display interface device 250, a display interface generator 254, a landmark pattern generator 258, the display 260, the display interface 261, a display controller of the display 260, by a processor of the display interface device 250, or by a combination thereof. The content displayed (block 262) on the display 260 and / or display interface 261 by the display interface device 250 may include one or more landmark patterns, which may be generated by the landmark pattern generator 214, by the landmark pattern generator 258, or by a combination thereof. The content displayed (block 262) on the display 260 and / or display interface 261 by the display interface device 250 may include a displayed interface generated by the touchscreen interface generator 254. The displayed interface generated by touchscreen interface generator 254 and displayed on display 260 of display interface device 250 may, in some cases, remain visible in the output image output by XR device 202. For example, XR device 202 may overlay a virtual interface over and / or replace certain content other than the displayed interface (e.g., a landmark pattern) displayed on display 260 and / or display interface 261 of display interface device 250 without overlaying a virtual interface over and / or replacing the displayed interface.An example of such a displayed interface includes the displayed interface 710 of Figures 7A-7B, which is still visible to a user 505 wearing an XR device 520, as shown in Figure 7B. The display interface can include one or more interactive touch-based interface elements, such as a button, slider, scroll bar, radio button, check box, knob, wheel, text field, touch-based keypad, touch-based keyboard, touch-based drawing area, or combinations thereof.

[0064] Display interface device 250 may receive 264 display interface input from display 260 and / or display interface 261. Receiving 254 display interface input may be performed by a display interface application 252 of display interface device 250, a display interface 261, a display 260, a display interface interpreter 256, a display interface controller of display interface 261, a display controller of display 260, by a processor of display interface device 250, or by a combination thereof. In some examples, display 260 and / or display interface 261 may include a touchscreen display having a display layer and a touch-sensitive layer, which may be referred to as a touch-sensing layer, a touch-sensitive surface, or a touch-sensing surface. The touch-sensitive layer may measure the display interface input capacitively, resistively, or a combination thereof. In embodiments where display 260 and / or display interface 261 are touch screens, they may be capacitive touch screens, resistive touch screens, or a combination thereof. Display interface application 252 may include a display interface input interpreter 256 that may interpret signals from a touch-sensitive layer of display 260 and / or display interface 261 into a display interface input identifier. The display interface input identifier may include, for example, coordinates that identify one or more portions of display 260 and / or display interface 261 that received the touch input or another type of display interface input.In some cases, the display interface input may be a tap or press at a single location, in which case the corresponding display interface input may identify a single set of coordinates or touch area that represents the portion of the display 260 and / or display interface 261 that received the tap or press. The touch area may represent, for example, the surface area of ​​a point of a finger or pointer that contacts the display 260 and / or display interface 261 during the display interface input. The touch area may in some cases be identified in the display interface input identifier by a set of coordinates and a radius, where the touch area includes an area having a radius around a point defined by the set of coordinates. In some cases, the display interface input may include a swipe gesture, a slide gesture, in which case the corresponding display interface input identifier(s) may include multiple sets of coordinates and / or areas that can be used to identify one or more paths of the swipe gesture and / or slide gesture as one or more portions of the display 260 and / or display interface 261 that received the display interface input(s).In some cases, the display interface 261 may include different types of display interfaces 261, such as a mouse to control a cursor or other display element on the display, a trackpad to control a cursor or other display element on the display, a keypad to control text input and / or cursor or other display elements on the display, a controller to control text input and / or cursor or other display elements on the display, a remote control to control text input and / or cursor or other display elements on the display, a hand tracker to hand track interaction with the display using a camera (e.g., of the XR device and / or display interface device), a hand tracker to hand track interaction with the display using an active depth sensor (e.g., RADAR, LIDAR, SONAR, SODAR, structured light, time of flight) (e.g., of the XR device and / or display interface device), a hand tracker to hand track interaction with the display using an ultrasonic sensor (e.g., of the XR device and / or display interface device), another input device 1245 to provide an input interface associated with the display, or a combination thereof. Display interface input may also include clicking, double-clicking, clicking and dragging, circling, pointing, hovering over, gesturing around or over, or combinations thereof.

[0065] In some instances, the display interface input interpreter 256 can compare the location of the display interface input to known locations where display content is displayed on the display 260 and / or display interface 261. For example, the display interface input interpreter 256 can compare the location of the display interface input to an interface element of the displayed interface, such as a button. If the location of the display interface input matches (e.g., overlaps) with a location of an interface element of the displayed interface, the display interface input interpreter 256 can identify (e.g., in the display interface input identifier) ​​that the display interface input activated or otherwise interacted with the interface element of the displayed interface. In some examples, the location of the display interface input matches (e.g., overlaps) with a particular portion of a landmark pattern displayed on the display 260 and / or display interface 261, and the display interface input interpreter 256 can identify (e.g., in the display interface input identifier) ​​with which the location of the display interface input matches (e.g., overlaps). In some examples, the same process can be performed at least in part by the XR device 202, for example, when the display interface interaction is based on hand tracking. An example of a display interface is display interface 710 of Figures 7A-7B.

[0066] The display interface device 250 can transmit display interface input data from the display interface device 250 to the XR device 202 using the wireless transceiver(s) 270 (block 272). The transmitting of the display interface input data (block 272) can be performed by the display interface application 252, the display input interpreter 258, the wireless transceiver(s) 270 of the display interface device 250, by a processor of the display interface device 250, or by a combination thereof. The XR device 202 can receive the display interface input data from the display interface device 250 using the wireless transceiver(s) 230 (block 228). The receiving of the display interface input data (block 228) can be performed by the XR application 204 of the XR device 202, the wireless transceiver(s) 230, by a processor of the XR device 202, or a combination thereof. The display interface input data may include one or more display interface input identifiers that identify one or more display interface inputs received at the display 260 and / or display interface 261 (block 264) and / or interpreted by the display interface input interpreter 256. In some cases, the display interface input identifiers may identify that the display interface input has interacted with a displayed interface, thereby causing the XR device 202 to perform actions such as modifying the virtual interface, modifying the virtual content, and / or modifying audio output to the user.

[0067] The XR application 204 can include a virtual interface generator 206 that can generate at least a portion of a virtual interface, such as virtual interfaces 535, 635, 735, and / or 835. The virtual interface can include interactive interface elements, such as buttons, sliders, scroll bars, radio buttons, checkboxes, knobs, wheels, text fields, touch-based keypads, touch-based keyboards, touch-based drawing areas, or combinations thereof. In some embodiments, the virtual interface generator 206 can resize, rotate, skew, distort, or otherwise adjust characteristics of the virtual interface based on the landmark pattern and / or the pose and / or size of the display 260 and / or the display interface device 250. In some embodiments, at least a portion of the virtual interface can be generated at the display interface device 250 (e.g., by the display interface application 252), transmitted (at block 272) from the display interface device 250 to the XR device 202, and / or received (at block 228) from the display interface device 250 at the XR device 202. Sending the virtual interface (block 272) may be performed by the display interface application 252 of the display interface device 250, the display interface generator 254, the virtual interface generator of the display interface device 250, the wireless transceiver(s) 270, by a processor of the display interface device 250, or by a combination thereof. Receiving the virtual interface (block 228) may be performed by the XR application 204 of the XR device 202, the wireless transceiver(s) 230, by a processor of the XR device 202, or a combination thereof.The XR application 204 may include an output image compositor 212 that generates an output image by combining (compositing) one or more images captured by the image sensor(s) 220 of the XR device 202 with virtual content, such as a virtual interface, other virtual content (e.g., other visual content generated using the virtual content generator 210), or a combination thereof. To generate the output image, the output image compositor 212 may overlay a virtual interface over a landmark pattern depicted in one or more images captured by the image sensor(s) 220 of the XR device 202. To generate the output image, the output image compositor 212 may replace a landmark pattern depicted in one or more images captured by the image sensor(s) 220 of the XR device 202 with a virtual interface.

[0068] The output image compositor 212 can position, move, resize, resample, rescale, upsample, upscale, downsample, downscale, magnify, reduce, rotate, skew, warp (e.g., perspective warping), and / or distort (e.g., perspective distortion) the virtual interface so that the pose and / or size of the virtual interface in the output image simulates the pose and / or size of the landmark pattern of the display interface device 250 and / or the display 260 and / or the display interface 261 (as determined using the display interface device tracking 224). The output image compositor 212 can warp, distort, and / or skew the virtual interface (e.g., using perspective warping and / or perspective distortion) to simulate a rotation of the virtual interface about an axis other than an axis perpendicular to the image captured by the image sensor(s) 220. For example, if the display 260 and / or the display interface device 250 and / or the landmark pattern are large in the image data, the virtual interface may be large. If the display 260 and / or display interface device 250 and / or landmark patterns are small in the image data, the virtual interface may be small. If the display 260 and / or display interface device 250 and / or landmark patterns are tilted and / or rotated and / or skewed according to a particular orientation or pose in the image data, the virtual interface may be tilted and / or rotated and / or skewed according to that orientation or pose.

[0069] In an example embodiment, the virtual interface may include a video playback control interface, which may include virtual buttons for pause, play, fast forward, rewind, skip, forward, and other video playback control functions. In some embodiments, the video playback control interface may include one or more sliders, such as a video scrubbing slider and / or a volume slider. In some embodiments, the virtual interface may include images, videos, and other media.

[0070] The virtual content generator 210 can generate virtual content other than a virtual interface. The virtual content can include, for example, images and / or videos presented by a virtual interface (e.g., representing an image viewer and / or a video viewer), one or more three-dimensional models, video game content, metadata of media played using the virtual interface, or combinations thereof. Examples of virtual content generated by the virtual content generator 210 can include the virtual content 720 of FIGS. 7A-7B, the virtual content 840 of FIGS. 8A-8B, certain aspects of the music player virtual interface 535 of FIG. 5B (e.g., album art, artist name, and / or song title), certain aspects of the music player virtual interface 535 of FIG. 5B (e.g., album art, artist name, and / or song title), the image viewer virtual interface 635 of FIG. 6B (e.g., an image), or combinations thereof. The output image compositor 212 can add the virtual content to the virtual interface or anywhere within the field of view of the image sensor(s) 220 of the XR device 202.

[0071] The XR application 204 can also include an occlusion detector 208 that can detect occlusion of the landmark pattern and / or another area of ​​the display, such as by a user's hand (e.g., occlusion 610 in FIGS. 6A-6B) and / or a user's finger (e.g., occlusion 620 in FIGS. 6A-6B). In some examples, the occlusion detector 208 can track the hand as a possible occlusion using hand tracking 222. In some examples, the occlusion detector 208 can track the landmark pattern depicted in an image captured by the image sensor(s) 220 of the XR device 202 using touch screen device tracking 224, and identify when a portion of the landmark pattern is occluded by comparing the landmark pattern depicted in an image captured by the image sensor(s) 220 of the XR device 202 to a reference landmark pattern stored in the XR device 202 (that is, not occluded). In some embodiments, the output image composer 212 can track the occlusion and ensure that the occlusion is visible in the output image, for example, by cropping out a portion of the virtual interface and / or virtual content at the location of the depicted occlusion, by overlaying a depiction of the occlusion over the virtual portion of the virtual interface and / or virtual content at the location of the depicted occlusion, or a combination thereof. For example, in FIG. 6B , occlusions 610 and 620 are still visible to user 505 through XR device 520. Similarly, in FIG. 8B , occlusion 810, which partially occludes virtual interface 835, is still visible to user 505 through XR device 520.

[0072] The output image compositor 212 can generate an output image by compositing a virtual interface (e.g., generated by the virtual interface generator 206) with an image captured by the image sensor(s) 220 of the XR device 202, by compositing a virtual content (e.g., generated by the virtual content generator 210) with an image captured by the image sensor(s) 220 of the XR device 202, by compositing an occlusion (e.g., detected by the occlusion detector 208) with the virtual interface and / or virtual content, or by a combination thereof. The XR device 202 can output the output image generated by the output image compositor 212 to one or more displays 234, and thus display the output image on one or more displays 234 (block 232). Displaying the output image (block 232) can be performed by the XR application 204 of the XR device 202, the display 234, the output image compositor 212, a display controller associated with the display 234, by a processor of the XR device 202, or a combination thereof. In an example embodiment, the XR device 202 can generate an output image based on an image of the scene and based on the pose and / or size of the display 260 of the display interface device 250 in the image data captured by the image sensor 220 of the XR device 202. For example, the output image may be a modified version of the captured image (captured by the image sensor 220 of the XR device 202) in which the XR device 202 overlays a virtual interface over the visual representation of the landmark pattern, and in some cases replaces the visual representation of the landmark pattern with the virtual interface. The XR device 202 can position, orient, resize, rotate, skew, warp, and / or distort the virtual interface to have a virtual, simulated pose based on the recognized pose of the landmark pattern and / or the recognized pose of the display 260 of the display interface device 250.The XR device 202 can position, orient, resize, rotate, skew, warp, and / or distort the virtual interface to have a virtual and / or simulated size based on the perceived size of the landmark pattern and / or the perceived size of the display 260 of the display interface device 250. The XR device 202 can display the output image to the user via one or more displays 234 of the XR device 250. Thus, to a user of the XR device 202, the virtual interface can appear to be displayed on the display 260 of the display interface device 250 over and / or in place of the landmark pattern.

[0073] In an example embodiment, the hand of a user of the XR device 202 may be occluding a portion of the landmark pattern from the perspective of the image sensor(s) 220. Because the landmark pattern is a known pattern (e.g., stored in the XR device 202 as a reference landmark pattern), the XR device 202 can determine which portions of the landmark pattern are occluded and which portions of the landmark pattern are not occluded. Based on this, the output image composer 212 of the XR device can generate an output image such that the virtual interface is occluded in the same way that the landmark pattern is occluded in the image(s) captured by the image sensor 220.

[0074] In some examples, the XR device 202 can include one or more other sensors other than the inertial sensor 216 and image sensor 220 shown in Figure 2. For example, the XR device 202 can include one or more Light Detection and Ranging (LIDAR) sensors, Radio Detection and Ranging (RADAR) sensors, Sound Detection and Ranging (SODAR) sensors, Sound Navigation and Ranging (SONAR) sensors, one or more Time of Flight (ToF) sensors, one or more structured light sensors, one or more microphones, one or more other sensors described herein, or combinations thereof. In some examples, sensor data from these sensors can also be used by the XR device 202 for inertial tracking 218, hand tracking 222, and / or touchscreen device tracking 224.

[0075] In some examples, the display interface device 250 includes one or more sensors 276. The one or more sensors 276 may include, for example, one or more image sensors (e.g., cameras), accelerometers, gyroscopes, inertial measurement units (IMUs), light detection and ranging (LIDAR) sensors, radio detection and ranging (RADAR) sensors, sound detection and ranging (SODAR) sensors, sound navigation and ranging (SONAR) sensors, one or more time-of-flight (ToF) sensors, one or more structured light sensors, one or more microphones, one or more other sensors described herein, or combinations thereof. The display interface device 250 can perform attitude tracking 278 of the attitude of the display interface device 250 using sensor data from the one or more sensors 276, for example, as described above with respect to the inertial tracking 218 performed by the XR device 202. In some examples, the display interface device 250 can also perform pose tracking 278 of the posture of the XR device 202 using sensor data from the one or more sensors 276, for example, based on a representation of the XR device 202 detected in the sensor data from the one or more sensors 276. Performing pose tracking 278 of the display interface device 250 and / or the XR device 202 can be performed by a display interface application 252 of the display interface device 250, by the sensor 278, by a posture tracking engine of the display interface device 250, by a processor of the display interface device 250, or by a combination thereof.

[0076] In some embodiments, the display interface device 250 can use one or more wireless transceivers 270 to transmit posture tracking data (e.g., identifying the posture and / or size of the display interface device 250 (and / or its display 260) and / or the posture and / or size of the XR device 202) from the display interface device 250 to the XR device 202 (block 272). Transmitting posture tracking data (block 272) can be performed by the display interface application 252 of the display interface device 250, the wireless transceiver(s) 270, by a processor of the display interface device 250, or by a combination thereof. In some embodiments, the XR device 202 can receive posture tracking data (e.g., identifying the posture and / or size of the display interface device 250 (and / or its display 260) and / or the posture and / or size of the XR device 202) from the display interface device 250 (block 228) using one or more wireless transceivers 230. Receiving posture tracking data (block 228) may be performed by the XR application 204 of the XR device 202, the wireless transceiver(s) 230, by a processor of the XR device 202, or by a combination thereof. In some embodiments, posture tracking data received by the XR device 202 from the display interface device 250 (e.g., identifying the posture of the display interface device 250 and / or the posture of the XR device 202) (at block 228) may be used by the XR device 202 for inertial tracking 218, hand tracking 222, and / or touchscreen device tracking 224.

[0077] As described above, the display interface device 250 can detect and interpret display interface input via the display 260 and / or the display interface 261 (e.g., touch input via the touchscreen display interface 261 of the display 260). The display interface device 250 can identify the display interface input and transmit a display interface input identifier to the XR device 202 (block 272), e.g., in the form of one or more coordinates on a coordinate grid of the display 260 and / or the display interface 261 that was touched, clicked, swiped, hovered over, gestured over, or otherwise received the display interface input to the display interface 261. Transmitting the display interface input (block 272) can be performed by the display interface application 252 of the display interface device 250, the wireless transceiver(s) 270, by a processor of the display interface device 250, or by a combination thereof. The XR device 202 can receive the display interface input identifier (block 228). Receiving the display interface input (block 228) may be performed by the XR application 204 of the XR device 202, by the wireless transceiver(s) 230, by a processor of the XR device 202, or by a combination thereof. The XR device 202 may adjust output content output by the XR device 202 to the user (e.g., displayed on the display 234) based on the display interface input. If the position of the display interface input aligns (e.g., overlaps and / or matches) with the position of an interface element of the virtual interface, the XR device 202 may identify that the display interface input has activated or otherwise interacted with an interface element of the virtual interface.In some examples, the display interface input identifier can identify that a position of the display interface input aligns with (e.g., overlaps and / or matches) a particular portion of the landmark pattern, and the XR device 202 can identify that a position of the display interface input aligns with (e.g., overlaps and / or matches) a corresponding portion of a virtual interface overlaid on the particular portion of the landmark pattern. The output content can include an output image, a future output image, one or more audio clips (e.g., music, sound effects), vibration feedback (e.g., haptic feedback, rumble feedback), or a combination thereof that the XR device 202 outputs to the user. The changes to the output content can include changes to the output image, changes to future output images, changes to audio clips that the XR device 202 outputs to the user, changes to the vibration feedback (e.g., to the pattern of vibrations), or a combination thereof.

[0078] In an example embodiment, if the XR device 202 determines, based on the display interface input, that the user has pressed a pause button on the virtual interface, the XR device can pause video playback at the current video frame. If the XR device 202 determines, based on the display interface input, that the user has pressed a fast-forward button on the virtual interface, the XR device 202 can begin fast-forward playback beginning with the next output frame. If the XR device 202 determines, based on the display interface input, that the user has moved a volume slider, the XR device 202 can adjust the volume of the audio that the XR device 202 is outputting to the user beginning with the next segment of audio.

[0079] In some embodiments, the display interface device 250 is configured to shift between displaying each of a set of different landmark patterns on the display interface device 250 at a predetermined frequency over time 905, as shown in FIG. 9. By detecting which landmark pattern of the set of different landmark patterns is depicted in the image and aligning it with the time that the display interface device 250 displayed that landmark pattern on the display 260, the XR device 202 and the display interface device 250 can synchronize timing. Based on the landmark pattern that the XR device 202 is overlaying the virtual interface, the XR device 202 can obtain a more accurate indication of when a particular display interface input was received at the display interface device 250 relative to what the XR device 202 is displaying as a virtual interface, and can increase the accuracy of the display interface input for use by the XR device.

[0080] In some cases, the XR device 202 may also use hand tracking 222 and / or finger tracking for input instead of or in addition to receiving display interface input from the display interface 261 of the display interface device 250. For example, the XR device 202 may use touch input detected by the touchscreen-based display interface 261 of the display interface device 250 as a first type of input and hand tracking-based detection of a user pointing at an element of a virtual interface as a second type of input, performed, for example, similar to hand tracking 218. Hand tracking input may be a fallback option, for example, when the display interface 261 of the display interface device 250 is faulty and / or cannot reliably detect display interface input (e.g., the touchscreen-based display interface 261 is wet). Hand tracking input may be a different type of input, for example, allowing a user to point and / or hover over an interface element of the virtual interface and / or display interface without actually touching the interface element, and allowing the pointing and / or hovering to be detected and interpreted as an interaction with the interface element. Hand tracking can include finger tracking. Hand tracking can be based on data from the inertial sensor 216, the image sensor 220, the sensor 276 of the display interface device 250, or a combination thereof. In some examples, hand tracking can be based on data from one or more cameras and / or one or more distance sensors (e.g., RADAR sensors, LIDAR sensors, SONAR sensors, SODAR sensors, time-of-flight sensors, structured light sensors) of the XR device 202 and / or the display interface device 250.

[0081] FIG. 3A is a perspective view 300 illustrating a head mounted display (HMD) 310 used as an extended reality (XR) device 202. The HMD 310 may be, for example, an augmented reality (AR) headset, a virtual reality (VR) headset, a mixed reality (MR) headset, an extended reality (XR) headset, or some combination thereof. The HMD 310 may be an embodiment of the XR device 202. The HMD 310 includes a first camera 330A and a second camera 330B along the front of the HMD 310. The first camera 330A and the second camera 330B may include the image sensor 220 of the XR device 202. In some embodiments, the HMD 310 may have only a single camera with a single image sensor 220. In some embodiments, the HMD 310 may include one or more additional cameras, which may also include the image sensor 220 of the XR device 202, in addition to the first camera 330A and the second camera 330B. In some embodiments, the HMD 310 may include one or more additional sensors in addition to the first camera 330A and the second camera 330B, which may be embodiments of the inertial sensor 216 of the XR device 202 and / or other sensors of the XR device 202 described herein.

[0082] The HMD 310 may include one or more displays 340 visible to the user 320 wearing the HMD 310 on the user's 320 head. The one or more displays 340 of the HMD 310 may be an embodiment of the display 234 of the XR device 202. In some embodiments, the HMD 310 may include one display 340 and two viewfinders. The two viewfinders may include a left viewfinder for the left eye of the user 320 and a right viewfinder for the right eye of the user 320. The left viewfinder may be oriented so that the left eye of the user 320 sees the left side of the display. The right viewfinder may be oriented so that the left eye of the user 320 sees the right side of the display. In some embodiments, the HMD 310 may include two displays 340 including a left display that displays content to the left eye of the user 320 and a right display that displays content to the right eye of the user 320.

[0083] The HMD 310 may include one or more earpieces 335 that may function as speakers and / or headphones to output audio to one or more ears of a user of the HMD 310. Although one earpiece 335 is shown in Figures 3A and 3B, it should be understood that the HMD 310 may include two earpieces, one earpiece for each ear (left and right) of the user. In some embodiments, the HMD 310 may also include one or more microphones (not shown). In some embodiments, audio output by the HMD 310 to the user through the one or more earpieces 335 may include or be based on audio recorded using one or more microphones.

[0084] FIG. 3B is a perspective view 350 showing the head mounted display (HMD) of FIG. 3A being worn by a user 320. The user 320 wears the HMD 310 on the user's 320 head over the user's 320 eyes. The HMD 310 can capture images using a first camera 330A and a second camera 330B. In some embodiments, the HMD 310 displays one or more output images to the user's 320 eyes. The output images may be examples of the display output images 232. The output images can be based on the images captured by the first camera 330A and the second camera 330B. The output images can provide a stereoscopic view of the environment, possibly with information overlaid and / or other modifications. For example, the HMD 310 can display a first display image based on the image captured by the first camera 330A to the right eye of the user 320. The HMD 310 can display a second display image based on the image captured by the second camera 330B to the left eye of the user 320. For example, the HMD 310 can provide overlay information in the display image overlaid on the images captured by the first camera 330A and the second camera 330B. An earpiece 335 of the HMD 310 is shown in the ear of the user 320. The HMD 310 may output audio to the user 320 through the earpiece 335 and / or through another earpiece (not shown) of the HMD 310 in the other ear (not shown) of the user 320.

[0085] 4A is a perspective view 400 showing the front of a mobile handset 410 that includes a forward-facing camera and can be used as an extended reality (XR) device 202 or a display interface device 250. The mobile handset 410 may be an example of an XR device 202. The mobile handset 410 may be an example of a display interface device 250. The mobile handset 410 may be, for example, a mobile phone, a satellite phone, a portable game console, a music player, a health tracking device, a wearable device, a wireless communication device, a laptop, a mobile device, any other type of computing device or computing system described herein, or a combination thereof.

[0086] The front face 420 of the mobile handset 410 includes a display 440. The front face 420 of the mobile handset 410 includes a first camera 430A and a second camera 430B. The first camera 430A and the second camera 430B may be examples of the image sensor 220 of the XR device 202. The first camera 430A and the second camera 430B may be examples of the sensor 276 of the display interface device 250. The first camera 430A and the second camera 430B are shown within a bezel around the display 440 on the front face 420 of the mobile handset 410. In some examples, the first camera 430A and the second camera 430B may be positioned in a notch or cutout cut out of the display 440 on the front face 420 of the mobile handset 410. In some embodiments, the first camera 430A and the second camera 430B may be under-display cameras disposed between the display 440 and the remainder of the mobile handset 410, such that light passes through a portion of the display 440 before reaching the first camera 430A and the second camera 430B. The first camera 430A and the second camera 430B in the perspective view 400 are forward-facing cameras. The first camera 430A and the second camera 430B face in a direction perpendicular to the plane of the front face 420 of the mobile handset 410. The first camera 430A and the second camera 430B may be two of the one or more cameras of the mobile handset 410. The first camera 430A and the second camera 430B may be the sensor 405A and the sensor 405B, respectively. In some embodiments, the front face 420 of the mobile handset 410 may have only a single camera. In some embodiments, the mobile handset 410 may include one or more additional cameras in addition to the first camera 430A and the second camera 430B. The one or more additional cameras may also be embodiments of the image sensor 220 of the XR device 202. The one or more additional cameras may also be embodiments of the sensor 276 of the display interface device 250.In some embodiments, the mobile handset 410 may include one or more additional sensors in addition to the first camera 430A and the second camera 430B. The one or more additional sensors may also be an embodiment of the inertial sensor 216 of the XR device 202 and / or other sensors of the XR device 202. The one or more additional sensors may also be an embodiment of the sensor 276 of the display interface device 250. The front surface 420 of the mobile handset 410 also includes a display 440. In some cases, the front surface 420 of the mobile handset 410 includes more than one display 440. The one or more displays 440 of the front surface 420 of the mobile handset 410 may be an embodiment of the display(s) 234 of the XR device 202. The one or more displays 440 of the front surface 420 of the mobile handset 410 may be an embodiment of the display(s) 260 and / or the display interface(s) 261 of the display interface device 250. For example, the one or more displays 440 may include one or more touchscreen displays, the touchscreen interface of which represents the display interface 261 of the touchscreen display.

[0087] The mobile handset 410 may include one or more speakers 435A and / or other audio output devices (e.g., earphones or headphones or connectors thereto) that can output audio to one or more ears of a user of the mobile handset 410. While one speaker 435A is shown in FIG. 4A, it should be understood that the mobile handset 410 can include more than one speaker and / or other audio devices. In some embodiments, the mobile handset 410 also includes one or more microphones (not shown). In some embodiments, audio output by the mobile handset 410 through the one or more speakers 435A and / or other audio output devices to a user can include or be based on audio recorded using the one or more microphones.

[0088] 4B is a perspective view 450 showing a back surface 460 of a mobile handset that includes a read-facing camera and can be used as the extended reality (XR) device 202 or the display interface device 250 of FIG. 2. The mobile handset 410 includes a third camera 430C and a fourth camera 430D on the back surface 460 of the mobile handset 410. The third camera 430C and the fourth camera 430D of the perspective view 450 are rear-facing. The third camera 430C and the fourth camera 430D may be examples of the image sensor 220 of the XR device 202 of FIG. 2. The third camera 430C and the fourth camera 430D may be examples of the sensor 276 of the display interface device 250 of FIG. 2. The third camera 430C and the fourth camera 430D are oriented perpendicular to the plane of the back surface 460 of the mobile handset 410. Although the back 460 of the mobile handset 410 does not have a display 440 as shown in the perspective view 450, in some embodiments, the back 460 of the mobile handset 410 can include one or more rear displays. The one or more rear displays of the back 460 of the mobile handset 410 may be examples of the display(s) 234 of the XR device 202. The one or more rear displays of the back 460 of the mobile handset 410 may be examples of the display interface display(s) 260 of the display interface device 250. If the back 460 of the mobile handset 410 includes one or more rear displays, any arrangement layout of the third camera 430C and the fourth camera 430D relative to the one or more rear displays can be used, as described with respect to the first camera 430A and the second camera 430B relative to the display 440 of the front 420 of the mobile handset 410.

[0089] The third camera 430C and the fourth camera 430D may be two of the one or more cameras of the mobile handset 410. In some embodiments, the back surface 460 of the mobile handset 410 may have only a single camera. In some embodiments, the mobile handset 410 may include one or more additional cameras in addition to the first camera 430A, the second camera 430B, the third camera 430C, and the fourth camera 430D. The one or more additional cameras may also be an embodiment of the image sensor 220 of the XR device 202. The one or more additional cameras may also be an embodiment of the sensor 276 of the display interface device 250. In some embodiments, the mobile handset 410 may include one or more additional sensors in addition to the first camera 430A, the second camera 430B, the third camera 430C, and the fourth camera 430D. The one or more additional sensors may also be an embodiment of the inertial sensor 216 of the XR device 202 and / or other sensors of the XR device 202. The one or more additional sensors may also be embodiments of sensors 276 of display interface device 250 .

[0090] The mobile handset 410 may include one or more speakers 435B and / or other audio output devices (e.g., earphones or headphones or connectors thereto) that can output audio to one or more ears of a user of the mobile handset 410. While one speaker 435B is shown in FIG. 4B, it should be understood that the mobile handset 410 can include more than one speaker and / or other audio devices. In some embodiments, the mobile handset 410 also includes one or more microphones (not shown). In some embodiments, audio output by the mobile handset 410 through the one or more speakers 435B and / or other audio output devices to a user can include or be based on audio recorded using the one or more microphones.

[0091] 5A is a perspective view 500 showing a user 505 wearing an extended reality (XR) device 520 and holding a display interface device 510 displaying a landmark pattern 530, according to some embodiments. The XR device 520 worn by the user 505 may be the HMD 310 of FIGS. 3A-3B, which may include one or more displays 525 of FIGS. 5A-9, which may be an embodiment of one or more displays 234 of FIG. 2, one or more displays 340 of FIGS. 3A-3B, or both. It should be understood that the XR device 520 is shown as the HMD 310 of FIGS. 3A-3B for illustrative purposes, and the XR device 520 may alternatively be the mobile handset 410 of FIGS. 4A-4B.

[0092] Display interface device 510 includes a touchscreen display 515 that is shown to display a landmark pattern 530 depicted as a QR code. Display interface device 510 of Figures 5A-5B, 6A-6B, 7A-7B, 8A-8B, and 9 is an example of display interface device 250. Touchscreen display 515 of Figures 5A-5B, 6A-6B, 7A-7B, 8A-8B, and 9 is an example of both display 260 of display interface device 250 and display interface 261 of display interface device 250. For example, the display layer(s) of touchscreen display 515 is an example of display 260, and the touch-sensitive layer(s) and / or touch-sensing layers of touchscreen display 515 are an example of display interface 261. In this example, touch input to the touchscreen display 515 of the display interface device 510 is an example of a display interface input to the display interface 261 of the display interface device 250. The landmark pattern 530 is what is actually displayed on the touchscreen display 515 of the display interface device 510 of FIG. 5A. Thus, to anyone in the environment other than the user 505 (viewing the environment through the XR device 520), the display interface display 515 of the display interface device 510 appears to display the landmark pattern 530. Thus, while no one in the environment other than the user 505 (viewing the environment through the XR device 520) can see the virtual interface 535, the user 505 (viewing the environment through the XR device 520) can see the virtual interface 535 through the XR device 520, as shown in FIG. 5B.The pose of the XR device 520 from the perspective of the XR device 520 (and any input images captured by the XR device 520), and therefore the pose of the landmark pattern 530 displayed on the touchscreen display 515 of the XR device 520 from the perspective of the XR device 520 (and any input images captured by the XR device 520), appears to be slightly rotated clockwise.

[0093] 5B is a perspective view 550 showing a field of view (FOV) 560 of a user 505 wearing the extended reality (XR) device 520 of FIG. 5A and viewing an environment through the XR device 520 of FIG. 5A, with a virtual interface 535 overlaid on top of the landmark pattern 530, according to some embodiments. The virtual interface 535 is a music player virtual interface and includes album art, artist name, and song title. The virtual interface 535 also includes interactive interface elements including a scrubbing slider, a pause / play button, a rewind button, a fast forward button, a volume up button, a volume down button, and a mute button. Interaction of the display interface device 510 with the touchscreen display 515 can cause the display interface device 510 to transmit a touch input identifier to the XR device 520 that identifies which portion(s) of the touchscreen display 515 was touched during the touch input and / or how the touch input was touched during the touch input (e.g., a tap, a double tap, a triple tap, a swipe, a long press, a gesture, a multi-finger variant of any of the touch inputs enumerated above, or a combination thereof), enabling the XR device 520 to identify whether any of the interactive interface elements of the virtual interface 535 of the XR device 520 was touched during the touch input. Because the orientation of the XR device 520 from the FOV 560, and therefore the orientation of the landmark pattern 530 displayed on the touchscreen display 515 of the XR device 520 from the FOV 560, appears to be rotated slightly clockwise, the virtual interface 535 also appears to be rotated slightly clockwise to align with the orientation of the XR device 520 and / or the touchscreen display 515 and / or the landmark pattern 530.

[0094] 6A is a perspective view 600 showing a user 505 wearing an extended reality (XR) device 520 and holding a display interface device 510 displaying two landmark patterns 630 that are partially occluded by occlusions 610 and 620 including the hands and fingers of the user 505, according to some embodiments. The display interface device 510 displays the two landmark patterns 630, each shown as a QR code, on its touchscreen display 515. Some areas of the touchscreen display 515, including areas of the two landmark patterns 630, are partially occluded by a first occlusion 610 (the left hand of the user 505 and / or the fingers of the left hand of the user 505) and a second occlusion 620 (the fingers of the right hand of the user 505). The XR device 520 can detect that the first occlusion 610 and the second occlusion 620 are partially occluding an area of ​​the touchscreen display 515 (e.g., including the area of ​​the two landmark patterns 630) at least in part by comparing a representation of the two landmark patterns 630 (and / or the touchscreen display 515 in general) in an image captured by an image sensor of the XR device 520 to a reference landmark pattern (and / or a reference copy of the content displayed on the touchscreen display 515) corresponding to the two landmark patterns 630 (and / or other content displayed on the touchscreen display 515) stored by the XR device 520. The touchscreen display 515 of the display interface device 510 displays the landmark pattern 630 to anyone in the environment other than the user 505 (who is viewing the environment through the XR device 520).

[0095] FIG. 6B is a perspective view 650 illustrating a field of view (FOV) 660 of a user 505 wearing the extended reality (XR) device 520 of FIG. 6A and viewing an environment through the XR device 520 of FIG. 6A, where a virtual interface 635 is overlaid on top of two landmark patterns 630 and occlusions 610 and 620 are visible on top of the virtual interface 635, according to some embodiments. The virtual interface 635 is an image viewer virtual interface and includes a row of images that can be interactively scrolled through using touch input. In some embodiments, tapping on an image can increase the size of the image and / or display metadata of the image. Interaction of the display interface device 510 with the touchscreen display 515 can cause the display interface device 510 to transmit a touch input identifier to the XR device 520 that identifies which portion(s) of the touchscreen display 515 were touched (and how) during the touch input, enabling the XR device 520 to identify the portion of the virtual interface 635 that was touched during the touch input. In one example implementation, the touch input identifier to the XR device 520 may identify a swipe or slide touch gesture that may cause the XR device 520 to scroll through images in the virtual interface 635. In another example implementation, the touch input identifier to the XR device 520 may identify a tap or press touch input on a particular image in the virtual interface 635, which may cause the XR device 520 to increase the size of the image and / or display metadata for the image.

[0096] The first occlusion 610 and the second occlusion 620 are still visible to the user 505 within the FOV 660 as occlusions of the virtual interface 635. The XR device 520, in some embodiments, can crop out portions of the virtual interface 635 at the locations of the first occlusion 610 and the second occlusion 620 before overlaying the virtual interface 635 over the landmark pattern 630. The XR device 520, in some embodiments, can overlay the virtual interface 635 over the landmark pattern 630 and then overlay image data corresponding to the first occlusion 610 and the second occlusion 620 over the virtual interface 635.

[0097] In some embodiments, the XR device 520 has knowledge of the expected landmark pattern 630. Any discrepancy between the expected landmark pattern 630 and the observed landmark pattern 630 in an image captured by the image sensor 220 of the XR device 520 can help the XR device 520 perform occlusion detection. This discrepancy calculation can be used alone for occlusion detection or as an input to one or more trained neural networks to identify occlusions. Occlusion detection can enable detailed and appropriate rendering of the virtual interface 635 and / or other virtual content from the XR device 520 with appropriate occlusions.

[0098] 7A is a perspective view 700 showing a user 505 wearing an extended reality (XR) device 520 and holding a display interface device 510 displaying a landmark pattern 530 and a displayed interface 710, according to some embodiments. The display interface device 510 displays the landmark pattern 730 (shown as a QR code) and the displayed interface 710 on its touchscreen display 515. The displayed interface 710 includes a play / pause button, a rewind button, and a fast forward button. The touchscreen display 515 of the display interface device 510 displays the landmark pattern 730 and the displayed interface 710 to anyone in the environment other than the user 505 (who is viewing the environment through the XR device 520).

[0099] FIG. 7B is a perspective view 750 showing a field of view (FOV) 760 of a user 505 wearing the extended reality (XR) device 520 of FIG. 5A and viewing an environment through the XR device 520 of FIG. 5A where a virtual interface 735 is overlaid on the landmark pattern 730 but the displayed interface 710 is still visible, according to some embodiments. The virtual interface 735 is a video player virtual interface and includes the video that is playing. The virtual interface 735 also includes interactive interface elements including a scrubbing slider. The displayed interface 710 is still visible to the user 505 because the XR device 520 does not overlay anything on top of or replace the displayed interface 710 with anything. The displayed interface 710 includes a play / pause button that can play the video or pause the video playback, a rewind button that can rewind the video playback, and a fast forward button that can fast forward the video playback.

[0100] Interaction of the display interface device 510 with the touchscreen display 515 can cause the display interface device 510 to send a touch input identifier to the XR device 520 that identifies which portion(s) of the touchscreen display 515 were touched during the touch input and how they were touched during the touch input, enabling the XR device 520 to identify whether any of the interactive interface elements of the virtual interface 735 of the XR device 520 (e.g., a scrubbing slider) were touched during the touch input. Interaction of the display interface device 510 with the touchscreen display 515 can also cause the display interface device 510 to detect any interaction with interface elements of the displayed interface, which the display interface device 510 can transmit to the XR device 520 as a touch input identifier that allows the XR device 520 to know, for example, whether the touch input interacted with a play / pause button (which causes the XR device 520 to play a video or pause the playback of a video), a rewind button (which causes the XR device 520 to rewind the playback of a video), and a fast forward button (which causes the XR device 520 to fast forward the playback of a video).

[0101] The video, when displayed to the user 505, is larger than the display interface device 510 and is not limited to the bounds of the touchscreen display 515, but can still be positioned and / or oriented based on the landmark pattern 730 of the display interface device 510 and / or the attitude of the touchscreen display 515. Additional virtual content 720, including the title of the video being played using the video player virtual interface 735 ("Speedy Track"), is also displayed within the FOV 760.

[0102] In some embodiments, a touch-based interface for the XR system 200 may thus include a mix of virtual content (e.g., the virtual interface 735 and / or other virtual content 720) rendered by the XR device 520 and the displayed interface 710 rendered by the display interface device 510. In some embodiments, this may conserve power and / or other computing resources for rendering some elements (e.g., the displayed interface 710) on the display interface device 510. For example, this may conserve power and / or other computing resources for complex 3D content rendered on a touchscreen device, since such content may be difficult to accurately modify to match the pose of the landmark pattern 730. On the other hand, this may conserve power and / or other computing resources for very clear and simple content rendered on a touchscreen device, since such content may be clearly visible and identifiable through the XR device 520, even with some degradation or image artifacts caused by recording the touchscreen display 515 using the image sensor(s) 220 of the XR device 520. Some use cases may involve making some content visible on the touchscreen device to other people in the environment (who may or may not be wearing the XR device itself), while some other parts of the content should only be visible to the device owner (who is wearing the XR device 520).

[0103] 8A is a perspective view 800 showing a user 505 wearing an extended reality (XR) device 520 and holding a display interface device 510 displaying two landmark patterns 830, according to some embodiments. The display interface device 510 displays on its touchscreen display 515 two landmark patterns 830, each depicted as a QR code. The two landmark patterns 830 are partially occluded by two occlusions 810 (fingers on the left and right hands of the user 505). The XR device 520 can detect that the occlusions 810 partially occlude the two landmark patterns 830 by comparing a representation of the two landmark patterns 830 in an image captured by an image sensor of the XR device 520 with reference landmark patterns corresponding to the two landmark patterns 830 stored by the XR device 520. The touchscreen display 515 of the display interface device 510 displays the landmark patterns 830 to anyone in the environment other than the user 505 (who is viewing the environment through the XR device 520).

[0104] 8B is a perspective view 850 showing a field of view (FOV) 860 of a user 505 wearing the extended reality (XR) device 520 of FIG. 8A and viewing an environment through the XR device 520 of FIG. 8A, with a virtual interface 835 overlaid on top of two landmark patterns 830, according to some embodiments. The virtual interface 835 is a video game controller virtual interface and includes interactive interface elements including four buttons and a directional pad (D-pad). Virtual content 840 including video game content is also displayed within the FOV 860. The virtual content 840 (video game content) shows a spaceship (representing a player character) flying through space, firing a laser gun at four flying saucers (representing enemies), and hitting one of the flying saucers with the laser gun. In some embodiments, pressing interactive interface elements of the video game controller virtual interface 835 can affect the virtual content 840 (video game content), for example, by controlling the flight direction of the spaceship (e.g., based on a direction input on a D-pad) and by controlling the firing of a laser gun (e.g., based on at least one of four buttons). The user 505 is shown pressing one of the four buttons in the video game controller virtual interface 835, which can, for example, cause the XR device 520 to display a spaceship firing a laser gun in the virtual content 840 (video game content).

[0105] The occlusion 810 remains visible to the user 505 in the FOV 860 as an occlusion of the virtual interface 835. The XR device 520, in some embodiments, can crop out a portion of the virtual interface 835 at the location of the occlusion 810 before overlaying the virtual interface 835 over the landmark pattern 830. The XR device 520, in some embodiments, can overlay the virtual interface 835 over the landmark pattern 830 and then overlay image data corresponding to the occlusion 810 over the virtual interface 835.

[0106] The display interface device 510 is shown in FIG. 8B as providing vibration feedback 845, with rounded lines on the corners of the display interface device 510 indicating that the display interface device 510 is vibrating. The vibration feedback 845 may include haptic feedback that may provide a small vibration when the user 505 touches a button, D-pad, or another interactive interface element of the virtual interface 835, for example. The haptic feedback may simulate the tactile sensation of a button press and may provide the user with confirmation that the user 505 has actually touched the interactive interface element. The vibration feedback 845 may include force feedback or rumble feedback that may cause the display interface device 510 to vibrate in response to an action or condition occurring within the virtual content 840 (within a video game). Actions or conditions may include, for example, a spaceship (player character) firing a laser gun, a spaceship (player character) hitting a flying saucer (enemy) laser gun, a spaceship (player character) crashing into a flying saucer (enemy), a spaceship (player character) exploding, a flying saucer (enemy) hitting a spaceship (player character) laser gun, a flying saucer (enemy) exploding, or combinations thereof.

[0107] 9 is a conceptual diagram 900 illustrating a display interface device 510 switching between displaying a plurality of different landmark patterns 930A-930D over time 905, according to some embodiments. The display interface device 510 is configured to shift between displaying each of the plurality of different landmark patterns 930A-930D over time 905. The display interface device 510 can be configured to shift between displaying each of the plurality of different landmark patterns 930A-930D over time 905 according to a predetermined timing and / or a predetermined frequency, for example, to change from one of the landmark patterns 930A-930D periodically every time a particular duration (e.g., one second or more, or one millisecond or more) has elapsed since a previous landmark pattern change, or every time a particular number of frames (e.g., one or more frames) have been captured by the image sensor(s) 220 of the XR device 520 since a previous landmark pattern change.

[0108] The first landmark pattern 930A shown in FIG. 9 is a first QR code. The second landmark pattern 930B shown in FIG. 9 is an Aztec code. The third landmark pattern 930C shown in FIG. 9 is a second QR code. The fourth landmark pattern 930D ​​shown in FIG. 9 is a dot matrix code. The XR device 520 can determine the exact timing of a touch input based on which of the different landmark patterns 930A-930D is displayed when a given touch input is received.

[0109] In some embodiments, each of the different landmark patterns 930A-930D may be a deterministic pattern defined by a known number of rounds of a secure hash algorithm (SHA) (e.g., SHA-1, SHA-2, SHA-3, or SHA-N) that varies at a known frequency. Both the XR device 520 and the display interface device 510 may have knowledge of the SHA seed and pattern sequence. When the display interface device 510 sends touch input identifier information to the XR device 520, the display interface device 510 may tag the touch input identifier information with the landmark pattern that was displayed when the touch input was detected and / or registered.

[0110] The XR device 520 may be responsible for rendering the XR content (e.g., virtual interface and / or virtual content). The XR device 520 may also observe a landmark pattern that changes dynamically in real time via the image sensor 220 of the XR device 520. Combining the landmark pattern with the tagged touch input identification information, the XR device 520 may know exactly what XR content (e.g., virtual interface and / or virtual content) is being overlaid and displayed on the display interface device 510 when the touch input is registered. This synchronization may be particularly important for dynamic touch interfaces. In an exemplary embodiment, when the user 505 is dragging a (virtually rendered) slider of the virtual interface, it is desirable for the rendered slider position to match the position of the user's 505 finger exactly.

[0111] By detecting which landmark pattern of a set of different landmark patterns 930A-930D is depicted in an image and aligning it with the time that the touch screen device displayed that landmark pattern, the XR device 202 and the display interface device 510 can synchronize timing. Based on which landmark pattern the XR device 202 overlays its virtual interface over, the XR device 202 can obtain a more accurate indication of when a particular touch input was received at the display interface device 510 relative to what the XR device 202 is displaying as a virtual interface, and can increase the accuracy of the touch input for use by the XR device.

[0112] FIG 10 is a swim lane diagram illustrating operations 1000 performed by an extended reality (XR) device 1005 and a display interface device 1010 to provide a virtual interface, according to some embodiments. The XR device 1005 may be an embodiment of the XR device 202 of FIG 2 and / or the XR device 520 of FIG 5A-5B, FIG 6A-6B, FIG 7A-7B, FIG 8A-8B, and FIG 9. The display interface device 1010 may be an embodiment of the display interface device 250 of FIG 2 and / or the display interface device 510 of FIG 5A-5B, FIG 6A-6B, FIG 7A-7B, FIG 8A-8B, and FIG 9.

[0113] In operation 1015, the XR device 1005 searches for the display interface device 1010, for example, by searching for a representation of the display interface device 1010 in the image(s) captured by the image sensor 220 of FIG. 2 of the XR device 1005, by wirelessly transmitting a search signal and / or message within range of the XR device 1005 (to the display interface device 1010 if it is within range of the XR device 1005), and / or based on waiting to receive a wireless signal and / or message from the display interface device 1010 indicating the presence of the display interface device 1010 in the vicinity (e.g., wireless signal range) of the XR device 1005. The display interface device 1010 can transmit such signals and / or messages to the XR device 1005 in some embodiments. The XR device 1005 can establish a line of communication with the display interface device 1010, and vice versa.

[0114] In operation 1020, the display interface device 1010 transmits a message to the XR device 1005 indicating the presence of the display interface device 1010 in the vicinity of the XR device 1005 and / or requesting landmark pattern data from the XR device 1005. The display interface device 1010. Operation 1020 can be followed by operation 1025 and / or operation 1035.

[0115] At operation 1025, the XR device 1005 generates landmark pattern data and transmits the landmark pattern data to the display interface device 1010. Operation 1025 may correspond to blocks 226 and 274 of FIG. 2. At operation 1030, the display interface device 1010 receives the landmark pattern data from the XR device 1005 and generates further landmark pattern data based on the received landmark pattern data. The further landmark pattern data may be the received landmark pattern data or may be generated based on the received landmark pattern data (and may be different from the received landmark pattern data). Operation 1020 may be followed by operation 1035 and / or operation 1040.

[0116] At operation 1035, the display interface device 1010 generates the landmark pattern and may also transmit the landmark pattern data to the XR device 1005. At operation 1040, the display interface device 1010 displays the landmark pattern at operation 1040 on its display (e.g., the display 260 as in block 262 of FIG. 2 and / or the touchscreen display 515 of FIGS. 5A-9). The landmark pattern generated at operation 1035 may be based on the landmark pattern data transmitted from the XR device 1005 at operation 1025 and received by the display interface device 1010 at operation 1030. The landmark pattern generated at operation 1035 may be based on further landmark pattern data generated by the display interface device 1010 at operation 1030. The landmark pattern may be generated by the display interface device 1010 based on landmark pattern data generated by the display interface device 1010 and not the XR device 1005, in which case the display interface device 1010 may send the landmark pattern data to the XR device 1005 to ensure that the XR device 1005 has (or can generate) a reference copy of the landmark pattern. The landmark pattern data in operations 1025, 1030, 1035, 1040, 1050, and / or 1055 may be any of the types of landmark pattern data described with respect to blocks 226 and 274 of FIG.

[0117] In operation 1045, the XR device 1005 captures one or more images using the one or more image sensors 220 of the XR device 1005. In operation 1050, the XR device 1005 detects landmark patterns in the image(s) captured by the one or more image sensors 220 of the XR device 1005 by comparing the captured image(s) to reference landmark pattern data. In operation 1055, the XR device 1005 identifies and / or tracks the pose and / or size of the landmark pattern (and / or the display interface device 1010, the display of the display interface device 1010) in the image(s) based on the position, size, orientation, and / or perspective distortion of the landmark pattern in the image(s).

[0118] In operation 1060, the XR device 1005 and / or the display interface device 1010 generate at least a portion of the virtual interface. In operation 1065, the XR device 1005 and / or the display interface device 1010 generate at least a portion of the output image(s) by overlaying the virtual interface over the landmark pattern. The pose of the virtual interface in the output image(s) is based on the pose of the landmark pattern (and / or the display interface device 1010, the display of the display interface device 1010) in the image(s). The size of the virtual interface in the output image(s) is based on the size of the landmark pattern (and / or the display interface device 1010, the display of the display interface device 1010) in the image(s).

[0119] At operation 1070, the XR device 1005 displays the one or more output image(s) using one or more displays (e.g., display 234) of the XR device 1005. Operation 1070 may correspond to block 232 of FIG.

[0120] In operation 1075, the display interface device 1010 receives a display interface input at a portion of the display interface (e.g., the display interface 261 as in block 264 of FIG. 2 and / or the touch screen display 515 as in FIG. 5A-FIG. 9) and transmits a display interface input identifier (e.g., a touch interface input identifier) ​​corresponding to the display interface input (e.g., a touch input) to the XR device 1005 (e.g., as in blocks 272 and / or 228). In operation 1080, the XR device 1005 and / or the display interface device 1010 update other portions (e.g., virtual content, displayed interface) of the virtual interface and / or output image(s) based on the display interface input. In operation 1085, the XR device 1005 displays the updated output image(s) using one or more displays (e.g., the display 234) of the XR device 1005. Operation 1070 may correspond to block 232 of FIG. 2.

[0121] FIG. 11 is a flow diagram illustrating an embodiment of a process 1100 for processing image data, according to some embodiments. The process 1100 can be performed by an imaging system. In some embodiments, the imaging system can be the XR device 202 of FIG. 2. In some embodiments, the imaging system can include, for example, the image capture and processing system 100, the image capture device 105A, the image processing device 105B, the image processor 150, the ISP 154, the host processor 152, the XR system 200 of FIG. 2, the XR device 202, the display interface device 250, the HMD 310 of FIG. 3A-3B, the mobile handset 410 of FIG. 4A-4B, the XR device 520 of FIG. 5A-9, the display interface device 510 of FIG. 5A-9, the XR device 1005 of FIG. 10, the display interface device 1010 of FIG. 10, the computing system 1200 of FIG. 12, the processor 1210, or a combination thereof.

[0122] In operation 1105, the imaging system is configured and can receive an input image of a scene. The input image is captured by an image sensor. Examples of scenes include scene 110, a scene including at least a portion of display 260 of display interface device 250, FOV 560, FOV 660, FOV 760, FOV 860, a scene captured by the image(s) captured in operation 1045, or a combination thereof. Examples of image sensors include image sensor 130, image sensor(s) 220, sensor 276, image sensors of cameras 330A-330B, image sensors of cameras 430A-430D, one or more image sensors of XR device 520, one or more image sensors of XR device 1005 (e.g., as used in operation 1045), or a combination thereof. Examples of input images include an image captured by the image sensor 220 of FIG. 2, an image captured by the image sensor of the XR device 520 of FIG. 5A-5B (not yet overlaid with the virtual interface 535), an image captured by the image sensor of the XR device 520 of FIG. 6A-6B (not yet overlaid with the virtual interface 635), an image captured by the image sensor of the XR device 520 of FIG. 7A-7B (not yet overlaid with the virtual interface 735 or the virtual content 720), an image captured by the image sensor of the XR device 520 of FIG. 8A-8B (not yet overlaid with the virtual interface 835 or the virtual content 840), an image(s) captured in operation 1045, or a combination thereof. In some embodiments, the imaging system can include a connector coupled to the image sensor, and the input image can be received using the connector. The connector can include a port, a jack, a wire, an input / output (IO) pin, a conductive trace on a printed circuit board (PCB), any other type of connector described herein, or any combination thereof. In some embodiments, the imaging system may include an image sensor that captures the input image.In some embodiments, the imaging system may include an image capture and processing system 100, an image capture device 105A, and / or an image processing device 105B to capture an input image. The operation 1105 may correspond to at least a subset of the operations 1015, 1025, and / or 1045 by the XR device 1005.

[0123] In operation 1110, the imaging system is configured to and capable of detecting, in an input image of the scene, a landmark pattern displayed on a first display in the scene. The landmark pattern can include at least one of a linear glyph, a linear barcode, a barcode, a two-dimensional (2D) glyph, a 2D barcode, a Quick Response (QR) code, a micro QR code, a barcode, a MaxiCode, an Aztec code, a PDF417 code, an ArUco code, a Data Matrix, a Grid Matrix, a Code One code, a stacked barcode, a Schott code, a JAB code, a High Capacity Color Barcode (HCCB), a checkerboard pattern, a three-dimensional (3D) glyph, a 3D barcode, one or more colors, or a combination thereof. Examples of landmark patterns include the landmark patterns generated by the landmark pattern generator 214 (and / or for which landmark pattern data is generated), the landmark patterns generated by the landmark pattern generator 258 (and / or for which landmark pattern data is generated), the landmark patterns displayed on the display 260 as part of displaying the content 262, the landmark patterns 530, the landmark patterns 630, the landmark patterns 730, the landmark patterns 830, the landmark patterns 930A-930D, the landmark patterns generated in acts 1025-1035, the landmark patterns displayed in act 1040, or combinations thereof. In some embodiments, the imaging system includes a reference copy of the landmark pattern and detects the landmark pattern in the input image by searching for features that match features of the reference copy of the landmark pattern in the input image. Act 1110 may correspond to act 1050 by the XR device 1005.

[0124] In operation 1115, the imaging system is configured to and can determine a pose of the landmark pattern (and / or the first display) in the input image. The pose of the landmark pattern can include the location of the landmark pattern (e.g., two-dimensional coordinates in the image and / or three-dimensional coordinates in the environment), the orientation of the landmark pattern (e.g., pitch, yaw, and / or roll), or a combination thereof. In some aspects, the imaging system is configured to and can determine a size of the landmark pattern (and / or the first display) in the input image. The size can include a three-dimensional volume, a two-dimensional area, and / or a one-dimensional measurement (e.g., height, width). The size can include an absolute size and / or a relative size (with respect to others in the image data). The size of the landmark pattern can be identified as an absolute size or as a size relative to another size in the input image (e.g., the size of the first display, the size of a display device displaying the landmark pattern on the first display, the size of a person). The size of the first display can be identified as an absolute size or as a size relative to another size in the input image (e.g., the size of the landmark pattern, the size of a display device displaying the landmark pattern on the first display, the size of a person). In some examples, the imaging system determines the pose of the landmark pattern (and / or the first display) in the input image by comparing a representation of the landmark pattern in the input image to a reference copy (without perspective distortion) of the landmark pattern in the input image, and determines the pose, at least in part, by determining the perspective distortion of the representation of the landmark pattern in the input image based on the comparison to the reference copy. In some examples, the imaging system determines the pose and / or size of the first display and / or a display device including the first display, at least in part, by receiving information from the display device indicating the pose and / or size of the first display and / or the display device. Operation 1115 can correspond to operation 1055 by the XR device 1005.

[0125] In operation 1120, the imaging system is configured and capable of causing the second display to display an output image based on the input image. The virtual interface is overlaid on top of the landmark pattern in the output image. The pose of the virtual interface in the output image is based on the pose of the landmark pattern (and / or the first display) in the input image. The imaging system can generate and / or modify the virtual interface and / or the output image to have a pose of the virtual interface based on the pose of the landmark pattern (and / or the first display) in the input image. In some aspects, the size of the virtual interface in the output image is based on the size of the landmark pattern in the input image and / or the size of the first display. The imaging system can generate and / or modify the virtual interface and / or the output image to have a size of the virtual interface based on the size of the landmark pattern (and / or the first display) in the input image. The imaging system can include a second display.

[0126] Examples of virtual interfaces include a virtual interface generated at least in part by virtual interface generator 206, virtual interface 535, virtual interface 635, virtual interface 735, virtual interface 835, a virtual interface generated in operation 1060, a virtual interface overlaid on a landmark pattern in operation 1065, a virtual interface updated in operation 1080, or a combination thereof. Examples of output images include an output image generated at least in part using output image composer 212, an output image(s) displayed by displaying output image(s) 232 on display(s) 234, an output image depicting FOV 560 and including virtual interface 535 and displayed to user 505 of XR device 520 via one or more displays of XR device 520, an output image depicting FOV 660 and including virtual interface 635 and occlusions 610 and occlusions 620 and displayed to user 505 of XR device 520 via one or more displays of XR device 520, an output image depicting FOV 760 and including virtual interface 735 and virtual The output image may include the content 720 and the displayed interface 710 and displayed to the user 505 of the XR device 520 via one or more displays of the XR device 520, the output image depicting the FOV 860 and including the virtual interface 835 and the occlusion 810 and the virtual content 840 and displayed to the user 505 of the XR device 520 via one or more displays of the XR device 520, the output image(s) generated in the operation 1065, the output image(s) generated in the operation 1070, the output image(s) updated in the operation 1080, the output image(s) displayed in the operation 1085, or a combination thereof. The operation 1120 may correspond to at least a subset of the operations 1060-1085 by the XR device 1005.

[0127] In some aspects, the imaging system is configured and capable of identifying that an object occludes an area of ​​the first display that includes at least a portion of the landmark pattern in the input image. As in operation 1120, causing the second display to display the output image can include the imaging system occluding a portion of the virtual interface corresponding to the area of ​​the first display in the output image. Examples of objects include occlusion 610 (e.g., a hand), occlusion 620 (e.g., a finger), and occlusion 810 (e.g., a finger). Examples of imaging systems occluding a portion of the virtual interface corresponding to the area of ​​the first display in the output image are shown at least in Figures 6B and 8B. For example, in Figure 6B, occlusions 610 and 620 occlude a portion of the virtual interface 635 corresponding to the area of ​​the touchscreen display 515 occluded by occlusions 610 and 620 in Figure 6A. In Figure 8B, occlusion 810 occludes a portion of the virtual interface 835 corresponding to the area of ​​the touchscreen display 515 occluded by occlusion 810 in Figure 8A.

[0128] In some aspects, the imaging system is configured and capable of generating at least a portion of the virtual interface. In some aspects, the imaging system is configured and capable of receiving at least a portion of the virtual interface from a display device including the first display. Examples of the display device may include the display interface device 250, the HMD 310, the mobile handset 410, the display interface device 520, the display interface device 1010, or a combination thereof. For example, in FIG. 10, the generation of the virtual interface in operation 1060 may be performed by the XR device 1005, the display interface device 1010, or a combination thereof.

[0129] In some aspects, the imaging system is configured and capable of generating at least a portion of the output image. In some aspects, generating at least a portion of the output image includes modifying the virtual interface using perspective distortion based on the pose of the landmark pattern (and / or the first display) in the input image. For example, the imaging system can perform display interface device tracking 224 to track the pose of the landmark pattern (and / or the first display) in the input image. In some examples, the imaging system can determine the pose of the first display based on the pose of the landmark pattern. The imaging system can include a virtual interface generator 206 and an output image compositor 212 that can perform perspective distortion of the virtual interface in the output image 232 based on the pose of the landmark pattern (and / or the first display) in the input image. An example of perspective distortion is shown in FIGS. 5A-5B, where the virtual interface 535 is rotated slightly clockwise based on the pose of the landmark pattern 530 and / or the touch screen display 515 and / or the display interface device 510 from the FOV 560 of the XR device 520. The virtual interface generator 206 and the output image combiner 212 may perform resizing of the virtual interface in the output image 232 based on the size of the landmark pattern (and / or the first display) in the input image.

[0130] In some aspects, the imaging system is configured and capable of generating landmark pattern data corresponding to the landmark pattern and, in response to receiving the landmark pattern data, transmitting the landmark pattern data to a display device, including a first display, for the display device to display the landmark pattern on the first display. Examples of display devices may include the display interface device 250, the HMD 310, the mobile handset 410, the display interface device 520, the display interface device 1010, or a combination thereof. The landmark pattern data may include a landmark pattern. The landmark pattern data may include data characterizing the landmark pattern and / or may be used to recreate a reference copy of the landmark pattern (e.g., data encoded by a QR code or barcode or other coding scheme of the landmark pattern). The landmark pattern data may be generated by the landmark pattern generator 214 and may be transmitted to the display device (display interface device 250) via the wireless transceiver(s) 230 (block 226).

[0131] In some aspects, the imaging system is configured to receive, and can receive, a display interface input identifier from a display device including a first display, the display interface indicating a portion of the first display that receives the display interface input via a display interface of the display device, the display interface being associated with the first display. Examples of display devices can include display interface device 250, HMD 310, mobile handset 410, display interface device 520, display interface device 1010, or combinations thereof. Examples of display interfaces corresponding to the first display include display interface 261 corresponding to display 260. An example of receiving a display interface input is shown in block 264 of FIG. 2.

[0132] The first display may be a display layer of a touchscreen display of a display device. The display interface may be a touch-sensitive layer of the touchscreen display. The display interface input may be a touch input detected by the touch-sensitive layer of the touchscreen display. An example of a touchscreen display is touchscreen display 515 of display interface device 510. The touch input may include, for example, a touch, a tap, a double tap, a triple tap, a swipe, a long press, a gesture, a multi-finger variant of any of the touch inputs listed above, or a combination thereof.

[0133] The display interface can control a cursor on the first display. The display interface input can be cursor input based on a position of the cursor on the first display. The display interface can include at least one of a mouse, a trackpad, a touch-sensitive surface, a touch screen, a joystick, a keypad, a keyboard, a button, a controller, a remote control, or a combination thereof. The cursor input can include, for example, a click, a double click, a triple click, a click-and-drag operation, circling with the cursor, pointing with the cursor, hovering over with the cursor, gesturing around or over with the cursor, or a combination thereof.

[0134] The display interface can perform hand tracking of the hand relative to the first display. The display interface input can indicate a position on the first display corresponding to the position of the hand. The display interface can include one or more cameras and / or one or more distance sensors. The display interface can perform hand tracking using one or more cameras and / or one or more distance sensors. The display interface input can be associated with at least one of a hand touching a position on the first display, a hand hovering over a position on the first display, a hand pointing to a position on the first display, a hand gesturing to a position on the first display, a finger of a hand touching a position on the first display, a finger of a hand hovering over a position on the first display, a finger of a hand pointing to a position on the first display, a finger of a hand gesturing to a position on the first display, or a combination thereof. The imaging system can include at least a subset of one or more cameras and / or one or more distance sensors. A display device including the first display can include at least a subset of one or more cameras and / or one or more distance sensors. The one or more distance sensors may include, for example, a RADAR sensor, a LIDAR sensor, a SONAR sensor, a SODAR sensor, a time-of-flight sensor, a structured light sensor, or a combination thereof.

[0135] In some aspects, the imaging system is configured and capable of identifying that a portion of the first display indicated by the display interface input identifier aligns with a portion of the virtual interface in the output image. For example, the imaging system can identify that the display interface input identifier indicates a display interface input at a particular set of coordinates on the first display, where the set of coordinates on the first display also aligns with a virtual interface element of the virtual interface (e.g., one or more buttons, sliders, scroll bars, radio buttons, check boxes, knobs, wheels, text fields, touch-based keypads, touch-based keyboards, touch-based drawing areas, or combinations thereof). For example, in FIG. 8B, the display interface input identifier may be a touch input identifier indicating that the thumb of the right hand of the user 505 touched a set of coordinates on the touchscreen display 515, and the imaging system can identify that the set of coordinates on the touchscreen display 515 corresponds to one of the controller buttons of the virtual interface 835.

[0136] The imaging system can automatically modify the virtual interface in response to identifying that a portion of the first display indicated by the display interface input identifier aligns with a portion of the virtual interface in the output image. For example, if the virtual interface is the music player virtual interface 535 of FIG. 5B, pressing a "skip" button on the virtual interface 535 via touch input to the touch screen 515 can trigger modification of the virtual interface to skip to the next song and thus identify the next song. Similarly, if the virtual interface is the music player virtual interface 535 of FIG. 5B, sliding a slider on the virtual interface 535 via touch input to the touch screen 515 can trigger modification of the virtual interface to move the slider. If the virtual interface is the video player virtual interface 735 of FIG. 7B, sliding a slider of the virtual interface 735 via touch input to the touch screen 515 or pressing any button of the virtual interface 735 (e.g., play, fast forward, rewind, pause) may move the slider and / or trigger modification of the virtual interface to update the video being played by the video player virtual interface 735 with the appropriate video frame. If the virtual interface is the image viewer virtual interface 635 of FIG. 6B, swiping down on the image viewer virtual interface 635 may scroll through images and thus trigger modification of the virtual interface to scroll through images. If the virtual interface is the image viewer virtual interface 635 of FIG. 6B, touching an image in the image viewer virtual interface 635 may zoom in or center on the selected image and thus trigger modification of the virtual interface to zoom in or center on the selected image.

[0137] The imaging system can automatically output an audio clip in response to identifying that a portion of the first display identified by the display interface input identifier aligns with a portion of the virtual interface in the output image. For example, if the virtual interface is the music player virtual interface 535 of FIG. 5B, a song can be played by pressing a "play" button on the virtual interface 535 via touch input to the touch screen 515, the song being an example of an audio clip. If the virtual interface is the video player virtual interface 735 of FIG. 5B, a song can be played by pressing a "play" button on the virtual interface 535 via touch input to the touch screen 515, the song being an example of an audio clip. If the virtual interface is the video player virtual interface 735 of FIG. 7B, sliding a slider on the virtual interface 735 or pressing any button (e.g., play, fast forward, rewind, pause) on the virtual interface 735 via touch input to the touch screen 515 can trigger playing an audio track of a video being played by the video player virtual interface 735 corresponding to an appropriate video frame, the audio track of a video being played being an example of an audio clip. If the virtual interface is the game controller virtual interface 835 of FIG. 8B, various game buttons on the virtual interface 835 may be pressed via touch input to perform actions in the game (e.g., fire a weapon) and play sound effects in the game, which are an example of an audio clip.

[0138] The imaging system can automatically output vibration in response to identifying that a portion of the first display identified by the display interface input identifier aligns with a portion of the virtual interface in the output image. An example of a vibration includes vibration feedback 845. For example, if the virtual interface is the game controller virtual interface 835 of FIG. 8B, actions can be performed in the game (e.g., firing a weapon) by pressing various game buttons on the virtual interface 835 via touch input, and vibration feedback 845 can be provided.

[0139] The imaging system can receive a second input image of the scene. The second input image can be captured by an image sensor after capture of the input image. The imaging system can cause a second display to display a second output image. The second output image includes virtual content overlaid on the second input image. The virtual content is automatically configured (e.g., selected, presented in the scene, sized in the scene) by the imaging system based on identifying that a portion of the first display indicated by the display interface input identifier aligns with a portion of the virtual interface in the output image. Examples of virtual content include virtual content generated by virtual content generator 210, virtual content composited into the second output image using output image compositer 212, virtual content 720, virtual content 840, or a combination thereof. Examples of the second output image include an output image generated at least in part using the output image composer 212, an output image(s) displayed by displaying the output image(s) 232 on the display(s) 234, an output image depicting the FOV 760 and including the virtual interface 735 and the virtual content 720 and the displayed interface 710 and displayed to the user 505 of the XR device 520 via one or more displays of the XR device 520, an output image depicting the FOV 860 and including the virtual interface 835 and the occlusion 810 and the virtual content 840 and displayed to the user 505 of the XR device 520 via one or more displays of the XR device 520, the output image(s) generated in operation 1065, the output image(s) generated in operation 1070, the output image(s) updated in operation 1080, the output image(s) displayed in operation 1085, or combinations thereof.

[0140] In some aspects, the imaging system may include means for receiving an input image of a scene captured by an image sensor, means for detecting, in the input image of the scene, a landmark pattern displayed on a first display in the scene, means for determining a pose of the landmark pattern in the input image, and means for causing the second display to display an output image based on the input image, where a virtual interface is overlaid over the landmark pattern in the output image, and the pose of the virtual interface in the output image is based on the pose of the landmark pattern in the input image. In some embodiments, the means for receiving the input image includes one or more image sensors of the image sensor 130, the image capture device 105A, the image processing device 105B, the image capture and processing device 100, the XR device 202, the image sensor 220, the XR application 204, the cameras 330A-330B, the cameras 430A-430D, the XR device 520, one or more image sensors of the XR device 1005 (e.g., as used in operation 1045), or a combination thereof. In some embodiments, the means for detecting the landmark pattern includes the image processing device 105B, the image capture and processing device 100, the image processor 150, the host processor 152, the image sensor 220, the display interface device tracking 224, a display interface device tracking engine of the XR device 202 running the display interface device tracking 224, the XR application 204, or a combination thereof. In some embodiments, the means for determining the pose of the landmark pattern includes the image processing device 105B, the image capture and processing device 100, the image processor 150, the host processor 152, the image sensor 220, the display interface device tracking 224, a display interface device tracking engine of the XR device 202 running the display interface device tracking 224, the XR application 204, or a combination thereof.In some embodiments, the means for causing the second display to display the output image includes a virtual interface generator 208, an occlusion detector 208, a virtual content generator 210, an output image composer 212, display output content 232, a display(s) 234, an XR application 204, a display controller(s) for the display(s) 234, or a combination thereof.

[0141] In some embodiments, the processes described herein (e.g., process 1100 and / or other processes described herein) may be performed by a computing device or apparatus. In some embodiments, process 1100 may be performed by XR system 200 of FIG. 2. In some embodiments, process 1100 may be performed by XR device 202 of FIG. 2, XR device 520 of FIGS. 5A-9, XR device 1005 of FIG. 10, or a combination thereof. In another embodiment, process 1100 may be performed by a computing device having computing system 1200 shown in FIG. 12.

[0142] The computing device may include any suitable device, such as a mobile device (e.g., a mobile phone), a desktop computing device, a tablet computing device, a wearable device (e.g., a VR headset, an AR headset, AR glasses, a network-connected watch or smartwatch, or other wearable device), a server computer, an autonomous vehicle or a computing device of an autonomous vehicle, a robotic device, a television, and / or any other computing device, having resource capabilities to perform the processes described herein, including process 1100. In some cases, the computing device or apparatus may include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and / or other components configured to perform steps of the processes described herein. In some examples, the computing device may include a display, a network interface configured to communicate and / or receive data, any combination thereof, and / or other components. The network interface may be configured to communicate and / or receive Internet Protocol (IP)-based data or other types of data.

[0143] Components of a computing device may be implemented in circuitry. For example, components may include and / or be implemented using electronic circuitry or other electronic hardware, which may include one or more programmable electronic circuits (e.g., a microprocessor, a graphics processing unit (GPU), a digital signal processor (DSP), a central processing unit (CPU), and / or other suitable electronic circuitry), and / or may include and / or be implemented using computer software, firmware, or any combination thereof, to perform various operations described herein.

[0144] Processes 1000 and 1100 are illustrated as logical flow diagrams, whose operations represent sequences of operations that may be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, the operations represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the described operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform particular functions or implement particular data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations may be combined in any order and / or in parallel to implement a process.

[0145] Additionally, processes 1000, 1100, and / or other processes described herein may be executed under the control of one or more computer systems configured with executable instructions and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that collectively execute on one or more processors, by hardware, or a combination thereof. As mentioned above, the code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program comprising a plurality of instructions executable by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.

[0146] Fig. 12 illustrates an example of a system for implementing some aspects of the present technology. In particular, Fig. 12 illustrates an example of a computing system 1200, which may be any computing device, such as an internal computing system, a remote computing system, a camera, or any components thereof, where the components of the system communicate with each other using a connection 1205. The connection 1205 may be a physical connection using a bus, or a direct connection to a processor 1210, such as in a chipset architecture. The connection 1205 may also be a virtual connection, a network connection, or a logical connection. Examples of the computing system 1200 may include, for example, the image capture and processing system 100, image capture device 105A, image processing device 105B, image processor 150, ISP 154, host processor 152 of FIG. 1, the XR system 200, XR device 202, display interface device 250 of FIG. 2, the HMD 310 of FIGS. 3A-3B, the mobile handset 410 of FIGS. 4A-4B, the XR device 520 of FIGS. 5A-9, the display interface device 510 of FIGS. 5A-9, the XR device 1005 of FIG. 10, the display interface device 1010 of FIG. 10, an imaging system that performs operation 1100, the computing system 1200, processor 1210 of FIG. 12, or a combination thereof.

[0147] In some embodiments, computing system 1200 is a distributed system in which the functionality described in this disclosure may be distributed across a data center, multiple data centers, a peer network, etc. In some embodiments, one or more of the described system components represent many components, each performing some or all of the functionality that is the subject of the component description. In some embodiments, the components may be physical or virtual devices.

[0148] The exemplary system 1200 includes at least one processing unit (CPU or processor) 1210 and connections 1205 coupling various system components to the processor 1210, including system memory 1215, such as read only memory (ROM) 1220 and random access memory (RAM) 1225. The computing system 1200 may include a cache 1212 of high speed memory either directly connected to the processor 1210, in close proximity to the processor 1210, or integrated as part of the processor 1210.

[0149] Processor 1210 may include any general-purpose processor, as well as hardware or software services, such as services 1232, 1234, and 1236 stored in storage device 1230, configured to control processor 1210, and special-purpose processors where software instructions are embedded in the actual processor design. Processor 1210 may essentially be a completely self-contained computing system, including multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0150] To enable user interaction, computing system 1200 includes input devices 1245, which can represent any number of input mechanisms, such as a microphone for speech, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, speech, etc. Computing system 1200 can also include output devices 1235, which can be one or more of several output mechanisms. In some cases, a multi-modal system can enable a user to provide multiple types of input / output to communicate with computing system 1200. Computing system 1200 can include a communication interface 1240, which can generally govern and manage user input and system output.The communications interface may be an audio jack / plug, a microphone jack / plug, a Universal Serial Bus (USB) port / plug, an Apple® Lightning® port / plug, an Ethernet port / plug, a fiber optic port / plug, a proprietary wired port / plug, BLUETOOTH® wireless signal transmission, BLUETOOTH® Low Energy (BLE) wireless signal transmission, IBEACON® wireless signal transmission, Radio Frequency Identification (RFID) wireless signal transmission, Near Field Communication (NFC) wireless signal transmission, Dedicated Short Range Communications (DSRC) wireless signal transmission, 802.11 The communication interface 1240 may perform or facilitate the reception and / or transmission of wired or wireless communications using wired and / or wireless transceivers, including those utilizing Wi-Fi wireless signal transmission, wireless local area network (WLAN) signal transmission, visible light communication (VLC), Worldwide Interoperability for Microwave Access (WiMAX), infrared (IR) communication wireless signal transmission, public switched telephone network (PSTN) signal transmission, integrated services digital network (ISDN) signal transmission, 3G / 4G / 5G / LTE cellular data network wireless signal transmission, ad-hoc network signal transmission, radio signal transmission, microwave signal transmission, infrared signal transmission, visible light signal transmission, ultraviolet light signal transmission, wireless signal transmission along the electromagnetic spectrum, or any combination thereof. The communication interface 1240 may also include one or more Global Navigation Satellite System (GNSS) receivers or transceivers used to determine a position of the computing system 1200 based on reception of one or more signals from one or more satellites associated with one or more GNSS systems. GNSS systems include, but are not limited to, the United States Global Positioning System (GPS), the Russian Global Navigation Satellite System (GLONASS), the Chinese BeiDou Navigation Satellite System (BDS), and the European Galileo GNSS.There is no constraint to operating on any particular hardware configuration, and therefore basic features herein may be easily substituted for improved hardware or firmware configurations as they are developed.

[0151] The storage device 1230 may be a non-volatile and / or non-transitory and / or computer readable memory device, such as a magnetic cassette, a flash memory card, a solid state memory device, a digital versatile disk, a cartridge, a floppy disk, a flexible disk, a hard disk, a magnetic tape, a magnetic strip / stripe, any other magnetic storage medium, a flash memory, a memristor memory, any other solid state memory, a compact disc read only memory (CD-ROM) optical disk, a rewritable compact disc (CD) optical disk, a digital video disk (DVD) optical disk, a blu-ray disc (BDD) optical disk, a holographic optical disk, another optical medium, a secure digital (SD) card, a micro secure digital (microSD) card, a memory stick card, a smart card chip, an EMV chip, a subscriber identity module (SIM) card, a mini / micro / nano / pico SIM card, another integrated circuit (ICC), a circuit, IC chip / card, random access memory (RAM), static RAM (SRAM), dynamic RAM (DRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash EPROM, cache memory (L1 / L2 / L3 / L4 / L5 / L#), resistive random-access memory (RRAM),The memory may be a hard disk or other type of computer readable medium capable of storing data that is accessible by a computer, such as a memory (RRAM / ReRAM), phase change memory (PCM), spin transfer torque RAM (STT-RAM), another memory chip or cartridge, and / or a combination thereof.

[0152] The storage devices 1230 may include software services, servers, services, etc. that cause the system to perform functions when code defining such software is executed by the processor 1210. In some embodiments, hardware services that perform specific functions may include software components stored in a computer-readable medium in conjunction with the necessary hardware components, such as the processor 1210, connections 1205, output devices 1235, etc., to perform the functions.

[0153] The term "computer-readable medium" as used herein includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, storing, or transporting instruction(s) and / or data. Computer-readable media may include non-transitory media on which data may be stored and which do not include carrier waves and / or transitory electronic signals propagating wirelessly or over wired connections. Examples of non-transitory media may include, but are not limited to, magnetic disks or tapes, optical storage media such as compact disks (CDs) or digital versatile disks (DVDs), flash memory, memory, or memory devices. A computer-readable medium may have code and / or machine-executable instructions stored thereon, which may represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. may be passed, forwarded, or transmitted using any suitable means including memory sharing, message passing, token passing, network transmission, etc.

[0154] In some embodiments, computer readable storage devices, media, and memories may include cables or wireless signals containing bit streams, etc. However, when stated, non-transitory computer readable storage media specifically excludes media such as energy, carrier signals, electromagnetic waves, and the signals themselves.

[0155] Specific details are provided in the above description to provide a thorough understanding of the embodiments and examples provided herein. However, it will be understood by those skilled in the art that the embodiments may be practiced without these specific details. For clarity of explanation, in some cases, the present technology may be presented as including individual functional blocks, including devices, device components, steps or routines in a method embodied in software, or functional blocks comprising a combination of hardware and software. Additional components other than those shown in the figures and / or described herein may be used. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form so as not to obscure the embodiments in unnecessary detail. In other cases, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail so as to avoid obscuring the embodiments.

[0156] Particular embodiments may be described above as a process or method that is depicted as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although the flowchart may describe operations as a sequential process, many of the operations may be performed in parallel or simultaneously. In addition, the order of operations may be rearranged. A process terminates when its operations are completed, but may have additional steps not included in the diagram. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, its termination may correspond to a return of the function to the calling function or to the main function.

[0157] The processes and methods according to the examples described above may be implemented using computer-executable instructions stored or otherwise available from a computer-readable medium. Such instructions may include, for example, instructions and data that cause a general-purpose computer, a special-purpose computer, or a processing device to perform a certain function or group of functions, or otherwise configure a general-purpose computer, a special-purpose computer, or a processing device to perform a certain function or group of functions. Portions of the computer resources used may be accessible over a network. The computer-executable instructions may be, for example, binary, intermediate format instructions, such as assembly language, firmware, source code, etc. Examples of computer-readable media that may be used to store instructions, information used, and / or information created during the methods according to the described examples include magnetic or optical disks, flash memory, USB devices with non-volatile memory, network-attached storage devices, etc.

[0158] Devices implementing the processes and methods according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, the program code or code segments (e.g., computer program product) to perform the necessary tasks may be stored in a computer-readable or machine-readable medium. A processor or processors may perform the necessary tasks. Typical examples of form factors include laptops, smartphones, mobile phones, tablet devices or other small form factor personal computers, personal digital assistants, rack-mounted devices, standalone devices, and the like. The functionality described herein may also be embodied in a peripheral device or an add-in card. Such functionality may also be implemented on a circuit board among different chips, or on different processes executing within a single device, as further examples.

[0159] The instructions, media for carrying such instructions, computing resources for executing such instructions, and other structures for supporting such computing resources are exemplary means for providing the functionality described in this disclosure.

[0160] In the above description, aspects of the present application are described with reference to specific embodiments thereof, but those skilled in the art will recognize that the present application is not limited thereto. Thus, while exemplary embodiments of the present application have been described in detail herein, it should be understood that the inventive concepts may be embodied and employed in various other ways, and that the appended claims are intended to be construed to include such variations, except as limited by the prior art. The various features and aspects of the present application described above may be used individually or jointly. Moreover, the embodiments may be utilized in any number of environments and applications other than those described herein without departing from the broader spirit and scope of the present specification. Thus, the present specification and drawings should be regarded as illustrative and not restrictive. For purposes of illustration, methods have been described in a particular order. It should be appreciated that in alternative embodiments, methods may be performed in an order different from that described.

[0161] Those skilled in the art will understand that the less than ("<") and greater than (">") symbols or terms used herein may be replaced with the less than or equal to ("≦") and greater than or equal to ("≧") symbols, respectively, without departing from the scope of this description.

[0162] When a component is described as being "configured to" perform some operation, such configuration may be achieved, for example, by designing electronic circuitry or other hardware to perform the operation, by programming a programmable electronic circuitry (e.g., a microprocessor or other suitable electronic circuitry) to perform the operation, or any combination thereof.

[0163] The phrase "coupled to" refers to any component that is physically connected, either directly or indirectly, to another component and / or that is in communication, either directly or indirectly, with another component (e.g., connected to the other component via a wired or wireless connection and / or other suitable communication interface).

[0164] Claim language or other language reciting "at least one of" a set and / or "one or more" of a set indicates that one member of the set or multiple members of the set (in any combination) satisfy the claim. For example, a claim language reciting "at least one of A and B" means A, B, or A and B. In another example, a claim language reciting "at least one of A, B, and C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language "at least one of" a set and / or "one or more" of a set does not limit the set to the items listed in the set. For example, a claim language reciting "at least one of A and B" can mean A, B, or A and B, and can further include items not listed in the set of A and B.

[0165] The various exemplary logic blocks, modules, circuits, and algorithm steps described with respect to the embodiments disclosed herein may be implemented as electronic hardware, computer software, firmware, or a combination thereof. To clearly illustrate this interchangeability of hardware and software, the various exemplary components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in various ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

[0166] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices, such as a general purpose computer, a wireless communication device handset, or an integrated circuit device having multiple uses, including applications in wireless communication device handsets and other devices. Any features described as modules or components may be implemented together in an integrated logic device, or separately as discrete but interoperable logic devices. If implemented in software, the techniques may be realized at least in part by a computer-readable data storage medium comprising program code including instructions that, when executed, perform one or more of the methods described above. The computer-readable data storage medium may form part of a computer program product, which may include packaging materials. The computer-readable medium may comprise a memory or data storage medium, such as a random access memory (RAM), such as a synchronous dynamic random access memory (SDRAM), a read-only memory (ROM), a non-volatile random access memory (NVRAM), an electrically erasable programmable read-only memory (EEPROM), a FLASH memory, a magnetic or optical data storage medium, etc. The techniques may additionally or alternatively be realized at least in part by a computer-readable communications medium, such as a propagated signal or wave, that carries or communicates program code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer.

[0167] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Such a processor may be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor, but alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. Thus, the term "processor" as used herein may refer to any of the above structures, any combination of the above structures, or any other structure or apparatus suitable for implementing the techniques described herein. Additionally, in some aspects, the functionality described herein may be provided in dedicated software or hardware modules configured for encoding and decoding, or may be incorporated into a composite video encoder-decoder (codec).

[0168] Exemplary aspects of the present disclosure include the following. Aspect 1: An apparatus for processing image data, comprising: a memory; and one or more processors coupled to the memory, wherein the one or more processors are configured to receive an input image of a scene captured by an image sensor; detect in the input image of the scene a landmark pattern displayed on a first display in the scene; determine a pose of the landmark pattern in the input image; and cause a second display to display an output image based on the input image, wherein a virtual interface is overlaid on top of the landmark pattern in the output image, and the pose of the virtual interface in the output image is based on the pose of the landmark pattern in the input image.

[0169] Aspect 2. The device of aspect 1, wherein the landmark pattern comprises at least one of a linear glyph, a linear barcode, a barcode, a two-dimensional (2D) glyph, a 2D barcode, a quick response (QR) code, a micro QR code, a barcode, MaxiCode, an Aztec code, a PDF417 code, an ArUco code, a data matrix, a grid matrix, a Code One code, a stacked barcode, a Schott code, a JAB code, a high capacity color barcode (HCCB), a checkerboard pattern, a three-dimensional (3D) glyph, a 3D barcode, and one or more colors.

[0170] Aspect 3. The apparatus of aspect 1 or 2, wherein the one or more processors are configured to identify that an object occludes an area of ​​the first display that includes at least a portion of a landmark pattern in the input image, and wherein to cause the second display to display the output image, the one or more processors are configured to occlude a portion of the virtual interface that corresponds to the area of ​​the first display in the output image.

[0171] Embodiment 4. The apparatus of any one of embodiments 1 to 3, wherein the one or more processors are configured to generate at least a portion of the virtual interface.

[0172] Embodiment 5. The apparatus of any one of embodiments 1 to 4, wherein the one or more processors are configured to receive at least a portion of the virtual interface from a display device including the first display.

[0173] Embodiment 6. An apparatus as described in any one of embodiments 1 to 5, wherein the one or more processors are configured to generate at least a portion of an output image.

[0174] Aspect 7. The apparatus of aspect 6, wherein the one or more processors are configured to modify the virtual interface using perspective distortion based on the pose of a landmark pattern in the input image to generate at least a portion of the output image.

[0175] Embodiment 8. The apparatus of any one of embodiments 1 to 7, wherein the one or more processors are configured to generate landmark pattern data corresponding to the landmark pattern, and, in response to receiving the landmark pattern data, transmit the landmark pattern data to a display device including the first display, so that the display device displays the landmark pattern on the first display.

[0176] Aspect 9. The apparatus of any one of aspects 1 to 8, wherein the one or more processors are configured to receive, from a display device including the first display, a display interface input identifier indicating a portion of the first display that receives display interface input via a display interface of the display device, and the display interface is associated with the first display.

[0177] Example 10. The apparatus of example 9, wherein the first display is a display layer of a touch screen display of the display device, the display interface is a touch sensitive layer of the touch screen display, and the display interface input is a touch input detected by the touch sensitive layer of the touch screen display.

[0178] Aspect 11. The apparatus of aspect 9 or 10, wherein the display interface controls a cursor on the first display, the display interface input is cursor input based on a position of the cursor on the first display, and the display interface includes at least one of a mouse, a trackpad, a touch-sensitive surface, a touch screen, a joystick, a keypad, a keyboard, a button, a controller, and a remote control.

[0179] Aspect 12. The apparatus of any one of aspects 9 to 11, wherein the display interface performs hand tracking relative to the first display, and the display interface input indicates a position on the first display corresponding to the position of the hand, and the display interface includes at least one of a camera and a distance sensor, and the display interface input is associated with at least one of a hand touching a position on the first display, a hand hovering over a position on the first display, a hand pointing to a position on the first display, and a hand gesturing relative to a position on the first display.

[0180] Embodiment 13. An apparatus as described in any one of embodiments 9 to 12, wherein the one or more processors are configured to identify that a portion of the first display indicated by the display interface input identifier aligns with a portion of the virtual interface in the output image.

[0181] Aspect 14. The apparatus of aspect 13, wherein the one or more processors are configured to automatically modify the virtual interface in response to identifying that a portion of the first display indicated by the display interface input identifier aligns with a portion of the virtual interface in the output image.

[0182] Aspect 15. The device of aspect 13 or 14, wherein one or more processors are configured to receive a second input image of the scene captured by the image sensor after capture of the input image, and cause the second display to display a second output image including virtual content overlaid on the second input image, the virtual content being automatically set based on identifying that a portion of the first display indicated by the display interface input identifier aligns with a portion of the virtual interface in the output image.

[0183] Aspect 16. An apparatus as described in any one of aspects 13 to 15, wherein the one or more processors are configured to automatically output an audio clip in response to identifying that a portion of a first display identified by the display interface input identifier aligns with a portion of a virtual interface in the output image.

[0184] Embodiment 17. An apparatus as described in any one of embodiments 13 to 16, wherein the one or more processors are configured to automatically output a vibration in response to identifying that a portion of a first display identified by a display interface input identifier aligns with a portion of a virtual interface in the output image.

[0185] Aspect 18. An apparatus as described in any one of aspects 1 to 17, wherein the one or more processors are configured to determine a size of a first display in the input image, and the size of the virtual interface in the output image is based on the size of the first display in the input image.

[0186] Aspect 19. An apparatus described in any one of aspects 1 to 18, wherein the one or more processors are configured to determine a size of a landmark pattern in the input image, and the size of the virtual interface in the output image is based on the size of the landmark pattern in the input image.

[0187] Aspect 20. The apparatus of any one of aspects 1 to 19, further comprising an image sensor.

[0188] Aspect 21. The device of any one of aspects 1 to 20, further comprising a second display.

[0189] Aspect 22. The apparatus of any one of aspects 1 to 21, wherein the apparatus includes at least one of a mobile handset, a wireless communication device, and a head mounted display (HMD).

[0190] Aspect 23. A method for processing image data, comprising: receiving an input image of a scene captured by an image sensor; detecting a landmark pattern displayed on a first display in the scene within the input image of the scene; determining a pose of the landmark pattern in the input image; and causing a second display to display an output image based on the input image, wherein a virtual interface is overlaid on top of the landmark pattern in the output image, and the pose of the virtual interface in the output image is based on the pose of the landmark pattern in the input image.

[0191] Aspect 24. The method of aspect 23, wherein the landmark pattern comprises at least one of a linear glyph, a linear barcode, a barcode, a two-dimensional (2D) glyph, a 2D barcode, a quick response (QR) code, a micro QR code, a barcode, MaxiCode, an Aztec code, a PDF417 code, an ArUco code, a data matrix, a grid matrix, a Code One code, a stacked barcode, a Shot code, a JAB code, a high capacity color barcode (HCCB), a checkerboard pattern, a three-dimensional (3D) glyph, a 3D barcode, and one or more colors.

[0192] Aspect 25. The method of aspect 23 or 24, further comprising identifying that the object occludes an area of ​​the first display that includes at least a portion of the landmark pattern in the input image, and causing the second display to display the output image includes occluding a portion of the virtual interface corresponding to the area of ​​the first display in the output image.

[0193] Aspect 26. The method of any one of aspects 23 to 25, further comprising generating at least a portion of the virtual interface.

[0194] Aspect 27. The method of any one of aspects 23 to 26, further comprising receiving at least a portion of the virtual interface from a display device including the first display.

[0195] Embodiment 28. The method of any one of embodiments 23 to 27, further comprising generating at least a portion of an output image.

[0196] Aspect 29. The method of aspect 28, wherein generating at least a portion of the output image includes modifying the virtual interface using perspective distortion based on the pose of a landmark pattern in the input image.

[0197] Aspect 30. The method of any one of aspects 23 to 29, further comprising: generating landmark pattern data corresponding to the landmark pattern; and, in response to receiving the landmark pattern data, transmitting the landmark pattern data to a display device including the first display, for the display device to display the landmark pattern on the first display.

[0198] Aspect 31. The method of any one of aspects 23 to 30, further comprising receiving a display interface input identifier from a display device including the first display, the display interface indicating a portion of the first display that receives display interface input via a display interface of the display device, the display interface being associated with the first display.

[0199] Aspect 32. The method of aspect 31, wherein the first display is a display layer of a touch screen display of the display device, the display interface is a touch sensitive layer of the touch screen display, and the display interface input is a touch input detected by the touch sensitive layer of the touch screen display.

[0200] Aspect 33. The method of aspect 31 or 32, wherein the display interface controls a cursor on the first display, the display interface input is cursor input based on a position of the cursor on the first display, and the display interface includes at least one of a mouse, a trackpad, a touch-sensitive surface, a touch screen, a joystick, a keypad, a keyboard, a button, a controller, and a remote control.

[0201] Aspect 34. The method of any one of aspects 31 to 33, wherein the display interface performs hand tracking relative to the first display, the display interface input indicates a position on the first display corresponding to the position of the hand, the display interface includes at least one of a camera and a distance sensor, and the display interface input is associated with at least one of a hand touching a position on the first display, a hand hovering over a position on the first display, a hand pointing to a position on the first display, and a hand gesturing relative to a position on the first display.

[0202] Aspect 35. The method of any one of aspects 31 to 34, further comprising identifying that a portion of the first display indicated by the display interface input identifier aligns with a portion of the virtual interface in the output image.

[0203] Aspect 36. The method of aspect 35, further comprising automatically modifying the virtual interface in response to identifying that a portion of the first display indicated by the display interface input identifier aligns with a portion of the virtual interface in the output image.

[0204] Aspect 37. The method of aspect 35 or 36, further comprising receiving a second input image of the scene captured by the image sensor after capture of the input image, and causing the second display to display a second output image including virtual content overlaid on the second input image, wherein the virtual content is automatically set based on identifying that a portion of the first display indicated by the display interface input identifier aligns with a portion of the virtual interface in the output image.

[0205] Aspect 38. The method of any one of aspects 35 to 37, further comprising automatically outputting an audio clip in response to identifying that a portion of the first display identified by the display interface input identifier aligns with a portion of the virtual interface in the output image.

[0206] Aspect 39. The method of any one of aspects 35 to 38, further comprising automatically outputting a vibration in response to identifying that a portion of the first display identified by the display interface input identifier aligns with a portion of the virtual interface in the output image.

[0207] Aspect 40. The method of any one of aspects 23 to 39, further comprising determining a size of the first display in the input image, wherein the size of the virtual interface in the output image is based on the size of the first display in the input image.

[0208] Aspect 41. A method according to any one of aspects 23 to 40, further comprising determining a size of a landmark pattern in the input image, wherein the size of the virtual interface in the output image is based on the size of the landmark pattern in the input image.

[0209] Embodiment 42. The method of any one of embodiments 23 to 41, performed by an apparatus including an image sensor.

[0210] Embodiment 43. The method of any one of embodiments 23 to 42, performed by an apparatus including a second display.

[0211] Aspect 44. The method of any one of aspects 23 to 43, performed by an apparatus including at least one of a mobile handset, a wireless communication device, and a head mounted display (HMD).

[0212] Aspect 45: A non-transitory computer-readable medium having instructions stored thereon that, when executed by one or more processors, cause the one or more processors to receive an input image of a scene captured by an image sensor, detect in the input image of the scene a landmark pattern displayed on a first display in the scene, determine a pose of the landmark pattern in the input image, and display on a second display an output image based on the input image, wherein a virtual interface is overlaid on top of the landmark pattern in the output image, and wherein the pose of the virtual interface in the output image is based on the pose of the landmark pattern in the input image.

[0213] Example 46: The non-transitory computer-readable medium of example 45, further comprising any of examples 2 to 22 and / or any of examples 24 to 44.

[0214] Aspect 47: An apparatus for image processing, comprising: means for receiving an input image of a scene captured by an image sensor; means for detecting, within the input image of the scene, a landmark pattern displayed on a first display within the scene; means for determining a pose of the landmark pattern in the input image; and means for displaying, on a second display, an output image based on the input image, wherein a virtual interface is overlaid on top of the landmark pattern in the output image, and the pose of the virtual interface in the output image is based on the pose of the landmark pattern in the input image.

[0215] Embodiment 48: The apparatus of embodiment 47, further comprising any of embodiments 2 to 22 and / or any of embodiments 24 to 44.

Claims

**Claim 1** An apparatus for processing image data, comprising: a memory; one or more processors coupled to the memory; wherein the one or more processors are configured to: receive an input image of a scene captured by an image sensor; detect, within the input image of the scene, a landmark pattern displayed on a first display within the scene, wherein a portion of the landmark pattern is blocked by an occluder; determine an orientation of the landmark pattern within the input image; compare the landmark pattern detected within the input image of the scene with an unblocked reference landmark pattern and identify a portion of the landmark pattern blocked by the occluder; cause an output image based on the input image to be displayed on a second display; wherein a virtual interface is overlaid on the landmark pattern within the output image, and an orientation of the virtual interface within the output image is based on the orientation of the landmark pattern within the input image, a portion of the virtual interface is blocked within the output image, and the portion of the virtual interface is selected based on the portion of the landmark pattern blocked by the occluder within the input image; An apparatus. **Claim 2** The apparatus according to claim 1, wherein the landmark pattern includes a Quick Response (QR) code. **Claim 3** The one or more processors are configured to: generate at least a portion of the virtual interface. The apparatus according to claim 1. **Claim 4** The one or more processors are configured to: receive at least a portion of the virtual interface from a display device including the first display. The apparatus according to claim 1. **Claim 5** The one or more processors are configured to: generate at least a portion of the output image, and in order to generate at least the portion of the output image, the one or more processors are configured to correct the virtual interface using a perspective distortion based on the orientation of the landmark pattern within the input image. **Claim 6** the one or more processors generate landmark pattern data corresponding to the landmark pattern, and in response to receiving the landmark pattern data, send the landmark pattern data to a display device including the first display for the display device to display the landmark pattern on the first display, the apparatus according to claim 1.

7. the one or more processors are configured to receive a display interface input identifier indicating a portion of the first display from a display device including the first display via a display interface of the display device, the display interface being associated with the first display, the apparatus according to claim 1.

8. the first display is a display layer of a touch screen display of the display device, the display interface is a touch sensing layer of the touch screen display, and the display interface input is a touch input detected by the touch sensing layer of the touch screen display, the one or more processors are configured to identify that a portion of the first display indicated by the display interface input identifier aligns with a portion of the virtual interface in the output image, the apparatus according to claim 7.

9. the one or more processors are configured to automatically correct the virtual interface in response to identifying that a portion of the first display indicated by the display interface input identifier aligns with the portion of the virtual interface in the output image, the apparatus according to claim 8.

10. the one or more processors receive a second input image of the scene captured by the image sensor after the capture of the input image, ​ ​ ​ ​ ​ ​ ​ configured to cause the second display to display a second output image including virtual content overlaid on the second input image, the virtual content being automatically set based on identifying that a portion of the first display indicated by the display interface input identifier aligns with a portion of the virtual interface within the output image The apparatus according to claim 8 **Claim 11** the one or more processors configured to automatically output an audio clip in response to identifying that a portion of the first display identified by the display interface input identifier aligns with a portion of the virtual interface within the output image The apparatus according to claim 8 **Claim 12** the one or more processors configured to determine a size of the first display within the input image, the size of the virtual interface within the output image being based on the size of the first display within the input image, or the one or more processors configured to determine a size of the landmark pattern within the input image, the size of the virtual interface within the output image being based on the size of the landmark pattern within the input image The apparatus according to claim 1 **Claim 13** further comprising the image sensor or further comprising the second display The apparatus according to claim 1 **Claim 14** The apparatus according to claim 1, wherein the apparatus includes at least one of a mobile handset, a wireless communication device, and a head-mounted display (HMD) **Claim 15** A method of processing image data, comprising receiving an input image of a scene captured by an image sensor detecting, within the input image of the scene, a landmark pattern displayed on a first display within the scene, a portion of the landmark pattern being obscured by an obstruction determining an orientation of the landmark pattern within the input image Comparing the landmark pattern detected in the input image of the scene with an unobstructed reference landmark pattern to identify a portion of the landmark pattern that is obstructed by an obstruction; Displaying an output image based on the input image on a second display; wherein a virtual interface is overlaid on the landmark pattern in the output image, a posture of the virtual interface in the output image is based on the posture of the landmark pattern in the input image, a portion of the virtual interface is obscured in the output image, and the portion of the virtual interface is selected based on the portion of the landmark pattern that is obscured by the obstruction in the input image. Method.

Images