Content reprojection between positions

Abstract

The present disclosure relates to content reprojection between locations. Various implementations disclosed herein include devices, systems, and methods that enable a user at a first location to view, from the user's point of view, reprojected content from a camera at a second location. For example, a process can render a first view from a first point of view within a first location within an environment. The first view can include an obstruction that prevents the first view from including a portion of a second location within the environment. The process can also detect a triggering condition associated with viewing the second location, and in response, the process can also render a second view of the environment generated based on image data captured from a second point of view within the second location.

Description

Technical Field

[0001] This disclosure relates in its entirety to systems, methods, and apparatuses that enable a user at one location to view, from the user's viewpoint, reprojected content from cameras at different locations. Background Technology

[0002] Existing technologies that enable users to view content from different locations can be improved relative to a true and accurate view to provide the desired viewing experience. Summary of the Invention

[0003] The various specific embodiments disclosed herein include devices, systems, and methods that enable a user at a first location to view, from the user's viewpoint, the reprojected content or image from a camera at a second location, as if the user could see the second location through an obstacle. For example, a user in a first room, while wearing a wearable device such as a head-mounted display (HMD), could view the reprojected content from a camera in a second room adjacent to the first room, as if the user could see the second location through a wall.

[0004] In some implementations, a view of reprojected content or images from a camera at a second location can be provided in an extended reality (XR) environment via a portal (such as, for example, a virtual opening in a wall). In some implementations, an existing portal can be initialized or triggered by manually placing (e.g., dragging) it to a location (such as a position on a wall in an XR environment). In some implementations, a portal can be initialized or triggered in response to a user gaze pointing at a location (such as a position on a wall in an XR environment). In some implementations, a portal can be initialized or triggered in response to a user gesture (e.g., hand gesture, finger gesture, head gesture, facial expression, etc.) performed relative to a location (such as a position on a wall in an XR environment).

[0005] In some implementations, the portal may be initialized (e.g., opened) slowly when a user engages with a portal or location on an obstacle (such as a wall). In other implementations, the portal may be initialized instantaneously when a user engages with a portal or location on an obstacle (such as a wall).

[0006] In some implementations, the previously mentioned process may use the pose of an image sensor (such as a camera) relative to the pose of a user using a viewing device (at the first location) (such as an HMD) to provide a line-of-sight view through a portal into a second location. In some implementations, the pose of the image sensor may be aligned with the user's viewpoint by positioning the image sensor, for example, using a servo system or robotic equipment.

[0007] In some embodiments, an electronic device has a processor (e.g., one or more processors) that executes instructions stored in a non-transitory computer-readable medium to perform a method. The method performs one or more steps or processes. In some embodiments, the electronic device presents a first view from a first viewpoint within a first location in an XR environment via one or more displays. The first view includes an obstacle that prevents the first view from including a portion of a second location within the XR environment. The second location is different from the first location. In some embodiments, triggering conditions associated with viewing the second location are triggered. In some embodiments, alignment criteria are determined based on the orientation of an image sensor capable of capturing image data from the second viewpoint within the second location. In some embodiments, based on the determination that alignment criteria are met, a second view of the XR environment is presented such that, based on image data from an image sensor that captures image data from the second viewpoint within the second location, the portion of the second location is included in the second view.

[0008] According to some embodiments, an apparatus includes one or more processors, non-transitory memory, and one or more programs; the one or more programs are stored in the non-transitory memory and configured to be executed by the one or more processors, and the one or more programs include instructions for performing or causing to perform any of the methods described herein. According to some embodiments, a non-transitory computer-readable storage medium stores instructions that, when executed by one or more processors of the apparatus, cause the apparatus to perform or cause to perform any of the methods described herein. According to some embodiments, an apparatus includes: one or more processors, non-transitory memory, and components for performing or causing to perform any of the methods described herein. Attached Figure Description

[0009] To enable those skilled in the art to understand this disclosure, more detailed descriptions can be made with reference to aspects of some exemplary embodiments, some of which are shown in the accompanying drawings.

[0010] Figures 1A to 1B Exemplary electronic devices operating in a physical environment according to some specific implementations are illustrated.

[0011] Figures 2A to 2D Examples of various configurations of user views of an XR environment, representing a device's display (such as an HMD), are shown, according to some specific implementations.

[0012] Figure 3This is a block diagram illustrating an example system of a user equipment according to some specific implementation, the user equipment being configured to enable a user at a first location to view, from the user's viewpoint, the reprojected content or image from a camera at a second location.

[0013] Figure 4 The flowchart represents an exemplary method according to some specific implementations, which enables a user at a first location to view, from the user's viewpoint, the reprojected content or image from a camera at a second location, as if the user could see the second location through an obstacle.

[0014] Figure 5 It is a block diagram based on some specific implementations of electronic devices.

[0015] As is customary practice, various features illustrated in the accompanying drawings may not be drawn to scale. Therefore, for clarity, the dimensions of various features may be arbitrarily expanded or reduced. Furthermore, some drawings may not depict all components of a given system, method, or apparatus. Finally, throughout the specification and drawings, the same reference numerals may be used to denote the same features. Detailed Implementation

[0016] Numerous details have been described to provide a thorough understanding of the exemplary embodiments illustrated in the accompanying drawings. However, the drawings illustrate only some exemplary aspects of this disclosure and should not be considered limiting. Those skilled in the art will understand that other effective aspects and / or variations do not include all the specific details described herein. Furthermore, well-known systems, methods, components, devices, and circuits have not been described exhaustively so as not to obscure further relevant aspects of the exemplary embodiments described herein.

[0017] Figures 1A to 1B Exemplary electronic devices 105 and 110 are illustrated in physical environment 100. Figures 1A to 1B In the example, physical environment 100 includes room 104, which includes table 120. Electronic devices 105 and 110 may include one or more cameras, microphones, depth sensors, or other sensors that can be used to capture and evaluate information about physical environment 100 and objects within it, as well as information about user 102 of electronic devices 105 and 110. Information about physical environment 100 and / or user 102 can be used to provide visual and audio content, and / or identify the current location of physical environment 100 and / or the location of the user within physical environment 100.

[0018] In some implementations, a view of the XR environment may be provided to one or more participants (e.g., user 102 and / or other participants not shown) via electronic device 105 (e.g., wearable device such as HMD, etc.) and / or 110 (e.g., handheld device such as mobile device, tablet computing device, laptop computer, etc.). Such an XR environment may include a view of a 3D environment generated based on camera images and / or depth camera images of the physical environment 100, and a representation of user 102 based on camera images and / or depth camera images of user 102. Such an XR environment may include virtual content positioned at a 3D location relative to a 3D coordinate system associated with the XR environment, which may correspond to the 3D coordinate system of the physical environment 100.

[0019] The various specific embodiments disclosed herein include devices, systems, and methods for implementing gaze tracking methods using image data. In some specific embodiments, imaging data can be used to track gaze to determine eye position or eye orientation using a pupil plus flash model, a depth camera with 3D point cloud alignment (e.g., stereo, structured light projection, time-of-flight (ToF), etc.), or an appearance-based model.

[0020] In some implementations, device 105 and / or device 110 enable, for example, a user 102 in room 104 to view a reprojected image from a camera in a room adjacent to room 104 (e.g., a room on the other side of any of the walls 104a…104c), as if the user could see the adjacent room through one of the walls 104a…104c. For example, a first view (e.g., an XR environment presented via device 105 and / or device 110) from a first viewpoint within room 104 (e.g., a viewpoint from device 105 and / or device 110) may be presented to the user 102. The first view may include obstacles (e.g., walls 104a…104c) that prevent the first view from allowing the user 102 to view a portion of an adjacent room located on the other side of any of the walls 104a…104c.

[0021] In some implementations, trigger conditions associated with viewing an adjacent room can be detected via sensor data obtained from sensors. For example, detecting trigger conditions may include determining that user 102 is looking at or gesturing toward an adjacent room. Alternatively or additionally, detecting trigger conditions may include detecting that user 102 is dragging or moving a user interface element (such as a portal) (e.g., within an XR environment) to a viewing location (e.g., on one of the walls 104a…104c), as described below relative to… Figures 2A to 2D As described.

[0022] In some implementations, alignment criteria can be determined based on the determined orientation of an image sensor (e.g., a camera) configured to capture image data from a second viewpoint in an adjacent room. For example, it can be determined that the camera in the adjacent room is adequately aligned (e.g., viewpoint aligned) in the same direction as device 105 and / or device 110, such that a realistic view (presented within an XR environment) of a specified quality level can be provided to user 102. The view of the specified quality level may, in particular, include views with limited apertures, limited illusions, limited reprojection errors, etc. In some implementations, the specified quality level can be determined using a threshold angular difference between the orientation of device 105 or 110 and the camera capture orientation.

[0023] In some implementations, a second view of the XR environment may be presented to user 102 based on determining that alignment criteria are met. In some implementations, a portion of the adjacent room is included in the second view based on image data from a camera capturing image data from a second viewpoint within the adjacent room. For example, this portion of the adjacent room in the second view can be generated by replacing a portion of a wall / obstacle with content obtained from a camera in the adjacent room. The content obtained from the camera in the adjacent room may be modified or reprojected to provide a view of the adjacent room corresponding to a first viewpoint rather than a second viewpoint. In some implementations, providing the second view may include providing a (virtual) opening or portal within the walls through which a second location can be viewed.

[0024] Figures 2A to 2D Views 200a-200d illustrate various configurations of the user view of the XR environment 227, which can be viewed via a display (such as an HMD) of device 211, according to some specific implementations.

[0025] Figure 2A View 200a is illustrated, representing a user 203 viewing an XR environment 227 via device 211. The XR environment 227 includes a physical environment 214 (e.g., a first room, such as a furnished living room), where walls 215a, 215b, and 215c create obstacles preventing viewing content within other physical environments. For example, user 203 may be located within physical environment 214, and wall 215c may obstruct viewing content within physical environment 219 (e.g., a second room, such as an office with a desk) relative to the user 203. In some implementations, physical environment 219 includes an image sensor 228a (such as, for example, a camera).

[0026] In some specific implementations, a user 203 within physical environment 214 may want to observe objects within physical environment 219 (e.g., a computer, television, table, people such as children), and therefore the user 203 may trigger a view of a portion of physical environment 219 via a virtual opening or portal placed on wall 215c to provide a view of that portion of physical environment 219 (e.g., reprojected content) aligned with the user 203's viewpoint, as described below relative to... Figures 2B to 2D As described.

[0027] In some specific implementations, a gesture / movement activation triggering mechanism may be enabled to activate the virtual opening (e.g., as shown in the image) by allowing the user 203 to use gestures (e.g., hand or finger gestures). Figure 2B The illustrated virtual opening 207 is used to enable the virtual opening structure and place (e.g., drag) the virtual opening structure to a specified location (e.g., a specified location on a wall) to view another physical location (e.g., a room on the other side of the wall).

[0028] In some specific implementations, a gaze activation triggering mechanism may be enabled to activate a virtual opening (e.g., as shown below) by using a gaze (direction) vector 206a pointing towards a portion of the wall 215c that intersects with the gaze of the image sensor 228a (of user 203). Figure 2B (Example: Virtual opening 207). In this case, as the user 203 looks towards the target area 236, the virtual opening may gradually fade in. For example, the virtual opening may initially be semi-transparent, and while the user's gaze remains locked on the target area 236 for a period of time (e.g., a few seconds), the virtual opening may fully open, presenting a completely opaque and accurate view of the physical environment 219, as shown. Figure 2B As illustrated, in some implementations, the virtual opening can adapt in real time as the user's eye moves (or focus shifts) to keep the view aligned with the user's viewpoint. Similarly, the virtual opening can automatically adjust its zoom, rotation, or angle based on the direction the user is looking.

[0029] In some specific implementations, a gaze activation triggering mechanism may be enabled to activate a virtual opening (e.g., as shown in the image) by using the gaze (direction) vector 206a (of user 203) and a virtual activation point 229 (or multiple virtual activation points) located at a specified position in the XR environment 227 (e.g., by user 203). Figure 2BThe illustrated virtual opening 207. For example, gaze vector 206a (i.e., the user's gaze direction in 3D space) may be projected toward virtual activation point 229 (e.g., invisible or highlighted when in the user 203's field of view). Virtual activation point 229 may be presented as one (or more) virtual objects within XR environment 227, such as a sphere. Virtual activation point 229 may have a fixed position or a dynamic position following the user 203's line of sight. Similarly, virtual activation point 229 may be repositioned within XR environment 227 (e.g., on wall 215c) such that when gaze vector 206 intersects virtual activation point 229 (e.g., when...). Figure 2A (As illustrated), this is registered as an activation event. Similarly, a fixed time period (e.g., user 203 must look at virtual activation point 229 for at least one second) can be used to avoid accidental activation events. In some specific implementations, the triggering mechanism for gaze activation can use a ray activation point (e.g., sphere) intersection algorithm to: (a) define activation point 229 by its center location and radius; (b) detect when gaze vector 206 intersects with activation point 229; and (c) define an activation threshold to avoid false alarms (e.g., fixed duration or alignment).

[0030] In some implementations, a virtual opening can be created by enabling user 203 to draw the shape and boundaries of the virtual opening directly on a physical surface (e.g., a wall, table, or other flat surface) via gestures, a stylus, or a controller in the XR environment 227.

[0031] In some implementations, visual indicators (e.g., luminous outlines) may be provided to indicate to the user the location where the virtual opening can appear.

[0032] In some implementations, the size, shape, and orientation of the virtual opening can be adjusted. For example, a pinch gesture can be used to reset the size of the virtual opening, while a rotation gesture or head movement can align the size of the virtual opening with a specific orientation.

[0033] Figure 2B View 200b representing an XR environment 227 is illustrated. This XR environment includes a physical environment 214 having a virtual opening 207 (e.g., a portal) at a first location 217a on a wall 215c and aligned with the user 203’s viewpoint to provide a view of the physical environment 219.

[0034] In some implementations, the view provided by the virtual opening 207 is created by reprojecting the image from the image sensor 228a (located in the physical environment 219) relative to the positioning of the device 211 with respect to the user 203. For example, the image sensor 228a may be configured to capture image data (in real time) representing portions of the physical environment 219, and the image data may be (e.g., by the device 211) transformed to match the user 203's current viewing angle, as if the user 203 (located in the physical environment 214) were seeing the physical environment 219 through a portal or window within the wall 215c.

[0035] In some implementations, the virtual opening 207 is created via video reprojection technology, which includes adjusting the video feed from the image sensor 228a using three-dimensional (3D) scene reconstruction techniques, depth information, and / or machine learning (ML)-based modeling and / or rule-based algorithms.

[0036] In some implementations, rule-based deterministic methods can be implemented to create virtual openings 207 using predefined rules (e.g., geometric rules, camera calibration parameters, etc.) for capturing, transforming, and displaying content. For example, tracking data associated with the user 203's head / gaze positioning and orientation relative to wall 215c within physical environment 214 can be used to establish the virtual viewing orientation of user 203 (within physical environment 214). Similarly, tracking data can be used to calculate the corresponding virtual viewpoint of (user 203) in physical environment 219 as if user 203 were looking through wall 215c along gaze vector 206a. Therefore, fixed geometric relationships (e.g., offset, rotation matrices) can be used to deterministically map the user 203's positioning and orientation within physical environment 214 to the viewpoint in physical environment 219. Subsequently, in cases where image sensor 228 is oriented and positioned at a distance from gaze vector 206 in different ways, video feeds from the image sensor (e.g., image sensor 228a) determined to be closest to the user's virtual viewpoint can be directly fed (e.g., in cases where image sensor 228a is oriented and positioned along gaze vector 206) or reprojected (e.g., using a predefined model, such as a perspective projection model) to match the virtual viewpoint.

[0037] In some implementations, the adjusted video feed can be configured to present a view of a portion of the physical environment 219 (e.g., via virtual opening 207), as if the user 203 were seeing the physical environment 219 through a physical window. Similarly, the adjusted video feed can accurately render depth and 3D effects (e.g., parallax).

[0038] In some specific implementations, 3D scene reconstruction techniques may include using a camera or multi-camera setup (e.g., image sensor 228a) within physical environment 219 to capture the 3D structure of the scene represented in physical environment 219 by converting the scene into a 3D model or point cloud and reprojecting the resulting 3D scene based on the viewpoint of the user 203 relative to virtual opening 207 and the position or pose of image sensor 228a. For example, image sensor 228a (e.g., an RGB camera) may be configured to capture video frames (representing the scene in physical location 219) relative to camera viewpoint (orientation) vector 221 and at a specific resolution and frame rate, and a monocular depth estimation algorithm may be used to generate a depth map for each frame. The depth map may be combined with intrinsic parameters of image sensor 228a (focal length, principal point, etc.) to generate a 3D point cloud of the scene. Subsequently, a transformation or transformation matrix may be computed based on the user viewpoint (e.g., exemplified by gaze vector 206a) relative to the camera viewpoint (e.g., exemplified by camera viewpoint vector 221). The transformation matrix may include the following transformation chains: (a) device 211 to world anchor, which includes a dynamic, real-time relationship defining the user's position and orientation within physical environment 214; (b) world anchor to room anchor, which includes a map defining the alignment of physical environment 219 relative to physical environment 214; and (c) room anchor to image sensor 228a pose, which defines the position and orientation of image sensor 228a within physical environment 219 (e.g., camera viewpoint (orientation) vector 221). In some implementations, the transformation matrix transforms the 3D point cloud from the viewpoint of image sensor 228a (e.g., camera viewpoint vector 221) to the viewpoint of user 203 (gaze vector 206a), and the point cloud is reprojected relative to the user's viewpoint for final frame rendering.

[0039] In some implementations, the viewpoint of user 203 can be determined based on a real-time 3D gaze signal that tracks the position of user 203 relative to the physical environment 214. For example, eye-tracking hardware (e.g., an inward-facing camera of device 211) can be configured to capture the gaze direction of user 203 (e.g., gaze vector 206a) by analyzing, in particular, pupil localization, corneal reflection, and head orientation, to infer the gaze direction and generate eye-tracking data.

[0040] In some implementations, the physical environment 219 may be 3D mapped using depth cameras, stereo cameras, and / or Simultaneous Localization and Mapping (SLAM) techniques to create a spatial model of the environment, including objects, surfaces, and their positions relative to the user 203.

[0041] In some implementations, eye-tracking data can be combined with a spatial model to determine where the user's gaze intersects with the physical environment 214, thereby indicating what the user 203 is focusing on in the real world, including its depth and spatial context.

[0042] In some cases, the angle between the user 203's viewpoint (e.g., gaze vector 206a) and the image sensor 228a's viewpoint (e.g., camera viewpoint vector 221) may become too steep, causing reprojection on the virtual opening 207 to distort the video feed, resulting in visual artifacts, mismatches in lighting or occlusion effects, etc. Therefore, the video feed is configured to fade out when the angle exceeds a threshold.

[0043] In some implementations, the stored model representing the physical environment 219 can be used to mitigate distortion caused by steep angles between the user 203's viewpoint (e.g., gaze vector 206a) and the image sensor 228a's viewpoint (e.g., camera viewpoint vector 221). For example, if the user 203 has previously accessed the physical environment 219 (e.g., within a threshold time period to ensure relatively recent data) and the device 211 has been exposed to its structure, this information can be used to enhance / fill holes or assist in the reprojection of images from the image sensor 228a.

[0044] In some specific implementations, a gaze activation triggering mechanism can be enabled to activate the virtual opening 207 by using the gaze direction (of user 203) and a virtual activation point, as described above relative to... Figure 2A As described. For example, a virtual activation point can be placed on a wall between rooms, and when user 203 looks at the virtual activation point, gaze vector 206 will intersect the virtual activation point to activate virtual opening 207.

[0045] In some implementations, a servo system or robotic component may be used to move the image sensor 228a to align the optical axis of the image sensor 228a with the line of sight or gaze of the user 203, so as to accurately reproject the content relative to the virtual opening 207. For example, the servo system may be used to rotate the image sensor to align it with the gaze direction of the user 203.

[0046] In some implementations, multiple image sensors or cameras can be used to capture the physical environment 219 from different perspectives. For example, selecting or blending video feeds from multiple image sensors based on the user's 293 viewpoint can enable the virtual opening 207 to present accurate content as the user's viewpoint shifts.

[0047] In some implementations, the content within the virtual opening 207 can be scaled toward or away from the user 203 to provide zoom and wide-angle view capabilities.

[0048] In some specific implementations, the virtual opening 207 can be configured to appear relative to different shapes and sizes. For example, the virtual opening 207 can be circular, rectangular, triangular, panoramic, a 3D floating window, or a cloud-like structure (e.g., as shown in the image). Figure 2D (as illustrated), any type of special shape (such as a star or a human figure), etc.

[0049] Figure 2C An example is illustrated in view 200c representing an XR environment 227, which includes a physical environment 219 having a virtual opening 207 that moves to a second position 217b on a wall 215c to provide a different view of the physical environment 214. In some specific implementations, the virtual opening 207 can be moved from a first position 217a (e.g., ...) Figure 2B (As illustrated) Drag (e.g., using an input device or gesture input) to the second location 217b to move the location of the virtual opening 207. In some implementations, the location of the virtual opening 207 can be moved by using a gaze activation trigger that follows the gaze direction, such as when the gaze vector 206b of user 203 intersects the virtual activation point, as described above relative to... Figure 2A and Figure 2B As described. In some implementations, the exposed area of ​​the virtual opening 207 may remain exposed after the user 203 looks into the distance. Alternatively, the exposed area of ​​the virtual opening 207 may be removed from the view after the user 203 looks into the distance.

[0050] In some implementations, virtual opening 207 can be configured to provide a virtual opening of any size. For example, virtual opening 207 can be configured to present a view of the physical environment 219 as if the entire wall 215c or a portion of wall 215c (e.g., larger than virtual opening 207) had been removed. Similarly, virtual opening 207 of any size can be configured to remain static so that it does not move relative to the user's gaze.

[0051] In some implementations, the virtual opening 207 can be configured to reveal any number of areas through the wall 215c in response to dynamic objects within the physical environment 219. For example, if two individuals move within the physical environment 219 and the user is looking at the wall 215c and / or a trigger point (e.g., as...), Figure 2A The virtual activation point 229 shown can reveal two areas (virtual openings), one area (virtual opening) for each individual / dynamic object.

[0052] In some implementations, user 203 may be able to define the areas to be displayed when looking at wall 215c. For example, an area including a specific type of furniture (e.g., a crib) may always be displayed via a virtual opening, regardless of the user 203's gaze direction. Similarly, an area associated with user 203's gaze may be displayed (via another virtual opening) in conjunction with a virtual opening associated with an area including a specific type of furniture. Thus, if user 203 is looking at wall 215c, portals around the crib and portals around the gaze location away from the crib may be activated and presented to user 203. Likewise, the portal for the crib may be stationary, while the portal for the user's gaze may move according to the gaze direction.

[0053] Figure 2D View 200d, representing an XR environment 227, is illustrated. This XR environment includes a physical environment 214 with a virtual opening 212, which includes different shapes activated at a location 223 on a wall 215a to provide a view of another physical environment 218, including an image sensor 228b (such as, for example, a camera).

[0054] In some specific implementations, the shape of the virtual opening 212 can be created by selecting a shape from multiple shape templates.

[0055] In some implementations, the shape of the virtual opening 212 can be created by enabling the user 203 to draw the shape and boundaries of the virtual opening 212 directly on a physical surface (e.g., wall 215a) via gestures, stylus or controllers in the XR environment 227.

[0056] In some implementations, the virtual opening 212 can be moved by dragging (e.g., using an input device or gesture input) from another location within the XR environment 227. In some implementations, the virtual opening 207 can be activated via a gaze-activated trigger that uses the gaze direction of the user 203 intersecting the virtual activation point, as described above relative to... Figure 2A and Figure 2B As described.

[0057] Figure 3 This is a block diagram illustrating an example system of user equipment 300, configured such that a user at a first location (e.g., a first room) can view, from the user's viewpoint, reprojected content or images from a camera at a second location (e.g., a second adjacent room), as if the user could see the second location through an obstacle (e.g., a wall) (e.g., via a doorway). User equipment 400 (e.g., an HMD) may include an input module 302, a content reprojection module 304, a display module 306, and a lightweight engine 308.

[0058] In some implementations, the input module 302 is configured to accept internal input data 305 associated with user characteristics (e.g., internal input data from sensors of the user equipment 300), such user characteristics being particularly things like a person's gaze direction and head posture in the physical environment. For example, the internal input data 305 may include images from an inward-facing camera of the HMD.

[0059] In some implementations, the input module 302 is configured to accept external input data 309 (e.g., external input data from sensors or storage devices outside the user equipment 300), including, for example, video feeds or images from another location (such as an adjacent room).

[0060] Input 310 from input module 302 can be fed to content reprojection module 304. Content reprojection module 304 may include one or more learning-based and / or non-learning-based models for perceiving, synthesizing, and inferring information. Those skilled in the art will understand that content reprojection module 304 may include any suitable number of rule-based processes to generate output 312 based on input 310.

[0061] In some embodiments, the content reprojection module 304 may be used in conjunction with, for example, calibrated camera parameters and fixed geometry, which can be used to deterministically map the user's positioning and orientation in a first room to the viewpoint of a camera in an adjacent room. In some embodiments, the content reprojection module 304 may be used to calculate a transformation between the known position of a camera with the current position of the user device 300 and the user's current gaze direction. This transformation extends from the position and orientation of the external device to the position and gaze direction of the user device 300.

[0062] In some implementations, user device 300 may optionally use lightweight engine 308 instead of content reprojection module 304 to deterministically map the user's location and orientation in the first room to a viewpoint in the second room to provide a view of the second room (e.g., via a virtual opening such as a portal). Lightweight engine 308 may be a non-learning network.

[0063] In some implementations, the display module 306 is configured to receive results from the content reprojection module 304 and / or the lightweight engine 308 to deterministically map the user’s position and orientation in the first room to a viewpoint in the second room to provide a view of the second room (e.g., via a virtual opening such as a portal).

[0064] For example, the output from the content reprojection module 304 and / or the lightweight engine 308 is provided to the display module 306, which provides a view of the reprojected content or image from the camera at the second location from the user's viewpoint, as if the user could see the second location through an obstacle (e.g., a wall) (e.g., via a portal).

[0065] Those skilled in the art will understand that the content reprojection module 304 may include any suitable machine learning model that is well-known or widely available, such as regression techniques, classification techniques, neural networks, and deep learning networks. For example, the content reprojection module 304 may include neural networks such as artificial neural networks (ANNs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), adversarial networks (GANs), reinforcement learning models (RLMs), encoder / decoder networks, and / or transformer-based models (e.g., bidirectional encoder representations from transformers (BERT), generative pre-trained transformers (GPTs), and / or multimodal large language models (LLMs)). Additionally or alternatively, those skilled in the art will understand that the content reprojection module 304 may be any suitable non-learning process, such as rule-based systems, heuristics, decision trees, knowledge-based systems, statistical or stochastic systems, and expert systems.

[0066] In the case where the content reprojection module 304 is based on a machine learning model, one or more well-known or widely available training techniques (such as supervised learning, semi-supervised learning, unsupervised learning, and / or reinforcement learning techniques) can be used to train the model so that a user in the first physical environment can view the reprojected content or image from a camera in the physical environment from the user's viewpoint, as if the user could see the second physical environment through an obstacle (e.g., a wall) (e.g., via a portal). Training data may include images and contextual information.

[0067] In some implementations, both synthetic and non-synthetic data can be used to train the content reprojection module 304.

[0068] In some implementations, the content reprojection module 304 may be deployed as one or more generative models, wherein content is automatically generated by one or more computers in response to a request to generate content. The automatically generated content may optionally be generated on the device (e.g., at least partially generated by the computer system that received the request to generate content) and / or generated off-device (e.g., at least partially generated by one or more nearby computers available via a local network or one or more computers available via the Internet). This automatically generated content may optionally include visual content (e.g., images, graphics, and / or video) and / or text content.

[0069] In some implementations, novel, automatically generated content is referred to as generative content (e.g., generative images, generative graphics, generative videos, and / or generative text). Generative content is typically generated based on cue input 310 to the content reprojection module 304. The input module 302 may optionally include one or more preprocessing steps (e.g., adjusting user-provided cue, creating system-generated cue, and / or selecting an AI model) to adjust the input before it is used by an AI model to generate an output image. The display module 306 may optionally include one or more postprocessing steps to adjust the output 312 of the content reprojection module 304 (e.g., passing AI model output to different AI models, scaling up, cropping, formatting, and / or adding or removing metadata) before it is used for other purposes (such as being provided to different software processes for further processing or (e.g., visually) presented to a user).

[0070] The cue input 310 used to generate generative content may include one or more of the following: one or more words (e.g., written or spoken natural language cues), one or more images, one or more pictures, and / or one or more videos. The generative pre-trained transformer model is a type of LLM that can efficiently generate novel generative content based on the cue input 310. In some embodiments, the content reprojection module 304 uses the cue input 310, which includes text, to generate different generative text, generative audio content, and / or generative visual content. In other embodiments, the content reprojection module 304 uses the cue input 310, which includes visual content and / or audio content, to generate generative text (e.g., transcriptions of audio and / or descriptions of visual content). In still other embodiments, the content reprojection module 304 uses the cue input 310, which includes multiple types of content (e.g., text, images, audio, video, and / or other sensor data), to generate generative content. The cue input 310 may sometimes also include values ​​for one or more parameters indicating the importance of various parts of the cue. Some of the prompts 310 include a structured set of instructions that can be provided to the content reprojection module 304, which includes wording, specified style, relevant context (e.g., starting content and / or one or more examples) and / or roles for the content reprojection module 304.

[0071] Generative content is typically based on a cue, but is not deterministically selected from pre-generated content; rather, it is generated using the cue as a starting point. In some implementations, pre-existing content (e.g., audio, text, and / or visual content) is used as part of the cue for creating the generative content (e.g., pre-existing content is used as a starting point for creating the generative content). For example, cue input 310 may request modifications to visual content to include or exclude content specified by the cue (e.g., removing identified features from the visual content, adding features described in the cue to the visual content, changing the visual style of the visual content, and / or creating additional visual elements based on the visual content beyond its spatial or temporal boundaries). In some implementations, a random or pseudo-random seed is used as part of the cue input 310 for creating the generative content (e.g., random or pseudo-random seed content is used as a starting point for creating the generative content). For example, when generating an image from a diffusion model, random noise patterns are iteratively denoised based on cue input 310 to generate the image. While a particular type of content reprojection module 304 has been described herein, it should be understood that a variety of different content reprojection modules can be used to generate generative content based on cue.

[0072] In cases where the content reprojection module 304 is a non-learning system, the non-learning-based content reprojection module 304 can use a predefined set of rules or a predefined structure to make decisions based on the input seen in the process. For example, the content reprojection module 304 can be used to guide a deterministic method to create a virtual opening using predefined rules (e.g., geometric rules, calibrated camera parameters, etc.) for capturing, transforming, and displaying content. For example, tracking data associated with the user 203's gaze / head positioning and orientation relative to the wall in the first room can be used to establish the user's virtual viewing position. Similarly, tracking data can be used to calculate the corresponding virtual viewpoint (of the user) in the second adjacent room, as if the user were looking through the wall. Thus, the user's positioning and orientation in the first room can be deterministically mapped to the viewpoint in the second adjacent room using fixed geometric relationships (e.g., offset, rotation matrices). Subsequently, reprojection (e.g., using a predefined model, such as a perspective projection model) is performed from the video feed from the image sensor (in the second adjacent room) determined to be closest to the user's virtual viewpoint to match the virtual viewpoint.

[0073] Some implementations described herein may include the use of learning-based and / or non-learning-based processes. This use may include collecting, preprocessing, encoding, labeling, organizing, analyzing, recommending, and / or generating data. Entities that collect, share, and / or otherwise utilize user data should provide transparency and / or obtain user consent when collecting such data. This disclosure recognizes that the use of data in rule-based or machine learning (ML) processes can be used to benefit users. For example, the data can be used to train models that can be deployed to improve the performance, accuracy, and / or functionality of applications and / or services. Therefore, the use of data enables rule-based processes or ML to adapt and / or optimize operations to provide a more personalized, efficient, and / or enhanced user experience. Such adaptation and / or optimization may include customizing content, recommendations, and / or interactions for individual users, as well as simplifying processes and / or implementing more intuitive interfaces. This disclosure also envisions the ability to further beneficially use data in rule-based processes.

[0074] This disclosure envisions that, in some embodiments, data used by rule-based processes may be used to include publicly available data. To protect user privacy, data may be anonymized, aggregated, and / or otherwise processed to remove or, where possible, limit any individual identification. As discussed herein, entities that collect, share, and / or otherwise utilize such data should obtain user consent before collecting such data and / or provide transparency in the collection process. Furthermore, this disclosure envisions that entities responsible for the use of data (including, but not limited to, data used in connection with rule-based processes) should strive to comply with robust privacy policies and / or privacy measures.

[0075] For example, such entities may implement and consistently follow strategies and measures deemed to meet or exceed industry standards and regulatory requirements for developing and / or training ML-based processes. In doing so, efforts should be made to ensure that all intellectual property and privacy considerations are maintained. Training should include measures to protect training data (such as personal information) by providing adequate safeguards against misuse or exploitation. Such strategies and measures should cover all phases of process development, training, and use, including data collection, data preparation, model training, model evaluation, model deployment, and ongoing monitoring and maintenance. Transparency and measurability should always be maintained. Such policies should be readily accessible to users and should be updated as data collection and / or use change. User data should be collected for the entity's lawful and reasonable use and not shared or sold outside of these lawful uses. Furthermore, such collection and sharing should be conducted in a transparent manner to users and / or with their informed consent. Additionally, such entities should consider taking any necessary steps to defend and safeguard access to such data and ensure that others with access to the data comply with their privacy policies and processes. Furthermore, such entities should be subject to appropriate third-party assessments for transparency purposes to demonstrate their compliance with widely accepted privacy policies and practices. In addition, policies and / or practices should be appropriate for the specific types of data collected and / or accessed, and tailored to specific use cases and applicable laws and standards, including jurisdiction-specific considerations.

[0076] In some implementations, the ML-based process may utilize a model that can be trained (e.g., in a supervised or unsupervised learning manner) using a variety of training data, including data collected using the user's device. This use of the user-collected data may be limited to operations on the user's device. For example, model training may be performed locally on the user's device, so that no part of the data is transmitted to another device. In other implementations, model training may be performed using one or more other devices besides the user's device (e.g., a server), but in a privacy-preserving manner, such as through multi-party computation, encrypted by secretly sharing data, or other means, ensuring that user data is not leaked to these other devices.

[0077] In some implementations, the trained model can be centrally stored on the user device or on multiple devices, as is the case in federated learning. This distributed storage can also be done in a privacy-preserving manner, for example, through encryption, where each piece of data is fragmented so that the data cannot be reassembled or used by a device alone (i.e., only in conjunction with another device) or only by the user device. In this way, user or device behavior patterns are not leaked, while leveraging the increased computing resources of other devices to train and execute the ML model. Therefore, user-collected data can be protected. In some specific implementations, data from multiple devices can be combined in a privacy-preserving manner to train the ML model.

[0078] In some implementations, this disclosure envisions that data used for the content reprojection module 304 may be kept strictly separate from the platform on which the content reprojection module 304 is deployed and / or used for user interaction and / or data processing. In such implementations, data used for offline training of the content reprojection module 304 may be kept in a secure data repository with restricted access and / or not retained for longer than necessary for the training purpose. In some implementations, the content reprojection module 304 may utilize a local memory cache to temporarily store data during a user session. The local memory cache can be used to improve the performance of rule-based processes. However, to protect user privacy, data stored in the local memory cache may be erased after the user session ends. Any temporary cached data used for online learning or inference may be quickly erased after processing. All data collection, transmission, and / or storage should use industry-standard encryption and / or secure communication.

[0079] In some implementations, as noted above, techniques such as federated learning, differential privacy, secure hardware components, homomorphic encryption and / or multi-party computation, and others, can be used to further protect personal information data during rule-based ML training and / or use. The content reprojection module 304 should be monitored for changes in the underlying data distribution, such as concept drift or data skew that may degrade the performance of the content reprojection module 304 over time.

[0080] In some implementations, a combination of offline and online training is used to train the content reprojection module 304. Offline training can use a carefully selected dataset to establish baseline model performance, while online training allows the content reprojection module 304 to be continuously adapted and / or improved. This disclosure recognizes the importance of maintaining strict data management measures throughout this process to ensure user privacy is protected.

[0081] In some implementations, the content reprojection module 304 process may be designed with safeguards to maintain adherence to its original intended purpose, even when the content reprojection module 304 is adapted based on new data. Any significant changes to the data collection and / or application used by the content reprojection module 304 may (and in some cases should) be transparently communicated to affected stakeholders, and / or user consent may be obtained when the manner of collecting and / or utilizing user data changes.

[0082] Regardless of the foregoing, this disclosure also contemplates implementation schemes for users to selectively restrict and / or block the use and / or access to data. That is, this disclosure contemplates that hardware and / or software elements may be provided to prevent or block access to data. For example, with respect to some services, the inventive technology should be configured to allow users to opt-in or opt-out to participate in data collection at any time during or after service registration. In another example, the inventive technology should be configured to allow users to opt out of providing certain data for training the content reprojection module 304 and / or for use as input during the inference phase of such systems. In yet another example, the inventive technology should be configured to allow users to choose to limit the length of time their data is maintained or to completely prohibit their data from being used by rule-based processes. In addition to providing "opt-in" and "opt-out" options, this disclosure also contemplates providing notifications related to access to or use of personal information. For example, users may be notified when their data is input into the content reprojection module 304 for training or inference purposes, and / or alerted when the content reprojection module 304 generates output or makes decisions based on its data.

[0083] This disclosure recognizes that the content reprojection module 304 should incorporate explicit limitations and / or oversight to mitigate risks that may still exist even when such systems have been designed, developed, and / or operated in accordance with industry best practices and standards. For example, outputs may be generated that could be considered erroneous, harmful, offensive, and / or biased; such outputs may not necessarily reflect the opinions or positions of the entities that developed or deployed these systems. Furthermore, in some cases, references to or non-references to third-party products and / or services in these outputs should not be construed as an endorsement or association by the entity providing the rules-based process with the third-party products and / or services. The generated content may be filtered to remove potentially inappropriate or dangerous material before being presented to users, while maintaining the ability for human oversight and / or to cover or correct erroneous or undesirable outputs as a safeguard.

[0084] This disclosure further envisions that users of the Content Reprojection Module 304 should avoid using the Service in any way that infringes upon, misappropriates, or violates the rights of any party. Furthermore, the Content Reprojection Module 304 should not be used for any illegal or unlawful activities, nor should it be used to develop any application or use case that will commit or assist in the committing of a crime or other tortious, illegal, or unlawful act, including misinformation, false information, false statements (e.g., deepfakes), deception, impersonation, and propaganda. The Content Reprojection Module 304 should not violate, misappropriate, or infringe upon any party's copyright, trademark, privacy and portrait rights, trade secrets, patents, or other proprietary or statutory rights, and should appropriately attribute them to the content as needed. Additionally, the Content Reprojection Module 304 should not interfere with any security, digital signature, digital rights management, content protection, verification, or authentication mechanisms. The Content Reprojection Module 304 should not incorrectly represent machine-generated output as human-generated.

[0085] Figure 4 This is a flowchart representation of an exemplary method 400 according to some specific implementations, which enables a user at a first location to view, from the user's viewpoint, the reprojected content or image from a camera at a second location, as if the user could see the second location through an obstacle. In some implementations, method 400 is performed by a device, such as a mobile device, desktop computer, laptop computer, HMD, or server device. In some implementations, the device has a screen for displaying images and / or a screen for viewing stereoscopic images, such as a head-mounted display (HMD, such as device 105 of Figure 1, for example). In some implementations, method 400 is performed by processing logic components, including hardware, firmware, software, or combinations thereof. In some implementations, method 400 is performed by a processor executing code stored in a non-transitory computer-readable medium (e.g., memory). Each block in method 400 can be implemented and performed in any order.

[0086] At box 402, method 400 presents a first view from a first viewpoint of a device (such as an HMD) within a first location in the environment. The first view includes obstacles that prevent the first view from including a portion of a second location within the environment (e.g., different from and adjacent to the first location). For example, a user 203 of wearable device 211 may be located within physical environment 214, and a wall 215c may create an obstacle relative to the view (by user 203) of the contents within physical environment 219, such as... Figure 2A exemplified.

[0087] At box 404, method 400 detects a triggering condition associated with viewing the second position. For example, a gaze activation triggering mechanism may be enabled to activate a virtual opening by using the gaze vector 206 (of user 203) and a virtual activation point 229 located at a specified position in the XR environment 227 (e.g., by user 203), as relative to... Figure 2A As described.

[0088] In some implementations, the triggering conditions associated with detecting and viewing a second location include determining that the user is looking at the second location.

[0089] In some implementations, the triggering conditions associated with detecting and viewing a second location include detecting that a user moves the virtual portal structure to a viewing location in the XR environment (e.g., on a wall).

[0090] In some implementations, the triggering conditions associated with detecting and viewing the second location include detecting user gestures pointing to the second location, such as gaze and / or hand or finger gestures.

[0091] In some implementations, the triggering conditions associated with detecting and viewing the second location can be detected via sensor data obtained from the sensor.

[0092] At box 406, method 400 (based on the detection of a trigger condition) presents a second view of the environment. This portion of the second location, captured from a second viewpoint within the second location, can be included in the second view based on image data from an image sensor. For example, a virtual opening 207 (e.g., a portal) can be activated at a first positioning 217a on wall 215c and aligned with the user 203's perspective to provide a view of the physical environment 219, such as relative to... Figure 2B As described. In some embodiments, a second view of the environment may include a first view of the environment and a portion of a second location. In some embodiments, this portion of the second location may be reprojected onto the user's viewpoint along the gaze direction.

[0093] In some specific implementations, method 400 determines alignment criteria based on the orientation of an image sensor capable of capturing image data from a second viewpoint within a second position. For example, the angular difference between the orientation of device 105 or 110 and the camera capture orientation may be smaller than that relative to... Figure 2A The described threshold angle difference. In some specific implementations, the second view of the presented environment may be further based on alignment criteria.

[0094] In some implementations, alignment criteria are further determined based on the orientation of the image sensor and the orientation of the electronic device. In some implementations, the orientation of the image sensor can be aligned with the orientation of the electronic device by moving the position of the image sensor. In some implementations, the position of the image sensor can be moved using electromechanical devices (such as, for example, servo systems, robotic devices, etc.). In some implementations, the angle between the orientation of the image sensor and the orientation of the electronic device may be less than a threshold angle.

[0095] In some implementations, the transformation between an image captured by an external camera and the user's viewpoint along the direction of gaze can be calculated. This transformation can extend from the position and orientation of the external device to the position of the user device and the direction of the user's gaze. The transformation can be calculated in response to determining that alignment criteria are met.

[0096] In some implementations, presenting a second view of the XR environment may include replacing a portion of the obstacle with content obtained from an image sensor within the second location.

[0097] In some implementations, the content obtained from the image sensor in the second location may include content that provides a changed second view relative to the first viewpoint.

[0098] In some implementations, presenting a second view may include providing a virtual opening within an obstacle through which a second location can be viewed.

[0099] In some specific implementations, a virtual opening can be provided instantaneously in response to triggering conditions and by determining that alignment criteria are met.

[0100] In some implementations, a virtual opening (e.g., a portal opening slowly) can be provided within a specified time frame in response to a triggering condition (e.g., a gaze) and by determining that alignment criteria are met.

[0101] Figure 5This is a block diagram of example device 500. Device 500 illustrates an exemplary device configuration for electronic devices 105 and 110 of FIG1. ​​Although certain specific features are illustrated, those skilled in the art will understand from this disclosure that various other features are not illustrated for the sake of brevity and to avoid obscuring more relevant aspects of the specific embodiments disclosed herein. Therefore, as a non-limiting example, in some specific implementations, device 500 includes one or more processing units 502 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, etc.), one or more input / output (I / O) devices and sensors 506, one or more communication interfaces 508 (e.g., USB, FireWire, Thunderbolt, IEEE 802.3x, IEEE 802.11x, IEEE 802.14x, GSM, CDMA, TDMA, GPS, IR, Bluetooth, ZIGBEE, SPI, I2C and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 510, output devices (e.g., one or more displays) 512, one or more internal and / or external image sensor systems 514, memory 520, and one or more communication buses 504 for interconnecting these components and various other components.

[0102] In some embodiments, one or more communication buses 504 include circuitry that interconnects system components and controls communication between system components. In some embodiments, one or more I / O devices and sensors 506 include at least one of the following: an inertial measurement unit (IMU), an accelerometer, a magnetometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., a blood pressure monitor, a heart rate monitor, a blood oxygen sensor, a blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptic engine, one or more depth sensors (e.g., structured light, time-of-flight, etc.), one or more cameras (e.g., an inward-facing camera and an outward-facing camera of an HMD), one or more infrared sensors, one or more thermal sensors, etc.

[0103] In some embodiments, one or more displays 512 are configured to present a view of a physical environment, a graphical environment, an extended reality environment, etc., to a user. In some embodiments, one or more displays 512 are configured to present content to a user (determined based on the user's determined user / object location within the physical environment). In some embodiments, one or more displays 512 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light-emitting field-effect transistor (OLET), organic light-emitting diode (OLED), surface-conducting electron emission display (SED), field emission display (FED), quantum dot light-emitting diode (QD-LED), microelectromechanical systems (MEMS), and / or similar display types. In some embodiments, one or more displays 512 correspond to waveguide displays such as diffraction, reflection, polarization, and holography. In one example, device 500 includes a single display. In another example, device 500 includes displays for each of the user's eyes.

[0104] In some embodiments, one or more image sensor systems 514 are configured to acquire image data corresponding to at least a portion of the physical environment 100. For example, one or more image sensor systems 514 include one or more RGB cameras, monochrome cameras, IR cameras, depth cameras, event-based cameras, etc. (e.g., having a complementary metal-oxide-semiconductor (CMOS) image sensor or a charge-coupled device (CCD) image sensor). In various embodiments, one or more image sensor systems 514 also include an illumination source emitting light, such as a flash. In various embodiments, one or more image sensor systems 514 also include an on-camera image signal processor (ISP) configured to perform multiple processing operations on the image data.

[0105] In some embodiments, sensor data may be acquired by devices (e.g., devices 105 and 110 of Figure 1) during a scan of a room in a physical environment. The sensor data may include a 3D point cloud and a sequence of 2D images corresponding to views of the room captured during the scan. In some embodiments, the sensor data includes image data (e.g., from an RGB camera), depth data (e.g., depth images from a depth camera), ambient light sensor data (e.g., from an ambient light sensor), and / or motion data from one or more motion sensors (e.g., accelerometers, gyroscopes, IMUs, etc.). In some embodiments, the sensor data includes visual inertial odometry (VIO) data determined based on the image data. The 3D point cloud can provide semantic information about one or more elements of the room. The 3D point cloud can provide information about the location and appearance of surface portions within the physical environment. In some embodiments, the 3D point cloud is acquired over time (e.g., during a scan of the room) and can be updated, with updated versions of the 3D point cloud obtained over time. For example, when the 3D representation is updated / adjusted over time (e.g., when a user scans a room), the 3D representation can be obtained (and analyzed / processed).

[0106] In some embodiments, sensor data may be positioning information, and some embodiments include VIO (Vehicle Identification and Odometry) to determine equivalent odometry information to estimate travel distance using sequential camera images (e.g., light intensity image data) and motion data (e.g., acquired from an IMU / motion sensor). Alternatively, some embodiments of this disclosure may include a Simultaneous Localization and Mapping (SLAM) system (e.g., a positioning sensor). This SLAM system may include a GPS-independent, multi-dimensional (e.g., 3D) laser scanning and range measurement system that provides real-time simultaneous localization and mapping. This SLAM system can generate and manage highly accurate point cloud data produced by reflections from laser scans of objects in the environment. Accurately tracking the movement of any points in the point cloud over time allows the SLAM system to use points in the point cloud as reference points for its location, maintaining an accurate understanding of its position and orientation as it travels through the environment.

[0107] In some embodiments, device 500 includes an eye-tracking system for detecting eye positioning and eye movement (e.g., eye gaze detection). For example, the eye-tracking system may include one or more infrared (IR) light-emitting diodes (LEDs), an eye-tracking camera (e.g., a near-infrared (NIR) camera), and an illumination source (e.g., an NIR light source) that emits light (e.g., NIR light) towards the user's eyes. Furthermore, the illumination source of device 500 may emit NIR light to illuminate the user's eyes, and the NIR camera may capture images of the user's eyes. In some embodiments, the images captured by the eye-tracking system may be analyzed to detect the positioning and movement of the user's eyes, or to detect other information about the eyes such as pupil dilation or pupil diameter. Furthermore, the gaze point estimated from the eye-tracking images enables gaze-based interaction with content displayed on a near-eye display of device 500.

[0108] Memory 520 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices. In some embodiments, memory 520 includes non-volatile memory, such as one or more disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 520 may optionally include one or more storage devices remotely located to one or more processing units 502. Memory 520 includes a non-transitory computer-readable storage medium.

[0109] In some embodiments, memory 520 or a non-transitory computer-readable storage medium of memory 520 stores an optional operating system 530 and one or more instruction sets 540. Operating system 530 includes procedures for handling various basic system services and for performing hardware-related tasks. In some embodiments, instruction set 540 includes executable software defined by binary information stored in charge. In some embodiments, instruction set 540 is software executable by one or more processing units 502 to implement one or more of the techniques described herein.

[0110] Instruction set 540 includes trigger detection instruction set 542 and virtual opening rendering set 544. Instruction set 540 can be represented as a single software executable file or multiple software executable files.

[0111] The trigger detection instruction set 542 is configured with instructions that can be executed by the processor to detect trigger conditions associated with viewing another location obscured by an obstacle. For example, trigger conditions may include determining that a user is looking at another location, detecting that the user has dragged the portal to a viewing location (on a wall) in the XR environment, etc.

[0112] The Virtual Opening Presentation Instruction Set 544 is configured with instructions that can be executed by the processor to provide an opening or portal in an obstacle (e.g., a wall) through which other locations can be viewed.

[0113] Although instruction set 540 is shown as residing on a single device, it should be understood that in other specific implementations, any combination of elements may reside on separate computing devices. Furthermore, Figure 5 This is intended more as a functional description of various features present in a particular specific implementation than as a structural diagram of the specific implementation described herein. As will be appreciated by those skilled in the art, the items shown individually can be combined, and some items can be separated. The actual number of instruction sets and how features are allocated therein will vary depending on the specific implementation and may depend in part on the specific combination of hardware, software, and / or firmware chosen for that particular implementation.

[0114] Those skilled in the art will understand that well-known systems, methods, components, devices, and circuits have not been described exhaustively so as not to obscure more relevant aspects of the specific embodiments of the examples described herein. Furthermore, other effective aspects and / or variations do not include all the details in the specific details described herein. Therefore, several details are described to provide a thorough understanding of the exemplary aspects illustrated in the accompanying drawings. Moreover, the drawings only illustrate some exemplary embodiments of this disclosure and should not be considered limiting.

[0115] While this specification contains numerous specific implementation details, these details should not be construed as limiting the scope of any invention or potentially claimed content, but rather as descriptions of features specific to particular embodiments of a particular invention. Certain features described in the context of different embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, while certain features may be described above as functioning in certain combinations and even initially claimed in this manner, one or more features of a claimed combination may be removed from that combination in some cases, and the claimed combination may involve sub-combinations or variations thereof.

[0116] Similarly, although the operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring such operations to be performed in a sequential order or the specific order shown, or requiring all illustrated operations to achieve the desired result. In some situations, multitasking and parallel processing may be advantageous. Furthermore, the partitioning of the various system components in the above embodiments should not be construed as requiring such partitioning in all embodiments, and it should be understood that the described program components and systems may generally be integrated together in a single software product or packaged into multiple software products.

[0117] Therefore, specific embodiments of the subject matter have been described. Other embodiments are also within the scope of the following claims. In some cases, the actions described in the claims can be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific order or sequence shown to achieve the desired result. In some embodiments, multitasking and parallel processing may be advantageous.

[0118] The embodiments of the subject matter and operation described in this specification may be implemented in digital electronic circuits or in computer software, firmware, or hardware (including the structures disclosed in this specification and their structural equivalents) or in a combination thereof. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, such as one or more modules of computer program instructions encoded on a computer storage medium for execution by or control of the operation of a data processing device. Alternatively or additionally, the program instructions may be encoded on artificially generated propagating signals, such as machine-generated electrical, optical, or electromagnetic signals generated to encode information for transmission to a suitable receiver device for execution by the data processing device. The computer storage medium may be or be included in a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination thereof. Furthermore, although the computer storage medium is not a propagating signal, it may be a source or destination of computer program instructions encoded in artificially generated propagating signals. The computer storage medium may also be or be included in one or more separate physical components or media (e.g., multiple CDs, disks, or other storage devices).

[0119] The term "data processing apparatus" encompasses all kinds of devices, apparatuses, and machines for processing data, including programmable processors, computers, systems-on-a-chip, or many or combinations of the foregoing. The apparatus may include special-purpose logic circuitry (e.g., FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits)). In addition to hardware, the apparatus may include code that creates an execution environment for the computer program under consideration, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, cross-platform runtime environments, virtual machines, or combinations thereof. The apparatus and execution environment can implement a variety of different computing model infrastructures, such as web services, distributed computing, and grid computing infrastructures. Unless otherwise specifically stated, it should be understood that throughout this specification, discussions using terms such as "processing," "computing," "calculating," "determining," and "identifying" refer to the actions or processes of computing devices, such as one or more computers or similar electronic computing devices, that manipulate or convert data represented as physical electronic or magnetic quantities within the memory, registers, or other information storage devices, transmission devices, or display devices of a computing platform.

[0120] The one or more systems discussed herein are not limited to any particular hardware architecture or configuration. A computing device may include any suitable arrangement of components that provide results conditioned on one or more inputs. Suitable computing devices include computer systems based on multi-purpose microprocessors that access stored software that programs or configures the computing system from a general-purpose computing device to a special-purpose computing device that implements one or more specific implementations of this subject. The teachings contained herein may be implemented in the software used for programming or configuring the computing device using any suitable programming, scripting, or other type of language or combination of languages.

[0121] Specific implementations of the methods disclosed herein can be performed in the operation of such computing devices. The order of the boxes presented in the examples above can be varied; for example, the boxes can be reordered, combined, and / or divided into sub-blocks. Some boxes or processes can be executed in parallel. The operations described in this specification can be implemented as operations performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.

[0122] The use of "applies to" or "configured to" in this document implies open and inclusive language, which does not exclude applicability to or configuration for performing additional tasks or steps. Additionally, the use of "based on" implies openness and inclusivity, as processes, steps, calculations, or other actions "based on" one or more of the stated conditions or values ​​may in practice be based on additional conditions or values ​​beyond those stated. The headings, lists, and numbering included herein are for illustrative purposes only and are not intended to be restrictive.

[0123] It will also be understood that while terms such as "first," "second," etc., may be used in this document to describe various elements, these elements should not be limited by these terms. These terms are merely used to distinguish one element from another. For example, a first node may be called a second node, and similarly, a second node may be called a first node, changing the meaning of the description, provided that all occurrences of "first node" are consistently renamed and all occurrences of "second node" are consistently renamed. First nodes and second nodes are both nodes, but they are not the same node.

[0124] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the claims. As used in the description of these embodiments and in the appended claims, the singular forms “a,” “an,” and “the” are intended to also cover the plural forms unless the context clearly indicates otherwise. It will also be understood that the term “and / or” as used herein refers to and covers any and all possible combinations of one or more of the associated listed items. It will be further understood that the term “comprising,” when used in this specification, specifies the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0125] As used herein, the term "if" can be interpreted as meaning "when the prerequisite is true" or "when the prerequisite is true" or "in response to determination" or "according to determination" or "in response to detection" that the prerequisite is true, depending on the context. Similarly, the phrases "if it is determined [the prerequisite is true]" or "if [the prerequisite is true]" or "when [the prerequisite is true]" can be interpreted as meaning "when it is determined that the prerequisite is true" or "in response to determination" or "according to determination" that the prerequisite is true or "when it is detected that the prerequisite is true" or "in response to detection" that the prerequisite is true, depending on the context.

Claims

1. A method, the method comprising: In electronic devices that have a processor and one or more displays: A first view is presented from a first viewpoint within a first location in the environment via the one or more displays, wherein the first view includes an obstacle that prevents the first view from including a portion of a second location within the environment, the second location being different from the first location; Detect and view the triggering conditions associated with the second location; as well as Based on the detection of the triggering condition, a second view of the environment is presented, wherein the portion of the second location is included in the second view based on image data from an image sensor, wherein the image sensor captures the image data from a second viewpoint within the second location.

2. The method of claim 1, wherein detecting the triggering condition associated with viewing the second location includes determining that the user is looking at the second location.

3. The method of claim 1, wherein detecting the triggering condition associated with viewing the second location includes detecting that the user moves the virtual portal structure to a viewing location in the XR environment.

4. The method of claim 1, wherein detecting the triggering condition associated with viewing the second location includes detecting a user gesture.

5. The method of claim 1, wherein detecting the triggering condition associated with viewing the second location includes detecting that the gaze vector intersects with a virtual activation point at the viewing location in the XR environment.

6. The method according to claim 1, further comprising: The alignment criteria are determined based on the orientation of the image sensor that captures the image data from the second viewpoint within the second position, wherein the second view presenting the environment is further based on the alignment criteria.

7. The method of claim 6, wherein determining that the alignment criteria are met is further based on the alignment of the image sensor orientation with the orientation of the electronic device.

8. The method of claim 7, wherein the orientation of the image sensor is aligned with the orientation of the electronic device by moving the positioning of the image sensor.

9. The method of claim 8, wherein the positioning of the image sensor is moved by using electromechanical equipment.

10. The method of claim 6, wherein the angle between the orientation of the image sensor and the orientation of the electronic device is less than a threshold angle.

11. The method of claim 1, wherein presenting the second view of the environment comprises replacing a portion of the obstacle with content obtained from the image sensor within the second location.

12. The method of claim 11, wherein the content obtained from the image sensor within the second location includes content providing a change in the second view relative to the first viewpoint.

13. The method of claim 1, wherein presenting the second view includes providing a virtual opening within the obstacle, through which the second location is viewed.

14. The method of claim 13, wherein the virtual opening is provided instantaneously in response to the triggering condition and the determination that the alignment criteria are met.

15. The method of claim 13, wherein the virtual opening is provided within a specified time range in response to the triggering condition and the determination that the alignment criteria are met.

16. A non-transitory computer-readable medium comprising instructions that, when executed by a processor, cause the processor to perform operations, the operations including: A first view is presented from a first viewpoint within a first location in an environment via one or more displays, wherein the first view includes an obstacle that prevents the first view from including a portion of a second location within the environment, the second location being different from the first location; Detect and view the triggering conditions associated with the second location; as well as Based on the detection of the triggering condition, a second view of the environment is presented, wherein the portion of the second location is included in the second view based on image data from an image sensor, wherein the image sensor captures image data from a second viewpoint within the second location.

17. An electronic device, the electronic device comprising: One or more displays; Non-transitory computer-readable storage medium; and One or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium includes program instructions that, when executed on the one or more processors, cause the electronic device to perform operations including: A first view is presented from a first viewpoint within a first location in the environment via the one or more displays, wherein the first view includes an obstacle that prevents the first view from including a portion of a second location within the environment, the second location being different from the first location; Detect and view the triggering conditions associated with the second location; as well as Based on the detection of the triggering condition, a second view of the environment is presented, wherein the portion of the second location is included in the second view based on image data from an image sensor, wherein the image sensor captures image data from a second viewpoint within the second location.

18. The electronic device of claim 17, wherein detecting the triggering condition associated with viewing the second location includes determining that the user is looking at the second location.

19. The electronic device of claim 17, wherein detecting the triggering condition associated with viewing the second location includes detecting that the user moves the virtual portal structure to a viewing location in the XR environment.

20. The electronic device of claim 17, wherein detecting the triggering condition associated with viewing the second location includes detecting a user gesture.

Images