Dynamic entry

Abstract

The invention relates to a dynamic entry. In one embodiment, a method of displaying a portal is performed by a device including a display, one or more processors, and a non-transitory memory. The method includes obtaining a user location in a first three-dimensional coordinate system of a first environment. The method includes obtaining a location of interest in a second three-dimensional second coordinate system of a second environment. The method includes transforming a location of interest to a transformed location of interest in a first three-dimensional coordinate system. The method includes determining an entry location in the first three-dimensional coordinate system based on the user location and the transformed location of interest. The method includes displaying, on the display, an entry corresponding to at least a partial view of the second environment at an entry location.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 544,586, filed on October 17, 2023, which is hereby incorporated by reference in its entirety. Technical Field

[0003] The present disclosure generally relates to systems, methods, and apparatus for automatically moving portals in an extended reality (XR) environment. Background Art

[0004] In various implementations, an extended reality (XR) environment presented by an electronic device includes a portal that displays a portion of a virtual environment. When a user moves from a first location to a second location, a view through the portal changes the portion of the virtual environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] So that the present disclosure can be understood by those of ordinary skill in the art, a more detailed description may be obtained with reference to aspects of some exemplary implementations, some of which are illustrated in the accompanying drawings.

[0006] Figure 1 is a block diagram of an exemplary operating environment according to some implementations.

[0007] FIG. 2A to FIG. 2H An XR environment is shown during various time periods according to some implementations.

[0008] Figure 3 is a flowchart representation of a method of displaying an entry according to some specific implementations.

[0009] Figure 4 is a block diagram of an example controller according to some implementations.

[0010] Figure 5 is a block diagram of an exemplary electronic device according to some implementations.

[0011] As is common practice, the various features shown in the drawings may not be drawn to scale. Therefore, for clarity, the sizes of the various features may be arbitrarily expanded or reduced. In addition, some drawings may not depict all components of a given system, method, or device. Finally, throughout the specification and drawings, similar reference numerals may be used to represent similar features. Summary of the invention

[0012] Various embodiments disclosed herein include devices, systems, and methods for displaying a portal. In various embodiments, the method is performed by a device including a display, one or more processors, and a non-volatile memory. The method includes obtaining a user position in a first three-dimensional coordinate system of a first environment. The method includes obtaining a location of interest in a second three-dimensional second coordinate system of a second environment. The method includes transforming the location of interest into a transformed location of interest in the first three-dimensional coordinate system. The method includes determining a portal location in the first three-dimensional coordinate system based on the user position and the transformed location of interest. The method includes displaying a portal corresponding to at least a partial view of the second environment at the portal location on the display.

[0013] According to some specific implementations, a device includes one or more processors, non-volatile memory, and one or more programs; the one or more programs are stored in the non-volatile memory and are configured to be executed by one or more processors, and the one or more programs include instructions for performing or causing the execution of any of the methods described herein. According to some specific implementations, a non-volatile computer-readable storage medium stores instructions that, when executed by one or more processors of the device, cause the device to perform or cause the execution of any of the methods described herein. According to some specific implementations, a device includes: one or more processors, non-volatile memory, and means for performing or causing the execution of any of the methods described herein. DETAILED DESCRIPTION

[0014] Many details are described in order to provide a thorough understanding of the example implementations shown in the accompanying drawings. However, the accompanying drawings only illustrate some example aspects of the present disclosure and should not be considered limiting. One of ordinary skill in the art will appreciate that other effective aspects and / or variations do not include all of the specific details described herein. In addition, well-known systems, methods, components, devices, and circuits are not described in detail so as not to obscure more relevant aspects of the example implementations described herein.

[0015] As described above, in various embodiments, as the user moves in the XR environment, the view of the virtual world through the portal within the XR environment changes. Therefore, as the user moves in the XR environment, static locations may not always be visible through the portal. Similarly, dynamic objects moving in the virtual world may not always be visible through the portal even if the user is not moving in the XR environment. Therefore, if the user wishes to maintain a view of a static location as the user moves in the XR environment, or to maintain a view of a dynamic object as the dynamic object moves in the virtual world, the user must also move the portal. However, in various embodiments, the portal is automatically moved based on the user's position in the XR environment and / or the static position or position of the dynamic object in the virtual world to maintain a view of the static location or dynamic object.

[0016] Figure 1 1 is a block diagram of an exemplary operating environment 100 according to some implementations. Although relevant features are shown, those of ordinary skill in the art will recognize from this disclosure that various other features are not illustrated for the sake of brevity and in order not to obscure more relevant aspects of the example implementations disclosed herein. To this end, as a non-limiting example, the operating environment 100 includes a controller 110 and an electronic device 120.

[0017] In some implementations, the controller 110 is configured to manage and coordinate the user's XR experience. In some implementations, the controller 110 includes a suitable combination of software, firmware, and / or hardware. Figure 5 Controller 110 is described in more detail. In some implementations, controller 110 is a computing device located locally or remotely relative to physical environment 105. For example, controller 110 is a local server located within physical environment 105. In another example, controller 110 is a remote server (e.g., a cloud server, a central server, etc.) located outside physical environment 105. In some implementations, controller 110 is communicatively coupled to electronic device 120 via one or more wired or wireless communication channels 144 (e.g., Bluetooth, IEEE 802.11x, IEEE802.16x, IEEE 802.3x, etc.). As another example, controller 110 is included in a housing of electronic device 120. In some implementations, the functionality of controller 110 is provided by electronic device 120 and / or is combined with the electronic device.

[0018] In some implementations, the electronic device 120 is configured to provide an XR experience to the user. In some implementations, the electronic device 120 includes a suitable combination of software, firmware, and / or hardware. According to some implementations, the electronic device 120 presents XR content to the user via a display 122 while the user is physically present within a physical environment 105, which includes a table 107 within a field of view 111 of the electronic device 120. In some implementations, the user holds the electronic device 120 in one or both of his / her hands. In some implementations, when providing XR content, the electronic device 120 is configured to display an XR object (e.g., an XR cylinder 109) and implement video pass-through of the physical environment 105 (e.g., including a representation 117 of the table 107) on the display 122. The following description is relative to Figure 5 The electronic device 120 is described in more detail.

[0019] According to some implementations, the electronic device 120 provides an XR experience to the user while the user is virtually and / or physically present within the physical environment 105 .

[0020] In some implementations, the user wears the electronic device 120 on his / her head. For example, in some implementations, the electronic device includes a head-mounted system (HMS), a head-mounted device (HMD), or a head-mounted housing (HME). Therefore, the electronic device 120 includes one or more XR displays configured to display XR content. For example, in various implementations, the electronic device 120 surrounds the user's field of view. In some implementations, the electronic device 120 is a handheld device (such as a smart phone or a tablet computer) configured to present XR content, and the user no longer wears the electronic device 120 but holds the device with the display facing the user's field of view and the camera facing the physical environment 105. In some implementations, the handheld device may be placed in a housing that can be worn on the user's head. In some implementations, the electronic device 120 is replaced with an XR cabin, housing, or room configured to present XR content, in which the user no longer wears or holds the electronic device 120.

[0021] FIG. 2A to FIG. 2H A top view 210 and a perspective view 220 of an XR environment generated by an electronic device are shown. The XR environment is based on the physical environment of a room. The top view 210 is from a top-down perspective, and the perspective view 220 is from a perspective of a user's eye position and is at least partially displayed on a display of the electronic device. For example, the eye position is where the user's eyes are when the device is worn on the head. For example, in various specific implementations, the eye position is a fixed distance vertically away from the display.

[0022] In various implementations, the electronic device includes multiple displays (e.g., a left display positioned in front of a user's left eye and a right display positioned in front of a user's right eye) configured to provide a stereoscopic view of the XR environment. FIG. 2A to FIG. 2H An XR environment is shown presented on a single display among multiple displays.

[0023] Figure 2A middle Figure 2H A top view 210 and a perspective view 220 of an XR environment are shown during a series of time periods. In various implementations, each time period is an instant, a fraction of a second, a few seconds, a few hours, a few days, or any length of time.

[0024] The XR environment is associated with a three-dimensional physical world coordinate system represented by axis 211 in top view 210 and axis 212 in perspective view 220. The XR environment includes a portal 230 that provides a view of the virtual environment. The virtual environment is associated with a three-dimensional virtual world coordinate system that is related to the three-dimensional physical world coordinate system through rotational and translational transformations. As shown in perspective view 220, portal 230 casts a shadow 231 on the floor of the XR environment.

[0025] The virtual environment includes a virtual bird 232 and a virtual bowl 233 that are visible through a portal 230 in the perspective view 220 at different time periods. The virtual bird 232 is associated with a set of dynamic three-dimensional coordinates in a three-dimensional virtual world coordinate system, and the virtual bowl is associated with a set of static three-dimensional coordinates in a three-dimensional world coordinate system. Because the three-dimensional virtual world coordinate system is related to the three-dimensional physical world coordinate system, the virtual bird 232 is also associated with a set of transformed dynamic three-dimensional coordinates in the three-dimensional physical world coordinate system, and the virtual bowl 233 is also associated with a set of transformed static three-dimensional coordinates in the three-dimensional physical world coordinate system.

[0026] Although virtual bird 232 and virtual bowl 233 are not visible in top view 210 , a projection of virtual bird 242 is displayed in top view 210 at the set of transformed dynamic three-dimensional coordinates, and a projection of virtual bowl 243 is displayed in top view 210 at the set of transformed static three-dimensional coordinates.

[0027] The XR environment also includes a user 250, and the user 250 has a field of view 251 and a gaze direction 252, which are not visible but are shown in the top view 210. In addition, the intersection of the field of view 253 intersects (and penetrates) the portal 230.

[0028] Figure 2A A top view 210 and a perspective view 220 of the XR environment during a first time period are shown. During the first time period, a user 250 has a first user position and a first user orientation in the XR environment, and a portal has a first portal position and a first portal orientation in the XR environment.

[0029] During the first time period, the intersection of field of view 253 includes the projection of virtual bird 242, and thus virtual bird 232 is visible through portal 230 in perspective view 220. Conversely, during the first time period, the intersection of field of view 253 does not include the projection of virtual bowl 243, and thus virtual bowl 233 is not visible through portal 230 in perspective view 220.

[0030] Figure 2B The top view 210 and the perspective view 220 of the XR environment are shown during a second time period after the first time period. Between the first time period and the second time period, the user 250 has moved from a first user position and a first user orientation to a second user position and a second user orientation (and neither the portal 230 nor the virtual bird 232 has moved).

[0031] During the second time period, the intersecting portion of field of view 253 intersects the projection of virtual bowl 243, but does not intersect the projection of virtual bird 242. Thus, during the second time period, virtual bowl 233 is visible through portal 230 in perspective view 220, but virtual bird 232 is not visible.

[0032] Figure 2C The top view 210 and the perspective view 220 of the XR environment during a third time period after the second time period are shown. Between the second time period and the third time period, the portal 230 has moved from the first portal position to the second portal position while maintaining the first portal orientation (and neither the user 250 nor the virtual bird 232 has moved). In various implementations, the portal 230 moves in response to user input provided by the user 250.

[0033] During the third time period, the intersecting portion of field of view 253 intersects the projection of virtual bird 242, but does not intersect the projection of virtual bowl 243. Thus, during the third time period, virtual bird 232 is visible through portal 230 in perspective view 220, but virtual bowl 233 is not visible.

[0034] Figure 2D The top view 210 and the perspective view 220 of the XR environment are shown during a fourth time period after the third time period. Between the third time period and the fourth time period, the virtual bird 232 has moved from a first bird position in the virtual environment to a second bird position in the virtual environment (and neither the user 250 nor the portal 230 has moved).

[0035] During the fourth time period, the intersecting portion of field of view 253 does not intersect the projection of virtual bird 242 or the projection of virtual bowl 243. Therefore, during the fourth time period, neither virtual bird 232 nor virtual bowl 233 is visible through portal 230 in perspective view 220.

[0036] Figure 2E The top view 210 and the perspective view 220 of the XR environment during a fifth time period after the fourth time period are shown. Between the fourth time period and the third time period, the portal 230 has moved from the second portal position and the first portal orientation to the third portal position and the second portal orientation (and neither the user 250 nor the virtual bird 232 has moved). In various implementations, the portal 230 moves in response to user input provided by the user 250.

[0037] During the fifth time period, the intersecting portion of field of view 253 intersects both the projection of virtual bird 242 and the projection of virtual bowl 243. Thus, during the fifth time period, both virtual bird 232 and virtual bowl 233 are visible through portal 230 in perspective view 220.

[0038] Therefore, if FIG. 2A to FIG. 2EAs shown, in order to view different virtual objects, the user may move or move portal 230 in the XR environment. Similarly, in order to maintain a view of a static object (such as virtual bowl 233) as the user moves in the XR environment, user 250 moves portal 230. Furthermore, in order to maintain a view of a dynamic object (such as virtual bird 232) even when the user is not moving in the XR environment, user 250 moves portal 230.

[0039] However, in various implementations, the electronic device automatically moves the portal 230 in response to the user moving the XR environment and / or the dynamic object moving in the virtual environment. Specifically, the electronic device moves the portal 230 so as to maintain visibility of a specific location in the virtual environment, which can be a dynamic location (such as the location of a dynamic object) or a static location (such as the location of a static object).

[0040] During the fifth time period, the electronic device locks the portal 230 to the position of the virtual bowl 233. Thus, the electronic device determines the position and orientation of the portal 230 based on the position of the virtual bowl 233 in the virtual environment and the position of the user 250 in the XR environment.

[0041] For example, in various implementations, the electronic device determines a set of user three-dimensional coordinates in the three-dimensional physical world coordinate system for the position of the user 250 in the XR environment. In addition, the electronic device determines a set of static three-dimensional coordinates in the virtual world coordinate system for the position of the virtual bowl 233 in the virtual environment, and transforms the set of static three-dimensional coordinates into a set of transformed static three-dimensional coordinates in the physical world coordinate system using rotation and translation transformations. The electronic device determines the position of the entrance 230 as a set of three-dimensional coordinate entrances in the middle of a line segment between the set of user and the set of transformed static coordinates in the three-dimensional physical world coordinate system, and determines the orientation of the entrance 230 as perpendicular to the line segment.

[0042] Figure 2F The top view 210 and the perspective view 220 of the XR environment are shown during a sixth time period after the fifth time period. Between the sixth time period and the fifth time period, the user 250 has moved from the second user position and the second user orientation back to the first user position and the first user orientation (and the virtual bird 232 has not moved).

[0043] Because the portal 230 is locked to the position of the virtual bowl 233, the electronic device automatically moves the portal 230 from the third portal position and the second portal orientation to the fourth portal position and the third portal orientation in response to the user 250 moving to a different position.

[0044] During the sixth time period, the intersecting portion of field of view 253 intersects both the projection of virtual bird 242 and the projection of virtual bowl 243. Thus, during the sixth time period, both virtual bird 232 and virtual bowl 233 are visible through portal 230 in perspective view 220.

[0045] During the sixth time period, the electronic device locks portal 230 to the position of virtual bird 232. Thus, the electronic device determines the position and orientation of portal 230 based on the position of virtual bird 232 in the virtual environment and the position of user 250 in the XR environment.

[0046] Figure 2G The top view 210 and perspective view 220 of the XR environment are shown during a seventh time period after the sixth time period. Between the seventh time period and the sixth time period, the virtual bird 242 has moved from the second bird position in the virtual environment back to the first bird position (and the user 250 has not moved).

[0047] Because portal 230 is locked to the position of virtual bird 232, in response to virtual bird 232 moving to a different position, the electronic device automatically moves portal 230 from the fourth portal position and third portal orientation back to the first portal position and first portal orientation.

[0048] During the seventh time period, the intersecting portion of field of view 253 intersects the projection of virtual bird 242, but does not intersect the projection of virtual bowl 243. Thus, during the seventh time period, virtual bird 232 is visible through portal 230 in perspective view 220, but virtual bowl 233 is not visible.

[0049] During the seventh time period, the electronic device locks the portal 230 to the position of the gaze point of the user 250, such as a position in the virtual environment that intersects the gaze direction 252. Therefore, the electronic device determines the position and orientation of the portal 230 based on the position of the gaze point of the user 250 in the virtual environment and the position of the user 250 in the XR environment.

[0050] Figure 2H The top view 210 and perspective view 220 of the XR environment are shown during an eighth time period following the seventh time period. Between the eighth time period and the seventh time period, the user 250 looks to the right, changing the gaze direction 252 but without changing the position or orientation (and the virtual bird 232 does not move).

[0051] Because portal 230 is locked to the position of user 250's gaze point, in response to a change in gaze direction 252, the electronic device automatically moves portal 230 from the first portal position and first portal orientation to a fifth portal position and fourth portal orientation.

[0052] During the eighth time period, the intersecting portion of field of view 253 intersects the projection of virtual bowl 243, but does not intersect the projection of virtual bird 242. Thus, during the eighth time period, virtual bowl 233 is visible through portal 230 in perspective view 220, but virtual bird 232 is not visible.

[0053] Figure 3 3 is a flowchart of a method 300 for displaying an entry according to some specific implementations. In various specific implementations, the method 300 is performed by an electronic device (such as Figure 1 The method 300 is performed by an electronic device 120 having a display, one or more processors, and a non-transitory memory. In some implementations, the method 300 is performed by a processing logic component (including hardware, firmware, software, or a combination thereof). In some implementations, the method 300 is performed by a processor executing instructions (e.g., code) stored in a non-transitory computer-readable medium (e.g., memory).

[0054] Method 300 begins at block 310, where a device obtains a user position in a first three-dimensional coordinate system of a first environment. In various implementations, the user position includes a set of user three-dimensional coordinates in the first three-dimensional coordinate system. In various implementations, the first environment is a physical environment including one or more physical objects. In various implementations, the first environment is a virtual environment including one or more virtual objects.

[0055] The method 300 continues in block 320, where the device obtains a location of interest in a second three-dimensional coordinate system of the second environment. In various implementations, the location of interest includes a set of three-dimensional coordinates of interest in the second three-dimensional coordinate system. In various implementations, the second environment is a virtual environment including one or more virtual objects.

[0056] In various implementations, the location of interest is the location of a static object in the second environment. Figure 2F In FIG. 2 , the location of the entrance 230 is based on the location of the static virtual bowl 233. In various implementations, the location of interest is the location of the dynamic object in the second environment. Figure 2G In FIG. 2 , the location of the entrance 230 is based on the location of the dynamic virtual bird 232. In various implementations, the location of interest is the user's gaze point in the second environment. Figure 2H , the location of the entrance 230 is based on the gaze direction 252. In various implementations, the user's gaze point is the location of an object in the second environment that the user is viewing. In various implementations, the gaze point is determined as the intersection of the user's gaze direction transformed from the first three-dimensional coordinate system to the second three-dimensional coordinate system.

[0057] Method 300 continues in box 330, where the device transforms the position of interest (in the second three-dimensional coordinate system) into a transformed position of interest in the first three-dimensional coordinate system. In various specific implementations, the transformed position of interest includes a set of transformed three-dimensional coordinates of interest in the first three-dimensional coordinate system. In various specific implementations, the first three-dimensional coordinate system and the second three-dimensional coordinate system are related by a coordinate system transformation. In various specific implementations, the coordinate system transformation is a rotation and translation transformation. For example, in various specific implementations, the transformation is stored as a set of three angles and three distances. In various specific implementations, the three angles and the three distances are all zero, and the coordinate system transformation is an identity transformation. In various specific implementations, the coordinate system transformation can be changed via user input.

[0058] Method 300 continues in box 340, where the device determines an entry position in a first three-dimensional coordinate system based on the user position and the transformed position of interest. In various specific implementations, the entry position includes a set of entry three-dimensional coordinates in the first three-dimensional coordinate system. In various specific implementations, the entry position is a point of a line segment between the user position and the transformed position of interest. In various specific implementations, the entry position is a point of a line segment at a predetermined distance from the user position. In various specific implementations, the predetermined distance can be changed via user input. In various specific implementations, the entry position is a point of a line segment at a predetermined distance from the transformed position of interest. In various specific implementations, the predetermined distance can be changed via user input. In various specific implementations, the entry position is a point of a line segment at a predetermined percentage of the distance between the user position and the transformed position of interest. For example, in various specific implementations, the predetermined percentage is 50%, and the entry position is the midpoint of the line segment. In various specific implementations, the predetermined percentage can be changed via user input.

[0059] In various implementations, method 300 also includes determining the orientation of the entrance in the first three-dimensional coordinate system based on the user position and the transformed position of interest. For example, in various implementations, the entrance orientation is perpendicular to the line segment between the user position and the transformed position of interest. In various implementations, the entrance size is a predetermined size. In various implementations, the predetermined size can be changed via user input. However, in various implementations, method 300 also includes determining the entrance size based on the user position and the transformed position of interest. For example, in various implementations, the entrance size is based on the distance between the user position and the entrance position. For example, in various implementations, the entrance size is an increasing function of the distance between the user position and the entrance position, so that the entrance size increases as the user moves further away from the entrance.

[0060] Method 300 continues in box 350, where the device displays a portal corresponding to at least a partial view of the second environment at the portal location on the display. In various embodiments, displaying the portal includes rendering the portal and compositing the rendering with an image of the physical environment. In various embodiments, rendering the portal includes determining a portal area in a two-dimensional coordinate system of the display. For example, in various embodiments, the portal has an portal shape, a portal size, a portal location, and a portal orientation for defining a set of points in a first three-dimensional coordinate system, and determining the portal area includes projecting the set of points into the two-dimensional coordinate system based on the user position. In various embodiments, rendering the portal includes determining a pixel value for each pixel of the portal area. In various embodiments, determining the pixel value includes transforming the user position into a transformed user position in a second three-dimensional coordinate system based on a coordinate system transformation, and rendering the second environment from the transformed user position for the pixel (e.g., via ray tracing from the transformed user position through the pixel position into the second environment).

[0061] In various implementations, the location of the portal is dynamically updated based on the user moving the first environment and / or the dynamic object moving the second environment. Therefore, in various implementations, method 300 also includes: obtaining an updated user location in the first three-dimensional coordinate system; determining an updated portal location in the first three-dimensional coordinate system based on the updated user location and the transformed location of interest; and displaying the portal at the updated portal location on the display.

[0062] In various specific implementations, method 300 also includes: obtaining an updated position of interest in a second three-dimensional coordinate system; transforming the updated position of interest into a transformed updated position of interest in a first three-dimensional coordinate system; determining an updated entry position in the first three-dimensional coordinate system based on the user position and the transformed updated position of interest; and displaying the entry at the updated entry position on the display.

[0063] Figure 4is a block diagram of an example of a controller 110 according to some implementations. While some specific features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the implementations disclosed herein. To this end, as a non-limiting example, in some embodiments, the controller 110 includes one or more processing units 402 (e.g., a microprocessor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a graphics processing unit (GPU), a central processing unit (CPU), a processing core, etc.), one or more input / output (I / O) devices 406, one or more communication interfaces 408 (e.g., a universal serial bus (USB), IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Global Positioning System (GPS), infrared (IR), Bluetooth, ZIGBEE, and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 410, a memory 420, and one or more communication buses 404 for interconnecting these components and various other components.

[0064] In some implementations, the one or more communication buses 404 include circuits that interconnect and control communications between system components. In some implementations, the one or more I / O devices 406 include at least one of a keyboard, a mouse, a trackpad, a joystick, one or more microphones, one or more speakers, one or more image sensors, one or more displays, and the like.

[0065] Memory 420 includes high-speed random access memory, such as dynamic random access memory (DRAM), static random access memory (SRAM), double data rate random access memory (DDR RAM), or other random access solid-state memory devices. In some specific implementations, memory 420 includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 420 optionally includes one or more storage devices remotely located from one or more processing units 402. Memory 420 includes non-transitory computer-readable storage media. In some specific implementations, memory 420 or non-transitory computer-readable storage media of memory 420 stores the following programs, modules, and data structures or their subsets, including an optional operating system 430 and an XR experience module 440.

[0066] The operating system 430 includes processes for handling various basic system services and for performing hardware-related tasks. In some implementations, the XR experience module 440 is configured to manage and coordinate single or multiple XR experiences of one or more users (e.g., a single XR experience of one or more users, or multiple XR experiences of corresponding groups of one or more users). To this end, in various implementations, the XR experience module 440 includes a data acquisition unit 442, a tracking unit 444, a coordination unit 446, and a data transmission unit 448.

[0067] In some specific implementations, the data acquisition unit 442 is configured to at least Figure 1 The electronic device 120 obtains data (eg, presentation data, interaction data, sensor data, location data, etc.). To this end, in various specific implementations, the data obtaining unit 442 includes instructions and / or logic components for instructions and heuristics and metadata for heuristics.

[0068] In some implementations, the tracking unit 444 is configured to map the physical environment 105 and to track at least the electronic device 120 relative to the physical environment 105. Figure 1 The location / position of the physical environment 105. To this end, in various implementations, the tracking unit 444 includes instructions and / or logic components for instructions and heuristics and metadata for the heuristics.

[0069] In some implementations, the coordination unit 446 is configured to manage and coordinate the XR experience presented to the user by the electronic device 120. To this end, in various implementations, the coordination unit 446 includes instructions and / or logic components for instructions and heuristics and metadata for the heuristics.

[0070] In some implementations, the data transmission unit 448 is configured to transmit data (e.g., presentation data, location data, etc.) to at least the electronic device 120. To this end, in various implementations, the data transmission unit 448 includes instructions and / or logic components for instructions as well as heuristics and metadata for the heuristics.

[0071] Although the data acquisition unit 442, tracking unit 444, coordination unit 446, and data transmission unit 448 are shown as residing on a single device (e.g., controller 110), it should be understood that in other specific implementations, any combination of the data acquisition unit 442, tracking unit 444, coordination unit 446, and data transmission unit 448 may be located in separate computing devices.

[0072] also, Figure 4It serves more as a functional description of various features that may be present in a particular implementation, as opposed to a schematic diagram of the structure of the implementation described herein. As one of ordinary skill in the art will recognize, items shown separately may be combined, and some items may be separated. For example, Figure 4 Some functional modules shown separately in the figure may be implemented in a single module, and the various functions of a single functional block may be implemented by one or more functional blocks in various specific implementations. The actual number of modules and the division of specific functions and how the features are distributed among them will vary depending on the specific implementation, and in some specific implementations, it depends in part on the specific combination of hardware, software and / or firmware selected for a specific specific implementation.

[0073] Figure 5 is a block diagram of an example of an electronic device 120 according to some implementations. While some specific features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the implementations disclosed herein. To this end, as a non-limiting example, in some embodiments, the electronic device 120 includes one or more processing units 502 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, etc.), one or more input / output (I / O) devices and sensors 506, one or more communication interfaces 508 (e.g., USB, FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, IR, BLUETOOTH, ZIGBEE, and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 510, one or more XR displays 512, one or more optional internal-facing and / or external-facing image sensors 514, memory 520, and one or more communication buses 504 for interconnecting these and various other components.

[0074] In some implementations, one or more communication buses 504 include circuits that interconnect and control communications between system components. In some implementations, one or more I / O devices and sensors 506 include an inertial measurement unit (IMU), an accelerometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., a blood pressure monitor, a heart rate monitor, a blood oxygen sensor, a blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptic engine, and / or one or more depth sensors (e.g., structured light, time of flight, etc.), etc.

[0075] In some implementations, one or more XR displays 512 are configured to provide an XR experience to the user. In some implementations, one or more XR displays 512 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light-emitting field effect transistor (OLET), organic light-emitting diode (OLED), surface conduction electron emission display (SED), field emission display (FED), quantum dot light-emitting diode (QD-LED), micro-electromechanical system (MEMS) and / or similar display types. In some implementations, one or more XR displays 512 correspond to diffraction, reflection, polarization, holographic and other waveguide displays. For example, the electronic device 120 includes a single XR display. As another example, the electronic device includes an XR display for each eye of the user. In some implementations, one or more XR displays 512 can present MR and VR content.

[0076] In some implementations, the one or more image sensors 514 are configured to acquire image data corresponding to at least a portion of the user's face (including the user's eyes) (and may be referred to as an eye-tracking camera). In some implementations, the one or more image sensors 514 are configured to face forward so as to acquire image data corresponding to the physical environment that the user would see when the electronic device 120 is not present (and may be referred to as a scene camera). The one or more optional image sensors 514 may include one or more RGB cameras (e.g., with a complementary metal oxide semiconductor (CMOS) image sensor or a charge coupled device (CCD) image sensor), one or more infrared (IR) cameras, one or more event-based cameras, and the like.

[0077] The memory 520 includes a high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices. In some specific implementations, the memory 520 includes a non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory 520 optionally includes one or more storage devices remotely located from the one or more processing units 502. The memory 520 includes a non-transitory computer-readable storage medium. In some specific implementations, the memory 520 or the non-transitory computer-readable storage medium of the memory 520 stores the following programs, modules, and data structures or a subset thereof, including an optional operating system 530 and an XR rendering module 540.

[0078] The operating system 530 includes processes for handling various basic system services and for performing hardware-related tasks. In some specific implementations, the XR rendering module 540 is configured to present XR content to the user via one or more XR displays 512. To this end, in various specific implementations, the XR rendering module 540 includes a data acquisition unit 542, an entry location unit 544, an XR rendering unit 546, and a data transmission unit 548.

[0079] In some specific implementations, the data acquisition unit 542 is configured to at least Figure 1 The controller 110 obtains data (eg, presentation data, interaction data, sensor data, location data, etc.) To this end, in various implementations, the data acquisition unit 542 includes instructions and / or logic components for instructions as well as heuristics and metadata for heuristics.

[0080] In some implementations, the portal location unit 544 is configured to determine the portal location based on the user location. To this end, in various implementations, the portal location unit 544 includes instructions and / or logic components for instructions and heuristics and metadata for the heuristics.

[0081] In some implementations, the XR rendering unit 546 is configured to display the portal at the determined portal location via the one or more XR displays 512. To this end, in various implementations, the XR rendering unit 546 includes instructions and / or logic for instructions and heuristics and metadata for the heuristics.

[0082] In some implementations, the data transmission unit 548 is configured to transmit data (e.g., presentation data, location data, etc.) to at least the controller 110. In some implementations, the data transmission unit 548 is configured to transmit authentication credentials to the electronic device. To this end, in various implementations, the data transmission unit 548 includes instructions and / or logic components for instructions and heuristics and metadata for the heuristics.

[0083] Although the data acquisition unit 542, the portal positioning unit 544, the XR rendering unit 546, and the data transmission unit 548 are illustrated as residing on a single device (e.g., the electronic device 120), it should be understood that in other specific implementations, any combination of the data acquisition unit 542, the portal positioning unit 544, the XR rendering unit 546, and the data transmission unit 548 may be located in separate computing devices.

[0084] also, Figure 5It is more of a functional description of various features that may be present in a particular implementation, as opposed to a schematic diagram of the structure of the implementation described herein. As one of ordinary skill in the art will recognize, items shown separately may be combined, and some items may be separated. For example, Figure 5 Some functional modules shown separately in the figure may be implemented in a single module, and the various functions of a single functional block may be implemented by one or more functional blocks in various specific implementations. The actual number of modules and the division of specific functions and how the features are distributed among them will vary depending on the specific implementation, and in some specific implementations, it depends in part on the specific combination of hardware, software and / or firmware selected for a specific specific implementation.

[0085] Although various aspects of specific implementations within the scope of the appended claims are described above, it should be apparent that the various features of the above-mentioned specific implementations can be embodied in a variety of forms, and any specific structures and / or functions described above are merely illustrative. Based on this disclosure, it should be understood by those skilled in the art that the aspects described herein can be implemented independently of any other aspects, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement a device and / or a method can be practiced. In addition, in addition to or different from one or more aspects set forth herein, other structures and / or functions can be used to implement such a device and / or such a method can be practiced.

[0086] It will also be understood that, although the terms "first", "second", etc. may be used to describe various elements in this article, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, a first node may be referred to as a second node, and similarly, a second node may be referred to as a first node, which changes the meaning of the description, as long as all occurrences of the "first node" are consistently renamed and all occurrences of the "second node" are consistently renamed. Both the first node and the second node are nodes, but they are not the same node.

[0087] The terms used herein are only for describing specific implementations and are not intended to limit the claims. As used in the description of this specific implementation and the appended claims, the singular forms of "one", "a kind of" and "the" are intended to also cover the plural forms, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" used herein refers to and covers any and all possible combinations of one or more items in the associated listed items. It will also be understood that the term "comprising" when used in this specification specifies the presence of stated features, integers, steps, operations, elements and / or parts, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts, and / or their groupings.

[0088] As used herein, the term “if” may be interpreted to mean “when the antecedent is true” or “when the antecedent is true” or “in response to determining” or “upon determining” or “in response to detecting” that the antecedent is true, depending on the context. Similarly, the phrase “if it is determined that [the antecedent is true]” or “if [the antecedent is true]” or “when [the antecedent is true]” is interpreted to mean “upon determining that the antecedent is true” or “in response to determining” or “upon determining” that the antecedent is true or “when detecting that the antecedent is true” or “in response to detecting” that the antecedent is true, depending on the context.

Claims

1. A method comprising: At a device having a display, one or more processors, and non-transitory memory: Obtaining a user position in a first three-dimensional coordinate system of a first environment; obtaining a location of interest in a second three-dimensional coordinate system of a second environment; transforming the location of interest into a transformed location of interest in the first three-dimensional coordinate system; determining a portal location in the first three-dimensional coordinate system based on the user location and the transformed location of interest; as well as A portal corresponding to at least a partial view of the second environment is displayed on the display at the portal location. 2 . The method of claim 1 , wherein the first environment is a physical environment and the second environment is a virtual environment. The method of claim 1 , wherein the location of interest is a location of a static object in the second environment. The method of claim 1 , wherein the location of interest is a location of a dynamic object in the second environment. The method according to claim 1 , wherein the location of interest is a gaze point of a user. The method of claim 1 , wherein the entry location is a point of a line segment between the user location and the transformed location of interest. The method of claim 6 , wherein the entry location is a point of the line segment that is a predetermined distance from the user location.

8. The method of claim 6, wherein the entry location is a point of the line segment that is a predetermined distance from the transformed location of interest.

9. The method of claim 6, wherein the entry location is a point of the line segment whose distance from the user location is a predetermined percentage of the distance between the user location and the transformed location of interest.

10. The method of claim 9, wherein the predetermined percentage is 50% and the entry location is a midpoint of the line segment.

11. The method of claim 1, further comprising determining a portal orientation in the first three-dimensional coordinate system based on the user location and the transformed location of interest.

12. The method of claim 11, wherein the portal orientation is perpendicular to a line segment between the user location and the transformed location of interest.

13. The method of claim 1, further comprising determining a portal size based on the user location and the transformed location of interest. The method of claim 13 , wherein the portal size is based on a distance between the user location and the portal location.

15. The method according to claim 1, further comprising: Obtaining an updated user position in the first three-dimensional coordinate system; determining an updated entry location in the first three-dimensional coordinate system based on the updated user location and the transformed location of interest; as well as The entry is displayed on the display at the updated entry location.

16. The method according to claim 1, further comprising: Obtaining an updated position of interest in the second three-dimensional coordinate system; transforming the updated position of interest to an updated transformed position of interest in the first three-dimensional coordinate system; determining an updated entry location in the first three-dimensional coordinate system based on the user location and the updated transformed location of interest; as well as The entry is displayed on the display at the updated entry location.

17. A device comprising: monitor; non-transitory memory; and One or more processors configured to: Obtaining a user position in a first three-dimensional coordinate system of a first environment; obtaining a location of interest in a second three-dimensional second coordinate system of the second environment; transforming the location of interest into a transformed location of interest in the first three-dimensional coordinate system; determining a portal location in the first three-dimensional coordinate system based on the user location and the transformed location of interest; as well as A portal is displayed on the display at the portal location.

18. The apparatus of claim 17, wherein the first environment is a physical environment and the second environment is a virtual environment.

19. The apparatus of claim 17, wherein the entry location is a point of a line segment between the user location and the transformed location of interest.

20. A non-transitory memory storing one or more programs that, when executed by one or more processors of a device including a display, cause the device to: Obtaining a user position in a first three-dimensional coordinate system of a first environment; obtaining a location of interest in a second three-dimensional second coordinate system of the second environment; transforming the location of interest into a transformed location of interest in the first three-dimensional coordinate system; determining a portal location in the first three-dimensional coordinate system based on the user location and the transformed location of interest; as well as A portal is displayed on the display at the portal location.

Images