Local viewing angle correction to mitigate vertical parallax

Abstract

In one implementation, a method of performing viewing angle correction is performed by a device having a three-dimensional device coordinate system and including a first image sensor, a first display, one or more processors, and a non-transitory memory. The method includes capturing a first image of a physical environment using the first image sensor. The method includes transforming the first image from a first view angle to a second view angle based on a difference between the first view angle and the second view angle of the first image sensor, wherein a first distance between the second viewing angle and a position corresponding to a first eye of a user is smaller than a second distance between the first viewing angle and the position corresponding to the first eye of the user. The method includes displaying the transformed first image of the physical environment on the first display.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 407,805 filed on September 19, 2022 and U.S. Provisional Patent Application No. 63 / 470,697 filed on June 2, 2023, both of which are incorporated by reference in their entirety. Technical Field

[0003] The present disclosure generally relates to systems, methods, and devices for performing local perspective correction. Background Art

[0004] In various embodiments, an extended reality (XR) environment is presented by a head mounted device (HMD). Various HMDs include a scene camera that captures an image of the physical environment (e.g., a scene) in which a user is present and a display that displays the image to the user. In some cases, the image or a portion thereof may be combined with one or more virtual objects to present an XR experience to the user. In other cases, the HMD may operate in a pass-through mode in which an image or a portion thereof is presented to the user without adding virtual objects. Ideally, the image of the physical environment presented to the user is substantially similar to what the user would see if the HMD were not present. However, due to the different positioning of the eyes, display, and camera in space, this may not occur, resulting in motion sickness, impaired distance perception, disorientation, and poor hand-eye coordination. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] In order that the present disclosure may be understood by those skilled in the art, a more particular description will be given with reference to some exemplary implementations, some of which are illustrated in the accompanying drawings.

[0006] Figure 1 is a block diagram of an example operating environment according to some implementations.

[0007] Figure 2 Example scenarios related to capturing images of a physical environment and displaying the captured images according to some implementations are illustrated.

[0008] Figure 3 A top-down view of the physical environment.

[0009] Figure 4A Illustrate what would be seen by the user's left eye if the user is not wearing an HMD Figure 3 view of the physical environment.

[0010] Figure 4B An example of a video captured by the left image sensor of the HMD Figure 3 An image of the physical environment.

[0011] Figure 4C and Figure 4D Illustrated Figure 4B A transformed version of an image.

[0012] Figures 5 to 7 A front view of the HMD with various perspective transformations is illustrated.

[0013] Figure 8 is a flowchart representation of a method of performing perspective correction according to some specific implementations.

[0014] Fig. 9 is a block diagram of an example controller according to some specific implementations.

[0015] Fig.10 is a block diagram of an example electronic device according to some implementations.

[0016] As is common practice, the various features illustrated in the drawings may not be drawn to scale. Therefore, the sizes of the various features may be arbitrarily expanded or reduced for clarity. In addition, some of the drawings may not depict all components of a given system, method, or device. Finally, throughout the specification and drawings, similar reference numerals may be used to represent similar features. Summary of the invention

[0017] Various embodiments disclosed herein include devices, systems, and methods for performing perspective correction. In various embodiments, the method is performed by a device having a three-dimensional device coordinate system and including a first image sensor, a first display, one or more processors, and a non-volatile memory. The method includes capturing a first image of a physical environment using the first image sensor. The method includes transforming the first image from the first perspective to the second perspective based on a difference between a first perspective and a second perspective of the first image sensor, wherein the second perspective is less than a first distance from a position corresponding to a first eye of a user than a second distance between the first perspective and the position corresponding to the first eye of the user. The method includes displaying the transformed first image of the physical environment on the first display.

[0018] According to some specific implementations, a device includes one or more processors, non-volatile memory, and one or more programs; one or more programs are stored in the non-volatile memory and are configured to be executed by one or more processors. One or more programs include instructions for executing or causing the execution of any method described herein. According to some specific implementations, a non-volatile computer-readable storage medium stores instructions that, when executed by one or more processors of the device, cause the device to execute or cause the execution of any of the methods described herein. According to some specific implementations, a device includes: one or more processors, non-volatile memory, and a component for executing or causing the execution of any of the methods described herein. DETAILED DESCRIPTION

[0019] Many details are described in order to provide a thorough understanding of the example implementations shown in the accompanying drawings. However, the accompanying drawings only illustrate some example aspects of the present disclosure and should not be considered limiting. One of ordinary skill in the art will appreciate that other effective aspects and / or variations do not include all of the specific details described herein. In addition, well-known systems, methods, components, devices, and circuits are not described in detail so as not to obscure more relevant aspects of the example implementations described herein.

[0020] As described above, in an HMD with a display and a scene camera, if the HMD does not exist due to the different positions of the eyes, display, and camera in space, the image of the physical environment presented to the user on the display may not always reflect what the user sees. In various cases, this leads to motion sickness, poor distance perception, disorientation, and poor hand-eye coordination when interacting with the physical environment. Therefore, in various specific implementations, the images from the scene camera are transformed so that they appear to be captured at the position of the user's eyes using a depth map. In various specific implementations, for each pixel of the image, the depth map represents the distance from the origin to the object represented by the pixel (e.g., from the location of the image sensor, another location of the HMD, or any other location in the physical environment).

[0021] In various cases, transforming the images so that they appear to be captured at the location of the user's eyes can introduce artifacts in the images, such as holes, distortions, flickering, and the like. In various implementations, rather than transforming the images so that they appear to be captured at the location of the user's eyes, partially transforming the images so that they appear to be captured at a location closer to the user's eye location than the location of the scene camera in one or more dimensions in the device's three-dimensional device coordinate system. In various cases, partial transforms introduce fewer artifacts. In addition, in various cases, partial transforms may also be more computationally efficient. Thus, the device is able to achieve a selected balance between user comfort, aesthetics, and power consumption.

[0022] Figure 1 1 is a block diagram of an example operating environment 100 according to some implementations. Although relevant features are shown, those of ordinary skill in the art will recognize from this disclosure that various other features are not illustrated for the sake of brevity and so as not to obscure more relevant aspects of the example implementations disclosed herein. To this end, as a non-limiting example, the operating environment 100 includes a controller 110 and an electronic device 120.

[0023] In some implementations, the controller 110 is configured to manage and coordinate the user's XR experience. In some implementations, the controller 110 includes a suitable combination of software, firmware, and / or hardware. Fig. 9 Controller 110 is described in more detail. In some implementations, controller 110 is a computing device located locally or remotely relative to physical environment 105. For example, controller 110 is a local server located within physical environment 105. In another example, controller 110 is a remote server (e.g., a cloud server, a central server, etc.) located outside physical environment 105. In some implementations, controller 110 is communicatively coupled to electronic device 120 via one or more wired or wireless communication channels 144 (e.g., Bluetooth, IEEE 802.11x, IEEE802.16x, IEEE 802.3x, etc.). In another example, controller 110 is included in a housing of electronic device 120. In some implementations, the functionality of controller 110 is provided by and / or combined with electronic device 120.

[0024] In some implementations, the electronic device 120 is configured to provide an XR experience to the user. In some implementations, the electronic device 120 includes a suitable combination of software, firmware, and / or hardware. According to some implementations, the electronic device 120 presents XR content to the user via a display 122 while the user is physically present within a physical environment 105, which includes a table 107 within a field of view 111 of the electronic device 120. In some implementations, the user holds the electronic device 120 in one or both of his / her hands. In some implementations, when providing XR content, the electronic device 120 is configured to display an XR object (e.g., an XR cylinder 109) and implement video pass-through of the physical environment 105 (e.g., including a representation 117 of the table 107) on the display 122. The following description is relative to Fig.10 The electronic device 120 is described in more detail.

[0025] According to some implementations, the electronic device 120 provides an XR experience to the user while the user is virtually and / or physically present within the physical environment 105 .

[0026] In some implementations, the user wears the electronic device 120 on his / her head. For example, in some implementations, the electronic device includes a head-mounted system (HMS), a head-mounted device (HMD), or a head-mounted housing (HME). Therefore, the electronic device 120 includes one or more XR displays configured to display XR content. For example, in various implementations, the electronic device 120 surrounds the user's field of view. In some implementations, the electronic device 120 is a handheld device (such as a smart phone or a tablet computer) configured to present XR content, and the user no longer wears the electronic device 120 but holds the device with the display facing the user's field of view and the camera facing the physical environment 105. In some implementations, the handheld device may be placed in a housing that can be worn on the user's head. In some implementations, the electronic device 120 is replaced with an XR cabin, housing, or room configured to present XR content, in which the user no longer wears or holds the electronic device 120.

[0027] Figure 2 An example scenario 200 is illustrated involving capturing an image of an environment and displaying the captured image according to some implementations. A user wears a device (e.g., Figure 1 The image sensor 230 captures an image of the physical environment, and the display 210 displays the image of the physical environment to the user's eye 220. The image sensor 230 has a viewing angle that is vertically offset from the user's viewing angle (e.g., where the user's eye 220 is located) by a vertical offset 241. In addition, the viewing angle of the image sensor 230 is longitudinally offset from the user's viewing angle by a longitudinal offset 242. In addition, the viewing angle of the image sensor 230 is laterally offset from the user's viewing angle by a lateral offset (e.g., toward Figure 2 within or outside of a page).

[0028] Figure 3 300. The physical environment 300 includes a structure 301 and a user 310 wearing an HMD 320. 4A to 4DAs shown in the described views and images, the structure 301 has squares, triangles and circles drawn on it. The left eye 311a of the user 310 is located at the left eye position in the device coordinate system, thereby providing a left eye perspective (e.g., at the center of the eye pupil). The right eye 311b of the user 310 is located at the right eye position, thereby providing a right eye perspective. The HMD 320 includes a left image sensor 321a located at the left image sensor position, which provides a left image sensor perspective (e.g., at the center of the entrance pupil of the image sensor). The HMD 320 includes a right image sensor 321b at the right image sensor position that provides a right image sensor perspective. Since the left eye 311a and the left image sensor 321a are located at different positions, they each provide different perspectives of the physical environment. The HMD 320 also includes a left eye display 331a in the field of view of the left eye 311a and a right eye display 331b in the field of view of the right eye 311b.

[0029] Figure 3 Axes 333 of a three-dimensional device coordinate system are also illustrated. In various implementations, the x-axis and y-axis are aligned with the horizontal u-axis and vertical v-axis of the left image sensor 321a (and / or the right image sensor 321b), and the z-axis is aligned with the optical axis of the left image sensor 321a (and / or the right image sensor 321b). In various implementations, the three-dimensional device coordinate system is not aligned with the left image sensor 321a and / or the right image sensor 321b.

[0030] Figure 4A Illustrated is a view 401 of the physical environment 300 as it would be seen by the left eye 311a of the user 310 if the user 310 was not wearing the HMD 320. In the view 401, a square, a triangle, and a circle can be seen on the structure 301.

[0031] Figure 4B An image 402 of the physical environment 300 captured by the left image sensor 321a is illustrated. In the image 402, similar to the view 401, a square, a triangle, and a circle can be seen on the structure 301. However, since the left image sensor 321a is to the left of the left eye 311a, the triangle and the circle on the structure 301 in the image 402 are at a position to the right of the corresponding positions of the triangle and the circle in the view 401. In addition, since the left image sensor 321a is higher than the left eye 311a, the square, the triangle, and the circle in the image 402 are at a position lower than the corresponding positions of the square, the triangle, and the circle in the view 401. In addition, since the left image sensor 321a is closer to the structure 301 than the left eye 311a, the square, the triangle, and the circle are larger in the image 402 than in the view 401.

[0032] In various implementations, the HMD 320 transforms the image 402 so that the image appears to be captured from a left eye perspective rather than from a left image sensor perspective, e.g., to look like view 401. In various implementations, the transformation includes a correction of the image 402 relative to a three-dimensional device coordinate system. In various implementations, the transformation is a perspective transformation. In various implementations, the HMD 320 transforms the image 402 based on a depth value associated with the image 402 and a difference between the left image sensor perspective and the left eye perspective. In various implementations, the depth value of a pixel of the image 402 represents a distance from the left image sensor 321a to an object represented by the pixel in the physical environment 300. In various implementations, the difference between the left image sensor perspective and the left eye perspective is determined during a calibration process.

[0033] In various implementations, HMD 320 transforms image 402 to make the image appear as if it was captured from a second perspective other than the left eye perspective, but closer to the left eye perspective than the left image sensor perspective in at least one dimension of the three-dimensional device coordinate system of HMD 320 .

[0034] In various implementations, transforming the image in any direction may increase artifacts. In various implementations, transforming the image in a particular direction may improve user comfort, the user's depth perception, and the user's sense of scale.

[0035] Thus, in various implementations, the HMD 320 transforms the image 402 only in the x-direction to make the image appear as if it was captured at the second perspective at a location having the same x-coordinate as the left eye location and the same y-coordinate and z-coordinate as the left image sensor location. In various implementations, the HMD 320 transforms the image 402 based on a depth value associated with the image 402 and a difference between the left image sensor perspective and the second perspective. In various implementations, the difference between the left image sensor perspective and the second perspective is determined during a calibration process.

[0036] Figure 4C A first transformed image 403 of the physical environment 300 generated by transforming the image 402 only in the x-direction is illustrated. In the image 402, the triangle and the circle are to the right of the corresponding positions of the triangle and the circle in the view 401, whereas in the first transformed image 403, the triangle and the circle are at the same horizontal positions as the corresponding horizontal positions of the triangle and the circle in the view 401. However, in the first transformed image 403 (similar to the image 402), the square, the triangle, and the circle are at a lower vertical position than the corresponding vertical positions of the square, the triangle, and the circle in the view 401. Furthermore, in the first transformed image 403 (similar to the image 402), the square, the triangle, and the circle are larger than the square, the triangle, and the circle in the view 401.

[0037] In various implementations, HMD 320 transforms image 402 only in the x- and z-directions to make the image appear as if it was captured at the second perspective at a location having the same x- and z-coordinates as the left eye location and the same y-coordinate as the left image sensor location.

[0038] Figure 4D A second transformed image 404 of the physical environment 300 generated by transforming the image 402 only in the x-direction and the z-direction is illustrated. In the second transformed image, the triangle and the circle are at the same horizontal positions as the corresponding horizontal positions of the triangle and the circle in the view 401. In addition, in the second transformed image, the square, the triangle, and the circle have the same size as the square, the triangle, and the circle in the view 401. However, in the second transformed image 403, the square, the triangle, and the circle are at lower vertical positions than the corresponding vertical positions of the square, the triangle, and the circle in the view 401.

[0039] In various implementations, the HMD 320 transforms the image 402 at least in part in each dimension to make the image appear, for example, as if it were captured at the second perspective at a location having the same x-coordinate of the left eye location, a y-coordinate from the left image sensor location to one-third of the left eye location, and a z-coordinate between the left image sensor location and the left eye location.

[0040] Figure 5 3 illustrates a front view of the HMD 320 with a first perspective transformation. Figure 5 , the image captured by the left image sensor 321a is transformed from a first perspective of the left image sensor 321a to a second perspective at a position 511a, which has the same x-coordinate as the left eye 311a and the same y-coordinate as the left image sensor 321a. Similarly, the image captured by the right image sensor 321b is transformed from a first perspective of the right image sensor 321b to a second perspective at a position 511b, which has the same x-coordinate as the right eye 311b and the same y-coordinate as the right image sensor 321b.

[0041] Thus, the position of the left eye 311a and the position of the second perspective 511a form a vector 512a, which is vertical and has a first length. The position of the left image sensor 321a and the position of the second perspective 511a form a vector 513a, which is horizontal and has a second length. The position of the right eye 311b and the position of the second perspective 511b form a vector 512b, which is vertical and has a first length. Vectors 512a and 512b have the same magnitude and the same direction. The position of the right image sensor 321b and the position of the second perspective 511b form a vector 513b, which is horizontal and has a second length. Vectors 513a and 513b have the same magnitude but opposite directions.

[0042] Figure 6 3 illustrates a front view of the HMD 320 with a second perspective transformation. Figure 6 In FIG, the HMD 320 is tilted so that the line passing through the left eye 311a and the right eye 311b is no longer parallel to the line passing through the left image sensor 321a and the right image sensor 321b. Figure 6 , the image captured by the left image sensor 321a is transformed from a first perspective of the left image sensor 321a to a second perspective at a position 611a, which has the same x-coordinate as the left eye 311a and the same y-coordinate as the left image sensor 321a. Similarly, the image captured by the right image sensor 321b is transformed from a first perspective of the right image sensor 321b to a second perspective at a position 611b, which has the same x-coordinate as the right eye 311b and the same y-coordinate as the right image sensor 321b.

[0043] Thus, the position of the left eye 311a and the position 611a of the second view form a vector 612a, which is vertical and has a first length. The position of the left image sensor 321a and the position 611a of the second view form a vector 613a, which is horizontal and has a second length. The position of the right eye 311b and the position 611b of the second view form a vector 612b, which is vertical and has a third length different from the first length. Vector 612a and vector 612b have the same direction but different magnitudes. This difference in magnitude results in vertical parallax, in which different eyes experience vertical transformations of different magnitudes. This may result in increased discomfort and decreased aesthetics, such as difficulty in binocular vision fusion. The position of the right image sensor 321b and the position 611b of the second view form a vector 613b, which is horizontal and has a fourth length, which may be the same or different from the second length. Vector 613a and vector 613b have opposite directions and may have the same magnitude or different magnitudes.

[0044] Figure 7A front view of the HMD 320 with a third perspective transformation is illustrated. Figure 7 In, as in Figure 6 As in FIG. 1 , the HMD 320 is tilted so that the line 710 passing through the left eye 311a and the right eye 311b is not parallel to the line 720 passing through the left image sensor 321a and the right image sensor 321b. Figure 7 , the image captured by the left image sensor 321a is transformed from the first perspective of the left image sensor 321a to the second perspective at position 711a. Similarly, the image captured by the right image sensor 321b is transformed from the first perspective of the right image sensor 321b to the second perspective at position 711b.

[0045] In various implementations, line 710 and line 720 may be skewed for reasons other than tilt of HMD 320, such as facial asymmetry, measurement / calibration errors, or extrinsic tolerances.

[0046] The HMD 320 determines the position 711a and the position 711b so that the vector 712a between the position of the left eye 311a and the position 711a of the second view and the vector 712b between the position of the right eye 311b and the position 711b of the second view have the same direction and the same magnitude. Therefore, the vector 712a and the vector 712b are parallel.

[0047] In various specific implementations, vector 712a and vector 712b have the same magnitude and the same direction as vector 712c between the midpoint of line 710 connecting left eye 311a and right eye 31b and the midpoint of line 720 connecting left image sensor 321a and right image sensor 321b. Therefore, vector 712a, vector 712b and vector 712c are parallel. Since vector 712a, vector 712b and vector 712c are parallel, vector 713a between left image sensor 321a and position 711a of the second viewing angle and vector 713b between right image sensor 321b and position 711b of the second viewing angle have the same magnitude but opposite directions. Therefore, vector 713a and vector 713b are parallel. In addition, since line 710 and line 720 are not parallel, vector 713a and vector 713b are not horizontal. In various specific implementations, vector 712a and vector 712b are not vertical.

[0048] Specifically, the x-component of vector 713a (and vector 713b) is half the difference between (1) the horizontal displacement of left eye 311a and right eye 311b (e.g., the x-component of line 710) and (2) the horizontal displacement of left image sensor 321a and right image sensor 321b (e.g., the x-component of line 720). Similarly, the y-component of vector 713a (and vector 713b) is half the difference between (1) the vertical displacement of left eye 311a and right eye 311b (e.g., the y-component of line 710) and (2) the vertical displacement of left image sensor 321a and right image sensor 321b (e.g., the y-component of line 720).

[0049] In various implementations, the z components of vectors 713a and 713b are determined as described above for the x and y components (e.g., using vector 712c as determined using the midpoints of lines 710 and 720 in three dimensions. In various implementations, the z components of vectors 713a and 713b are set to zero.

[0050] Figure 8 800 is a flowchart representation of a method for performing local perspective correction of an image according to some specific implementations. In various specific implementations, method 800 is performed by a device (e.g., Figure 1 In some implementations, the method 800 is performed by a processing logic (including hardware, firmware, software, or a combination thereof). In some implementations, the method 800 is performed by a processor executing instructions (e.g., code) stored in a non-transitory computer-readable medium (e.g., memory).

[0051] Method 800 begins in block 810 where a device captures a first image of a physical environment using a first image sensor.

[0052] The method 800 continues at block 820, where the device transforms, using one or more processors, the first image of the physical environment based on a difference between a first perspective and a second perspective of the image sensor, wherein the second perspective is less than a first distance from a position corresponding to a first eye of the user than a second distance between the first perspective and the position corresponding to the first eye of the user. In various implementations, the device transforms the first image of the physical environment at an image pixel level, an image tile level, or a combination thereof.

[0053] In various implementations, a device transforms a first image of a physical environment based on a depth map, the depth map comprising a plurality of depths respectively associated with a plurality of pixels of the first image of the physical environment. In various implementations, the depth map comprises a dense depth map, the dense depth map representing, for each pixel of the first image, an estimated distance between the first image sensor and an object represented by the pixel. In various implementations, the depth map comprises a sparse depth map, the sparse depth map representing, for each pixel in a subset of pixels of the first image, an estimated distance between the first image sensor and the object represented by the pixel. In various implementations, the device generates a sparse depth map from the dense depth map by sampling the dense depth map (e.g., selecting a single pixel in each N×N pixel block).

[0054] In various implementations, the device obtains multiple depths from a depth sensor. In various implementations, the device obtains multiple depths using stereo matching (e.g., using an image of the physical environment captured by a left scene camera and another image of the physical environment captured by a right scene camera). In various implementations, the device obtains multiple depths through eye tracking, e.g., the intersection of the gaze directions of the user's two eyes indicates the depth of the object the user is looking at.

[0055] In various implementations, the device obtains multiple depths from the three-dimensional model of the physical environment, such as via rasterization of the three-dimensional model and / or ray tracing from an image sensor to various features of the three-dimensional model.

[0056] In various implementations, for at least one dimension of the device coordinate system, the second viewing angle and the position corresponding to the user's first eye have the same coordinate value. Figure 4C In , image 402 is transformed only in the x dimension, and the second perspective and the left eye 311a share the x coordinate. In various implementations, the second perspective and the position corresponding to the user's first eye have the same coordinate values ​​for both dimensions of the device coordinate system. For example, in Figure 4D In FIG. 4 , the image 402 is transformed in the x dimension and the z dimension, and the second perspective and the left eye 311 a share the x coordinate and the z coordinate.

[0057] In various implementations, for less than three dimensions of the device coordinate system, the second viewing angle and the position corresponding to the user's first eye have the same coordinate values. Figure 4D , image 402 is transformed in the x-dimension and the z-dimension, and the second perspective and the left eye 311a have different y-coordinates. In various implementations, for less than two dimensions of the device coordinate system, the second perspective and the position corresponding to the user's first eye have the same coordinate values. For example, in Figure 4C, image 402 is transformed only in the x dimension, and the second perspective and the left eye 311a have different y coordinates and different z coordinates. In various implementations, for less than one dimension of the device coordinate system, the second perspective and the position corresponding to the user's first eye have the same coordinate values. Thus, in various implementations, for all three dimensions, the second perspective and the position corresponding to the user's first eye have different coordinate values. For example, in Figure 7 , the position 711a of the second viewing angle and the left eye 311a have different x-coordinates, different y-coordinates, and different z-coordinates.

[0058] In various embodiments, a first ratio between (1) the displacement in a first dimension of a device coordinate system between a first perspective and a second perspective and (2) the displacement in the first dimension between the first perspective and a position corresponding to a user's first eye is different from a second ratio between (1) the displacement in a second dimension of a device coordinate system between the first perspective and the second perspective and (2) the displacement in the second dimension between the first perspective and a position corresponding to the user's first eye. In various embodiments, the first ratio is approximately zero. In various embodiments, the first ratio is approximately one. In various embodiments, the first ratio is between zero and one. For example, in various embodiments, the first ratio is approximately between 0.25 and 0.75. For example, in Figure 5 In , the ratio between (1) the y-dimensional displacement between the left image sensor 321a and the position 511a of the second viewing angle and (2) the y-dimensional displacement between the image sensor 321a and the left eye 311a is approximately zero. As another example, in Figure 5 , the ratio between (1) the x-dimensional displacement between the image sensor 321a and the position 511a of the second viewing angle and (2) the x-dimensional displacement between the left image sensor 321a and the left eye 311a is approximately one. Thus, in various implementations, the first dimension is the x-dimension, the second dimension is the y-dimension, and the first ratio is greater than the second ratio. As another example, in Figure 7 , the ratio between (1) the y-dimensional displacement between the right image sensor 321b and the position 511b of the second viewing angle and (2) the y-dimensional displacement between the right image sensor 321b and the right eye 311b is between zero and one (eg, approximately 0.1).

[0059] In various implementations, the device performs a projective transformation based on the depth map and a difference between a first perspective and a second perspective of the first image sensor.

[0060] In various implementations, the projective transformation is a forward mapping, wherein for each pixel of the first image of the physical environment at the pixel position in the untransformed space, a new pixel position is determined in the transformed space of the transformed first image. In various implementations, the projective transformation is a reverse mapping, wherein for each pixel of the transformed first image at the pixel position in the transformed space, a source pixel position is determined in the untransformed space of the first image of the physical environment.

[0061] In various implementations, the source pixel position is determined according to the following equation, where x1 and y1 are the pixel positions in the untransformed space, x2 and y2 are the pixel positions in the transformed space, P2 is the 4×4 view projection matrix of the second perspective, P1 is the 4×4 view projection matrix of the first perspective of the image sensor, and d is the depth map value at the pixel position:

[0062]

[0063] In various implementations, for each pixel in the first image of the physical environment, the source pixel position is determined using the above equation. In various implementations, for less than each pixel of the first image of the physical environment, the source pixel position is determined using the above equation.

[0064] In various implementations, the device determines a view projection matrix for the second perspective and a view projection matrix for the first perspective during calibration, and stores data indicating the view projection matrices (or a product thereof) in a non-transitory memory. The product of the view projection matrices is a transformation matrix representing a difference between the first perspective and the second perspective of the first image sensor.

[0065] Thus, in various implementations, transforming the first image of the physical environment includes, for a plurality of pixels of the transformed first image having corresponding pixel positions, determining a corresponding plurality of source pixel positions. In various implementations, determining the corresponding plurality of source pixel positions includes, for each of the plurality of pixels of the transformed first image, multiplying a vector including the corresponding pixel position and a multiplicative inverse element of a corresponding element of the depth map by a transformation matrix representing a difference between a first perspective and a second perspective of the image sensor.

[0066] Using the source pixel positions in the untransformed space and the pixel values ​​of the pixels of the first image of the physical environment, the device generates a pixel value for each pixel position of the transformed first image using interpolation or other techniques.

[0067] In various implementations, method 800 includes determining a second viewing angle. In various implementations, method 800 includes determining a second viewing angle based on a position corresponding to a first eye of a user. Thus, in various implementations, method 800 includes determining a position corresponding to a first eye of a user. In various implementations, the device measures a position corresponding to a first eye of a user based on a current image (obtained while capturing an image of a physical environment) including the first eye of the user. In various implementations, the device predicts a position corresponding to a first eye of a user based on a previous image (obtained before capturing an image of an environment) including the first eye of the user. In various implementations, the device estimates a position corresponding to a first eye of a user based on an IMU (inertial measurement unit) of the device. For example, if the IMU indicates that the device is horizontal, the device estimates a position corresponding to a first eye of the user as a fixed distance from the center of the display in a vertical direction. However, if the IMU indicates that the device is tilted, the device estimates a position corresponding to a first eye of the user as a fixed distance from the center of the display in a vertical direction in a lateral deviation.

[0068] The method 800 continues in block 830, where the device displays the transformed first image of the physical environment on the first display. In various implementations, the transformed first image includes XR content. In some implementations, the XR content is added to the first image of the physical environment prior to the transformation (at block 820). In some implementations, the XR content is added to the transformed first image.

[0069] In various implementations, method 800 includes performing splay mitigation. For example, in various implementations, method 800 includes capturing a second image of the physical environment using a second image sensor. Method 800 includes transforming the second image from a third perspective to a fourth perspective based on a difference between a third perspective and a fourth perspective of the second image sensor. The method includes displaying the transformed second image of the physical environment on a second display.

[0070] In various implementations, the vector between the second viewing angle and the position corresponding to the user's first eye is parallel to the vector between the fourth viewing angle and the position corresponding to the user's second eye. Figure 7 , vector 712b is parallel to vector 712a. In various implementations, the vector between the second viewing angle and the position corresponding to the user's first eye is parallel to the midpoint vector between (1) the midpoint between the position corresponding to the user's first eye and the position corresponding to the user's second eye and (2) the midpoint between the first image sensor and the second image sensor. For example, in Figure 7, vector 712b is parallel to vector 712c. In various implementations, the vector between the second perspective and the position corresponding to the user's first eye is the same magnitude as the midpoint vector. In various implementations, the vector between the first perspective and the second perspective is parallel to the vector between the third perspective and the fourth perspective. For example, in Figure 7 , vector 713b is parallel to vector 713a.

[0071] In various implementations, the third distance of the fourth viewing angle from the position corresponding to the second eye of the user is less than the fourth distance between the second image sensor and the position corresponding to the second eye of the user. In various implementations, the third distance of the fourth viewing angle from the position corresponding to the second eye of the user is greater than the fourth distance between the second image sensor and the position corresponding to the second eye of the user. Therefore, the distance between the position 711b of the second viewing angle and the right eye 311a is less than the distance between the right image sensor 321a and the right eye 311b. In various implementations, the distance between the position 711a of the second viewing angle and the left eye 311a may be less than or greater than the distance between the left image sensor 321a and the left eye 311a, depending on the amount of vertical displacement between the left eye 311a and the right eye 311b.

[0072] Fig. 9 is a block diagram of an example of a controller 110 according to some implementations. While certain specific features are illustrated, those skilled in the art will recognize from this disclosure that various other features are not illustrated for the sake of brevity and so as not to obscure more relevant aspects of the implementations disclosed herein. To this end, as a non-limiting example, in some embodiments, the controller 110 includes one or more processing units 902 (e.g., a microprocessor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a graphics processing unit (GPU), a central processing unit (CPU), a processing core, and / or the like), one or more input / output (I / O) devices 906, one or more communication interfaces 908 (e.g., a universal serial bus (USB), FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Global Positioning System (GPS), infrared (IR), Bluetooth, ZIGBEE, and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 910, a memory 920, and one or more communication buses 904 for interconnecting these and various other components.

[0073] In some implementations, the one or more communication buses 904 include circuits that interconnect system components and control communications between system components. In some implementations, the one or more I / O devices 906 include at least one of a keyboard, a mouse, a touchpad, a joystick, one or more microphones, one or more speakers, one or more image sensors, one or more displays, etc.

[0074] The memory 920 includes a high-speed random access memory, such as a dynamic random access memory (DRAM), a static random access memory (SRAM), a double data rate random access memory (DDR RAM), or other random access solid-state memory devices. In some specific implementations, the memory 920 includes a non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory 920 optionally includes one or more storage devices remotely located from one or more processing units 902. The memory 920 includes a non-transitory computer-readable storage medium. In some specific implementations, the memory 920 or the non-transitory computer-readable storage medium of the memory 920 stores the following programs, modules, and data structures or subsets thereof, including an optional operating system 930 and an XR experience module 940.

[0075] The operating system 930 includes procedures for handling various basic system services and for performing hardware-related tasks. In some specific implementations, the XR experience module 940 is configured to manage and coordinate single or multiple XR experiences of one or more users (e.g., a single XR experience of one or more users, or multiple XR experiences of corresponding groups of one or more users). To this end, in various specific implementations, the XR experience module 940 includes a data acquisition unit 942, a tracking unit 944, a coordination unit 946, and a data sending unit 948.

[0076] In some specific implementations, the data acquisition unit 942 is configured to at least Figure 1 The electronic device 120 obtains data (eg, presentation data, interaction data, sensor data, location data, etc.). To this end, in various specific implementations, the data acquisition unit 942 includes instructions and / or logic components for instructions and heuristics and metadata for heuristics.

[0077] In some implementations, the tracking unit 944 is configured to map the physical environment 105 and to track at least the electronic device 120 relative to the physical environment 105. Figure 1 The location / position of the physical environment 105. To this end, in various specific implementations, the tracking unit 944 includes instructions and / or logic components for instructions and heuristics and metadata for the heuristics.

[0078] In some implementations, the coordination unit 946 is configured to manage and coordinate the XR experience presented to the user by the electronic device 120. To this end, in various implementations, the coordination unit 946 includes instructions and / or logic components for instructions and heuristics and metadata for heuristics.

[0079] In some implementations, the data transmission unit 948 is configured to transmit data (e.g., presentation data, location data, etc.) to at least the electronic device 120. To this end, in various implementations, the data transmission unit 948 includes instructions and / or logic components for instructions as well as heuristics and metadata for the heuristics.

[0080] Although the data acquisition unit 942, the tracking unit 944, the coordination unit 946, and the data sending unit 948 are shown as residing on a single device (e.g., the controller 110), it should be understood that in other specific implementations, any combination of the data acquisition unit 942, the tracking unit 944, the coordination unit 946, and the data sending unit 948 may be located in separate computing devices.

[0081] also, Fig. 9 It serves more as a functional description of various features that may be present in a particular implementation, rather than as a structural schematic diagram of the implementation described herein. As one of ordinary skill in the art will recognize, items shown separately may be combined, and some items may be separated. For example, Fig. 9 Some functional modules shown separately in the figure may be implemented in a single module, and the various functions of a single functional block may be implemented by one or more functional blocks in various specific implementations. The actual number of modules and the division of specific functions and how the features are distributed among them will vary depending on the specific implementation, and in some specific implementations, it depends in part on the specific combination of hardware, software and / or firmware selected for a specific specific implementation.

[0082] Fig.10is a block diagram of an example of an electronic device 120 according to some implementations. While certain specific features are illustrated, those skilled in the art will recognize from this disclosure that various other features are not illustrated for the sake of brevity and so as not to obscure more relevant aspects of the implementations disclosed herein. To this end, as a non-limiting example, in some embodiments, the electronic device 120 includes one or more processing units 1002 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, etc.), one or more input / output (I / O) devices and sensors 1006, one or more communication interfaces 1008 (e.g., USB, FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, IR, Bluetooth, ZIGBEE, and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 1010, one or more XR displays 1012, one or more optional internal-facing and / or external-facing image sensors 1014, memory 1020, and one or more communication buses 1004 for interconnecting these and various other components.

[0083] In some implementations, the one or more communication buses 1004 include circuits that interconnect system components and control communications between system components. In some implementations, the one or more I / O devices and sensors 1006 include at least one of an inertial measurement unit (IMU), an accelerometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., a blood pressure monitor, a heart rate monitor, a blood oxygen sensor, a blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptic engine, one or more depth sensors (e.g., structured light, time of flight, etc.), etc.

[0084] In some implementations, one or more XR displays 1012 are configured to provide an XR experience to the user. In some implementations, one or more XR displays 1012 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light-emitting field effect transistor (OLET), organic light-emitting diode (OLED), surface conduction electron emission display (SED), field emission display (FED), quantum dot light-emitting diode (QD-LED), micro-electromechanical system (MEMS) and / or similar display types. In some implementations, one or more XR displays 1012 correspond to diffraction, reflection, polarization, holographic and other waveguide displays. For example, the electronic device 120 includes a single XR display. In another example, the electronic device includes an XR display for each eye of the user. In some implementations, one or more XR displays 1012 can present MR and VR content.

[0085] In some implementations, the one or more image sensors 1014 are configured to obtain image data corresponding to at least a portion of the user's face, including the user's eyes (any of which may be referred to as an eye-tracking camera). In some implementations, the one or more image sensors 1014 are configured to face forward so as to obtain image data corresponding to the physical environment that the user would see when the electronic device 120 is not present (and may be referred to as a scene camera). The one or more optional image sensors 1014 may include one or more RGB cameras (e.g., with a complementary metal oxide semiconductor (CMOS) image sensor or a charge coupled device (CCD) image sensor), one or more infrared (IR) cameras, one or more event-based cameras, and the like.

[0086] Memory 1020 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices. In some specific implementations, memory 1020 includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 1020 optionally includes one or more storage devices remotely located from one or more processing units 1002. Memory 1020 includes non-transitory computer-readable storage media. In some specific implementations, memory 1020 or non-transitory computer-readable storage media of memory 1020 stores the following programs, modules, and data structures or subsets thereof, including an optional operating system 1030 and an XR rendering module 1040.

[0087] The operating system 1030 includes processes for handling various basic system services and for performing hardware-related tasks. In some specific implementations, the XR rendering module 1040 is configured to present XR content to the user via one or more XR displays 1012. To this end, in various specific implementations, the XR rendering module 1040 includes a data acquisition unit 1042, a perspective transformation unit 1044, an XR rendering unit 1046, and a data sending unit 1048.

[0088] In some specific implementations, the data acquisition unit 1042 is configured to at least Figure 1 The controller 110 obtains data (eg, presentation data, interaction data, sensor data, location data, etc.) To this end, in various implementations, the data acquisition unit 1042 includes instructions and / or logic components for instructions and heuristics and metadata for heuristics.

[0089] In some implementations, the perspective transformation unit 1044 is configured to perform local perspective correction. To this end, in various implementations, the perspective transformation unit 1044 includes instructions and / or logic components for instructions and heuristics and metadata for the heuristics.

[0090] In some implementations, the XR rendering unit 1046 is configured to display the transformed image via one or more XR displays 1012. To this end, in various implementations, the XR rendering unit 1046 includes instructions and / or logic for instructions and heuristics and metadata for the heuristics.

[0091] In some implementations, the data sending unit 1048 is configured to send data (e.g., presentation data, location data, etc.) to at least the controller 110. In some implementations, the data sending unit 1048 is configured to send authentication credentials to the electronic device. To this end, in various implementations, the data sending unit 1048 includes instructions and / or logic components for instructions and heuristics and metadata for the heuristics.

[0092] Although the data acquisition unit 1042, the perspective transformation unit 1044, the XR rendering unit 1046, and the data sending unit 1048 are shown as residing on a single device (e.g., the electronic device 120), it should be understood that in other specific implementations, any combination of the data acquisition unit 1042, the perspective transformation unit 1044, the XR rendering unit 1046, and the data sending unit 1048 may be located in separate computing devices.

[0093] also, Fig.10 It is more of a functional description of various features that may be present in a particular implementation, as opposed to a schematic diagram of the structure of the implementation described herein. As one of ordinary skill in the art will recognize, items shown separately may be combined, and some items may be separated. For example, Fig.10 Some functional modules shown separately in the figure may be implemented in a single module, and the various functions of a single functional block may be implemented by one or more functional blocks in various specific implementations. The actual number of modules and the division of specific functions and how the features are distributed among them will vary depending on the specific implementation, and in some specific implementations, it depends in part on the specific combination of hardware, software and / or firmware selected for a specific specific implementation.

[0094] Although various aspects of specific implementations within the scope of the appended claims are described above, it should be apparent that the various features of the above-mentioned specific implementations can be embodied in a variety of forms, and any specific structures and / or functions described above are merely illustrative. Based on this disclosure, it should be understood by those skilled in the art that the aspects described herein can be implemented independently of any other aspects, and two or more of these aspects can be combined in various ways. For example, a device and / or a method can be implemented using any number of aspects set forth herein. In addition, in addition to or different from one or more aspects set forth herein, such a device and / or such a method can be implemented using other structures and / or functions.

[0095] It will also be understood that, although the terms "first", "second", etc. may be used to describe various elements in this article, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, a first node may be referred to as a second node, and similarly, a second node may be referred to as a first node, which changes the meaning of the description, as long as all occurrences of the "first node" are consistently renamed and all occurrences of the "second node" are consistently renamed. Both the first node and the second node are nodes, but they are not the same node.

[0096] The terms used herein are only for describing specific implementations and are not intended to limit the claims. As used in the description of this specific implementation and the appended claims, the singular forms of "one", "a kind of" and "the" are intended to also cover the plural forms, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" used herein refers to and covers any and all possible combinations of one or more items in the associated listed items. It will also be understood that the term "comprising" when used in this specification specifies the presence of stated features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or their grouping.

[0097] As used herein, the term "if" may be interpreted to mean "when the antecedent is true" or "when the antecedent is true" or "in response to determining" or "upon determining" or "in response to detecting" that the antecedent is true, depending on the context. Similarly, the phrase "if it is determined that [the antecedent is true]" or "if [the antecedent is true]" or "when [the antecedent is true]" is interpreted to mean "upon determining that the antecedent is true" or "in response to determining" or "upon determining" that the antecedent is true or "when detecting that the antecedent is true" or "in response to detecting" that the antecedent is true, depending on the context.

Claims

1. A method, comprising: At a device having a three-dimensional device coordinate system and comprising a first image sensor, a first display, one or more processors, and a non-transitory memory: capturing a first image of a physical environment using the first image sensor; transforming the first image from a first perspective to a second perspective based on a difference between a first perspective and a second perspective of the first image sensor, wherein the second perspective is a first distance from a position corresponding to a first eye of a user that is less than a second distance between the first perspective and the position corresponding to the first eye of the user; as well as The transformed first image of the physical environment is displayed on the first display. 2 . The method of claim 1 , wherein the second viewing angle and the position corresponding to the first eye of the user have the same coordinate value for at least one dimension of the device coordinate system. 3 . The method of claim 2 , wherein the second viewing angle and the position corresponding to the first eye of the user have the same coordinate values ​​for two dimensions of the device coordinate system.

4. The method of any one of claims 1 to 3, wherein for less than three dimensions of the device coordinate system, the second viewing angle and the position corresponding to the first eye of the user have the same coordinate values. 5 . The method of claim 4 , wherein the second viewing angle and the position corresponding to the first eye of the user have the same coordinate values ​​for less than two dimensions of the device coordinate system. 6 . The method of claim 5 , wherein the second viewing angle and the position corresponding to the user's first eye have different coordinate values ​​for each dimension of the device coordinate system.

7. A method according to any one of claims 1 to 6, wherein a first ratio between (1) the displacement in a first dimension of the device coordinate system between the first perspective and the second perspective and (2) the displacement in the first dimension between the first perspective and the position corresponding to the first eye of the user is different from a second ratio between (1) the displacement in a second dimension of the device coordinate system between the first perspective and the second perspective and (2) the displacement in the second dimension between the first perspective and the position corresponding to the first eye of the user. The method of claim 7 , wherein the first ratio is approximately zero.

9. The method of claim 7, wherein the first ratio is approximately one.

10. The method of claim 7, wherein the first ratio is between zero and one.

11. The method of any one of claims 7 to 10, wherein the first dimension is an x-dimension, the second dimension is a y-dimension, and the first ratio is greater than the second ratio.

12. The method according to any one of claims 1 to 11, further comprising: capturing a second image of the physical environment using a second image sensor; transforming the second image from the third perspective to the fourth perspective based on a difference between the third perspective and the fourth perspective of the second image sensor; as well as The transformed second image of the physical environment is displayed on a second display.

13. The method of claim 12, wherein a vector between the second viewing angle and the position corresponding to the first eye of the user is parallel to a vector between the fourth viewing angle and the position corresponding to the second eye of the user.

14. A method according to claim 12 or 13, wherein the vector between the second viewing angle and the position corresponding to the first eye of the user is parallel to a midpoint vector between (1) a midpoint between the position corresponding to the first eye of the user and the position corresponding to the second eye of the user and (2) a midpoint between the first image sensor and the second image sensor.

15. The method of claim 14, wherein the vector between the second viewing angle and the position corresponding to the first eye of the user has the same magnitude as the midpoint vector.

16. The method according to any one of claims 12 to 15, wherein a vector between the first perspective and the second perspective is parallel to a vector between the third perspective and the fourth perspective.

17. A method according to any one of claims 12 to 16, wherein the fourth viewing angle is less than a third distance corresponding to the position of the user's second eye than a fourth distance between the second image sensor and the position corresponding to the user's second eye.

18. The method of any one of claims 12 to 17, wherein the fourth viewing angle is greater than a third distance from a position corresponding to a second eye of a user than a fourth distance between the second image sensor and the position corresponding to the second eye of the user.

19. A device, comprising: a first image sensor; a first display; non-transitory memory; and One or more processors configured to: capturing a first image of a physical environment using the first image sensor; transforming the first image from a first perspective to a second perspective based on a difference between a first perspective and a second perspective of the first image sensor, wherein a first ratio between (1) a displacement in a first dimension of the device coordinate system between the first perspective and the second perspective and (2) a displacement in the first dimension between the first perspective and a position corresponding to the first eye of the user is different from a second ratio between (1) a displacement in a second dimension of the device coordinate system between the first perspective and the second perspective and (2) a displacement in the second dimension between the first perspective and the position corresponding to the first eye of the user; as well as The transformed first image of the physical environment is displayed on the first display.

20. A non-transitory computer readable memory having encoded thereon instructions that, when executed by one or more processors of a device comprising a first image sensor, a second image sensor, a first display, and a second display, cause the device to: capturing a first image of a physical environment using the first image sensor; capturing a second image of the physical environment using the second image sensor; transforming the first image from the first perspective to the second perspective based on a difference between a first perspective and a second perspective of the first image sensor; transforming the second image from a third perspective to a fourth perspective based on a difference between a third perspective and a fourth perspective of the second image sensor, wherein a vector between the second perspective and a position corresponding to the first eye of the user is parallel to a vector between the fourth perspective and a position corresponding to the second eye of the user; displaying the transformed first image of the physical environment on the first display; as well as The transformed second image of the physical environment is displayed on the second display.

21. A device having a three-dimensional device coordinate system, the device comprising: a first image sensor; a first display; one or more processors; non-transitory memory; and One or more programs stored in the non-transitory memory, which, when executed by the one or more processors, cause the device to perform any of the methods according to claims 1 to 18.

22. A non-volatile memory storing one or more programs which, when executed by one or more processors of a device having a three-dimensional device coordinate system and comprising a first image sensor and a first display, cause the device to perform any of the methods described in claims 1 to 18.

23. A device having a three-dimensional device coordinate system, the device comprising: a first image sensor; a first display; one or more processors; non-transitory memory; and Means for causing the apparatus to perform any of the methods according to claims 1 to 18.

Images