Horizontal image shift for stereoscopic viewing comfort

Abstract

Various implementations disclosed herein include devices, systems, and methods for controlling stereoscopic effects of image content using horizontal shifts of the left-eye and right-eye image content. For example, a process may include obtaining image data including an image to be rendered in a stereoscopic effect at a virtual screen location within a three-dimensional (3D) viewing environment. The process may further obtain depth data corresponding to the distances of the elements of the scene depicted in the image. The process may further determine a horizontal positioning characteristic for rendering the image in a stereoscopic effect at the virtual screen based on the depth data, and a view of the image is rendered in a stereoscopic effect at the virtual screen within the 3D environment based on the horizontal positioning characteristic. The image may additionally be presented with a blur effect, an illumination effect, a gradation effect, and / or a vignetting effect.

Description

Technical Field

[0001] This disclosure relates in its entirety to systems, methods, and apparatus for controlling stereoscopic effects for viewing using parallax shift of left-eye and right-eye image content. Background Technology

[0002] It can improve existing content presentation systems to provide an accurate, expected, and enhanced viewing experience. Summary of the Invention

[0003] The various specific embodiments disclosed herein include devices, systems, and methods for displaying stereoscopic views (e.g., left-eye and right-eye views) of image content at a virtual screen location within a three-dimensional (3D) environment. For example, the stereoscopic view may be presented on a virtual screen at a designated location (e.g., a few feet) in front of a user in an extended reality (XR) environment presented via a head-mounted device (HMD).

[0004] In some implementations, horizontal shifting or movement of the left-eye and / or right-eye image content can be implemented to control or correct the amount or type of stereoscopic effect applied to the image content to provide the user with a comfortable or otherwise desired viewing experience. For example, horizontal shifting of the left-eye and / or right-eye image content can be used to achieve or influence the amount by which objects appear to bulge or recede in a stereoscopic view of the image content.

[0005] In some implementations, the amount of horizontal shift can be selected to provide a comfortable or otherwise desired viewing experience. In some implementations, the amount of horizontal shift can be selected based on the depth of objects depicted in the image content. For example, the depth of an object within the image content can be associated with a determined distance of the object from the image capturing device (e.g., a device that captures image data from which the content is generated). In this case, the depth can be determined based on image analysis or information from the image capturing device (e.g., a camera), such as, in particular, the camera's focal length, interpupillary distance (IPD) parameter, etc.

[0006] In some implementations, the amount of horizontal shift to be applied to the image content may depend on the playback environment, such as, in particular, the size and / or position of the image content and / or virtual screen relative to the viewer, and its position associated with the user's view. In some implementations, the amount of horizontal shift can be determined such that the image content is comfortably viewed at its minimum and maximum depths. In some implementations, when the depth range exceeds a threshold, the horizontal shift can be determined by prioritizing specific portions of the image scene. For example, a saliency map defining visual attributes (e.g., visually interesting parts of the image scene) associated with the user's view can be used to prioritize specific portions of the image scene.

[0007] In some implementations, based on the characteristics of the overall video (e.g., determining that the video does not contain an excessive number of minimum and maximum depth variations), the minimum and maximum depths of all frames of the video can be used to determine the horizontal shift for all frames.

[0008] In some specific implementations, image content can be rendered with blurring effects, lighting effects, vignetting effects, and / or gradient effects to address issues such as overlapping user interfaces (UIs), window violations, and inconsistent left / right eye views.

[0009] In some embodiments, an HMD has a processor (e.g., one or more processors) that executes instructions stored in a non-transitory computer-readable medium to perform a method. The method performs one or more steps or procedures. In some embodiments, image data depicting a scene is obtained. The image data includes one or more images to be rendered stereoscopically at a virtual screen location within a 3D viewing environment. In some embodiments, depth data corresponding to distances to one or more elements of the scene depicted in the one or more images is obtained. The distances are relative to a reference location. In some embodiments, horizontal positioning characteristics for rendering one or more images stereoscopically at a virtual screen are determined based on the depth data. In some embodiments, a view is provided for rendering one or more images at a virtual screen within a 3D environment. The one or more images are rendered stereoscopically based on the horizontal positioning characteristics.

[0010] According to some embodiments, an apparatus includes one or more processors, non-transitory memory, and one or more programs; the one or more programs are stored in the non-transitory memory and configured to be executed by the one or more processors, and the one or more programs include instructions for performing or causing to perform any of the methods described herein. According to some embodiments, a non-transitory computer-readable storage medium stores instructions that, when executed by one or more processors of the apparatus, cause the apparatus to perform or cause to perform any of the methods described herein. According to some embodiments, an apparatus includes: one or more processors, non-transitory memory, and components for performing or causing to perform any of the methods described herein. Attached Figure Description

[0011] To enable those skilled in the art to understand this disclosure, more detailed descriptions can be made with reference to aspects of some exemplary embodiments, some of which are shown in the accompanying drawings.

[0012] Figure 1 Exemplary electronic devices operating in a physical environment according to some specific implementations are illustrated.

[0013] Figure 2 An example pipeline for calculating disparity maps during playback (from stereo image pairs) to achieve real-time disparity management is illustrated according to some specific implementations.

[0014] Figure 3 An example is given of an environment where, according to some specific implementations, a user views 3D content rendered behind a portal.

[0015] Figures 4A to 4E Examples are given of environments representing different types of parallax and associated parallax correction techniques according to some specific implementations.

[0016] Figure 5 This is a flowchart representation of an exemplary method for controlling stereoscopic effects for viewing by providing parallax shift of image content for the left and right eyes according to some specific implementations.

[0017] Figure 6 It is a block diagram based on some specific implementations of electronic devices.

[0018] As is customary practice, various features illustrated in the accompanying drawings may not be drawn to scale. Therefore, for clarity, the dimensions of various features may be arbitrarily expanded or reduced. Furthermore, some drawings may not depict all components of a given system, method, or apparatus. Finally, similar reference numerals may be used throughout the specification and drawings to denote similar features. Detailed Implementation

[0019] Numerous details have been described to provide a thorough understanding of the exemplary embodiments illustrated in the accompanying drawings. However, the drawings illustrate only some exemplary aspects of this disclosure and should not be considered limiting. Those skilled in the art will understand that other effective aspects and / or variations do not include all the specific details described herein. Furthermore, well-known systems, methods, components, devices, and circuits have not been described exhaustively so as not to obscure further relevant aspects of the exemplary embodiments described herein.

[0020] Figure 1 An exemplary electronic device 105 operating in physical environment 100 is illustrated. Figure 1 In the example, physical environment 100 is a room including table 120. Electronic device 105 may include one or more cameras, microphones, depth sensors, or other sensors that can be used to capture and evaluate information about physical environment 100 and objects within it, as well as information about user 102 of electronic device 105. Information about physical environment 100 and / or user 102 can be used to provide visual and audio content, and / or identify the current location of physical environment 100 and / or the location of the user within physical environment 100.

[0021] In some implementations, a view of an extended reality (XR) environment may be provided to one or more participants (e.g., user 102 and / or other participants not shown) via an electronic device 105 (e.g., a wearable device such as an HMD). Such an XR environment may include a view of a 3D environment generated based on camera images and / or depth camera images of the physical environment 100, and a representation of user 102 based on camera images and / or depth camera images of user 102. Such an XR environment may include virtual content positioned at a 3D location relative to a 3D coordinate system associated with the XR environment, which may correspond to the 3D coordinate system of the physical environment 100.

[0022] In some implementations, the electronic device 105 can be configured to display a stereoscopic view of image content at a virtual screen location within a 3D environment. The stereoscopic view can be adjusted by shifting the horizontal position of the image content within the 3D (e.g., XR) viewing environment to control the stereoscopic effect applied to the image content for the user's viewing.

[0023] In some implementations, image data depicting the scene may be obtained, in particular, from image sensors such as cameras, data storage systems, etc. Image data may include one or more images (e.g., frames of a video) that will be presented in a stereoscopic effect at a virtual screen location within the 3D viewing environment. For example, image data may include photographs, videos, etc.

[0024] In some implementations, depth data (e.g., generated based on analysis of image data, camera focal length, IPD parameters, etc.) corresponding to the distances to elements (e.g., objects) of the scene depicted in the image can be determined or obtained. The distances to the elements of the scene can correspond to reference locations such as, in particular, the position of the capturing device or camera. In some implementations, the distances can be considered relative to the position of a user or viewer of a virtual screen or image presented via a wearable device such as an HMD.

[0025] In some implementations, depth data can be used to determine horizontal positioning characteristics, such as displacement or parallax, for presenting images at the virtual screen with the applied stereoscopic effect. Horizontal positioning characteristics can correspond to, in particular, the size and / or position of the image content / virtual screen relative to the viewer of the content, where the viewer is currently looking, etc.

[0026] In some implementations, a view of an image can be presented at a virtual screen within a 3D environment, such that the image is presented in a stereoscopic effect based on horizontal positioning characteristics. For example, a view of an image presented in a stereoscopic effect may include horizontally shifting one or more of the left-eye or right-eye images in a stereoscopic image pair to control the amount of protrusion or indentation of objects in the image (relative to their 3D position).

[0027] In some implementations, the amount of horizontal shift can be determined based on the image content (of the image) located at the minimum and maximum depths of the depth data.

[0028] In some implementations, the depth range of the depth data may be determined to exceed a threshold, and therefore, the amount of horizontal shift may be determined based on prioritizing a specified portion of the image data. This prioritization may be based on a saliency map defining visual attributes associated with the user's view.

[0029] In some implementations, images can be additionally rendered with blur effects, lighting effects, vignetting effects, and / or gradient effects to address issues such as UI overlap, window conflicts, and inconsistencies between left and right eye views.

[0030] Figure 2 An example pipeline 200 is illustrated, which calculates a disparity map 207 (e.g., depth) from a stereoscopic image pair 202 (i.e., left-eye image 202a and right-eye image 202b) of image content (e.g., spatial photograph or video content) during playback (for a user) to achieve real-time disparity management. The left-eye image 202a (e.g., a frame of a video) includes a left-eye view of background content 205a (e.g., background content such as plants, flowers, and grass) and foreground content 204a (e.g., a bird). Similarly, the right-eye image 202b (e.g., a frame of a video) includes a right-eye view of background content 205b (e.g., background content such as plants, flowers, and grass) and foreground content 204b (e.g., a bird).

[0031] In some specific implementations, by obtaining information from users (e.g., Figure 1 A stereoscopic image pair 202 is generated by using a left-eye view (i.e., left-eye image 202a) associated with the left-eye viewpoint of user 102 and a right-eye view (i.e., right-eye image 202b) associated with the right-eye viewpoint, wherein the left-eye view and the right-eye view are relative to the device displaying the left-eye image 202a and the right-eye image 202b (e.g., ...). Figure 1 The device 110 or 105). Therefore, when viewed via, for example, an HMD, the combination of the left-eye image 202a and the right-eye image 202b forms a stereoscopic output image pair 202, which depicts a 3D video / representation of the content of the stereoscopic image pair 202 (e.g., background content such as plants, flowers and grass and foreground content such as birds) for viewing on a stereoscopic display of a device such as an HMD.

[0032] In some implementations, (e.g., via image analysis) the left-eye image 202a and the right-eye image 202b are analyzed to calculate a disparity (depth) map 207 representing the amount of disparity to be adjusted to provide sufficient user comfort. The disparity map 207 may include a depth image (e.g., a low-resolution 3D model) that includes depth values ​​at the original pixel locations mapped to a subset of the pixel locations of the stereo image pair 202.

[0033] In some embodiments, region 207a represents a region of parallax map 207 associated with a first depth region (e.g., a background region such as grass). In some embodiments, region 207b represents a region of parallax map 207 associated with a second depth region (e.g., an intermediate region of medium depth such as plants). In some embodiments, region 207c represents a region of parallax map 207 associated with a third depth region (e.g., a foreground region such as flowers and birds).

[0034] In some implementations, depth can be calculated based on information from the camera during image capture. For example, information from the camera may include camera focal length, interpupillary distance (IPD) parameters, etc.

[0035] In some implementations, the minimum and maximum depths 209 (relative to the region of interest) of all frames of the video (image content) (e.g., creating a depth range for the entire video) can be used to determine the horizontal shift to be used for all frames. Similarly, scene data can be used to determine the horizontal shift. For example, indoor and outdoor scene data, camera focus, camera depth of field, etc., can be used to determine the horizontal shift.

[0036] Figure 3 An environment 300 is illustrated, including a user viewpoint location 302 for providing a view to the user (e.g., via an HMD). The view from the user viewpoint location 302 will include 3D content 308 rendered behind a portal 304 within the 3D environment 300. Figure 3 Examples of positional relationships (e.g., between user viewpoint 302, 3D content 308, and portal 304) are illustrated, which can be used to determine how (e.g., in an HMD) the user view of the 3D environment 300 is displayed, such that the 3D content 308 is displayed within that view in a manner that provides a comfortable or otherwise desired user experience.

[0037] In some embodiments, the size of 3D content 308 rendered in the user view (e.g., on an HMD) (e.g., a portion of 3D content 308, such as an object (e.g., a bird rendering) 310) can be determined. In some embodiments, rendering 3D content 308 behind a portal 304 in the user view (e.g., on an HMD) allows the system to determine the size of the 3D content 308. In some embodiments, the size of the portal 304 and the associated distance 312a of the 3D content 308 relative to the portal 304 can also be determined. In some embodiments, the distance between the user's eyes (e.g., IPD) can be determined. Similarly, the distance 314 between the user's viewpoint position 302 and the portal 304 and the distance 312 between the user's viewpoint position 302 and the 3D content 308 and / or the object 310 can be determined. The aforementioned sizes and distances (e.g., distances 312, 312a, 314, IPD, etc.) of 3D content 308, portal 304, etc., can change continuously during content viewing because portal 304 can move or resize within environment 300 based on different viewing conditions. For example, different viewing conditions can be caused by, in particular, movement of the user's head, which can change the user's viewpoint position 302 within the 3D environment, or can cause different applications associated with different settings and different hardware platforms to affect playback conditions, which may require parallax adjustments to 3D content 308.

[0038] Figures 4A to 4E Examples of 3D environments 400a-400e are shown based on some specific implementations of different types of parallax and associated parallax correction techniques. Figures 4A to 4E Examples of 3D environments 400a-400e are shown, which include a user viewpoint position 402 for a view to be provided to the user (e.g., via an HMD). The view from the user viewpoint position 402 will include 3D image content (e.g., a 3D object such as a bird 410) rendered behind a portal 404 within the 3D environment 400. Similarly, Figures 4A to 4E Examples of positional relationships (e.g., between user viewpoint 402, 3D content 408, and portal 404) are illustrated. These positional relationships can be used to determine how (e.g., in an HMD) the user view of the 3D environments 400a-400e is displayed, such that the 3D content 408 is displayed within that view in a manner that provides a comfortable or otherwise desired user experience.

[0039] Figure 4AAn example is illustrated of generating 3D image content 408 by presenting a pair of stereoscopic images 409a (i.e., left-eye image 408a and right-eye image 408b) representing image content (e.g., spatial photograph or video content) during playback for user 402. The left-eye image 408a (e.g., a frame of a video) includes a left-eye view of background content 411a (e.g., background content such as flowers) and foreground content 410a (e.g., a bird). Similarly, the right-eye image 408b (e.g., a frame of a video) includes a right-eye view of background content 411b (e.g., background content such as flowers) and foreground content 410b (e.g., a bird).

[0040] In some specific implementations, a stereoscopic image pair 409a can be generated by generating a left-eye view (i.e., left-eye image 408a) associated with the left-eye viewpoint of user view 402 (on the HMD) and a right-eye view (i.e., right-eye image 408b) associated with the right-eye viewpoint, the user view being relative to a device displaying the left-eye image 408a and the right-eye image 408b via portal 404 (e.g., Figure 1 The device 105). Therefore, when viewed via, for example, an HMD, the combination of the left-eye image 408a and the right-eye image 408b forms a stereoscopic output image pair 409a, which depicts a 3D video / representation of the content of the stereoscopic image pair 409a (e.g., background content 411 such as flowers and foreground content such as birds 410) for viewing on a stereoscopic display of a device such as an HMD.

[0041] In some embodiments, the left-eye image 408a and the right-eye image 408b provide a stereoscopic effect (e.g., via a view on an HMD) by controlling the amount by which the stereoscopic image appears to bulge (e.g., bird 410) or recess (e.g., the stereoscopic image appears to 409a's background content 411). In some embodiments, the amount by which the stereoscopic image appears to bulge (e.g., bird 410) or recess (e.g., the stereoscopic image appears to 409a's background content 411) can be controlled by adjusting a horizontal shift or movement relative to the left-eye image 408a and the right-eye image 408b (e.g., in horizontal directions 426a and 426b or 428a and 428b). In some embodiments, the observed parallax may depend on this horizontal shift and the depth of objects in the view (e.g., relative to directions 425 and 427). In some embodiments, parallax may affect user comfort, causing objects to appear more rounded or elongated depending on how far the object is relative to the view and the nature or amount of the horizontal shift. Parallax can additionally affect stereoscopic comfort and the sense of proportion.

[0042] In some implementations, the amount of horizontal shift used to correct for parallax problems may be based on the type of content that user 402 is viewing (e.g., a bird in a background scene). In some implementations, the view of the content may be altered for a video file or stream based on the size of portal 404 or the distance between the user's eyes and the portal. For example, calculations related to the gaze direction (of the user's eyes) regarding the parallax of a single pixel (e.g., the beak of bird 410) may be used to extend the user's gaze (associated with both eyes) via rays 414 and 416, so that the object (i.e., bird 410) can be perceived in 3D at location 418, the intersection of rays 414 and 416.

[0043] Figure 4B An example of a 3D environment 400b is shown. (Compared to...) Figure 4A Compared to the 3D environment 400a, environment 400b represents a horizontal shift in directions 428a and 428b such that the background content 411a and foreground content 410a are located at a greater distance from each other (for the stereoscopic image pair 409b) in the left-eye image 408a and right-eye image 408b. Therefore, because the user's eyes are looking further apart, the rays 414 and 416 do not intersect, thus preventing objects (e.g., bird 410) from being rendered in 3D. Similarly, this can lead to parallax, including double vision (e.g., relative to foreground content 410a and 410b (e.g., bird) and background content 411a and 411b (e.g., flower)), caused by the excessive horizontal distance between the left-eye image 408a and right-eye image 408b. In some specific implementations, a horizontal shift can be applied to move the left-eye image 408a and the right-eye image 408b in directions 426a and 426b, thereby bringing them closer together so that rays 414 and 416 begin to intersect, thus resolving parallax and realizing objects (such as, for example) Figure 4A 3D view of the bird (310).

[0044] Figure 4C An example of a 3D environment 400c is shown. (Compared to...) Figure 4B Compared to the 400b 3D environment, Figure 4CThe 3D environment 400c represents a horizontal shift in directions 426a and 426b such that foreground content 410a and 410b and background content 411a and 411b are positioned closer to each other, causing the light rays 414 and 416 to intersect at position 418b in the left-eye image 408a and right-eye image 408b (of the stereoscopic image pair 409c). Therefore, the object 410 (e.g., a bird) is presented in 3D such that the portal 404 is perceived as being behind the object 410 but in front of the position of the stereoscopic image pair 409c, resulting in a depth parallax that may be uncomfortable for the user. In some specific implementations, a horizontal shift can be applied to move the left-eye image 408a and right-eye image 408b in directions 428a and 428b, further separating them (horizontally), causing the object 410 to move in direction 427 to a position inside or behind the portal 404, thereby resolving the parallax and achieving a comfortable 3D view of the object 410.

[0045] Figure 4D An example of a 3D environment 400d is shown. (Compared to...) Figure 4B Compared to the 400b 3D environment, Figure 4D The 3D environment 400d represents a window conflict problem, which causes a point of interest (e.g., a bird in the foreground content 410a and 410b) to be visible in one eye of the user but not in the other. For example, the user's left eye (associated with ray 414) may be able to see the bird, but the user's right eye (associated with ray 416) may not be able to see the bird because the side 404a of the portal 404 is obstructing the user's right eye's field of vision. In some implementations, this parallax can be resolved by applying a feathering effect to the inner edge of the portion 404a to dynamically adjust the width of the portal 404 and achieve a view of the bird.

[0046] Figure 4E An example of a 3D environment 400e is shown. (Compared to...) Figure 4C Compared to the 3D environment of 400c, Figure 4E The 3D environment 400e represents excessive negative parallax associated with rendering object 410 at a position too close to the user's eyes, causing the user to squint to see object 410. In this case, a horizontal shift can be applied to move the left-eye image 408a and right-eye image 408b in directions 428a and 428b or 426a and 426b, thereby moving object 410 to a position inside or behind portal 404 in direction 427, thus resolving the parallax and achieving a comfortable 3D view of object 410.

[0047] Figure 5This is a flowchart illustrating an exemplary method 500 for controlling stereoscopic effects for viewing by providing parallax shift of image content for the left and right eyes according to some specific implementations. In some specific implementations, method 500 is provided by a device (such as a mobile device, desktop computer, laptop computer, HMD, or server device, e.g., Figure 1 The device 110) performs the operation. In some specific implementations, the device has a screen for displaying images and / or a screen for viewing stereoscopic images, such as a head-mounted display (HMD, such as, for example) Figure 1 (Device 105). In some embodiments, method 500 is performed by processing logic components, including hardware, firmware, software, or combinations thereof. In some embodiments, method 500 is performed by a processor that executes code stored in a non-transitory computer-readable medium (e.g., memory). Each block in method 500 can be enabled and executed in any order.

[0048] In some implementations, method 500 provides a user viewpoint position for a view to be provided to a user (e.g., via an HMD). The view from the user viewpoint position may include 3D content rendered behind a portal within a 3D environment. In some implementations, positional relationships (e.g., between the user viewpoint position, the 3D content, and the portal) can be used to determine how (e.g., in the HMD) the user view of the 3D environment is displayed such that the 3D content is displayed within that view in a manner that provides a comfortable or otherwise desired user experience.

[0049] At box 502, method 500 obtains image data depicting the scene. In some specific implementations, the image data (e.g., photographs, videos, etc.) includes a virtual screen that will be used within the three-dimensional (3D) viewing environment (e.g., as shown in the image). Figure 4A The image at the location of the described portal 404 is presented in a stereoscopic effect (such as regarding...). Figure 4A The described stereoscopic image is for 409a).

[0050] At box 504, method 500 obtains depth data corresponding to the distances to one or more elements of the scene depicted in the image (e.g., as per the context of...). Figure 2 The disparity diagram 207 is described. In some specific implementations, the distance can be relative to a reference position, such as regarding... Figure 1 The described location of the capture device / camera. In some specific implementations, distance may be considered relative to the HMD user / viewer's location / viewpoint relative to the virtual screen / image.

[0051] In some specific implementations, depth data can be determined based on the analysis of image data.

[0052] In some implementations, depth data can be determined based on data from the image sensor used to capture image data. Examples include camera focal length and IPD parameters.

[0053] At box 506, method 500 is based on, as per [the relevant information] Figure 2 The described depth data determines the horizontal positioning characteristics used to render images in a stereoscopic effect on a virtual screen.

[0054] In some specific implementations, the horizontal positioning characteristic may correspond to the amount of horizontal displacement applied to at least one image in the image.

[0055] In some implementations, the amount of horizontal shift can be determined based on the characteristics of the playback environment associated with the presented view. For example, the size and / or position of the image content / virtual screen relative to the viewer, where the viewer is looking, etc., as per [the context of the image]. Figure 3 As described.

[0056] In some specific implementations, the amount of horizontal shift can be determined based on the image content of at least one image located at the minimum and maximum depths of the depth data. For example, as regarding Figure 2 The minimum and maximum depths 209 of the described parallax map 207.

[0057] In some implementations, the amount of horizontal shift can be determined based on the image content of all images in the image located at the minimum and maximum depths of the depth data.

[0058] In some implementations, the depth range of the depth data may exceed a threshold, and the amount of horizontal shift may be determined based on a specified portion of the image data that is given priority.

[0059] In some implementations, specific portions of the image data can be prioritized based on a saliency map that defines visual attributes associated with the user's view.

[0060] At box 508, method 500 presents a view of the image on a virtual screen within the 3D environment, such that the image is presented in a stereoscopic effect based on horizontal positioning characteristics, as per [reference to...]. Figures 4A to 4E As described.

[0061] In some specific implementations, a blurring effect can be used to present images to resolve issues such as UI overlap, window conflicts, and inconsistencies between left and right eye views.

[0062] In some specific implementations, lighting effects can be used to render images to resolve issues such as UI overlap, window conflicts, and inconsistent left / right eye views.

[0063] In some implementations, a vignetting effect can be used to render images to resolve issues such as UI overlap, window conflicts, and inconsistencies in left / right eye views. For example, brightness or saturation can be reduced at the edges or periphery of an image compared to the center of a video frame.

[0064] In some implementations, images can be rendered with a gradient effect (e.g., around the perimeter of the image) to resolve UI overlap, window conflicts, and inconsistencies in left / right eye views.

[0065] In some implementations, images can be rendered using any combination of blur effects, lighting effects, vignetting effects, and / or gradient effects to resolve UI overlap, window conflicts, and inconsistencies in left / right eye views.

[0066] Figure 6 This is a block diagram of example device 600. Device 600 illustrates a device used for... Figure 1 Exemplary device configurations of electronic devices 105 and 110. Although certain specific features have been illustrated, those skilled in the art will recognize from this disclosure that various other features have not been illustrated for the sake of brevity and so as not to obscure further relevant aspects of the specific embodiments disclosed herein. Therefore, as a non-limiting example, in some specific implementations, device 600 includes one or more processing units 602 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, and / or processing cores, etc.), one or more input / output (I / O) devices and sensors 606, one or more communication interfaces 608 (e.g., USB, FireWire, Thunderbolt, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, IR, Bluetooth, ZigBee, SPI, I2C, and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 610, output devices (e.g., one or more displays) 612, one or more internal and / or external image sensor systems 614, memory 620, and one or more communication buses 604 for interconnecting these components and various other components.

[0067] In some embodiments, one or more communication buses 604 include circuitry for interconnecting system components and controlling communication between system components. In some embodiments, one or more I / O devices and sensors 606 include at least one of the following: an inertial measurement unit (IMU), an accelerometer, a magnetometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., a blood pressure monitor, a heart rate monitor, a blood oxygen sensor, a blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptic engine, one or more depth sensors (e.g., structured light, time-of-flight, etc.), one or more cameras (e.g., an inward-facing camera and an outward-facing camera of an HMD), one or more infrared sensors, one or more thermal sensors, and / or the like.

[0068] In some embodiments, one or more output devices 612 include one or more displays configured to present a view of a 3D environment to a user. In some embodiments, one or more displays are configured to present a view of a physical environment, a graphical environment, an extended reality environment, etc., to a user. In some embodiments, one or more displays are configured to present content to a user (determined based on the user / object's position within the physical environment). In some embodiments, one or more displays 612 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light-emitting field-effect transistor (OLET), organic light-emitting diode (OLED), surface-conducting electron emission display (SED), field emission display (FED), quantum dot light-emitting diode (QD-LED), microelectromechanical systems (MEMS), and / or similar display types. In some embodiments, one or more displays correspond to waveguide displays such as diffraction, reflection, polarization, and holography. In one example, device 600 includes a single display. In another example, device 600 includes displays for each of the user's eyes.

[0069] In some embodiments, one or more output devices 612 include one or more audio generating devices. In some embodiments, one or more output devices 612 include one or more speakers, surround sound speakers, speaker arrays, or headphones for generating spatialized sound (e.g., 3D audio effects). Such devices can virtually place sound sources in a 3D environment, including behind, above, or below one or more listeners. Generating spatialized sound may involve transforming sound waves (e.g., using head-related transfer functions (HRTF), reverberation, or cancellation techniques) to simulate natural sound waves (including reflections from walls and floors) emanating from one or more points in the 3D environment. Spatialized sound can induce the listener's brain to interpret sound as if it occurred at one or more points in the 3D environment (e.g., from one or more specific sound sources), even if the actual sound may be generated by speakers in other locations. One or more output devices 612 may additionally or alternatively be configured to generate haptic feedback.

[0070] In some embodiments, one or more image sensor systems 614 are configured to acquire image data corresponding to at least a portion of the physical environment 100. For example, one or more image sensor systems 614 include one or more RGB cameras (e.g., having a complementary metal-oxide-semiconductor (CMOS) image sensor or a charge-coupled device (CCD) image sensor), monochrome cameras, IR cameras, depth cameras, event-based cameras, etc. In various embodiments, one or more image sensor systems 614 also include an illumination source emitting light, such as a flash. In various embodiments, one or more image sensor systems 614 also include an on-camera image signal processor (ISP) configured to perform multiple processing operations on the image data.

[0071] In some embodiments, device 600 includes an eye-tracking system for detecting eye positioning and eye movement (e.g., eye gaze detection). For example, the eye-tracking system may include one or more infrared (IR) light-emitting diodes (LEDs), an eye-tracking camera (e.g., a near-infrared (NIR) camera), and an illumination source (e.g., an NIR light source) that emits light (e.g., NIR light) towards the user's eyes. Furthermore, the illumination source of device 600 may emit NIR light to illuminate the user's eyes, and the NIR camera may capture images of the user's eyes. In some embodiments, the images captured by the eye-tracking system may be analyzed to detect the positioning and movement of the user's eyes, or to detect other information about the eyes such as pupil dilation or pupil diameter. Furthermore, the gaze point estimated from the eye-tracking images enables gaze-based interaction with content displayed on a near-eye display of device 600.

[0072] Memory 620 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices. In some embodiments, memory 620 includes non-volatile memory, such as one or more disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 620 optionally includes one or more storage devices remotely located to one or more processing units 602. Memory 620 includes a non-transitory computer-readable storage medium.

[0073] In some embodiments, memory 620 or a non-transitory computer-readable storage medium of memory 620 stores an optional operating system 630 and one or more instruction sets 640. Operating system 630 includes procedures for handling various basic system services and for performing hardware-related tasks. In some embodiments, instruction set 640 includes executable software defined by binary information stored in charge. In some embodiments, instruction set 640 is software executable by one or more processing units 602 to implement one or more of the techniques described herein.

[0074] Instruction set 640 includes horizontal positioning instruction set 642 and parallax correction rendering instruction set 644. Instruction set 640 can be embodied in a single software executable file or multiple software executable files.

[0075] The horizontal positioning instruction set 642 is configured with instructions that can be executed by the processor to determine horizontal positioning characteristics, such as shifts, for rendering an image in terms of resolved parallax.

[0076] The parallax correction rendering instruction set 644 is configured with instructions that can be executed by the processor to render an image in a stereoscopic effect based on horizontal positioning characteristics.

[0077] Although instruction set 640 is shown as residing on a single device, it should be understood that in other specific implementations, any combination of elements may reside in separate computing devices. Furthermore, Figure 6 This is intended more as a functional description of various features present in a particular implementation than as a structural diagram of the specific implementation described herein. As will be appreciated by those skilled in the art, the items shown individually can be combined, and some items can be separated. The actual number of instruction sets and how features are allocated therein will vary depending on the specific implementation and may depend in part on the specific combination of hardware, software, and / or firmware chosen for that particular implementation.

[0078] Those skilled in the art will understand that well-known systems, methods, components, devices, and circuits have not been described exhaustively so as not to obscure more relevant aspects of the specific embodiments of the examples described herein. Furthermore, other effective aspects and / or variations do not include all the details in the specific details described herein. Therefore, several details are described to provide a thorough understanding of the exemplary aspects illustrated in the accompanying drawings. Moreover, the drawings only illustrate some exemplary embodiments of this disclosure and should not be considered limiting.

[0079] While this specification contains numerous specific implementation details, these details should not be construed as limiting the scope of any invention or potentially claimed content, but rather as descriptions of features specific to particular embodiments of a particular invention. Certain features described in the context of different embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, while certain features may be described above as functioning in certain combinations and even initially claimed in this manner, one or more features of a claimed combination may be removed from that combination in some cases, and the claimed combination may involve sub-combinations or variations thereof.

[0080] Similarly, although the operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring such operations to be performed in a sequential order or the specific order shown, or requiring all illustrated operations to achieve the desired result. In some situations, multitasking and parallel processing may be advantageous. Furthermore, the partitioning of the various system components in the above embodiments should not be construed as requiring such partitioning in all embodiments, and it should be understood that the described program components and systems may generally be integrated together in a single software product or packaged into multiple software products.

[0081] Therefore, specific embodiments of the subject matter have been described. Other embodiments are also within the scope of the following claims. In some cases, the actions described in the claims can be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific order or sequence shown to achieve the desired result. In some embodiments, multitasking and parallel processing may be advantageous.

[0082] The embodiments of the subject matter and operation described in this specification may be implemented in digital electronic circuits or in computer software, firmware, or hardware (including the structures disclosed in this specification and their structural equivalents) or in a combination thereof. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, such as one or more modules of computer program instructions encoded on a computer storage medium for execution by or control of the operation of a data processing device. Alternatively or additionally, the program instructions may be encoded on artificially generated propagating signals, such as machine-generated electrical, optical, or electromagnetic signals, which are generated to encode information for transmission to a suitable receiver device for execution by the data processing device. The computer storage medium may be or be included in a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination thereof. Furthermore, although the computer storage medium is not a propagating signal, it may be a source or destination of computer program instructions encoded in artificially generated propagating signals. The computer storage medium may also be or be included in one or more separate physical components or media (e.g., multiple CDs, disks, or other storage devices).

[0083] The term "data processing apparatus" encompasses all kinds of devices, apparatuses, and machines for processing data, including, for example, programmable processors, computers, systems-on-a-chip, or many or combinations of the foregoing. The apparatus may include special-purpose logic circuitry (e.g., FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits)). In addition to hardware, the apparatus may also include code that creates an execution environment for the computer program under consideration, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, cross-platform runtime environments, virtual machines, or combinations thereof. The apparatus and execution environment can implement a variety of different computing model infrastructures, such as web services, distributed computing, and grid computing infrastructures. Unless otherwise specifically stated, it should be understood that throughout this specification, discussions using terms such as "processing," "computing," "calculating," "determining," and "identifying" refer to the actions or processes of computing devices, such as one or more computers or similar electronic computing devices, that manipulate or convert data represented as physical electronic or magnetic quantities within the memory, registers, or other information storage devices, transmitting devices, or display devices of a computing platform.

[0084] The one or more systems discussed herein are not limited to any particular hardware architecture or configuration. A computing device may include any suitable arrangement of components that provide results conditioned on one or more inputs. Suitable computing devices include computer systems based on multi-purpose microprocessors that access stored software that programs or configures the computing system from a general-purpose computing device to a special-purpose computing device that implements one or more specific embodiments of the subject matter of this invention. The teachings contained herein may be implemented in the software used for programming or configuring the computing device using any suitable programming, scripting, or other type of language or combination of languages.

[0085] Specific implementations of the methods disclosed herein can be performed in the operation of such computing devices. The order of the boxes presented in the examples above can be varied; for example, the boxes can be reordered, combined, and / or divided into sub-blocks. Some boxes or processes can be executed in parallel. The operations described in this specification can be implemented as operations performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.

[0086] The use of "applies to" or "configured to" in this document implies open and inclusive language, which does not exclude applicability to or configuration to devices performing additional tasks or steps. Furthermore, the use of "based on" implies openness and inclusivity, as processes, steps, calculations, or other actions "based on" one or more of the stated conditions or values ​​may in practice be based on additional conditions or values ​​beyond those stated. The headings, lists, and numbering included herein are for illustrative purposes only and are not intended to be restrictive.

[0087] It will also be understood that while terms such as "first," "second," etc., may be used in this document to describe various elements, these elements should not be limited by these terms. These terms are merely used to distinguish one element from another. For example, a first node may be called a second node, and similarly, a second node may be called a first node, changing the meaning of the description, provided that all occurrences of "first node" are consistently renamed and all occurrences of "second node" are consistently renamed. First nodes and second nodes are both nodes, but they are not the same node.

[0088] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the claims. As used in the description of these embodiments and in the appended claims, the singular forms “a,” “an,” and “the” are intended to also cover the plural forms unless the context clearly indicates otherwise. It will also be understood that the term “and / or” as used herein refers to and covers any and all possible combinations of one or more of the associated listed items. It will be further understood that the term “comprising,” as used in this specification, specifies the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0089] As used herein, the term "if" can be interpreted as meaning "when the prerequisite is true" or "when the prerequisite is true" or "in response to determination" or "according to determination" or "in response to detection" that the prerequisite is true, depending on the context. Similarly, the phrase "if it is determined [the prerequisite is true]" or "if [the prerequisite is true]" or "when [the prerequisite is true]" can be interpreted as meaning "when it is determined that the prerequisite is true" or "in response to determination" or "according to determination" that the prerequisite is true or "when it is detected that the prerequisite is true" or "in response to detection" that the prerequisite is true, depending on the context.

Claims

1. A method, the method comprising: At head-mounted devices (HMDs) with processors: Obtain image data depicting the scene, wherein the image data includes one or more images that will be presented in a stereoscopic effect at a virtual screen location within a three-dimensional (3D) viewing environment; Obtain depth data corresponding to the distances to one or more elements of the scene depicted in the one or more images, wherein the distances are relative to a reference position; Based on the depth data, determine the horizontal positioning characteristics for presenting the one or more images at the virtual screen with the stereoscopic effect; as well as A view of one or more images is presented at the virtual screen within the 3D environment, wherein the one or more images are presented with the stereoscopic effect based on the horizontal positioning characteristics.

2. The method according to claim 1, wherein the depth data is determined based on the analysis of the image data.

3. The method of claim 1, wherein the depth data is determined based on data from an image sensor used to capture the image data.

4. The method of claim 1, wherein the horizontal positioning feature corresponds to the amount of horizontal shift applied to at least one of the one or more images.

5. The method of claim 4, wherein the amount of horizontal shift is determined based on characteristics of the playback environment associated with presenting the view.

6. The method of claim 4, wherein the amount of horizontal shift is determined based on the image content of at least one of the one or more images located at the minimum and maximum depths of the depth data.

7. The method of claim 4, wherein the amount of horizontal shift is determined based on the image content of all images in the one or more images located at the minimum and maximum depths of the depth data.

8. The method of claim 4, wherein the depth range of the depth data exceeds a threshold, and wherein the amount of horizontal shift is determined based on preferential consideration of a specified portion of the image data.

9. The method of claim 8, wherein the specified portion of the image data is preferentially considered based on a saliency map defining visual attributes associated with the user's view.

10. The method of claim 1, wherein the one or more images are presented with a blurred effect.

11. The method of claim 1, wherein the one or more images are presented with lighting effects.

12. The method of claim 1, wherein the one or more images are presented with a vignetting effect.

13. The method of claim 1, wherein the one or more images are presented with a gradient effect.

14. A head-mounted device (HMD), said head-mounted device (HMD) comprising: Non-transitory computer-readable storage medium; and One or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium includes program instructions that, when executed on the one or more processors, cause the HMD to perform operations, the operations including: Obtain image data depicting the scene, wherein the image data includes one or more images that will be presented in a stereoscopic effect at a virtual screen location within a three-dimensional (3D) viewing environment; Obtain depth data corresponding to the distances to one or more elements of the scene depicted in the one or more images, wherein the distances are relative to a reference position; Based on the depth data, determine the horizontal positioning characteristics for presenting the one or more images at the virtual screen with the stereoscopic effect; and A view of one or more images is presented at the virtual screen within the 3D environment, wherein the one or more images are presented with the stereoscopic effect based on the horizontal positioning characteristics.

15. The HMD of claim 14, wherein the depth data is determined based on analysis of the image data.

16. The HMD of claim 14, wherein the depth data is determined based on data from an image sensor used to capture the image data.

17. The HMD of claim 14, wherein the horizontal positioning feature corresponds to the amount of horizontal shift applied to at least one of the one or more images.

18. The HMD of claim 17, wherein the amount of horizontal shift is determined based on characteristics of the playback environment associated with presenting the view.

19. The HMD of claim 17, wherein the amount of horizontal shift is determined based on the image content of at least one of the one or more images located at the minimum and maximum depths of the depth data.

20. A non-transitory computer-readable storage medium storing program instructions executable via one or more processors of a head-mounted device (HMD) to perform operations including: Obtain image data depicting the scene, wherein the image data includes one or more images that will be presented in a stereoscopic effect at a virtual screen location within a three-dimensional (3D) viewing environment; Obtain depth data corresponding to the distances to one or more elements of the scene depicted in the one or more images, wherein the distances are relative to a reference position; Based on the depth data, determine the horizontal positioning characteristics for presenting the one or more images at the virtual screen with the stereoscopic effect; as well as A view of one or more images is presented at the virtual screen within the 3D environment, wherein the one or more images are presented with the stereoscopic effect based on the horizontal positioning characteristics.

Images