Application of single image to stereo conversion

Abstract

The invention relates to application of single image to stereo conversion. Various implementations disclosed herein include devices, systems, and methods for dynamically applying 3D effects to 2D assets. For example, a process may obtain an image depicting two-dimensional (2D) content. The process may also determine to apply a three-dimensional (3D) effect to the image via a head-mounted device (HMD). The process may also apply the 3D effect to the image according to the determination, presenting a view of the 3D environment including the image. The image may be positioned at a location within the 3D environment, and the view may depict the image using the 3D effect.

Description

Technical Field

[0001] The present disclosure generally relates to systems, methods, and apparatus for dynamically applying 3-dimensional (3D) effects to 2-dimensional (2D) assets. Background Art

[0002] Existing techniques for viewing 2D image content may not adequately facilitate viewing of such image content with improved realism or other effects of the image content to provide a desired viewing experience. Summary of the invention

[0003] Various implementations disclosed herein include devices, systems, and methods for dynamically applying 3D effects to 2D assets, such as images (e.g., standard images, panoramic images, etc.) or videos, in particular, from a photo library, a web page, a web browser, and the like. The 3D effect may be applied to the 2D asset in real time or after an elapsed time period for display via a head mounted device (HMD). Application of the 3D effect may be manually triggered in response to, in particular, a user activating a button or switch. Alternatively, application of the 3D effect may be triggered by a user initiating a gesture in response to a glow effect or icon being presented adjacent to the image on a web page. For example, a glow effect or icon presented adjacent to the image on a web page may be triggered in response to a detected user gaze direction relative to an image depicting the 2D asset (e.g., the user is looking at the 2D asset). In response to the glow effect or icon being presented adjacent to the image, the user may activate a button, a switch, an icon, an input command (e.g., a gesture such as a picking gesture, etc.), and the like.

[0004] In some implementations, the application of the 3D effect may be automatically triggered based on a detected user gaze direction relative to an image depicting the 2D asset (e.g., the user is looking at the 2D asset). Alternatively, the application of the 3D effect may be automatically triggered based on the content depicted in the 2D asset. For example, content (e.g., of a photograph) depicting objects in nature (e.g., mountains, canyons, rivers, etc.) and / or people may automatically trigger the application of the 3D effect to the 2D asset, while a receipt (in a photograph) will not trigger the application of the 3D effect to the 2D asset. In some implementations, the application of the 3D effect may be automatically triggered based on an expected quality of the 3D effect. For example, the expected quality of the 3D effect for a photograph may be determined to be low (e.g., producing error-prone or noisy results), and therefore the application of the 3D effect is not triggered. In some implementations, the application of the 3D effect may be automatically triggered based on metadata (e.g., including instructions) of the 2D asset. In some implementations, application of the 3D effect can be automatically triggered based on settings initiated by an application developer within an application that renders an image depicting the 2D content.

[0005] In some implementations with respect to web-based applications, a web browser may be configured to evaluate an image of a web page to determine whether the image is suitable for 3D rendering, and based on the result of the evaluation, may provide an indication to the user that the 3D effect may be applied to the image. For example, a glow effect or icon (indicating suitable 3D properties) may be presented adjacent to the image.

[0006] In some specific implementations relative to web-based applications, the 3D effect can be used to depict an image without using the 3D effect to depict any other portion of the web page. For example, the 3D effect can be applied only to the image without applying the 3D effect to the web page.

[0007] In some implementations with respect to web-based applications, the 3D effect can be used to depict an image without depicting any other portion of the web page. For example, portions of the web page surrounding the image can be hidden or blurred, the image can be depicted in full screen mode, the image can be surrounded by a feathering or dimming effect so that the web page is not rendered within the background surrounding the image, and so on.

[0008] In some implementations with respect to web-based applications, the 3D effect may be applied to the image based on a priority given to the importance of the image relative to other portions of the web page. For example, the image may be larger than other images of the web page, and therefore it may be determined that applying the 3D effect to the image should be prioritized over the other images.

[0009] In some implementations, an initial view of an image (for image preview) may depict a low-resolution version of the image using a 3D effect, and subsequent views may depict a high-resolution version of the image using the 3D effect.

[0010] In some implementations, an adjustable baseline (eg, relative to the user IPD) can be used to provide different 3D effects (eg, realistic, exaggerated, etc.) to the 2D asset.

[0011] In some implementations, a 3D effect is provided by changing the 2D asset relative to the left eye view and the right eye view via a display of the HMD. Changing the image relative to the left eye view and the right eye view can provide a vergence effect to provide the 3D effect.

[0012] In some implementations, the 3D effect is provided by providing a changing view of the 2D asset based on different viewpoints within the 3D environment (e.g., via an HMD). Providing a changing view of the 2D asset based on different viewpoints within the 3D environment can provide a parallax effect to provide the 3D effect.

[0013] In some implementations, an HMD has a processor (e.g., one or more processors) that executes instructions stored in a non-transitory computer-readable medium to perform a method. The method performs one or more steps or processes. In some implementations, the HMD obtains an image depicting 2D content. In some implementations, it is determined to apply a 3D effect to the image via the HMD. Based on the determination to apply the 3D effect to the image, a view of a 3D environment including the image is presented to the user. The image can be positioned at a location within the 3D environment, and the view can depict the image using the 3D effect.

[0014] According to some specific implementations, a device includes one or more processors, non-volatile memory, and one or more programs; the one or more programs are stored in the non-volatile memory and are configured to be executed by the one or more processors, and the one or more programs include instructions for executing or causing the execution of any of the methods described herein. According to some specific implementations, a non-volatile computer-readable storage medium stores instructions that, when executed by one or more processors of the device, cause the device to execute or cause the execution of any of the methods described herein. According to some specific implementations, a device includes: one or more processors, non-volatile memory, and a device for executing or causing the execution of any of the methods described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] So that the present disclosure may be understood by those of ordinary skill in the art, a more detailed description may be obtained with reference to aspects of some illustrative implementations, some of which are shown in the accompanying drawings.

[0016] Figure 1A to Figure 1B Example electronic devices are shown operating in a physical environment according to some implementations.

[0017] Figure 2 An example of application of a 3D effect to a retrieved 2D image for viewing by a user is shown according to some implementations.

[0018] Figure 3 Shown are left-eye and right-eye image versions presented within an application user interface (UI) positioned within a 3D environment according to some implementations.

[0019] FIG. 4A to FIG. 4C A diagram showing application of a 3D effect to a 2D image obtained from a web page for enhanced viewing by a user is shown according to some implementations.

[0020] Figure 5 is a flowchart representation of an exemplary method for dynamically applying 3-dimensional (3D) effects to 2-dimensional content, according to some implementations.

[0021] Figure 6 is a block diagram of an electronic device according to some specific implementations.

[0022] As is common practice, the various features shown in the drawings may not be drawn to scale. Therefore, the sizes of the various features may be arbitrarily expanded or reduced for clarity. In addition, some of the drawings may not depict all of the components of a given system, method, or device. Finally, throughout the specification and drawings, similar reference numerals may be used to represent similar features. DETAILED DESCRIPTION

[0023] Many details are described in order to provide a thorough understanding of the exemplary implementations shown in the accompanying drawings. However, the accompanying drawings only illustrate some exemplary aspects of the present disclosure and should not be considered limiting. One of ordinary skill in the art will appreciate that other effective aspects and / or variations do not include all of the specific details described herein. In addition, well-known systems, methods, components, devices, and circuits are not described in detail in order to avoid obscuring more relevant aspects of the exemplary implementations described herein.

[0024] Figure 1A to Figure 1B Exemplary electronic devices 105 and 110 are shown operating in physical environment 100. Figure 1A to Figure 1B In the example of FIG. 1 , the physical environment 100 is a room including a table 120. The electronic devices 105 and 110 may include one or more cameras, microphones, depth sensors, or other sensors that can be used to capture and evaluate information about the physical environment 100 and objects therein, as well as information about the user 102 of the electronic devices 105 and 110. The information about the physical environment 100 and / or the user 102 can be used to provide visual and audio content and / or to identify the current location of the physical environment 100 and / or the location of the user within the physical environment 100.

[0025] In some implementations, a view of an extended reality (XR) environment may be provided to one or more participants (e.g., user 102 and / or other participants not shown) via electronic devices 105 (e.g., wearable devices such as HMDs, etc.) and / or 110 (e.g., handheld devices such as mobile devices, tablet computing devices, laptop computers, etc.). Such an XR environment may include a view of a 3D environment generated based on a camera image and / or a depth camera image of the physical environment 100 and a representation of the user 102 based on the camera image and / or the depth camera image of the user 102. Such an XR environment may include virtual content positioned at a 3D position relative to a 3D coordinate system associated with the XR environment, which may correspond to a 3D coordinate system of the physical environment 100.

[0026] In some implementations, an HMD (e.g., device 105) communicatively coupled to a server or other external device may be configured to accept a 2D input image (e.g., a standard image or panoramic image from a photo library or web page / web browser) and apply a (synthesized) 3D (stereoscopic) output effect (based on a left-eye image and a right-eye image) (in real time or after a dwell time) for rendering via the HMD during a viewing session. The synthesized 3D stereoscopic output effect (based on the left-eye image and the right-eye image) may be applied for rendering via the HMD using a monoscopic to stereoscopic conversion technique based on machine learning (ML). An adjustable baseline [e.g., user interpupillary distance (IPD)] may be used to provide different effects (e.g., realistic, exaggerated, etc.) associated with the 3D output effect. Applying the 3D output effect (for rendering) may be triggered manually or automatically. The manual process of applying the 3D output effect may include activating a button or switch (by the user) when the user intends to apply the 3D output effect to the 2D input image.

[0027] In some implementations, a glow effect or icon can be triggered relative to an image on a web page in response to a detected user gaze. The glow effect or icon can indicate that the image is suitable for applying the 3D output effect and can be presented adjacent to the image (via the display of the HMD). In response to the glow effect or icon being presented adjacent to the image, the user can activate a button, switch, icon, input command (e.g., gesture, mouse click, etc.) associated with the glow effect, etc.

[0028] The automated process of applying the 3D output effect may include selecting a 2D input image based on detecting a user's gaze direction relative to the 2D input image (e.g., the user is looking at the 2D input image). Similarly, the application of the 3D effect may be automatically triggered based on the content depicted within the 2D input image. For example, content (e.g., a photo) depicting natural objects (e.g., mountains, trees, etc.) and / or people may automatically trigger the application of the 3D output effect to the 2D input image. Similarly, text (e.g., a receipt) may not trigger the application of the 3D output effect to the 2D input image. Alternatively, the application of the 3D effect may be automatically triggered based on the expected quality of the 3D output effect. For example, a 2D input image such as a photo may be determined to be error-prone or noisy (e.g., random variations in the photo caused by poor lighting conditions), and therefore the 3D output effect is not triggered. Some specific implementations include automatically triggering the 3D output effect based on metadata (e.g., including instructions) of the 2D input image. Some implementations include automatically triggering the 3D output effect based on an application developer initiated setting within an application that renders the 2D input image. Applying the 3D effect to the 2D input image may not require the use of any depth information.

[0029] In some implementations, a 3D (stereoscopic) output effect is applied to a 2D input image of a web page / web browser for 3D rendering via an HMD during a viewing session. Some implementations enable a web browser to evaluate an image of a web page to determine whether the image is suitable for 3D rendering. If the image is determined to be suitable for 3D rendering, an indication is displayed to the user that the 3D output effect may be applied to the image (a glow effect or icon adjacent to the image). Similarly, if the image is determined to be unsuitable for 3D rendering, a different indication is displayed to the user that the 3D effect may not be applied to the image (an alternative glow effect or icon adjacent to the image).

[0030] In some implementations, a 3D output effect may be used to depict an image without using the 3D output effect to depict any other portion of a web page. For example, the 3D output effect may be applied only to the image without applying the 3D output effect to the web page. In some implementations, a 3D output effect may be used to depict an image without depicting any other portion of a web page. For example, the web page may be hidden or blurred, the image may be depicted in full screen mode, the image may be surrounded by a feathering or dimming effect to obscure the web page from background areas surrounding the image, and so on.

[0031] In some implementations, the 3D output effect may be applied to the image based on prioritizing the importance of the image relative to other portions of the web page. For example, the image may be larger than other images of the web page, and therefore it may be determined that applying the 3D output effect to the image should be prioritized over the other images.

[0032] In some implementations, an initial view of an image can use the 3D output effect to depict a low-resolution version of the image, and subsequent views can use the 3D output effect to depict a high-resolution version of the image.

[0033] Figure 2An example of an application of a 3D effect to an original 2D image 202 retrieved (e.g., from a photo application of an HMD) for viewing by a user is shown according to some implementations (relative to providing a convergence effect). The original 2D image 202 includes a photo of a person 208 in a foreground and a mountain 204 in a background. Applying the 3D effect to the original 2D image 202 includes altering the original 2D image 202 by providing (for viewing by the user via, e.g., an HMD) a left-eye image version 202a (of the original 2D image 202) and a right-eye image version 202b (of the original 2D image 202). The left-eye image version 202a represents a view 208a of a person 208 located at a first position that is different from an original position 207 of the user 208 in the original 2D image 202 (e.g., horizontally offset in a direction 212a). The right eye image version 202b represents a view 208b of the person 208 at a second position that is different from the original position 207 of the user 208 in the original 2D image 202 (e.g., horizontally offset in direction 212b). The first position represents the user 208 at a different location within the left eye image version 202a than the second position within the right eye image version 202b. Thus, when viewed via an HMD, the combination of the left eye image version 202a and the right eye image version 202b is presented (to the user) as a merged image that represents a 3D effect relative to the person 208.

[0034] Figure 3 An example of an application program presented within an application user interface (UI) 302 positioned within a 3D environment 300 is shown according to some implementations. Figure 2 1 . The 3D environment includes a physical environment (e.g., physical environment 100 of FIG. 1 ) and virtual objects, the physical environment including table 320, and the virtual objects including application UI 302. Application UI 302 may include any type of software application UI, including, among others, photo application UI, social media application UI, etc. By providing (for viewing by a user via, for example, an HMD) a left eye view 305a (of the 3D environment 300) and a right eye view 305b (of the original 2D image 202), a 3D effect is applied to the 3D environment including ( Figure 2 The left eye image version 202a and the left eye image version 202b of the image 202 Figure 2202 of the image 202 of the present invention. Left eye view 305a represents a view of application UI 302 at a first position (e.g., horizontally offset in direction 312a). Right eye view 305b represents a view of application UI 302 at a second position that is different from the first position of application UI 302 presented in left eye view 305a (e.g., horizontally offset in direction 312b). The different positions of left eye view 305a and right eye view 305b enable the user (when viewing via the HMD) to view the combination of left eye view 305a and right eye view 305b as a merged image that represents a view of application UI 302 relative to application UI 302 and (e.g., horizontally offset in direction 312b). Figure 2 3D effect of person 208 in image 202 .

[0035] FIG. 4A to FIG. 4C Views 401a-401c are shown representing the application of a 3D effect to a 2D image (e.g., image 410) obtained from a web page 408 for enhanced 3D viewing by a user, according to some implementations. Each of views 401a-401c includes an exemplary 3D environment 400 (e.g., an XR environment) that includes a physical environment (e.g., the physical environment 100 of FIG. 1 , which can be viewed as a pass-through video or can be a direct view through a transparent display or a semi-transparent display) and a virtual environment (e.g., including web page 408). Each of views 401a-401c includes a web browser 404 that displays web page 408. Web page 408 includes images 410, 412, 414, 416, and 418 (e.g., photos) and a text portion 440.

[0036] exist Figure 4A In the example of , at a first moment corresponding to view 401a, a user gaze direction / position (shown by ray 425) is detected relative to image 410, thereby indicating a predicted user intent relative to viewing image 410. Alternatively, a finger direction (shown by hand 429 relative to ray 426) may be detected relative to image 410 (instead of or in combination with the user gaze direction / position shown by ray 425), thereby indicating a predicted user intent relative to viewing image 410. The predicted user intent relative to viewing image 410 may be configured to trigger an indication that a 3D effect may or may not be applied to image 410 (e.g., a glow effect or icon), as described below relative to Figure 4B In some implementations, the image for applying the 3D effect may be selected based on prioritizing the importance of the image relative to other parts of the web page. Figure 4A, image 410 may be selected for 3D application because it includes a larger image relative to images 412 , 414 , 416 , and 418 , and therefore it may be determined that applying the 3D effect to image 410 should take precedence over applying the 3D effect to images 412 , 414 , 416 , and 418 .

[0037] exist Figure 4B In the example of , at a second moment corresponding to view 401b, the adjacent image 410 presents / displays an indication that a 3D effect may or may not be applied to the image 410 (e.g., via effect 422 and / or icon 242). In some implementations, the web browser 404 may be configured to evaluate the image 410 to determine whether a 3D effect may or may not be applied to the image 410. For example, the web browser 404 may evaluate the image 410 and, based on the evaluated image attributes (e.g., the image includes too many details, the image has a border, the image is a generic / decorative image, etc.) and / or the evaluated image quality (e.g., a low-resolution image), may determine that the image 410 is not suitable for 3D presentation. Conversely, the web browser 404 may evaluate the image 410 and, based on the evaluated image attributes (e.g., an image of an indoor scene or an outdoor scene, etc.) and / or the evaluated image quality (e.g., a high-resolution image), may determine that the image 410 is suitable for 3D presentation. Subsequently, the effect 422 and / or icon 424 associated with the evaluated image is presented to the user (adjacent to the image 410). Effects 422 may include, among other things, a semi-transparent glow effect over or around image 420, a border around image 420, etc. Icons 424 may include (virtual) buttons, activators, and the like.

[0038] If effect 422 and / or icon 424 indicates that image 410 is not suitable for 3D rendering, effect 422 and / or icon 424 may be presented as a first specific indicator type, such as, among other things, a first specific color (e.g., red), a first specific transparency level, a first specified shape, etc. The user may then gaze at or point to another image (e.g., one of images 412, 414, 416, or 418) to select and trigger the evaluation process to determine whether a 3D effect may or may not be applied to the newly selected image.

[0039] If the effect 422 and / or icon 424 indicates that the image 410 is suitable for 3D presentation, the effect 422 and / or icon 424 may be presented as a second specific indicator type (i.e., different from the first specific indicator type), such as, in particular, a second specific color (e.g., green), a second specific transparency level, a second specific shape, etc. In response to the indication that the image 410 is suitable for 3D presentation, the user may select the image 410 to apply the 3D effect. The image 410 may be selected (for 3D effect application) by receiving a gesture input from the user. For example, the user may initiate an input command to select the image 410 and apply the 3D effect by performing a gesture such as a pinch gesture 427 (e.g., fingers of hand 428 are brought together and touching) while moving hand 428 in any of the X, Y, and Z directions 432 (e.g., movement of hand 428 relative to the Z direction to lift or pick up image 410 relative to webpage 408). Alternatively, image 410 may be selected (for 3D effect application) by activating icon 424 (eg, activating a virtual button or switch) via a hand gesture or gaze gesture.

[0040] exist Figure 4C In the example of , at a third moment corresponding to view 401c, an expanded view 410a of image 410 is presented / displayed to the user. Expanded view 410a represents an expanded 3D version of image 410 presented / displayed after the 3D effect is applied, such that the 3D effect is used to depict image 410 without depicting any other portion of web page 408 (e.g., images 412, 414, 416, and 418 and text portion 440). In this example, image 410 is displayed at a larger size in full screen mode, thereby obscuring most of the content of web page 408 from the background area surrounding the image to enhance the user viewing experience. Alternatively, the 3D effect can be used to (relative to Figure 4A and Figure 4B ) such that all portions of the web page 408 surrounding the image 410 are blurred (e.g., out of focus), obscured via a feathering or dimming effect, etc.

[0041] In some implementations, an initial view of image 410 can depict a low-resolution version of the image using a 3D effect (e.g., for preview purposes), and subsequent views of image 410 can depict a high-resolution version of the image using the 3D effect.

[0042] Figure 51 is a flowchart representation of an exemplary method 500 for dynamically applying 3-dimensional (3D) effects to 2-dimensional content according to some specific implementations. In some specific implementations, method 500 is performed by a device such as a mobile device, a desktop computer, a laptop computer, an HMD, or a server device. In some specific implementations, the device has a screen for displaying images and / or a screen for viewing stereoscopic images, such as a head-mounted display (HMD, such as, for example, device 105 of Figure 1). In some specific implementations, method 500 is performed by a processing logic component including hardware, firmware, software, or a combination thereof. In some specific implementations, method 500 is performed by a processor executing code stored in a non-transitory computer-readable medium (e.g., a memory). Each box in the box in method 500 can be enabled and executed in any order.

[0043] At block 502 , method 500 obtains an image depicting two-dimensional (2D) content, such as, among other things, a photograph, a panoramic image, a video, or the like.

[0044] At block 504, method 500 determines to apply a three-dimensional (3D) effect to the image in real time. In some implementations, determining to apply the 3D effect to the image includes receiving input from a user of a device (e.g., an HMD) that manually initiates the 3D effect. For example, the user may manually activate a button or switch to manually initiate the 3D effect being applied to the image.

[0045] In some implementations, determining to apply the 3D effect to the image can include an automated process to apply the 3D effect to the image based on certain criteria, such as, in particular, in response to a certain application, such as a photo gallery application, being enabled.

[0046] In some implementations, determining to apply the 3D effect to the image can include an automated process to apply the 3D effect to the image based on detecting a gaze direction relative to the image depicting the 2D content.

[0047] In some implementations, determining to apply the 3D effect to the image may include an automated process for applying the 3D effect to the image based on content depicted within the image. For example, content depicting objects and / or humans in nature (e.g., a photograph) may automatically trigger application of the 3D effect to the image.

[0048] In some implementations, determining to apply the 3D effect to the image may include an automated process for applying the 3D effect to the image based on an expected quality of the 3D effect depicting the image. For example, a photo may be determined to be error-prone or noisy and therefore not trigger application of the 3D effect.

[0049] In some implementations, determining to apply the 3D effect to the image can include an automated process to apply the 3D effect to the image based on metadata (including instructions) of the image depicting the 2D content.

[0050] In some implementations, determining to apply the 3D effect to the image may include an automated process to apply the 3D effect to the image based on settings initiated by an application developer within an application that renders the image depicting the 2D content.

[0051] In some implementations, in response to receiving a first input from a user of the HMD (e.g., gaze input, gesture input, etc.), providing an indication that the 3D effect can be applied to the image (e.g., a glow effect or icon presented adjacent to the image on a webpage, etc.), such that determining to apply the 3D effect to the image includes: receiving a second input from a user who manually initiates the 3D effect in response to the indication. (e.g., in response to the user activating a button or icon). The indication can be provided to the user instantaneously in real time or after a specified period of time has elapsed.

[0052] In some implementations, the image can be located within a web page such that determining to apply the 3D effect to the image is based on prioritizing the image relative to its importance relative to other portions of the web page (e.g., the image is larger than other images on the web page).

[0053] In some implementations, the image may be located within a web page, and in response to receiving a first input from a user (e.g., a gaze input, a gesture input, etc.), web browser-based evaluation data associated with an ability to apply the 3D effect to the image (e.g., whether the image is suitable for 3D presentation) may be obtained. Based on the web browser-based evaluation data, an indication is provided to the user that the 3D effect may be applied to the image (e.g., a glow effect or icon adjacent to the 2D presentation on the web page, etc.), such that determining to apply the 3D effect to the image includes: receiving a second input from a user of the HMD manually initiating the 3D effect in response to the indication (e.g., a pinch gesture with hand movement in the Z direction).

[0054] At block 506 (based on determining to apply the 3D effect to the image), the method presents a view of the 3D environment including the image. In some implementations, the image is positioned at a location within the 3D environment, and the view depicts the image using the 3D effect.

[0055] In some implementations, the image is positioned within a plane region within the 3D environment. In some implementations, the image is positioned within a plane user interface within the 3D environment. The plane user interface may be provided by an application. In some implementations, the image may be a frame within a video so that the view uses the 3D effect to depict the video.

[0056] In some implementations, an adjustable baseline [eg, relative to the user's interpupillary distance (IPD)] can be used to provide different effect types (eg, realistic, exaggerated, etc.) for the 3D effect.

[0057] In some implementations, a 3D effect is provided by changing the image relative to a left-eye view and a right-eye view via the display to provide a convergence effect.

[0058] In some implementations, the 3D effect is provided by providing changing views of the image based on different viewpoints within the 3D environment to provide a parallax effect.

[0059] In some implementations, the image can be located within a web page, and the view can use the 3D effect to depict the image without using the 3D effect to depict any other portion of the web page (e.g., applying the 3d effect only to the image and not to the web page).

[0060] In some embodiments, the image can be located within a web page, and the view can use the 3D effect to depict the image without depicting any other portion of the web page (e.g., hiding or blurring the web page, depicting the image in full screen mode, surrounding the image with a feathering effect or dimming effect to obscure the web page from the background, etc.).

[0061] In some implementations, based on determining to apply the 3D effect to the image, an initial view of the 3D environment including the image is presented such that the image is positioned at a location within the 3D environment, and the initial view depicts a low-resolution version of the image using the 3D effect. After depicting the low-resolution version of the image using the 3D effect, the view may depict a high-resolution version of the image using the 3D effect.

[0062] Figure 61 is a block diagram of an exemplary device 600. Device 600 illustrates an exemplary device configuration for electronic devices 105 and 110 of FIG1. ​​Although some specific features are shown, those skilled in the art will appreciate from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the specific implementations disclosed herein. To this end, as a non-limiting example, in some embodiments, the device 600 includes one or more processing units 602 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, etc.), one or more input / output (I / O) devices and sensors 604, one or more communication interfaces 608 (e.g., USB, FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE802.14x, GSM, CDMA, TDMA, GPS, IR, Bluetooth, ZIGBEE, SPI, I2C, and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 610, output devices (e.g., one or more displays) 612, one or more internal-facing and / or external-facing image sensor systems 614, memory 620, and one or more communication buses 604 for interconnecting these and various other components.

[0063] In some implementations, the one or more communication buses 604 include circuits that interconnect and control communications between system components. In some implementations, the one or more I / O devices and sensors 606 include at least one of the following: an inertial measurement unit (IMU), an accelerometer, a magnetometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., a blood pressure monitor, a heart rate monitor, a blood oxygen sensor, a blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptic engine, one or more depth sensors (e.g., structured light, time of flight, etc.), one or more cameras (e.g., an internally facing camera and an externally facing camera of the HMD), one or more infrared sensors, one or more thermal map sensors, etc.

[0064] In some implementations, one or more displays 612 are configured to present a view of a physical environment, a graphical environment, an extended reality environment, etc. to the user. In some implementations, one or more displays 612 are configured to present content (determined based on the determined user / object position of the user within the physical environment) to the user. In some implementations, one or more displays 612 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light emitting field effect transistor (OLET), organic light emitting diode (OLED), surface conduction electron emission display (SED), field emission display (FED), quantum dot light emitting diode (QD-LED), micro-electromechanical system (MEMS) and / or similar display types. In some implementations, one or more displays 612 correspond to diffraction, reflection, polarization, holographic, etc. waveguide displays. In one example, the device 600 includes a single display. In another example, the device 600 includes a display for each eye of the user.

[0065] In some implementations, the one or more image sensor systems 614 are configured to obtain image data corresponding to at least a portion of the physical environment 100. For example, the one or more image sensor systems 614 include one or more RGB cameras (e.g., with complementary metal oxide semiconductor (CMOS) image sensors or charge coupled device (CCD) image sensors), monochrome cameras, IR cameras, depth cameras, event-based cameras, etc. In various implementations, the one or more image sensor systems 614 also include an illumination source that emits light, such as a flash. In various implementations, the one or more image sensor systems 614 also include an on-camera image signal processor (ISP) that is configured to perform multiple processing operations on the image data.

[0066] In some implementations, sensor data may be obtained by a device (e.g., devices 105 and 110 of FIG. 1 ) during a scan of a room of a physical environment. The sensor data may include a 3D point cloud and a 2D image sequence corresponding to a view of the room captured during the scan of the room. In some implementations, the sensor data includes image data (e.g., from an RGB camera), depth data (e.g., a depth image from a depth camera), ambient light sensor data (e.g., from an ambient light sensor), and / or motion data from one or more motion sensors (e.g., an accelerometer, a gyroscope, an IMU, etc.). In some implementations, the sensor data includes visual inertial odometry (VIO) data determined based on the image data. The 3D point cloud may provide semantic information about one or more elements of the room. The 3D point cloud may provide information about the positioning and appearance of a surface portion within the physical environment. In some implementations, the 3D point cloud is obtained over time (e.g., during a scan of the room), and the 3D point cloud may be updated, and an updated version of the 3D point cloud is obtained over time. For example, the 3D representation may be obtained (and analyzed / processed) as it updates / adjusts over time (eg, as a user scans a room).

[0067] In some implementations, the sensor data may be positioning information, and some implementations include VIO to determine equivalent odometer information using sequence camera images (e.g., light intensity image data) and motion data (e.g., obtained from an IMU / motion sensor) to estimate the distance traveled. Alternatively, some implementations of the present disclosure may include a simultaneous localization and simultaneous mapping (SLAM) system (e.g., a positioning sensor). The SLAM system may include a multi-dimensional (e.g., 3D) laser scanning and range measurement system that is independent of GPS and provides real-time simultaneous positioning and mapping. The SLAM system may generate and manage data of a very accurate point cloud generated by reflections of laser scans from objects in the environment. Over time, the movement of any point in the point cloud is accurately tracked so that the SLAM system can use points in the point cloud as reference points for location, maintaining an accurate understanding of its position and orientation as it travels through the environment.

[0068] In some implementations, the device 600 includes an eye tracking system for detecting eye positioning and eye movement (e.g., eye gaze detection). For example, the eye tracking system may include one or more infrared (IR) light emitting diodes (LEDs), an eye tracking camera (e.g., a near infrared (NIR) camera), and an illumination source (e.g., a NIR light source) that emits light (e.g., NIR light) to the user's eyes. In addition, the illumination source of the device 600 may emit NIR light to illuminate the user's eyes, and the NIR camera may capture images of the user's eyes. In some implementations, the images captured by the eye tracking system may be analyzed to detect the positioning and movement of the user's eyes, or to detect other information about the eyes such as pupil dilation or pupil diameter. In addition, the gaze point estimated from the eye tracking image may enable gaze-based interaction with content shown on the near-eye display of the device 600.

[0069] The memory 620 includes a high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices. In some implementations, the memory 620 includes a non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory 620 optionally includes one or more storage devices remotely located from the one or more processing units 602. The memory 620 includes a non-transitory computer-readable storage medium.

[0070] In some implementations, the memory 620 or a non-transitory computer-readable storage medium of the memory 620 stores an optional operating system 630 and one or more instruction sets 640. The operating system 630 includes processes for handling various basic system services and for performing hardware-related tasks. In some implementations, the instruction set 640 includes executable software defined by binary information stored in the form of electrical charge. In some implementations, the instruction set 640 is software that can be executed by one or more processing units 602 to implement one or more of the techniques described herein.

[0071] The instruction set 640 includes a 3D effect application instruction set 642 and a 3D effect rendering instruction set 644. The instruction set 640 may be embodied as a single software executable file or multiple software executable files.

[0072] The 3D effect application instruction set 642 is configured with instructions executable by the processor to determine to apply a 3D effect (eg, via manual instructions or automated instructions) to a 2D image via the HMD (eg, in real time).

[0073] The 3D effect rendering instruction set 644 is configured with instructions executable by the processor to render a view of the 3D environment including the 2D image (located at a position within the 3D environment) rendered using the 3D effect.

[0074] Although the instruction set 640 is shown as residing on a single device, it should be understood that in other implementations, any combination of elements may be located in separate computing devices. Figure 6 More so as to serve as functional descriptions of various features present in a particular implementation that are distinct from the structural schematics of the implementations described herein. As will be appreciated by one of ordinary skill in the art, items shown separately may be combined, and some items may be separated. The actual number of instruction sets and how features are distributed among them will vary from one implementation to another, and may depend in part on the specific combination of hardware, software, and / or firmware selected for a particular implementation.

[0075] Those of ordinary skill in the art will understandably describe well-known systems, methods, components, devices, and circuits so as not to obscure more relevant aspects of the example implementations described herein. In addition, other effective aspects and / or variations do not include all of the details in the specific details described herein. Therefore, several details are described in order to provide a thorough understanding of the exemplary aspects shown in the accompanying drawings. In addition, the accompanying drawings only illustrate some exemplary embodiments of the present disclosure and therefore should not be considered as limiting.

[0076] Although this specification includes many specific implementation details, these specific implementation details should not be understood as limitations on the scope of any invention or possible content to be protected, but should be understood as descriptions of features specific to a particular embodiment of a particular invention. Certain features described in the context of different embodiments in this specification may also be implemented in combination in a single embodiment. On the contrary, the various features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in the form of any suitable sub-combination. In addition, although certain features may be described above as acting in certain combinations and even initially claimed in this way, one or more features of the claimed combination may be removed from the combination in some cases, and the claimed combination may involve a sub-combination or a variation of the sub-combination.

[0077] Similarly, although the operations are shown in a specific order in the accompanying drawings, this should not be understood as requiring such operations to be performed in a sequential order or in the specific order shown, or requiring the execution of all the operations shown to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. In addition, the division of each system component in the above-mentioned embodiment should not be understood as requiring such division in all embodiments, and it should be understood that the program components and the system can be generally integrated together in a single software product or encapsulated in multiple software products.

[0078] Thus, specific embodiments of the subject matter have been described. Other embodiments are also within the scope of the following claims. In some cases, the actions described in the claims can be performed in a different order and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In some specific implementations, multitasking and parallel processing may be advantageous.

[0079] The embodiments of the subject matter and operations described in this specification may be implemented in digital electronic circuits or in computer software, firmware or hardware (including the structures disclosed in this specification and their structural equivalents) or in a combination of one or more of them. The embodiments of the subject matter described in this specification may be implemented as one or more computer programs, for example, one or more modules of computer program instructions encoded on a computer storage medium, for execution by a data processing device or for controlling the operation of a data processing device. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagation signal, such as a machine-generated electrical signal, optical signal or electromagnetic signal, which is generated to encode information so as to be sent to a suitable receiver device for execution by a data processing device. The computer storage medium may be or be included in a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more of them. In addition, although the computer storage medium is not a propagation signal, the computer storage medium may be the source or destination of the computer program instructions encoded in the artificially generated propagation signal. The computer storage medium may also be or be included in one or more separate physical components or media (e.g., multiple CDs, disks or other storage devices).

[0080] The term "data processing device" covers all kinds of devices, equipment and machines for processing data, including, for example, programmable processors, computers, systems on chip, or multiple or combinations of the foregoing. The device may include a dedicated logic circuit (e.g., FPGA (field programmable gate array) or ASIC (application-specific integrated circuit)). In addition to hardware, the device may also include code for creating an execution environment for the computer program under consideration, such as code constituting a processor firmware, a protocol stack, a database management system, an operating system, a cross-platform operating environment, a virtual machine, or a combination of one or more of them. The device and the execution environment can implement a variety of different computing model infrastructures, such as web services, distributed computing, and grid computing infrastructures. Unless otherwise specifically stated, it should be understood that throughout the specification, discussions using terms such as "processing", "calculating", "calculating", "determining", and "identifying" refer to the actions or processes of a computing device, such as one or more computers or similar electronic computing devices, which manipulate or convert data represented as physical electronic quantities or magnetic quantities within the memory, registers, or other information storage devices, transmitting devices, or display devices of a computing platform.

[0081] The one or more systems discussed herein are not limited to any particular hardware architecture or configuration. A computing device may include any suitable arrangement of components that provide results conditioned on one or more inputs. Suitable computing devices include multi-purpose microprocessor-based computer systems that access stored software that programs or configures the computing system from a general-purpose computing device to a dedicated computing device that implements one or more specific implementations of the subject matter of the present invention. Any suitable programming, scripting, or other type of language or combination of languages ​​may be used to implement the teachings contained herein in software for programming or configuring a computing device.

[0082] The specific implementation of the method disclosed herein can be performed in the operation of such a computing device. The order of the boxes presented in the above examples can be changed, for example, the boxes can be reordered, combined and / or divided into sub-blocks. Some boxes or processes can be executed in parallel. The operations described in this specification can be implemented as operations performed by a data processing device on data stored on one or more computer-readable storage devices or received from other sources.

[0083] The use of "suitable for" or "configured to" herein is meant to be open and inclusive language, which does not exclude devices that are suitable for or configured to perform additional tasks or steps. In addition, the use of "based on" is meant to be open and inclusive, as a process, step, calculation or other action "based on" one or more stated conditions or values ​​may in practice be based on additional conditions or values ​​beyond those stated. The headings, lists and numbering included herein are for ease of explanation only and are not intended to be limiting.

[0084] It will also be understood that, although the terms "first", "second", etc. may be used to describe various elements in this article, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, a first node may be referred to as a second node, and similarly, a second node may be referred to as a first node, which changes the meaning of the description, as long as all occurrences of the "first node" are consistently renamed and all occurrences of the "second node" are consistently renamed. Both the first node and the second node are nodes, but they are not the same node.

[0085] The terms used herein are only for describing specific implementations and are not intended to limit the claims. As used in the description of the specific implementations and the appended claims, the singular forms of "one", "a kind of" and "the" are intended to also cover the plural forms, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" used herein refers to and covers any and all possible combinations of one or more items in the associated listed items. It will also be understood that the term "comprising" when used in this specification specifies the presence of stated features, integers, steps, operations, elements and / or parts, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts, and / or their groupings.

[0086] As used herein, the term "if" may be interpreted to mean "when" or "at" or "in response to determining" or "upon determining" or "in response to detecting" that the antecedent is true, depending on the context. Similarly, the phrase "if it is determined that [the antecedent is true]" or "if [the antecedent is true]" or "when [the antecedent is true]" is interpreted to mean "upon determining" or "in response to determining" or "upon determining" or "when detecting" or "in response to detecting" that the antecedent is true, depending on the context.

Claims

1. A method comprising: At a head mounted device (HMD) having a processor and a display: obtaining an image depicting two-dimensional (2D) content; determining, via the HMD, to apply a three-dimensional (3D) effect to the image; as well as Based on determining to apply the 3D effect to the image, presenting a view of a 3D environment including the image, wherein the image is positioned at a location within the 3D environment and the view depicts the image using the 3D effect.

2. The method of claim 1, wherein the determining to apply the 3D effect to the image comprises: Input is received from a user of the HMD manually initiating the 3D effect.

3. The method of claim 1 , wherein the determining to apply the 3D effect to the image comprises: Applying the 3D effect to the image is determined automatically based on a criterion or based on detecting a gaze direction to the image depicting the 2D content.

4. The method of claim 1, wherein the determining to apply the 3D effect to the image comprises: Applying the 3D effect to the image is automatically determined based on content depicted within the image or based on an expected quality of the 3D effect depicting the image.

5. The method of claim 1, wherein the determining to apply the 3D effect to the image comprises: Applying the 3D effect to the image is determined automatically based on metadata of the image depicting the 2D content or based on an application developer initiated setting within an application presenting the image depicting the 2D content.

6. The method according to claim 1, further comprising: In response to receiving a first input from a user of the HMD, providing an indication that the 3D effect may be applied to the image, wherein the determining to apply the 3D effect to the image comprises: receiving a second input from the user of the HMD manually initiating the 3D effect in response to the indication.

7. The method of claim 6, wherein the indication that the 3D effect may be applied to the image is provided to the user instantaneously in real time or after a specified period of time has elapsed.

8. The method of claim 6, wherein the image is within a web page, and wherein the view depicts the image using the 3D effect without depicting any other portion of the web page using the 3D effect.

9. The method of claim 6, wherein the image is within a web page, and wherein the view depicts the image using the 3D effect without depicting any other portion of the web page.

10. The method of claim 6, wherein the image is located within a web page, and wherein the determining to apply the 3D effect to the image is based on prioritizing the image based on an importance of the image relative to other portions of the web page.

11. The method of claim 1 , wherein the image is located within a web page, and wherein the method further comprises: In response to receiving a first input from a user of the HMD, obtaining web browser-based assessment data associated with an ability to apply the 3D effect to the image; as well as Based on the web browser-based evaluation data, an indication is provided that the 3D effect may be applied to the image, wherein the determining to apply the 3D effect to the image comprises: receiving, in response to the indication, a second input from the user of the HMD manually initiating the 3D effect (e.g., a pinch gesture with hand movement in the Z direction).

12. The method according to claim 1, further comprising: Based on determining to apply the 3D effect to the image, presenting an initial view of a 3D environment including the image, wherein the image is positioned at a location within the 3D environment and the initial view depicts a low-resolution version of the image using the 3D effect, and wherein after depicting the low-resolution version of the image using the 3D effect, the view depicts a high-resolution version of the image using the 3D effect.

13. The method of claim 1, wherein the image is positioned within a planar region within the 3D environment.

14. The method of claim 1, wherein the image is positioned within a planar user interface within the 3D environment.

15. The method of claim 1, wherein the image is a frame within a video, and wherein the view depicts the video using the 3D effect.

16. The method of claim 1, wherein an adjustable baseline is used to provide different effect types for the 3D effect.

17. The method of claim 1, wherein the 3D effect is provided by changing the image via the display relative to a left eye view and a right eye view to provide a convergence effect.

18. The method of claim 1, wherein the 3D effect is provided by providing changing views of the image based on different viewpoints within the 3D environment to provide a parallax effect.

19. A system comprising: processor; A computer readable medium storing instructions that, when executed by the processor, cause the processor to perform operations including the method of claim 1.

20. A non-transitory computer readable medium comprising instructions that, when executed by a processor, cause the processor to perform operations including the method of claim 1.

Images