Transparent display based on camera

By using the rear camera and position detection module to determine the viewpoint relationship in a multi-function electronic device, image warping display is achieved, which solves the problem of seamless alignment of images and real environments in the device and provides a more realistic view effect.

CN118945309APending Publication Date: 2024-11-12APPLE INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411023006.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2017-09-08
Filing Date
2018-09-07
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

When using multi-function electronics, the images captured by the camera are inconsistent with the user's viewpoint, causing the images to display incorrect positions and zooms on the screen, and cannot be seamlessly aligned with the real environment.

Method used

The scene image is captured through the rear camera of the electronic device, and the position detection module is used to determine the spatial relationship between the viewpoint and the device. The warping function is used to render the image on the display, so that the image looks seamlessly connected to the real environment, realizing that Warp display of the image.

Benefits of technology

A more realistic view is provided, allowing the user to look transparent from any viewpoint of the electronic device, or the image of the scene is basically seamlessly engaged with the real environment behind it.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118945309A_ABST
    Figure CN118945309A_ABST
Patent Text Reader

Abstract

The invention relates to a camera-based transparent display. Presenting an image of a scene may include capturing an image of a scene by a camera of an electronic device, where the electronic device includes the camera and a display, and where the camera and the display have a first spatial relationship that determines a second spatial relationship between a viewpoint and the display of the electronic device, the image is warped to obtain an image of a first portion of the scene based on the first spatial relationship and the second spatial relationship, and the warped image is presented on the display, where the first portion of the scene is warped from a viewpoint. The image of the first portion of the scene substantially adjoins a second portion of the scene visible outside the edge of the electronic device.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] This application is a divisional application of the PCT international application with an international filing date of September 7, 2018, Chinese application number 201880057861.X, and invention name “Camera-based transparent display” entering the Chinese national phase. Background Art

[0002] The present disclosure relates generally to the field of digital image processing, and more particularly to the field of providing camera-based transparent displays.

[0003] Many multifunction electronic devices, such as smartphones and tablets, include cameras and display devices that allow a user to preview and view images captured by the camera using a display. When a conventional multifunction device is used in a preview mode, the image displayed on the screen represents the image from the camera's viewpoint. Therefore, when the user holds the device in front of the scene and views the display, the image on the screen is not aligned with the scene behind it. That is, the user's viewpoint is different from the angle of the lens that captures the image, resulting in a disconnect between the user's viewpoint and the real environment. That is, the image may appear in the wrong position and in the wrong scale for the user. Summary of the invention

[0004] In one embodiment, a method for presenting an image of a scene is described. The method may include capturing an image of the scene by a camera of an electronic device, wherein the electronic device includes the camera and a display, and wherein the camera and the display have a first spatial relationship, the spatial relationship determining a second spatial relationship between a viewpoint and the display of the electronic device, warping the image to obtain an image of a first portion of the scene based on the first spatial relationship and the second spatial relationship, and presenting the warped image on the display, wherein, from the viewpoint, the image of the first portion of the scene is substantially adjacent to a second portion of the scene visible beyond an edge of the electronic device.

[0005] In another embodiment, the method may be embodied in a computer executable program code and stored in a non-transitory storage device. In another embodiment, the method may be implemented in an electronic device having image capture capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 A simplified electronic device according to one or more embodiments is shown in block diagram form.

[0007] Figure 2 An example method for displaying a warped image in a real environment according to one or more embodiments is shown in flowchart form.

[0008] Figure 3 A method for warping an image of a real environment according to one or more embodiments is shown in flowchart form.

[0009] Figure 4 An exemplary setup for viewing an image of a real-world environment using a device in accordance with one or more embodiments is illustrated in system diagram form.

[0010] Figure 5 An example system diagram illustrating a method of displaying a warped image of a real environment is shown in accordance with one or more embodiments.

[0011] Figure 6 An example system diagram illustrating a warping method according to one or more embodiments is shown.

[0012] Figure 7 A simplified multifunction device according to one or more embodiments is shown in block diagram form. DETAILED DESCRIPTION

[0013] The present disclosure relates to systems, methods, and computer-readable media for presenting a warped image of a scene. In general, techniques are disclosed to provide a warped image of a scene so that the image of the environment in the scene appears seamless with the real environment on a display from a viewpoint. According to one or more embodiments, displaying the warped image can provide a more realistic view of the real environment on an electronic device.

[0014] According to one or more embodiments, the rear camera of the electronic device can capture an image of a view of the real environment. The spatial relationship between the viewpoint and the electronic device, as well as the spatial relationship between the camera and the display of the electronic device can be determined. A warping function can be applied to the image based on the spatial relationship. The warped image can then be presented on the front display on the electronic device so that the image of the real environment appears to be substantially adjacent to the background (i.e., the rest of the real environment). That is, except for any edges or boundaries of the electronic device surrounding the display, the scene may appear adjacent in the real world and the display. Given a viewpoint relative to the screen of the electronic device, the image captured by the camera can be transformed or reprojected relative to the screen so that from the user's viewpoint, at least one object in the scene has the same position and sales as the real object behind the scene. Therefore, according to one or more embodiments, the electronic device may appear to be transparent, or the image of the scene may appear to be substantially seamless with the scene behind it from the user's viewpoint.

[0015] In the following description, for the purpose of explanation, a lot of specific details are set forth in order to provide a thorough understanding of the disclosed concept. As part of this specification, some drawings of the present disclosure represent structures and devices in the form of block diagrams to avoid blurring the novel aspects of the disclosed embodiments. In this context, it should be understood that references to numbered drawing elements without associated identifiers (e.g., 100) refer to all instances of drawing elements with identifiers (e.g., 100a and 100b). In addition, as part of this specification, some drawings of the present disclosure can be provided in the form of flow charts. The blocks in any particular flow chart can be presented in a specific order. However, it should be understood that the specific flow of any flow chart is only used to illustrate an embodiment. In other embodiments, any various components depicted in the flow chart can be deleted, or components can be executed in different orders, or even executed simultaneously. In addition, other embodiments may include additional steps that are not shown as a part of the flow chart. The language used in the present disclosure is mainly selected for readability and guidance purposes, and it may not be selected to depict or limit the disclosed subject matter. Reference in this disclosure to "one embodiment" or "an embodiment" is meant to include a particular feature, structure, or characteristic described in connection with that embodiment in at least one embodiment, and multiple references to "one embodiment" or "an embodiment" should not be construed as necessarily all referring to the same or different embodiments.

[0016] It will be appreciated that in the development of any actual implementation (as in any development project), many decisions must be made to achieve the developer's specific goals (e.g., to meet system and business-related constraints), and that these goals will vary from one implementation to another. It will also be appreciated that such a development effort may be complex and time-consuming, but will nevertheless be a routine undertaking for those of ordinary skill in the art who benefit from the image capture disclosed herein.

[0017] For the purposes of this disclosure, the term "lens" refers to a lens assembly, which may include multiple lenses. In one or more embodiments, the lens may be moved to different positions to capture images with different focal points. Additionally, in one or more embodiments, the lens may refer to any type of lens, such as a telescopic lens or a wide-angle lens. Thus, the term lens may refer to a single optical element or multiple elements configured as a stack or other arrangement.

[0018] For purposes of this disclosure, the term "camera system" refers to one or more lens assemblies and one or more sensor elements and other circuitry for capturing images. For purposes of this disclosure, a "camera" may include more than one camera system, such as a stereo camera system, a multi-camera system, or other camera system capable of sensing the depth of a captured scene.

[0019] See also Figure 1 , which shows a simplified block diagram of an electronic device 100 according to one or more embodiments of the present disclosure. The electronic device 100 may be part of a multi-function device such as a mobile phone, a tablet computer, a personal digital assistant, a portable music / video player, a wearable device, or any other electronic device including a camera system. Figure 1 An overall view of a system diagram capable of supporting image warping according to one or more embodiments is shown in block diagram form. Specifically, Figure 1 The electronic device 100 is shown as a computer system. The electronic device 100 can be connected to other network devices such as mobile devices, tablet devices, desktop devices, and network storage devices such as servers through a network.

[0020] The electronic device 100 may include a processor 130, such as a central processing unit (CPU). The processor 130 may be a system on a chip such as those found in mobile devices, and may include one or more dedicated graphics processing units (GPUs). In addition, the processor 130 may include multiple processors of the same or different types. The electronic device 100 may also include a memory 140. The memories 140 may each include one or more different types of memory that can be used to perform device functions in conjunction with the processor 130. For example, the memory 140 may include a cache, a ROM, and / or a RAM. The memory 140 may store various programming modules during execution, including the warping module 155 and the position detection module 160.

[0021] The electronic device 100 may also include one or more cameras, such as a front camera 110 and a rear camera 120. The cameras 110 and 120 may each include an image sensor, a lens stack, and other components that can be used to capture images. In one or more embodiments, the cameras may point to different directions in the electronic device. For example, the front camera 110 may be located in or on a first surface of the electronic device 100, and the rear camera 120 may be located in or on a second surface of the electronic device 100. In one or more embodiments, the first surface and the second surface may be opposite surfaces of the electronic device 100. For another example, the rear camera may be configured to capture images of the environment from the back of the electronic device 100, while the front camera is configured to capture images of the user or other objects in front of the device. Although two cameras are shown, as will be described in more detail below, in one or more embodiments, the electronic device 100 may include a single camera, or may include additional cameras, such as two rear cameras and one front camera. In one or more embodiments, the electronic device 100 may also include a display 150. The display 150 may be any kind of display device, such as an LCD (liquid crystal display), an LED (light emitting diode) display, an OLED (organic light emitting diode) display, etc. In addition, the display 150 may be a semi-opaque display, such as a head-up display, etc. The display 150 may be located on the same surface of the electronic device 100 as the front camera 110.

[0022] Although electronic device 100 is described as including the above numerous components, in one or more embodiments, various components may be distributed across multiple devices. In addition, additional components may be used, and some combinations of the functions of any component may also be combined.

[0023] In one or more embodiments, the position detection module 160 is configured to determine the spatial relationship between the user and the device. For example, the position detection module 160 can detect the user's viewpoint using sensor data such as from the rear camera 120 or the sensor 175. For example, the position detection module 160 can track the relative position of the viewpoint compared to the device. In one or more embodiments, the viewpoint can be a point in space from which a scene is viewed via a display. For example, the viewpoint can be located at or near the user's eyes, such as in the middle of the user's eyes, or at a position near the user's eyes. In one or more embodiments, the position detection module 160 can track the relative posture of the eyes to the display 150 of the electronic device 100. That is, the position detection module 160 can implement eye tracking and / or gaze detection technology. In one or more embodiments, the viewpoint can be regarded as a position in space where the user's eyes are located or near. The posture of the user's eyes and the relationship between the eyes and the electronic device can be determined using the front camera 110 to capture an image of the user's eyes. The camera that captures the image of the user's eyes can be a traditional camera, including a lens stack and a sensor, or can be a lensless camera. According to one or more embodiments, the relationship between the eye posture and the display can be based on one or more intermediate measurements. For example, the position detection module 160 can determine the relationship between the viewpoint and the front camera 110 or other components of the electronic device. From there, the position detection module 160 can determine the spatial relationship between the viewpoint and the display 150 based on the known spatial relationship between the various components of the electronic device 100. In addition, in one or more embodiments, a sensor (e.g., a depth sensor) can be used to determine the distance between the viewpoint and the electronic device, as well as the angular position and orientation between the viewpoint and the electronic device.

[0024] In one or more embodiments, the position detection module 160 may additionally or alternatively utilize other sensor data from the sensor 175 that can track the movement and position of a viewpoint, such as the user's eyes. According to one or more embodiments, the position detection module 160 may also track the target at which the eyes are looking. Gaze detection methods may be utilized to identify areas or objects of interest in the real environment. For example, the position detection module 160 may determine the location on the display 150 that the user is looking at, and identify an object or area in the real environment associated with the location.

[0025] According to one or more embodiments, the warping module 155 may include instructions that can be executed by the processor 130 to generate a warped real environment image captured by the rear camera 120. In one or more embodiments, the warping module may obtain an image of the real environment, for example, through the rear camera 120. The warping module 155 may generate a warped real environment view based on the viewpoint determined by the position detection module 160, so that the image of the environment appears to be substantially adjacent to or seamless with the real environment from the viewpoint. In other words, in one or more embodiments, the image may be warped so that the image of the scene on the display of the electronic device appears to be substantially adjacent to the scene viewed around the electronic device from the viewpoint. For example, a first portion of the scene may be visible from a viewpoint in the display, and may appear to be substantially adjacent to a second portion of the scene visible around the electronic device. If the view of the scene is interrupted from the viewpoint by the boundary of the display and / or the electronic device, the scene may appear to be substantially adjacent.

[0026] In one or more embodiments, the warping module 155 warps the image by projecting the image based on the camera intrinsics to obtain a 2D mapping for the 3D environment. The back projection may be based on the target depth. The target depth may be predetermined or determined based on an area of ​​interest or a point of interest in the image. For example, the depth may be selected based on the depth of the object of interest. In one or more embodiments, the function of the electronic device 100 may be used to select the depth. For example, in one or more embodiments, the depth may be selected based on the detected object, for example, using the autofocus operation of the electronic device 100. As another example, the warping module 155 may use sensor data to detect the depth of the field of the scene of the real environment. Sensor data may be used to determine the depth based on stereo methods, structured light, flight time, etc. Stereoscopic perspective methods may include using a stereo camera system to determine the depth in the field of view, such as by stereo estimation. Structured light operations may involve projecting a pattern onto a scene and determining the depth of the scene based on the appearance of the projected pattern. Flight time operations may include determining the distance based on determining the time it takes for a wave to travel to an object in the scene. A depth map that can be used to determine the depth may be determined or obtained. For example, the depth may be determined from a depth map of a particular point or area of ​​interest. According to one or more embodiments, the rear projected image may include identifying four image corners where the truncated image from the camera intersects the determined depth. Thus, based on the determined depth, four points in space may be identified. The resulting plane may be determined as the rear projected image.

[0027] In one or more embodiments, the warping module 155 also re-projects the post-projection image based on the determined relationship between the viewpoint and the display. In one or more embodiments, the re-projected image is displayed on the screen so that the image of the scene matches the "background" scene of the real world. In other words, the image is warped so that the plane defined by the four points is displayed on the display 150. Therefore, when viewed from the viewpoint, the portion of the scene hidden by the electronic device is visible in the display. The remainder of the scene can be visible from the viewpoint in the real world surrounding the electronic device. For another example, when viewed from the viewpoint, an object in the scene can be partially obfuscated by the electronic device. A first portion of the object can be visible in the real environment from the viewpoint, and a second portion of the object can be visible in the image of the environment displayed in the display.

[0028] In one or more embodiments, the warping module may also warp the image to better match the scene in the real environment. For example, the warping module may perform color correction on the image, such as performing a white balance function, so that the coloring of the image better matches the coloring in the real environment.

[0029] Figure 2 A method for providing a warped real environment display is shown in flowchart form. For the purpose of explanation, Figure 1 The following steps are described in the context of . However, it should be understood that various actions can be performed by alternative components. In addition, various actions can be performed in different orders. In addition, some actions can be performed simultaneously, and some actions may not be required, or other actions may be added.

[0030] The flowchart begins at 205, where the warping module 155 obtains an image of a scene in a real environment. In one or more embodiments, the warping module 155 can obtain the image from the rear camera 120. In one or more embodiments, the warping module 155 can obtain the image from another source such as an external camera or a peripheral camera. The image can include a still image, a video stream, a series of frames, a real-time image, etc.

[0031] The flowchart continues at 210, where the warping module 155 determines a spatial relationship between the viewpoint and the device. In one or more embodiments, the spatial relationship is obtained from the position detection module 160. According to one or more embodiments, the spatial relationship can be based on a relationship between a position of the viewpoint (such as a position at or near one or both eyes of a user) and a display 150 of the electronic device 100. In one or more embodiments, the eye from which the viewpoint is determined can be predetermined, for example, based on a dominant eye of a user, based on a position between two eyes of a user, and the like.

[0032] According to one or more embodiments, the viewpoint may change over time. For example, the user may move the electronic device relative to their face. The viewpoint may be tracked, for example, with a sensor on the electronic device facing the user (such as the front camera 110) or another sensor from the sensor 175 in the electronic device. For example, the user's viewpoint may be determined based on the detection of the user's gaze by performing head or eye tracking based on sensor data from the RGBD sensor data. In one or more embodiments, the sensor data may be used to track other facial landmarks, such as eye position, to determine the user's relative position from the device and the user's gaze direction. As another example, an RGB sensor only (i.e., an RGB sensor that does not capture depth information) may be used to track head position, facial landmarks, etc. In one or more embodiments, other devices (such as a time-of-flight laser ranging sensor) may be used to determine the user's depth information. Depth may also be inferred by features in an image captured by a front camera on the user's face, such as the spacing between the user's eyes based on the user's interpupillary distance. According to one or more embodiments, data about the user's gaze direction and the user's relative position may be used to determine the user's viewpoint.

[0033] The flowchart continues at 215, and the warping module 215 warps the image based on the spatial relationship between the viewpoint and the device together with the known intrinsics of the camera. In one or more embodiments, the warping module 215 can back-project the image based on the intrinsics of the camera that captured the image. The warping module 215 can then re-project the back-projected image based on the determined viewpoint.

[0034] The flowchart continues at 220, and the warping module 155 displays the warped image. As described above, the warping module 155 may display a single warped image based on a single determined viewpoint. Additionally, for example, if the display 150 includes a stereoscopic display, the warping module 155 may display two warped images, one corresponding to a viewpoint from each of the user's eyes.

[0035] At 225, a determination is made as to whether movement is detected. In one or more embodiments, a determination is made as to whether movement of the viewpoint relative to the electronic device is detected. If it is determined at 225 that movement is not detected, the warped area continues to be displayed, as shown at 220. In one or more embodiments, continuing to display the warped image may include continuing to generate an updated warped image based on updated image data received by the electronic device. In other words, if movement between the viewpoint and the electronic device is not detected, the image data may continue to be warped in the same manner. Thus, for example, if the scene changes, or the relationship between the scene and the electronic device changes, such as if the user translates the electronic device, the image may continue to be warped in the same or similar manner.

[0036] Returning to 225, if it is determined that movement is detected, the flowchart continues at 210 where a new spatial relationship between the viewpoint and the device is determined. Thus, in one or more embodiments, the new spatial relationship between the viewpoint and the device may require warping the image in a different manner to appear substantially contiguous with the real world environment.

[0037] See now Figure 3 , shows a flow chart of a method for warping an image of a scene in a real environment so that the image appears substantially contiguous or seamless with the real environment from a viewpoint according to one or more embodiments. In one or more embodiments, various actions occur as part of warping the image based on the spatial relationship between the viewpoint and the device and known intrinsics of the device, such as Figure 2 However, various actions may be performed in Figure 2 For the purpose of explanation, Figure 1 The following steps are described in the context of the various components described in . However, it should be understood that various actions can be performed by alternative components. In addition, various actions can be performed in different orders. In addition, some actions can be performed simultaneously, and some actions may not be required, or other actions may be added.

[0038] The flowchart starts at 305, and the warping module 155 selects the depth of the scene as the target depth. In one or more embodiments, in order to accurately transform (or reproject) the camera image, 3D information from each pixel in the image may be required. However, a 2D camera image may no longer have any depth information. Therefore, a single depth may be selected for each pixel in the image. In one or more embodiments, the depth may be selected based on the depth of the main object in the scene, such as a person or an object. In one or more embodiments, the depth may be based on the location in the scene that the user can view. Selecting the depth of the main object in the scene ensures that the main object that the user is viewing has approximately the correct scale and position so as to maintain the illusion of transparency of the electronic device. In one or more embodiments, the depth of the main object in the scene relative to the electronic device can be estimated by any method of determining depth. For example, if the rear camera 120 actually includes two cameras, the depth of the scene can be estimated using stereo estimation from a pair of cameras facing the scene. As another example, another sensor such as a time-of-flight laser ranging sensor or other depth sensing technology can be used to determine the depth of the scene.

[0039] The flow chart continues at 310, and the warping module back-projects the image based on the intrinsics of the camera and the selected depth. As shown, in one or more embodiments, the back-projected image may include various actions. As shown, at 315, the back-projected image may include identifying four image corners based on the intrinsics of the camera and the selected depth. At 320, the four image corners intersect with the plane defined by the display device to obtain a 2D image transformation. Then, at 325, the 2D image points are mapped to the 3D scene based on the 2D image transformation.

[0040] The flowchart continues at 330, where the position detection module 160 determines a viewpoint. For example, a viewpoint may be a point in space from which a user is viewing a scene. In one or more embodiments, a viewpoint may be a point in three spaces from which a scene is viewed, and may include directional information. For example, the directional information may identify the direction in which a user is looking in a scene or on a display. As shown in 330, in one or more embodiments, determining a viewpoint may include various actions. At 335, the position detection module 160 captures sensor data from a sensor facing the user. Then, at 340, the position detection module 160 detects a viewpoint based on the sensor data.

[0041] The flowchart continues at 345, where the warping module reprojects the image to the display device based on the user's viewpoint and the selected depth. The flowchart ends at 350, where the warping module adjusts the color processing of the image. In one or more embodiments, if the warping module 155 is able to determine the depth of the scene (e.g., if the rear camera 120 is actually a stereo camera, or if the sensor 175 includes an additional sensor that provides depth information), then each pixel can be reprojected based on the actual depth information and can appear more accurate than selecting a single depth at 305.

[0042] According to one or more embodiments, if the rear camera 120 includes a stereo camera or multiple cameras, each camera may have multiple viewpoints. Figure 3 The various steps of can be reproduced for each camera. In addition, according to one or more embodiments, Figure 3 Each of the various steps of may be completed for each eye. Thus, when determining the spatial relationship between the viewpoint and the device, the viewpoint for each eye may be determined at 210. Additionally, if the display 150 is a stereoscopic display, a warped image may be displayed for each eye at 220.

[0043] See now Figure 4 , shows a system diagram of an example setup for presenting a warped real environment image using a device according to one or more embodiments. Figure 4A user 410 is shown utilizing an electronic device 100 including a front camera 110 and a rear camera 120. In one or more embodiments, the user can view the real environment from a viewpoint through the front camera 110 of the electronic device 100. In one or more embodiments, the image of the real environment 405 includes a field of view similar to the user's field of view 425 if the user were to look directly at the real environment 405.

[0044] Figure 5 An example system diagram of an augmented reality device for warping an image according to one or more circumstances is shown. Figure 5 Three time points during various parts of practicing the embodiment are shown. At 505, a first version of the electronic device 100 is shown. The electronic device 100 includes a display 150, which can display a preview image 530A of two frames 535 and 540 in a real environment. At 505, the electronic device 100 captures an image 530A, including two frames 535 and 540. The image 530A can be captured by the rear camera 120.

[0045] At 510, in one or more embodiments, the position detection module may determine the viewpoint of the user 410. In one or more embodiments, the viewpoint may be detected using the front camera 110 or some other sensor on the electronic device 100. In addition, in one or more embodiments, the warping module 155 also back-projects the image 530A to a selected depth 545. In one or more embodiments, the selected depth may be determined using a set depth or based on an object detected in the scene. For example, the depth may be determined using autofocus or a similar function to determine the depth of a particular point of interest in the scene. The image may be back-projected by intersecting the four points defining the camera view frustum with a plane defined by the selected depth. The warping module then reprojects the image packet to the display device based on the viewpoint of the user 410 and the selected depth 545.

[0046] At 515, an exemplary flow chart shows an updated image 530B of the electronic device 100 from the viewpoint of the user 410. As shown, both boxes 535 and 540 are displayed as if the display device is out of the way, because it turns out that box 540 is only partially displayed in image 530B, but the rest of the box is still visible and looks seamless in the view of the adjacent device in the real environment. According to one or more embodiments, if the object in the image appears with substantially the same zoom and position, the scene looks seamless in the image and the real environment, as if the user is looking at the scene from the same viewpoint rather than through the display. In one or more embodiments, if the environment in the image looks substantially adjacent to the real environment from the viewpoint, the image looks seamless. For example, if the user views the real environment through the electronic device with the same zoom and distance as when the electronic device does not exist, the image can be substantially adjacent to the real environment. Therefore, any frame around the display or electronic device can interfere with the view of the real environment, but the image can still be considered to be seamless or substantially adjacent to the background real environment.

[0047] See now Figure 6 , which presents an example system diagram showing a warping method according to one or more embodiments. At 600, a user at viewpoint 410 views a scene including boxes 535 and 540 through electronic device 100. At 600, four corners of the truncated head from the camera to the determined scene depth are identified. As described above, the depth can be determined using a variety of methods, such as object detection technology, autofocus, user input, etc. The four corners of the truncated head are shown as 602A, 602B, 602C, and 602D. The plane defined by the four corners 602 can then be back-projected onto the display of the electronic device 100. In one or more embodiments, the image captured in the plane can be back-projected based on the spatial relationship between the camera and the display of the electronic device 100. In one or more embodiments, the spatial relationship between the camera and the display can be understood or encoded, for example, by the electronic device 100, or can be determined in other ways.

[0048] Turning to 610, the image plane defined by the four corners 602 is reprojected onto the display of the electronic device 100. In one or more embodiments, the reprojection is based on the viewpoint, such as the position and gaze direction of the eyes of the user 410. Therefore, based on the truncation from the viewpoint to the plane, the reprojection includes four corners 604A, 604B, 604C and 604D. In one or more embodiments, the reprojected image generates an image for display in the electronic device 100. In one or more embodiments, reprojecting the image based on the viewpoint shows that the scene including the frames 535 and 540 can cause the frames 535 and 540 to appear through the electronic device 100 with the same scale and position as they appear in the real world environment. For example, if the frame 540 is partially enclosed by the electronic device from the gaze direction and position of the viewpoint, the enclosed portion of the frame 540 will appear in the display of the electronic device 100 as if the electronic device 100 does not enclose the portion of the image. In addition, in one or more embodiments, the enclosed portion and the unenclosed portion of the frame 540 can appear substantially adjacent from the viewpoint. For example, the frame 540 may appear contiguous except for the sides of the electronic device 100 surrounding the display.

[0049] Reference now Figure 7 , which shows a simplified functional block diagram of an exemplary multifunction device 700 according to one embodiment. The multifunction electronic device 700 may include a processor 705, a display 710, a user interface 715, graphics hardware 720, device sensors 725 (e.g., proximity sensor / ambient light sensor, accelerometer and / or gyroscope), a microphone 730, one or more audio codecs 735, one or more speakers 740, communication circuitry 745, digital image capture circuitry 750 (e.g., including a camera system), one or more video codecs 755 (e.g., supporting a digital image capture unit), a memory 760, a storage device 765, and a communication bus 770. The multifunction electronic device 700 may be, for example, a digital camera or a personal electronic device such as a personal digital assistant (PDA), a personal music player, a mobile phone, or a tablet computer.

[0050] The processor 705 may execute instructions necessary for implementing or controlling the operation of many functions performed by the device 700 (e.g., generation and / or processing of images such as disclosed herein). The processor 705 may, for example, drive the display 710 and may receive user input from the user interface 715. The user interface 715 may allow a user to interact with the device 700. For example, the user interface 715 may take a variety of forms, such as buttons, keypads, turntables, click-touch turntables, keyboards, display screens, and / or touch screens. The processor 705 may also, for example, be a system on a chip, such as those present in a mobile device, and include a dedicated graphics processing unit (GPU). The processor 705 may be based on a reduced instruction set computer (RISC) or complex instruction set computer (CISC) architecture or any other suitable architecture, and may include one or more processing cores. The graphics hardware 720 may be dedicated computing hardware for processing graphics and / or assisting the processor 705 to process graphics information. In one embodiment, the graphics hardware 720 may include a programmable GPU.

[0051] The image capture circuit 750 may include two (or more) lens assemblies 780A and 780B, wherein each lens assembly may have a separate focal length. For example, relative to the focal length of lens assembly 780B, lens assembly 780A may have a short focal length. Each lens assembly may have a separate associated sensor element 790. Alternatively, two or more lens assemblies may share a common sensor element. The image capture circuit 750 may capture still images and / or video images. The output from the image capture circuit 750 may be processed at least in part by one or more video codecs 755 and / or processor 705 and / or graphics hardware 720, and / or a dedicated image processing unit or pipeline incorporated within the circuit 765. The images captured in this way may be stored in the memory 760 and / or storage device 765.

[0052] The sensor and camera circuit system 750 can capture still images and video images that can be processed in accordance with the present disclosure at least in part by the following devices: one or more video codecs 755 and / or processors 705 and / or graphics hardware 720 and / or dedicated image processing units incorporated into the circuit 750. The images captured in this way can be stored in memory 760 and / or storage device 765. Memory 760 may include one or more different types of media used by processor 705 and graphics hardware 720 to perform device functions. For example, memory 760 may include memory cache, read-only memory (ROM) and / or random access memory (RAM). Storage device 765 can store media (such as audio files, image files, and video files), computer program instructions or software, preference information, device profile information, and any other suitable data. The storage device 765 may include one or more non-transitory computer-readable storage media, including, for example, magnetic disks (fixed hard disks, floppy disks, and removable disks) and tapes, optical media (such as CD-ROMs and digital video disks (DVDs)), and semiconductor storage devices (such as electrically programmable read-only memories (EPROMs) and electrically erasable programmable read-only memories (EEPROMs). The memory 760 and the storage device 765 may be used to tangibly retain computer program instructions or codes organized into one or more modules and written in any desired computer programming language. When executed by, for example, the processor 705, such computer program code may implement one or more of the methods described herein.

[0053] According to one or more embodiments, the camera-based transparent display can have a variety of uses. For example, the above-described techniques can be used for image stabilization. That is, the warping function used to warp an image can also be used to stabilize the image. In another embodiment, the camera-based transparent display can be used for machine learning. For example, a camera can capture an image of a scene and classify the scene and / or objects within the scene. In one or more embodiments, by tracking the object that the user's eyes are looking at, the object that should be classified can be identified.

[0054] In one or more embodiments, a camera-based transparent display can be used to present virtual information on a display, such as for augmented reality purposes. For example, digital information associated with a particular location can be displayed at a more accurate location perceived through the display from a particular viewpoint. For example, an image of a scene can be enhanced by additional text or image data at a specific point within the scene in a warped image. In one or more embodiments, the digital information can be associated with a specific object in the scene. When an object falls within the warped image (or when the object is visible in the display), the digital information associated with the object can be displayed in the warped image together with the object. In one or more embodiments, when the warped image changes, whether the scene changes or the warping changes due to a change in the viewpoint or the position of the device relative to the scene, the digital information can also change.

[0055] The scope of the subject matter disclosed in this disclosure should be determined with reference to the appended claims, along with the full scope of equivalents to which the claims are entitled. In the appended claims, the terms "including" and "in which" are used as the plain-English equivalents of the respective terms "comprising" and "wherein."

Claims

1. A method for presenting an image of a scene, comprising: capturing an image of a scene via a camera of an electronic device, wherein the electronic device includes the camera and a stereoscopic display, and wherein the camera and the stereoscopic display have a first spatial relationship; determining a second spatial relationship between a first eye and the stereoscopic display of the electronic device; warping the image based on the first spatial relationship and the second spatial relationship to obtain a first warped image; determining a third spatial relationship between a second eye and the stereoscopic display; warping the image based on the first spatial relationship and the third spatial relationship to obtain a second warped image; as well as The first warped image and the second warped image are presented on the stereoscopic display.

2. The method of claim 1, wherein the first eye and the second eye correspond to a viewpoint, and wherein from the viewpoint, the image of the scene is substantially contiguous with the scene visible beyond an edge of the stereoscopic display.

3. The method of claim 1 , wherein warping the image further comprises: The image is back-projected based on intrinsics of the camera and depth in the scene.

4. The method according to claim 3, further comprising: selecting the depth using at least one selected from the group consisting of: an autofocus operation, a stereo estimation, a time-of-flight operation, and a structured light operation; as well as The image is back-projected based on the selected depth.

5. The method of claim 3, wherein rear projecting the image comprises: identifying four image corners based on the intrinsics of the camera and the depth; as well as The four image corners are intersected with a plane defined by the stereoscopic display to obtain a 2D image transform.

6. The method according to claim 1, further comprising: determining the gaze direction of the first eye, Wherein, the second spatial relationship is based on the gaze direction of the first eye. The method of claim 1 , wherein warping the image further comprises performing a white balancing operation on the image.

8. The method according to claim 1, further comprising: identifying objects in the scene; as well as Wherein warping the image to obtain a first warped image comprises: warping the image so that a first portion of the object is visible in the scene from a viewpoint of the first eye, and an image of a second portion of the object is visible on the stereoscopic display from a viewpoint of the first eye, and wherein the first portion and the second portion are substantially aligned.

9. A non-transitory computer readable medium comprising computer readable code for presenting an image of a scene, the computer readable code being executable by one or more processors to: capturing an image of a scene via a camera of an electronic device, wherein the electronic device includes the camera and a stereoscopic display, and wherein the camera and the stereoscopic display have a first spatial relationship; determining a second spatial relationship between a first eye and the stereoscopic display of the electronic device; warping the image based on the first spatial relationship and the second spatial relationship to obtain a first warped image; determining a third spatial relationship between a second eye and the stereoscopic display; warping the image based on the first spatial relationship and the third spatial relationship to obtain a second warped image; as well as The first warped image and the second warped image are presented on the stereoscopic display.

10. The non-transitory computer-readable medium of claim 9, wherein the first eye and the second eye correspond to a viewpoint, and wherein from the viewpoint, an image of a first portion of the scene is substantially contiguous with a second portion of the scene visible beyond an edge of the stereoscopic display.

11. The non-transitory computer readable medium of claim 9, wherein the computer readable code for warping the image further comprises computer readable code for: The image is back-projected based on intrinsics of the camera and depth in the scene.

12. The non-transitory computer readable medium of claim 11 , further comprising computer readable code for: selecting the depth using at least one selected from the group consisting of: an autofocus operation, a stereo estimation, a time-of-flight operation, and a structured light operation; and The image is back-projected based on the selected depth.

13. The non-transitory computer readable medium of claim 11, wherein the computer readable code for post-projecting the image comprises computer readable code for: identifying four image corners based on the intrinsics of the camera and the depth; and The four image corners are intersected with a plane defined by the stereoscopic display to obtain a 2D image transform.

14. The non-transitory computer readable medium of claim 9, further comprising computer readable code for: determining the gaze direction of the first eye, in, The second spatial relationship is based on the gaze direction of the first eye.

15. The non-transitory computer readable medium of claim 9, wherein warping the image further comprises performing a white balance operation on the image.

16. The non-transitory computer readable medium of claim 9, further comprising computer readable code for: identifying objects in the scene; Wherein the computer readable code for warping the image to obtain a first warped image comprises computer readable code for: warping the image so that a first portion of the object is visible in the scene from a viewpoint of the first eye, and an image of a second portion of the object is visible on the stereoscopic display from a viewpoint of the first eye, and wherein the first portion and the second portion are substantially aligned.

17. A system comprising: one or more processors; as well as One or more computer readable media including computer readable code for presenting an image of a scene, the computer readable code being executable by the one or more processors to: capturing an image of a scene via a camera of an electronic device, wherein the electronic device includes the camera and a stereoscopic display, and wherein the camera and the stereoscopic display have a first spatial relationship; determining a second spatial relationship between a first eye and the stereoscopic display of the electronic device; warping the image based on the first spatial relationship and the second spatial relationship to obtain a first warped image; determining a third spatial relationship between a second eye and the stereoscopic display; warping the image based on the first spatial relationship and the third spatial relationship to obtain a second warped image; as well as The first warped image and the second warped image are presented on the stereoscopic display.

18. The system of claim 17, wherein the first eye and the second eye correspond to a viewpoint, and wherein from the viewpoint, an image of a first portion of the scene is substantially contiguous with a second portion of the scene visible beyond an edge of the stereoscopic display.

19. The system of claim 17, wherein the computer readable code for warping the image further comprises computer readable code for: The image is back-projected based on intrinsics of the camera and depth in the scene.

20. The system of claim 19, further comprising computer readable code for: selecting the depth using at least one selected from the group consisting of: an autofocus operation, a stereo estimation, a time-of-flight operation, and a structured light operation; and The image is back-projected based on the selected depth.