Dynamic resolution of depth conflicts in telepresence

Abstract

Systems and methods are described for determining a capture volume associated with image content captured by at least one camera, determining a depth associated with the image content, defining a viewing range, depicting a stereo effect in the viewing range when viewing the image content, determining a depth conflict between the image content and a boundary associated with the viewing range, the determining comprising detecting that at least a portion of the image content extends beyond the boundary associated with the viewing range and resolving the depth conflict for the at least a portion using the viewing range and at least one user interface element, and generating modified image content for rendering using the resolved depth conflict.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to and the benefit of U.S. Provisional Patent Application No. 63 / 198,473, filed on October 21, 2020, entitled “DYNAMIC RESOLUTION OF DEPTHCONFLICTS IN TELEPRESENCE,” the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] The present specification generally relates to methods, devices, and algorithms for resolving depth conflicts in three-dimensional (3D) telepresence systems. Background Art

[0004] Stereoscopic display devices typically provide content and convey a sense of depth to a viewer. Such displays may include a frame with an edge around the display screen that may unnaturally cut off the view of a portion of the content to be rendered on the display screen. The cut-off view may create conflicting visual cues for the viewer. Such a conflict may weaken the 3D effect, which may cause visual fatigue to the viewer. Summary of the invention

[0005] A system having one or more computers may be configured to perform specific operations or actions by having software, firmware, hardware, or a combination thereof installed on the system, which, in operation, causes the system to perform the actions. One or more computer programs may be configured to perform specific operations or actions by including instructions that, when executed by a data processing device, cause the device to perform the actions.

[0006] In one general aspect, systems and methods are described that utilize at least one processing device to perform operations, the operations including: determining a capture volume associated with image content captured by at least one camera; determining a depth associated with the image content; defining a viewing range within the capture volume based on the depth, wherein a stereoscopic effect is depicted when viewing the image content; determining a depth conflict between the image content and a boundary associated with the viewing range, the determination including: detecting that at least a portion of the image content extends beyond the boundary associated with the viewing range, and in response to determining the depth conflict, resolving at least a portion of the depth conflict using the viewing range and at least one user interface element; and generating modified image content for rendering with the resolved depth conflict, the modified image content including a portion of the image content replaced by the at least one user interface element.

[0007] These and other aspects may include, alone or in combination, one or more of the following. For example, the system and method may include: detecting a depth conflict between at least a portion of the image content and a boundary associated with the viewing range includes: using at least some of the determined depths associated with the image content to generate a plurality of three-dimensional voxels, the plurality of three-dimensional voxels representing locations in a plane of a display rendering the image content, the distance being from the at least a portion to the boundary, wherein at least one user interface element is selected based on the distance.

[0008] In some implementations, the boundary is associated with at least one edge of a lenticular display device; a depth conflict is determined based on a tracked head position of a user viewing image content at a remote lenticular display device; and resolving the depth conflict includes: adjusting the size of a user interface element based on the tracked user head position of the user.

[0009] In some implementations, resolving the depth conflict includes generating at least one user interface element as a box that overlays at least some of the image content, the at least one box adapted to accommodate movement depicted in the image content. In some implementations, a side of the box corresponding to at least a portion that extends beyond the border is placed in a different plane parallel to and in front of the remainder of the box to generate a visually perceived tilt of the box from vertical to a non-zero angle from vertical.

[0010] In some implementations, the at least one user interface element depicts a user interface layer having a thumbnail image of an attached software program, the software program being executed in memory by the at least one processing device when the image content is accessed. In some implementations, the user interface element includes a blur overlay that begins at the border and ends at a predefined position associated with a size of a display device depicting the image content, wherein a blur radius associated with the blur overlay is increased at a threshold distance from the border.

[0011] In some implementations, the blurred overlay includes a user interface layer having a thumbnail image of an attached software program, the software program being executed in memory by at least one processing device when the image content attachment is accessed, and the blurred overlay is elliptical. In some implementations, the blurred overlay is a gradient blur that fades from a left center portion of the overlay to a left edge of the image content and from a right center portion of the overlay to a right edge of the video content. In some implementations, the gradient blur is placed at a central location associated with the depth conflict and gradually blurs outward to a first edge and a second edge associated with the depth conflict. In some implementations, resolving the depth conflict includes animating at least one user interface element to hide at least a portion of the image content with the modified image content.

[0012] The above-described systems and aspects may be configured to perform any combination of the above-described aspects, wherein each aspect may be implemented with any suitable combination of the above-listed features and aspects.

[0013] Implementations of the described techniques may include hardware, methods or processes or computer software on a computer-accessible medium. Details of one or more implementations are set forth in the accompanying drawings and the following description. Other features will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a block diagram illustrating an example 3D content system for displaying image content on a display device according to implementations described throughout this disclosure.

[0015] Figure 2 is a block diagram of an example system for dynamically resolving depth conflicts for stereoscopic displays according to implementations described throughout this disclosure.

[0016] Figure 3 is an example display device illustrating depth conflicts for a user according to implementations described throughout this disclosure.

[0017] Figure 4 is a block diagram illustrating examples of local capture volumes and ranges of movement within the capture volumes according to implementations described throughout this disclosure.

[0018] Figure 5 is a block diagram illustrating an example of a remote capture volume relative to a local capture volume according to implementations described throughout this disclosure.

[0019] Figure 6 is a block diagram illustrating an example of display edge clipping of a capture volume according to implementations described throughout this disclosure.

[0020] Figures 7A to 7C is a block diagram illustrating an example of visually perceived tilt of a display device according to implementations described throughout this disclosure.

[0021] Figures 8A to 8B is a block diagram illustrating an example of resolving depth conflicts with synthesized image content according to implementations described throughout this disclosure.

[0022] Figures 9A to 9B is a block diagram illustrating an example of resolving depth conflicts by dynamically adjusting a display window of a display device according to implementations described throughout the present disclosure.

[0023] Figures 10A to 10G is a block diagram illustrating an example of resolving depth conflicts by adjusting borders and / or edges according to implementations described throughout this disclosure.

[0024] Fig.11 is a block diagram illustrating an example of resolving depth conflicts using segmented virtual content according to implementations described throughout this disclosure.

[0025] Fig.12 is a block diagram illustrating an example of application content placed on virtual content according to implementations described throughout this disclosure.

[0026] Fig.13 is a flow chart illustrating one example of a process for resolving depth conflicts in a 3D content system according to implementations described throughout this disclosure.

[0027] Fig.14 Examples of computer devices and mobile computer devices that may be used with the techniques described herein are shown.

[0028] Like reference numbers in the various drawings indicate like elements. DETAILED DESCRIPTION

[0029] In general, this document describes examples related to detecting, analyzing, and correcting depth conflicts within three-dimensional (3D) video content. Depth conflicts may occur relative to user interface (UI) elements within the video content. For example, in 3D video content (e.g., stereoscopic video), the depth perceived by a user may vary within a portion of the video. Such depth variations may result in depth conflicts at portions of the video, between UI elements in the video, and / or at edges associated with the content in the video and / or a display depicting the content.

[0030] Depth conflicts cause discomfort to the user, which may cause difficulty for the user when focusing or panning portions of image content and / or focus within UI elements. For example, a depth conflict may occur if portions of a user are cut off at the edge of a display screen depicting image content based on the maximum capture space that a camera capturing such content has. The disappearance of a hand (or other object or user portion) beyond the edge of the display may be inconsistent with what the user's eyes (i.e., brain) expect to occur. In particular, movement beyond an edge (or other defined boundary) may cause depth conflicts including artifacts that cause depth cues to the user to conflict with the expected results of movement in the content (i.e., as perceived by the user's brain).

[0031] The systems and methods described herein are configured to maximize the user's comfort zone by minimizing or eliminating depth conflicts. Maximizing the user's comfort zone may include evaluating the depth of the area both in front of and behind the display screen so as to place UI elements within the 3D video (or other image content) so that the placed UI elements comply with depth rules and / or minimize violations of such rules. In some implementations, the systems and methods described herein may reduce or eliminate depth conflicts by generating and rendering transparent, translucent, blurred, partially blurred, gradient-based blurred UI elements, and the like.

[0032] The systems and methods described herein can provide several advantages over conventional video rendering systems. For example, the systems and methods described herein can dynamically modify the depth of UI elements or objects depicted as image content and / or video content based on the depth of other objects within the 3D video. Unlike conventional systems that remove content from a video when the content causes a depth conflict, the systems and methods described herein are used to improve the view of content that may cause a depth conflict, as described in examples throughout the present disclosure.

[0033] As used herein, depth may refer to the perceived distance from the location of content depicted on a display screen. As used herein, depth cues may refer to an indication or representation of distance that aids in the visual perception of depth that a user may understand through the eyes. Example depth cues may include any or all of vergence, monocular motion parallax, binocular parallax, linear perspective, texture gradients, blending, retinal image size, overlay, chromaticity, shading, and aerial perspective.

[0034] As used herein, a depth conflict may refer to: an inconsistency between depth cues perceived by a user. For example, a user may perceive depth based on any number of depth cues corresponding to the user's field of view. A depth conflict may occur when two or more such depth cues are incompatible with each other. Example depth conflicts may include, but are not limited to, near conflicts, far conflicts, image conflicts, and / or occlusion and stereoscopic conflicts.

[0035] Example corrections to depth conflicts may include, but are not limited to, eliminating depth conflicts, modifying pixels or voxels to modify depth conflicts, reducing depth conflicts, and / or generating and / or moving content to reduce or eliminate depth conflicts, etc. In some implementations, the systems and methods described herein select one or more depth conflict corrections from any number of depth conflict correction techniques. For example, the systems and methods described herein may combine two or more UI elements to correct depth conflicts.

[0036] In some implementations, for example, the techniques described herein can be used to synthesize depth-corrected images that appear accurate and realistic for display on a screen of a 2D or 3D display used in a multi-way video conference. The techniques described herein can be used to generate and display accurate and realistic views of users, objects, and UI content (e.g., image content, video content), and correct for user movement or 3D display conflicts.

[0037] Figure 1 1 is a block diagram illustrating an example 3D content system 100 for displaying content in a stereoscopic display device according to implementations described throughout this disclosure. The 3D content system 100 may be used by one or more users to, for example, conduct a video conference communication in which the content is viewed in 3D (e.g., a telepresence session) on a single 3D display or other device. In general, Figure 1 The system can be used to capture video and / or images of users and / or objects during a video conference and use the systems and techniques described herein to correct depth conflicts that may occur in the display of users, objects, and / or other additional UI content.

[0038] The system 100 can benefit from the use of the techniques described herein because such techniques can generate, modify, update, and display a corrected (e.g., resolved depth) view of a capture volume associated with a particular display screen device. In some implementations, the capture volume can be used to determine how to correct depth conflicts. For example, the resolved depth of the view can be displayed to another user via the system 100 in 2D and / or 3D.

[0039] As used herein, a capture volume may refer to a physical volume of space that may be bounded by one or more boundaries imposed by one or more cameras that capture image / video content within the defined one or more boundaries. In some implementations, a capture volume may refer to a viewing volume in which a user may be continuously tracked by multiple image sensors (e.g., cameras).

[0040] like Figure 1 As shown, the 3D content system 100 is being used by a first user 102 and a second user 104. For example, the users 102 and 104 are using the 3D content system 100 to participate in a 3D telepresence session. In such an example, the 3D content system 100 may allow each of the users 102 and 104 to see a highly realistic and visually coordinated representation of the other user, thereby facilitating the users to interact in a manner similar to being physically present to each other. The system 100 may access a depth conflict resolver to improve, correct, reduce, or otherwise modify depth conflicts that may occur during a session of the 3D telepresence session.

[0041] Each user 102, 104 may have a corresponding 3D system. Here, user 102 has 3D system 106, and user 104 has 3D system 108. 3D systems 106, 108 may provide functions related to 3D content, including but not limited to: capturing images for 3D display, processing and presenting image information, and processing and presenting audio information. 3D system 106 and / or 3D system 108 may constitute: a collection of sensing devices integrated into a unit. 3D system 106 and / or 3D system 108 may include reference Figure 2 and 14 Some or all of the components described.

[0042] The 3D content system 100 may include one or more 2D or 3D displays. Here, a 3D display 110 is provided for the 3D system 106, and a 3D display 112 is provided for the 3D system 108. The 3D displays 110, 112 may use any of a variety of types of 3D display technologies to provide an autostereoscopic view for a corresponding viewer (here, for example, user 102 or user 104). In some implementations, the 3D displays 110, 112 may be stand-alone units (e.g., self-supporting or hung on a wall). In some implementations, the 3D displays 110, 112 may include or utilize wearable technology (e.g., a controller, a head-mounted display, smart glasses, a watch, etc.). In some implementations, the displays 110, 112 may be 2D displays.

[0043] In general, displays such as displays 110, 112 can provide images that approximate the 3D optical properties of physical objects in the real world without using a head mounted display (HMD) device. In general, the displays described herein include a flat panel display, a lenticular lens (e.g., a microlens array), and / or a parallax barrier to redirect images to multiple different viewing areas associated with the display.

[0044] In some implementations, the displays 110, 112 may include high-resolution and glasses-free lenticular 3D displays. For example, the displays 110, 112 may include a microlens array (not shown) including a plurality of lenses (e.g., microlenses) with glass spacers coupled (e.g., bonded) to the microlenses of the display. The microlenses may be designed such that from a selected viewing position, a left eye of a user of the display may view a first set of pixels while a right eye of the user may view a second set of pixels (e.g., where the second set of pixels is mutually exclusive to the first set of pixels).

[0045] In some example displays, there may be a single position that provides a 3D view of the image content (e.g., user, object, content, etc.) provided by such a display. The user can sit in a single position to experience appropriate parallax, minimal distortion, and a true 3D image. If the user moves to a different physical position (or changes the head position or eye gaze position), the image content (e.g., user, object worn by the user, and / or other object) may begin to appear less real, 2D, and / or distorted. The systems and techniques described herein can reconfigure the image content projected from the display to ensure that the user can move around while still experiencing appropriate parallax, low distortion, minimal depth conflict, and real-time, true 3D images. Therefore, the systems and techniques described herein provide the following advantages: maintaining and providing 3D image content and objects for display to the user, regardless of any user movement that occurs while the user is viewing the 3D display.

[0046] like Figure 1 As shown, 3D content system 100 can be connected to one or more networks. Here, network 114 is connected to 3D system 106 and to 3D system 108. Network 114 can be a publicly available network (e.g., the Internet) or a private network, to name just two examples. Network 114 can be wired or wireless, or a combination of both. Network 114 can include or utilize one or more other devices or systems, including, but not limited to, one or more servers (not shown).

[0047] The 3D systems 106, 108 may include a plurality of components associated with the capture, processing, transmission or reception of 3D information and / or the presentation of 3D content. The 3D systems 106, 108 may include one or more cameras for capturing image content of an image to be included in the 3D presentation. Here, the 3D system 106 includes cameras 116 and 118. For example, the camera 116 and / or the camera 118 may be substantially disposed within the housing of the 3D system 106, such that the objective lens or lens of the corresponding camera 116 and / or 118 captures the image content through one or more openings in the housing. In some implementations, the cameras 116 and / or 118 may be separate from the housing, such as in the form of a stand-alone device (e.g., with a wired and / or wireless connection to the 3D system 106). The cameras 116 and 118 may be positioned and / or oriented so as to capture a sufficiently representative view of a user (e.g., user 102).

[0048] The placement of cameras 116 and 118 may be arbitrarily selected, while cameras 116 and 118 generally do not obstruct the view of 3D display 110 to user 102. For example, one of cameras 116, 118 may be located somewhere above the face of user 102, while the other camera may be located somewhere below the face. For example, one of cameras 116, 118 may be located somewhere to the right of the face of user 102, while the other camera may be located somewhere to the left of the face. For example, 3D system 108 may include cameras 120 and 122 in a similar manner. Additional cameras may be used. For example, a third camera may be placed near or behind display 110.

[0049] In some implementations, the 3D systems 106, 108 may include one or more depth sensors to capture depth data to be used in a 3D presentation. Such depth sensors may be considered part of a depth capture component in the 3D content system 100 for characterizing the scene captured by the 3D systems 106 and / or 108 in order to correctly represent the scene on the 3D display. In addition, the system may track the position and orientation of the viewer's head so that the 3D presentation may be rendered with an appearance corresponding to the viewer's current viewing point. Here, the 3D system 106 includes a depth sensor 124. In a similar manner, the 3D system 108 may include a depth sensor 126. Any of a variety of types of depth sensing or depth capture may be used to generate and / or modify depth data.

[0050] In some implementations, assisted stereo depth capture is performed. For example, a light point may be used to illuminate the scene, and stereo matching may be performed between two corresponding cameras. This illumination may be performed using waves of a selected wavelength or range of wavelengths. For example, infrared (IR) light may be used. In some implementations, for example, when generating a view on a 2D device, a depth sensor may not be utilized.

[0051] The depth data may include or be based on any information about a scene that reflects the distance between a depth sensor (e.g., depth sensor 124) and an object or UI element in the scene. For content in an image corresponding to an object in the scene, the depth data reflects the distance (or depth) to the object. For example, the spatial relationship between a camera and a depth sensor may be known and may be used to correlate an image from the camera with a signal from the depth sensor to generate depth data for the image.

[0052] like Figure 1As shown, the system 100 may include or may have access to an image management system 140. The image management system 140 may obtain or otherwise access and / or store image content, video content, algorithms, and / or UI content for provision and rendering on a display screen. The image management system 140 includes a depth conflict resolver 142, a range detector 144, and a UI generator 146. The depth conflict resolver 142 may include any number of algorithms for generating and / or modifying UI elements, such as to resolve depth conflicts for a user viewing the 3D display 110 or content on the 3D display 112. The depth conflict resolver 142 may use the UI generator 146 to generate a UI element that is used to mitigate, resolve, minimize, or otherwise modify perceived depth conflicts.

[0053] For example, range detector 144 may determine the capture volume and comfort range of a user associated with local 3D system 106. Similarly, for example, remote range detector may determine the capture volume and comfort range of a user associated with remote 3D system 108. For example, range detector 144 is configured to determine the line of sight of a user viewing content and determine where particular content may be cropped from the edge of the display. Such determination may be used to determine whether a depth conflict may occur. A remote range detector (not shown) may perform similar functions for remote device access.

[0054] Images captured by the 3D content system 100 may be processed and then displayed as a 3D presentation. Figure 1 As depicted in the example of , 3D image 104' is presented on 3D display 110. As such, user 102 may perceive 3D image 104' as a 3D representation of user 104, who may be located away from user 102. 3D image 102' is presented on 3D display 112. As such, user 104 may perceive 3D image 102' as a 3D representation of user 102.

[0055] 3D content system 100 can allow participants (e.g., users 102, 104) to conduct audio communications with each other and / or with others. In some implementations, 3D system 106 includes a speaker and a microphone (not shown). For example, 3D system 108 can similarly include a speaker and a microphone. In this way, 3D content system 100 can allow users 102 and 104 to conduct a 3D telepresence session with each other and / or with others. In general, the systems and techniques described herein can work with system 100 to generate image content and / or video content for display between users of system 100.

[0056] Figure 22 is a block diagram of an example system for dynamically resolving depth conflicts for stereoscopic displays according to implementations described throughout the present disclosure. System 200 can be used as or included in one or more implementations described herein, and / or can be used to perform one or more examples of compositing, processing, modifying, or presenting image content described herein. The overall system 200 and / or one or more of its individual components can be implemented according to one or more examples described herein.

[0057] System 200 may include one or more 3D systems 202. In the depicted example, 3D systems 202A, 202B, through 202N are shown, where the index N indicates an arbitrary number. 3D system 202 may provide capture of visual information and audio information for 2D or 3D presentations, and forward the 2D or 3D information for processing. Such information may include images of a scene (e.g., images and / or video), depth data about the scene, and audio from the scene. For example, 2D / 3D system 202 may be used as or included in system 106 and 2D / 3D display 110 ( Figure 1 )Inside.

[0058] System 200 may include multiple cameras, as indicated by camera 204. Any type of light sensing technology may be used to capture images, such as the type of image sensors used in common digital cameras. Cameras 204 may be of the same type or of different types. For example, the camera positions may be placed anywhere on a 3D system such as system 106. In some implementations, camera 204 may include multiple stereo cameras.

[0059] System 202A includes a depth sensor 206. In some implementations, depth sensor 206 operates by broadcasting IR signals onto a scene and detecting a response signal. For example, depth sensor 206 can generate and / or detect light beams 128A-B and / or 130A-B. In some implementations, depth sensor 206 is an optional component, for example, in a 2D video conferencing application that does not utilize depth sensing. In some implementations, depth sensor 206 of any of systems 202 can send and receive depth (e.g., depth 232) to server 214 (e.g., executing image management system 140). System 202A also includes at least one microphone 208 and speaker 210. In some implementations, microphone 208 and speaker 210 can be part of system 106.

[0060] The system 202 additionally includes a 3D display 212 that can present 3D images. In some implementations, the 3D display 212 can be a stand-alone display. In some implementations, the 3D display can be a lenticular display. In some implementations, the 3D display 212 operates using a parallax barrier technology. For example, a parallax barrier can include parallel vertical strips of a substantially non-transparent material (e.g., an opaque film) placed between the screen and the viewer. Due to the parallax between the viewer's respective eyes, different portions of the screen (e.g., different pixels) are viewed by the respective left and right eyes. In some implementations, the 3D display 212 operates using lenticular lenses. For example, alternating rows of lenses can be placed in front of the screen that align light from the screen toward the viewer's left and right eyes, respectively.

[0061] System 200 includes a server 214 that can perform specific tasks of data processing, data modeling, data coordination, and / or data transmission. Server 214 and / or its components may include reference Fig.14 In general, server 214 may receive information from tracking module 216, which may include head / eye tracker 218, hand tracker 220, and / or motion detector 222, any of which may be received from any of 2D / 3D systems 202. Server 214 may receive such tracking information in order to correct, eliminate, reduce, or otherwise modify specific detected depth conflicts within image content captured by system 202.

[0062] like Figure 2 As shown, server 214 includes image management system 140. Image management system 140 can generate 2D and / or 3D information in the form of image content, video content, and / or other UI content. This can include receiving such content (e.g., from 3D system 202A), processing the content, and / or forwarding the (processed and depth-corrected) content to another participant (e.g., to another of 3D systems 202). In some implementations, image management system 140 can enable delivery of image and / or video content to a user via a display device of a computing device.

[0063] Image management system 140 includes a depth conflict resolver (e.g., such as depth conflict resolver 142), a UI generator (e.g., such as UI generator 146), and UI element data 226. Depth conflict resolver 142 can analyze the capture volume size using capture volume detector 228. Depth conflict resolver 142 can also analyze the range between UI elements, and / or the range between a user viewing the content and the depicted content, using range detector 144, for example.

[0064] The depth conflict resolver 142 may generate and modify specific image content 234 received from any of the 2D / 3D systems 202 to ensure that the image content 234 is depicted with appropriate depth perception for the system 202. For example, the system 202A may send image content (e.g., a video of the user) during a remote presentation session with a user of the system 202B. The system 202B may evaluate (e.g., track) the position of the head or eyes of the user of the system 202A and / or 202B to generate UI content 236, virtual content 238, and / or visual effects 240. Generating such content 236, 238, and / or 240 may include accessing image content 234 provided by one or more of the systems 202 and modifying such accessed image content 234 with the content 236-240 to ensure a comfortable viewing environment for the user of any of the systems 202. Modification of image content 234 using content 236 - 240 element ranges (generated by UI generator 146 ) may take into account specific UI element ranges 242 , UI element depths 244 , and voxels 246 , as will be described in detail below.

[0065] In some implementations, the depth conflict resolver 142 can analyze image content (e.g., stereoscopic video content) to determine a depth 232 associated with the image content 234. In some implementations, determining such a depth 232 can include using optical flow techniques to estimate the correspondence between the left-eye view and the right-eye view of each image frame of the image content 234. The depth 232 can be determined relative to the pixels (and / or generated voxels 246) associated with a particular image frame. For example, the depth sensor 206 and / or the camera 204 can detect a particular distance between a boundary 230, an object, a portion of an object, a UI element, or other captured content in an image and / or video. The depth 232 can be estimated, calculated, or otherwise determined in real time or near real time.

[0066] In some implementations, the system 200 may define a viewing range within a particular capture volume based on a determined depth associated with the captured image content. The viewing range defines a volume that depicts a stereoscopic effect when viewing the captured image content. In some implementations, the viewing range may refer to a portion of a display screen. In some implementations, the viewing range may refer to the entire display screen if such a screen provides a stereoscopic effect throughout the screen.

[0067] In some implementations, the viewing range and depth can be used to generate voxels that define or model a specific environment associated with the displayed image content. For example, each voxel can represent a cube inside a 3D model that contains a position and a single color value inside a 3D grid. Each point in the environment can be represented as a voxel that includes volume symbol data. Surfaces and boundaries associated with the environment and objects (e.g., UI elements, virtual content, etc.) can be rendered by extracting isosurfaces from the volume symbol data. When image content (e.g., UI elements, objects, and / or surfaces) changes position within the environment, the content can be re-rendered to update the 3D model of the environment. For example, the image management system 140 can iteratively generate a surface network that represents the volume symbol data, and the surface network can be updated when the volume symbol data is updated. When new or updated depth information about the environment becomes available, similar updates can trigger 3D model updates.

[0068] At some point during playback (e.g., streaming, video session, etc.) of the image content, the image management system 140 can detect a depth conflict occurring in the image content. For example, the depth conflict resolver 142 can detect a depth conflict between the image content and a boundary associated with the viewing range. Such detection can include determining that at least a portion of the image content (e.g., a user's hand) extends beyond a boundary associated with the viewing range (e.g., a bottom edge of a display screen). In some implementations, the boundary can refer to an edge of a capture volume.

[0069] In response to determining a depth conflict, the viewing range and at least one user interface element may be used to resolve the depth conflict of at least a portion (e.g., a hand). For example, detecting a depth conflict between at least a portion of the image content (e.g., a hand) and a boundary (the bottom edge of the display screen) may include: using at least some of the determined depths associated with the image content to generate a plurality of three-dimensional voxels representing corresponding positions in the plane of the display depicting at least a portion. The determined depth may be related to the portion of the hand that is cut off when the hand extends beyond the boundary edge of the display. The UI element may be a frame element sized based on the determined distance from the hand to the boundary. Such a distance may be selected to ensure that the hand is hidden, thereby removing the depth conflict from the view of the user viewing the image content. The frame may represent UI content 236, which may be generated and displayed within the depicted image content via software. The UI content 236 (i.e., UI element) may include virtual content 238 or virtual content 238 that is overlaid, placed below, or otherwise merged with other objects or UI content within the image content. In some implementations, UI elements (e.g., UI content 236) may include virtual content, blurred content, inserted content, and / or other user interface effects for mitigating, dissolving, and / or reducing depth conflicts. Visual effects 240 may be applied to further reduce, correct, or eliminate depth conflicts.

[0070] The image management system 140 can generate modified image content with resolved depth conflicts. The modified image content can include a portion of the image content replaced by at least one user interface element. In the above example, the UI element can include a frame. Other examples are certainly possible, some of which are described in detail throughout this disclosure. The modified image content can be rendered and presented to the user as rendered content 250.

[0071] The above exemplary components are described herein as being implemented in a server 214, which may be connected to the server 214 via a network 260 (the network 260 may be connected to the server 214). Figure 1 140) communicates with one or more of the 3D systems 202. In some implementations, the image management system 140 and / or its components may alternatively or additionally be implemented in some or all of the 3D systems 202. For example, the depth conflict correction and / or related processing described above may be performed by the system that created the 3D information before forwarding the 3D information to one or more receiving systems. As another example, the creating system may forward the images, modeling data, pixels, voxels, depth data, and / or corresponding information to one or more receiving systems that may perform the processing described above. Combinations of these approaches may be used.

[0072] System 200 is an example of a system that includes a camera (e.g., camera 204), a depth sensor (e.g., depth sensor 206), and a 3D content generator (e.g., image management system 140) having a processor that executes instructions stored in a memory. Such instructions may cause the processor to use depth data included in the 3D information (e.g., via a depth processing component) to identify image content in an image of a scene included in the 3D information. The processor may generate modified 3D information by detecting depth conflicts and correcting (e.g., resolving) or minimizing such depth conflicts. For example, the modified 3D information may be generated based on UI elements 226 and image content 234 and may be provided to UI generator 146 to appropriately generate rendered content 250. Rendered content 250 may be provided to one or more systems 202.

[0073] The rendered content 250 represents a 3D stereoscopic image (or video portion) of a particular object (e.g., a user image) based at least in part on modifications generated by the image management system 140, the 3D stereoscopic image (or video portion) having appropriate parallax, corrected or eliminated depth conflicts, and a viewing configuration for both eyes associated with a user accessing a display device, as described herein.

[0074] In some implementations, the processor (not shown) of the server 214 and the system 202 may include (or communicate with) a graphics processing unit (GPU). In operation, the processor may include (or may access memory, storage, and other processors (e.g., CPU)). To facilitate graphics generation and image generation, the processor may communicate with the GPU to display images on a display device (e.g., display 212). The CPU and GPU may be connected via a high-speed bus such as PCI, AGP, or PCI-Express. The GPU may be connected to the display via another high-speed interface such as HDMI, DVI, or DisplayPort. Typically, the GPU may render image content in pixel form. The display 212 may receive image content from the GPU and may display the image content on a display screen.

[0075] Figure 3304 is an example display device illustrating a depth conflict for a user according to implementations described throughout the present disclosure. Here, a lenticular display 300 depicts a user 302 during a 3D video conferencing session. Display 300 is associated with a particular capture volume in which a remote user can best view a 3D representation of user 302. In the 3D representation of the video conferencing session, the view of user 302 may extend beyond the borders of the display at edge 304, which may cause a depth conflict for a user viewing the content shown in display 300. For example, a depth conflict may occur at edge 304 because in a 2D representation the user's hand may appear to be cut off at edge 304, but in a 3D representation, the digital voxel representation of the hand may appear to extend beyond edge 304. Because the hand extends beyond the bounds of the lenticular display from the user's current viewpoint, the system described herein may not be able to generate and display the hand. Therefore, the arm of user 302 appears without a hand in 3D, as shown by depth conflicts 306, 308, 310, and 312. Such a view may cause an interruption or conflict in what the viewing user expects to see. Similar depth conflicts may occur at any edge of display 300, for example, if during a session, user 302 moves, stands, etc. Such movement may cause one or more portions of the user to extend beyond the edges of display 300. System 200 may correct, minimize, or remove such depth conflicts.

[0076] Figure 4 4 is a block diagram illustrating an example of a local capture volume and a range of movement within the capture volume according to implementations described throughout the present disclosure. The depicted view shows a representative display screen volume 402 with a particular capture volume 404 in the yz plane. The capture volume 404 represents a local capture volume generated based on a triangulated view of captured data from at least three stereo camera boxes (e.g., cameras 204). The capture volume 404 includes an optimal viewing range 406. Such range 406 may be determined using the cameras 204, the depth sensor 206, and / or the range detector 144.

[0077] The user displayed on the capture volume 402 is represented by a head 408 and a hand 410. In this example, the hand 410 is partially captured within the volume 402. In this example, the capture volume is shown as being a distance 412 away from a remote user (not shown). The capture volume is shown at an altitude 414. Such measurements are configurable depending on the display screen size and the capture volume defined by the camera capturing the image content.

[0078] Figure 5is a block diagram illustrating an example of a remote capture volume 502 relative to a local capture volume according to implementations described throughout the present disclosure. For example, the remote capture volume 502 may be approximated to compute depth conflicts by flipping the capture volume 402 in the z-plane and translating the volume from about 1.25 meters to about 1.35 meters. The remote capture volume 502 includes an optimal viewing range 504. The remote user is approximated by a head 506 and a hand 508. Similar to the hand 410, the remote user's hand 412 is also cropped by the bottom edge of the capture (approximated by the capture volume 502). This cropping may cause depth conflicts to a user viewing the remote user (i.e., represented by the head 506 and the hand 508). In this example, the two represented capture volumes 402 and 502 are placed a distance 412 from each other. The distance 412 may simulate a user standing about four to eight feet away from another user to participate in a conversation as if the user were physically present. The content captured and displayed on the representative display screen volume 402 may be displayed to the remote user to replicate this physical presence. Similarly, content captured and displayed on representative display screen volume 502 may be displayed to a user of screen volume 402 to replicate the same physical presence.

[0079] Figure 6 is a block diagram illustrating an example of a display edge of a cropped capture volume according to implementations described throughout the present disclosure. In this example, a user line of sight 602 is determined (i.e., based on the user head 408). The line of sight 602 indicates that the entire hand 508 of the remote user (when viewed by the user head 408) can be cropped by the display edge of the display represented by the volume 502. In this case, a depth conflict can occur at the bottom edge of the display represented by the volume 502.

[0080] The image management system 140 can generate a solution to resolve or minimize the detected depth conflict. For example, the system 140 can generate a dark window or box to hide the depth conflict. In some implementations, hiding the depth conflict includes resolving the depth conflict by animating at least one user interface element to hide at least a portion of the image content with the modified image content. In some implementations, a floating black bar 604 can be rendered at z=0.4 meters along the bottom edge of the display screen, thereby creating a perception that an object (e.g., bar 604) between a subject (e.g., user head 408) and the viewer is clipping the view, but because the bar 604 is dark and insets the view, the user does not perceive the depth conflict.

[0081] Figures 7A to 7C is a block diagram illustrating an example of visually perceived tilt of a display device according to implementations described throughout this disclosure. Fig. 7A, another perspective view of bar 604 is shown by bar 702. The top edge of black bar 702 may be perceived as landing at z=0.4 meters because of how the cropping changes as head 408 moves. The bottom edge of volume 502 may be ambiguous, and because there is no interior detail of the parallax cue within black bar 702, black bar 702 may be perceived at different depths. If display frame bar 702 is black, the bottom edge of bar 702 may be perceived by the user as landing at z=0 meters, and in this case, bar 702 may be perceived as tilted.

[0082] refer to Figure 7B , a frame strip solution for alleviating depth conflict is shown. The solution includes: using a frame 704 overlay as a UI element around the image content. In this example, a portion 706 of the window frame 704 is placed in front of the image content 708, while the remaining portion of the frame 704 is located behind the image content 708.

[0083] Although frame 704 and portion 706 appear in different planes, Figure 7C User 710, shown, when viewing image content on a display device, for example, will perceive the frame and image content to be at an angle (e.g., tilted), as shown in content 712. This illusion may correct for previously perceived depth conflicts.

[0084] Figures 8A to 8B is a block diagram illustrating an example of resolving depth conflicts using synthetic image content according to implementations described throughout this disclosure. Fig. 8A As shown, user 802 is depicted as image content in a 3D video conferencing session on a remote presentation device 804 housing a display screen 806, as shown for Figure 1 To correct for depth conflicts that may occur at the bottom edge of the display screen 806, the portion 808 of the table 810 may be physically painted to appear as if the table 810 has a dark stripe.

[0085] like Figure 8B As shown in FIG, the dark strip 808 may appear tilted and floating to a user viewing the content on the display 806 at a right angle (or nearly right angle) to the display screen 806. Such a configuration can correct or eliminate depth conflicts that occur due to cutting off portions of the displayed user 802.

[0086] Figures 9A to 9B is a block diagram showing an example of resolving depth conflicts by dynamically adjusting a display window of a display device according to implementations described throughout this disclosure. Fig.9AAs shown, capture volume 404 includes an optimal viewing range 406. Similarly, a user displayed on display screen volume 402 is represented by head 902. In this example, a wrist or upper arm is partially captured within volume 402, which causes a portion of the captured content to be clipped by the display edge. In this example, capture volume 402 is shown as a distance 412 away from remote capture volume 502, which includes the user represented by head 506 and wrist 508. In this example, the user (i.e., head 902) may be standing and therefore at the top portion of optimal viewing range 406. Because head 902 is higher in capture volume 406 than when the user is seated (or at a lower height), when viewing the remote content or portion of the remote user shown in capture volume 502, depth conflicts may be perceived, as shown by viewing angle 904. For example, such viewing angle 904 may view depth conflicts associated with wrist 508 because the wrist may be perceived as missing when head 902 views the remote content in capture volume 502. For example, a depth conflict may be determined to have occurred based on the tracked head position (of head 902) of a local user viewing remote content (in capture volume 502). For example, a depth conflict may be determined by depth sensor 206 of system 202A viewing content received from system 202B. Both systems 202A and 202B may include a lenticular display device having four or more edge borders, wherein if, for example, user movement occurs including a change in head position, a depth conflict may be determined to have occurred. In the depicted example, frame 908a may include surrounding, enclosing, and / or overlapping Fig.9A 508a, which ensures that the line of sight 904 stays above content that may cause a depth conflict (e.g., wrist 508), as shown by the distance 910 from the bottom edge of the capture volume 502 to the inner edge of the box 908a.

[0087] To correct the depth conflict detected above, the depth conflict resolver 142 may work with the UI generator 146 to provide a visual effect 240, such as Fig. 9B For example, the depth conflict resolver 142 may determine and trigger a visual effect that raises a frame portion extending from the bottom edge of the capture volume 502 to a higher fixed position to eliminate depth conflicts due to the viewing wrist 508. The frame portion may include a frosted area or volume, or a blurred area or volume to eliminate depth conflicts.

[0088] In some implementations, the depth conflict resolver 142 can trigger a resizing of a particular user interface element. For example, the UI element data 226 can include information about the range 242, depth 244, and / or voxels 246 of the frame UI element. Such information can be used to resize the frame. Therefore, the depth conflict resolver 142 can trigger a resizing of a frame (e.g., a window surrounding, enclosing, and / or overlapping content) associated with the content depicted in the capture volume 502 upon determining that the tracked head position of the head 902 has moved to a position that can generate one or more depth conflicts. The resizing of the frames 908a to 908b is indicated by the line of sight 912 (modified from the line of sight 910) and the distance 914 from the bottom edge of the capture volume 502 to the inner edge of the frame 908b. In this example, the distance 908b is increased relative to the distance 908a to avoid the depth conflict.

[0089] In some implementations, the depth conflict resolver 142 can work with the range detector 144 to determine a particular boundary 230 and / or a depth 232 associated with an object near or at the boundary 230. For example, the depth conflict resolver 142 can detect a depth conflict between a portion of the image content and a boundary 230 associated with the viewing range (e.g., the wrist 508 at the boundary 906) by using at least some of the determined depths of the wrist 508 associated with the image content in the volume 502. For example, the depth conflict resolver 142 can use the depth of the wrist 508 to generate a plurality of three-dimensional voxels (as described above) that represent the position of the wrist 508 in the plane of the display depicting the wrist. In some implementations, the user interface element can be a box, a blurred wall, or other UI element selected based on a distance (e.g., the distance between the wrist outside the boundary and the boundary).

[0090] In some implementations, visual effects 240 may determine the speed at which the frame resizing (e.g., from 908a to 908b) is performed. For example, depth conflict resolver 142 may determine whether the user is seated, standing, actively moving, etc., and may select the frame resizing speed accordingly. Similarly, range detector 144 and UI element data 226 may be used to select the actual amount (smaller or larger) by which the frame is resized.

[0091] Figures 10A to 10B is a block diagram illustrating an example of resolving depth conflicts by adjusting boundaries and / or edges of a capture volume according to implementations described throughout this disclosure. Fig. 10A A portion of a room 1000 housing a telepresence display device in which a user 1002 is within a display screen 1004 is depicted.

[0092] If image management system 140 does not provide depth conflict management, then border edges 1006, 1008, 1010, and 1012 may cause depth conflicts to a user viewing user 1002 (or viewing content within screen 1004) if portions of user 1002 are clipped by the edges. Similarly, a user viewing user 1002 (or viewing content within screen 1004) may perceive depth conflicts at portions of image content that appear to float, such as border 1014.

[0093] In some implementations, the image management system 140 can trigger the visual effects 240 and / or the virtual content 238 and / or the UI content 236 to mitigate or correct depth conflicts that occur at such borders and boundary edges. For example, the depth conflict resolver 142 can generate a mesh-textured blur wall 1016 for a particular boundary of the screen 1004 to blur the boundary edge to ensure that a user viewing content in the screen 1004 cannot view around or beyond the boundary edge of the screen 1004. Although the wall 1016 is shown on a single boundary of the screen 1004, any or all boundaries in the screen 1004 may include a blur wall, such as the wall 1016. In some implementations, the depth conflict resolver 142 can generate additional pixels (e.g., regions and / or volumes) to be blurred throughout the screen 1004 to prevent depth conflicts.

[0094] refer to Fig. 10B , user 1002 is shown with screen 1004 and box 1006 to define capture volume 1008. In this example, depth conflict resolver 142 can generate a box surrounding any portion of capture volume 1008 to ensure that depth conflicts are minimized or eliminated. In this example, box 1006 can be generated by image management system 140 via UI generator 146, for example.

[0095] The depth conflict resolver 142 also generates a blur shape 1010. The blur shape 1010 can be blurred in the following manner: a partial transparency of the content is blurred to make the remote view of the user 1002 appear to be extended, thereby increasing the presence of the remote user viewing the user 1002. In this example, the radius 1013 of the blur shape 1010 is selected to be a less realistic place where the content of the bounding volume 1008 is not depicted. Although the blur shape 1010 is a partial ellipse, other shapes are certainly possible. In some implementations, the surface of the blur shape described herein is angled so that z=0 at the bottom edge of the blur shape. For example, the blur shape 1010 is at Fig. 10C are shown at angles having the same radius 1013.

[0096] refer to Fig. 10D, another example UI element / shape for mitigating depth conflicts when viewing content within the volume 1008 includes a blur shape 1014. Blur shape 1014 is a frosted, partially transparent, half-angled trapezoid. In particular, the surface of shape 1014 can be angled and rounded 1016 to provide lateral translation to prevent clipping by the left display border 1006 and the right display border 1010.

[0097] refer to Fig.10E , another example UI element / shape for mitigating depth conflicts when viewing content within volume 1008 includes blur shape 1018. Blur shape 1018 is a frosted, partially transparent shell that provides for additional content to be displayed near shape 1018 in the front center portion of volume 1008 while limiting the bottom portion to the side boundaries of volume 1008.

[0098] refer to Fig.10F , another example UI element for mitigating (e.g., hiding, removing, correcting) depth conflicts when viewing content within volume 1020 includes shape 1022. Shape 1022 and the size of shape 1022 can be determined by depth sensor 206, which can detect possible depth conflicts when image and / or video content is being depicted in volume 1020. In this example, sensor 206 can provide depth 232 to depth conflict resolver 142. Resolver 142 can determine specific areas in which depth conflicts may occur for a particular user viewing content being depicted in volume 1020. For example, resolver 142 can work with a depth sensor and camera on another system 202 to determine head pose data associated with particular captured data. For example, such data can be used to dynamically calculate depth conflicts in order to determine whether a depth conflict will be visible on a display device depicting volume 1020.

[0099] refer to Figure 10G , the depth conflict resolver 142 determines that a portion of the user's 1024 arm may cause a particular depth conflict. In response, the depth conflict may be mitigated as needed. For example, an animation via the visual effect 240 may be faded in or out based on the particular detected depth conflict. In some implementations, a selected shape such as shape 1026 may have a gradient blur. In this example, the image management system 140 using the UI generator 146 may, for example, use a distortion map to generate a blur radius that increases toward the bottom boundary of the capture volume 1020, as shown by the shape 1026.

[0100] The shape 1026 may be generated when it is detected that the user 1024 is moving forward. For example, when the user 1024 moves forward, the image management system 140 (e.g., using a shader) detects a depth conflict on the lower border of the volume 1020 and triggers the UI generator 146 to generate a visual effect 240 to fade the blur and opacity of the frosted shape 1026. In some implementations, similar transitions may be used for side walls in the boundary system for limbs or objects outside the capture volume 1020.

[0101] Blur shape 1022 is a frosted, partially transparent shell that provides for additional content to be displayed adjacent shape 1018 in the front and center portion of volume 1008 , while confining the bottom portion to the side boundaries of volume 1008 .

[0102] Fig.11 1 is a block diagram illustrating an example of using segmented virtual content to resolve depth conflicts according to implementations described throughout this disclosure. Segmented front depth conflict mitigation is shown here. Display 1004 is shown together with capture volume 1008. If the large frosted wall that appears in front is too visually distracting when the hand briefly enters the boundary zone, the image management system 140 can trigger the depth conflict resolver 142 via the UI generator 146 to generate a segmented grid element 1100 and fade the partitions in and out, which is comfortable for the user. Such a segmented grid element 1100 may include blurred portions 1102 and 1104 and a semi-opaque portion 1106. Such a UI element can also provide an overlay for permanent objects depicted in volume 1008, such as a laptop on a table.

[0103] In some implementations, the grid element 1100 can be a blurred overlay with a gradient blur that fades from a left center portion 1108 of the overlay (e.g., the grid element 1100) to a left edge 1110 of the image content, and from a right center portion 1112 of the overlay to a right edge 1114 of the image content.

[0104] Fig.12 1008 is a block diagram illustrating example application content placed on virtual content according to implementations described throughout the present disclosure. In this example, a shaped translucent UI element 1202 is depicted to mitigate a specific detected depth conflict. For example, UI element 1202 may be blurred, partially blurred, or translucent. Such an element may be shaped based on display volume 1008, covered content, or a specific detected depth conflict. For example, although UI element 1202 is depicted as a rectangle, other shapes are possible.

[0105] UI element 1202 may be used as a location to depict additional content to a user viewing volume 1008. For example, if user 1008 and user 1020 are accessing systems 202A and 202B, respectively, the two users may wish to share application data, screen data, and the like. Image management system 140 may trigger shapes generated to mitigate depth conflicts to begin depicting application content, such as content 1204 and 1206. Although two application windows are depicted, any number of windows, content, applications, shortcuts, icons, and the like may be depicted. In some implementations, applications and / or content depicted within a UI element such as element 1202 may include additional UI elements that are determined to be open during a session viewing image content. For example, UI element 1202 may depict a user interface layer with thumbnail images (e.g., content 1204 and content 1206) of additional software programs that are being executed in memory by at least one processing device while accessing image content. Such content 1204 and content 1206 may represent software applications that are open in memory and executed while the user is operating a remote presentation session with user 1002 .

[0106] In some implementations, to prevent the mitigation described herein from itself generating depth conflicts where the mitigation intersects the display edge, there are minimum parallax cues that can be used to place the mitigation at a specific z-height. For example, applied translucent surfaces are perceived as overlaying a rendered person, but are perceptually flexible as to the exact height at which they are perceived. In particular, vertical matte designs and curved matte designs end up looking similar. The perceptual pinning of the vertical matte design to the display edge can make a flat vertical surface appear to be curved.

[0107] In some implementations, the gradient blur variants described herein provide the advantage of avoiding sharp upper edges. The blur can provide an improved amount of depth conflict reduction. In some implementations, the gradient blur can be applied at specific areas where depth conflicts are detected, but not outside of these areas. That is, instead of the entire bottom edge of the gradient blur, the gradient blur can be placed at the portion where the content at the edge of the display (e.g., user portion, object, etc.) creates a depth conflict. For example, the system described herein can use a detected head pose in conjunction with rendered content to detect a depth conflict. Then, a gradient blur effect can be added at the depth conflict portion. The type of blur can vary based on the detected level of depth conflict. The gradient can be adjustable to gradually decrease away from the edge of the display.

[0108] Fig.131 is a flow chart illustrating an example of a process 1300 for resolving depth conflicts in a 3D content system according to implementations described throughout the present disclosure. In some implementations, the process 1300 may utilize an image processing system having at least one processing device and a memory storing instructions that, when executed, cause the processing device to perform a plurality of operations and computer-implemented steps described in the claims. In general, the systems 100, 200, and / or 1400 may be used for the description and execution of the process 1400. In some implementations, each of the systems 100, 200, and / or 1400 may represent a single system. In some implementations, the remote presentation system described in the system 202 may perform the operations described in the claims. In some implementations, the server 214 accessed by the system 202 may alternatively perform the operations described in the claims.

[0109] In general, process 1300 utilizes the systems and algorithms described herein to detect and correct depth conflicts for a 3D display. In some implementations, depth conflict detection is based on head and / or eye tracking and captured depth image pixels. In some implementations, depth conflict detection is based on other user movements (e.g., hand movement or placement within a capture volume, and / or body movement or placement within a capture volume). In some implementations, UI elements are generated to mitigate detected depth conflicts. In general, the described process 1300 can be performed in image content, video content, virtual content, UI elements, application content, or other camera captured content.

[0110] At block 1302, process 1300 includes utilizing at least one processing device to perform operations including determining a capture volume associated with captured image content. For example, capture volume detector 228 may determine capture volume 502 ( Figure 6 ) size. The capture volume may be used to determine whether image content resides within or exceeds certain boundaries defined by the capture volume. Such image content that extends beyond the boundaries may cause depth conflicts to a user viewing the image content within the capture volume 502. Additionally, UI element data 226 as well as range determinations from range detector 144 may be considered when evaluating depth conflicts.

[0111] At block 1304, process 1300 includes determining a depth associated with the captured image content. For example, range detector 144 may calculate depth 232 based on images captured by camera 204 and / or data captured by depth sensor 206. Depth may relate to an object, a user, a portion of a user, a UI element, or other content captured within system 202.

[0112] At block 1306, process 1300 includes defining a viewing range within the capture volume based on the depth within which a stereoscopic effect is depicted when viewing the captured image content. For example, range detector 144 may utilize depth 232 and capture volume 502 size to determine a viewing range, which may be a viewing range (e.g., size, window, volume) for 3D stereoscopic effects and 3D content within a 3D display (such as a lenticular display) for viewing rendered image content. Determining such a viewing range may enable system 200 to appropriately account for where particular depth conflicts may occur.

[0113] At block 1308, process 1300 includes determining a depth conflict between the captured image content and a boundary associated with the viewing range. For example, depth conflict resolver 142 may detect that hand 508 ( Figure 6 ) at the bottom edge of the capture volume 502 (i.e., the bottom edge of the display device, such as Fig. 10A Here, in some implementations, the depth conflict resolver 142 may use the capture volume detector 228 to detect at least a portion of the captured image content (i.e., Figure 6 The portion of hand 508 in FIG. 504 extends beyond boundary 1012 associated with a viewing range defined by capture volume 502 and / or volume 504.

[0114] In some implementations, detecting a depth conflict between at least a portion of the image content (e.g., a portion of the hand 508) and a boundary associated with the viewing range (e.g., boundary 1012) can include: using at least some of the determined depths 232 associated with the image content (e.g., the head 506 and the hand 508) to generate a 3D voxel that represents a location in the plane of the display that renders the captured image content (e.g., the z-plane of the display). In this example, the depth used to detect and / or correct the depth conflict can include the depth of the hand 508. The distance can include the distance from the boundary 1012 to the portion of the hand outside the boundary 1012. A UI element for correcting depth perception can be selected based on the distance. In some implementations, the boundary 1012 is associated with at least one edge of the lenticular display device.

[0115] In some implementations, voxels (e.g., voxels 246) can be generated using UI element data 226. Voxels 246 can be derived from a point cloud defined in 3D space. Voxels 246 can include a pixel grid that defines a plurality of cells having fixed sizes and discrete coordinates. Voxels 246 can be used to determine which portions of particular image content may cause depth conflicts, and which portions of these portions should be corrected, resolved, blurred, or otherwise modified to avoid depth conflicts.

[0116] In some implementations, a depth conflict can be determined based on a tracked head position of a user viewing image content at a remote lenticular display device. Figure 6 The depth conflict associated with the hand 508 may be determined based on the viewing angle (e.g., line of sight 602) of the head 408 in the image. For example, the tracked head position may be determined or provided via the tracking module 216 using the head / eye tracker 218, the hand tracker 220, and / or the motion detector 222.

[0117] At block 1310, process 1300 includes, in response to determining a depth conflict, resolving at least a portion of the depth conflict using a viewing range and at least one UI element. For example, a depth conflict resolver may use the determined viewing range relative to hand 508 to select a particular UI element to generate and / or modify. Such a UI element may be generated by UI generator 146 and provided for rendering with the image content. In some implementations, resolving the depth conflict includes generating a UI element that represents a box around the image content within volume 502. The box (e.g., Fig. 10B The frame 1006 in the image content may be adapted to accommodate the movement depicted in the captured image content. For example, the size, shape, or other factors of the frame 1006 may be modified to resolve (e.g., cover) depth conflicts. In some implementations, the size and / or shape of other generated UI elements may be adjusted to resolve depth conflicts. In some implementations, adjusting the size of a particular UI element is based on the tracked head position of the user viewing the particular image content.

[0118] In some implementations, one side of a UI element, such as the frame 1006 corresponding to at least a portion of the frame (the hand 508) that extends beyond the boundaries of the capture volume 502, can be placed in a different plane that is parallel to and in front of the rest of the frame to generate a visually perceived tilt of the frame from vertical to a non-zero angle relative to vertical, such as Figures 7A to 7B shown.

[0119] At box 1312, processing 1300 includes generating modified image content for rendering using the resolved depth conflicts. The modified image content may include a portion of the image content replaced by at least one UI element. For example, the UI element may include a blurred overlay. The blurred overlay may be generated by UI generator 146. The blurred overlay may be 2D or 3D. For example, the blurred overlay may start at the boundary of the capture volume 502 and may end at a predefined position associated with the size of the display device depicting the image content. For example, the size of the display may include a predefined minimum and maximum size of a frame, overlay, or UI element.

[0120] In some implementations, UI elements such as blur overlays can be defined by depth conflict resolver 142 and UI generator 146, where a blur radius associated with the UI element can increase at a threshold distance from a border. For example, the blur radius can be animated based on the movement of the image content. In such examples, animating the blur radius (or other UI elements) can resolve and / or hide depth conflicts.

[0121] In some implementations, the blurred overlay can be shaped according to the determined depth conflict. In some implementations, the blurred overlay can be shaped according to the size or shape of the depicted image content. Example shapes can include, but are not limited to, square, rectangle, ellipse, semicircle, semi-ellipse, trapezoid, etc.

[0122] In some implementations, the blurred overlays described herein may include additional UI elements that are determined to be opened during a session of viewing image content. For example, the additional UI elements may include software programs that are accessed (i.e., executed in memory) by at least one processing device when accessing image content on a display device. The software program / application may be displayed as a selectable UI element superimposed on the blurred overlay. The user may select a specific application to depict the application in a larger form and begin using the application within, around, or near the depicted image and / or video content.

[0123] In some implementations, such as Fig.11 As shown, the blurred overlay is a gradient blur that fades from the left center portion of the overlay to the left edge of the image content and from the right center portion of the overlay to the right edge of the video content. Other variations of the gradient blur are possible, and Fig.11 Only one example of gradient blur is shown.

[0124] In some implementations, the lower boundary of the capture volume can be used as an interactive zone where gestures and the like often occur. For example, image management system 140 can distinguish between the lower portion of the display screen, where depth conflicts are more problematic, and the upper-middle portion of the display screen, where interactive elements may reside.

[0125] Fig.14An example of a computer device 1400 and a mobile computer device 1450 that can be used with the described technology is shown. The computing device 1400 may include a processor 1402, a memory 1404, a storage device 1406, a high-speed interface 1408 connected to the memory 1404 and a high-speed expansion port 1410, and a low-speed interface 1412 connected to a low-speed bus 1414 and the storage device 1404. The components 1402, 1404, 1406, 1408, 1410, and 1412 are interconnected using various buses and can be mounted on a common motherboard or in other appropriate ways. The processor 1402 can process instructions for execution within the computing device 1400, including instructions stored in the memory 1404 or on the storage device 1406 to display graphical information of a GUI on an external input / output device, such as a display 1416 coupled to the high-speed interface 1408. In some embodiments, multiple processors and / or multiple buses and multiple memories and multiple types of memories can be used as appropriate. Furthermore, multiple computing devices 1400 may be connected, with each device providing portions of the necessary operations (eg, as a server bank, a blade server bank, or a multi-processor system).

[0126] Memory 1404 stores information within computing device 1400. In one embodiment, memory 1404 is one or more volatile memory units. In another embodiment, memory 1404 is one or more non-volatile memory units. Memory 1404 may also be another form of computer-readable medium, such as a magnetic disk or optical disk.

[0127] Storage device 1406 can provide mass storage for computing device 1400. In one embodiment, storage device 1406 can be or include a computer readable medium, such as a floppy disk device, a hard disk device, an optical disk device or a tape device, a flash memory or other similar solid-state memory device, or a device array, including a storage area network or other configured device. A computer program product can be tangibly embodied in an information carrier. A computer program product can also include instructions for performing one or more methods when executed, such as the methods described herein. The information carrier is a computer readable medium or a machine readable medium, such as memory 1404, storage device 1406, or a memory on processor 1402.

[0128] High-speed controller 1408 manages bandwidth-intensive operations of computing device 1400, while low-speed controller 1412 manages less bandwidth-intensive operations. This allocation of functions is exemplary only. In one embodiment, high-speed controller 1408 is coupled to memory 1404, display 1416 (e.g., through a graphics processor or accelerator), and to high-speed expansion port 1410, which can accept various expansion cards (not shown). Low-speed controller 1412 can be coupled to storage device 1406 and low-speed expansion port 1414. The low-speed expansion port, which can include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), can be coupled to one or more input / output devices, such as a keyboard, a pointing device, a scanner, or a network device, such as a switch or a router, for example, through a network adapter.

[0129] Computing device 1400 can be implemented in many different forms, as shown. For example, it can be implemented as a standard server 1420, or multiple times in a group of such servers. It can also be implemented as part of a rack server system 1424. In addition, it can be implemented in a personal computer such as a laptop computer 1422. Alternatively, components from computing device 1400 can be combined with other components in a mobile device (not shown) such as device 1450. Each of these devices can contain one or more of computing devices 1400, 1450, and the entire system can be composed of multiple computing devices 1400 and 1450 that communicate with each other.

[0130] The computing device 1450 includes a processor 1452, a memory 1464, an input / output device such as a display 1454, a communication interface 1466, and a transceiver 1468, among other components. The device 1450 may also be provided with a storage device, such as a microdrive or other device, to provide additional storage. Each of the components 1450, 1452, 1464, 1454, 1466, and 1468 are interconnected using various buses, and several of these components may be mounted on a common motherboard or in other suitable ways.

[0131] Processor 1452 is capable of executing instructions within computing device 1450, including instructions stored in memory 1464. The processor may be implemented as a chipset that includes separate multiple analog processors and digital processors. For example, the processor may provide coordination of other components of device 1450, such as control of a user interface, applications running on device 1450, and wireless communications of device 1450.

[0132] The processor 1452 can communicate with the user through the control interface 1458 and the display interface 1456 coupled to the display 1454. For example, the display 1454 can be a TFT LCD (thin film transistor liquid crystal display) or an OLED (organic light emitting diode) display or other appropriate display technology. The display interface 1456 may include appropriate circuits for driving the display 1454 to present graphics and other information to the user. The control interface 1458 can receive commands from the user and convert the commands for submission to the processor 1452. In addition, the external interface 1462 can communicate with the processor 1452 to enable near-field communication of the device 1450 with other devices. In some embodiments, the external interface 1462 may provide, for example, multiple interfaces for wired or wireless communication.

[0133] The memory 1464 stores information within the computing device 1450. The memory 1464 may be implemented as one or more of one or more computer-readable media, one or more volatile memory units, or one or more non-volatile memory units. An expansion memory 1484 may also be provided and connected to the device 1450 via an expansion interface 1482, which may include, for example, a SIMM (single-line memory module) card interface. Such an expansion memory 1484 may provide additional storage space for the device 1450, or may also store applications or other information for the device 1450. Specifically, the expansion memory 1484 may include instructions for executing or supplementing the above-mentioned processing, and may also include security information. Therefore, for example, the expansion memory 1484 may be a security module of the device 1450, and may be programmed with instructions that allow the secure use of the device 1450. In addition, security applications and additional information may be provided via a SIMM card, such as placing identification information on a SIMM card in an unbreakable manner.

[0134] The memory may include, for example, flash memory and / or NVRAM memory, as described below. In one embodiment, a computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods, such as the methods described above. The information carrier is a computer readable medium or a machine readable medium, such as memory 1464, expansion memory 1484, or memory on processor 1452, which may be received, for example, via transceiver 1468 or external interface 1462.

[0135] Device 1450 may communicate wirelessly via communication interface 1466, which may include digital signal processing circuitry where necessary. Communication interface 1466 may provide communication in various modes or protocols, such as GSM voice calls, SMS, EMS or MMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000, or GPRS, etc. Such communication may occur, for example, via radio frequency transceiver 1468. In addition, short-range communication may occur, such as using Bluetooth, Wi-Fi, or other such transceivers (not shown). In addition, GPS (Global Positioning System) receiver module 1480 may provide additional navigation and location-related wireless data to device 1450, which may be used appropriately by applications running on device 1450.

[0136] Device 1450 may also communicate audibly using audio codec 1460, which may receive voice information from a user and convert the voice information into usable digital information. Audio codec 1460 may similarly generate audible sounds for the user, such as through a speaker, for example, in an earpiece of device 1450. Such sounds may include sounds from voice phone calls, may include recorded sounds (e.g., voice messages, music files, etc.), and may also include sounds generated by applications operating on device 1450.

[0137] The computing device 1450 can be implemented in a variety of different forms, as shown. For example, the computing device 1450 can be implemented as a cellular phone 1480. The computing device 1450 can also be implemented as part of a smart phone 1482, a personal digital assistant, or other similar mobile device.

[0138] Various implementations of the systems and techniques described herein can be realized in digital electronic circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementations in one or more computer programs executable and / or interpretable on a programmable system, the programmable system comprising at least one programmable processor, which can be special purpose or general purpose, coupled to receive data and instructions from and send data and instructions to a storage system, at least one input device, and at least one output device.

[0139] These computer programs (also referred to as programs, software, software applications or code) include machine instructions for a programmable processor and can be implemented in high-level procedural and / or object-oriented programming languages ​​and / or in assembly / machine language. As used herein, the terms "machine-readable medium", "computer-readable medium" refer to any computer program product, apparatus and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0140] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user, and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, voice, or tactile input.

[0141] The systems and techniques described herein may be implemented in a computing system that includes a back-end component (e.g., as a data server) or includes a middleware component (e.g., an application server) or includes a front-end component (e.g., a client computer with a graphical user interface or a web browser through which a user can interact with embodiments of the systems and techniques described herein), or any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), and the Internet.

[0142] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises from computer programs running on the respective computers and having a client-server relationship to each other.

[0143] In some embodiments, Fig.14 The computing device depicted in FIG. 144 may include sensors that interface with a virtual reality or AR headset (VR headset / AR headset / HMD device 1490). For example, included in computing device 1450 or Fig.14One or more sensors on the other computing devices depicted may provide input to the VR headset 1490, or generally to the VR space. Sensors may include, but are not limited to, touch screens, accelerometers, gyroscopes, pressure sensors, biometric sensors, temperature sensors, humidity sensors, and ambient light sensors. Computing device 1450 may use sensors to determine an absolute position and / or detected rotation of the computing device in the VR space, which may then be used as input to the VR space. For example, computing device 1450 may be incorporated into the VR space as a virtual object, such as a controller, laser pointer, keyboard, weapon, etc. When incorporated into the VR space, positioning of the computing device / virtual object by the user may allow the user to position the computing device to view the virtual object in a particular manner in the VR space.

[0144] In some embodiments, one or more input devices included on or connected to computing device 1450 may be used as input to the VR space. Input devices may include, but are not limited to, a touch screen, a keyboard, one or more buttons, a trackpad, a touchpad, a pointing device, a mouse, a trackball, a joystick, a camera, a microphone, headphones or earphones with input capabilities, a game controller, or other connectable input devices. When the computing device is incorporated into the VR space, a user interacting with the input devices included on computing device 1450 may cause specific actions to occur in the VR space.

[0145] In some embodiments, one or more output devices included on the computing device 1450 may provide output and / or feedback to a user of the VR headset 1490 in the VR space. The output and feedback may be visual, auditory, or audio. The output and / or feedback may include, but are not limited to, rendering the VR space or virtual environment, vibrating, turning on and off or flashing and / or shining one or more lights or flashes, sounding an alarm, playing a chime, playing a song, and playing an audio file. The output devices may include, but are not limited to, vibration motors, vibration coils, piezoelectric devices, electrostatic devices, light emitting diodes (LEDs), flashes, and speakers.

[0146] In some embodiments, computing device 1450 may be placed within a VR headset 1490 to create a VR system. VR headset 1490 may include one or more positioning elements that allow a computing device 1450, such as a smartphone 1482, to be placed in an appropriate position within VR headset 1490. In such embodiments, the display of smartphone 1482 may be capable of rendering a stereoscopic image representing a VR space or virtual environment.

[0147] In some embodiments, computing device 1450 can appear as other objects in a computer-generated 3D environment. User interactions with computing device 1450 (e.g., rotating, shaking, touching a touch screen, swiping a finger on a touch screen) can be interpreted as interactions with objects in the VR space. As just one example, the computing device can be a laser pointer. In such an example, computing device 1450 appears as a virtual laser pointer in a computer-generated 3D environment. When the user manipulates computing device 1450, the user in the VR space sees the movement of the laser pointer. The user receives feedback from the interaction with computing device 1450 in the VR environment on computing device 1450 or VR headset 1490.

[0148] In some embodiments, computing device 1450 may include a touch screen. For example, a user may be able to interact with the touch screen in a particular manner that may simulate what is happening on the touch screen using what is happening in the VR space. For example, a user may use a pinching motion to zoom the content displayed on the touch screen. This pinching motion on the touch screen may cause the information provided in the VR space to be zoomed. In another example, the computing device may be rendered as a virtual book in a computer-generated 3D environment. In the VR space, the pages of the book may be displayed in the VR space, and the swiping of the user's finger across the touch screen may be interpreted as turning / flipping the pages of the virtual book. When each page is turned / flipped, in addition to seeing the page content change, audio feedback may also be provided to the user, such as the sound of the pages turning in the book.

[0149] In some embodiments, one or more input devices (e.g., a mouse, a keyboard) other than a computing device may be rendered in a computer-generated 3D environment. The rendered input devices (e.g., a rendered mouse, a rendered keyboard) may be used to render in the VR space to control objects in the VR space.

[0150] Computing device 1400 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. Computing device 1450 is intended to represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, and other similar computing devices. The components shown here, their connections and relationships, and their functions are exemplary only and are not meant to limit the disclosed embodiments.

[0151] In addition, the logic flows depicted in the accompanying drawings do not require the particular order or sequence shown to achieve the desired results. In addition, other steps can be provided from the described flows, or steps can be eliminated, and other components can be added to or removed from the described systems. Therefore, other embodiments are within the scope of the appended claims.

Claims

1. A computer-implemented method utilizing at least one processing device to perform operations, the operations comprising: determining a capture volume in the form of a physical volume of space bounded by one or more boundaries, the capture volume associated with image content captured by at least one camera and bounded within the one or more boundaries; determining a depth associated with the image content; Within the capture volume, defining a viewing range based on the depth, in which a stereoscopic effect occurs; Determining a depth violation between the image content and a boundary associated with the viewing range includes: detecting that at least a portion of the image content extends beyond the boundary associated with the viewing range to determine which portions of particular image content cause a depth violation; In response to determining the depth conflict, resolving the depth conflict for the at least one portion using the viewing range and at least one user interface element, wherein resolving the depth conflict comprises selecting at least one user interface element based on a distance of the at least one portion from the boundary; and Modified image content is generated for rendering using the resolved depth conflicts, the modified image content comprising a portion of the image content replaced by the at least one user interface element.

2. The method according to claim 1, wherein: Determining depth conflicts between the image content and a boundary associated with the viewing range further includes generating a plurality of three-dimensional voxels using at least some of the determined depths associated with the image content, the plurality of three-dimensional voxels representing locations in a plane of a display rendering the image content.

3. The method according to claim 1, wherein: The boundary is associated with at least one edge of the lenticular display device; The depth conflict is determined based on a tracked head position of a user viewing the image content at a remote lenticular display device; as well as Resolving the depth conflict includes adjusting a size of the user interface element based on the tracked head position of the user.

4. The method according to claim 1, wherein: Resolving the depth conflict includes generating the at least one user interface element as a box overlaying at least some of the image content, the at least one box adapted to accommodate movement depicted in the image content such that the at least one box is modified in size and / or shape.

5. The method according to claim 4, wherein: A side of the frame corresponding to the at least a portion extending beyond the border is placed in a different plane parallel to and in front of the remainder of the frame to generate a visually perceived tilt of the frame from vertical to a non-zero angle from vertical.

6. The method according to claim 1, wherein: The at least one user interface element depicts a user interface layer having a thumbnail image of an additional software program that is executed in memory by the at least one processing device when the image content is accessed.

7. The method according to any one of claims 1 to 6, wherein: The user interface element includes a blur overlay that begins at the border and ends at a predefined position associated with a size of a display device depicting the image content, wherein a blur radius associated with the blur overlay increases at a threshold distance from the border.

8. The method according to claim 7, wherein: The blurred overlay includes a user interface layer having a thumbnail image of an additional software program that is executed in memory by the at least one processing device when the image content is accessed.

9. The method according to claim 7, wherein: The fuzzy overlay is oval in shape.

10. The method according to claim 7, wherein: The blurred overlay is a gradient blur that gradually changes from a left central portion of the overlay to a left edge of the image content and from a right central portion of the overlay to a right edge of the image content.

11. The method according to claim 1, wherein: Resolving the depth conflict includes animating the at least one user interface element to hide the at least a portion of the image content with the modified image content.

12. An image processing system, comprising: at least one processing device; Multiple stereo cameras; as well as a memory storing instructions that, when executed, cause the system to perform operations comprising: determining a capture volume in the form of a physical volume of space bounded by one or more boundaries, the capture volume associated with image content captured by the plurality of stereo cameras and bounded within the one or more boundaries; determining a depth associated with the image content; Within the volume, defining a viewing range within which a stereoscopic effect occurs based on the depth; Determining a depth violation between the image content and a boundary associated with the viewing range includes: detecting that at least a portion of the image content extends beyond the boundary associated with the viewing range to determine which portions of particular image content cause a depth violation; In response to determining the depth conflict, resolving the depth conflict for the at least one portion using the viewing range and at least one user interface element, wherein resolving the depth conflict comprises selecting at least one user interface element based on a distance of the at least one portion from the boundary; and Modified image content is generated for rendering using the resolved depth conflicts, the modified image content comprising a portion of the image content replaced by the at least one user interface element.

13. The system of claim 12, wherein: Determining depth conflicts between the image content and a boundary associated with the viewing range further includes generating a plurality of three-dimensional voxels using at least some of the determined depths associated with the image content, the plurality of three-dimensional voxels representing locations in a plane of a display rendering the image content.

14. The system of claim 12, wherein: The boundary is associated with at least one edge of the lenticular display device; The depth conflict is determined based on a tracked head position of a user viewing the image content at a remote lenticular display device; as well as Resolving the depth conflict includes adjusting a size of the user interface element based on the tracked head position of the user.

15. The system of claim 12, wherein: Resolving the depth conflict includes generating the at least one user interface element as a box surrounding the image content, the at least one box adapted to accommodate movement depicted in the image content such that the at least one box is modified in size and / or shape.

16. A system according to any one of claims 12 to 15, wherein: The user interface element includes a blur overlay that begins at the border and ends at a predefined position associated with a size of a display device depicting the image content, wherein a blur radius associated with the blur overlay increases at a threshold distance from the border.

17. A non-transitory machine-readable medium having stored thereon instructions that, when executed by a processor, cause a computing device to: When capturing video content with multiple stereo cameras: determining a capture volume in the form of a physical volume of space bounded by one or more boundaries, the capture volume associated with the captured video content bounded within the one or more boundaries; determining a depth associated with the captured video content; Within the capture volume, defining a viewing range within which a stereoscopic effect occurs based on the depth; Determining a depth violation between the captured video content and a boundary associated with the viewing range includes: detecting that at least a portion of the captured video content extends beyond the boundary associated with the viewing range to determine which portions of particular image content cause a depth violation; In response to determining the depth conflict, resolving the depth conflict for the at least one portion using the viewing range and at least one user interface element, wherein resolving the depth conflict comprises selecting at least one user interface element based on a distance of the at least one portion from the boundary; and Modified video content is generated for rendering using the resolved depth conflicts, the modified video content including a portion of the video content replaced by the at least one user interface element.

18. The machine-readable medium of claim 17, wherein: Determining depth conflicts between the captured video content and a boundary associated with the viewing range further includes generating a plurality of three-dimensional voxels using at least some of the determined depths associated with the captured video content, the plurality of three-dimensional voxels representing locations in a plane of a display rendering the captured video content.

19. The machine-readable medium of claim 17, wherein: The boundary is associated with at least one edge of the lenticular display device; The depth conflict is determined based on a tracked head position of a user viewing the captured video content at a remote lenticular display device; as well as Resolving the depth conflict includes adjusting a size of the user interface element based on the tracked head position of the user.

20. The machine-readable medium of claim 17, wherein: The user interface element includes a blur overlay that begins at the border and ends at a predefined position associated with a size of a display device depicting the captured video content, wherein a blur radius associated with the blur overlay is increased at a threshold distance from the border.

Images