Three-dimensional (3D) facial feature tracking for an autostereoscopic telepresence system

Abstract

1. A method comprising: capturing at least one image using a plurality of cameras; identifying a plurality of facial features associated with the at least one image; stabilizing a position of at least one facial feature among the plurality of facial features; converting the position of the at least one facial feature to a three-dimensional (3D) position; predicting a future position of the at least one facial feature; and rendering a 3D image on a flat panel display using the future position of the at least one facial feature.

Description

[Technical field]

[0001] Field SUMMARY OF THE DISCLOSURE The embodiments relate generally to methods, devices, and algorithms used in generating content for display on a display. [Background technology]

[0002] background Three-dimensional (3D) telepresence systems may rely on 3D pose information of the user's head to determine where to display video and where to project audio. The 3D pose information must be accurate. For example, some systems may be accurate but require the user to wear a 3D marker ball. Furthermore, these systems are very expensive and have a large hardware footprint. Other systems may use small consumer devices that can be used for gaming, for example. However, the accuracy and speed of these systems do not meet the requirements of a 3D telepresence system. There is a need for a 3D pose information system with the desired accuracy and speed performance that can be used in a 3D telepresence system without the need for a wearable (e.g., 3D marker ball, head-mounted display, etc.) device. Summary of the Invention

[0003] overview In a general aspect, a device, system, non-transitory computer-readable medium (storing computer-executable program code that can be executed on a computer system), and / or method can perform a process using a method that includes capturing at least one image using multiple cameras, identifying a plurality of facial features associated with the at least one image, stabilizing a position of at least one facial feature of the plurality of facial features, converting the position of the at least one facial feature to a three-dimensional (3D) position, predicting a future position of the at least one facial feature, and rendering a 3D image on a flat panel display using the future position of the at least one facial feature.

[0004] Implementations may include one or more of the following features. For example, the at least one image may include two or more images, and the two or more images may be captured at least one of sequentially by a same camera and simultaneously by two or more cameras. Identifying the facial features may include identifying a face associated with the at least one image, generating a plurality of landmarks on the face corresponding to the facial features, and associating a location with each of the plurality of landmarks. Identifying the facial features may include identifying a first face associated with the at least one image, identifying a second face associated with the at least one image, generating a first plurality of landmarks corresponding to the facial features on the first face, generating a second plurality of landmarks corresponding to the facial features on the second face, associating a first location with each of the first plurality of landmarks, and associating a second location with each of the second plurality of landmarks.

[0005] Stabilizing a position of the at least one facial feature may include selecting a facial feature from at least two of the at least one image, selecting at least two landmarks associated with the facial feature, and averaging positions of the at least two landmarks. To stabilize may include generating a plurality of landmarks corresponding to the plurality of facial features, selecting a subset of the plurality of landmarks, and determining a facial movement based on a velocity associated with a position of each landmark of the subset of the plurality of landmarks, and stabilizing a position of the at least one facial feature is based on the movement of the face. Converting the position of the at least one facial feature to a 3D position may include triangulating the position of the at least one facial feature based on position data generated using images captured by three or more of the plurality of cameras.

[0006] The method may further include applying a double exponential filter to the 3D position of the at least one facial feature to reduce noise associated with the 3D position of the at least one facial feature. Predicting a future position of the at least one facial feature may include applying a double exponential filter to the sequence of 3D positions of the at least one facial feature and selecting a value at a time greater than zero (0) based on the sequence of double exponential filtered 3D positions prior to time zero (0). The method may further include generating audio using the future position of the at least one facial feature.

[0007] In a general aspect, a device, system, non-transitory computer-readable medium (storing computer-executable program code that can be executed on a computer system), and / or method can perform a process using a device including a memory including code segments representing a plurality of computer instructions and a processor configured to execute the code segments, the computer instructions including capturing at least one image using a plurality of cameras, identifying a plurality of facial features associated with the at least one image, stabilizing a position of at least one facial feature of the plurality of facial features, converting the position of the at least one facial feature to a three-dimensional (3D) position, predicting a future position of the at least one facial feature, and rendering a 3D image on a flat panel display using the future position of the at least one facial feature.

[0008] Implementations may include one or more of the following features. For example, the at least one image may include two or more images, and the two or more images may be captured at least one of sequentially by a same camera and simultaneously by two or more cameras. Identifying the facial features may include identifying a face associated with the at least one image, generating a plurality of landmarks on the face corresponding to the facial features, and associating a location with each of the plurality of landmarks. Identifying the facial features may include identifying a first face associated with the at least one image, identifying a second face associated with the at least one image, generating a first plurality of landmarks corresponding to the facial features on the first face, generating a second plurality of landmarks corresponding to the facial features on the second face, associating a first location with each of the first plurality of landmarks, and associating a second location with each of the second plurality of landmarks.

[0009] Stabilizing a position of the at least one facial feature may include selecting a facial feature from at least two of the at least one image, selecting at least two landmarks associated with the facial feature, and averaging positions of the at least two landmarks. Stabilizing a position of the at least one facial feature may include generating a plurality of landmarks corresponding to the plurality of facial features, selecting a subset of the plurality of landmarks, and determining facial movement based on a velocity associated with a position of each landmark of the subset of the plurality of landmarks, where stabilizing a position of the at least one facial feature is based on the movement of the face. Converting the position of the at least one facial feature to a 3D position may include triangulating the position of the at least one facial feature based on position data generated with images captured by three or more of the plurality of cameras.

[0010] The computer instructions may further include applying a double exponential filter to the 3D position of the at least one facial feature to reduce noise associated with the 3D position of the at least one facial feature. Predicting a future position of the at least one facial feature may include applying a double exponential filter to the sequence of 3D positions of the at least one facial feature and selecting a value at a time greater than zero (0) based on the sequence of double exponential filtered 3D positions prior to time zero (0). The computer instructions may further include generating audio using the future position of the at least one facial feature.

[0011] The illustrative embodiments will be more fully understood from the following detailed description herein and the accompanying drawings, in which like elements are represented by like reference numerals, and which are provided by way of example only and therefore not as limitations of the illustrative embodiments. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a block diagram illustrating an example 3D content system for displaying content on a display device in accordance with at least one example implementation. [Diagram 2] FIG. 1 is a block diagram of an example system for modeling content for rendering on a display device in accordance with at least one example implementation. [Diagram 3] FIG. 1 is a block diagram of a system for tracking facial features and locating position(s) based on facial features in accordance with at least one exemplary embodiment. [Figure 4] FIG. 2 is a pictorial illustration of facial feature locations in accordance with at least one exemplary embodiment. [Diagram 5] FIG. 1 is a block diagram of a method for tracking facial features and locating position(s) based on facial features according to at least one exemplary embodiment. [Figure 6] FIG. 1 illustrates an example of a computing device and a mobile computing device in accordance with at least one exemplary embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Detailed Description of the Embodiments It should be noted that these drawings are intended to illustrate general features of methods, structures and / or materials utilized in some exemplary embodiments, as well as to supplement the descriptions provided below. However, these drawings are not to scale, may not precisely reflect the exact structural or performance characteristics of any embodiment, and should not be construed as defining or limiting the range of values ​​or properties encompassed by the exemplary embodiments. For example, the relative thicknesses and positions of molecules, layers, regions and / or structural elements may be reduced or exaggerated for clarity. The use of similar or identical reference numbers in various drawings is intended to indicate the presence of similar or identical elements or features.

[0014] Augmented reality and virtual reality (AR / VR) systems that do not use wearable devices (e.g., head-mounted displays (HMDs)), such as autostereoscopic / glasses-free telepresence systems that use stationary 3D displays, may rely on having accurate, up-to-date 3D poses of the user's facial features (e.g., eyes and ears). For example, these systems (e.g., 3D telepresence systems) may use accurate eye tracking to modify the displayed scene by positioning a virtual camera and projecting separate left and right stereo images to the left and right eyes, respectively. Such systems may also use ear tracking to accurately process audio output and input to achieve spatial audio without headphones. Instead, these systems may use, for example, stationary speaker pairs.

[0015] In a 3D telepresence system, the facial feature tracking system should be at least high speed, low latency, low noise and jitter, markerless, etc. As an example, a high speed tracking system should be able to update every 8.33 ms on a 120 Hz display system. A low latency tracking system should report the user's position such that the updated rendering on the screen matches the current user position. Otherwise, if the latency is too high, the user may become dizzy and experience severe visual artifacts. A low noise and jitter tracking system should minimize scene shaking that can cause visual discomfort. A markerless tracking system should enable the user to participate in the telepresence system without the discomfort of a device attached to the user (e.g., without an undesirable user experience).

[0016] An example implementation may include a system for accurately and cost-effectively tracking facial features of a user (e.g., a user of a 3D telepresence system) without the use of a wearable device (e.g., an HMD, a marker, and / or the like). The facial feature tracking system may capture images at regular intervals using at least one camera (e.g., a monochrome camera, an infrared camera, and / or the like). The regular intervals may be short (e.g., less than 6 ms, less than 8 ms, less than 10 ms, etc.). These images may be used to locate the face and facial features and generate virtual camera positions, display positions, audio positions, and / or the like. Advantages of this tracking system are that the user does not need to wear a device for facial feature tracking (improved user experience), the tracking system can acquire features quickly, and the tracking system includes software for generating accurate (e.g., minimal noise) location information.

[0017] FIG. 1 is a block diagram illustrating an exemplary 3D content system 100 for capturing and displaying content on a stereoscopic display device, according to implementations described throughout this disclosure. The 3D content system 100 can be used by multiple users to conduct, for example, video conference communication in 3D (e.g., telepresence sessions). In general, the system of FIG. 1 can be used to capture video and / or images of users during a video conference and generate virtual camera, viewing, and audio positions using the systems and techniques described herein. The system 100 can benefit from the use of position generation systems and techniques described herein, because such techniques can be used to project video and audio content to improve video conferences. For example, video can be projected to render 3D video based on the viewer's position and audio can be projected based on the video conference participant's position.

[0018] 1, a 3D content system 100 is being used by a first user 102 and a second user 104. For example, users 102 and 104 are participating in a 3D telepresence session using the 3D content system 100. In such an example, the 3D content system 100 can enable each of users 102 and 104 to view highly realistic and visually consistent representations of each other, thereby facilitating the users to interact with each other in a manner similar to being physically present.

[0019] Each user 102, 104 may have a corresponding 3D system, where user 102 has 3D system 106 and user 104 has 3D system 108. The 3D systems 106, 108 may provide functions related to 3D content, including but not limited to capturing images for 3D display, processing and displaying image information, and processing and displaying audio information. The 3D systems 106 and / or 108 may constitute a collection of sensing devices integrated as one unit. The 3D systems 106 and / or 108 may include some or all of the components described with reference to FIG. 3.

[0020] The 3D system 106, 108 may include multiple components related to capturing, processing, transmitting, positioning or receiving 3D information, and / or displaying 3D content. The 3D system 106, 108 may include one or more cameras for capturing image content of images included in the 3D display and / or for capturing faces and facial features. Here, the 3D system 106 includes cameras 116 and 118. For example, the camera 116 and / or the camera 118 may be disposed substantially within the housing of the 3D system 106 such that the objective lens or lens of the respective camera 116 and / or 118 captures image content through one or more apertures of the housing. In some implementations, the camera 116 and / or 118 is separable from the housing, such as in the form of a stand-alone device (e.g., having a wired and / or wireless connection with the 3D system 106). As shown in FIG. 1, at least one camera 114, 114' and / or 114'' is shown as separate from the housing as a standalone device that can be communicatively coupled to the 3D system 106 (e.g., using a wired and / or wireless connection).

[0021] In an example implementation, at least one image can be captured using multiple cameras. Two or more images can be captured using multiple cameras. For example, two or more images can be captured sequentially by the same camera (e.g., for tracking). Two or more images can be captured simultaneously by two or more cameras (e.g., used for triangulation).

[0022] The cameras 114, 114', 114", 116 and 118 may be positioned and / or oriented to capture a sufficiently representative view of a user (e.g., user 102). Although the cameras 114, 114', 114", 116 and 118 generally do not obscure the view of the 3D display 110 for the user 102, the placement of the cameras 114, 114', 114", 116 and 118 may be selected arbitrarily. For example, one of the cameras 116, 118 may be positioned somewhere above the face of the user 102 and the other may be positioned somewhere below the face. The cameras 114, 114' and / or 114" may be positioned to the left, right and / or above the 3D system 106. For example, one of the cameras 116, 118 may be positioned somewhere on the right side of the face of the user 102 and the other may be positioned somewhere on the left side of the face. The 3D system 108 may similarly include cameras 120, 122, 134, 134' and / or 134'', for example. Additional cameras are possible. For example, a third camera may be located near or behind the display 110.

[0023] The 3D content system 100 may include one or more 2D or 3D displays. Here, a 3D display 110 is provided for the 3D system 106, and a 3D display 112 is provided for the 3D system 108. The 3D displays 110, 112 may provide autostereoscopic viewing to the respective viewer (here, e.g., user 102 or user 104) using any of several types of 3D display technology. In some implementations, the 3D displays 110, 112 may be standalone units (e.g., freestanding or wall-hung). In some implementations, the 3D displays 110, 112 may include or have access to wearable technology (e.g., controllers, head-mounted displays, etc.).

[0024] In general, a 3D display, such as displays 110, 112, can provide images that approximate the 3D optical properties of physical objects in the real world without the use of a head-mounted display (HMD) device. Generally, the displays described herein include flat panel displays, lenticular lenses (e.g., microlens arrays), and / or parallax barriers for redirecting images to multiple different viewing regions associated with the display.

[0025] In some implementations, the displays 110, 112 may be flat panel displays including high resolution, glasses-free, lenticular three-dimensional (3D) displays. For example, the displays 110, 112 may include a microlens array (not shown) including a plurality of lenses (e.g., microlenses) with glass spacers coupled (e.g., glued) to the microlenses of the display. The microlenses may be designed such that, from a selected viewing position, the left eye of a user of the display can see a first set of pixels and the right eye of the user can see a second set of pixels (e.g., the second set of pixels is mutually exclusive with the first set of pixels).

[0026] In some exemplary 3D displays, there may be one location that provides a 3D view of the image content (e.g., user, objects, etc.) provided by such a display. The user may sit at this one location and experience proper parallax, minimal distortion, and realistic 3D imagery. As the user moves to a different physical location (or changes head or eye position), the image content (e.g., the user, objects worn by the user, and / or other objects) may begin to appear less realistic, 2D, and / or distorted. Thus, the techniques described herein may enable accurate determination of the user position (e.g., the user's eyes) to enable realistic 3D generation. The systems and techniques described herein may reconstruct the image content projected from the display such that the user moves around but still reliably experiences proper parallax, low distortion rates, and realistic 3D imagery in real time. Thus, the systems and techniques described herein provide the advantage of maintaining and providing 3D image content and objects for viewing to the user regardless of the user's movements that occur while the user is viewing the 3D display.

[0027] As shown in Figure 1, the 3D content system 100 can be connected to one or more networks. Here, a network 132 is connected to the 3D system 106 and the 3D system 108. The network 132 can be a publicly available network (e.g., the Internet) or a private network, to name just two examples. The network 132 can be wired, or wireless, or a combination of the two. The network 132 can include or use one or more other devices or systems, including but not limited to one or more servers (not shown).

[0028] The 3D systems 106, 108 may include a face finder / recognition tool. The 3D systems 106, 108 may include a facial feature extraction tool. For example, the 3D systems 106, 108 may include a machine learning (ML) tool (e.g., software) configured to identify faces in an image and extract facial features and their locations (or x, y, z locations). The image(s) may be captured using cameras 114, 114', 114", 116 and / or 118 (for the 3D system 106) and using cameras 120, 122, 134, 134' and / or 134" (for the 3D system 108).

[0029] The 3D systems 106, 108 may include one or more depth sensors for capturing depth data used for the 3D display. Such depth sensors may be considered part of a depth capture component in the 3D content system 100 that is used to characterize the scene captured by the 3D systems 106 and / or 108 in order to accurately represent the scene on a 3D display. Additionally, the system may track the position and orientation of the viewer's head so that the 3D display may be rendered with an appearance that corresponds to the viewer's current viewpoint. Here, the 3D system 106 includes a depth sensor 124. Similarly, the 3D system 108 may include a depth sensor 126. Any of several types of depth sensing or depth capture may be used to generate the depth data.

[0030] In some implementations, assisted stereo depth capture is performed. A dot of light can be used to illuminate the scene, and stereo matching can be performed, for example, between two respective cameras. The illumination can be done with waves of a selected wavelength or wavelength range. For example, infrared (IR) light can be used. In some implementations, a depth sensor may not be utilized in generating the view, for example, on a 2D device. The depth data can include or be based on any information about the scene that reflects the distance between a depth sensor (e.g., depth sensor 124) and an object in the scene. For content in the image that corresponds to an object in the scene, the depth data reflects the distance (or depth) to the object. For example, the spatial relationship between the camera(s) and the depth sensor can be known and used to correlate the image from the camera(s) with a signal from the depth sensor to generate depth data for the image.

[0031] The images captured by the 3D content system 100 may be processed and then displayed as a 3D representation. As shown in the example of FIG. 1, a 3D image 104′ having an object (glasses 104″) is displayed on a 3D display 110. In this manner, the user 102 may perceive the 3D image 104′ and the glasses 104″ as a 3D representation of the user 104, who may be located remotely from the user 102. A 3D image 102′ is displayed on a 3D display 112. In this manner, the user 104 may perceive the 3D image 102′ as a 3D representation of the user 102.

[0032] The 3D content system 100 can enable participants (e.g., users 102, 104) to engage in audio communications with each other and / or others. In some implementations, the 3D system 106 includes a speaker and a microphone (not shown). For example, the 3D system 108 may include a speaker and a microphone as well. In this manner, the 3D content system 100 can enable users 102 and 104 to participate in a 3D telepresence session with each other and / or others.

[0033] 2 is a block diagram of an example system 200 for modeling content for rendering on a 3D display device, according to implementations described throughout this disclosure. System 200 can function as or be included in one or more implementations described herein, and / or can be used to perform the operation(s) of one or more examples of 3D processing, modeling, or display described herein. System 200 as a whole and / or one or more of its individual components can be implemented according to one or more examples described herein.

[0034] System 200 includes one or more 3D systems 202. In the illustrated example, 3D systems 202A, 202B through 202N are shown, where the subscript N denotes any number. 3D systems 202 can provide for the capture of visual and audio information for 3D display and transfer the 3D information for processing. Such 3D information can include images of a scene, depth data about the scene, and audio from the scene. For example, 3D system 202 can function as 3D system 106 and 3D display 110 (FIG. 1) or can be included within 3D system 106 and 3D display 110.

[0035] System 200 may include multiple cameras, as illustrated by camera 204. Any type of light sensing technology may be used to capture images, such as image sensors of the types used in typical digital cameras, monochrome cameras, and / or infrared cameras. Cameras 204 may be of the same type or different types. The camera locations may be located within any location on (or external to) a 3D system, such as 3D system 106, for example.

[0036] The system 202A includes a depth sensor 206. In some implementations, the depth sensor 206 operates by propagating an IR signal over a scene and detecting a response signal. For example, the depth sensor 206 can generate and / or detect beams 128A-B and / or 130A-B.

[0037] The system 202A also includes at least one microphone 208 and a speaker 210. For example, these may be integrated into a head-mounted display worn by a user. In some implementations, the microphone 208 and the speaker 210 may be part of the 3D system 106 or may not be part of the head-mounted display.

[0038] The system 202 further includes a 3D display 212 capable of stereoscopically displaying 3D images. In some implementations, the 3D display 212 can be a standalone display, and in some other implementations, the 3D display 212 can be included in a head-mounted display unit configured to be worn by a user to experience a 3D display. In some implementations, the 3D display 212 operates using parallax barrier technology. For example, the parallax barrier can include parallel vertical stripes of a substantially non-transparent material (e.g., an opaque film) disposed between a screen and a viewer. Due to the parallax between each eye of the viewer, different portions of the screen (e.g., different pixels) are seen by the left and right eyes, respectively. In some implementations, the 3D display 212 operates using a lenticular lens. For example, alternating rows of lenses can be disposed in front of the screen, which direct light from the screen to the left and right eyes of the viewer, respectively.

[0039] The system 202A includes a tracking and location 214 block. The tracking and location 214 block may be configured to track a user's position in a room. In some implementations, the tracking and location 214 block may track the user's eye position. In some implementations, the tracking and location 214 block may track the user's head position. The tracking and location 214 block may be configured to determine the position of the user, microphone, camera, etc. in the system 202. In some implementations, the tracking and location 214 block may be configured to generate a virtual position based on the user's face and / or facial features. For example, the tracking and location block 214 may be configured to generate a virtual camera position based on the user's face and / or facial features. In some implementations, the tracking and location 214 block can be implemented using cameras 114, 114', 114", 116 and / or 118 (for 3D system 106) and using cameras 120, 122, 134, 134' and / or 134" (for 3D system 108). The tracking and location 214 block may be configured to implement the methods and techniques described in this disclosure within the 3D system(s) 202.

[0040] The system 200 may include a server 216 that may perform specific tasks of data processing, data modeling, data federation, and / or data transmission. The server 216 and / or its components may include some or all of the components described with reference to FIG.

[0041] The server 216 includes a 3D content generator 218 that can be responsible for rendering the 3D information in one or more ways. This may include receiving 3D content (e.g., from the 3D system 202A), processing the 3D content, and / or forwarding the (processed) 3D content to another participant (e.g., to another 3D system 202).

[0042] Some aspects of the functions performed by the 3D content generator 218 may be implemented for execution by a shader 218. The shader 218 may be responsible for applying shading with respect to particular portions of an image, and may also be responsible for performing other services related to the image for which shading has been or will be provided. For example, the shader 218 may be utilized to counteract or hide certain artifacts that may otherwise be generated by the 3D system(s) 202.

[0043] Shading refers to one or more parameters that define the appearance of image content, including but not limited to the color of objects, surfaces, and / or polygons in an image. In some implementations, shading can be applied to or adjusted for one or more portions of image content to change how those image content portion(s) appear to a viewer. For example, shading can be applied / adjusted to make the image content portion(s) darker, lighter, transparent, etc.

[0044] The 3D content generator 218 may include a depth processing component 220. In some implementations, the depth processing component 220 can apply shading (e.g., darker, lighter, transparent, etc.) to image content based on one or more depth values ​​associated with that content and based on one or more received inputs (e.g., content model inputs).

[0045] The 3D content generator 218 may include an angle processing component 222. In some implementations, the angle processing component 222 may apply shading to image content based on the orientation (e.g., angle) of the content relative to the camera capturing the image content. For example, shading may be applied to content that points away from the camera angle at angles greater than a predetermined threshold angle. This may allow the angle processing component 222 to reduce and fade out the brightness of surfaces as they move away from the camera, to name just one example.

[0046] The 3D content generator 218 includes a renderer module 224. The renderer module 224 may render content to one or more 3D system(s) 202. For example, the renderer module 224 may render an output / composite image that may be displayed, for example, in the system 202.

[0047] 2, the server 216 also includes a 3D content modeler 230 that may be responsible for modeling the 3D information in one or more ways. This may include receiving 3D content (e.g., from the 3D system 202A), processing the 3D content, and / or forwarding the (processed) 3D content to another participant (e.g., to another 3D system 202). The 3D content modeler 230 may utilize an architecture 400 to model objects, as described in more detail below.

[0048] Pose 232 may represent a pose associated with the captured content (e.g., an object, a scene, etc.). In some implementations, pose 232 may be detected and / or otherwise determined by a tracking system associated with system 100 and / or 200 (e.g., may be realized using cameras 114, 114', 114", 116 and / or 118 (for 3D system 106) and using cameras 120, 122, 134, 134' and / or 134" (for 3D system 108). Such a tracking system may include sensors, cameras, detectors, and / or markers for tracking the position of all or a portion of the user. In some implementations, the tracking system may track the position of the user within a room. In some implementations, the tracking system may track the position of the user's eyes. In some implementations, the tracking system may track the position of the user's head.

[0049] In some implementations, a tracking system may track the user's position (or the user's eye or head position), for example, relative to the display device 212, in order to display images with the appropriate depth and parallax. In some implementations, the head position associated with the user may be detected and used as an orientation for simultaneously projecting images to the user on the display device 212, for example via microlenses (not shown).

[0050] The categories 234 may represent classifications of particular objects 236. For example, the category 234 may be glasses, and the objects may be blue glasses, clear glasses, round glasses, etc. Any category and object may be represented by the models described herein. The categories 234 may be used as a basis for training a generative model on the objects 236. In some implementations, the categories 234 may represent a dataset that can be used to synthetically render a 3D object category under different viewpoints that gives access to a set of ground truth poses, color space images, and masks for multiple objects of the same category.

[0051] Three-dimensional (3D) proxy geometry 238 represents both a (coarse) geometric approximation of a set of objects and one or more latent textures 239 of the objects mapped to the respective object geometries. The coarse geometries and the mapped latent textures 239 may be used to generate images of one or more objects in a category of objects. For example, the systems and techniques described herein may generate objects for a 3D telepresence display by rendering the latent textures 239 onto a target viewpoint and accessing a neural rendering network (e.g., a differential deferred rendering neural network) to generate a target image on a display. To learn such latent textures 239, the systems described herein may learn a low-dimensional latent space of neural textures and a shared deferred neural rendering network. The latent space encompasses all instances of a class of objects and allows for the interpolation of instances of the object, which may enable the reconstructing of instances of the object from a small number of viewpoints.

[0052] The neural texture 244 represents the learned feature map 240 that is trained as part of the image capture process. For example, when an object is captured, the neural texture 244 may be generated using the feature map 240 and the 3D proxy geometry 238 for that object. In operation, the system 200 may generate the neural texture 244 for a particular object (or scene) and store it as a map on the 3D proxy geometry 238 for that object. For example, the neural texture may be generated based on the latent code associated with each instance of the identified category and the view associated with the pose.

[0053] The geometric approximation 246 may represent a shape-based proxy for the object geometry. The geometric approximation 246 may be a mesh-based, shape-based (e.g., triangle, diamond, square, etc.), free-form version of the object.

[0054] Neural renderer 250 may generate intermediate representations of objects and / or scenes to render, for example, utilizing a neural network. Neural texture 244 may be used to jointly learn features on texture maps (e.g., feature map 240) with a 5-layer U-Net, such as neural network 242, operating with neural renderer 250. Neural renderer 250 may incorporate view-dependent effects, for example, by modeling the difference between true appearance (e.g., ground truth) and diffuse reprojection using object-specific convolutional networks. Such effects may be difficult to predict based on scene knowledge, and thus a GAN-based loss function may be used to render realistic output.

[0055] The RGB color channels 252 (e.g., a color image) represent three output channels. For example, the three output channels may include a red color channel, a green color channel, and a blue color channel (e.g., RGB) that represent a color image. In some implementations, the color channels 252 may be a YUV map that indicates which colors to render for a particular image. In some implementations, the color channels 252 may be a CIE map. In some implementations, the color channels 252 may be an ITP map.

[0056] Alpha (α) 254 represents an output channel (e.g., a mask) that describes how a particular pixel color merges with other pixels when overlaid, for any number of pixels in an object. In some implementations, alpha 254 represents a mask that defines the transparency level (e.g., translucency, opacity, etc.) of the object.

[0057] The above exemplary components are described herein as being implemented in a server 216 that can communicate with one or more of the 3D systems 202 by a network 260 (which may be similar or identical to the network 132 of FIG. 1). In some implementations, the 3D content generator 216 and / or its components can instead or additionally be implemented in some or all of the 3D systems 202. For example, the modeling and / or processing described above can be performed by a system that generates the 3D information before forwarding the 3D information to one or more receiving systems. As another example, the generating system can forward the images, modeling data, depth data, and / or corresponding information to one or more receiving systems that can perform the above processing. A combination of these approaches can be used.

[0058] Thus, system 200 is an example of a system including a camera (e.g., camera 204), a depth sensor (e.g., depth sensor 206), and a 3D content generator (e.g., 3D content generator 218) having a processor executing instructions stored in a memory. Such instructions can cause the processor to use depth data included in the 3D information (e.g., by depth processing component 220) to identify image content within images of a scene included in the 3D information. Image content can be identified as associated with depth values ​​that meet a criteria. The processor can generate modified 3D information, for example, by applying a model generated by 3D content modeler 230, which can be provided to 3D content generator 218 to properly render composite image 256.

[0059] The composite image 256 represents a 3D stereoscopic image of the particular object 236 with the appropriate parallax and viewing configuration for both eyes associated with a user accessing a display (e.g., display 212) based at least in part on the tracked position of the user's head. At least a portion of the composite image 256 may be determined based on output from the 3D content modeler 230, for example, using the system 200 each time the user moves the position of their head while looking at the display. In some implementations, the composite image 256 represents the object 236 and other objects, users, or image content in the view that captures the object 236.

[0060] In some implementations, the processor (not shown) of the systems 202 and 216 may include (or may communicate with) a graphics processing unit (GPU). In operation, the processor may include (or have access to) memory, storage, and other processors (e.g., CPUs). To facilitate graphics and image generation, the processor may communicate with the GPU to display images on a display device (e.g., display device 212). The CPU and GPU may be connected via a high-speed bus such as PCI, AGP, or PCI-Express. The GPU may be connected to a display via another high-speed interface such as HDMI, DVI, or Display Port. In general, the GPU may render image content in pixel format. The display device 212 may receive image content from the GPU and display this image content on a display screen.

[0061] FIG. 3 illustrates a block diagram of a system for tracking facial features and locating a position(s) based on facial features, according to at least one exemplary embodiment. As shown in FIG. 3, the system 300 includes a feature tracker 305 block, a 3D feature triangulation 325 block, a double exponential filter with prediction 330 block, a virtual camera position 335 block, a display position 340 block, and an audio position 345 block. The feature tracker 305 block includes a camera 310 block, a 2D facial feature extraction 315 block, and a 2D feature stabilizer 320 block. An exemplary implementation may include multiple feature trackers (shown as a feature tracker 305-1 block, a feature tracker 305-2 block, a feature tracker 305-3 block, a feature tracker 305-4 block, ..., and a feature tracker 305-n block). An exemplary implementation may include using at least two (2) feature trackers 305. For example, an implementation may use four (4) feature trackers 305 to optimize (e.g., improve) accuracy, optimize (e.g., reduce) noise, and optimize (e.g., expand) capture volume compared to systems that use fewer than four (4) feature trackers 305.

[0062] The camera 310 may be a monochrome camera operating at 120 frames per second. If more than one camera is used, the cameras 310 may be connected to a hardware trigger to ensure that the cameras 310 operate simultaneously. The resulting image frames may be referred to as a frameset, with each frame in a frameset being taken at the same instant in time. The camera 310 may also be an infrared camera with similar operating characteristics as a monochrome camera. The camera 310 may be a combination of monochrome and infrared cameras. The camera 310 may be a fixed camera (e.g., cameras 116, 118, 120, 122) in the 3D content system. The camera 310 may be a free-standing camera (e.g., cameras 114, 114', 114", 134, 134', 134") coupled to the 3D content system. The camera 310 may be a combination of fixed and free-standing cameras. Multiple cameras may be implemented in multiple feature trackers 305.

[0063] The 2D facial feature extraction 315 may be configured to extract facial features from an image captured using the camera 310. Thus, the 2D facial feature extraction 315 may be configured to identify a face of a user (e.g., a participant in a 3D telepresence communication) and extract facial features of the identified face. A face detector (face finder, face locator, and / or other) may be configured to identify a face in the image. The face detector may be implemented as a function call in a software application. The function call may return the Cartesian coordinates of the location of the face. The face detector may be configured to pick out one face if there is more than one user in the image.

[0064] From the identified face, facial features can be extracted. The facial features can be extracted using a 2D ML algorithm or model. The facial feature extractor can be implemented as a function call in the software application. The function call can return the locations of facial landmarks (or key points) of the face. The facial landmarks can include eyes, mouth, ears, and / or others. The facial recognition and facial feature extraction can be implemented as a single function call that returns the facial landmarks.

[0065] FIG. 4 pictorially illustrates facial feature locations. In other words, FIG. 4 pictorially illustrates landmarks using numbers. For example, numbers 0 and 1 can be the pupils of the left and right eyes, respectively. The facial feature extractor can output two variables including a landmark number and a pixel number. The pixel number can be based on the resolution (e.g., number of pixels) of the image. For example, the image can have a resolution of 2048×1536 (e.g., a 3 megapixel camera). Thus, the image can have 3,145,728 pixels, with the first pixel being in the top left corner of the image and the last pixel being in the bottom right corner of the image.

[0066] An exemplary bivariate landmark output may be (1,123456), which may identify the landmark as the pupil of the right eye, which is the 123456th pixel. The pixel location (e.g., 123456) may be mapped to an x ​​and y location as needed (e.g., for use in some calculations). The facial feature extractor may output a triple that includes the landmark number, the x location, and the y location in the image (e.g., a 2048×1526 image) using a 2D coordinate system. For example, the triple (1,23,218) may identify the landmark as the pupil of the right eye, which is at x location 23 and y location 218. These are just two possible examples for locating facial landmarks.

[0067] In at least one example implementation, the eyes of a user (e.g., a participant in a 3D telepresence communication) may be landmarks of interest. The eyes may be landmarks of interest because they may indicate the pose of the user's head. Where the user is looking may not be of interest. Thus, the pupils (e.g., landmark 0 and landmark 1) may not be used to indicate the pose of the user's head. However, the user's eye sockets may be used to indicate the pose of the user's head. For example, landmarks 402, 414, 406, and 410 may indicate the position of the left eye, and landmarks 404, 416, 408, and 412 may indicate the position of the right eye. Each of the landmarks (e.g., 402, 414, 406, and 410) has a location. Multiple locations may be used to create an averaged location (e.g., the center of the eye). However, other mathematical combinations of landmark locations are possible. For example, the landmark locations may be weighted, normalized, averaged, interpolated (e.g., midpoint or non-midpoint interpolation), and / or other.

[0068] In at least one example implementation, the user's ears can be landmarks of interest. Thus, landmark 418 and landmark 420 can be landmarks of interest to indicate the positions of the left and right ears, respectively.

[0069] Returning to FIG. 3, the 2D feature stabilizer 320 may be configured to reduce noise associated with landmarks. For example, a filter may be applied to the 2D feature positions (e.g., landmark positions) to stabilize the 2D features. In an exemplary implementation, a filter may be applied to reduce noise associated with eye positions (e.g., landmarks 402, 414, 406, 410 and landmarks 404, 416, 408, 412). In at least one exemplary implementation, at least two images are used for feature stabilization.

[0070] In an example implementation, stabilizing the eyes (e.g., as positions of 2D features) may include determining a center of each eye by averaging the positions of landmarks around each eye. Averaging landmarks around the eyes can reduce noise associated with two or more landmarks, since noise associated with these landmarks may not be correlated.

[0071] Stabilizing the position of the facial features (e.g., eyes) may be based on facial movement. Facial movement (e.g., velocity) may be used to further reduce noise. A set of all facial landmarks may be generated (see FIG. 4), and a subset of facial landmarks that are substantially stable may be determined. For example, landmarks that are particularly noisy may be excluded. For example, ears and cheeks may exhibit high levels of noise and imprecision. Thus, to generate a substantially stable subset of facial landmarks, landmarks associated with ears (e.g., landmark 418 and landmark 420) and cheeks (e.g., landmark 436 and landmark 438) may be excluded. Determining the average facial movement may include calculating an average 2D velocity of the subset of facial landmarks. Considering that the eye sockets are fixed relative to the rest of the face, the average velocity of the face should be close to the average velocity of the eye sockets. Thus, the averaged eye center velocity and the average velocity of the subset of facial landmarks may be summed. The averaged eye center velocity and the average velocity of the subset of facial landmarks may be summed with a preselected set of weights. For example, the face velocity may be weighted at 90% and the eye center velocity may be weighted at 10%. The stabilized eye features may be based on the original positions of the eyes (e.g., landmarks 402, 414, 406, 410 and landmarks 404, 416, 408, 412) and the calculated average velocity.

[0072] The 3D feature triangulation 325 may be configured to obtain 3D positions of the landmarks, in other words, the 2D locations (or positions) of the facial features may be transformed into three-dimensional (3D) locations (or positions). In an example implementation, the positions and orientations (relative to another camera and / or display) of the cameras (e.g., cameras 114, 114′, 114″, 116, 118) and the 3D displays (e.g., displays 110, 112) are known (e.g., using calibration when setting up the 3D telepresence system). The X and Y coordinates in the 2D image space of each camera used to capture an image including facial features (e.g., landmarks) can be determined. A ray can be generated for each camera-landmark pair. For example, a ray can be drawn (e.g., using function calls in software) starting from the pixel location of the landmark (e.g., eye) toward each camera. If there are four cameras, four rays can be generated. The 3D position of the landmark can be determined based on these rays (e.g., the four rays). For example, the location where the rays intersect (or where the rays approach an intersection) may indicate the 3D position of the landmark (e.g., the left eye).

[0073] The double exponential filter with prediction 330 may be configured to reduce noise associated with the 3D landmarks. The double exponential filter with prediction 330 may also be configured (or used to predict) future positions of the 3D landmarks. Although the landmark noise was reduced using the 2D feature stabilizer 320, there may be residual noise associated with the 3D landmark locations (or positions). The residual noise may be amplified, for example, by environmental conditions or user aspects (e.g., glasses, facial hair, and / or other). The double exponential filter with prediction 330 may be configured to reduce this residual noise.

[0074] In an exemplary implementation, a modified band filter may define a band for each axis (x, y, and z) from n0 to n1, thus having a width of n1 to n0. If the new value is less than n0, n0 may be set to the new value and n1 may be set to n0+width. Otherwise, if the new value is greater than n1, n1 may be set to the new value and n0 is set to n1-width. If the new value is between n0 and n1, nothing changes. The actual value reported may be the center of the band, i.e., (n0+n1) / 2. This modified band filter may remove jitter. However, the wider the band, the less responsive the signal may be, and the greater the potential error. Therefore, the band should be as narrow as possible.

[0075] In a 3D telepresence system, the tracking-display system may have an inherent latency. This latency may be from the time that the photons capturing the user's new position are received by the head-tracking camera to the time that the newly calculated position is sent to the renderer and finally to the display where the display pixels change row by row. In some cases, the delay may be around 60 milliseconds. Thus, the user's movements can be used to render the scene by predicting the correct position and accurately project the stereo image to the user's eyes through an autostereo display. The predictive double exponential filter 330 may include a trend signal that is used to predict the future (e.g., predict the future positions of landmarks).

[0076] The output of the modified bandpass filter may be input to a double exponential filter that further smooths the signal from additional noise and predicts future signals. The double exponential filter may be two recursive applications of an exponential filter. The double exponential smoothing may use a term that accounts for trends in the input data (e.g., a series of data). This term may be a gradient component that may be updated as a result of the exponential smoothing.

[0077]

number

[0078] Other algorithms can be used for exponential smoothing (e.g., linear exponential smoothing). In an exemplary implementation, the current value of the location of the landmark (e.g., facial feature) is x t and the value of the future location of the landmark is F t+m In other words, predicting the future location of a facial feature (e.g., a landmark) can be done by calculating a value at time greater than zero (0) (e.g., at time m>0) (e.g., F) based on a sequence of double exponentially filtered 3D locations before time zero (0) (e.g., based on data up to time t). t+m ).

[0079] The virtual camera position 335, display position 340, and audio position 345 can be used as data for the tracking process, with the current and / or future position values ​​of the landmarks as binaries that can be used to drive a display, as input to a renderer for displaying the virtual scene (e.g., to determine the position of the left eye scene and the right eye scene), and for projecting audio (e.g., to determine the stereo balance between the left and right ear).

[0080] The above description of FIG. 3 is for identifying and determining the location of one face. However, the exemplary implementation is not limited to identifying and determining the location of one participant in a communication. In other words, two or more participants can be identified and located. Thus, the exemplary implementation may include identifying and determining the location of two or more faces along with the eyes, ears, and / or mouth of each face for purposes of driving a 3D display and 3D rendering system and / or for purposes of driving an audio system.

[0081] 3D rendering may include rendering a 3D scene from a desired viewpoint (POV) (e.g., determined as a face position) using a 3D image rendering system. For example, head tracking techniques described herein may be used to determine two POVs (one for the left eye and one for the right eye) for each user viewing the display. These viewpoints may then be inputs to the 3D rendering system for rendering the scene. Additionally, an autostereo display may require capturing images for each user's left and right eye and mapping them to individual screen pixels. The resulting image, rendered on an LCD panel (e.g., beneath a lenticular panel), may appear as if many images are interleaved. The mapping may be determined by the optical properties of the lens (e.g., how a pixel maps to a ray of light in space along which it is visible). An autostereo display may include any autostereo display capable of displaying separate images to the left and right eyes of a viewer. One such type of display may be realized by placing a lenticular lens array in front of an LCD panel, offset by a small distance.

[0082] FIG. 5 illustrates a block diagram of a method according to at least one exemplary embodiment. The steps described with respect to FIG. 5 may be performed by execution of software code stored in a memory associated with an apparatus (e.g., as shown in FIGS. 1 and 2) and executed by at least one processor (e.g., at least one processor) associated with the apparatus. However, alternative embodiments are contemplated, such as a system embodied as a dedicated processor. Although the steps described below are described as being performed by a processor, the steps are not necessarily performed by the same processor. In other words, at least one processor may perform the steps described below with respect to FIG. 5.

[0083] FIG. 5 illustrates a block diagram of a method for tracking facial features and locating position(s) based on facial features, according to at least one exemplary embodiment. This method for tracking facial features and locating position(s) based on facial features may be repeatedly triggered while performing 3D telepresence communication (e.g., in a 3D telepresence system). As shown in FIG. 5, in step S505, two-dimensional (2D) image(s) are captured. For example, multiple cameras may be used in a 3D telepresence system. With reference to FIG. 1, the image(s) may be captured using cameras 114, 114', 114", 116 and / or 118 (for the 3D system 106) and using cameras 120, 122, 134, 134' and / or 134" (for the 3D system 108).

[0084] In step S510, face(s) are detected. For example, the captured image can be used as input to a face detection algorithm. A face detector (face finder, face locator, and / or other) algorithm can be configured to identify a face in the image. The face detector algorithm can be implemented as a function call in a software application. The function call can return the Cartesian coordinates of the location of the face. The face detector algorithm can be a machine learning algorithm or model. The face detector algorithm can be configured to pick out one face if there is more than one user in the image. The face can be associated with a participant communicating using the 3D telepresence system.

[0085] In step S515, facial features are identified. For example, features associated with the detected face(s) may be identified. The facial features may be extracted using a 2D ML algorithm or model. The facial feature extractor may be implemented as a function call in a software application. The function call may return the location of facial landmarks (or key points) of the face. The facial landmarks may include eyes, mouth, ears, and / or others. The facial recognition and facial feature extraction may be implemented as a single function call that returns the facial landmarks. Identifying facial features is described in more detail above with respect to FIG. 4.

[0086] In step S520, the facial features are stabilized. For example, stabilizing the facial features can reduce noise associated with landmarks representing the facial features. For example, to stabilize the 2D features, a filter can be applied to the 2D feature positions (e.g., landmark positions). In an exemplary implementation, a filter can be applied to reduce noise associated with eye positions (e.g., landmarks 402, 414, 406, 410 and landmarks 404, 416, 408, 412). In at least one exemplary implementation, at least two images are used for feature stabilization.

[0087] In an example implementation, stabilizing the eyes (e.g., as 2D features) may include determining a center of each eye by averaging the positions of landmarks around each eye. Averaging landmarks around the eyes can reduce noise associated with two or more landmarks, since noise associated with these landmarks may not be correlated.

[0088] Facial motion (e.g., velocity) can be used to further reduce noise. Determining facial motion can include calculating an average 2D velocity of a subset of facial landmarks. Considering that the eye sockets are fixed relative to the rest of the face, the facial motion should be close to the eye socket motion. Thus, the averaged eye center velocity and the average velocity of the subset of facial landmarks can be summed. The averaged eye center and the average velocity of the subset of facial landmarks can be summed with a preselected set of weights. For example, the facial velocity can be weighted at 90% and the eye center velocity can be weighted at 10%. The stabilized eye features can be based on the original position of the eyes (e.g., landmarks 402, 414, 406, 410 and landmarks 404, 416, 408, 412) and the calculated average velocity.

[0089] In step S525, the 2D features are converted to 3D features, for example, triangulation techniques can be used to convert the 2D facial feature positions to 3D facial feature positions. In an example implementation, the positions and orientations (relative to another camera and / or display) of the cameras (e.g., cameras 114, 114′, 114″, 116, 118) and the 3D displays (e.g., displays 110, 112) are known (e.g., using calibration when setting up the 3D telepresence system). The X and Y coordinates in the 2D image space of each camera used to capture an image including facial features (e.g., landmarks) can be determined. A ray can be generated for each camera-landmark pair. For example, a ray can be drawn (e.g., using function calls in software) starting from the pixel location of the landmark (e.g., eye) toward each camera. If there are four cameras, four rays can be generated. The 3D position of the landmark can be determined based on these rays (e.g., the four rays). For example, the location where the rays intersect (or where the rays approach an intersection) may indicate the 3D position of the landmark (e.g., the left eye).

[0090] In step S530, the 3D features are filtered. For example, a double exponential filter can be used to reduce noise associated with the 3D landmarks, as described in more detail above. The double exponential filter with prediction 330 can also be configured (or used to predict) future positions of the 3D landmarks. Although the landmark noise was reduced using the 2D feature stabilizer 320, there may be residual noise associated with the 3D landmark locations (or positions). The residual noise may be amplified, for example, by environmental conditions or user aspects (e.g., glasses, facial hair, and / or other). The double exponential filter with prediction 330 can be configured to reduce this residual noise and predict future positions of the 3D landmarks.

[0091] The camera position, display position, and audio position can be used as data for the tracking process, such as current and / or future position values ​​of landmarks (e.g., eye positions, ear positions, and / or others) as binaries usable to drive a display, as input to a renderer for displaying the virtual scene (e.g., to determine the positions of the left eye scene and right eye scene), and for projecting audio (e.g., to determine the stereo balance between the left and right ear).

[0092] Thus, in step S535, a tracking process based on the positions of landmarks associated with the eyes is used to determine a virtual camera position. This tracking process may be based on predicted positions of the eyes. In step S540, a tracking process based on the positions of landmarks associated with the eyes is used to determine a display position. This tracking process may be based on predicted positions of the eyes. In step S545, a tracking process based on the positions of landmarks associated with the ears is used to determine an audio position. This tracking process may be based on predicted positions of the ears. The landmark positions may be 3D landmark positions that are double exponentially filtered and used to predict future landmark positions.

[0093] In a general aspect, a device, system, non-transitory computer-readable medium (storing computer-executable program code that can be executed on a computer system), and / or method can perform a process using a method that includes capturing at least one image using multiple cameras, identifying a plurality of facial features associated with the at least one image, stabilizing a position of at least one facial feature of the plurality of facial features, converting the position of the at least one facial feature to a three-dimensional (3D) position, predicting a future position of the at least one facial feature, and rendering a 3D image on a flat panel display using the future position of the at least one facial feature.

[0094] Implementations may include one or more of the following features. For example, the at least one image may include two or more images, and the two or more images may be captured at least one of sequentially by a same camera and simultaneously by two or more cameras. Identifying the facial features may include identifying a face associated with the at least one image, generating a plurality of landmarks on the face corresponding to the facial features, and associating a location with each of the plurality of landmarks. Identifying the facial features may include identifying a first face associated with the at least one image, identifying a second face associated with the at least one image, generating a first plurality of landmarks corresponding to the facial features on the first face, generating a second plurality of landmarks corresponding to the facial features on the second face, associating a first location with each of the first plurality of landmarks, and associating a second location with each of the second plurality of landmarks.

[0095] Stabilizing a position of the at least one facial feature may include selecting a facial feature from at least two of the at least one image, selecting at least two landmarks associated with the facial feature, and averaging positions of the at least two landmarks. To stabilize may include generating a plurality of landmarks corresponding to the plurality of facial features, selecting a subset of the plurality of landmarks, and determining a facial movement based on a velocity associated with a position of each landmark of the subset of the plurality of landmarks, and stabilizing a position of the at least one facial feature is based on the movement of the face. Converting the position of the at least one facial feature to a 3D position may include triangulating the position of the at least one facial feature based on position data generated using images captured by three or more of the plurality of cameras.

[0096] The method may further include applying a double exponential filter to the 3D position of the at least one facial feature to reduce noise associated with the 3D position of the at least one facial feature. Predicting a future position of the at least one facial feature may include applying a double exponential filter to the sequence of 3D positions of the at least one facial feature and selecting a value at a time greater than zero (0) based on the sequence of double exponential filtered 3D positions prior to time zero (0). The method may further include generating audio using the future position of the at least one facial feature.

[0097] In a general aspect, a device, system, non-transitory computer-readable medium (storing computer-executable program code that can be executed on a computer system), and / or method can perform a process using a device including a memory including code segments representing a plurality of computer instructions and a processor configured to execute the code segments, the computer instructions including capturing at least one image using a plurality of cameras, identifying a plurality of facial features associated with the at least one image, stabilizing a position of at least one facial feature of the plurality of facial features, converting the position of the at least one facial feature to a three-dimensional (3D) position, predicting a future position of the at least one facial feature, and rendering a 3D image on a flat panel display using the future position of the at least one facial feature.

[0098] Implementations may include one or more of the following features. For example, the at least one image may include two or more images, and the two or more images may be captured at least one of sequentially by a same camera and simultaneously by two or more cameras. Identifying the facial features may include identifying a face associated with the at least one image, generating a plurality of landmarks on the face corresponding to the facial features, and associating a location with each of the plurality of landmarks. Identifying the facial features may include identifying a first face associated with the at least one image, identifying a second face associated with the at least one image, generating a first plurality of landmarks corresponding to the facial features on the first face, generating a second plurality of landmarks corresponding to the facial features on the second face, associating a first location with each of the first plurality of landmarks, and associating a second location with each of the second plurality of landmarks.

[0099] Stabilizing a position of the at least one facial feature may include selecting a facial feature from at least two of the at least one image, selecting at least two landmarks associated with the facial feature, and averaging positions of the at least two landmarks. Stabilizing a position of the at least one facial feature may include generating a plurality of landmarks corresponding to the plurality of facial features, selecting a subset of the plurality of landmarks, and determining facial movement based on a velocity associated with a position of each landmark of the subset of the plurality of landmarks, where stabilizing a position of the at least one facial feature is based on the movement of the face. Converting the position of the at least one facial feature to a 3D position may include triangulating the position of the at least one facial feature based on position data generated with images captured by three or more of the plurality of cameras.

[0100] The computer instructions may further include applying a double exponential filter to the 3D position of the at least one facial feature to reduce noise associated with the 3D position of the at least one facial feature. Predicting a future position of the at least one facial feature may include applying a double exponential filter to the sequence of 3D positions of the at least one facial feature and selecting a value at a time greater than zero (0) based on the sequence of double exponential filtered 3D positions prior to time zero (0). The computer instructions may further include generating audio using the future position of the at least one facial feature.

[0101] 6 illustrates an example of a computing device 600 and a mobile computing device 650 that can be used with the techniques described herein. Computing device 600 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. Computing device 650 is intended to represent various forms of mobile devices, such as personal digital assistants, mobile phones, smartphones, and other similar computing devices. The components illustrated herein, their connections and relationships, and their functions are intended to be merely exemplary and are not intended to limit the implementation of the inventions described and / or claimed herein.

[0102] The computing device 600 includes a processor 602, a memory 604, a storage device 606, a high-speed interface 608 that connects to the memory 604 and a high-speed expansion port 610, and a low-speed interface 612 that connects to a low-speed bus 614 and the storage device 606. Each of the components 602, 604, 606, 608, 610, and 612 may be interconnected using various buses and mounted on a common motherboard or otherwise as appropriate. The processor 602 may process instructions that are executed within the computing device 600, including instructions stored in the memory 604 or on the storage device 606 for displaying graphical information for a GUI on an external input / output device, such as a display 616 coupled to the high-speed interface 608. In some implementations, multiple processors and / or multiple buses may be used as appropriate, along with multiple memories and types of memories. Also, multiple computing devices 600 may be connected, with each device providing a portion of the required operations (e.g., as a server bank, a group of blade servers, or a multiprocessor system).

[0103] The memory 604 stores information within the computing device 600. In one implementation, the memory 604 is one or more volatile memory units. In another implementation, the memory 604 is one or more non-volatile memory units. The memory 604 may also be another form of computer-readable medium, such as a magnetic disk or optical disk.

[0104] The storage device 606 can provide mass storage for the computing device 600. In one implementation, the storage device 606 may be or include a computer-readable medium such as a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or an array of devices including devices in a storage area network or other configuration. The computer program product may be tangibly embodied in an information carrier. The computer program product may also include instructions that, when executed, perform one or more methods as described above. The information carrier is a computer-readable or machine-readable medium such as the memory 604, the storage device 606, or a memory on the processor 602.

[0105] The high-speed controller 608 manages bandwidth-intensive operations for the computing device 600, while the low-speed controller 612 manages less bandwidth-intensive operations. Such allocation of functions is merely exemplary. In one implementation, the high-speed controller 608 is coupled to the memory 604, the display 616 (e.g., via a graphics processor or accelerator), as well as to a high-speed expansion port 610 that may accept various expansion cards (not shown). In this implementation, the low-speed controller 612 is coupled to the storage device 606 and to a low-speed expansion port 614. The low-speed expansion port, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), may be coupled to one or more input / output devices, such as a keyboard, pointing device, scanner, or a networking device, such as a switch or router, for example, via a network adapter.

[0106] Computing device 600 may be implemented in many different forms as shown in the figure. For example, it may be implemented as a standard server 620 or multiple times in a group of such servers. It may also be implemented as part of a rack server system 624. In addition, it may be implemented as a personal computer, such as a laptop computer 622. Alternatively, components from computing device 600 may be combined with other components in a mobile device (not shown), such as device 650. Each such device may include one or more of computing devices 600, 650, and the entire system may be made up of multiple computing devices 600, 650 in communication with each other.

[0107] Computing device 650 includes, among other components, a processor 652, a memory 664, input / output devices such as a display 654, a communication interface 666, and a transceiver 668. Device 650 may be provided with a storage device such as a microdrive or other device to provide additional storage. Each of components 650, 652, 664, 654, 666, and 668 are interconnected using various buses, and some of these components may be mounted on a common motherboard or in other manners as appropriate.

[0108] The processor 652 can execute instructions within the computing device 650, including instructions stored in the memory 664. The processor may be implemented as a chipset of chips including separate analog and digital processors. The processor may provide coordination for other components of the device 650, such as, for example, control of a user interface, applications executed by the device 650, and wireless communication by the device 650.

[0109] The processor 652 may communicate with a user through a control interface 658 and a display interface 656 coupled to a display 654. The display 654 may be, for example, a TFT LCD (thin film transistor liquid crystal display) or an OLED (organic light emitting diode) display, or other suitable display technology. The display interface 656 may include appropriate circuitry for driving the display 654 to display graphical and other information to the user. The control interface 658 may receive commands from the user and translate and send these commands to the processor 652. Additionally, an external interface 662 may be provided in communication with the processor 652 to enable near-area communication between the device 650 and other devices. The external interface 662 may, for example, provide for wired communication in some implementations and wireless communication in other implementations, and multiple interfaces may be used.

[0110] The memory 664 stores information within the computing device 650. The memory 664 may be realized as one or more of one or more computer readable media, one or more volatile memory units, or one or more non-volatile memory units. An expansion memory 674 may also be provided and connected to the device 650 via an expansion interface 672, which may include, for example, a SIMM (single in-line memory module) card interface. Such expansion memory 674 may provide additional storage space for the device 650 or may store applications or other information for the device 650. In particular, the expansion memory 674 may include instructions for performing or complementing the above processes, and may also include secure information. Thus, the expansion memory 674 may be provided, for example, as a security module for the device 650 and may be programmed with instructions that enable secure use of the device 650. In addition, secure applications may be provided via a SIMM card along with additional information, such as by placing identifying information on the SIMM card in an unhackable manner.

[0111] The memory may include, for example, flash memory and / or NVRAM memory, as described below. In one implementation, a computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable or machine-readable medium, such as memory 664, expansion memory 674, or memory on processor 652, which may be received, for example, via transceiver 668 or external interface 662.

[0112] Device 650 may communicate wirelessly via communication interface 666, which may include digital signal processing circuitry as necessary. Communication interface 666 may provide communications under various modes or protocols, such as GSM voice telephony, SMS, EMS or MMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000, or GPRS, among others. Such communications may occur, for example, via radio frequency transceiver 668. In addition, short-range communications may occur, such as using Bluetooth, Wi-Fi, or other such transceivers (not shown). In addition, a GPS (Global Positioning System) receiver module 670 may provide additional wireless data related to navigation and location to device 650, which may be used as appropriate by applications executing on device 650.

[0113] Device 650 may also communicate audibly using a voice codec 660, which may receive spoken information from a user and convert it into usable digital information. Voice codec 660 may also generate audible sounds for the user, such as through a speaker in a handset of device 650. Such sounds may include sounds from a voice telephone call, recorded sounds (e.g., voice messages, music files, etc.), and may also include sounds generated by applications running on device 650.

[0114] The computing device 650 may be implemented in many different forms, as shown in the figure, for example as a mobile phone 680, or as part of a smartphone 682, personal digital assistant, or other similar mobile device.

[0115] While exemplary embodiments may include various modifications and alternative forms, embodiments thereof are shown by way of example in the drawings and will be described in detail herein. It is to be understood, however, that there is no intention to limit the exemplary embodiments to the particular forms disclosed, but rather, the exemplary embodiments cover all modifications, equivalents, and alternatives falling within the scope of the claims. Like numbers refer to like elements throughout the description of the drawings.

[0116] Various implementations of the systems and techniques described herein may be realized in digital electronic circuitry, integrated circuits, specially designed ASICs (Application Specific Integrated Circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs executable and / or interpretable on a programmable system, which may include at least one programmable processor, at least one input device, and at least one output device, which may be a dedicated or general purpose processor coupled to transmit and receive data and instructions from a storage system. Various implementations of the systems and techniques described herein may be realized and / or generally referred to herein as circuits, modules, blocks, or systems that may combine software and hardware aspects. For example, a module may include functions / acts / computer program instructions executed on a processor (e.g., a processor formed on a silicon substrate, GaAs substrate, etc.) or other programmable data processing apparatus.

[0117] Some of the above exemplary embodiments are described as a process or method that is depicted as a flowchart. Although the flowchart describes operations as a sequential process, many of the operations may be performed in parallel, simultaneously, or concurrently. Additionally, the order of operations may be rearranged. A process may terminate when its operations are completed, but may have additional steps not included in the drawing. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc.

[0118] The above-described methods, some of which are illustrated by flowcharts, may be implemented by hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. If implemented as software, firmware, middleware, or microcode, the program code or code segments to perform the necessary tasks may be stored in a machine- or computer-readable medium, such as a storage medium. A processor(s) may perform the necessary tasks.

[0119] Specific structural and functional details disclosed herein are merely representative for purposes of describing example embodiments, however, example embodiments may be embodied in many alternative forms and should not be construed as being limited to only the embodiments set forth herein.

[0120] Terms such as "first", "second", etc. may be used herein to describe various elements, but it will be understood that these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first element can be referred to as a second element, and similarly, a second element can be referred to as a first element, without departing from the scope of the exemplary embodiments. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0121] When an element is referred to as being connected or coupled to another element, it will be understood that the element may be directly connected or coupled to the other element, or there may be intervening elements. In contrast, when an element is referred to as being directly connected or directly coupled to another element, there are no intervening elements. Other words used to describe the relationship between elements should be construed in a similar manner (e.g., "between" and "directly between," "adjacent" and "directly adjacent," etc.).

[0122] The terminology used herein is merely for the purpose of describing certain embodiments and is not intended to limit the exemplary embodiments. As used herein, the singular forms "a", "an" and "the" are intended to include the plural, unless the context clearly dictates otherwise. It will be further understood that, as used herein, the terms "comprises", "comprising" and / or "includes", "including" specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0123] Also, in some alternative implementations, the functions / acts noted may occur out of the order noted in the figures. For example, two figures shown in succession may in fact be executed concurrently or in the reverse order, depending on the functions / acts involved.

[0124] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art to which the exemplary embodiments belong. Furthermore, for example, terms defined in commonly used dictionaries should be interpreted to have a meaning consistent with their meaning in the context of the relevant art, and will not be interpreted in an idealized or overly formal sense unless expressly defined herein.

[0125] Portions of the above exemplary embodiments and corresponding detailed description have been presented in terms of software or algorithms and symbolic representations of operations on data bits within a computer memory. These descriptions and representations are the ones intended to enable those skilled in the art to effectively convey the substance of their work to others skilled in the art. An algorithm, as the term is used here, and as it is used generally, is conceived to be a self-consistent sequence of steps leading to a desired result. The steps are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of optical, electrical, or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. It has proven convenient at times, primarily for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like.

[0126] In the above illustrative embodiments, references to symbolic representations of acts and operations that may be implemented as program modules or functional processes (e.g., in the form of flowcharts) include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types and that may be written and / or implemented using existing hardware in existing structural elements. Such existing hardware may include one or more central processing units (CPUs), digital signal processors (DSPs), application specific integrated circuits, or field programmable gate array (FPGA) computers, etc.

[0127] It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless otherwise indicated, or as will be apparent from the description, the terms processing or computing or calculation or determination of display refer to the actions and processes of a computer system or similar electronic computing device that manipulates data represented as physical electronic quantities in the computer system's registers and memory and converts that data into other data that are similarly represented as physical quantities in the computer system's memory, or registers, or other such information storage, transmission or display devices.

[0128] Also, the software-implemented aspects of the exemplary embodiments are typically encoded on some form of non-transitory program storage medium or implemented on some type of transmission medium. The program storage medium may be magnetic (e.g., a floppy disk or a hard drive) or optical (e.g., a compact disk read only memory or CD ROM) and may be read-only or random access. Similarly, the transmission medium may be twisted wire pairs, coaxial cable, optical fiber, or some other suitable transmission medium known in the art. The exemplary embodiments are not limited by these aspects of a given implementation.

[0129] Finally, although the appended claims recite particular combinations of features described herein, the scope of the disclosure is not limited to the specific combinations claimed below, but instead is extended to encompass any combination of features or embodiments disclosed herein, regardless of whether that particular combination is currently specifically recited in the appended claims.

Claims

1. capturing at least one image using a plurality of cameras; identifying a plurality of facial features associated with the at least one image; stabilizing a position of at least one facial feature of the plurality of facial features based on at least two landmarks associated with the at least one facial feature; converting the position of the at least one facial feature into a three-dimensional (3D) position; predicting a future position of the at least one facial feature; and rendering a 3D image on a flat panel display using the future position of the at least one facial feature based on the at least two landmarks.

2. the at least one image includes two or more images; The method of claim 1 , wherein the two or more images are captured at least one of sequentially by a same camera and simultaneously by two or more cameras.

3. Identifying the plurality of facial features includes: identifying a face associated with the at least one image; generating a plurality of landmarks on the face corresponding to the plurality of facial features; and associating a location with each of the plurality of landmarks.

4. Identifying the plurality of facial features includes: identifying a first face associated with the at least one image; identifying a second face associated with the at least one image; and generating a first plurality of landmarks corresponding to the plurality of facial features on the first face; generating a second plurality of landmarks corresponding to the facial features on the second face; and Associating a first location with each of the first plurality of landmarks; and associating a second location with each of the second plurality of landmarks.

5. Stabilizing a position of the at least one facial feature includes: selecting facial features from at least two of the at least one image; selecting the at least two landmarks associated with the facial feature; and Averaging the positions of the at least two landmarks.

6. Stabilizing a position of the at least one facial feature includes: generating a plurality of landmarks corresponding to the plurality of facial features; selecting a subset of the plurality of landmarks; determining facial movement based on a velocity associated with a position of each landmark of the subset of the plurality of landmarks; The method of any one of claims 1 to 5, wherein stabilizing the position of the at least one facial feature is based on the movement of the face.

7. Transforming the position of the at least one facial feature into a 3D position includes:

7. The method of claim 1, further comprising triangulating the position of the at least one facial feature based on position data generated using images captured by three or more of the plurality of cameras.

8. The method of claim 7 , further comprising applying a double exponential filter to the 3D position of the at least one facial feature to reduce noise associated with the 3D position of the at least one facial feature.

9. Predicting a future position of the at least one facial feature includes: applying a double exponential filter to the sequence of 3D positions of the at least one facial feature; and selecting a value at time greater than zero (0) based on the sequence of 3D positions filtered with the double exponential filter before time zero (0).

10. Determining an audio location; The method of any one of claims 1 to 9, further comprising: generating audio using the future position of the at least one facial feature and the audio position.

11. 1. A three-dimensional (3D) content system, comprising: a memory containing a code segment representing a plurality of computer instructions; and one or more processors configured to execute the code segments, the computer instructions being executed by the one or more processors to cause the 3D content system to perform the method of any one of claims 1 to 10.

12. A program that causes one or more processors of a system to execute the method according to any one of claims 1 to 10.

Images