Neural blending for novel view synthesis

Abstract

Systems and methods for receiving a plurality of input images, a plurality of depth images, and a plurality of view parameters to generate a virtual view of a target subject are described. The systems and methods can generate a plurality of warped images based on at least one of the plurality of input images, the plurality of view parameters, and the plurality of depth images. In response to providing the plurality of depth images, the plurality of view parameters, and the plurality of warped images to a neural network, the systems and methods can receive, from the neural network, blending weights for assigning colors to pixels of the virtual view of the target subject, and can generate a composite image from the view parameters based on the blending weights and the virtual view.

Description

Technical Field

[0001] This specification generally relates to the methods, apparatus and algorithms used in compositing three-dimensional (3D) content. Background Technology

[0002] Conventional object rendering typically involves a computationally intensive workload to generate realistic images. If the object is in motion, additional computational effort can be used to generate a realistic image of the object. This rendering can include using neural networks to model the appearance of the object. However, the model can generate images with foreign noise and geometric artifacts. Summary of the Invention

[0003] The systems and methods described herein can perform image-based rendering using input images and predefined view parameters to generate (e.g., synthesize) novel (e.g., unseen) views of video and / or images based on the input images. Image-based rendering of unseen views can utilize warping processes for the received input images. In general, warping processes can cause geometric inaccuracies and view- and / or image-related effects, which can produce artifacts when contributions from different input views are blended together. The systems and methods described herein employ deep learning techniques that use neural networks (NNs) to blend image content to obtain image-based rendering of novel views. Specific blending weights are learned and used to combine input image contributions into a final synthetic view. The blending weights are generated to provide the advantage of generating synthetic images exhibiting reduced view- and / or image-related effects and a reduced number of image artifacts.

[0004] A potential technical challenge when using neural networks (NNs), warping processes, and / or blending weights is the lack of sufficiently accurate geometry that allows the NN (e.g., a convolutional neural network) to select appropriate blending weights to avoid image artifacts. The systems and methods described in this paper address this challenge by using learned blending of color and depth views of the input image and / or employing multi-resolution blending techniques to select pixel colors that provide an accurate image with reduced image artifacts. For example, blending weights can be applied to pixel colors that are likely to be correctly and accurately projected relative to the ground truth image (e.g., probabilistically provided), while reducing the weights on pixel colors that are unlikely to be correctly and accurately projected for a given ground truth image.

[0005] To employ such hybrid techniques, the systems and methods described herein can utilize one or more witness cameras in addition to specific airborne system cameras (e.g., color cameras, infrared cameras, etc.). The witness cameras can supervise the content used to generate the novel view. For example, a witness camera can be a high-resolution camera capable of providing ground-based data. The generated novel view is compared with ground-based data received from (e.g., acquired by) the witness cameras. In some implementations, the image detail of the novel view can be scored based on the image detail captured by the witness cameras during the generation of the novel view.

[0006] In some implementations, the systems and methods described in this paper take into account training loss. For example, the system can generate training data with various capture scenarios to minimize the loss in order to provide high-quality novel view synthesis while reducing temporal flicker artifacts in the synthesized view. In some implementations, the systems and methods described in this paper can also employ occlusion inference to correct artifacts in the synthesized novel view.

[0007] By installing software, firmware, hardware, or a combination thereof on a system, one or more computers can be configured to perform specific operations or actions, which, individually or collectively, cause the system to perform said actions during operation. One or more computer programs can be configured to perform specific operations or actions by including instructions that, when executed by a data processing device, cause said device to perform said actions.

[0008] In one general aspect, a system and method are described for receiving a plurality of input images, receiving a plurality of depth images associated with a target subject in at least one of the plurality of input images, receiving a plurality of view parameters for generating a virtual view of the target subject, and generating a plurality of warped images based on the plurality of input images, the plurality of view parameters, and at least one of the plurality of depth images. In response to providing the plurality of depth images, the plurality of view parameters, and the plurality of warped images to a neural network, the system and method can receive from the neural network a blending weight for assigning color to pixels of the virtual view of the target subject. The system and method can generate a synthetic image based on the blending weight and the virtual view according to the view parameters.

[0009] These and other aspects can individually or in combination include one or more of the following. In some implementations, the system and method may include using a geometric fusion process on multiple depth images to reconstruct a desired surface to generate a geometric fusion model, generating multiple reprojected images based on multiple input images and the desired surface, and in response to providing a neural network with multiple depth images, multiple view parameters, and multiple reprojected images, the system and method may receive additional blending weights from the neural network for assigning colors to pixels in the synthesized image.

[0010] In some implementations, the system and method may further include providing the neural network with a depth difference between a geometric fusion model and depths observed in multiple depth images, and the method further includes correcting for occlusions detected in the synthetic image based on the depth difference. In some implementations, the multiple input images are color images captured according to predefined view parameters associated with at least one camera capturing the multiple input images, and / or each of the multiple depth images includes a depth map associated with at least one camera capturing at least one of the multiple input images, at least one occlusion map, and / or a depth map associated with a ground-based image captured by at least one witness camera at a time corresponding to the capture of at least one of the multiple input images. In some implementations, the blending weights are configured to assign a blending color to each pixel of the synthetic image.

[0011] In some implementations, the neural network is trained based on minimizing an occlusion loss function between a synthetic image generated by the neural network and a live ground image captured by at least one witness camera. In other implementations, the synthetic image is an uncaptured view of the target subject generated for a 3D video conference.

[0012] In some implementations, generating multiple warped images based on at least one of multiple input images, multiple view parameters, and multiple depth images includes using at least one of the multiple depth images to determine candidate projections of colors associated with the multiple input images into an uncaptured view, wherein the uncaptured view includes at least a portion of image features of at least one of the multiple input images.

[0013] In another general aspect, an image processing system is described, particularly for performing the method as described in any one of the preceding claims. The image processing system may include at least one processing device and a memory storing instructions that, when executed, cause the system to perform operations including: receiving a plurality of input images captured by the image processing system; receiving a plurality of depth images captured by the image processing system; receiving a plurality of view parameters associated with an uncaptured view that is also associated with at least one of the plurality of input images; and generating a plurality of warped images based on at least one of the plurality of input images, the plurality of view parameters, and the plurality of depth images. In response to providing the plurality of depth images, the plurality of view parameters, and the plurality of warped images to a neural network, the system may include receiving from the neural network mixing weights for assigning colors to pixels of the uncaptured view. The system may further include generating a synthetic image based on the mixing weights, wherein the synthetic image corresponds to the uncaptured view.

[0014] These and other aspects can individually or in combination include one or more of the following. In some implementations, the multiple input images are color images captured by an image processing system according to predefined view parameters associated with the image processing system and / or the multiple depth images include a depth map associated with at least one camera that has captured at least one of the multiple input images, at least one occlusion map, and / or a depth map associated with a witness camera of the image processing system.

[0015] In some implementations, the blending weights are configured to assign a blending color to each pixel of the synthesized image. In some implementations, the neural network is trained based on minimizing an occlusion loss function between the synthesized image generated by the neural network and a live ground image captured by at least one eyewitness camera. In some implementations, the synthesized image is a novel view generated for 3D video conferencing.

[0016] In another general aspect, a non-transitory machine-readable medium is described as having instructions stored thereon that, when executed by a processor, cause a computing device to receive a plurality of input images, a plurality of depth images associated with a target subject in at least one of the plurality of input images, and a plurality of view parameters for generating a virtual view of the target subject. The machine-readable medium can also be configured to use a geometric fusion process on the plurality of depth images to reconstruct a desired surface to generate a geometric fusion model of the target subject, and to generate a plurality of reprojected images based on the plurality of input images, the plurality of view parameters, and the desired surface. In response to providing the plurality of depth images, the plurality of view parameters, and the plurality of reprojected images to a neural network, the machine-readable medium can receive from the neural network mixing weights for assigning color to pixels of the virtual view of the target subject, and generate a synthetic image based on the mixing weights and the virtual view according to the view parameters.

[0017] These and other aspects can individually or in combination include one or more of the following. In some implementations, the machine-readable medium further includes providing the neural network with a depth difference between a geometric fusion model and depths observed in multiple depth images, and correcting for occlusions detected in the synthetic image based on the depth difference. In some implementations, the multiple input images are color images captured according to predefined view parameters associated with at least one camera that captures the multiple input images, and / or the multiple depth images include a depth map associated with at least one camera that captures at least one of the multiple input images, at least one occlusion map, and / or a depth map associated with a ground-based image captured by at least one witness camera at a time corresponding to the capture of at least one of the multiple input images.

[0018] In some implementations, the blending weights are configured to assign a blending color to each pixel of the synthesized image. In some implementations, the neural network is trained based on minimizing an occlusion loss function between the synthesized image generated by the neural network and a ground-based live image captured by at least one witness camera. In some implementations, the synthesized image is a novel view used for 3D video conferencing. In some implementations, the neural network is further configured to perform multi-resolution blending to assign pixel colors to pixels in the synthesized image, the multi-resolution blending triggering receiving an image pyramid as input to the neural network to trigger the reception of multi-resolution blending weights for multiple scales and opacity values ​​associated with each scale.

[0019] These and other aspects may include, individually or in combination, one or more of the following aspects. Depending on some aspects, the methods, systems, and computer-readable media claimed herein may include one or more (e.g., all) of the following features (or any combination thereof).

[0020] Implementations of the technology may include hardware, methods or processes, or computer software on a computer-accessible medium. Details of one or more embodiments are given in the following description and drawings. Other features will become apparent from the description, drawings, and claims. Attached Figure Description

[0021] Figure 1 This is a block diagram illustrating an example 3D content system for displaying composite content on a display device according to embodiments described throughout this disclosure.

[0022] Figure 2 This is a block diagram of an exemplary system for compositing content for rendering on a display, according to embodiments described throughout this disclosure.

[0023] Figure 3 This is a block diagram illustrating an example of reprojecting an input image onto a target camera viewpoint according to an embodiment described throughout this disclosure.

[0024] Figure 4 This is a block diagram of an example flowchart for generating synthetic content for rendering on a display, based on embodiments described throughout this disclosure, using neural hybridization techniques.

[0025] Figure 5 It is a block diagram of an example flowchart for generating hybrid weights according to embodiments described throughout this disclosure.

[0026] Figure 6 This is a flowchart illustrating an example of a process for generating synthetic content using neural hybrid techniques according to the implementation described throughout this disclosure.

[0027] Figure 7 Examples of computer devices and mobile computer devices that can be used with the techniques described herein are shown.

[0028] The same reference numerals in different figures indicate the same elements. Detailed Implementation

[0029] This document describes examples related to generating novel (e.g., unseen) views of image content. The examples described herein can synthesize (e.g., generate) real-time novel views based on captured video content and / or image content. For example, image-based rendering techniques can be used to synthesize novel views of moving image content (e.g., objects, users, scene content, image frames, etc.) using a learned blend of color and depth views.

[0030] The systems and methods described in this paper can generate novel color images with fewer artifacts compared to conventional systems. For example, the systems and methods described in this paper can correct specific image noise and loss function analysis to generate novel images with less depth inaccuracy and less occlusion. Correction can be performed by employing a neural network (NN) to learn to detect and correct image regions containing visibility errors. Furthermore, the NN is capable of using a mixture algorithm to learn and predict color values ​​for novel views, which constrains the output values ​​to a linear combination of reprojected input colors retrieved from the color input image.

[0031] In operation, the process can retrieve (e.g., capture, acquire, receive, etc.) multiple input images and data (e.g., target view parameters) to predict a novel view (e.g., an unseen color image) by combining a stream of color images (e.g., views) from the same scene (e.g., image content within the scene). The color image stream can be fed to the neural network (NN) to employ neural rendering techniques to enhance low-quality output from a real-time image capture system (e.g., a 3D video conferencing system such as a telepresence system). For example, the novel view could be a predicted color image generated by the systems and techniques described herein. A predicted image can be generated by feeding an input image and the combined color image stream (e.g., and / or a reprojection or representation of such input images) to the NN, allowing the NN to learn specific mixing weights to assign pixel colors to the predicted color image. The learned mixing weights can be applied to generate pixel colors for the novel color image. The learned mixing weights can also be used to generate additional novel views of image content represented in one or more provided input images.

[0032] In some implementations, the NN described herein can model view-related effects to predict future user movement (e.g., motion) in order to mitigate erroneous projections of artifacts caused by the noise nature of the specific geometric information used to generate the user's image and / or the geometric information received from the camera that captures the user and / or the information received from image processing performed on the user's image.

[0033] In some implementations, the systems and methods described herein are capable of training one or more neural networks (e.g., convolutional neural networks, such as U-net) to predict images in the viewpoint of a single eyewitness camera, which can be used, for example, to provide supervision of the output color image. The eyewitness camera can act as a ground-based live camera for the image capture and / or processing system described herein. In some implementations, two or more eyewitness cameras can be used as training data for the neural network. Two or more eyewitness cameras can represent a pair or many pairs of eyewitness cameras.

[0034] In some implementations, the system and method may utilize captured input images, predefined parameters associated with the desired novel output view, and / or an occlusion map containing depth differences and a depth map. Depth differences can be generated using views from a color camera between the surface closest to the novel view and the surface of the camera view. Depth differences can be used for occlusion inference to correct occluded views and / or other errors in the generated images. In some implementations, the depth map may include a depth map from a view captured by a witness camera.

[0035] In some implementations, the systems and methods described herein can reconstruct a consensus surface (e.g., a geometric surface) through geometric fusion of an input depth image. In some implementations, the systems and methods described herein can use depth information, such as separately captured depth images and / or consensus surfaces, to determine the projection of an input color into a novel view.

[0036] In some implementations, the systems and methods described herein can generate a color image (e.g., a color image) for the novel view by assigning a blend color to each pixel in the novel view. The blend color can be determined using the color input image and blend weights determined by the neural network described herein. In some implementations, the blend weights are regularized using a loss function. In some implementations, the novel view is a weighted combination of one or more pixel color values ​​from the original input image projected onto the image in the novel view.

[0037] As used herein, a novel (e.g., unseen) view may include image and / or video content that has been interpreted (e.g., synthesized, interpolated, simulated, etc.) based on one or more frames of camera-captured image and / or video content. For example, the interpretation of camera-captured image and / or video content may be used in conjunction with the techniques described herein to create unseen versions and views of the captured image and / or video content (e.g., poses, expressions, angles, etc.).

[0038] In some implementations, for example, the techniques described herein can be used to synthesize images that are accurately and realistically represented for display on the screen of a 2D or 3D display used in multi-channel 2D or 3D video (e.g., telephone) conferences. The techniques described herein can be used to generate and display accurate and realistic views (e.g., image content, video content) of users during video conferences. Views include previously unseen views that are conventionally difficult to depict in 3D without significant image artifacts.

[0039] The system and method described in this paper offer the advantage of generating novel views without significant image artifacts by learning blending weights based on multi-view color input images and noisy occlusion cues using one or more witness cameras and a neural network. The learned blending weights ensure that occlusion and color artifacts are corrected in the resulting output image. Furthermore, the learned blending weights and one or more witness cameras can be used by the system described in this paper to ensure that image content not captured in the input image can be used to accurately predict novel views associated with image content in the input image. For example, because the blending weights are learned and evaluated relative to witness camera images, accurate predictions can be made for image portions of the scene that were not captured or represented in the original input image.

[0040] In some implementations, the techniques described herein can be used for entertainment purposes in films, videos, short films, game content, virtual or augmented reality content, or other formats including user images that may benefit from the prediction techniques described herein. For example, the techniques described herein can be used to generate novel views of moving characters rendered in image and / or video content.

[0041] In some implementations, the techniques described herein may be used by virtual assistant devices or other intelligent agents that can perform image processing to identify objects, recreate objects, and / or generate synthetic images based on those objects using the techniques described herein.

[0042] Figure 1This is a block diagram illustrating an example 3D content system 100 for displaying content in a stereoscopic display device according to embodiments described throughout this disclosure. For example, the 3D content system 100 can be used by multiple users to perform video conferencing communication in 3D (e.g., telepresence sessions) and / or access augmented and / or reality content. Generally, Figure 1 The system can be used to capture video and / or images and / or scenes of a user during 2D or 3D video conferencing, and uses the systems and techniques described herein to generate novel views based on the captured content, facilitating the rendering of accurate images depicting the novel views within the video conferencing session. System 100 can benefit from the use of the techniques described herein, as such techniques can, for example, generate and display a real-time novel view within a video conference, accurately representing the user. For example, the novel view can be provided to another user's display in 2D and / or 3D via system 100.

[0043] like Figure 1 As shown, a first user 102 and a second user 104 access a 3D content system 100. For example, users 102 and 104 can access the 3D content system 100 to participate in a 3D telepresence session. In this example, the 3D content system 100 can allow each of users 102 and 104 to see a highly realistic and visually consistent representation of each other, thereby facilitating interaction between users in a manner similar to their physical presence.

[0044] Each user 102, 104 can use the corresponding 3D system for a 3D telepresence session. Here, user 102 accesses 3D system 106, while user 104 accesses 3D system 108. 3D systems 106, 108 can provide functions related to 3D content, including but not limited to capturing images for 3D display, processing and presenting image information, and processing and presenting audio information. 3D systems 106 and / or 3D system 108 can constitute a collection of sensing devices integrated into a single unit. 3D systems 106 and / or 3D system 108 may include reference... Figure 2 And some or all of the components shown in Figure 8.

[0045] 3D content system 100 may include one or more 2D or 3D displays. Here, 3D display 110 is depicted for 3D system 106, and 3D display 112 is depicted for 3D system 108. 3D displays 110, 112 may use any of a variety of 3D display technologies to provide a stereoscopic view to a corresponding viewer (e.g., user 102 or user 104). In some embodiments, 3D displays 110, 112 may be stand-alone units (e.g., self-supporting or wall-mounted). In some embodiments, 3D displays 110, 112 may include or have access to wearable technologies (e.g., controllers, AR glasses, etc.). In some embodiments, displays 110, 112 may be 2D displays.

[0046] Generally, displays 110 and 112 can provide images that approximate the 3D optical properties of physical objects in the real world without using a head-mounted display (HMD) device. Generally, displays described herein may include flat panel displays covered with biconvex lenses (e.g., microlens arrays) and / or parallax barriers to redirect images to multiple different viewing areas associated with the display.

[0047] In some embodiments, displays 110, 112 may include high-resolution, glasses-free biconvex lens 3D displays. For example, displays 110, 112 may include microlens arrays (not shown) comprising a plurality of lenses (e.g., microlenses) having glass spacers coupled (e.g., joined) to the displays. The microlenses may be designed such that, from a selected viewing position, the user's left eye can view a first set of pixels, while the user's right eye can view a second set of pixels (e.g., where the second set of pixels is independent of the first set of pixels).

[0048] In some exemplary displays, there may be a single location providing a 3D view of image content (e.g., a user, an object, etc.) provided by such a display. A user can sit in this single location to experience appropriate parallax, minimal distortion, and realistic 3D images. If the user moves to a different physical location (or changes head position or eye gaze position), the image content (e.g., the user, objects worn by the user, and / or other objects) may begin to appear less realistic, 2D, and / or distorted. The systems and techniques described herein can reconfigure the image content projected from the display to ensure that the user can move around but still experience appropriate parallax, low distortion, and realistic 3D images in real time. Therefore, the systems and techniques described herein offer the advantage of maintaining and providing 3D image content and objects for display to a user regardless of any user movement that occurs while the user is viewing the 3D display.

[0049] like Figure 1As shown, the 3D content system 100 can be connected to one or more networks. Here, network 114 is connected to 3D systems 106 and 108. Network 114 can be a publicly available network (such as the Internet) or a private network, merely as two examples. Network 114 can be wired, wireless, or a combination of both. Network 114 may include or utilize one or more other devices or systems, including but not limited to one or more servers (not shown).

[0050] 3D systems 106 and 108 may include multiple components related to the capture, processing, transmission, or reception of 3D information and / or the presentation of 3D content. 3D systems 106 and 108 may include one or more cameras for capturing image content and / or video (e.g., visible and IR image data) of images to be included in the 3D presentation. In the depicted example, 3D system 106 includes cameras 116 and 118. For example, cameras 116 and / or 118 may be substantially housed within the housing of 3D system 106 such that the objectives or lenses of the respective cameras 116 and / or 118 capture image content through one or more openings in the housing. In some embodiments, cameras 116 and / or 118 may be separable from the housing, such as as stand-alone devices (e.g., having wired and / or wireless connections to 3D system 106). Cameras 116 and 118 may be positioned and / or oriented to capture a sufficiently representative view of a user (e.g., user 102).

[0051] While cameras 116 and 118 will generally not obstruct the view of user 102's 3D display 110, their placement can be arbitrary. For example, one of cameras 116 and 118 can be positioned above user 102's face, and the other below. Alternatively, one of cameras 116 and 118 can be positioned to the right of user 102's face, and the other to the left. Similarly, 3D system 108 can include cameras 120 and 122 in a similar manner. Additional cameras may also be included. For example, a third camera can be placed near or behind display 110.

[0052] In some implementations, 3D systems 106, 108 may include one or more witness cameras 119, 121. Witness cameras 119, 121 may be used to capture high-quality images (e.g., witness camera image 132) that may represent ground-based images. Images captured by witness cameras 119 and / or 121 may be used in conjunction with the techniques described herein to serve as comparisons when generating novel views and calculating losses and corrections for such losses. Generally, images captured by witness cameras 119, 121 may be captured at a substantially similar time to other images (e.g., frames) captured by cameras 116, 118, 120, 122, 124 and / or 126, and combinations of such cameras and / or camera pods. In some implementations, witness camera image 134 may be captured and used as training data for one or more neural networks to generate novel views.

[0053] In some implementations, 3D systems 106, 108 may include one or more depth sensors to capture depth data to be used in 3D rendering. Such depth sensors can be considered as part of a depth capture component in the 3D content system 100 to characterize the scene captured by 3D systems 106 and / or 108, thereby correctly representing the scene on a 3D display. Furthermore, the system can track the position and orientation of the viewer's head, enabling the rendering of the 3D presentation to correspond to the viewer's current perspective. Here, 3D system 106 includes a depth sensor 124, which may also represent an infrared camera. Similarly, 3D system 108 may include a depth sensor 126. Any of several types of depth sensing or depth capture can be used to generate depth data.

[0054] In some implementations, each camera 116, 118, 119, and 124 may represent multiple cameras within a bay. For example, depth sensor 124 may be housed in a camera bay along with camera 116 and / or camera 118. In some implementations, three or more camera bays may be placed around and / or behind display 110, and each bay may include camera 124 (e.g., depth sensor / camera) and one or more cameras 116, 118. Similarly, three or more camera bays may be placed around and / or behind display 112, and each bay may include camera 126 (e.g., depth sensor / camera) and one or more cameras 120, 122.

[0055] In the operation of system 106, assisted stereo depth capture can be performed. For example, a scene can be illuminated using a spotlight, and stereo matching can be performed between two corresponding cameras. This illumination can be accomplished using waves of a selected wavelength or wavelength range. For example, infrared (IR) light can be used. For example, depth data can include or be based on any scene-related information reflecting the distance between a depth sensor (e.g., depth sensor 124) and objects in the scene. For the content in an image corresponding to an object in the scene, the depth data reflects the distance (or depth) to said object. For example, the spatial relationship between the camera and the depth sensor can be known and can be used to correlate images from the camera with signals from the depth sensor to generate depth data for the image.

[0056] It can process images captured by the 3D content system 100 and then display them as a 3D rendering. For example... Figure 1 As shown in the example, a 3D image of user 104 is presented on a 3D display 110. Thus, user 102 can perceive (e.g., user's) 3D image 104' as a 3D representation of user 104, even when user 104 is away from user 102. Similarly, a 3D image 102' is presented on a 3D display 112. Thus, user 104 can perceive the 3D image 102' as a 3D representation of user 102.

[0057] The 3D content system 100 allows participants (e.g., users 102, 104) to engage in audio communication with each other and / or with others. In some embodiments, the 3D system 106 includes a speaker and a microphone (not shown). For example, the 3D system 108 may similarly include a speaker and a microphone. Thus, the 3D content system 100 allows users 102 and 104 to participate in 3D telepresence sessions with each other and / or with others. Generally, the systems and techniques described herein can operate with system 100 to generate image content and / or video content for display among users of system 100.

[0058] In the operation of system 100, a set of input images 132 may be captured by cameras 116, 118, 119, 124 and / or 120, 121, 122 and 126. For example, the input images may include eyewitness camera image 134 and RGB color image 136. In some implementations, system 100 may also generate and / or otherwise obtain a depth image 138. In one example, the depth image 138 may be generated by performing one or more stereo calculations based on a pair of IR images retrieved from an IR camera, as described above. The input images 132 may be used as the basis for predicting an output image that is a linear combination of reprojected colors from the input images. In some implementations, the input images 132 may include two or more color images representing reprojected color images (e.g., red, green, blue (RGB)) captured according to known (e.g., predetermined, predefined) view parameters. In some implementations, the input images 132 may also include one or more depth images 138 calculated (e.g., generated) according to known view parameters. The input image 132 can be used in combination with specific camera parameters, view parameters, and / or NN mixing algorithm 140 to generate novel views for display on displays 110 and / or 112.

[0059] Figure 2 This is a block diagram of an example system for compositing content for rendering on a display, based on embodiments described throughout this disclosure. System 200 may be used as or included in one or more embodiments described herein, and / or may be used to perform one or more examples of operations for compositing, processing, simulating, or presenting the image content described herein. The entire system 200 and / or one or more of its various components may be implemented according to one or more examples described herein.

[0060] System 200 may include one or more 3D systems 202. In the depicted example, 3D systems 202A, 202B through 202N are shown, where index N indicates an arbitrary number. 3D system 202 can provide the capture of visual and audio information for 2D or 3D rendering and can forward 2D or 3D information for processing. Such information may include images of the scene, depth data about the scene, parameters associated with the image capture, and / or audio from the scene. 2D / 3D system 202 can be used as systems 106 and 108 and 2D / 3D displays 110 and 112 (…). Figure 1 ), or be included in. Although systems 202B and 202N do not depict the same modules as those depicted in system 202A, each module in system 202A may also exist in systems 202B and 202N.

[0061] System 200 may include multiple cameras, as shown in camera 204. Any type of light sensing technology can be used to capture images, such as the image sensor type used in ordinary digital cameras. Camera 204 may be of the same type or different types. For example, the camera position may be placed anywhere on a 3D system (e.g., system 106). In some implementations, each system 202A, 202B, and 202N includes three or more camera bays, each including a depth camera (e.g., a pair or more pairs of IR cameras, such as depth sensor 206 and / or its contents, analyzed using stereo algorithms to infer depth images) and one or more color cameras. In some implementations, systems 202A, 202B, and 202N also include one or more witness cameras (not shown) that can capture images to be used as ground reality images when generating novel views and / or, for example, for training neural networks.

[0062] System 202A includes a depth sensor 206. In some embodiments, the depth sensor 206 operates by propagating an IR signal onto the scene and detecting a response signal. For example, the depth sensor 206 may generate and / or detect beams 128A and / or 128B and / or 130A and / or 130B. In some embodiments, the depth sensor 206 may be used to calculate an occlusion map. System 202A also includes at least one microphone 208 and a speaker 210. In some embodiments, the microphone 208 and speaker 210 may be part of system 106.

[0063] In addition, system 202 includes a 3D display 212 capable of displaying 3D images. In some embodiments, the 3D display 212 may be a stand-alone display, while in other embodiments, it may be integrated into AR glasses, head-mounted displays, etc. In some embodiments, the 3D display 212 operates using parallax barrier technology. For example, a parallax barrier may comprise parallel vertical stripes of a substantially opaque material (e.g., an opaque film) placed between the screen and the viewer. Due to parallax between the viewer's individual eyes, different portions of the screen (e.g., different pixels) are viewed by the corresponding left and right eyes. In some embodiments, the 3D display 212 operates using biconvex lenses. For example, alternating rows of lenses may be placed in front of the screen, directing light from the screen toward the viewer's left and right eyes respectively.

[0064] System 200 may include a computing system 214 capable of performing certain tasks such as data processing, data modeling, data coordination, and / or data transmission. In some implementations, computing system 214 may also generate images, mix weights, and perform neural processing tasks. In some implementations, computing system 214 is an image processing system. Computing system 214 and / or its components may include some or all of the components described with reference to FIG8.

[0065] The computing system 214 includes an image processor 216 capable of generating 2D and / or 3D information. For example, the image processor 216 may receive (e.g., acquire) one or more input images 132 and / or view parameters 218 and may generate image content for further processing by an image warping engine 220, a hybrid weight generator 222, and / or a neural network 224. The input image 132 may include a captured color image (e.g., RGB, YUV, CMYK, CIE, RYB).

[0066] View parameters 218 may include camera parameters associated with the capture of a particular input image 132 and / or with the capture of an image to be generated (e.g., synthesized). Generally, view parameters 218 may represent approximations of a camera model. View parameters 218 may include any or all of the camera's view orientation, pose, camera angle, lens distortion, and / or intrinsic and extrinsic parameters.

[0067] Image processor 216 also includes (and / or generates and / or receives) occlusion map 226, depth map 228, UV map 230, target view parameters 232, loss function 234, and mesh proxy geometry 236.

[0068] Occlusion map 226 can encode signed distances between surface points identified as closest to the target viewpoint and the camera capturing those surfaces. Positive values ​​can indicate that a point is occluded by the view. Therefore, system 200 can configure the blending weight generator 222 (and NN 224) not to use positive distances when determining blending weights 242, because such occluded image content would not provide accurate reproduction data when generating new or novel views based on the captured images. In some implementations, occlusion map 226 can be used to evaluate the depth difference between the depth observed in a particular view and the depth difference associated with the geometric fusion model of that view.

[0069] Depth map 228 represents one or more images containing information about the distance of a surface of a particular scene object from the selected viewpoint. In some implementations, depth map 228 corresponds to the depth from the target viewpoint to the nearest surface point determined for each output pixel in each of three color camera images and / or in a composite (e.g., novel) view.

[0070] UV map 230 is generated from the visible content in input image 132. In particular, UV map 230 represents the projection of a 2D image onto the surface of a 3D model in order to perform texture mapping to generate features that can be used to generate synthetic images (e.g., novel views).

[0071] Target view parameter 232 represents the view parameters of the novel synthetic image (i.e., the view parameters used to generate a virtual view of the target subject). Target view parameter 232 may include image parameters and / or camera parameters associated with the image to be generated (e.g., synthesized). Target view parameter 232 may include view orientation, pose, camera angle, etc.

[0072] Loss function 234 can evaluate the difference between the ground reality image and the predicted image, wherein the predicted image is predicted based on a combination of visible light information captured for the frame, IR light captured for the frame, and mixing weights associated with color and / or depth. Loss function 234 may include functions describing any or all image errors, image gaps, image error projection artifacts, etc.

[0073] In some implementations, the loss function 234 can include a reconstruction loss based on the reconstruction difference between the segmented ground reality image mapped to the activations of each layer in the neural network and the segmented predicted image mapped to the activations of each layer in the neural network. The segmented ground reality image can be segmented by a ground reality mask to remove background pixels, while the segmented predicted image can be segmented by a prediction mask to remove background pixels. The prediction mask can be predicted based on a combination of visible light information for frame capture and IR light for frame capture.

[0074] The mesh agent geometry 236 can represent a set of K agents {P} i,1 ,…,P i,K A coarse geometry (i.e., a mesh of rectangles, triangles, etc., with UV coordinates). For example, a 2D image can be projected onto the surface of a 3D proxy model to generate a mesh proxy geometry 236. The proxy can be used as a version of the actual geometry to represent specific image content. In operation, system 200 uses the principles of proxy geometry to encode the geometry using a set of coarse proxy surfaces (e.g., mesh proxy geometry 236) along with shape, albedo, and view-dependent effects.

[0075] Image warping engine 220 can be configured to receive one or more input images (e.g., frames, streams) and / or other capture / feature parameter data and generate one or more output images (e.g., frames, streams) that preserve the features. Image warping engine 220 can utilize the capture / feature parameter data to reconstruct the input image in some way. For example, image warping engine 220 can generate reconstructed candidate color images from the input images, where each pixel in the reconstructed image is a candidate pixel for a new synthetic image corresponding to one or more input images.

[0076] In some implementations, the image warping engine 220 can perform functions on the input image at the pixel level to preserve small-scale image features. In some implementations, the image warping engine 220 can use nonlinear or linear functions to generate the reconstructed image.

[0077] The blending weight generator 222 includes a blending algorithm 238 and a visibility score 240. The blending algorithm 238 can be used to generate the blending weights 242. Specifically, the blending algorithm 238 can be accessed via the neural network 224 to generate the blending weights 242. The blending weights 242 represent the values ​​of specific pixels in the image, which can be used to contribute to various aspects of the pixels in the result (e.g., the final, novel image). The blending algorithm 238 includes a trial-and-error algorithm for computing blending weights for coloring a specific set of depth images and / or fused geometry representing the depth images. The blending algorithm receives multi-view color images and noise occlusion cues as input to learn output blending weights for novel views (e.g., novel synthetic images). In some implementations, texture information (e.g., received from a camera pod) about the target view and the input image, along with the visibility score 240, may also be provided as input to the blending algorithm 238.

[0078] Visibility scores 240 can represent the visibility of specific pixels or features of an object in a captured image. Each visibility score 240 can represent a single scalar value that indicates which parts of the image (e.g., pixels, features, etc.) are visible in a particular view of the input image. For example, if the leftmost side of a user's face is not visible in the user's input image, the visibility score 240 representing the leftmost pixel of the user's face can be weighted low, while other areas that are well-viewed and / or captured in the input image can be weighted high. Visibility scores can be taken into account when generating blending weights 242 for novel views (e.g., images).

[0079] Neural network 224 includes an embedder network 244 and a generator network 246. The embedder network 244 includes one or more convolutional layers and downsampling layers. The generator network 246 includes one or more convolutional layers and upsampling layers.

[0080] The in-painter 254 can generate content (e.g., pixels, regions, etc.) that may be missing from a particular texture or image based on the local neighborhood of pixels surrounding a specific missing content area. In some implementations, the in-painter 254 can utilize blending weights 242 to determine how to in-paint a specific pixel, region, etc. The in-painter 254 can utilize the output from the neural network 224 to predict a specific background / foreground mask for rendering. In some implementations, the in-painter 254 can be used in conjunction with the image correction engine 252 to pull-push hole filling. This can be performed on images with regions / pixels that have missing depth information, which can result in no output color predicted by the neural network 224. The image correction engine 252 can trigger the in-painter to colorize a specific region / pixel in the image.

[0081] Once the blending weights 242 are determined, system 214 can provide these weights to neural renderer 248. Neural renderer 248 can generate intermediate representations of objects (e.g., users) and / or scenes, for example, utilizing NN 244 (or another NN). For example, using an object-specific convolutional network, neural renderer 248 can incorporate view-related effects by simulating the difference between realistic appearance (e.g., baseline truth) and diffuse reprojection.

[0082] In operation, system 200 can receive a stereo fusion pipeline that generates (1) a depth map corresponding to each of the three color camera images and (2) a depth value D in the composite view from the target viewpoint to the nearest surface point determined for each output pixel. t For example, an image capture system may include at least three camera bays. Each camera bay may include one or more color cameras and depth cameras (e.g., camera 204, depth sensor 206). In some implementations, the image capture system may additionally include a witness camera bay. In this example, the system may perform geometric warping to transform information from the three camera bays into a target image space for the witness camera bay. Specifically, for each k of the three color cameras, reproducing a color (e.g., RGB) image can be calculated... The value is represented by the depth of the target viewpoint reprojected into 3D space (e.g., depth map 228) to represent the reprojected RGB image (e.g., reprojected image 404). Additionally, it can be calculated... The values ​​represent occlusion maps (e.g., occlusion map 226) that encode the signed distance between the surface point closest to the predefined target viewpoint and the camera. Positive values ​​indicate that the image point is occluded by the view. Negative values ​​indicate that the image point is not occluded by the view.

[0083] Then Value and Value and D tThe first fully two-dimensional convolutional layer, cascaded into an N224, contains 13-channel tensors per pixel. The network can predict an image W, where each channel is a non-negative floating-point value for each pixel of the input color image. System 214 can then construct the output image I. N .

[0084] In some implementations, system 214 may use multi-resolution blending engine 256 to perform multi-resolution blending. Multi-resolution blending engine 256 may take an image pyramid as input to a convolutional neural network (e.g., NN 224 / 414), which generates blending weights at multiple scales based on opacity values ​​associated with each scale. In operation, multi-resolution blending engine 256 may employ a two-stage trained end-to-end convolutional network process. Engine 256 may utilize multiple source cameras.

[0085] As described herein, based at least in part on the calculated mixing weights 242, the synthetic view 250 represents a 3D stereoscopic image of content (e.g., VR / AR objects, users, scenes, etc.) with appropriate parallax and viewing configuration for the eyes of a user associated with accessing the display (e.g., display 212). Each time the user moves their head position while viewing the display and / or each time a particular image changes on the display, the system 214 can determine at least a portion of the synthetic view 250 based on the output from a neural network (e.g., NN 244). In some embodiments, the synthetic view 250 represents the user's face and other features of the user surrounding the face within the view capturing the user's face. In some embodiments, for example, the synthetic view 250 represents the entire field of view captured by one or more cameras associated with the telepresence system 202A.

[0086] In some implementations, the processors (not shown) of systems 202 and 214 may include (or communicate with) a graphics processing unit (GPU). In operation, the processor may include or have access to memory, storage, and other processors (e.g., a CPU). To aid in graphics and image generation, the processor may communicate with the GPU to display images on a display device (e.g., display device 212). The CPU and GPU may be connected via a high-speed bus such as PCI, AGP, or PCI-Express. The GPU may be connected to the display via another high-speed interface such as HDMI, DVI, or DisplayPort. Typically, the GPU renders image content in pixel form. Display device 212 may receive image content from the GPU and may display the image content on the display screen.

[0087] Despite Figure 2Not depicted, but additional maps such as feature maps can be provided to one or more NNs 224 to generate image content. Feature maps can be generated by analyzing the image to generate features for each pixel of the image. Such features can be used to generate feature maps and texture maps, which can be provided to the hybrid weight generator 222 and / or NN 224 to help generate hybrid weights 242.

[0088] Figure 3 This is a block diagram illustrating an example of reprojection of an input image to a target camera viewpoint according to an implementation described throughout this disclosure. For example, system 200 may be used to generate a reprojection of an image that is to be used as an input image for a neural network (NN). Image warping may include reprojecting a captured input image 132 to a target camera viewpoint using a fused depth (from a depth image) to aim at the camera viewpoint. In some implementations, the input image 132 is already in the form of a reprojected image. In some implementations, image warping engine 220 performs the warping.

[0089] For example, image warping engine 220 can back-project target image point x 302 onto a ray. Image warping engine 220 can then find point X 304 at a distance d from target camera 308. Next, image warping engine 220 can project X onto cabin image point x'306 at a distance d' from cabin camera 310. The following equations [1]-[3] describe this calculation:

[0090]

[0091]

[0092]

[0093] Next, the image warping engine 220 can perform bilinear sampling of the texture camera image at x', as shown by the following equations [4] and [5]:

[0094]

[0095]

[0096] Figure 4 This is a block diagram of an example flowchart 400 for generating synthetic content for rendering on a display using neural mixing techniques, based on an implementation described throughout this disclosure. Figure 400 can generate data (e.g., multi-view color images, noise occlusion cues, depth data, etc.) to be fed to a mixing algorithm via a neural network. The neural network is then able to learn and output mixing weights.

[0097] In this example, multiple input images 402 can be obtained (e.g., received). For example, system 202A can capture multiple input images 402 (e.g., image frames, video). Input images 402 can be color images. Input images 402 can also be associated with depth images captured at substantially the same time as the input images. For example, the depth image can be captured by an infrared camera.

[0098] The computing system 214 can use the input image color and depth images to warp (e.g., reproject) the input image 402 into a reprojected image 404. For example, the warping engine 220 can reproject the input image 402 into an output view representing the desired novel view. Specifically, the warping engine 220 can retrieve colors from the input image 402 and warp the colors into the output view using the depth view associated with the input image. In general, each input image can be warped into a single reprojected view. Thus, if four input images are retrieved, the warping engine 220 can generate four reprojected views, each associated with a single input image. The reprojected image 404 serves as a candidate color that can be selected for pixels in the novel synthetic output image. The depth view, captured at approximately the same time as the input image 402, can be used to generate a depth map 406 and an occlusion map 408 (similar to depth map 228 and occlusion map 226).

[0099] The reprojected image 404 can be used to generate a weighted sum image 410, which represents a weighted combination of colors used for pixels. The weighted sum image 410 may also take into account a ground reality image 412. The ground reality image 412 may be captured by one or more eyewitness cameras.

[0100] The reprojected image 404, depth map 406, and occlusion map 408 can be provided to the NN 414, as follows: Figure 4 The diagram shows a convolutional neural network with a U-Net shape. Other neural networks are, of course, possible. In a non-limiting example, the input to NN 414 could include three color RGB images, an occlusion map, and a depth map of the target view, which could utilize approximately fourteen channels.

[0101] In some implementations, multiple view parameters 415 may also be provided to the NN 414. The view parameters 415 may be related to the desired novel view (e.g., an image). The view parameters 415 may include any or all of the following: view orientation, pose, camera angle, lens distortion, and / or intrinsic and extrinsic parameters of the camera (virtual or physical camera).

[0102] NN 414 can generate blending weights 416 for each reprojected image 404 to determine how to combine the colors of the reprojected image 404 to generate an accurate novel output image. The reprojected image 404 can be computed by warping the input image 402 into a novel view based on the depth image 406. NN 414 can use the blending weights 416 and the reprojected image 404 to generate a blended texture image 418, for example, by blending at least a portion of the reprojected image 404 with each other using the blending weights 416. The blended texture image 418 can be used to generate an image associated with each camera bay associated with the input image 402 and therefore with the reprojected image 404. In this example, three camera bays are used to capture three color images (e.g., input image 402) and three depth images (e.g., represented by depth map 406). Therefore, three corresponding image views are output, as shown by image 420. Novel views can be synthesized using images 418 and 420, as shown by synthesized image 422.

[0103] In operation, the NN 414 can use mixing weights 416 to determine how to combine the reprojected colors associated with the reprojected image 404 to generate an accurate synthetic image 422. The NN 414 can determine the mixing weights by learning in the space of a predefined output view.

[0104] The network architecture of NN 414 can be a deep neural network with all convolutional layers having the same padding values ​​and rectified linear unit activation functions. The output can include mixed weights 416 for three reprojected images 404, one channel per camera module, wherein the output weights are generated according to equation [6]:

[0105] W' = 10 -2 *W+1 / 3, clamp to [0,1] [6]

[0106] Figure 400 can be executed to take into account the training loss. For example, the resulting synthetic image 422 can be improved by determining and using reconstruction loss, perceptual loss on the mixed color image, and integrity loss.

[0107] In operation, system 200 can utilize several aspects to generate per-pixel loss values. For example, the novel view image I for texture camera i can be represented as shown in equation [7]. N and neural hybrid weight W i :

[0108]

[0109] And an invalid target depth mask without input RGB values ​​can be represented as I. Mask .

[0110] In particular, the example loss function can be expressed by the following equation [8]:

[0111]

[0112] Where D:=α r L1+α c VGG represents the reconstruction and perceptual loss. In other words, it can be represented as shown in equation [9]:

[0113] D: = L1 + αL VGG ,D(I N ⊙I Mask I Witness ⊙I Mask [9]

[0114] The integrity loss on the network output mixing weights for each x, y pixel coordinate can be expressed as shown in equation

[10] :

[0115] L compl (W)=∑ x ∑ y |(∑ k w k，x，y )-1|

[10]

[0116] The occlusion loss on the network can be expressed as shown in equation

[11] :

[0117] L occl (C, O) = ||c x，y ||1, if o x，y >τ

[11]

[0118] In some implementations, the NN 414 can be trained based on minimizing the occlusion loss function (i.e., equation [8]) between the synthetic image 422 generated by the NN 414 and the ground reality image 412 captured by at least one witness camera.

[0119] Figure 5 This is a block diagram of an example flowchart for generating hybrid weights based on the implementation described throughout this disclosure. For example, this example could employ a convolutional neural network (e.g., a convolutional U-Net) to process pixels of each input view. A multilayer perceptron (MLP) could be used to generate hybrid weights to assign each pixel of the proposed synthetic view. The hybrid weights generated by the MLP can be used to combine features from the input image / view.

[0120] In some implementations, generating blending weights may involve the use of multi-resolution blending techniques. Multi-resolution blending techniques employ a two-stage, end-to-end convolutional network process. These techniques utilize multiple source cameras. For example, system 202A can capture one or more input images (e.g., RGB color images) from each of the first camera bay 502, the second camera bay 504, and the third camera bay 506. Similarly, and at substantially the same time, bays 502-504 are each capable of capturing (or computing) a depth image corresponding to a specific input image.

[0121] At least three color source input images and at least three source depth images can be provided to convolutional networks 508A, 508B, and 508C (e.g., convolutional U-Net) to generate feature maps embedding view-related information. For example, one or more feature maps (not shown) can represent features of the input images in feature space. Specifically, for each input image / depth image 502-504, extracted features of the image can be used to generate feature maps (e.g., feature maps 510A, 510B, and 510C). In some implementations, the input image may include two color source images and a single depth image. In such an example, system 500 can use a single depth image to reproject each of the two color input images into the output view.

[0122] Feature maps 510A-510C can be used to generate UV maps 512A, 512B, and 512C. For example, feature maps 510A-510C can be used to generate UV maps 512A-C from the visible content in input images 502-504. UV maps 512A-512C represent projections of 2D images onto the surface of a 3D model to perform texture mapping to generate features that can be used to generate synthetic images (e.g., novel views). The output neural texture remains in the source camera image coordinates.

[0123] The corresponding feature maps 510A-510C can each be sampled together with the corresponding UV maps 512A-512C and the witness camera parameters 514. For example, system 500 can use a witness camera as a target camera for generating synthetic novel images. The witness (e.g., target) camera parameters 514 can be predefined. Each of the corresponding sampled feature maps 510A-510C and UV maps 512A-C can be used with parameter 514 and sampled together with an occlusion map and a depth map 516. Sampling may include differentiable sampling layers that warp each neural texture using UV maps 512A-512C pre-computed from a fusion geometry (e.g., mesh proxy geometry 236).

[0124] The sampled content can be used by a per-pixel multilayer perceptron (MLP) NN 518 to generate occlusion maps, depth maps, etc., of sampled features from all source camera views. Based on these maps, the MLP 518 can generate a set of blending weights 520. For example, the per-pixel MLP 518 map can include sampled features from any number of source camera views, which can be used to generate a set of blending weights 520. Such blending weights 520 can be used to generate a synthetic image.

[0125] In some implementations, the process described herein can be incorporated into a multi-resolution blending technique. For example, the multi-resolution blending technique can be performed, for instance, by a multi-resolution blending engine 256, and can take an image pyramid as input to a convolutional neural network (e.g., NN 224 / 414) that generates blending weights at multiple scales based on opacity values ​​associated with each scale.

[0126] The output blending weights at each scale are used to construct the output color image from the input reprojected color image at that scale, thus forming an output image pyramid. Each level of this pyramid is then weighted by an associated opacity value and upsampled to the original scale. The resulting set of images is then summed to construct the final output image. This is advantageous because of the fact that small holes exist in the input reprojected image (due to missing geometry), and the shrinking and then enlarging process fills the missing areas with neighboring pixel values. Additionally, the process can generate softer contours that are visually more appealing than conventional blending techniques.

[0127] In some implementations, the input pyramid can be constructed by downsampling the bilinear reprojection color of the reprojected image, without pre-multiplying by the downsampled effective depth mask (e.g., graph), upsampling back to a predefined (e.g., original) resolution, and without pre-multiplying by the upsampled effective depth mask. For each layer, the flowchart can add an output layer decoder (for mixing weights and α), upsample to a predefined (e.g., original resolution), adjust the additional background α at the highest resolution, normalize α using a soft max function, and blend with the reprojection color and background.

[0128] Multi-resolution fusion techniques employ a two-stage, end-to-end convolutional network process. For each stage, for example, multi-resolution fusion techniques may add an output layer decoder (e.g., on the fusion weights and alpha loss). These techniques compute an RGB image, add the loss, multiply by alpha, and concatenate them to determine candidate RGB images. The candidate RGB images can be upsampled. The upsampled candidate images can be used to generate the output image (e.g., novel view / synchronous image) while taking the loss into account.

[0129] In operation, the technology utilizes multiple source cameras. For example, system 202A can capture one or more input images (e.g., RGB color images) from each of the first camera bay 502, the second camera bay 504, and the third camera bay 506. Similarly, and at substantially the same time, bays 502-504 are each capable of capturing a depth image corresponding to a specific input image.

[0130] Regardless of how the output viewpoint moves, multi-resolution blending can use the same 3D point at the same location in the scene map as in the feature map. This ensures that 2D convolutions are not performed, and therefore the output includes the same blending weights for point locations because the input features are fixed.

[0131] Figure 6 This is a flowchart illustrating an example of a process 600 for generating synthetic content using neural hybrid techniques according to an implementation described throughout this disclosure. Process 600 is about... Figure 1 and Figure 2 The example implementations of systems 100 and / or 200 and / or systems 500 and / or 800 are described herein, but it should be understood that the method can be implemented by systems with other configurations. Generally, one or more processors and memories on system 202 and / or computing system 214 can be used to execute process 600.

[0132] At a higher level, process 600 may utilize a color input image, a depth image corresponding to the input image, and view parameters associated with a desired novel view corresponding to at least a portion of the content within the input image. Process 600 may provide the aforementioned elements or versions thereof to a neural network to receive blending weights for determining specific pixel colors and depths for the desired novel view. The view may be used in conjunction with the blending weights to generate a novel output image.

[0133] In box 602, process 600 may include receiving multiple input images. For example, system 202A (or other image processing system) may use a camera (e.g., camera 204) to capture input images from two or more camera bays. Generally, the multiple input images are color images captured according to predefined view parameters. However, in some implementations, the multiple input images may be gradient images of a single color (e.g., sepia, grayscale, or other gradient colors). Predefined view parameters may include camera parameters associated with the capture of a particular input image 132 (e.g., input image 402) and / or with the capture of an image to be generated (e.g., composited). In some implementations, view parameters may include any or all of the camera's view orientation, pose, camera angle, lens distortion, and / or intrinsic and extrinsic parameters. In some implementations, the multiple input images may include multiple target subjects captured within a frame of the image. Target subjects may include users, backgrounds, foregrounds, physical objects, virtual objects, gestures, hairstyles, wearable devices, etc.

[0134] At block 604, process 600 may include receiving multiple depth images associated with a target subject in at least one of a plurality of input images. For example, system 202A may capture depth image 138 at substantially the same capture time as the input image (e.g., RGB color image 136). The depth images may capture the target subject also captured in one or more of the plurality of input images. Each depth image may include a depth map (e.g., FIG. 228) associated with at least one camera 204 that captured at least one of the plurality of input images 132, at least one occlusion map 226, and (e.g., via target view parameter 232) a depth map associated with a ground-based image captured by at least one witness camera at a time corresponding to the capture of at least one of the plurality of input images. In short, system 200 may take into account the depth of the input images and the depth of the desired target view (or other determined target view) of the witness camera when generating blending weights 242 for the target view.

[0135] At box 606, process 600 may include receiving multiple view parameters for generating a virtual view of a target subject. For example, the view parameters may be related to a desired novel view (e.g., a novel synthetic image associated with a novel (e.g., virtual) view not previously captured by a camera). For example, the view parameters may include target parameters for a witness camera that captures content at substantially the same time as color image 136 and depth image 138. The view parameters may include predefined lens parameters, viewing direction, pose, and specific intrinsic and / or extrinsic parameters of the camera configured to capture the novel view.

[0136] At box 608, process 600 may include generating multiple warped images based on at least one of multiple input images, multiple view parameters, and multiple depth images. For example, image warping engine 220 may generate warped images using input image 132 by reprojecting it into a reprojected version of image 132. Warping may be performed to determine the projection of the input color of input image 132 into the novel view using depth information (e.g., a separate depth image or a geometrically desirable surface). Warping may generate a reprojected image (e.g., image 404) by obtaining colors from one or more original input views and manipulating the colors for the novel view (e.g., image 406) using depth images (e.g., depth map 406 and occlusion map 408). Each input image may be used to generate a separate reprojection. The reprojected image (e.g., image 404) may represent pixels of candidate colors that can be used in the novel composite image.

[0137] In some implementations, process 600 may include determining candidate projections of colors associated with the multiple input images 402 into an uncaptured view (i.e., a novel view / image, a virtual view / image) using at least one of multiple depth images (e.g., depth map 406 and occlusion map 408) to generate multiple warped images based on the multiple input images, multiple view parameters, and at least one of the multiple depth images. The uncaptured view may include at least a portion of image features from at least one of the multiple input images. For example, if the input images include objects, the uncaptured view may take into account at least a portion of the object, its color, pixels, etc.

[0138] In box 610, process 600 may include receiving blending weights 416 from a neural network (e.g., NN 224, NN 414, NN 508A-C) for assigning pixel colors to a virtual view (e.g., unseen image / uncaptured view) of a target subject (e.g., user 104'). In some implementations, the target subject may include or be based on at least one element captured in at least one frame of a plurality of input images 402. The blending weights 416 may be received in response to providing the NN 414 with a plurality of depth images (e.g., depth image 138 and / or depth map 406 and / or occlusion map 408), a plurality of view parameters 415, and a plurality of warped images (e.g., reprojected image 404). The NN 414 may generate the blending weights 416 to indicate a probabilistic manner in which the colors of the reprojected image 404 are combined to provide a probable and realistic output image that realistically represents the target subject. In some implementations, blending weights 416 are configured to assign blending colors to each pixel of a virtual view (i.e., a novel and / or unseen and / or previously uncaptured view), resulting in such blending colors being assigned to an output composite image (e.g., composite image 422). For example, blending weights 416 are used to blend at least portions of reprojected image 404 with each other.

[0139] In box 612, process 600 may include generating a composite image based on hybrid weights and virtual views according to view parameters. Composite image 422 may represent an image captured using parameters associated with an uncaptured view (e.g., not captured by a physical camera, generated as a virtual view from a virtual or physical camera, etc.), which may represent an unseen view (e.g., not captured by any camera in the imaging system, but instead composited). Composite image 422 may be generated for and / or during 3D (e.g., telepresence) video conferencing. For example, composite image 422 may be generated in real-time during a video conference to provide an error-corrected and accurate image of a user or content being captured by a camera associated with the video conference. In some implementations, composite image 422 represents a novel view generated for a 3D video conference. In some implementations, composite image represents an uncaptured view of a target subject generated for a 3D video conference.

[0140] In operation, blending weights are applied to pixels in the virtual view based on view parameters. The resulting virtual view may include pixel colors generated using blending weights tailored to the target subject. For example, a color image of the virtual view can be used to generate a composite view based on view parameters associated with a virtual camera.

[0141] In some implementations, process 600 may additionally perform a geometry fusion process. In some implementations, process 600 may perform a geometry fusion process instead of providing separate depth images per input image. For example, process 600 may use a geometry fusion process on multiple depth images to reconstruct a desired surface (e.g., a geometry proxy) to generate a geometry fusion model.

[0142] A geometric fusion model can be used to replace multiple views of depth image data (e.g., depth views of captured image content) with an updated (e.g., computed) view of the depth image data. The updated depth view can be generated as a view of the image content, containing depth data from the captured depth views and additionally including image and / or depth information from each of any other available captured depth views of the image content. One or more updated depth views can be used by the NN 414, for example, to synthesize additional (and new) blending weights to synthesize additional (and new) views of the object by utilizing the geometrically fused depth image data and image and depth information associated with multiple other views of the object. Any number of algorithms can be used to fuse the depth image data to replace each (input) depth view with a new depth view incorporating depth data information from several other depth views. In some implementations, system 200 is capable of using the geometric fusion model to generate depth data (e.g., depth maps) that can be used to correct for such occlusion loss regarding the cause of occlusion.

[0143] Process 600 may then generate multiple reprojected images based on multiple input images and a desirable surface for generating geometrically fused depth image data, and provide the geometrically fused depth image data (along with multiple view parameters 415 and multiple reprojected images 404) to NN 414. In response, process 600 may include receiving blending weights 416 from NN 414 and / or additional blending weights generated using the desired surface depth image data for assigning colors to pixels in the composite image 422.

[0144] In some implementations, process 600 may further include providing the NN 414 with a depth difference between the geometric fusion model and the depths observed in multiple depth images. For example, the depth difference may be used to correct for occlusion detected in the synthetic image 422. In some implementations, the NN 414 may be trained based on minimizing an occlusion loss function between the synthetic image generated by the NN 414 and a ground-based real-world image 412 captured by at least one witness camera (e.g., associated with system 202A), as per [reference to...]. Figure 4 A detailed description is provided. In some implementations, process 400 can be performed using a single depth image instead of multiple depth images.

[0145] In some implementations, the NN 414 is further configured to perform multi-resolution blending to assign pixel colors to pixels in the synthesized image. In operation, multi-resolution blending may trigger the feeding of an image pyramid as input to the NN 414 to trigger the receiving of multi-resolution blending weights (e.g., additional blending weights 520) from the NN 414 for multiple scales, and may additionally receive opacity values ​​associated with each scale.

[0146] Figure 7 Examples of a computer device 700 and a mobile computer device 750 that can be used with the described technology are shown. The computing device 700 may include a processor 702, a memory 704, a storage device 706, a high-speed interface 708 connected to the memory 704 and multiple high-speed expansion ports 710, and a low-speed interface 712 connected to a low-speed bus 714 and the storage device 706. Components 702, 704, 706, 708, 710, and 712 are interconnected using various buses and may be mounted on a common motherboard or, as appropriate, otherwise installed. The processor 702 may process instructions for execution within the computing device 700, including instructions stored in the memory 704 or the storage device 706, to display graphical information for a GUI on an external input / output device, such as a display 716 coupled to the high-speed interface 708. In some embodiments, multiple processors and / or multiple buses may be used in conjunction with multiple memories and various types of memory, as appropriate. In addition, multiple computing devices 700 can be connected, each providing a part of the necessary operation (e.g., as a server library, blade server cluster, or multiprocessor system).

[0147] The memory 704 stores information in the computing device 700. In one embodiment, the memory 704 is a volatile memory cell or multiple volatile memory cells. In another embodiment, the memory 704 is a non-volatile memory cell or multiple non-volatile memory cell cells. The memory 704 may also be another form of computer-readable medium, such as a magnetic disk or optical disk.

[0148] Storage device 706 provides mass storage for computing device 700. In one embodiment, storage device 706 may be a computer-readable medium or may contain computer-readable media, such as a floppy disk device, hard disk device, optical disk device, magnetic tape device, flash memory or other similar solid-state storage device, or an array of devices, including devices in a storage area network or other configuration. A computer program product may be tangibly embodied in an information carrier. A computer program product may also contain instructions that, when executed, perform one or more methods such as those described herein. The information carrier may be a computer-readable or machine-readable medium, such as memory 704, storage device 706, or memory on processor 702.

[0149] High-speed controller 708 manages bandwidth-intensive operations of computing device 700, while low-speed controller 712 manages lower bandwidth-intensive operations. This functional allocation is merely exemplary. In one embodiment, high-speed controller 708 is coupled to memory 704, display 716 (e.g., via a graphics processor or accelerator), and high-speed expansion port 710, which can accommodate various expansion cards (not shown). Low-speed controller 712 may be coupled to storage device 706 and low-speed expansion port 714. The low-speed expansion port, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, Wireless Ethernet), may be coupled to one or more input / output devices, such as keyboards, pointing devices, scanners, or networking devices, such as switches or routers, via a network adapter.

[0150] As shown in the accompanying drawings, computing device 700 can be implemented in a variety of different forms. For example, it can be implemented as a standard server 720, or multiple times in a group of such servers. Furthermore, it can be implemented as part of a rack server system 724. Additionally, it can be implemented in a personal computer such as a laptop computer 722. Alternatively, components from computing device 700 can be combined with other components (not shown) in mobile devices, such as device 750. Each of these devices can contain one or more computing devices 700, 750, and the entire system can consist of multiple computing devices 700, 750 communicating with each other.

[0151] Among other components, computing device 750 includes processor 752, memory 764, input / output devices—such as display 754, communication interface 766, and transceiver 768. Device 750 may also be provided with storage devices, such as microdrives or other devices, to provide additional storage. Each of components 750, 752, 764, 754, 766, and 768 is interconnected using various buses, and some components may be mounted on a common motherboard or otherwise, depending on the circumstances.

[0152] Processor 752 can execute instructions in computing device 750, including instructions stored in memory 764. The processor can be implemented as a chipset comprising multiple separate analog and digital processors. The processor can, for example, provide cooperation with other components of device 750, such as control of the user interface, applications running through device 750, and wireless communication through device 750.

[0153] Processor 752 can communicate with the user via control interface 758 and display interface 756 coupled to display 754. Display 754 may be, for example, a TFT-LCD (Thin Film Transistor Liquid Crystal Display) or OLED (Organic Light Emitting Diode) display or other suitable display technology. Display interface 756 may include appropriate circuitry for driving display 754 to present graphics and other information to the user. Control interface 758 can receive commands from the user and translate them for submission to processor 752. Furthermore, external interface 762 can communicate with processor 752, enabling near-field communication between device 750 and other devices. For example, in some embodiments, external interface 762 may provide wired or wireless communication, and multiple interfaces may be used.

[0154] Memory 764 stores information in computing device 750. Memory 764 can be implemented as one or more computer-readable media, volatile memory cells, or non-volatile memory cells. Furthermore, expansion memory 784 can be provided and connected to device 750 via expansion interface 782, which may include, for example, a SIMM (Single In-line Memory Module) card interface. This expansion memory 784 can provide additional storage space for device 750 and can also store applications or other information for device 750. Specifically, expansion memory 784 may include instructions to perform or supplement the above-described processes, and may also include security information. Therefore, for example, expansion memory 784 can be a security module for device 750 and can be programmed via instructions that allow secure use of device 750. Furthermore, secure applications can be provided via a SIMM card along with additional information, such as placing identification information into the SIMM card in a non-hacking manner.

[0155] The memory may include, for example, flash memory and / or NVRAM memory, as described below. In one embodiment, the computer program product may be tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods such as those described above. The information carrier may be a computer-readable or machine-readable medium, such as memory 764, extended memory 784, or memory on processor 752, which may be received, for example, via transceiver 768 or external interface 762.

[0156] Device 750 can communicate wirelessly via communication interface 766, which may include digital signal processing circuitry if necessary. Communication interface 766 can provide communication under various modes or protocols, such as GSM voice calls, SMS, EMS or MMS messages, CDMA, TDMA, PDC, WCDMA, CDMA2000, or GPRS, etc. This communication can be performed, for example, via radio frequency transceiver 768. Furthermore, short-range communication can be performed using transceivers such as Bluetooth, WiFi, or others (not shown). Additionally, GPS (Global Positioning System) receiver module 780 can provide device 750 with additional navigation and location-related wireless data, which can be used, as appropriate, through an application running on device 750.

[0157] Device 750 can also communicate audibly using audio codec 760, which can receive spoken information from a user and convert it into usable digital information. Similarly, audio codec 760 can generate audible sound for the user, such as through a speaker, for example, in headphones of device 750. This sound can include sounds from voice telephone calls, recorded sounds (e.g., voice messages, music files, etc.), and sounds generated by applications operating on device 750.

[0158] As shown in the attached diagram, the computing device 750 can be implemented in a variety of different forms. For example, it can be implemented as a cellular phone 780. It can also be implemented as a smartphone 782, a personal digital assistant, or part of another similar mobile device.

[0159] Various implementations of the systems and techniques described herein can be implemented in digital electronic circuits, integrated circuits, specially designed ASICs (Application-Specific Integrated Circuits), computer hardware, firmware, software, and / or combinations thereof. These different implementations may include implementations in one or more computer programs executable and / or interpretable on a programmable system, the programmable system including at least one programmable processor, which may be dedicated or general-purpose, coupled to receive data and instructions from a storage system, at least one input device, and at least one output device, and to transfer data and instructions to the storage system, at least one input device, and at least one output device.

[0160] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in high-level programming and / or object-oriented programming languages ​​and / or assembly / machine languages. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program product, device, and / or apparatus (e.g., disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term “machine-readable signal” refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0161] To provide interaction with the user, the systems and techniques described herein can be implemented on a computer having a display device (such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and pointing device (such as a mouse or trackball) from which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (such as visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including sound, speech, or tactile input.

[0162] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), middleware components (e.g., application servers), frontend components (e.g., client computers with graphical user interfaces or web browsers, through which users can interact with embodiments of the systems and technologies described herein), or any combination of these backend, middleware, or frontend components. The components of the system can be interconnected via any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0163] A computing system can include clients and servers. Clients and servers are generally geographically separated and typically interact through communication networks. The relationship between clients and servers is generated by computer programs running on various computers that have a client-server relationship with each other.

[0164] In some embodiments, Figure 7 The computing device shown may include sensors that interface with a virtual reality headset (VR headset / HMD device 790). For example, computing device 750 or Figure 7One or more sensors included in the other computing devices shown can provide input to the VR headset 790 or, more generally, to the VR space. Sensors may include, but are not limited to, touchscreens, accelerometers, gyroscopes, pressure sensors, biometric sensors, temperature sensors, humidity sensors, and ambient light sensors. The computing device 750 can use the sensors to determine its absolute position and / or detected rotation in the VR space, and then use this as input to the VR space. For example, the computing device 750 can be incorporated as a virtual object into the VR space, such as a controller, laser pointer, keyboard, weapon, etc. When incorporated into the VR space, user positioning of the computing device / virtual object allows the user to position the computing device to view the virtual object in some way within the VR space.

[0165] In some embodiments, one or more input devices included or connected to the computing device 750 can be used as inputs to the VR space. Input devices may include, but are not limited to, touchscreens, keyboards, one or more buttons, touchpads, pointing devices, mice, trackballs, joysticks, cameras, microphones, headphones or earbuds with input functionality, game controllers, or other connectable input devices. When the computing device is incorporated into the VR space, users interacting with the input devices included on the computing device 750 can cause specific actions to occur in the VR space.

[0166] In some embodiments, one or more output devices included on the computing device 750 may provide output and / or feedback to a user of the VR headset 790 in the VR space. The output and feedback may be visual, tactile, or audio. The output and / or feedback may include, but is not limited to, rendering the VR space or virtual environment, vibration, turning on and off or flashing and / or blinking one or more lights or flashlights, emitting alarms, playing ringtones, playing songs, and playing audio files. Output devices may include, but are not limited to, vibration motors, vibration coils, piezoelectric devices, electrostatic devices, light-emitting diodes (LEDs), flashlights, and speakers.

[0167] In some embodiments, a computing device 750 may be placed within a VR headset 790 to create a VR system. The VR headset 790 may include one or more positioning elements that allow the computing device 750 (such as a smartphone 782) to be placed in the appropriate position within the VR headset 790. In this embodiment, the display of the smartphone 782 may render stereoscopic images representing a VR space or virtual environment.

[0168] In some embodiments, the computing device 750 may appear as another object in a computer-generated 3D environment. User interactions with the computing device 750 (e.g., rotation, shaking, touching a touchscreen, swiping a finger across a touchscreen) can be interpreted as interactions with an object in the VR space. As merely an example, the computing device could be a laser pointer. In this example, the computing device 750 appears as a virtual laser pointer in the computer-generated 3D environment. When the user manipulates the computing device 750, the user in the VR space perceives the laser pointer as moving. The user receives feedback from interactions with the computing device 750 within the VR environment on the computing device 750 or the VR headset 790.

[0169] In some embodiments, computing device 750 may include a touchscreen. For example, a user may interact with the touchscreen in a manner that mimics events occurring on the touchscreen and in VR space. For instance, a user may use a pinch gesture to zoom in on content displayed on the touchscreen. Such a pinch gesture on the touchscreen causes information provided in VR space to be magnified. In another example, the computing device may be rendered as a virtual book in a computer-generated 3D environment. In VR space, pages of the book may be displayed, and a user's finger swiping across the touchscreen may be interpreted as turning / flipping pages of the virtual book. With each page turned / flipped, in addition to seeing the page content change, audio feedback, such as the sound of turning pages in a book, may be provided to the user.

[0170] In some embodiments, one or more input devices (e.g., a mouse, a keyboard) other than a computing device can be rendered in a computer-generated 3D environment. The rendered input devices (e.g., a rendered mouse, a rendered keyboard) can be used to render objects in VR space to control those objects.

[0171] The purpose of computing device 700 is to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The purpose of computing device 750 is to represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the disclosed embodiments.

[0172] Furthermore, the logical flow shown in the accompanying drawings does not require a specific or sequential order to achieve the desired result. Additionally, other steps can be provided to the shown flow, other steps can be eliminated from the shown flow, and other components can be added to or removed from the system. Therefore, other embodiments fall within the scope of the appended claims.

Claims

1. A computer-implemented method, comprising: Receive multiple input images; Receive multiple depth images captured by multiple depth cameras and associated with a target subject in at least one of the multiple input images, each of the depth images including at least one occlusion map, the occlusion map encoding a signed distance between a surface point determined to be closest to the target viewpoint corresponding to the synthetic image and the camera that captured the surface; Receive multiple view parameters for generating a virtual view of the target subject; Multiple warped images are generated based on the multiple input images, the multiple view parameters, and at least one depth image among the multiple depth images; In response to providing the plurality of depth images, the plurality of view parameters, and the plurality of warped images to the neural network, a blending weight is received from the neural network for assigning color to pixels of the virtual view of the target subject, wherein the neural network does not use positive distance when determining the blending weight; A composite image is generated based on the hybrid weights and the virtual view according to the view parameters; and Occlusion detected in the synthesized image is corrected based on the depth difference between the geometric fusion model and the depth observed in the plurality of depth images.

2. The computer-implemented method according to claim 1, further comprising: A geometric fusion process is used on the multiple depth images to reconstruct a desirable surface to generate a geometric fusion model; Multiple reprojection images are generated based on the multiple input images and the desired surface; as well as In response to providing the neural network with the plurality of depth images, the plurality of view parameters, and the plurality of reprojection images, additional blending weights for assigning colors to pixels in the synthesized image are received from the neural network.

3. The computer-implemented method of claim 2, further comprising providing the neural network with the depth difference between the geometric fusion model and the depths observed in the plurality of depth images.

4. The computer-implemented method according to claim 1, wherein: The plurality of input images are color images captured according to predefined view parameters associated with at least one camera that captured the plurality of input images; And / or Each of the plurality of depth images includes a depth map associated with at least one camera that captured at least one of the plurality of input images and / or a depth map associated with a ground-based image captured by at least one witness camera at the time corresponding to the capture of at least one of the plurality of input images.

5. The computer-implemented method according to claim 1, wherein, The blending weights are configured to assign a blending color to each pixel of the synthesized image.

6. The computer-implemented method according to claim 1, wherein, The neural network is trained based on minimizing a loss function between the synthetic image generated by the neural network and a ground-based image captured by at least one eyewitness camera.

7. The computer-implemented method according to claim 1, wherein, The synthesized image is an uncaptured view of the target subject generated for a 3D video conference.

8. The computer-implemented method according to any one of claims 1 to 7, wherein, Generating the plurality of warped images based on the plurality of input images, the plurality of view parameters, and at least one depth image from the plurality of depth images includes using at least one depth image from the plurality of depth images to determine candidate projections of colors associated with the plurality of input images into an uncaptured view, the uncaptured view including at least a portion of image features from at least one of the plurality of input images.

9. An image processor system for performing the method according to any one of the preceding claims, the system comprising: At least one processing device; as well as The memory stores instructions that, when executed, cause the system to perform operations, including: Receive multiple input images captured by the image processing system; Receive multiple depth images captured by multiple depth cameras of the image processing system, each of the depth images including at least one occlusion map, the occlusion map encoding a signed distance between a surface point determined to be closest to a target viewpoint corresponding to the synthetic image and the camera capturing the surface; Receive multiple view parameters associated with an uncaptured view, the uncaptured view being associated with at least one of the multiple input images; Multiple warped images are generated based on the multiple input images, the multiple view parameters, and at least one depth image among the multiple depth images; In response to providing the plurality of depth images, the plurality of view parameters, and the plurality of warped images to the neural network, a blending weight for assigning color to pixels of the uncaptured view is received from the neural network, wherein the neural network does not use positive distance when determining the blending weight; A composite image is generated based on the hybrid weights, wherein the composite image corresponds to the uncaptured view; and Occlusion detected in the synthesized image is corrected based on the depth difference between the geometric fusion model and the depth observed in the plurality of depth images.

10. The image processing system according to claim 9, wherein: The plurality of input images are color images captured by the image processing system according to predefined view parameters associated with the image processing system; and / or The plurality of depth images include a depth map associated with at least one camera that captured at least one of the plurality of input images and / or a depth map associated with a witness camera of the image processing system.

11. The image processing system according to claim 9, wherein, The blending weights are configured to assign a blending color to each pixel of the synthesized image.

12. The image processing system according to claim 9, wherein, The neural network is trained based on minimizing a loss function between the synthetic image generated by the neural network and a ground-based image captured by at least one eyewitness camera.

13. The image processing system according to any one of claims 9 to 12, wherein, The synthesized image is a novel view generated for 3D video conferencing.

14. A non-transitory machine-readable medium having instructions stored thereon, the instructions, when executed by a processor, causing a computing device to: Receive multiple input images; Receive multiple depth images captured by multiple depth cameras and associated with a target subject in at least one of the multiple input images, each of the depth images including at least one occlusion map, the occlusion map encoding a signed distance between a surface point determined to be closest to the target viewpoint corresponding to the synthetic image and the camera that captured the surface; Receive multiple view parameters for generating a virtual view of the target subject; Multiple warped images are generated based on the multiple input images, the multiple view parameters, and at least one depth image among the multiple depth images; In response to providing the plurality of depth images, the plurality of view parameters, and the plurality of warped images to the neural network, a blending weight is received from the neural network for assigning color to pixels of the virtual view of the target subject, wherein the neural network does not use positive distance when determining the blending weight; A composite image is generated based on the hybrid weights and the virtual view according to the view parameters; and Occlusion detected in the synthesized image is corrected based on the depth difference between the geometric fusion model and the depth observed in the plurality of depth images.

15. The non-transitory machine-readable medium according to claim 14, wherein: The plurality of input images are color images captured according to predefined view parameters associated with at least one camera that captured the plurality of input images; And / or The plurality of depth images include a depth map associated with at least one camera that captured at least one of the plurality of input images and / or a depth map associated with a ground-based image captured by at least one witness camera at the time corresponding to the capture of at least one of the plurality of input images.

16. The non-transitory machine-readable medium according to claim 14, wherein, The blending weights are configured to assign a blending color to each pixel of the synthesized image.

17. The non-transitory machine-readable medium according to claim 14, wherein, The neural network is trained based on minimizing a loss function between the synthetic image generated by the neural network and a ground-based image captured by at least one eyewitness camera.

18. The non-transitory machine-readable medium according to claim 14, wherein, The synthesized image is a novel view for 3D video conferencing.

19. The non-transitory machine-readable medium according to claim 14, wherein, The neural network is further configured to perform multi-resolution blending to assign pixel colors to pixels in the synthesized image. The multi-resolution blending trigger provides an image pyramid as input to the neural network to trigger the receipt of multi-resolution blending weights for multiple scales and opacity values ​​associated with each scale from the neural network.

20. The non-transitory machine-readable medium according to any one of claims 14 to 19, wherein, When executed by the processor, the instructions cause the computing device to perform the method according to any one of claims 2 to 3.

Images