Hierarchical view synthesis system and method

Abstract

A method of computer-implemented synthetic view image generation and a synthetic view image generation system provide hierarchical view synthesis. The method includes: receiving an input image having a plurality of pixels, the plurality of pixels having color values; generating an expanded depth map by expanding a depth map associated with the input image, the depth map having a depth value associated with each pixel in the input image, respectively; determining a repair mask using the expanded depth map; performing a repair operation based on the repair mask and the input image to generate a background image; and rendering the synthetic view image using the background image, the input image, and the expanded depth map.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 348,450, filed on June 2, 2022, the entire contents of which are incorporated herein by reference. Background Art

[0003] In order to perceive a scene in three dimensions, the left and right eyes see image views of the scene from slightly different perspectives. Each eye has a slightly different viewpoint, so that objects at different depths are "shifted" in position between the images perceived by the left and right eyes. Therefore, in order for the observer to perceive the image as a three-dimensional (3D) image, two different perspectives need to be presented to the two eyes. In AR / VR headsets, this is accomplished by displaying the left and right eye perspectives on the left and right screens of the glasses. Similarly, glasses-free 3D displays turn a separate view to each eye, allocating a subset of the display pixels to each view. In addition, a multi-view display can be provided in which different viewing perspectives are provided to three or more viewing directions, so that viewers perceive different perspective views as they move around the multi-view display.

[0004] At the same time, while 3D or multi-view cameras do exist, they typically capture a single two-dimensional (2D) image, providing a single perspective view of a scene. Therefore, it is desirable to take a single 2D image of a scene and generate images of one or more additional viewing perspectives so that the scene can be visualized in 3D.

[0005] Methods for generating synthetic perspective view images have been reported previously, but these methods often produce visual artifacts (e.g., striping artifacts and dilation artifacts) in the synthetic images. In addition, previously reported methods are often not robust to multi-level occlusion, i.e., different features in the image that partially occlude each other correspond to a range of depths. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] The various features of examples and embodiments according to the principles described herein may be more readily understood by reference to the following detailed description taken in conjunction with the accompanying drawings, in which like reference numerals represent like structural elements, and in which:

[0007] Figure 1a A flow chart showing steps of a method of computer-implemented synthetic view image generation, according to an embodiment consistent with principles described herein.

[0008] Figure 1b The interrelationships between different data objects used in the computer-implemented method of synthetic view image generation of FIG. 1 are shown.

[0009] Figure 2An example input image consistent with the principles described herein is shown.

[0010] Figure 3 An example depth map consistent with the principles described herein is shown.

[0011] Figure 4a Another example depth map consistent with the principles described herein is shown.

[0012] Figure 4b Shows Figure 4a A magnified portion of the depth map.

[0013] Figure 5a Shows the corresponding Figure 4a Example of a depth map of FIG. 10 . An example of a dilated depth map of FIG.

[0014] Figure 5b Shows Figure 5a An enlarged portion of the dilated depth map.

[0015] Figure 6 An image not derived by a method according to embodiments disclosed herein is shown, where streak artifacts are visible.

[0016] Figure 7 An example repair mask consistent with the principles described herein is shown.

[0017] Figure 8 An example background image is shown consistent with the principles described herein.

[0018] Fig. 9 An example foreground image is shown consistent with the principles described herein.

[0019] Fig.10 Example synthetic view images consistent with the principles described herein are shown.

[0020] Figures 11a to 11c Rendered synthetic view images are shown provided according to various methods, including computer-implemented methods of synthetic view image generation consistent with the principles described herein.

[0021] Fig.12 A schematic block diagram is shown depicting one example illustration of a computing device that may be used to perform a method of computer-implemented synthetic view image generation, according to an embodiment consistent with principles described herein.

[0022] Certain examples and embodiments have other features that are in addition to or in lieu of the features shown in the above drawings. These and other features are described in detail below with reference to the above drawings. DETAILED DESCRIPTION

[0023] According to examples and embodiments of the principles described herein, a computer-implemented method for generating a synthetic view image is provided. With the method described herein, an input image including a plurality of pixels having color values ​​is received, and a dilated depth map is generated by dilating a depth map associated with the input image, the depth map including depth values ​​associated with each pixel in the input image. The depth map can be generated based on the input image. A blending map can also be generated based on the depth map, the blending map including blending values ​​associated with each pixel in the depth map. A repair mask is determined using the dilated depth map, and a repair operation is performed based on the repair mask and the input image to generate a background image. A synthetic view image is then rendered using the background image, the input image, the dilated depth map, and (if the blending map has been generated) the blending map. A computer system and a computer program product are also described.

[0024] By the described method, it has been found that visual artifacts in the synthesized image can be mitigated or in some cases eliminated. Furthermore, the described method has been found to be more robust to artifacts caused by multiple levels of occlusion within the input image.

[0025] In this document, a "two-dimensional image" or "2D image" is defined as a set of pixels, each pixel having an associated intensity and / or color value. For example, a 2D image may be a 2D RGB image, where, for each pixel in the image, the relative intensity of red (R), green (G), and blue (B) is provided. A 2D image will typically represent a perspective view of a scene or object.

[0026] In contrast, a stereoscopic image is defined as a pair of images corresponding to perspective views of a scene or object observed from the viewpoint of each of the left and right eyes of a viewer, respectively. By further contrast, a "multi-view image" is an image comprising different view images, where each view image represents a different perspective view of the scene or object of the multi-view image. A multi-view image explicitly provides three or more perspective views.

[0027] As used herein, a "multi-view display" is defined as an electronic display or display system configured to provide different views of a multi-view image in or from different view directions. The multi-view display may be provided as part of a variety of devices, including but not limited to mobile phones (e.g., smart phones), watches, tablet computers, mobile computers (e.g., laptop computers), personal computers and computer monitors, automotive display consoles, camera displays, and various other mobile and substantially non-mobile display applications and devices. The multi-view display may display a multi-view image by providing different views of the multi-view image in different view directions relative to the multi-view display.

[0028] In this document, a "depth map" is defined as a map that provides information indicating the absolute or relative distance of objects depicted in an image to a camera (or equivalently to a viewpoint to which the image corresponds). By definition, a depth map includes a plurality of pixels, each pixel having a depth value, which is a value indicating the distance of an object at that pixel within the depth map relative to the viewpoint of the image. A depth map may have a one-to-one correspondence with an image, that is, for each pixel in the image, the depth map provides a depth value at the corresponding pixel. However, as will be appreciated, a depth map may provide a coarser granularity, and a depth map may have a lower resolution than a corresponding image, wherein each pixel within a depth map provides depth values ​​for multiple pixels within the image. A depth map having a lower resolution than its corresponding image may be referred to as a downsampled depth map.

[0029] The disparity map can be used in an equivalent manner to the depth map described above. Disparity refers to the apparent displacement of objects in a scene when viewed from two different viewpoints, such as from a left-eye viewpoint and a right-eye viewpoint. Disparity information and depth information are related and can be mapped onto each other through the geometry of the corresponding viewpoints of the disparity map. In view of this close relationship and the fact that one can be converted into the other, the terms "depth map" and "depth values" used throughout this specification are understood to include depth information as well as disparity information. That is, depth and disparity can be used interchangeably in the methods described below.

[0030] In this article, "occlusion" is defined as a foreground object in an image covering at least a portion of the background, making the background invisible. In addition, in this article, "disocclusion" is defined as the area of ​​the image that is no longer occluded by the foreground object when the position of the foreground object is moved from its original position within the image according to the shift of the viewpoint or perspective.

[0031] In addition, as used herein, the articles "a" and "an" are intended to have their ordinary meaning in the patent field, i.e., "one or more". For example, "an image" means one or more "images", and therefore, "the image" means "(one or more) images" in this article. In addition, any reference to "top", "bottom", "upper", "lower", "up", "lower", "front", "back", "first", "second", "left" or "right" in this article is not intended to be a limitation of this article. In this article, the term "about" when applied to a value generally means within the tolerance range of the device used to generate the value, or may mean plus or minus 10%, or plus or minus 5%, or plus or minus 1%, unless otherwise expressly stated. In addition, the term "substantially" as used herein means most, or almost all, or all, or an amount in the range of about 51% to about 100%. In addition, the examples in this article are intended to be illustrative only and are presented for discussion purposes rather than as limitations.

[0032] According to some embodiments of the principles described herein, there is provided a method of computer-implemented synthetic view image generation. Figure 1a A flow chart showing the steps of method 100 is shown. Figure 1b , which depicts the relationship between different data objects used and generated in the present method. The steps of method 100 (which will each be described in more detail below) are as follows.

[0033] First, in step 101, an input image 200 including a plurality of pixels having color values ​​is received. Then, in step 103, a dilated depth map 350 is generated by dilating a depth map 300 associated with the input image, the depth map including depth values ​​respectively associated with each pixel in the input image. The depth map 300 may be generated according to the input image 200, such as Figure 1a As shown in optional step 102 in Figure 1b , as shown by the dashed arrow connecting the input image 200 and the depth map 300. In some embodiments, in optional step 104, a blending map 360 is generated according to the depth map 300, and the blending map 360 includes blending values ​​respectively associated with each pixel in the depth map 300. In step 105, a repair mask 700 is determined using the dilated depth map 350. Next, in step 106, a repair operation is performed based on the repair mask and the input image to generate a background image 800. In step 107, a synthetic view image 1000 is rendered using the background image 800, the input image 200, and the dilated depth map 350. Rendering the synthetic view image 1000 may include generating a foreground image 900 using the input image 200 and the dilated depth map 350, and the foreground image 900 is combined with the background image 800, as shown in FIG. Figure 1b In an embodiment where the blending map 360 has been generated, the blending map 360 may also be used to render the synthetic view image 1000 .

[0034] In other words, in the method described herein, depth estimation can be performed based on a single input image. An inpainting mask can then be formed, where the inpainting mask highlights the areas that need inpainting so that the deocclusion can be filled later. The depth map is then dilated and a blending value is determined. Next, to render a synthetic view image, a foreground is rendered and an inpainted background image is rendered. The deocclusion holes in the foreground image are then filled using the background image so that a synthetic view image is rendered.

[0035] The method will now be explained in more detail, taking Figure 1a The steps of the method shown in .

[0036] First, in Figure 1aAt step 101 in , an input image 200 is received. The input image 200 may be a 2D RGB image. That is, for each pixel in the input image 200, a color value (e.g., red, green, and blue) is assigned. The input image 200 may be received from any number of sources. For example, the input image 200 may be captured by a 2D static camera. The input image 200 may be a single frame of a 2D video. The image 200 may be a generated image, such as an image generated by a deep learning model or a generative AI (e.g., OpenAI's DALL-E model, etc.).

[0037] To facilitate discussion of this method, Figure 2 An exemplary input image 200 is shown in . The image includes background 201 and foreground objects 202 and 203. In this exemplary image, foreground object 203 is in front of foreground object 202, which in turn is in front of background 201.

[0038] like Figure 1a As indicated by optional step 102 in , after receiving the input image 200, depth estimation can be performed on the image to generate a depth map 300. Monocular depth estimation techniques are capable of estimating dense depth based on a single 2D (RGB) image. Many methods directly utilize a single image or estimate an intermediate 3D representation (such as a point cloud). Some other methods combine a 2D image with, for example, a sparse depth map or a normal map to estimate a dense depth map. These methods are trained on a large-scale dataset generated including RGB-D images, which are images in which a color (RGB) value and a depth (D) value are provided for each pixel. One depth estimation technique suitable for the present method is the Midas technique disclosed in "Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer" IEEE transactions on pattern analysis and machine intelligence, 2020 by Ranftl et al., which is incorporated herein by reference. Depth estimation techniques may provide a depth value for each pixel within the input image, such that depth map 300 includes a depth value associated with each pixel in the input image, each depth value being an estimate of the depth associated with an object at that pixel in the image.

[0039] Of course, it should be understood that the depth map 300 may not be generated based on the received input image, but instead provided by other means. For example, the depth map 300 may be captured when the input image is captured using a depth sensor (such as a time-of-flight sensor, etc.). As another example, the depth map 300 may be generated by a different application or by an operating system when the input image is captured or later. In either case, the depth map 300 may be received together with the input image.

[0040] To facilitate discussion of this method, Figure 3 , where the shading of each pixel in the depth map 300 represents the depth value (i.e., estimated depth) of each corresponding pixel in the input image 200, where darker shading indicates larger depth values ​​(i.e., at locations of the imaged scene farther from the "viewer"), and lighter shading indicates smaller depth values ​​(i.e., at locations closer to the "viewer"). In this case, region 301 of the depth map 300 corresponds to the background, region 302 corresponds to the foreground object 202, and region 303 corresponds to the foreground object 203.

[0041] Typically, the depth values ​​in a depth map will not have sharp (or step-like) transitions from foreground to background depth. Instead, there will be visible transitional depth values ​​near the edges of objects. This is especially true for depth maps that have a lot of depth information. Figure 4a and Figure 4b Shown in. Figure 4a Another exemplary depth map 400 is shown, and Figure 4b is corresponding to Figure 4a FIG. 4 is an enlarged portion of the depth map 400 as shown in FIG. Figure 4b As can be seen in FIG, between the brightest shadow area (right-hand side of the image, corresponding to the foreground object) and the darkest shadow area (left-hand side of the image, corresponding to the background area of ​​the image), there are pixels with transitional depth values, i.e., pixels with depth values ​​that fall between the depth values ​​of the foreground object and the background area. The inventors of the present invention have recognized that these foreground-background transitions may cause visual artifacts when rendering a new view in a synthesized image using forward or backward mapping.

[0042] For example, when forward mapping is used, streaking artifacts may appear due to transitional depth values. This is because each transitional depth value causes a slightly further displacement of the associated pixel of the foreground object, spread across the region being disoccluded. Additionally, the edges of the foreground object are corrupted because some pixels near the edge may be displaced away from the rest of the object. Figure 6 An example of this streaking artifact is shown in , which shows a forward mapped rendering of a foreground image (without inpainting the deoccluded areas). As can be seen, "streaks" of pixels are visible in the unoccluded areas near the foreground object.

[0043] In the present method, after the depth map 300 has been received with or generated from an input image, a dilated depth map 350 is generated from the image at step 103. Generating a dilated depth map may provide sharp transitions between regions of different depth values.

[0044] Generally speaking, the process of generating the dilated depth map 350 according to the depth map 300 is to convert the gradual transition between the foreground area and the background area in the depth map 300 into a sharp transition in the dilated depth map 350 .

[0045] In some embodiments, the process for generating the dilated depth map 350 is as follows.

[0046] A local minimum depth value and a local maximum depth value are identified. Transition depth values ​​are also identified, each transition depth value having a value between the local minimum depth value and the local maximum depth value. For pixels in the depth map 300 having a transition depth value, the depth value of the corresponding pixel in the dilated depth map 350 is set to the local maximum depth value.

[0047] In some embodiments, this is only performed when the difference between the local maximum depth value and the local minimum depth value exceeds a certain threshold depth difference. That is, where the transition depth value falls within a small range of depth values ​​(defined by the threshold difference in depth values), then the pixel in the dilated depth map corresponding to the pixel in the depth map having the transition depth value is not set to the local maximum, but is instead set to the transition depth value of the corresponding pixel in the depth map 300. This can help limit the computational requirements of the method.

[0048] For pixels in the depth map 300 having a local minimum depth value or a local maximum depth value, the corresponding pixels in the dilated depth map 350 are set to the local minimum depth value and the local maximum depth value, respectively.

[0049] The above process can be applied iteratively on multiple regions within the image.

[0050] Figure 5a and Figure 5b An exemplary dilated depth map is shown in . Figure 5a A dilated depth map 500 corresponding to the depth map 400 is shown. Figure 5b is corresponding to Figure 5a The enlarged portion of the depth map 500 is the dashed rectangle in FIG. Figure 5b and Figure 4b By comparison, it can be seen that a sharp transition between foreground objects and background areas has been provided in the enlarged depth map. Figure 5a and Figure 4aIt can also be seen from the comparison that for image regions with a smoother depth value transition, the smooth transition is maintained between the depth map 400 and the dilated depth map 500 (e.g., between the depth map 400 and the dilated depth map 500). Figure 5a ).

[0051] In some embodiments of the method, at optional step 104, a blending map 360 is generated from the depth map 300. The blending map will be used to blend the transition between the foreground area and the background area in the final rendered synthetic view image 1000. Using the blending map 360, any dilation artifacts that may otherwise be visible after rendering can be mitigated or even completely avoided. The blending map 360 includes a blending value for each pixel in the input image. The blending map 360 can be used as an alpha mask for rendering the synthetic view image at step 107. In this way, the blending map divides the image into three regions: a background region (corresponding to a minimum blending value, e.g., α=0.0), a foreground region (corresponding to a maximum blending value, e.g., α=1.0), and a transition region (corresponding to blending values ​​between the maximum blending value and the minimum blending value, e.g., 0.0<α<1.0). When rendering the synthetic view image, the blending map 360 can be applied as an alpha mask to smooth the transition between the foreground layer and the background layer, where the blending value determines the opacity of the foreground pixels covering the background layer. For example, a foreground pixel corresponding to a blend value of α = 0.0 may be fully transparent (i.e., for this pixel in the rendered image, only the color information from the background layer is used), and a foreground pixel corresponding to a blend value of α = 1.0 may be opaque (i.e., for this pixel in the rendered image, only the color information from the foreground layer is used). For pixels corresponding to intermediate blend values ​​of 0.0 < α < 1.0, the foreground pixel will be partially transparent (i.e., for this pixel in the rendered image, the color information for each RGB channel will be the value of the corresponding channel in the foreground pixel multiplied by the value of α plus the value of the corresponding channel in the background pixel multiplied by (1-α)).

[0052] A blend map 360 may be generated by determining a local minimum depth value, a local maximum depth value, and a transition depth value, each transition depth value having a value between the local minimum depth value and the local maximum depth value. The blend value of each corresponding pixel in the blend map is set by scaling the depth value, such that the local maximum depth value is scaled to a global maximum blend value (e.g., α=1.0) and the local minimum depth value is scaled to a global minimum blend value (e.g., α=0.0). The transition depth value is scaled to a value between the global maximum blend value and the global minimum blend value (e.g., 0.0<α<1.0). This process may be iterated over multiple regions within the image.

[0053] In the present method, at step 105, an inpainting mask 700 is determined based on the dilated depth map 350. The inpainting mask 700 identifies areas of the input image 200 that may become unoccluded when a transformation corresponding to a shift in the perspective view is applied. For each pixel in the input image 200, the inpainting mask 700 includes a value indicating whether the pixel will be inpainted in the inpainting operation. In other words, these are areas in the image that may become unoccluded in the foreground image when the foreground object moves according to the shift in the perspective view. The inpainting mask 700 identifies areas of the input image that will be inpainted to provide a background image.

[0054] In some embodiments, the inpainting mask 700 may be generated by identifying depth transitions in the dilated depth mask 350 that exceed a threshold depth difference; and adding one or more pixels to the inpainting mask 700, the one or more added pixels corresponding to pixels of the dilated depth mask 350 that are adjacent to the transition and on the side of the transition that has a lower depth value. That is, where a sharp transition in depth is identified in the dilated depth map 350, pixels are added to the inpainting mask 700 adjacent to the location of the transition on the shallower side of the transition. The threshold depth difference used in this step may be the same threshold depth difference used when generating the dilated depth map at step 103, or may be a different threshold depth difference.

[0055] In some embodiments, only transitions in one direction (horizontal or vertical) are identified, and one or more added pixels are in the horizontal direction or vertical direction, respectively, relative to the transition. This may be achieved in cases where only horizontal parallax or vertical parallax is to be provided from the synthetic view image 1000 (that is, where the shift in perspective view will only be in the horizontal direction or vertical direction), since only areas of the image adjacent to the depth transition in the direction of the perspective shift will potentially be disoccluded.

[0056] In other words, to generate horizontally spaced views, the process iterates over the dilated depth map and whenever a sudden increase or decrease is reached, the mask is horizontally positioned at the pixels on the higher side of that transition (ie, the side with the lower depth value).

[0057] To facilitate discussion of this method, Figure 7 An exemplary inpainting mask 700 derived from the depth map 300 is shown in FIG. White areas in the inpainting mask 700 indicate areas to be inpainted in an inpainting operation. In this example, only horizontal depth transitions in the depth map 300 have been identified to add pixels to the inpainting mask.

[0058] After the inpainting mask has been generated, at step 106, an inpainting operation is performed to generate a background image 900. In some embodiments, this is achieved by providing the input image 200 and the inpainting mask 700 to an inpainting neural network.

[0059] In some embodiments, the inpainting network is a depth-aware inpainting network. By depth-aware inpainting, it is meant that both color values ​​and depth values ​​are generated for the region of the background image being inpainted. The input image 200 is provided as an RGB-d image (i.e., each pixel has RGB color information and a depth value D derived from the depth map 300 or from the dilated depth map 350). The inpainting network will inpaint the image region defined by the inpainting mask to generate a color (RGB) value and a depth value for each pixel in the inpainted region.

[0060] In some embodiments, the inpainting network is a generative adversarial network (GAN). Multiple suitable inpainting networks can be used. One such network is the LaMa inpainting network disclosed in “Large scale image completion via co-modulated generative adversarial networks” by Zhao et al. (International Conference on Learning Representations (ICLR), 2021), which is incorporated herein by reference.

[0061] The LaMa network can be modified for RGB-D inpainting and trained on a combination of random inpainting masks (i.e., masks that include randomly generated mask regions) and disocclusion inpainting masks (i.e., masks that have been derived from the inpainting mask generation process described above). Using random inpainting masks (in addition to disocclusion inpainting masks) allows for better training of general inpainting, which allows the network to handle larger masks that may occur on multi-level disocclusion.

[0062] In some embodiments, a second image inpainting operation is used that is different from the first image inpainting operation, wherein the first image inpainting operation generates pixels having depth values ​​that indicate the presence of multiple levels of disocclusion when compared to reference depth values ​​derived from the dilated depth map 350. Alternatively, in some other embodiments, the reference depth values ​​may be derived from the depth map 300.

[0063] In some embodiments, for example, the reference depth value is the depth value of the pixel on the darker side of the transition (i.e., the side of the depth transition with the larger depth value). The depth value of the pixel generated in the inpainting operation can be compared to the reference value. In the case where the depth value difference between the inpainted pixel and the reference depth value exceeds a certain threshold difference in depth values, then multi-level removal of occlusion can be assumed, in which case a different inpainting operation can be used. For example, a simple reflection inpainting can be used as the second inpainting operation.

[0064] To facilitate discussion of the method, an exemplary background image 800 is shown in FIG. Figure 8 , and is the output of an inpainting operation using the inpainting mask 700 and the input image 200. Region 805 corresponds to the region identified in the inpainting mask that has been inpainted by the inpainting operation.

[0065] After the background image has been generated, the process may continue at step 107 to render a synthetic view image 1000 corresponding to an image having a different viewpoint than the input image. A transform may be applied to the input image 200 using the depth values ​​from the dilated depth map 350 to generate the foreground image 900. This may be accomplished by calculating, for each pixel in the input image 200, a shift in the position of the pixel within the image that will result from the position change of the viewpoint and the depth value of the pixel from the dilated depth map 350. Each pixel is shifted according to the position change calculated from the depth value in the depth map to generate the foreground image 900 (that is, the color information from one pixel is transposed to another pixel according to the calculated position change). According to the shift in position, this causes a shift in the position of the pixel group corresponding to the object at the foreground depth, and will also cause a deocclusion hole consisting of a pixel area that is deoccluded due to the viewpoint difference between the foreground image and the input image.

[0066] To facilitate discussion of this method, Fig. 9 An exemplary foreground image 900 is shown in Figure 3 The depth map of the dilated depth map derived from Figure 2 350. The foreground objects 202 and 203 have been horizontally shifted in position according to the change in viewpoint compared to the input image. The horizontal shift in the position of pixels associated with this object between the input image 200 and the foreground image 800 corresponds to the depth values ​​of those pixels in the dilated depth map 350. It can be seen that a deocclusion hole 905 (shown in dark grey) has been left in the image.

[0067] The deocclusion holes are filled using information from the background image 900 by filling the deocclusion holes in the foreground image with information from the corresponding pixels of the background image. In some embodiments, before filling the deocclusion holes from the background image, a transformation is applied to the background image based on the change in viewpoint from the input image (i.e., the pixel is shifted according to the depth associated with that pixel in the depth map).

[0068] In embodiments where a blend map 360 has been generated, the blend map is used to smooth transitions between regions of an image derived from the foreground image and the background image. As described above in the discussion of step 104, the blend map is applied as an alpha mask.

[0069] In some embodiments, an inpainting operation may be performed to fill any holes left near the edges of the rendered synthetic view image 1000. Such holes may appear because neither the foreground image nor the background image will be mapped to these areas. Because these holes are relatively small and near the edges of the image, reflection inpainting is used in these remaining areas. This inpainting method is computationally cheap and effective for this task.

[0070] To facilitate discussion of this method, Fig.10 An exemplary composite view image 1000 based on the foreground image 900 and the background image 800 is shown in FIG. The disocclusion holes 905 have been filled using information from the background image 800.

[0071] Once rendered, the synthetic view image 1000 may be displayed on a display screen. The synthetic view image may be displayed as part of a stereoscopic image pair (on a stereoscopic display, virtual reality headset, etc.) with the input image or with another synthetic view image corresponding to a perspective view from a different viewpoint.

[0072] It should be understood that a plurality of synthetic view images may be provided, each synthetic view image corresponding to a perspective view from a different viewpoint. These may be displayed on a multi-view display as a set of different views of a multi-view image. The input image may or may not provide one of the views of the multi-view image.

[0073] Fig.121 is a schematic block diagram of an example illustration of a computing device 1200 that provides a multi-view display according to various embodiments of the present disclosure. The computing device 1200 may include a system of components that perform various computing operations for a user of the computing device 1200. The computing device 1200 may be a laptop, a tablet, a smart phone, a touch screen system, an intelligent display system, or other client device. The computing device 1200 may include various components, such as, for example, (one or more) processors 1203, a memory 1206, (one or more) input / output (I / O) components 1209, a display 1212, and possibly other components. These components may be coupled to a bus 1215, which serves as a local interface to allow the components of the computing device 1200 to communicate with each other. Although the components of the computing device 1200 are shown as being contained within the computing device 1200, it should be understood that at least some of the components may be coupled to the computing device 1200 via an external connection. For example, a component may be plugged into the computing device 1200 from the outside or otherwise connected to the computing device 1200 via an external port, socket, plug, or connector.

[0074] The processor 1203 may be a central processing unit (CPU), a graphics processing unit (GPU), or any other integrated circuit that performs computational processing operations. The processor(s) 1203 may include one or more processing cores. The processor(s) 1203 include circuits that execute instructions. Instructions include, for example, computer code, programs, logic, or other machine-readable instructions that are received and executed by the processor(s) 1203 to perform the computational functions embodied in the instructions. The processor(s) 1203 may execute instructions to operate on data. For example, the processor(s) 1203 may receive input data (e.g., input images), process the input data according to an instruction set, and generate output data (e.g., synthetic view images). As another example, the processor(s) 1203 may receive instructions and generate new instructions for subsequent execution.

[0075] Memory 1206 may include one or more memory components. Memory 1206 is defined herein as including either or both of volatile and non-volatile memory. Volatile memory components are those memories that do not retain information when power is off. Volatile memory may include, for example, random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), magnetic random access memory (MRAM), or other volatile memory structures. System memory (e.g., main memory, cache, etc.) may be implemented using volatile memory. System memory refers to fast memory that can temporarily store data or instructions for fast read and write access to assist (one or more) processors 1203.

[0076] A non-volatile memory component is a component that retains information when power is removed. Non-volatile memory includes read-only memory (ROM), a hard drive, a solid-state drive, a USB flash drive, a memory card accessed via a memory card reader, a floppy disk accessed via an associated floppy disk drive, an optical disk accessed via an optical drive, a magnetic tape accessed via an appropriate magnetic tape drive. ROM may include, for example, a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or other similar memory devices. Storage memory may be implemented using non-volatile memory to provide long-term retention of data and instructions.

[0077] The memory 1206 may refer to a combination of volatile and non-volatile memory for storing instructions and data. For example, data and instructions may be stored in non-volatile memory and loaded into volatile memory for processing by (one or more) processors 1203. The execution of instructions may include, for example, a compiler, source code, source code, etc., wherein the compiler is translated into machine code, the format of which can be loaded from non-volatile memory into volatile memory and then run by the processor 1203, the source code is converted in a suitable format into an object code such as an object code that can be loaded into volatile memory for execution by the processor 1203, the source code is interpreted by another executable program to generate instructions in volatile memory and executed by the processor 1203. Instructions may be stored or loaded in any part or component of the memory 1206, which includes, for example, RAM, ROM, system memory, storage devices, or any combination thereof.

[0078] Although the memory 1206 is shown as being separate from the other components of the computing device 1200, it should be understood that the memory 1206 may be at least partially embedded or otherwise integrated into one or more components. For example, the processor(s) 1203 may include onboard memory registers or caches to perform processing operations.

[0079] The I / O component(s) 1209 include, for example, a touch screen, a speaker, a microphone, a button, a switch, a dial, a camera, a sensor, an accelerometer, or other components that receive user input or generate output directed to the user. The I / O component(s) 1209 may receive user input and convert it into data to be stored in the memory 1206 or processed by the processor(s) 1203. The I / O component(s) 1209 may receive data output by the memory 1206 or the processor 1203 and convert them into a format perceived by the user (e.g., sound, tactile response, visual information, etc.).

[0080] A specific type of I / O component 1209 is a display 1212. The display 1212 may include a multi-view display, a multi-view display combined with a 2D display, or any other display that presents an image. A capacitive touch screen layer used as an I / O component 1209 may be layered within the display to allow a user to provide input while simultaneously perceiving visual output. The processor(s) 1203 may generate data formatted as an image for presentation on the display 1212. The processor(s) 1203 may execute instructions to render an image on the display for user perception.

[0081] The bus 1215 facilitates communication of instructions and data between the processor(s) 1203, the memory 1206, the I / O component(s) 1209, the display 1212, and any other components of the computing device 1200. The bus 1215 may include address translators, address decoders, structures, conductive traces, conductive lines, ports, plugs, sockets, and other connectors to enable communication of data and instructions.

[0082] The instructions within the memory 1206 may be embodied in various forms in a manner that implements at least a portion of a software stack. For example, the instructions may be embodied as an operating system 1231, (one or more) applications 1234, device drivers (e.g., display drivers 1237), firmware (e.g., display firmware 1240), or other software components. The operating system 1231 is a software platform that supports the basic functions of the computing device 1200, such as scheduling tasks, controlling I / O components 1209, providing access to hardware resources, managing power, and supporting applications 1234.

[0083] (One or more) applications 1234 are executed on the operating system 1231, and can obtain access to the hardware resources of the computing device 1200 via the operating system 1231. In this regard, the execution of (one or more) applications 1234 is at least partially controlled by the operating system 1231. (One or more) applications 1234 can be user-level software programs that provide advanced functions, services and other functions to users. In some embodiments, the application 1234 can be a dedicated "app" that can be downloaded or otherwise accessed by the user on the computing device 1200. The user can start (one or more) applications 1234 via the user interface provided by the operating system 1231. Applications 1234 can be developed by developers and defined in various source code formats. Applications 1234 can be developed using a variety of programming or scripting languages, such as, for example, C, C++, C#, Objective C, Java®, Swift, JavaScript®, Perl, PHP, Visual Basic®, Python®, Ruby, Go or other programming languages. (One or more) applications 1234 can be compiled into target code by a compiler or interpreted by an interpreter for execution by (one or more) processors 1203.

[0084] Device drivers, such as, for example, display drivers 1237, include instructions that allow the operating system 1231 to communicate with various I / O components 1209. Each I / O component 1209 may have its own device driver. Device drivers may be installed so that they are stored in a storage device and loaded into system memory. For example, after installation, the display driver 1237 converts high-level display instructions received from the operating system 1231 into low-level instructions implemented by the display 1212 to display an image.

[0085] Firmware (such as, for example, display firmware 1240) may include machine code or assembly code that allows the I / O component 1209 or the display 1212 to perform low-level operations. The firmware may convert electrical signals of a particular component into higher-level instructions or data. For example, the display firmware 1240 may control how the display 1212 activates individual pixels at a low level by adjusting a voltage or current signal. The firmware may be stored in a non-volatile memory and executed directly from the non-volatile memory. For example, the display firmware 1240 may be embodied in a ROM chip coupled to the display 1212, such that the ROM chip is separated from other storage and system memory of the computing device 1200. The display 1212 may include processing circuitry for executing the display firmware 1240.

[0086] The operating system 1231, (one or more) applications 1234, drivers (e.g., display driver 1237), firmware (e.g., display firmware 1240), and potentially other instruction sets may each include instructions that can be executed by (one or more) processors 1203 or other processing circuits of the computing device 1200 to perform the functions and operations discussed above. Although the instructions described herein may be embodied in software or code executed by (one or more) processors 1203, as described above, as an alternative, the instructions may also be embodied in dedicated hardware or a combination of software and dedicated hardware. For example, the functions and operations performed by the instructions discussed above may be implemented as circuits or state machines using any one or a combination of a variety of technologies. These technologies may include, but are not limited to, discrete logic circuits having logic gates for implementing various logic functions when one or more data signals are applied, application specific integrated circuits (ASICs) with appropriate logic gates, field programmable gate arrays (FPGAs), or other components, etc.

[0087] In some embodiments, instructions to perform the functions and operations discussed above may be embodied in a non-transitory computer-readable storage medium. The computer-readable storage medium may or may not be part of the computing device 1200. Instructions may include, for example, statements, codes, or declarations that may be retrieved from a computer-readable medium and executed by a processing circuit (e.g., processor(s) 1203). In the context of the present disclosure, a "computer-readable medium" may be any medium that may contain, store, or maintain the instructions described herein for use by or in connection with an instruction execution system (e.g., computing device 1200).

[0088] Computer-readable media may include any of a number of physical media, such as magnetic, optical, or semiconductor media. More specific examples of suitable computer-readable media may include, but are not limited to, magnetic tape, magnetic floppy disk, magnetic hard drive, memory card, solid-state drive, USB flash drive, or optical disk. In addition, the computer-readable medium may be a random access memory (RAM), including, for example, static random access memory (SRAM) and dynamic random access memory (DRAM) or magnetic random access memory (MRAM). In addition, the computer-readable medium may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or other types of memory devices.

[0089] The computing device 1200 may perform any of the operations or implement the above-described functions. For example, the flowcharts and process flows discussed above may be performed by a computing device 1200 that executes instructions and processes data. Although the computing device 1200 is shown as a single device, the present disclosure is not limited thereto. In some embodiments, the computing device 1200 may offload processing of instructions in a distributed manner so that multiple computing devices 1200 operate together to execute instructions that may be stored or loaded in a distributed arrangement. For example, at least some instructions or data may be stored, loaded, or executed in a cloud-based system operating in conjunction with the computing device 1200.

[0090] The present disclosure also provides a computer program product corresponding to each embodiment of the method for computer-implemented synthetic view image generation described herein. Such a computer program product includes instructions that, when executed by a computer, cause the computer to implement any method disclosed herein. The computer program product can be embodied in a non-transitory computer-readable storage medium.

[0091] Experimental Results

[0092] The present method for synthetic view image generation is compared with previously reported methods. In particular, the method including steps 101, 102, 103, 104, 105, 106 and 107 is compared with two previously reported methods, namely the SynSin method disclosed in "SynSin: End-to-end view synthesis from a single image" by Wiles et al., Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pp. 7467-7477, 2020, and the Slide method disclosed in "Slide: Single image 3D photography with soft layering and depth-aware inpainting" by Jampani et al., Proceedings of the IEEE / CVF International Conference on Computer Vision, pp. 12518-12527, 2021. The two previously reported methods and the present method are applied to the Holopix50k dataset, and multiple metrics are used to determine the effectiveness of each method. The Holopix50 dataset is publicly available in Hua et al., “Holopix50k: A large-scale in-the-wild stereo image dataset”, arXiv preprint arXiv:2003.11172, 2020.

[0093] The following four evaluation metrics are determined: 1. Mean squared error (MSE). 2. Peak signal-to-noise ratio (PSNR). 3. Structural similarity index measure (SSIM). 4. Learned perceptual image patch similarity (LPIPS). The details of these metrics are given in Zhang et al., “The Unrasonated Effectiveness of Deep Features as a Perceptual Metric”, CVPR, 2018, which is incorporated by reference into this paper. The results are shown in Table 1.

[0094]

[0095] Table 1. Evaluation of our method compared to previously reported view synthesis methods and standard completion on the Holopix50k dataset.

[0096] From the results in Table 1, it can be seen that the present method provides improvements over previously reported methods according to all four metrics.

[0097] The improvements over previously reported methods are also apparent from qualitative analysis through visual comparison of renderings of synthetic view images from different methods. Figure 11a-Figure 11c The same rendered composite view images provided from different methods are shown, where Fig.11a shows the image produced by the SynSin method, Fig.11b shows the image produced by the Slide method, and Fig.11c Images produced by the present method are shown. In each case, a magnified view of the area indicated by the solid rectangle is provided. Fig.11c Images and Fig.11a As can be seen from the comparison of the images, the present method does not cause significant distortion, as is the case with the images produced by the SynSin method (see for example by Fig.11a The dotted rectangle in the figure indicates the distortion of the traffic light. Fig.11c The enlarged part and Fig.11b As can be seen from the comparison of the enlarged parts of , the images of the present method do not have the visual artifacts produced by the Slide method. Therefore, it should be understood that the present method provides an improvement over the two previously reported methods in terms of qualitative and quantitative evaluation.

[0098] Thus, examples and embodiments of methods and systems for providing hierarchical view synthesis have been described. The above examples are merely illustrations of some of the many specific examples of the principles described herein. Obviously, those skilled in the art can easily design many other arrangements without departing from the scope defined by the appended claims.

Claims

1. A computer-implemented method for generating a synthetic view image, the method comprising: receiving an input image comprising a plurality of pixels having color values; generating a dilated depth map by dilating a depth map associated with the input image, the depth map comprising depth values ​​respectively associated with each pixel in the input image; determining an inpainting mask using the dilated depth map; performing an inpainting operation based on the inpainting mask and the input image to generate a background image; as well as A synthetic view image is rendered using the background image, the input image, and the dilated depth map.

2. The computer-implemented method of generating a synthetic view image according to claim 1, further comprising: A depth map for the input image is generated by performing depth estimation within the image to determine depth values ​​respectively associated with each pixel.

3. The method according to claim 1, wherein: Generating a dilated depth map includes: A local minimum depth value, a local maximum depth value, and a transition depth value are determined, each transition depth value having a value between the local minimum depth value and the local maximum depth value.

4. The computer-implemented method of generating a synthetic view image according to claim 3, wherein: Generating a dilated depth map also includes: The depth value of the pixel in the dilated depth corresponding to the pixel having the transition depth value in the depth map is set to the local maximum depth value.

5. The computer-implemented method of generating a synthetic view image according to claim 4, wherein: Setting the depth value of the pixel in the dilated depth map is performed when a difference between the local minimum depth value and the local maximum depth value of the depth map exceeds a predetermined threshold difference in depth values.

6. The method of computer-implemented synthetic view image generation of claim 1 , further comprising generating a blending map from the depth map, the blending map comprising blending values ​​respectively associated with each pixel in the depth map, and wherein The composite view image is rendered using the blending map.

7. The computer-implemented method of generating a synthetic view image according to claim 6, wherein: Generating a hybrid graph involves: determining a local minimum depth value, a local maximum depth value, and a transition depth value, each transition depth value having a value between the local minimum depth value and the local maximum depth value; and The mixing value of each corresponding pixel in the mixing map is set by scaling the depth value, so that the local maximum depth value is scaled to the global maximum mixing value, and the local minimum depth value is scaled to the global minimum mixing value.

8. The computer-implemented method of generating a synthetic view image according to claim 6, wherein: Rendering the synthetic view image comprises: generating a foreground image according to the input image and the dilated depth map; and generating the synthetic view image by combining the foreground image with the background image, The generating of the synthetic view image comprises: smoothing the transition between the area of ​​the synthetic view image corresponding to the foreground image and the area of ​​the synthetic view image corresponding to the background image according to the mixing value in the mixing map.

9. The computer-implemented method of generating a synthetic view image according to claim 8, wherein: The foreground image corresponds to an image having a different viewpoint from the input image, the foreground image having a deocclusion hole composed of a pixel group corresponding to an area deoccluded due to a viewpoint difference between the foreground image and the input image.

10. The method of computer-implemented synthetic view image generation of claim 9, wherein combining the foreground image with the background image comprises: The deocclusion holes in the foreground image are filled with information from corresponding pixels of the background image.

11. The computer-implemented method of generating a synthetic view image according to claim 10, wherein: The background image includes a depth value associated with each pixel, and filling the deocclusion hole in the foreground image includes: A transformation is applied to the background image based on different viewpoints. 12 . The method of computer-implemented synthetic view image generation of claim 8 , wherein smoothing the transitions between regions of the synthetic view image comprises using a blending mask as an alpha mask.

13. The computer-implemented method of synthetic view image generation according to claim 8, wherein: Rendering each of the one or more synthetic view images further comprises: An inpainting operation is performed to fill holes left near edges of the image, wherein the inpainting operation is a reflective inpainting operation.

14. The computer-implemented method of synthetic view image generation according to claim 1, wherein: Determine the repair mask including: identifying a depth transition in the dilated depth map that exceeds a threshold depth difference; and One or more pixels are added to the inpainting mask, the one or more added pixels corresponding to pixels of the dilated depth map adjacent to the transition and on a side of the transition having a lower depth value.

15. The method for computer-implemented synthetic view image generation of claim 14, wherein the identified depth transition is a transition in only one of a horizontal direction or a vertical direction within the depth map, and wherein the one or more added pixels are in the horizontal direction or the vertical direction, respectively, relative to the transition.

16. The computer-implemented method of synthetic view image generation according to claim 1, wherein: For each pixel in the input image, the inpainting mask includes a value indicating whether the pixel is to be inpainted in the inpainting operation.

17. The computer-implemented method of synthetic view image generation according to claim 1, wherein: The steps of performing the repair operation include: An input image and the inpainting mask are provided to an inpainting network, which generates the background image, wherein the inpainting network is a depth-aware inpainting network.

18. The computer-implemented method of synthetic view image generation according to claim 17, wherein: The repair network has been trained on data including: a first data set comprising images and associated corresponding deocclusion masks; and A second data set comprising images and associated corresponding random inpainting masks.

19. The computer-implemented method of synthetic view image generation according to claim 1, wherein: Performing a repair operation also includes: The depth value generated for each repaired pixel is compared with the depth value of the reference depth value derived from the dilated depth map, and when the difference between the depth value generated for the repaired pixel and the depth value of the corresponding pixel in the depth map exceeds a threshold depth difference, a second repair operation is used to replace the pixel in the background image, preferably, wherein the second repair operation is a reflection repair operation.

20. The computer-implemented method of synthetic view image generation of claim 1, further comprising displaying at least one of the one or more synthetic view images on a display screen.

21. The computer-implemented method of synthetic view image generation of claim 20, wherein the display screen is a multi-view display screen.

22. A computer program product comprising instructions, which, when executed by a computer, causes the computer to generate a synthetic view image by: generating a dilated depth map by dilating a depth map associated with a received input image, the input image comprising a plurality of pixels having color values, and the depth map comprising depth values ​​respectively associated with each pixel in the input image; determining an inpainting mask using the dilated depth map; performing an inpainting operation based on the inpainting mask and the input image to generate a background image; as well as A synthetic view image is rendered using the background image, the input image, and the dilated depth map.

23. The computer program product of claim 22, wherein: When the program is executed by a computer, the instructions cause the computer to further implement synthetic view image generation by: A depth map for the input image is generated by performing depth estimation within the image to determine depth values ​​respectively associated with each pixel.

24. The computer program product of claim 22, wherein: When the program is executed by a computer, the instructions cause the computer to further implement synthetic view image generation by: A blending map is generated from the depth map, the blending map comprising blending values ​​respectively associated with each pixel in the depth map, and wherein the synthetic view image is rendered using the blending map.

25. A synthetic view image generation system, the system comprising: processor; as well as a memory storing instructions that, when executed, cause the processor to: generating a dilated depth map by dilating a depth map associated with a received input image, the input image comprising a plurality of pixels having color values, and the depth map comprising depth values ​​respectively associated with each pixel in the input image; determining an inpainting mask using the dilated depth map; performing an inpainting operation based on the inpainting mask and the input image to generate a background image; as well as A synthetic view image is rendered using the background image, the input image, and the dilated depth map.

26. The synthetic view image generation system of claim 25, further comprising a multi-view display screen, and wherein: The plurality of instructions, when executed, further cause the synthetic view image to be displayed on the multi-view display screen.

Images