Hierarchical view synthesis system and method

Abstract

A method for computer-implemented synthetic view image generation and a synthetic view image generation system provide hierarchical view synthesis. The method includes receiving an input image including a plurality of pixels having color values, generating an inflated depth map by inflating a depth map associated with the input image, where the depth map has depth values respectively associated with each pixel in the input image, determining a repair mask using the inflated depth map, performing a repair operation based on the repair mask and the input image to generate a background image, and rendering a synthetic view image using the background image, the input image, and the inflated depth map.

Description

Technical Field

[0001] Cross - Reference to Related Applications This application claims the benefit of U.S. Provisional Patent Application No. 63 / 348,450, filed on June 2, 2022, the entire content of which is incorporated herein by reference.

Background Art

[0002] To perceive a scene in three dimensions, the left and right eyes view the image views of the scene from slightly different viewpoints. Each eye has a slightly different viewpoint, and objects at different depths have their positions "shift" between the images perceived by the left and right eyes. Therefore, in order for an observer to perceive an image as a three - dimensional (3D) image, it is necessary to present two different viewpoints to both eyes. In AR / VR headsets, this is done by displaying the viewpoints of the left and right eyes on the left and right screens of the glasses. Similarly, a 3D display without glasses directs a separate view to each eye and assigns a subset of the display pixels to each view. Furthermore, a multi - view display can be provided that provides different view viewpoints in three or more view directions so that different viewpoint views are perceived when a viewer moves around the multi - view display.

[0003] On the other hand, while 3D or multi - view cameras exist, it is common to obtain a single two - dimensional (2D) image that provides a single viewpoint view of the scene. Therefore, it is desirable to generate an image of one or more additional view viewpoints so that a single 2D image of the scene can be obtained and the scene can be visualized in 3D.

[0004] Methods for generating synthetic view images have been reported previously, but these methods often produce visual artifacts in the synthetic images, such as striping and dilation artifacts. Further, previously reported methods are not generally robust to multi-level occlusion, i.e., different features in an image that partially occlude each other correspond to a range of depths.

Brief Description of the Drawings

[0005] The various features of the examples and embodiments according to the principles described herein can be more readily understood by reference to the following detailed description in conjunction with the accompanying drawings, where like reference numerals indicate like structural elements.

[0006]

Figure 1a

[0007]

Figure 1b

[0008]

Figure 2

[0009]

Figure 3

[0010]

Figure 4a

[0011]

Figure 4b

[0012]

Figure 5a

[0013]

Figure 5b

[0014]

Figure 6

[0015]

Figure 7

[0016]

Figure 8

[0017]

Figure 9

[0018]

Figure 10

[0019]

Figure 11a

Figure 11b

Figure 11c

[0020]

Figure 12

[0021] Certain examples and embodiments have other features in addition to, and instead of, the features shown in the figures referenced above. These features and other features are described in detail below with reference to the figures referenced above.

Best Mode for Carrying Out the Invention

[0022] Examples and embodiments according to the principles described herein that provide a method for computer-implemented synthetic view image generation. By the method described herein, an input image including a plurality of pixels having color values is received, an inflated depth map is generated by inflating a depth map associated with the input image, the depth map including depth values respectively associated with each pixel in the input image. The depth map may be generated from the input image. A blending map may also be generated from the depth map, the blending map including blending values respectively associated with each pixel in the depth map. The inflated depth map is used to determine a repair mask, and a repair operation is performed based on the repair mask and the input image to generate a background image. The synthetic view image is then rendered using the background image, the input image, the inflated depth map, and (if a blending map is generated) the blending map. A computer system and a computer program product are also described.

[0023] It has been found that the described method can reduce or, in some cases, remove visual artifacts in the synthetic image. Further, the described method has been found to be robust against artifacts resulting from multi-level occlusion in the input image.

[0024] As used herein, a "two-dimensional image" or "2D image" is defined as a set of pixels, each pixel having an associated intensity and / or color value. For example, a 2D image may be a 2D RGB image in which, for each pixel in the image, relative intensities of red (R), green (G), and blue (B) are provided. A 2D image generally represents a perspective view of a scene or object.

[0025] In contrast, as used herein, a stereoscopic image is defined as a pair of images respectively corresponding to perspective views of a scene or object from each of the viewer's left and right eye viewpoints. Even more in contrast, as used herein, a "multi-view image" is an image that includes different view images, each view image representing a different perspective view of the scene or object of the multi-view image. A multi-view image explicitly provides three or more perspective views.

[0026] As used herein, a "multi-view display" is defined as an electronic display or electronic display system configured to provide different views of a multi-view image in various view directions or from various view directions. A multi-view display can be provided as part of various devices including, but not limited to, cellular phones (e.g., smartphones), watches, tablet computers, mobile computers (e.g., laptop computers), personal computers and computer monitors, automotive display consoles, camera displays, and various other mobile as well as substantially non-mobile display applications and devices. A multi-view display may display a multi-view image by providing different views of the multi-view image in different view directions with respect to the multi-view display.

[0027] As used herein, a "depth map" is defined as a map that provides information indicating the absolute or relative distance between an object depicted in an image and a camera (or equivalent to the viewpoint to which the image corresponds). By definition, a depth map includes a plurality of pixels, each pixel having a depth value, which is a value indicating the distance of the object at that pixel within the depth map relative to the viewpoint of the image. A depth map may have a one-to-one correspondence with the image, i.e., for each pixel in the image, the depth map provides a depth value at the corresponding pixel. However, as will be understood, depth maps often provide a coarser granularity and often have a lower resolution than the corresponding image, and each pixel within the depth map provides the depth values of a plurality of pixels within the image. A depth map having a lower resolution than the corresponding image may be referred to as a downsampled depth map.

[0028] A disparity map can be used equivalently to the depth map described above. Disparity refers to the apparent shift of an object within a scene when observed from two different viewpoints, e.g., the left-eye and right-eye viewpoints. Disparity information and depth information are related and can be mapped to each other conditional on the geometric shape of each viewpoint of the disparity map. Considering this close relationship and the fact that one can be converted to the other, the terms "depth map" and "depth value" used throughout the description are understood to include depth information as well as disparity information. That is, depth and disparity can be used interchangeably in the manner described below.

[0029] As used herein, "occlusion" is defined as a foreground object within an image that covers at least a portion of the background such that the background is not visible. Further, as used herein, "disocclusion" is defined as an area of the image that is no longer obscured by a foreground object when the position of the foreground object has moved from its original position within the image according to a shift in the viewpoint or viewing point.

[0030] Furthermore, as used herein, the articles "a" and "an" are intended to have their ordinary meaning in patent technology, i.e., meaning "one or more". For example, "image" means one or more "images", and thus, in this specification, "image" means "image(s)". Also, references herein to "top", "bottom", "upper", "lower", "up", "down", "front", "back", "first", "second", "left", or "right" are not intended to be limiting in this specification. As used herein, the term "about", when applied to a value, generally means within the tolerance of the equipment used to generate that value, or, unless otherwise specified, can mean plus or minus 10%, plus or minus 5%, or plus or minus 1%. Further, the term "substantially" as used herein means mostly, or almost all, or all, or an amount in the range of about 51% to about 100%. Further, the examples in this specification are intended for illustration only and are presented for explanatory purposes and not as limitations.

[0031] According to some embodiments of the principles described herein, a method for computer-implemented synthetic view image generation is provided. FIG. 1a shows a flowchart of the steps of method 100. See also FIG. 1b, which shows the relationships between the different data objects used and generated in this method. The steps of method 100 (each of which will be described in detail below) are as follows.

[0032] First, in step 101, an input image 200 including a plurality of pixels having color values is received. Next, in step 103, a dilated depth map 350 is generated by dilating a depth map 300 associated with the input image, and the depth map includes depth values respectively associated with each pixel in the input image. The depth map 300 may be generated from the input image 200 as shown by an optional step 102 in FIG. 1a and by a dashed arrow connecting the input image 200 and the depth map 300 in FIG. 1b. In some embodiments, in an optional step 104, a blending map 360 is generated from the depth map 300, and the blending map 360 includes blending values respectively associated with each pixel in the depth map 300. In step 105, the dilated depth map 350 is used to determine a repair mask 700. Next, in step 106, a repair operation is performed based on the repair mask and the input image to generate a background image 800. In step 107, a synthetic view image 1000 is rendered using the background image 800, the input image 200, and the dilated depth map 350. Rendering of the synthetic view image 1000 may include using the input image 200 and the dilated depth map 350 to generate a foreground image 900 that is combined with the background image 800 as shown in FIG. 1b. In embodiments where the blending map 360 is generated, the blending map 360 can also be used for rendering the synthetic view image 1000.

[0033] In other words, in the method described herein, depth estimation may be performed based on a single input image. Next, a repair mask may be formed, which emphasizes areas that need to be repaired later to fill in occlusions. Next, the depth map is dilated and blending values are determined. Next, to render the synthetic view image, the foreground is rendered and the repaired background image is rendered. Next, occlusions holes in the foreground image are filled using the background image so that the synthetic view image is rendered.

[0034] Next, the steps of the method shown in FIG. 1a are executed in order to explain the method in detail.

[0035] First, in step 101 of FIG. 1a, an input image 200 is received. The input image 200 may be a 2D RGB image. That is, for each pixel of the input image 200, color values (e.g., red, green, and blue) are assigned. The input image 200 may be received from any number of sources. For example, the input image 200 may be captured by a 2D still camera. The input image 200 may be a single frame of a 2D video. The image 200 may be a generated image, e.g., an image generated by a deep learning model or generative AI (such as OpenAI's DALL-E model).

[0036] To facilitate the explanation of the method, an exemplary input image 200 is shown in FIG. 2. The image includes a background 201 and foreground objects 202 and 203. In this exemplary image, the foreground object 203 is in front of the foreground object 202, and the foreground objects are in front of the background 201.

[0037] As shown by optional step 102 of FIG. 1a, after receiving the input image 200, depth estimation may be performed on the image to generate a depth map 300. Monocular depth estimation techniques can estimate dense depth based on a single 2D (RGB) image. Many methods either directly utilize a single image or estimate intermediate 3D representations, such as point clouds. Some other methods combine the 2D image with, for example, a sparse depth map or a normal map to estimate a dense depth map. These methods are trained on large-scale generated datasets that include RGB-D images, i.e., images where each pixel color (RGB) value and depth (D) value are provided. One depth estimation technique suitable for this method is the Midas technique disclosed by Ranftl et al. in "Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer" in IEEE transactions on pattern analysis and machine intelligence in 2020, which is incorporated herein by reference. The depth estimation technique may provide a depth value for each pixel in the input image such that the depth map 300 contains depth values associated with each pixel in the input image, and each depth value is an estimated value of the depth associated with the object at that pixel in the image.

[0038] Of course, it will be understood that the depth map 300 may not be generated from the received input image, but instead may be provided by other means. For example, a depth sensor (such as a time-of-flight sensor) may be used to image the depth map 300 at the time of imaging the input image. As a further example, the depth map 300 may be generated by a different application or by the operating system at the time of or after imaging the input image. In any case, the depth map 300 may be received together with the input image.

[0039] To facilitate the description of the method, an exemplary depth map 300 is shown in FIG. 3. The shading of each pixel of the depth map 300 represents the depth value (i.e., the estimated depth) of each corresponding pixel of the input image 200. Dark shading indicates a large depth value (i.e., a further position within the scene imaged from the "viewer"), and light shading indicates a small depth value (i.e., a position closer to the "viewer"). In this case, area 301 of the depth map 300 corresponds to the background, area 302 corresponds to the foreground object 202, and area 303 corresponds to the foreground object 203.

[0040] Generally, the depth values in a depth map do not have a sharp (or step-like) transition from foreground to background depth. Instead, transitional depth values can be seen near the edges of objects. This is shown in FIGS. 4a and 4b. FIG. 4a shows another exemplary depth map 400, and FIG. 4b is an enlargement of a portion of the depth map 400 corresponding to the dashed rectangle in FIG. 4a. As seen in FIG. 4b, between the lightest shading area (corresponding to the foreground object, on the right side of the image) and the darkest shading area (corresponding to the background area of the image, on the left side of the image), there are pixels with transitional depth values, i.e., pixels having depth values that fall between the depth value of the foreground object and the depth value of the background area. The inventors of the present invention have identified that these foreground-background transitions can cause visual artifacts when using forward or inverse mapping to render a new view in a synthetic image.

[0041] For example, when using forward mapping, striping artifacts may occur due to transitional depth values. This is because each transitional depth value causes a slight further displacement of the relevant pixels of the foreground object, which spreads across the occluded area. Further, the edges of the foreground object are damaged because some of the pixels near the edges may be displaced away from the rest of the object. An example of this striping artifact is shown in FIG. 6, which shows the forward-mapped rendering of the foreground image (without repair of the occluded area). As can be seen, "stripes" of pixels can be seen in the occluded area near the foreground object.

[0042] In this method, the depth map 300 is received together with the input image or generated from the input image, and then, in step 103, an inflated depth map 350 is generated from the image. Generating an inflated depth map can result in sharp transitions between areas of different depth values.

[0043] Generally speaking, the process of generating the inflated depth map 350 from the depth map 300 is to convert the gradual transition between the foreground area and the background area in the depth map 300 into a sharp transition in the inflated depth map 350.

[0044] In some embodiments, the process of generating the inflated depth map 350 is as follows.

[0045] Local minimum depth values and local maximum depth values are identified. Transition depth values are also identified, and each transition depth value has a value between the local minimum depth value and the local maximum depth value. For the pixels in the depth map 300 having a transition depth value, the depth value of the corresponding pixel in the inflated depth map 350 is set to the local maximum depth value.

[0046] In some embodiments, this is only performed when the difference between the local maximum depth value and the local minimum depth value exceeds a specific threshold difference in depth. That is, when the transition depth value is within a small range of depth values (defined by the threshold difference in depth values), the pixels in the dilated depth map corresponding to the pixels in the depth map having the transition depth value are not set to the local maximum value, but instead are set to the transition depth value of the corresponding pixels in the depth map 300. This can help limit the computational requirements of the method.

[0047] For pixels in the depth map 300 having local minimum or local maximum depth values, the corresponding pixels in the dilated depth map 350 are set to the local minimum and local maximum depth values respectively.

[0048] The above process may be repeatedly applied across multiple areas in the image.

[0049] Exemplary dilated depth maps are shown in FIGS. 5a and 5b. FIG. 5a shows a dilated depth map 500 corresponding to the depth map 400. FIG. 5b is an enlargement of a portion of the depth map 500 corresponding to the dashed rectangle in FIG. 5a. As can be seen by comparing FIG. 5b with FIG. 4b, a sharp transition between the foreground object and the background area is brought about in the dilated depth map. By comparing FIG. 5a with FIG. 4a, it can also be seen that in areas of the image where the depth values transition smoothly, that smooth transition is maintained between the depth map 400 and the dilated depth map 500 (e.g., within the area indicated by the dashed ellipse in FIG. 5a).

[0050] In some embodiments of the present method, in optional step 104, a blending map 360 is generated from the depth map 300. The blending map is used to blend the transition between the foreground area and the background area in the finally rendered composite view image 1000. The use of the blending map 360 can reduce or even avoid any inflation artifacts that might normally be visible after rendering. The blending map 360 includes blending values for each pixel in the input image. The blending map 360 may be used as an alpha mask when rendering the composite view image in step 107. Thus, the blending map divides the image into three regions: a background region (corresponding to a minimum blending value, e.g., α = 0.0), a foreground region (corresponding to a maximum blending value, e.g., α = 1.0), and a transition region (a blending value between the maximum and minimum blending values, e.g., corresponding to 0.0 < α < 1.0). When rendering the composite view image, the blending map 360 may be applied as an alpha mask to smooth the transition between the foreground layer and the background layer, and the blending value determines the opacity of the foreground pixels covering the background layer. For example, a foreground pixel corresponding to a blending value α = 0.0 may be completely transparent (i.e., only the color information from the background layer is used for that pixel in the rendered image), and a foreground pixel corresponding to a blending value α = 1.0 may be opaque (i.e., only the color information from the foreground layer is used for that pixel in the rendered image). In the case of pixels corresponding to an intermediate blending value 0.0 < α < 1.0, the foreground pixels become partially transparent (i.e., for that pixel in the rendered image, the color information takes the value obtained by adding the corresponding channel value in the foreground pixel multiplied by α and the corresponding channel value in the background pixel multiplied by (1 - α) for each RGB channel).

[0051] The blending map 360 may be generated by determining a local minimum depth value, a local maximum depth value, and a transition depth value, where each transition depth value has a value between the local minimum depth value and the local maximum depth value. The local maximum depth value is scaled to the global maximum blending value (e.g., α = 1.0), and the local minimum depth value is scaled to the global minimum blending value (e.g., α = 0.0), such that the blending value of each corresponding pixel in the blending map is set by scaling the depth value. The transition depth values are scaled to a value between the global maximum blending value and the global minimum blending value (e.g., 0.0 < α < 1.0). This process may be repeated over multiple areas in the image.

[0052] In this method, in step 105, the repair mask 700 is determined from the inflated depth map 350. The repair mask 700 identifies areas of the input image 200 that may be occluded when a transformation corresponding to a shift in the viewpoint view is applied. The repair mask 700 includes, for each pixel in the input image 200, a value indicating whether that pixel will be repaired in the repair operation. In other words, these are areas in the image that may be occluded in the foreground image as the foreground object moves according to the shift in the viewpoint view. The repair mask 700 identifies the areas of the input image that will be repaired to provide the background image.

[0053] In some embodiments, the repair mask 700 may be generated by identifying depth transitions within the dilated depth mask 350 that exceed a threshold difference in depth and by adding one or more pixels to the repair mask 700, where the one or more added pixels are adjacent to the transition and on the side of the transition having a lower depth value, corresponding to pixels of the dilated depth mask 350. That is, when a sharp transition in depth is identified in the dilated depth map 350, pixels are added to the repair mask 700 adjacent to the location of that transition on the shallower side of the transition. The threshold difference in depth used in this step may be the same threshold difference in depth as that used in step 103 when generating the dilated depth map, or may be a different threshold difference in depth.

[0054] In some embodiments, only transitions in one (horizontal or vertical) direction are identified, and each of the one or more added pixels is in the horizontal or vertical direction relative to the transition. This can be performed when only horizontal or vertical parallax is provided from the synthetic view image 1000 (i.e., when the shift of the viewpoint view is only in the horizontal or vertical direction), since only areas of the image adjacent to depth transitions in the direction of the viewpoint shift are potentially occluded.

[0055] In other words, to generate views separated in the horizontal direction, the process iterates across the dilated depth map each time a sudden increase in decrease is reached, and pixels horizontally positioned on the high side of this transition (i.e., the side having a lower depth value) are masked.

[0056] To facilitate the explanation of this method, an exemplary repair mask 700 derived from the depth map 300 is shown in FIG. 7. The white areas in the repair mask 700 indicate the areas to be repaired in the repair operation. In this example, only horizontal depth transitions in the depth map 300 are identified to add pixels to the repair mask.

[0057] After the repair mask is generated, at step 106, a repair operation is performed to generate the background image 900. In some embodiments, this is achieved by providing the input image 200 and the repair mask 700 to a repair neural network.

[0058] In some embodiments, the repair network is a depth-aware repair network. Depth-aware repair means that for the area of the repaired background image, both color values and depth values are generated. The input image 200 is provided as an RGB-d image (i.e., each pixel has RGB color information and a depth value D derived from the depth map 300 or the dilated depth map 350). The repair network will repair the area of the image defined by the repair mask to generate color (RGB) values and depth values for each pixel within the repair area.

[0059] In some embodiments, the repair network is an adversarial generative network (GAN). Some suitable repair networks may be employed. One such network is the LaMa repair network disclosed by Zhao et al. in "Large Scale image completion via co-modulated generative adversarial networks". The 2021 International Conference on Learning Representations (ICLR) is incorporated herein by reference.

[0060] The LaMa network can be modified for RGB-D repair and trained with a combination of a random repair mask (i.e., a mask containing randomly generated mask areas) and a disocclusion repair mask (i.e., a mask derived from the above-described repair mask generation process). The use of the random repair mask (in addition to the disocclusion repair mask) enables suitable training for general repair where the network can handle large masks that may occur in multi-level disocclusion.

[0061] In some embodiments, a second repair operation different from the first repair operation is used, and the first repair operation generates pixels having depth values, which, when compared to a reference depth value derived from the inflated depth map 350, indicates the presence of multilevel occlusion. Alternatively, in some other embodiments, the reference depth value can be derived from the depth map 300.

[0062] In some embodiments, for example, the reference depth value is the depth value of a pixel on the deeper side of the transition (i.e., the side of the depth transition having a larger depth value). The depth value of the pixels generated by the repair operation may be compared to this reference value. If the difference in depth values between the repaired pixel and the reference depth value exceeds a specific threshold difference in depth values, multilevel occlusion can be assumed, and in that case, a different repair operation can be used. For example, as the second repair operation, simple reflection repair can be used.

[0063] To facilitate the explanation of the method, an exemplary background image 800 is shown in FIG. 8, which is the output of a repair operation using the repair mask 700 and the input image 200. Area 805 corresponds to the area identified in the repair mask repaired by the repair operation.

[0064] After the background image is generated, the process can proceed to render a composite view image 1000 corresponding to an image having a viewpoint different from that of the input image in step 107. To generate the foreground image 900, a transformation can be applied to the input image 200 using the depth values from the inflated depth map 350. This may be achieved by calculating, for each pixel in the input image 200, the shift in the position of that pixel within the image resulting from the change in the position of the viewpoint from the inflated depth map 350 and the depth value of that pixel. To generate the foreground image 900, each pixel is shifted according to the change in position calculated from the depth values of the depth map (i.e., the color information from the pixel is moved to another pixel according to the calculated change in position). This causes a shift in position in groups of pixels corresponding to objects at the foreground depth according to the shift in position, and also causes a disocclusion hole including an area of pixels that are occluded due to the difference in viewpoint between the foreground image and the input image.

[0065] For ease of explanation of the method, FIG. 9 shows an exemplary foreground image 900 corresponding to the transformation of the input image 200 of FIG. 2 using the inflated depth map derived from the depth map of FIG. 3. The foreground objects 202 and 203 are shifted horizontally in position according to the change in viewpoint compared to the input image. This horizontal shift in the position of the pixels associated with these objects between the input image 200 and the foreground image 800 corresponds to the depth values of those pixels in the inflated depth map 350. As can be seen, a disocclusion hole 905 (shown in dark gray) remains in the image.

[0066] The occlusion hole is filled using information from the background image 900 by filling the occlusion hole in the foreground image with information from the corresponding pixels of the background image. In some embodiments, a transformation is applied to the background image based on a change in the view point from the input image before the occlusion hole is filled from the background image (i.e., pixels are shifted according to the depth associated with that pixel in the depth map).

[0067] In embodiments where a blending map 360 is generated, the blending map is used to smooth the transition between areas of the images derived from the foreground and background images. The blending map is applied as an alpha mask as described above in the description of step 104.

[0068] In some embodiments, a repair operation can be performed to fill any holes remaining near the edges of the rendered composite view image 1000. Such holes can occur because neither the foreground nor the background image is mapped to these areas. Since these holes are relatively small and near the edges of the image, reflection repair is used in these remaining areas. This repair method is computationally inexpensive and effective for this task.

[0069] To facilitate the explanation of the method, an exemplary composite view image 1000 is shown in FIG. 10 based on the foreground image 900 and the background image 800. The occlusion hole 905 is filled using information from the background image 800.

[0070] Once rendered, the composite view image 1000 may be displayed on a display screen. The composite view image may be displayed as part of a stereoscopic image pair (such as a stereoscopic display, virtual reality headset, etc.) using the input image or another composite view image corresponding to a viewpoint view from a different view point.

[0071] It will be appreciated that several synthetic view images can be provided that each correspond to a perspective view from a different viewpoint. These may be displayed on a multi-view display screen as a set of different views of a multi-view image. The input image may or may not provide one of the views of the multi-view image.

[0072] FIG. 12 is a schematic block diagram depicting an illustration of an example of a computer device 1200 that provides a multi-view display according to various embodiments of the present disclosure. The computer device 1200 may include a system of components that perform various computer operations for a user of the computer device 1200. The computer device 1200 may be a laptop, a tablet, a smartphone, a touch screen system, an intelligent display system, or other client device. The computer device 1200 may include various components, such as one or more processors 1203, a memory 1206, input / output (I / O) components 1209 (plural possible), a display 1212, and potentially other components. These components may be coupled to a bus 1215 that functions as a local interface that enables the components of the computer device 1200 to communicate with each other. The components of the computer device 1200 are shown as being housed within the computer device 1200, but it should be understood that at least some of the components may be coupled to the computer device 1200 via an external connection. For example, the components may be externally plugged into the computer device 1200 via an external port, socket, plug, or connector, or otherwise connected.

[0073] Processor 1203 may be a central processing unit (CPU), a graphics processing unit (GPU), or any other integrated circuit that performs computing operations. The processor(s) 1203 may include one or more processing cores. The processor(s) 1203 includes circuitry for executing instructions. The instructions include, for example, computer code, programs, logic, or other machine-readable instructions that are received and executed by the processor(s) 1203 to perform a computing function incorporated in the instructions. The processor(s) 1203 may execute instructions for operating on data. For example, the processor(s) 1203 may receive input data (e.g., an input image), process the input data according to a set of instructions, and generate output data (e.g., a synthesized view image). As another example, the processor(s) 1203 may receive instructions and generate new instructions for subsequent execution.

[0074] Memory 1206 may include one or more memory components. Memory 1206 is defined herein as including either or both volatile and non-volatile memory. Volatile memory components do not retain information when power is lost. Volatile memory may include, for example, random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), magnetic random access memory (MRAM), or other volatile memory structures. System memory (e.g., main memory, cache, etc.) may be implemented using volatile memory. System memory refers to high-speed memory that may temporarily store data or instructions for rapid read and write access to assist the processor(s) 1203.

[0075] A non-volatile memory component is one that retains information during power loss. Non-volatile memory includes read-only memory (ROM), hard disk drives, solid-state drives, USB flash drives, memory cards accessed via a memory card reader, floppy disks accessed via an associated floppy disk drive, optical disks accessed via an optical disk drive, and magnetic tapes accessed via an appropriate tape drive. ROM may include, for example, programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or other similar memory devices. Storage memory may be implemented using non-volatile memory to provide long-term retention of data and instructions.

[0076] Memory 1206 may refer to a combination of volatile and non-volatile memory used to store instructions and data. For example, data and instructions may be stored in non-volatile memory and loaded into volatile memory for processing by the processor(s) 1203. Execution of instructions may include, for example, a compiled program that is loaded from non-volatile memory into volatile memory and then converted to machine code in a format executable by the processor 1203, source code that is converted to a suitable format, object code that may be loaded into volatile memory for execution by the processor 1203, or source code that is interpreted by another executable program to generate instructions in volatile memory and executed by the processor 1203. Instructions may be stored or loaded in any portion or component of memory 1206, including, for example, RAM, ROM, system memory, storage, or any combination thereof.

[0077] Although memory 1206 is shown as being separate from the other components of computer device 1200, it should be understood that memory 1206 can be at least partially embedded in one or more components or otherwise integrated. For example, processor(s) 1203 may include on-board memory registers or caches for performing processing operations.

[0078] I / O component(s) 1209 includes, for example, a touch screen, speaker, microphone, button, switch, dial, camera, sensor, accelerometer, or other component that receives user input or generates an output directed to the user. I / O component(s) 1209 may convert user input into data for receipt by memory 1206 for storage or for processing by processor(s) 1203. I / O component(s) 1209 may receive data output by memory 1206 or processor(s) 1203 and convert them into a format perceivable by the user (e.g., sound, tactile response, visual information, etc.).

[0079] A particular type of I / O component 1209 is display 1212. Display 1212 may include a multi-view display, a multi-view display combined with a 2D display, or any other display that presents images. A capacitive touch screen layer that functions as an I / O component 1209 is layered within the display to enable the user to provide input while simultaneously perceiving the visual output. Processor(s) 1203 may generate data formatted as an image for presentation on display 1212. Processor(s) 1203 may execute instructions for rendering an image on the display for perception by the user.

[0080] Bus 1215 facilitates communication of instructions and data among the processor(s) 1203, memory 1206, I / O component(s) 1209, display 1212, and any other components of computer device 1200. Bus 1215 may include an address translator, address decoder, fabric, conductive traces, wires, ports, plugs, sockets, and other connectors to enable communication of data and instructions.

[0081] Instructions within memory 1206 may be embodied in various forms to implement at least a portion of a software stack. For example, the instructions may be embodied as an operating system 1231, application(s) 1234, device driver (e.g., display driver 1237), firmware (e.g., display firmware 1240), or other software components. Operating system 1231 is a software platform that supports basic functions of computer device 1200, such as task scheduling, control of I / O component 1209, provision of access to hardware resources, power management, and support of applications 1234.

[0082] The application(s) 1234 runs on the operating system 1231 and may access the hardware resources of the computer device 1200 via the operating system 1231. In this regard, the execution of the application(s) 1234 is at least partially controlled by the operating system 1231. The application(s) 1234 may be a user-level software program that provides high-level functions, services, and other functions to the user. In some embodiments, the application 1234 may be a dedicated "app" that can be downloaded or otherwise accessed by the user on the computer device 1200. The user may launch the application(s) 1234 via the user interface provided by the operating system 1231. The application(s) 1234 may be developed by developers and may be defined in various source code formats. The application 1234 may be developed using a number of programming languages or scripting languages, such as C, C++, C#, Objective C, Java®, Swift, JavaScript®, Perl, PHP, Visual Basic®, Python®, Ruby, Go, or other programming languages. The application(s) 1234 may be compiled into object code by a compiler or may be interpreted by an interpreter for execution by the processor(s) 1203.

[0083] A device driver, for example, display driver 1237, includes instructions that enable operating system 1231 to communicate with various I / O components 1209. Each I / O component 1209 may have its own device driver. The device driver can be installed to be stored in storage and loaded into system memory. For example, during installation, display driver 1237 converts high-level display instructions received from operating system 1231 into low-level instructions to be executed by display 1212 to display an image.

[0084] Firmware, for example, display firmware 1240, may include machine code or assembly code that enables I / O component 1209 or display 1212 to perform low-level operations. The firmware can convert electrical signals of a specific component into high-level instructions or data. For example, display firmware 1240 may control how individual pixels are activated at a low level by display 1212 adjusting voltage or current signals. The firmware can be stored in non-volatile memory and executed directly from the non-volatile memory. For example, display firmware 1240 can be embodied in a ROM chip coupled to display 1212 such that the ROM chip is separated from other storage and system memory of computer device 1200. Display 1212 may include processing circuitry for executing display firmware 1240.

[0085] The operating system 1231, the application(s) 1234, the driver (e.g., the display driver 1237), the firmware (e.g., the display firmware 1240), and potentially other instruction sets can each include instructions executable by the processor(s) 1203 or other processing circuitry of the computer device 1200 to perform the functions and operations described above. The instructions described herein may be embodied as software or code executed by the processor(s) 1203 as described above, but alternatively, the instructions may also be embodied in dedicated hardware or in a combination of software and dedicated hardware. For example, the functions and operations performed by the instructions described above may be implemented as a circuit or state machine using any one of several techniques or a combination thereof. These techniques may include, but are not limited to, discrete logic circuits having logic gates for performing various logical functions when one or more data signals are applied, application specific integrated circuits (ASICs) having appropriate logic gates, field programmable gate arrays (FPGAs), or other components, etc.

[0086] In some embodiments, the instructions for performing the functions and operations described above may be embodied in a non-transitory computer-readable storage medium. The computer-readable storage medium may or may not be part of the computer device 1200. The instructions may include, for example, statements, code, or declarations that can be fetched from the computer-readable medium and executed by the processing circuitry (e.g., the processor(s) 1203). In the context of the present disclosure, a "computer-readable medium" may be any medium that can contain, store, or maintain the instructions described herein for use by or in connection with an instruction execution system, such as the computer device 1200.

[0087] A computer-readable medium can include any one of many physical media, such as magnetic media, optical media, or semiconductor media. More specific examples of suitable computer-readable media can include, but are not limited to, magnetic tapes, magnetic floppy disks, magnetic hard drives, memory cards, solid state drives, USB flash drives, or optical disks. Also, the computer-readable medium can be, for example, random access memory (RAM) including static random access memory (SRAM) and dynamic random access memory (DRAM), or magnetic random access memory (MRAM). Further, the computer-readable medium can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or other types of memory devices.

[0088] The computer device 1200 may perform any of the operations described above or implement the functions described above. For example, the flowcharts and process flows described above may be implemented by the computer device 1200 that executes instructions and processes data. Although the computer device 1200 is shown as a single device, the present disclosure is not so limited. In some embodiments, the computer device 1200 may offload the processing of instructions in a distributed manner such that multiple computer devices 1200 operate together to execute instructions that may be stored or loaded in a distributed arrangement. For example, at least some instructions or data may be stored, loaded, or executed in a cloud-based system that operates in conjunction with the computer device 1200.

[0089] The present disclosure also provides a computer program product corresponding to every embodiment of the computer-implemented synthetic view image generation method described herein. Such a computer program product, when executed by a computer, comprises instructions for causing the computer to implement any of the methods disclosed herein. The computer program product may be embodied on a non-transitory computer-readable storage medium.

[0090] Experimental results This method of synthetic view image generation was compared with previously reported methods. In particular, the method comprising steps 101, 102, 103, 104, 105, 106 and 107 was compared with two previously reported methods, namely the so-called SynSin method disclosed by Wiles et al. in "SynSin: End-to-end view synthesis from a single image" on pages 7467 - 7477 of the proceedings of the 2020 IEEE / CVF Conference on Computer Vision and Pattern Recognition, and the Slide method disclosed by Jampani et al. in "Slide: Single image 3d photography with soft layering and depth-aware inpainting" on pages 12518 - 12527 of the proceedings of the 2021 IEEE / CVF International Conference on Computer Vision. The two previously reported methods and this method were applied to the Holopix50k dataset, and several metrics were used to determine the effectiveness of each method. The Holopix50 dataset is disclosed by Hua et al. in "Holopix50k: A large-scale in-the-wild stereo image dataset" in the 2020 arXiv preprint arXiv:2003.11172.

[0091] 1. Mean Squared Error (MSE), 2. Peak Signal-to-Noise Ratio (PSNR), 3. Structural Similarity Index Measure (SSIM), 4. Learned Perceptual Image Patch Similarity (LPIPS), four evaluation metrics were determined. Details of these metrics are in "The unreasonable effectiveness of deep features as a perceptual metric" by Zhang et al. in CVPR 2018, which is incorporated herein by reference. The results are shown in Table 1.

[0092]

Table 1

[0093] From the results in Table 1, it can be seen that this method brings improvements over previously reported methods according to all four metrics.

[0094] Improvements over previously reported methods are also evident from qualitative analysis by visual comparison of the rendering of synthetic view images from different methods. FIGS. 11a - 11c show the same rendered synthetic view images provided from different methods, where FIG. 11a shows the image generated by the SynSin method, FIG. 11b shows the image generated by the Slide method, and FIG. 11c shows the image generated by the present method. In all cases, the area indicated by the solid-line rectangle is enlarged and shown. From the comparison between the image of FIG. 11c and the image of FIG. 11a, it can be seen that the present method does not cause significant distortion as in the case of the image generated by the SynSin method (for example, refer to the distortion of the traffic signal indicated by the dashed-line rectangle in FIG. 11a). From the comparison between the enlarged part of FIG. 11c and the enlarged part of FIG. 11b, it can be seen that the image of the present method has no visual artifacts resulting from the slide method. Therefore, it will be understood that the present method provides improvements over the two previously reported methods in both qualitative and quantitative evaluations.

[0095] Accordingly, examples and embodiments of methods and systems for providing hierarchical view synthesis have been described. The above examples are merely illustrative of some of the many specific examples that represent the principles described herein. Clearly, those skilled in the art can readily devise many other configurations without departing from the scope defined by the following claims.

Claims

1. A method for generating a computer-implemented composite view image, wherein the method is The steps include receiving an input image containing multiple pixels having color values, A step of generating an expanded depth map by expanding the depth map associated with the input image, wherein the depth map includes depth values ​​associated with each pixel in the input image. The steps include determining a repair mask using the aforementioned expansion depth map, To generate a background image, a step is performed to carry out a repair operation based on the repair mask and the input image, The steps include rendering a composite view image using the background image, the input image, and the expanded depth map, and Includes, A computer-implemented composite view image generation method comprising the step of generating an expanded depth map, the step of determining a local minimum depth value, a local maximum depth value, and a plurality of transition depth values, wherein each transition depth value has a value between the local minimum depth value and the local maximum depth value.

2. A method for generating a computer-implemented composite view image according to claim 1, further comprising the step of generating the depth map of the input image by performing depth estimation in the image to determine the depth value associated with each pixel.

3. The step of generating an expanded depth map is A method for generating a computer-implemented composite view image according to claim 1, further comprising the step of setting the depth value of a pixel in the expanded depth corresponding to a pixel in the depth map having a plurality of transition depth values ​​to the local maximum depth value.

4. The method for generating a computer-implemented composite view image according to claim 3, wherein the step of setting the depth value of the pixels in the expanded depth map is performed when the difference between the local minimum depth value and the local maximum depth value of the depth map exceeds a predetermined threshold difference of depth values.

5. A step of generating a blending map from the depth map, further comprising the steps of: the blending map includes blending values ​​associated with each pixel in the depth map, and the composite view image is rendered using the blending map; The step of generating a blending map is A step of determining a local minimum depth value, a local maximum depth value, and a plurality of transition depth values, wherein each transition depth value has a value between the local minimum depth value and the local maximum depth value. The process includes setting the blending value for each corresponding pixel in the blending map based on scaling the depth values ​​such that the local maximum depth is scaled to a global maximum blending value and the local minimum depth is scaled to a global minimum blending value. The step of rendering the composite view image is: The steps include generating a foreground image from the input image and the expanded depth map, The step of generating the composite view image by combining the foreground image and the background image is included, The step of generating the composite view image includes the step of smoothing the transition between the area of ​​the composite view image corresponding to the foreground image and the area of ​​the composite view image corresponding to the background image, according to the blending value in the blending map. The foreground image corresponds to an image having a different viewpoint from the input image, and the foreground image has a disocclusion hole that includes a set of pixels corresponding to an area that is occluded due to the difference in viewpoints between the foreground image and the input image. The step of combining the foreground image and the background image is, The step includes filling the disocclusion holes in the foreground image with information from the corresponding pixels of the background image, The background image includes depth values ​​associated with each pixel, and the step of filling the disocclusion holes in the foreground image is: The step includes applying a transformation to the background image based on the different viewpoints, The step of smoothing the transition between the areas in the composite view image is The step includes using the blending mask as an alpha mask, The step of rendering each of the one or more composite view images is: A step of performing a repair operation to fill a hole remaining near the edge of the aforementioned image, wherein the repair operation is a reflective repair operation, further comprising the step of A method for generating a computer-implemented composite view image according to claim 1.

6. The repair mask determination step is, Steps include identifying depth transitions within the expanded depth map that exceed a depth threshold difference, A step of adding one or more pixels to the repair mask, wherein the one or more added pixels correspond to the pixels of the expanded depth mask that are adjacent to the transition and on the side of the transition having a low depth value. The method for generating a computer-implemented composite view image according to claim 1, wherein the identified depth transition is a transition in either the horizontal or vertical direction within the depth map, and the one or more additional pixels are each located in the horizontal or vertical direction relative to the transition.

7. The step of performing the repair operation is, A step of providing an input image and the repair mask to a repair network, wherein the repair network generates the background image, and the repair network is a depth-aware repair network. The aforementioned repair network, A first dataset containing images and their respective associated disocclusion masks, A second dataset containing the image and each associated random repair mask. A method for generating a computer-implemented composite view image according to claim 1, which is trained with data including the following.

8. The steps to perform the repair operation are: A method for generating a computer-implemented composite view image according to claim 1, further comprising the step of comparing the depth value generated for each restored pixel with the depth value of a reference depth value derived from the expanded depth map, and using a second restoration operation to replace the pixel in the background image when the difference between the depth value generated for the restored pixel and the depth value of the corresponding pixel in the depth map exceeds a depth threshold difference, wherein the second restoration operation is a reflection restoration operation.

9. The further step includes displaying at least one of the one or more composite view images on a display screen, The method for generating a computer-implemented composite view image according to claim 1, wherein the display screen is a multi-view display screen.

10. The method for generating a computer-implemented composite view image according to claim 1, wherein the repair mask includes a value for each pixel in the input image indicating whether or not the pixel will be repaired in the repair operation.

11. A computer program product, wherein when the program is executed by a computer, the computer, The method for generating an expanded depth map is to expand a depth map associated with a received input image, wherein the input image includes a plurality of pixels having color values, and the depth map includes depth values ​​associated with each pixel in the input image. The repair mask is determined using the aforementioned expansion depth map, To generate a background image, a repair operation is performed based on the repair mask and the input image. A composite view image is rendered using the background image, the input image, and the expanded depth map. It includes a command to perform the generation of a composite view image, The step of generating an expanded depth map includes the step of determining a local minimum depth value, a local maximum depth value, and a plurality of transition depth values, wherein each transition depth value has a value between the local minimum depth value and the local maximum depth value. Computer program products.

12. When the program is executed by the computer, the instruction is sent to the computer. The computer program product according to claim 11, which generates a depth map of the input image by performing depth estimation in the image in order to determine the depth value associated with each pixel, thereby performing further composite view image generation.

13. To generate an expansion depth map, The computer program product according to claim 11, further comprising setting the depth value of a pixel in the expanded depth corresponding to a pixel in the depth map having a plurality of transition depth values ​​to the local maximum depth value.

14. The computer program product according to claim 11, wherein setting the depth value of the pixel in the expanded depth map is performed when the difference between the local minimum depth value and the local maximum depth value of the depth map exceeds a predetermined threshold difference of depth values.

15. When the program is executed by the computer, the instruction is sent to the computer. The computer program product according to claim 11, which causes the computer program product to perform further composite view image generation by generating a blending map from the depth map, wherein the blending map includes blending values ​​associated with each pixel in the depth map, and the composite view image is rendered using the blending map.

16. A composite view image generation system, wherein the system Processor and When executed, the processor: The method for generating an expanded depth map is to expand a depth map associated with a received input image, wherein the input image includes a plurality of pixels having color values, and the depth map includes depth values ​​associated with each pixel in the input image. The repair mask is determined using the aforementioned expansion depth map, To generate a background image, a repair operation is performed based on the repair mask and the input image. A composite view image is rendered using the background image, the input image, and the expanded depth map. A memory that stores the instructions to execute and Equipped with, A composite view image generation system comprising: generating an expanded depth map, determining a local minimum depth value, a local maximum depth value, and a plurality of transition depth values, wherein each transition depth value has a value between the local minimum depth value and the local maximum depth value.

17. When the instruction is executed, the processor: To determine the depth value associated with each pixel, depth estimation is performed within the image to generate the depth map of the input image. The composite view image generation system according to claim 16, further comprising the implementation of the above.

18. To generate an expansion depth map, The composite view image generation system according to claim 16, further comprising setting the depth value of the pixel in the expanded depth corresponding to the pixel in the depth map having a plurality of transition depth values ​​to the local maximum depth value.

19. The composite view image generation system according to claim 18, wherein setting the depth value of the pixels in the expanded depth map is performed when the difference between the local minimum depth value and the local maximum depth value of the depth map exceeds a predetermined threshold difference of depth values.

20. The composite view image generation system according to claim 16, wherein the system further comprises a multi-view display screen, and when the plurality of commands are executed, the system further displays the multi-view display screen on the composite view image.

Images