Enhancement of texture and alpha channels in multi-plane images

Abstract

Image processing techniques are directed to improve the quality of a visual image generated by rendering a multi-plane image having a plurality of pixels and represented by a plurality of layers corresponding to different respective distances from a reference camera position. In an example embodiment, the image processing technique includes one or more of the following: (A) for a first set of pixels, scaling respective weights of the layers such that a sum of the scaled weights is normalized to one; (B) for a second set of pixels, replacing the respective alpha values and texture values in these layers with corresponding local mean values; and (C) for a third set of pixels, scaling the corresponding texture values in the layers such that, for a resulting visual image rendered for the reference camera position, the texture values of the third set match the corresponding texture values of the source image captured from the reference camera position.

Description

[0001] 1. Cross-reference to related applications

[0002] This application claims the benefit of priority to U.S. Provisional Application Serial No. 63 / 357,669, filed on July 1, 2022, and European Application No. 22182507.8, filed on July 1, 2022, each of which is incorporated herein by reference in its entirety. 2. Technical field

[0003] Various example embodiments relate generally to multi-planar imaging (MPI), and more particularly, but not exclusively, to editing multi-planar images. 3. Background technology

[0004] Multiplane images embody a relatively new approach for storing volumetric content. MPI can be used to render still images and video, and uses, for example, 32 planes of texture and transparency (alpha) information per camera to represent a three-dimensional (3D) scene within the viewing frustum. Example applications of MPI include computer vision and graphics, image editing, photo animation, robotics, and virtual reality. Summary of the invention

[0005] Various embodiments of image processing techniques are disclosed herein that are intended to improve the quality of a visible image generated by rendering a multi-plane image having a plurality of pixels and consisting of a plurality of layers corresponding to different respective distances from a reference camera position. In an example embodiment, the image processing technique includes one or more of the following operations: (A) for a first set of pixels, scaling the respective weights of the layers so that the sum of the scaled weights is normalized to one; (B) for a second set of pixels, replacing the respective alpha values ​​and texture values ​​in the layers with the respective local averages; and (C) for a third set of pixels, scaling the respective texture values ​​in the layers so that for a resulting visible image rendered for the reference camera position, the texture values ​​of the third set match the respective texture values ​​of a source image captured from the reference camera position.

[0006] According to an example embodiment, there is provided an apparatus for enhancing a first multi-plane image represented by a plurality of layers corresponding to different respective distances from a reference camera position, the apparatus comprising: at least one processor; and at least one memory comprising program code; and wherein the at least one memory and the program code are configured to, together with the at least one processor, cause the apparatus to at least: for each pixel in a first set of pixels, scale the respective weights of the layers so that a sum of the scaled weights is equal to a predetermined fixed value; for each pixel in a second set of pixels, replace the respective alpha values ​​and texture values ​​in the layers with the corresponding local averages; and for each pixel in a third set of pixels, scale the respective texture values ​​in the layers so that, for a resulting visible image rendered for the reference camera position, the texture value of each pixel in the third set matches the respective texture value of a reference image captured from the reference camera position.

[0007] According to another example embodiment, a method for enhancing a first multi-plane image represented by a plurality of layers corresponding to different respective distances from a reference camera position is provided, the method comprising: for each pixel in a first set of pixels, scaling the respective weights of the layers so that a sum of the scaled weights is a predetermined fixed value, the scaling of the respective weights being performed using at least one processor and at least one memory comprising program code; for each pixel in a second set of pixels, replacing the respective alpha values ​​and texture values ​​in the layers with the respective local averages, the replacement being performed using the at least one processor and the at least one memory; and for each pixel in a third set of pixels, scaling the respective texture values ​​in the layers so that for a resulting visible image rendered for the reference camera position, the texture value of each pixel in the third set matches the respective texture value of a reference image captured from the reference camera position, the scaling of the respective texture values ​​being performed using the at least one processor and the at least one memory.

[0008] According to yet another example embodiment, a non-transitory computer-readable medium storing instructions is provided that, when executed by a processor, causes the processor to perform operations including a method for enhancing a first multi-plane image represented by a plurality of layers corresponding to different respective distances from a reference camera position, the method comprising: for each pixel in a first set of pixels, scaling respective weights of the layers so that a sum of the scaled weights is a predetermined fixed value, the scaling of the respective weights being performed using at least one processor and at least one memory including program code; for each pixel in a second set of pixels, replacing respective alpha values ​​and texture values ​​in the layers with corresponding local averages, the replacement being performed using the at least one processor and the at least one memory; and for each pixel in a third set of pixels, scaling respective texture values ​​in the layers so that, for a resulting visible image rendered for the reference camera position, the texture value of each pixel in the third set matches a respective texture value of a reference image captured from the reference camera position, the scaling of the respective texture values ​​being performed using the at least one processor and the at least one memory. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Other aspects, features, and advantages of the various disclosed embodiments will become more fully apparent from the following detailed description and the accompanying drawings, by way of example, in which:

[0010] Figure 1 Depicts an example process of a video / image transfer pipeline.

[0011] Figure 2 The diagram illustrates a 3D scene representation using multi-plane images according to an embodiment.

[0012] FIG. 3A to FIG. 3B Pseudo code that may be used to implement multi-plane image rendering according to an embodiment is shown.

[0013] Figure 4 is a flow chart illustrating a method of editing a multi-plane image according to an embodiment.

[0014] FIG. 5A to FIG. 5D The analysis results corresponding to example artifacts according to an embodiment are graphically illustrated.

[0015] Figure 6 It is shown that the method according to the embodiment can be used to at least implement Figure 4 Pseudocode for the first part of the method's processing.

[0016] FIG. 7A to FIG. 7B It is shown that the embodiment can be used in Figure 4 Pseudocode for implementing local averaging in the method.

[0017] Figure 8 It is shown that the method according to the embodiment can be used to at least implement Figure 4 Pseudocode for the second part of the method's processing.

[0018] Fig. 9 It is shown that the method according to the embodiment can be used to at least implement Figure 4 Pseudocode for the third part of the method processing.

[0019] FIG. 10A to FIG. 10B The diagram shows a method according to an embodiment of the present invention. Figure 4 Example visual improvements corresponding to the processing blocks of the method.

[0020] FIG. 11A to FIG. 11B The diagram shows a method according to an embodiment of the present invention. Figure 4 Another processing block of the method corresponds to an example visual improvement.

[0021] Fig.12 is a block diagram illustrating a computing device according to an embodiment. DETAILED DESCRIPTION

[0022] The present disclosure and its various aspects may be embodied in various forms, including: hardware, devices or circuits controlled by computer-implemented methods, computer program products, computer systems and networks, user interfaces and application programming interfaces; as well as hardware-implemented methods, signal processing circuits, memory arrays, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), etc. The foregoing is intended only to give a general idea of ​​the various aspects of the present disclosure and is not intended to limit the scope of the present disclosure in any way.

[0023] In the following description, many details such as device configuration, timing, operation, etc. are set forth to provide an understanding of one or more aspects of the present disclosure. It is obvious to those skilled in the art that these specific details are only exemplary and are not intended to limit the scope of the present application.

[0024] In addition, although the present disclosure focuses on examples of using various circuits in digital projection systems, it should be understood that these are only examples. It should be further understood that the disclosed systems and methods can be used in any device that requires projection light, such as theaters, consumer and other commercial projection systems, heads-up displays, virtual reality displays, etc.

[0025] Video Coding According to Example Embodiments

[0026] Figure 1An example process of a video transmission pipeline (100) according to an embodiment is depicted, which shows various stages from video / image capture to video / image content display. An image generation block (105) can be used to capture or generate a sequence of video / image frames (102). The frames (102) can be digitally captured (e.g., by a digital camera) or generated by a computer (e.g., using computer animation) to provide video and / or image data (107). Alternatively, the frames (102) can be captured on film by a film camera. The film can then be converted into a digital format to provide video / image data (107).

[0027] During the production phase (110), the data (107) may be edited to provide a video / image production stream (112). The data of the video / image production stream (112) may be provided to a processor (or one or more processors, such as a central processing unit CPU) at a post-production block (115) for post-production editing. The post-production editing of block (115) may include, for example, adjusting or modifying the color or brightness in specific areas of the image to enhance the image quality or achieve a specific look of the image in accordance with the creative intent of the video creator. This portion of post-production editing is sometimes referred to as "color timing" or "color grading." Other edits (e.g., scene selection and sequencing, image cropping, adding computer-generated visual effects, removing artifacts, etc.) may be performed at block (115) to produce a "final" version (117) of the work for distribution. Enhancements to textures and alpha channels in the multi-plane images disclosed herein below may be performed at block (115). During post-production editing (115), the video and / or images may be viewed on a reference display (125).

[0028] After post-production (115), the data of the final version (117) can be transmitted to an encoding block (120) for further downstream transmission to decoding and playback devices such as televisions, set-top boxes, movie theaters, etc. In some embodiments, the encoding block (120) may include audio and video encoders (such as those defined by ATSC, DVB, DVD, Blu-ray and other transmission formats) to generate an encoded bitstream (122). In the receiver, the encoded bitstream (122) is decoded by a decoding unit (130) to generate a corresponding decoded signal (132) that is a copy or a close approximation of the representation signal (117). The receiver can be attached to a target display (140), which can have slightly different or completely different characteristics from the reference display (125). In this case, a display management (DM) block (135) can be used to map the decoded signal (132) to the characteristics of the target display (140) by generating a display mapping signal (137). Depending on the embodiment, the decoding unit (130) and the display management block (135) may include separate processors or may be based on a single integrated processing unit.

[0029] Multiplanar imaging

[0030] A multi-plane image includes multiple image planes, each of which is a "snapshot" of a 3D scene at a specific depth relative to the camera position. The information stored in each plane includes texture information (e.g., represented by R values, G values, B values) and transparency information (e.g., represented by alpha (A) values). In this document, the acronyms R, G, B represent red, green, and blue, respectively. There are different ways to generate a multi-plane image. For example, two or more input images from two or more cameras located at different known viewpoints can be processed together to generate a corresponding multi-plane image. Alternatively, a single view synthesis of a multi-plane image can be performed using a source image captured by a single camera. For at least some multi-plane image generation algorithms, when rendered on a reference display (125) or a target display (140), the corresponding multi-plane image may disadvantageously exhibit one or more types of artifacts. The various embodiments disclosed herein can be used to beneficially reduce the appearance of such artifacts and / or substantially completely suppress such artifacts.

[0031] Figure 2A 3D scene representation using a multi-plane image (200) according to an embodiment is illustrated in a picture. The multi-plane image (200) has D planes or layers (P0, P1, ..., P(D-1)), where D is an integer greater than one. The planes (layers) are indexed so that the layer farthest from the reference camera position (RCP) is indexed as layer 0 and maintains a certain distance (or depth) d0 from the RCP along the Z dimension of the 3D scene. For each next layer that is closer to the RCP, the index is incremented by one. The plane (layer) closest to the RCP has an index value of (D-1) and maintains a certain distance (or depth) d from the RCP along the Z dimension. D-1 Each of the planes (P0, P1, ..., P(D-1)) is orthogonal to a base plane (202) parallel to the XZ coordinate plane. The RCP is at a vertical height h above the base plane (202). Figure 2 The XYZ triplet shown indicates the overall orientation of the multi-plane image (200) and the planes (PO, PI, ..., P(D-1)) relative to the X, Y, and Z dimensions of the 3D scene.

[0032] We denote the RGB value of the i-th layer as C i , where the lateral dimensions of the layer are H×W, where H is the height of the layer (Y dimension) and W is the width of the layer (X dimension). The pixel value (x, y) of color channel c is represented as C i (x, y, c). The α value of the i-th layer is represented by A i , and the corresponding pixel value (x, y) of the alpha channel is represented as A i (x, y). Denote the depth distance from the i-th layer to the reference camera position (RCP) as d i . Denote the source image from the original reference view (where the camera is fixed at RCP) as R, and denote the texture pixel value as R(x, y, c). Note that in MPI, the effective distance between two adjacent layers usually has a fixed value, for example, different layers (P0, P1, ..., P(D-1)) of the multi-plane image (200) are equidistantly spaced in disparity (inverse depth).

[0033] As already indicated above, a multi-plane image, such as the multi-plane image (200), may be generated using single view synthesis of a single source image R or multi-view synthesis using two or more source images. Such synthesis may be performed, for example, during the production stage (110). The corresponding (multiple) MPI synthesis algorithm may generally be expressed as {(C i , A i ), where i=0, ..., D-1}, a multi-plane image (200) containing XYZ resolved pixel values ​​is output.

[0034] By processing {(C i , Ai ), where i=0, ..., D-1}, the MPI rendering algorithm can generate a visible image corresponding to the RCP or corresponding to a new virtual camera position different from the RCP. An example MPI rendering algorithm that can be used for this purpose (commonly referred to as an "MPI viewer") can include the steps of distortion and synthesis. Other suitable MPI viewers can also be used. The rendered multi-plane image (200) can be viewed, for example, on a reference display (125).

[0035] During the warping step of the MPI rendering algorithm, each layer (C i , A i ) can be viewed from the RCP viewpoint position (v s ) is warped to the new viewpoint position (v t ), for example, as follows:

[0036]

[0037]

[0038] in, is a warping function; and σ is a consistent scale (to minimize error). In an exemplary embodiment, the warping function It can be expressed as follows:

[0039]

[0040] Among them, v s =(u s , v s ) and v t =(u t , v t ). Function K s and K t The functions R and t represent the extrinsic camera models for rotation and translation, respectively. n represents the normal vector [0 01] T a represents the depth σd i The distance to the plane parallel to the front face of the source camera.

[0041] During the composition step of the MPI rendering algorithm, a new visible image C may be generated, for example, using processing operations corresponding to the following equation: t :

[0042]

[0043] Among them, the weight Expressed as:

[0044]

[0045] The disparity map D corresponding to the source view s can be calculated as:

[0046]

[0047] Among them, the weight Expressed as:

[0048]

[0049] The MPI rendering algorithm can also be used to generate a visual image C corresponding to the RCP s In this case, the warping step is omitted and the image C s is calculated as:

[0050]

[0051] FIG. 3A to FIG. 3B Pseudocode (300, 302) that can be used to implement equation (8) according to an embodiment is shown. More specifically, the pseudocode (300) defines a first function that can be called to generate a weight based on the alpha value of the multi-plane image (200). Pseudo code (302) defines a second function C that can be called to render image C s The pseudo code (302) calls the first function at its "step-1". In some cases, the weight {W i} may be known from some other process. In this case, there is no need to call pseudo code (300) during execution of pseudo code (302).

[0052] In the exemplary embodiment, post-production editing (115) includes steps to adjust the image C s , so that any difference between the image and the source image R is approximately minimized, from which the corresponding multi-plane image (200) is generated. In mathematical terms, the goal of this processing can be expressed as follows. The processing is used to convert the value { where i=0, ..., D-1} is converted to the value { where i = 0, ..., D-1}. After such a transformation, the resulting rendered view corresponding to the reference camera position (RCP) Given by equation (9):

[0053]

[0054] Among them, the weight Expressed as:

[0055]

[0056] The optimization criterion for finding a suitable "best" conversion algorithm can then be expressed using equation (11) as follows:

[0057]

[0058] However, it should be noted that equation (11) represents an under-defined problem because the range of adjustable parameters is too large to obtain a deterministic solution. Therefore, according to various disclosed embodiments, the introduction of additional (e.g., implicit) constraints is used to obtain a near-optimal solution. The effectiveness of such additional constraints has been experimentally verified, and representative results according to various example embodiments will be described below with reference to Figures 10 to 11.

[0059] Post-production editing of multi-plane images

[0060] Figure 4 The present invention is a flowchart illustrating a method (400) of editing a multi-plane image (200) according to an embodiment. The method (400) uses a multi-plane image (200) as input, which multi-plane image can be generated, for example, as previously described. The editing method (400) is applied to process the input multi-plane image (200), thereby converting the input multi-plane image into a corresponding output multi-plane image (440). When the multi-plane image (440) is rendered using an MPI rendering algorithm, for example as described above, artifacts appearing in the corresponding visible image may be beneficially reduced or completely suppressed compared to a similar rendering of the input multi-plane image (200).

[0061] The method (400) includes a first processing block (410), wherein the alpha channel of the multi-plane image (200) is subjected to normalization. The resulting multi-plane image (412) is applied to a second processing block (420) of the method (400), wherein the alpha channel and the texture channel of the image (412) are subjected to alpha channel and texture channel refinement. The resulting multi-plane image (422) is applied to a third processing block (430) of the method (400), wherein the texture channel of the image (422) is subjected to scaling. The output of the third processing block (430) is a multi-plane image (440). Example embodiments of the processing blocks (410, 420, 430) are described in more detail below.

[0062] In some embodiments of the method (400), one of the processing blocks (410, 420, 430) may not be present. In some other embodiments of the method (400), two of the processing blocks (410, 420, 430) may not be present. In some embodiments of the method (400), the order in which the processing blocks (410, 420, 430) are executed may be different from Figure 4 The order indicated in .

[0063] Alpha channel normalization

[0064] One example type of artifact that may be observed when rendering a multiplane image (200) is a "dark pixel" artifact. For alpha channel values ​​corresponding to a "dark pixel" artifact, i Analysis of the {x, y)} reveals that the alpha channel values ​​of pixels corresponding to such artifacts are significantly lower than the alpha channel values ​​of pixels without artifacts in the same multiplane image layer, while the value ranges of the texture channels of both sets of pixels are similar. There is a similar trend in .

[0065] FIG. 5A to FIG. 5D Example analysis results according to an embodiment are graphically illustrated, which illustrate certain characteristics of "dark pixel" artifacts. The analysis performed includes calculating, for each pixel position (x, y), the mean of the absolute differences (MAD) between the source image R and the rendered image according to equation (12):

[0066]

[0067] The analysis performed further comprises calculating, for each pixel position (x, y), the sum of the weights according to equation (13):

[0068]

[0069] FIG. 5A to FIG. 5C The calculated MAD(x, y, c) and the color subchannels R, G and B of the texture channel are shown respectively. A scatter plot of . Figure 5D Shows The corresponding histogram of . Figure 5D As observed in The values ​​of are concentrated around 1. As becomes smaller, the distribution of MAD(x, y, c) becomes wider, and the error between the source image and the rendered image becomes larger. FIG. 5A to FIG. 5C This is clearly seen in the scatter plot of FIG. 5A to FIG. 5D The results of the analysis shown graphically in Figure 2 show that the correction (Therefore, and ) can improve the quality of the rendered image.

[0070] If the purpose of constructing the multi-plane image (440) is to Figure 2 ) matches the source image R when rendered, then for the pixel position (x, y) and the optimal parameters and Approximately satisfies the following equation:

[0071]

[0072] For most pixels without artifacts, the weight close to 1 (see, for example, Figure 5D Based on this observation, the processing block (410) is configured to weight Normalized to 1. The normalization condition can be expressed as follows:

[0073]

[0074] Equation (15) may be used to set a first example constraint for approximately solving the optimization problem formulated using equation (11).

[0075] In an example embodiment of processing block (410), the weight W of the pixel (x, y) may be performed according to equation (16): i Normalization of (x, y):

[0076]

[0077] in, represents the normalized weight. Equation (16) is conditionally applicable to layers i=0,...,D-1 and satisfying the condition to avoid division by zero in equation (16).

[0078] In an example embodiment of processing block (410), the normalized weights The corresponding modified alpha value It can be calculated by recursive backpropagation based on equation (17):

[0079]

[0080] Equations (18a)-(18e) provide explicit forms of equation (17) for different layers (from layer (D-1) to layer 0) of the corresponding multi-plane image:

[0081]

[0082]

[0083]

[0084]

[0085] …

[0086]

[0087] Starting from equation (18a), the modified alpha value for the (D-1)th layer can be calculated using equation (19):

[0088]

[0089] Once calculated The modified alpha value of the (D-2)th layer can be calculated based on equation (18b) as follows:

[0090]

[0091] Once calculated The modified alpha value of the (D-3)th layer can be calculated based on equation (18c) as follows:

[0092]

[0093] A person skilled in the art will readily appreciate that the recursive back-propagation process illustrated by equations (20) and (21) can be repeated for the remaining layers. Equation (22) provides the final formula for the recursive back-propagation process according to an embodiment:

[0094]

[0095] Figure 6 Pseudocode (600) is shown that can be used to implement at least some of the processing of the processing block (410) according to an embodiment. The pseudocode (600) includes three code blocks, respectively labeled as step-1, step-2 and step-3. Step-1 performs the conversion of alpha channel values ​​to weights. In an example embodiment, the pseudocode (300) (see Figure 3A ) to implement step-1 of pseudo code (600). Step-2 Calculate the normalized weight In an exemplary embodiment, step-2 of pseudo code (600) may be implemented based on equation (16). Step-3 Returns the modified alpha channel value In an example embodiment, step-3 of pseudocode (600) may be implemented using recursive back-propagation defined by equations (19)-(22).

[0096] Alpha channel and texture channel refinement

[0097] Although the processing implemented in the first processing block (410) of the method (400) is capable of removing a relatively large number of artifacts alone, some types of artifacts still exist after the processing of the artifacts is completed. One of such artifacts is a "black hole" artifact, for which the RGB pixel value of the corresponding pixel in the visible image generated by rendering the multi-plane image (412) is empty (i.e., (0,0,0)), while there is no feature in the source image R that may cause the empty value. It should be noted that the "black hole" artifact usually appears as an isolated small "hole" and does not appear in large quantities.

[0098] Since the rendered pixel value is a weighted linear combination of the corresponding pixel values ​​from D layers, and the weights in all D layers and texture channel value (C i (x, y, c)) are all non-negative, so the corresponding mathematical equations (23) and (24) are irreconcilable:

[0099]

[0100] R(x,y,c)> (24)

[0101] In other words, the result expressed by equation (24) cannot be achieved by equation (23) with positive weights and texture channel values. In an example embodiment, the second processing block (420) of method (400) solves this problem by employing local averaging as further described below. The disclosed local averaging is beneficially capable of substantially removing at least these "black hole" artifacts.

[0102] FIG. 7A to FIG. 7B Pseudocode (710, 720) that may be used to implement local averaging in the second processing block (420) of the method (400) according to an embodiment is shown. More specifically, Fig. 7A The pseudo code (710) is configured to perform local averaging on the alpha channel. Figure 7B The pseudo-code (720) is similarly configured to perform local averaging on the texture channels.

[0103] Both pseudocodes (710, 720) use a mean filter, which is a filter that calculates the average value within a sliding window. In some embodiments, the size B of the sliding window used in pseudocode (710) is a The size B of the sliding window used in the pseudo code (720) may be different c For example, the sliding window size can be B a =19 and B c=9. In some other embodiments, the sliding window size may be the same. The local channel average calculated using pseudo code (710, 720) may be expressed as follows:

[0104]

[0105]

[0106] Figure 8 Pseudocode (800) is shown that can be used to implement at least some of the processing of processing block (420) according to an embodiment. Pseudocode (800) includes three code blocks, respectively labeled as step-1, step-2 and step-3. Step-1 of pseudocode (800) is configured to perform a local average corresponding to equations (25), (26), and pseudocodes (710, 720) (see Fig. 7A , Figure 7B ) is implemented. Step-2 of the pseudo code (800) is configured to (i) search for "black hole" artifacts, and (ii) replace the "black hole" pixel values ​​with the corresponding local average values ​​calculated at step-1 of the pseudo code (800) using the corresponding local average values.

[0107] The "black hole" artifact that can be used to implement the search (i) of step-2 can be expressed as follows:

[0108] C(N)(x,y,c)=0 and R(x,y,c)>0 (27)

[0109] The pixel value replacement in step-2 (ii) results in the alpha channel value being and texture channel value C i (x, y, c) are converted into corresponding refined values ​​respectively and Equations (28) and (29) provide example mathematical formulas for implementing this refinement:

[0110]

[0111]

[0112] Step-3 of the pseudo code (800) may include applying the pseudo code (300, 302) to the refined value generated at step-2 of the pseudo code (800) and More specifically, first, the alpha channel value Normalization is required to generate the corresponding normalized value Second, use the updated alpha channel To calculate the weight Third, the corresponding composite multi-plane image can be rendered To generate a visual image C (HN) .

[0113] Texture channel scaling

[0114] Fig. 9 Pseudo code (900) is shown that may be used to implement at least some of the processing of processing block (430) according to an embodiment. FIG. 11A to FIG. 11B As illustrated, the general processing of the processing block (430) and the specific processing of the pseudo code (900) can be beneficially used to reduce object boundary artifacts in a visual image generated by rendering an output multi-plane image (440). The processing of the processing block (430) can also reduce blur in some portions of the visual image.

[0115] The input of the pseudo code (900) may include the source image R and the refined texture channel values ​​calculated at step-2 of the pseudo code (800) The scaling operation of pseudo code (900) is selectively applied to the visible image C (HN) Pixels that meet the following conditions:

[0116] C (HN) (x,y,c)>0 and R(x,y,c)>0 (30)

[0117] As already indicated above, the visible image C can be calculated at step-3 of the pseudo code (800) (HN) .

[0118] The effect of the scaling factor β applied in pseudocode (900) can be more clearly illustrated by equation (31):

[0119] βC (HN) (x,y,c)=R(x,y,c) (31)

[0120] More specifically, equation (31) explicitly shows that the scaling factor β is selected so that after scaling, the visible image C (HN) The scaled pixel value of is the same as the corresponding pixel value of the source image R. Rearrangement of equation (31) provides a formula for calculating the scaling factor β as follows:

[0121]

[0122] The processing of pseudocode (900) is configured to replace pixel values ​​that satisfy condition (30) with corresponding scaled pixel values. Pixel values ​​that do not satisfy condition (30) remain unchanged. Equation (33) provides an example mathematical formula for such conditional scaling:

[0123]

[0124] This formula can be used for Fig. 9 The pseudo code indicated in (900).

[0125] Improved Example

[0126] FIG. 10A to FIG. 10B An example visual improvement corresponding to the first processing block (410) of the method (400) is illustrated according to an embodiment. More specifically, Fig. 10A A portion of a visual image generated by rendering an input multi-plane image (200) depicting a violinist playing is shown. The "dark pixel" artifacts described above are clearly visible in the contour area around the violinist's upper arm. Fig. 10B The same portion of the visual image generated by rendering the multi-plane image (412) is shown. Fig. 10A The contour area of ​​the image shown is Fig. 10B Comparison of corresponding areas of the images shown provides a visual indication of the extent to which "dark pixel" artifacts can be corrected by the alpha channel normalization of the first processing block (410) of the method (400).

[0127] FIG. 11A to FIG. 11B An example visual improvement corresponding to the third processing block (430) of the method (400) is illustrated according to an embodiment. More specifically, Fig.11A shows the rendering from Fig. 10A A portion of a visible image generated by obtaining a multi-plane image (422) corresponding to the input multi-plane image (200) of a "violinist playing the violin". Fig. 11B The same portion of the visible image generated by rendering the output multi-plane image (440) is shown. Fig.11A The contour area of ​​the image shown is Fig. 11B Comparison of corresponding areas of the images shown provides a visual indication of the extent to which "object boundary" and "blurring" artifacts can be corrected by the texture channel scaling of the third processing block (430) of the method (400).

[0128] Example Hardware

[0129] Fig.12is a block diagram illustrating a computing device (1200) according to an embodiment. The device (1200) can be used, for example, at a post-production block (115). The device (1200) includes an input / output (I / O) device (1210), an image enhancement engine (IEE, 1220), and a memory (1230). The I / O device (1210) can be used to enable the device (1200) to receive at least a portion of a video / image production stream (112) and output at least a portion of a final video / image stream (117). The I / O device (1210) can also be used to connect the device (1200) to a reference display (125).

[0130] The memory (1230) may have a buffer for receiving the input multi-plane image (200), for example, via the video / image production stream (112). The input multi-plane image (200) may be in the form of an image file, for example. Once the input multi-plane image (200) is received, the memory (1230) may provide the image file to the IEE (1220) for processing therein. The IEE (1220) includes a processor (1222) and a memory (1224). The memory (1224) may store program codes therein, which, when executed by the processor (1222), enable the IEE (1220) to perform the method (400). The program codes may include, in particular, program codes that embody the various pseudo codes described above. Once the IEE (1220) converts the input multi-plane image (200) into a corresponding output multi-plane image (440) by executing the method (400), the IEE (1220) can perform rendering processing on the output multi-plane image and provide a corresponding visible image for viewing on the reference display (125). The visible image can be, for example, in the form of a suitable image file output through the I / O device (1210).

[0131] According to the above (for example, in the Summary section and / or reference Figures 1 to 12 ) discloses an exemplary embodiment of some or all of the figures in any one or any combination thereof, providing an apparatus for enhancing a first multi-plane image represented by a plurality of layers corresponding to different respective distances from a reference camera position, the apparatus comprising: at least one processor (e.g., Fig.12 1222); and at least one memory including program code (e.g., Fig.12 1224); and wherein at least one memory and program code are configured to, together with at least one processor, cause the apparatus to at least: for each pixel in the first set of pixels, scale the corresponding weight of the layer (e.g., Figure 6Step -2) so that the sum of the scaling weights is equal to a predetermined fixed value; for each pixel in the second set of pixels, replace the corresponding alpha value and texture value in the layer with the corresponding local average value (e.g., Figure 8 Step -2); and for each pixel in the third set of pixels, the corresponding texture value in the scaling layer (eg, Fig. 9 Step -1), so that for the resulting visual image rendered for the reference camera position, the texture value of each pixel in the third set matches the corresponding texture value of the reference image captured from the reference camera position. In various embodiments, the predetermined fixed value can be one or other appropriately selected positive fixed values.

[0132] In some embodiments of the above device, the second set is an empty set. For example, when there is no "black hole" artifact, the second pixel set is empty. The first pixel set and the third pixel set are usually not empty.

[0133] In some embodiments of any of the above apparatus, at least one memory and program code are configured to, with at least one processor, further cause the apparatus to generate a second multi-plane image (e.g., Figure 4 412): converting the alpha value of the first multi-plane image into a corresponding weight value (eg, Figure 6 Step -1); identifying a first set of pixels based on the corresponding weight values; and calculating an alpha value of a second multi-plane image using the scaled weights (e.g., Figure 6 Step-3).

[0134] In some embodiments of any of the above devices, at least one memory and program code are configured to, together with at least one processor, further cause the device to calculate alpha values ​​for the second multi-plane image by recursively back-propagating the scaling weights (e.g., equations (19)-(22)).

[0135] In some embodiments of any of the above-mentioned devices, at least one memory and program code are configured to, together with at least one processor, further enable the device to identify a second set of pixels by at least finding one or more empty texture values ​​in a visible image generated based on the second multi-plane image (e.g., equation (23)).

[0136] In some embodiments of any of the above apparatus, at least one memory and program code are configured to, with at least one processor, further cause the apparatus to generate a third multi-planar image based on the second multi-planar image (e.g., Figure 4 422), the second pixel set of the third multi-plane image has a corresponding local average value as the pixel value in the second pixel set.

[0137] In some embodiments of any of the above apparatus, the at least one memory and the program code are configured to, with the at least one processor, further cause the apparatus to perform alpha channel normalization on the third multi-plane image (e.g., Figure 8 Step-3).

[0138] In some embodiments of any of the above apparatus, the at least one memory and the program code are configured to, with the at least one processor, further cause the apparatus to perform an alpha to weight conversion to implement alpha channel normalization.

[0139] In some embodiments of any of the above apparatus, at least one memory and program code are configured to, with at least one processor, further cause the apparatus to generate a fourth multi-planar image based on the third multi-planar image (e.g., Figure 4 440), a third set of pixels of the fourth multi-plane image has alpha values ​​and texture values ​​that result in a match.

[0140] In some embodiments of any of the above apparatus, the at least one memory and the program code are configured to, with the at least one processor, further cause the apparatus to generate another visible image by rendering a fourth multiplane image for a virtual camera position different from the reference camera position.

[0141] According to the above (for example, in the Summary section and / or reference Figures 1 to 12 Another exemplary embodiment disclosed in any one or any combination of some or all of the figures in , provides a method for enhancing a first multi-plane image represented by a plurality of layers corresponding to different respective distances from a reference camera position, the method comprising the steps of: for each pixel in a first set of pixels, scaling the respective weights of the layers (e.g., Figure 6 Step -2) so that the sum of the scaled weights is a predetermined fixed value, and the scaling of the corresponding weights is performed using at least one processor (e.g., Fig.12 1222) and at least one memory including program code (e.g., Fig.12 1224) is performed; for each pixel in the second pixel set, the corresponding alpha value and texture value in the layer are replaced with the corresponding local average value (for example, Figure 8 Step -2), the replacement is performed using at least one processor and at least one memory; and for each pixel in the third set of pixels, the corresponding texture value in the scaling layer (e.g., Fig. 9 Step -1) such that, for a resulting visible image rendered for a reference camera position, a texture value for each pixel in the third set matches a corresponding texture value of a reference image captured from the reference camera position, scaling the corresponding texture values ​​being performed using at least one processor and at least one memory.

[0142] In some embodiments of the above method, the method further comprises generating a second multi-plane image (e.g., Figure 4 412): converting the alpha value of the first multi-plane image into a corresponding weight value (eg, Figure 6 Step -1); identifying a first set of pixels based on the corresponding weight values; and calculating an alpha value of a second multi-plane image using the scaled weights (e.g., Figure 6 Step-3).

[0143] In some embodiments of any of the above methods, the method further includes calculating alpha values ​​for the second multi-plane image by recursively back-propagating the scaling weights (eg, equations (19)-(22)).

[0144] In some embodiments of any of the above methods, the method further includes identifying a second set of pixels by finding one or more null texture values ​​in at least a visible image generated based on the second multi-planar image (eg, equation (23)).

[0145] In some embodiments of any of the above methods, the method further includes generating a third multi-planar image based on the second multi-planar image (e.g., Figure 4 422), the second pixel set of the third multi-plane image has a corresponding local average value as the pixel value in the second pixel set.

[0146] In some embodiments of any of the above methods, the method further includes performing alpha channel normalization on the third multi-plane image (e.g., Figure 8 Step-3).

[0147] In some embodiments of any of the above methods, the method further includes performing an alpha to weight conversion to achieve alpha channel normalization.

[0148] In some embodiments of any of the above methods, the method further includes generating a fourth multi-planar image based on the third multi-planar image (e.g., Figure 4 440), a third set of pixels of the fourth multi-plane image has alpha values ​​and texture values ​​that result in a match.

[0149] In some embodiments of any of the methods above, the method further comprises generating another viewable image by rendering a fourth multiplane image for a virtual camera position different from the reference camera position.

[0150] According to the above (for example, in the Summary section and / or reference Figures 1 to 12Yet another example embodiment disclosed in any one or any combination of some or all of the figures in , provides a non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform operations including a method for enhancing a first multi-plane image represented by a plurality of layers corresponding to different respective distances from a reference camera position, the method comprising the steps of: for each pixel in a first set of pixels, scaling a respective weight of the layer (e.g., Figure 6 Step -2) so that the sum of the scaled weights is a predetermined fixed value, and the scaling of the corresponding weights is performed using at least one processor (e.g., Fig.12 1222) and at least one memory including program code (e.g., Fig.12 1224) is performed; for each pixel in the second pixel set, the corresponding alpha value and texture value in the layer are replaced with the corresponding local average value (for example, Figure 8 Step -2), the replacement is performed using at least one processor and at least one memory; and for each pixel in the third set of pixels, the corresponding texture value in the scaling layer (e.g., Fig. 9 Step -1) such that, for a resulting visible image rendered for a reference camera position, a texture value for each pixel in the third set matches a corresponding texture value of a reference image captured from the reference camera position, scaling the corresponding texture values ​​being performed using at least one processor and at least one memory.

[0151] With respect to the processes, systems, methods, heuristics, etc. described herein, it should be understood that although the steps of these processes, etc. have been described as being performed in a particular ordered order, these processes can be practiced with the described steps performed in an order different from that described herein. It should further be understood that certain steps can be performed simultaneously, other steps can be added, or certain steps described herein can be omitted. In other words, the process descriptions herein are provided for the purpose of illustrating certain embodiments and should in no way be construed as limiting the claims.

[0152] Accordingly, it should be understood that the above description is intended to be illustrative rather than restrictive. By reading the above description, many embodiments and applications other than the examples provided will be apparent. The scope should not be determined with reference to the above description, but should be determined with reference to the full range of equivalents to which the appended claims and these claims are eligible for consideration. It is expected and desired that the technology discussed herein will develop in the future, and the disclosed systems and methods will be incorporated into such future embodiments. In a word, it should be understood that the present application is capable of modification and change.

[0153] All terms used in the claims are intended to be given their broadest reasonable interpretations and ordinary meanings as understood by those skilled in the art described herein, unless an explicit indication to the contrary appears herein. In particular, use of singular articles such as "a," "an," "the," and "said" should be understood to recite one or more of the indicated elements unless a claim recites an explicit limitation to the contrary.

[0154] The abstract of the present disclosure is provided to allow the reader to quickly determine the nature of the technical disclosure. This abstract is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. In addition, in the aforementioned detailed description, it can be seen that various features are combined together into various embodiments for the purpose of connecting the present disclosure as a whole. The method of the present disclosure should not be interpreted as reflecting the intention that the claimed embodiments incorporate more features than those explicitly recited in each claim. On the contrary, as reflected in the attached claims, the inventive subject matter lies in less than all the features of a single disclosed embodiment. Therefore, the attached claims are hereby incorporated into the detailed description, and each claim stands alone as a separately claimed subject matter.

[0155] Although the present disclosure includes references to illustrative embodiments, the description is not intended to be interpreted as limiting. Various modifications of the described embodiments that are obvious to those skilled in the art to which the present disclosure relates, as well as other embodiments within the scope of the present disclosure, are considered to be within the principles and scope of the present disclosure, for example, as expressed in the claims.

[0156] Some embodiments may be embodied in the form of methods and apparatus for practicing the methods. Some embodiments may also be embodied in the form of program code recorded in a tangible medium, such as a magnetic recording medium, an optical recording medium, a solid-state memory, a floppy disk, a CD-ROM, a hard drive, or any other non-transitory machine-readable storage medium, wherein when the program code is loaded into and executed by a machine (such as a computer, etc.), the machine becomes an apparatus for practicing the patented invention(s). When implemented on a general-purpose processor, the program code segments combine with the processor to provide a unique device that operates similarly to a specific logic circuit.

[0157] Unless expressly stated otherwise, each numerical value and range should be interpreted as being approximate as if the value or range were preceded by the word "about" or "approximately."

[0158] The use of figure numbers and / or reference numerals in the claims is intended to identify one or more possible embodiments of the claimed subject matter to facilitate the interpretation of the claims. Such use should not be construed as necessarily limiting the scope of these claims to the embodiments shown in the corresponding figures.

[0159] Although elements in the method claims (if any) are recited in a specific order with corresponding labels, these elements are not necessarily intended to be limited to being implemented in the specific order unless the claim recitation otherwise implies a specific order for implementing some or all of these elements.

[0160] References herein to "one embodiment" or "an embodiment" mean that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the present disclosure. In this specification, the appearance of the phrase "in one embodiment" in various places does not necessarily refer to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. The same applies to the term "implementation".

[0161] Unless otherwise specified herein, the use of ordinal adjectives "first," "second," "third," etc. to refer to an object among multiple similar objects merely indicates that different instances of such similar objects are being referred to, and is not intended to imply that the similar objects so referred to must be in a corresponding order or sequence, whether temporally, spatially, in ranking, or in any other manner.

[0162] Unless otherwise specified herein, in addition to its plain meaning, the conjunction "if" may also or alternatively be interpreted to mean "when" or "at" or "in response to determining" or "in response to detecting", which interpretation may depend on the corresponding specific context. For example, the phrase "if it is determined that..." or "if [the stated condition] is detected" may be interpreted to mean "upon determining that..." or "in response to determining that..." or "upon detecting [the stated condition or event]" or "in response to detecting [the stated condition or event]".

[0163] Also for the purposes of this description, the terms "couple", "coupling", "coupled", "connect", "connecting", or "connected" refer to any means known in the art or later developed to allow energy to be transferred between two or more elements, and the insertion of one or more additional elements is contemplated, although this is not required. In contrast, the terms "directly coupled", "directly connected", etc. imply the absence of such additional elements.

[0164] The functions of the various elements shown in the figure, including any functional blocks marked or referred to as including "processors" and / or "controllers", can be provided by using dedicated hardware and hardware capable of executing software associated with appropriate software. When provided by a processor, the function can be provided by a single dedicated processor, by a single shared processor, or by multiple separate processors (some of which can be shared). In addition, the explicit use of the term "processor" or "controller" should not be interpreted as exclusively referring to hardware capable of executing software, and may implicitly include but is not limited to digital signal processor (DSP) hardware, network processors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), read-only memories (ROMs), random access memories (RAMs) and non-volatile memories for storing software. Other traditional and / or customized hardware may also be included. Similarly, any switches shown in the figure are only conceptual. Their functions can be performed by the operation of program logic, by dedicated logic, by the interaction of program control and dedicated logic, or even manually, and specific techniques can be selected by the implementer, as more specifically understood from the context.

[0165] As used in this application, the terms "circuit" and "circuitry" may refer to one or more or all of the following: (a) a pure hardware circuit implementation (such as an implementation in a pure analog circuit and / or digital circuit); (b) a combination of hardware circuit and software, such as (if applicable): (i) a combination of analog hardware circuit and / or digital hardware circuit and software / firmware, and (ii) any part of a hardware processor with software (including a digital signal processor), software and memory, which work together to enable a device (such as a mobile phone or server, etc.) to perform various functions; and (c) a hardware circuit and / or processor, such as a microprocessor or a part of a microprocessor, which requires software (e.g., firmware) to operate, but the software may not be present when it is not required for operation. This definition of circuit applies to all uses of the term in this application (including all claims). As a further example, as used in this application, the term "circuit" also covers an implementation of only one hardware circuit or one processor (or multiple processors) or a part of a hardware circuit or processor and its accompanying software and / or firmware. The term "circuitry" also covers, for example, a baseband integrated circuit or a processor integrated circuit for a mobile device, or a similar integrated circuit in a server, cellular network device, or other computing or networking device, if applicable to a particular claim element.

[0166] Those skilled in the art will recognize that any block diagrams herein represent conceptual views of illustrative circuits that embody the principles of the present disclosure. Similarly, it will be recognized that any flow charts, flow diagrams, state transition diagrams, pseudocodes, etc. represent various processes that can be substantially represented in a computer-readable medium and thus executed by a computer or processor, whether or not the computer or processor is explicitly shown.

[0167] The Summary of the Invention in this specification is intended to introduce some example embodiments, and additional embodiments are described in the Detailed Description and / or with reference to one or more figures. The Summary of the Invention is not intended to identify essential elements or features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.

Claims

1. An apparatus for enhancing a first multi-plane image, the first multi-plane image being represented by a plurality of layers corresponding to different respective distances from a reference camera position, the apparatus comprising: at least one processor; as well as at least one memory including program code; and The at least one memory and the program code are configured to, together with the at least one processor, cause the apparatus to at least: For each pixel in the first set of pixels, scaling the corresponding weight of the layer so that the sum of the scaled weights is equal to a predetermined fixed value; For each pixel in the second set of pixels, replacing the corresponding alpha value and texture value in the layer with the corresponding local average value; and For each pixel in a third set of pixels, a corresponding texture value in the layer is scaled so that, for a resulting visible image rendered for the reference camera position, the texture value of each pixel in the third set matches a corresponding texture value of a reference image captured from the reference camera position.

2. The device according to claim 1, wherein: The second set is an empty set.

3. The device according to claim 1 or 2, wherein: The at least one memory and the program code are configured to, with the at least one processor, further cause the apparatus to generate a second multi-plane image by at least: converting the alpha value of the first multi-plane image into a corresponding weight value; identifying a first set of pixels based on the corresponding weight values; as well as An alpha value for the second multi-planar image is calculated using the scaling weights.

4. The device according to claim 3, wherein: The at least one memory and the program code are configured to, with the at least one processor, further cause the apparatus to calculate an alpha value for the second multi-plane image by recursively back-propagating the scaling weights.

5. The device according to claim 3 or 4, wherein: The at least one memory and the program code are configured to, with the at least one processor, further cause the apparatus to identify the second set of pixels by finding one or more null texture values ​​in at least a visible image generated based on the second multi-plane image.

6. The device according to any one of claims 3 to 5, wherein: The at least one memory and the program code are configured to, with the at least one processor, further cause the apparatus to generate a third multiplanar image based on the second multiplanar image, the second set of pixels of the third multiplanar image having corresponding local averages as pixel values ​​in the second set of pixels.

7. The device according to claim 6, wherein: The at least one memory and the program code are configured to, with the at least one processor, further cause the apparatus to perform alpha channel normalization on the third multi-plane image.

8. The device according to claim 7, wherein: The at least one memory and the program code are configured to, with the at least one processor, further cause the apparatus to perform an alpha-to-weight conversion to implement the alpha channel normalization.

9. The device according to any one of claims 6 to 8, wherein: The at least one memory and the program code are configured to, together with the at least one processor, further cause the apparatus to generate a fourth multiplanar image (e.g., 440 of FIG. 4) based on the third multiplanar image, a third set of pixels of the fourth multiplanar image having alpha values ​​and texture values ​​that result in the match.

10. The device of claim 9, wherein: The at least one memory and the program code are configured to, with the at least one processor, further cause the apparatus to generate another visible image by rendering the fourth multi-plane image for a virtual camera position different from the reference camera position.

11. A method for enhancing a first multi-plane image, the first multi-plane image being represented by a plurality of layers corresponding to different respective distances from a reference camera position, the method comprising: For each pixel in the first set of pixels, scaling the corresponding weights of the layer so that a sum of the scaled weights is a predetermined fixed value, the scaling of the corresponding weights being performed using at least one processor and at least one memory including program code; for each pixel in the second set of pixels, replacing a corresponding alpha value and a texture value in the layer with a corresponding local average, the replacing being performed using the at least one processor and the at least one memory; as well as For each pixel in a third set of pixels, scaling a corresponding texture value in the layer so that, for a resulting visible image rendered for the reference camera position, the texture value of each pixel in the third set matches a corresponding texture value of a reference image captured from the reference camera position, wherein the scaling of the corresponding texture values ​​is performed using the at least one processor and the at least one memory.

12. The method of claim 11, further comprising generating a second multi-planar image by at least: converting the alpha value of the first multi-plane image into a corresponding weight value; identifying a first set of pixels based on the corresponding weight values; and An alpha value for the second multi-planar image is calculated using the scaling weights.

13. The method of claim 12, further comprising calculating an alpha value for the second multi-plane image by recursively back-propagating the scaling weights.

14. The method of claim 12 or claim 13, further comprising identifying the second set of pixels by finding one or more null texture values ​​in at least a visible image generated based on the second multi-plane image.

15. The method of any one of claims 12 to 14, further comprising A third multiplanar image is generated based on the second multiplanar image, the second set of pixels of the third multiplanar image having a corresponding local average value as a pixel value in the second set of pixels.

16. The method of claim 15, further comprising performing alpha channel normalization on the third multi-planar image.

17. The method of claim 16, further comprising performing an alpha to weight conversion to achieve the alpha channel normalization.

18. The method of any one of claims 15 to 17, further comprising A fourth multiplanar image is generated based on the third multiplanar image, a third set of pixels of the fourth multiplanar image having alpha values ​​and texture values ​​that result in the match.

19. The method of claim 18, further comprising Another viewable image is generated by rendering the fourth multiplane image for a virtual camera position different from the reference camera position.

20. A non-transitory computer readable medium storing instructions that, when executed by a processor, cause the processor to perform operations including the method of any one of claims 11 to 19.

Images