Method and electronic device for generating multi-plane image from video

Abstract

The generation of a multi-plane image from a video is provided. This method may comprise the steps of: generating a coarse multi-plane image of any one video frame from among video frames on the basis of the similarity between the video frames; and generating a fine multi-plane image of any one video frame by performing inpainting on the coarse multi-plane image.

Description

Method and electronic device for generating multi-planar images from video

[0001] The present disclosure relates to image processing. More specifically, the present disclosure relates to a method for generating a multi-planar image using video frames.

[0002] With the advent of digital cameras, image processing technologies have rapidly advanced. Image processing algorithms have been developed to improve image quality, such as correcting lens aberrations, enhancing color, and enhancing sharpness.

[0003] Beyond improving image quality, algorithms for image synthesis are being developed. These algorithms enable the creation of composite 3D images from 2D images. However, these algorithms have the limitation of using 2D images captured at the same moment in time to create composite 3D images.

[0004] According to one aspect of the present disclosure, a method for generating a multi-planar image from a video may be provided. The method may include generating a coarse multi-planar image of one of the video frames based on similarity between the video frames. The method may include generating a fine multi-planar image of one of the video frames by performing an inpainting process on the coarse multi-planar image.

[0005] According to one aspect of the present disclosure, an electronic device for generating a multi-planar image from a video may be provided. The electronic device may include a memory storing one or more instructions and one or more processors for executing the one or more instructions stored in the memory. The one or more processors may be configured to generate a coarse multi-planar image of one of the video frames based on similarities between the video frames by executing the one or more instructions. The one or more processors may be configured to generate a fine multi-planar image of one of the video frames by performing an inpainting process on the coarse multi-planar image by executing the one or more instructions.

[0006] According to one aspect of the present disclosure, a computer-readable recording medium having recorded thereon a program for executing any one of the above-described and below-described methods for generating a multi-planar image from a video can be provided.

[0007] FIG. 1 is a diagram illustrating a multi-planar image generated from a video according to one embodiment.

[0008] FIG. 2 is a drawing illustrating operations of an electronic device according to one embodiment.

[0009] Figures 3 and 4 are drawings explaining the operations of a coarse MPI generation module and a fine MPI generation module according to embodiments.

[0010] FIG. 5 is a diagram illustrating the operations of a rough MPI generation module according to one embodiment.

[0011] FIG. 6 is a diagram illustrating the operations of a spatiotemporal feature block processing module according to one embodiment.

[0012] Figure 7 is a drawing explaining transmittance according to one embodiment.

[0013] FIG. 8 is a diagram for explaining the operations of a fine MPI generation module according to one embodiment.

[0014] FIG. 9 is a drawing illustrating a method for generating a mask based on transmittance according to one embodiment.

[0015] FIGS. 10 and 11 are drawings illustrating a method for generating a mask based on transmittance according to embodiments.

[0016] FIG. 12 is a flowchart of a method for generating multi-planar images from a video according to one embodiment.

[0017] Figure 13 is a diagram for comparing the performance of the proposed method and the conventional method according to one embodiment.

[0018] Figure 14 is a drawing for explaining the performance of the proposed method according to one embodiment.

[0019] Figures 15a and 15b are block diagrams of electronic devices according to embodiments.

[0020] Hereinafter, terms used in this specification will be briefly described, and the present disclosure will be described in detail. In this disclosure, the expression “at least one of a, b, or c” can refer to “a,” “b,” “c,” “a and b,” “a and c,” “b and c,” “all of a, b, and c,” or variations thereof.

[0021] The terms used in this disclosure are selected from widely used, common terms, taking into account the functions of the disclosure. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, terms may be arbitrarily selected by the applicant, in which case their meanings will be described in detail in the relevant description. Therefore, the terms used in this disclosure should not be defined simply as names, but rather based on the meanings of the terms and the overall content of the disclosure.

[0022] Singular expressions may include plural expressions unless the context clearly indicates otherwise. Terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art described herein. Furthermore, terms containing ordinal numbers, such as "first" or "second," used herein may be used to describe various components, but such components should not be limited by such terms. Such terms are used solely to distinguish one component from another.

[0023] When a part of the specification is said to "include" a component, unless otherwise specifically stated, this does not exclude other components but rather implies the inclusion of other components. Furthermore, terms such as "part" and "module" used in the specification refer to a unit that processes at least one function or operation, which may be implemented in hardware, software, or a combination of hardware and software.

[0024] In the present disclosure, a 'multi-planar image' may mean an image expressed by image planes of different depths.

[0025] In the present disclosure, a 'rough multi-planar image' may mean a multi-planar image on which inpainting processing has not been performed.

[0026] In the present disclosure, a 'micro multi-planar image' may mean a multi-planar image on which inpainting processing has been performed.

[0027] In the present disclosure, a 'spatiotemporal feature block' may mean a feature representation representing spatiotemporal characteristics of a video frame.

[0028] Below, with reference to the attached drawings, embodiments of the present disclosure are described in detail so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein.

[0029] The present disclosure will be described in detail with reference to the attached drawings below.

[0030] FIG. 1 is a drawing illustrating a multi-plane image (110) generated from a video according to one embodiment.

[0031] A video file may contain video frames. When a video file is played, the video frames may be displayed sequentially on a display panel.

[0032] Each video frame can represent a still 2D image. The 2D image can be a monochrome image or a color image. A color image can consist of at least two color channels. For example, the color image can be an RGB image.

[0033] FIG. 1 illustrates a multi-planar image (110) generated from a video frame (103) of a video file. The multi-planar image (110) may include a plurality of image planes (111 to 113). The image planes (111 to 113) may represent information of the video frame (103) at different depths.

[0034] A multi-planar image (110) can be generated from a video frame (103) of a video file and other video frames (101, 102, 104, 105). The video frames (101, 102, 104, 105) can be frames that include information for generating a multi-planar image (110) of the video frame (103). For example, the video frames (101, 102, 104, 105) can be frames in which at least one subject identical to the video frame (103) is recorded. For example, the video frames (101 to 105) can be consecutive frames of the video frame (103).

[0035] According to one embodiment, a sophisticated multi-planar image (110) can be generated by utilizing spatiotemporal information provided from a video frame (103) and other video frames (101, 102, 104, 105).

[0036] Below, various embodiments of a method for generating a multi-planar image using video frames are described.

[0037] FIG. 2 is a drawing illustrating operations of an electronic device (200) according to one embodiment.

[0038] In one embodiment, the electronic device (200) may include a coarse MPI (multi-planar image) generation module (210) and a fine MPI (multi-planar image) generation module (220).

[0039] The modules (210, 220) illustrated in FIG. 2 may be configurations implemented by at least one processor included in the electronic device (200) executing a program or command stored in a memory included in the electronic device (200). Accordingly, the operations described below as being performed by the modules (210, 220) of the electronic device (200) may actually be performed by at least one processor included in the electronic device (200).

[0040] The rough MPI generation module (210) can generate a rough multi-planar image from video frames. In other words, the rough MPI generation module (210) can generate a rough multi-planar image of a video frame at a single moment from video frames at multiple time periods.

[0041] The rough MPI generation module (210) can generate a rough multi-planar image based on similarities between video frames. The rough MPI generation module (210) can generate a rough multi-planar image of a video frame based on similarities between one of the video frames and other video frames.

[0042] The fine MPI generation module (220) can generate a fine multi-planar image by performing inpainting processing on a coarse multi-planar image. The fine MPI generation module (220) can generate a corresponding image plane of the fine multi-planar image by performing inpainting processing on each image plane of the coarse multi-planar image.

[0043] FIG. 3 is a diagram illustrating the operations of a coarse MPI generation module (310) and a fine MPI generation module (320) according to one embodiment.

[0044] The coarse MPI generation module (310) can generate a coarse multi-planar image (Coarse MPI) from video frames and depth maps. Here, the depth maps can be depth maps of each of the video frames.

[0045] In one embodiment, the coarse MPI generation module (310) may utilize consecutive video frames and their depth maps. In another embodiment, the coarse MPI generation module (310) may utilize skipped video frames and their depth maps.

[0046] In one embodiment, the coarse MPI generation module (310) may utilize video frames and their depth maps of the same view. In another embodiment, the coarse MPI generation module (310) may utilize video frames and their depth maps of a different view.

[0047] Electronic devices synthesize images (from coarse multi-planar images (Course MPI)) ) and composite images ( ) of the depth map( ) can be created.

[0048] The fine MPI generation module (320) can generate a fine multi-planar image (Fine MPI) by performing inpainting processing on a coarse multi-planar image (Course MPI). The fine MPI generation module (320) can perform inpainting processing using a deep learning-based inpainting model.

[0049] A mask (M) based on transmittance may be used to generate training images for training an inpainting model. In one embodiment, a coarse multi-planar image (Course MPI) masked by the mask (M) may be used as a training image for training an inpainting model.

[0050] The electronic device synthesizes images from fine multi-planar images (Fine MPI). ) can be generated. In one embodiment, a synthetic image ( ) may be images of different views than the video frames input to the rough MPI generation module (310). In one embodiment, the electronic device uses homography warping and volume rendering to create a synthetic image ( ) can be created.

[0051] In another embodiment, a synthetic image ( ) may be an image of a light field. In another embodiment, a synthetic image ( ) may be an image that forms a stereo view with a video frame input to the rough MPI generation module (310). In another embodiment, a synthetic image ( ) may be an image constituting a holography. In another embodiment, the composite image may be an image constituting a 3D image.

[0052] FIG. 4 is a diagram illustrating the operations of a coarse MPI generation module (410) and a fine MPI generation module (420) according to one embodiment.

[0053] In Fig. 4, the rough MPI generation module (410) generates video frames (I) of t-T1 to t+T2. t-T1 , ..., I t , ..., I t+T2 ) and depth maps (D t-T1 , ..., D t , ..., D t+T2 ) from the video frame (I) at time t t ) of coarse multi-planar images (Course MPI t ) is shown as an example of generating a

[0054] T1 and T2 can be any numbers. If T1 = 0, the rough MPI generation module (410) generates video frames (I t , ..., I t+T2 ), using the most advanced video frame (I t ) of coarse multi-planar images (Course MPI t ) can be generated. When T2 = 0, the rough MPI generation module (410) generates video frames (I t-T1 , ..., I t ), using the latest video frame (I t ) of coarse multi-planar images (Course MPI t) can be generated. When T1=T2, the rough MPI generation module (410) generates video frames (I t-T1 , ..., I t , ..., I t+T2 ), using the middle video frame (I t ) of coarse multi-planar images (Course MPI t ) can be created.

[0055] The fine MPI generation module (420) generates a coarse multi-planar image (Course MPI t ) by performing inpainting processing on each image plane of the video frame (I) at time t. t ) of fine multi-planar images (Fine MPI) t ) can be created.

[0056] Figure 4 shows a rough multi-planar image (Course MPI t ) from N image planes of fine multi-planar image (Fine MPI) t ) is shown as an example of generating N image planes. The fine MPI generation module (420) generates a coarse multi-planar image (Course MPI t ) by performing inpainting processing on the first image plane of the fine multi-planar image (Fine MPI t ) can generate the first image plane. In the same way, the fine MPI generation module (420) can generate a coarse multi-plane image (Course MPI t ) by performing inpainting processing on the 2nd to Nth image planes, respectively, to obtain fine multi-planar images (Fine MPI t ) can generate the 2nd to Nth image planes respectively.

[0057] FIG. 5 is a diagram illustrating the operations of a rough MPI generation module (500) according to one embodiment.

[0058] The rough MPI generation module (500) can generate a rough multi-planar image by synthesizing information of video frames based on similarities between the video frames. More specifically, the rough MPI generation module (500) can generate a rough multi-planar image for a video frame by synthesizing information of video frames based on similarities between one video frame and other video frames.

[0059] Similarity between video frames can indicate similarity between pixel values ​​and depth values ​​of the video frames. Here, the pixel values ​​can include color information or brightness information of the video frame. Additionally, the depth values ​​can include disparity information of the depth map of the video frame.

[0060] The rough MPI generation module (500) can generate a rough multi-planar image through operations in feature space. To this end, the rough MPI generation module (500) can include an encoder (510), a spatiotemporal feature block processing module (520), and a decoder (530).

[0061] In Fig. 5, a rough MPI generation module (500) generates pixel values ​​(P) of video frames from t-T1 to t+T2. t-T1 , ..., P t+T2 ) and depth values ​​(d t-T1 , ..., d t+T2 ) to obtain a coarse multi-planar image of a video frame at time t (Course MPI t ) is shown as an example of generating a

[0062] The encoder (510) can generate spatiotemporal feature blocks by embedding pixel values ​​and depth values ​​of each video frame. For example, the encoder (510) can embed pixel values ​​(P) of a video frame at time t-T1. t-T1 ) and depth values ​​(d t-T1) can be used to generate spatiotemporal feature blocks of time t-T1. In the same manner, the encoder (510) can generate spatiotemporal feature blocks of each time by embedding pixel values ​​and depth values ​​of the video frame of each time.

[0063] Spatiotemporal feature blocks generated from video frames can represent spatial characteristics derived from pixel values ​​and depth values ​​of the video frames and temporal characteristics derived from changes in pixel values ​​and depth values ​​over time of the video frames.

[0064] The spatiotemporal feature block processing module (520) can perform operations on spatiotemporal feature blocks generated by the encoder (510). The operations performed by the spatiotemporal feature block processing module (520) on the spatiotemporal feature blocks can include multiplication, element-wise multiplication, weighted sum, scale, dimensional transformation, or nonlinear operations, and are not limited to the examples listed.

[0065] The spatiotemporal feature block processing module (520) can calculate similarity between video frames through operations between spatiotemporal feature blocks. For example, similarity between a video frame at time t and a video frame at time t-1 can be calculated through operations between spatiotemporal feature blocks generated from a video frame at time t and spatiotemporal feature blocks generated from a video frame at time t-1.

[0066] The spatiotemporal feature block finally generated by the spatiotemporal feature block processing module (520) may include information on the video frame of time t and information reflecting the similarity of the video frames of times t-T1 to t-1 and t+1 to t+T2 with the video frame of time t. For example, the spatiotemporal feature block finally generated may include pixel values ​​(P) of the video frame of time t. t ) and depth values ​​(d t) may include information about the video frame at time t+1. In addition, the finally generated spatiotemporal feature block may include pixel values ​​(P) of the video frame at time t. t ) and depth values ​​(d t ), pixel values ​​(P) that reflect the similarity with t+1 ) and depth values ​​(d t+1 ) may contain information about.

[0067] The decoder (530) de-embeds the spatiotemporal feature block finally generated by the spatiotemporal feature block processing module (520) to produce a coarse multi-planar image (Course MPI) of the video frame of time t. t ) can be created.

[0068] FIG. 6 is a diagram illustrating the operations of a spatiotemporal feature block processing module (600) according to one embodiment.

[0069] In one embodiment, the operations performed by the spatiotemporal feature block processing module (600) to generate a rough multi-planar image from video frames may be based on an attention mechanism.

[0070] FIG. 6 illustrates an example of operations of a spatiotemporal feature block processing module (600) to generate a rough multi-planar image of a video frame at time t from video frames at t-T1 to t+T2 based on an attention mechanism.

[0071] The spatiotemporal feature blocks embedded by the encoder may include a query block (610), a key block (620), and a value block (630). The query block (610) includes a spatiotemporal feature block (f) in which pixel values ​​and depth values ​​of a video frame at time t are embedded. q_t ) may be included. The key block (620) includes spatiotemporal feature blocks (f) in which pixel values ​​and depth values ​​of video frames of time t-T1 to t+T2 are embedded. k_t-T1 , ..., f k_t+T2) may be included. The value block (630) includes spatiotemporal feature blocks (f) in which pixel values ​​and depth maps of video frames from t-T1 to t+T2 are embedded. v_t-T1 , ..., f v_t+T2 ) may be included.

[0072] The spatiotemporal feature block processing module (600) can perform an attention mechanism for the query block (610), the key block (620), and the value block (630). The spatiotemporal feature block processing module (600) can process the spatiotemporal feature block (f) of the query block (610). q_t ), the spatiotemporal feature blocks (f) of the key block (620) k_t-T1 , ..., f k_t+T2 ) can be calculated respectively. The spatiotemporal feature block processing module (600) uses the calculated similarity as a weight to calculate the spatiotemporal feature blocks (f) of the value block (630). v_t-T1 , ..., f v_t+T2 ) can be weighted and added. The spatiotemporal feature block processing module (600) processes spatiotemporal feature blocks (f v_t-T1 , ..., f v_t+T2 ) can be used to generate a final spatiotemporal feature block (640).

[0073] The spatiotemporal feature block processing module (600) processes the spatiotemporal feature block (f q_t ) from the query matrix (f q (.)) and generate spatiotemporal feature blocks (f k_t-T1 , ..., f k_t+T2 ) from the key matrix (f k (.)) can be generated. The spatiotemporal feature block processing module (600) generates a spatiotemporal feature block (f q_t ) and spatiotemporal feature blocks (f k_t-T1 , ..., f k_t+T2 ) by transforming the dimension of the query matrix (f q (.)) and key matrix (f k (.)) can be created respectively.

[0074] The spatiotemporal feature block processing module (600) processes the query matrix (f q (.)) and key matrix (f k From the product of (.)), the attention score matrix can be obtained. The attention score matrix is ​​a spatiotemporal feature block (f q_t ) and spatiotemporal feature blocks (f k_t-T1 , ..., f k_t+T2 ), that is, the similarity between the pixel values ​​and depth values ​​of the video frame at time t and the pixel values ​​and depth values ​​of the video frames at times t-T1~t-1 and t+1~t+T2.

[0075] The spatiotemporal feature block processing module (600) can apply a softmax function to an attention score matrix. The spatiotemporal feature block processing module (600) can obtain an attention distribution from the output of the softmax function.

[0076] The spatiotemporal feature block processing module (600) processes spatiotemporal feature blocks (f v_t-T1 , ..., f v_t+T2 ) from the value matrix (f v (.)) can be generated. The spatiotemporal feature block processing module (600) generates spatiotemporal feature blocks (f v_t-T1 , ..., f v_t+T2 ) by transforming the dimension of the value matrix (f v (.)) can be created.

[0077] The spatiotemporal feature block processing module (600) converts the attention distribution into a value matrix (f v By multiplying (.) by the attention value matrix, an attention value matrix can be obtained. The spatiotemporal feature block processing module (600) can generate a final spatiotemporal feature block (640) from the attention value matrix. The spatiotemporal feature block processing module (600) can generate a final spatiotemporal feature block (640) by converting the dimension of the attention value matrix.

[0078] In one embodiment, the spatiotemporal feature block processing module (600) may utilize patches of any size when performing the attention mechanism. For example, the spatiotemporal feature block processing module (600) may utilize patches of size kxk.

[0079] In one embodiment, the spatiotemporal feature block processing module (600) can perform operations between patches (611) of a query block (610) and corresponding patches (621, 631) of a key block (620) and a value block (630).

[0080] In one embodiment, the spatiotemporal feature block processing module (600) processes the spatiotemporal feature block (f q_t ) patches (611) and spatiotemporal feature blocks (f k_t-T1 , ..., f k_t+T2 ) can calculate patch-wise similarity with patches (621). That is, the spatiotemporal feature block processing module (600) can calculate patch-wise similarity between a patch of a video frame at time t and corresponding patches of video frames at times t-T1 to t-1 and t+1 to t+T2 based on an attention mechanism.

[0081] In one embodiment, the spatiotemporal feature block processing module (600) weights the similarity of the produced patch units to spatiotemporal feature blocks (f v_t-T1 , ..., f v_t+T2 ) can be weighted and added to the corresponding patches (631).

[0082] In one embodiment, the spatiotemporal feature block processing module (600) can set a patch for each of the pixels of the query block (610), the key block (620), and the value block (630). For example, for any pixel of the query block (610), the spatiotemporal feature block processing module (600) can set a kxk sized patch centered on the arbitrary pixel. The spatiotemporal feature block processing module (600) can generate a final spatiotemporal feature block (640) by applying an attention mechanism to all of the set patches.

[0083] The decoder can generate a coarse multi-planar image of a video frame at time t by de-embedding the final spatiotemporal feature block (640).

[0084] The attention score matrix of the spatiotemporal feature block processing module (600) represents the similarity between the video frame of time t and the video frames of times t-T1 to t-1 and t+1 to t+T2. In addition, the attention value matrix of the spatiotemporal feature block processing module (600) includes information of the video frames in which the similarity between the video frames is reflected. Therefore, the spatiotemporal feature block processing module (600) can generate a rough multi-planar image of the video frame of time t by synthesizing the information of the video frames of times t-T1 to t+T2, taking into account the similarity between the video frame of time t and the video frames of times t-T1 to t-1 and t+1 to t+T2. In other words, the spatiotemporal feature block processing module (600) can generate a rough multi-planar image by synthesizing the information of the video frames, taking into account the temporal and spatial similarity between the video frames.

[0085] In addition, the spatiotemporal feature block processing module (600) can utilize the patch-wise similarity between a patch of a video frame at time t and corresponding patches of video frames at times t-T1 to t-1 and t+1 to t+T2 to generate a coarse multi-planar image. The patch-wise similarity can provide a higher level of spatial similarity than the pixel-wise similarity. Therefore, the spatiotemporal feature block processing module (600) can generate a coarse multi-planar image by synthesizing information of video frames by considering a high level of spatial similarity.

[0086] Figure 7 is a drawing explaining the transmittance according to one embodiment.

[0087] The fine MPI generation module can calculate the transmittance of a multi-planar image. The multi-planar image can include volume density information along with color information. The fine MPI generation module can calculate the transmittance of the multi-planar image using the volume density information of the multi-planar image.

[0088] In one embodiment, the fine MPI generation module can calculate the transmittance of a multi-planar image using Equation 1.

[0089]

[0090] In mathematical expression 1, α j is a value calculated based on the volume density, which represents the probability that an object exists in the j-th image plane, and T i represents the transmittance of the i-th image plane.

[0091] Transmittance can be expressed as a value ranging from 0 to 1. The transmittance of an image plane can decrease depending on the degree to which information is blocked by preceding image planes. A pixel with a transmittance of 1 can indicate a pixel whose information is not blocked by preceding image planes, while a pixel with a transmittance of 0 can indicate a pixel whose information is completely blocked by preceding image planes.

[0092] Referring to Fig. 7, since there is no object on the first vector (701), the pixels on the first vector (701) have a transmittance of 1, and the transmittance does not decrease even as the depth increases. However, since there is an object on the second vector (702), the information transmitted from the front image planes to the back image planes is blocked. Accordingly, the transmittance of the pixels on the second vector (702) decreases as the depth increases.

[0093] FIG. 8 is a diagram for explaining the operations of a fine MPI generation module according to one embodiment.

[0094] The fine MPI generation module can generate a fine multi-planar image from a coarse multi-planar image. The fine MPI generation module can generate image planes of the fine multi-planar image by performing inpainting processing on each image plane of the coarse multi-planar image.

[0095] FIG. 8 illustrates an embodiment of performing an inpainting process on an image plane. In one embodiment, the image plane may include an image (800) that includes an occluded region (801) and a non-occluded region (802).

[0096] The closed region (801) may be an area having a transmittance less than or equal to a predetermined value due to the preceding image planes. In other words, the closed region (801) may be an opaque region. The non-closed region (802) may be an area having a transmittance higher than the predetermined value. In other words, the non-closed region (802) may be a transparent region. In other words, the non-closed region (802) may be a background region excluding the closed region (801). For example, the predetermined value may be 0 or a small number close to 0.

[0097] The fine MPI generation module can perform inpainting processing by restoring at least a portion of a closed area (801) using only pixels of an unclosed area (802).

[0098] The unclosed area (802) is an area where pixel information is known. Therefore, by using only the pixels of the unclosed area (802) for inpainting processing, the closed area (801) can be restored more accurately.

[0099] FIG. 9 is a drawing illustrating a method for generating a mask (903) based on transmittance according to one embodiment.

[0100] The fine MPI generation module can generate a mask (901) based on the transmittance of the image. In one embodiment, the image can include areas with a transmittance of 0, i.e., closed areas, and areas with a transmittance of 1, i.e., unclosed areas. In one embodiment, the mask (901) can have the transmittance of the image as a pixel value. In FIG. 9, the black areas of the mask (901) can represent areas with a transmittance of 0 in the image, and the white areas can represent areas with a transmittance of 1 in the image.

[0101] The fine MPI generation module can perform min pooling on the mask (902). The fine MPI generation module can perform min pooling without changing the size of the mask (902) by using zero padding and stride adjustment. The size of the window (904) for min pooling can be an adjustable parameter depending on the degree to which the closed region is to be expanded.

[0102] The fine MPI generation module can obtain a mask (903) with an expanded black area from the result of minimum pooling. The fine MPI generation module can expand a closed area of ​​an image by masking the image with the mask (903).

[0103] FIG. 10 is a drawing illustrating a method for generating a mask (1005) based on transmittance according to one embodiment.

[0104] The fine MPI generation module can generate a mask (1001) based on the transmittance of the image. In one embodiment, the image can include an area with a transmittance of 0, i.e., a closed area, and an area with a transmittance of 1, i.e., an unclosed area. In one embodiment, the mask (1001) can have the transmittance of the image as a pixel value. In FIG. 10, the black area of ​​the mask (1001) can represent an area with a transmittance of 0 in the image, and the white area can represent an area with a transmittance of 1 in the image.

[0105] The fine MPI generation module can invert the mask (1001). In the inverted mask (1002), a black area can represent an area where the image's transparency is 1, and a white area can represent an area where the image's transparency is 0.

[0106] The fine-grained MPI generation module can perform max pooling on the mask (1003). The fine-grained MPI generation module can perform max pooling without changing the size of the mask (1003) by using zero padding and stride adjustment. The size of the window (1006) for max pooling can be an adjustable parameter depending on the degree to which the closed region is to be expanded.

[0107] The fine MPI generation module can obtain a mask (1004) with an expanded white area from the result of max pooling. The fine MPI generation module can invert the mask (1004).

[0108] In the inverted mask (1005), a white area may represent an area where the image's transparency is 1, and a black area may represent an area where the image's transparency is 0. The fine MPI generation module may expand a closed area of ​​the image by masking the image with the mask (1005).

[0109] FIG. 11 is a diagram illustrating a method for generating a learning image (1100b) according to one embodiment.

[0110] The fine-grained MPI generation module can perform inpainting processing using a deep learning-based inpainting model. In one embodiment, the fine-grained MPI generation module can be configured to train the inpainting model. In another embodiment, the fine-grained MPI generation module can be configured to use a pre-trained inpainting model.

[0111] In one embodiment, the fine MPI generation module can train an inpainting model using a first training image (1100a) and a second training image (1100b). In one embodiment, the first training image (1100a) can be obtained from a coarse multi-planar image generated by the coarse MPI generation module. In another embodiment, the first training image (1100a) can be any image having a closed region.

[0112] In one embodiment, the first learning image (1100a) may include a closed region (1101a) and a non-closed region (1102a). The closed region (1101a) may be a region in the first learning image (1100a) that has a transmittance less than or equal to a predetermined value. The non-closed region (1102a) may be a region in the first learning image (1100a) that has a transmittance greater than the predetermined value. For example, the predetermined value may be 0.

[0113] In one embodiment, the fine MPI generation module can generate a second training image (1100b) by expanding a closed region (1101a) based on the transmittance of the first training image (1100a). The fine MPI generation module can expand the closed region (1101a) by masking the first training image (1100a) using a mask (1100c). The method described with reference to FIGS. 9 and 10 can be used to generate the mask (1100c).

[0114] The second learning image (1100b) may include an occluded region (1101b), a non-occluded region (1102b), and an expanded occluded region (1103b). The occluded region (1101b) of the second learning image (1100b) may be identical to the occluded region (1101a) of the first learning image (1100a). The expanded occluded region (1103b) may be a region that was part of the non-occluded region (1102a) of the first learning image (1100a). The expanded occluded region (1103b) may be a region masked by a mask (1100c).

[0115] The fine MPI generation module can train an inpainting model to restore an extended closed region (1103b) of the second training image (1100b) using information from an unclosed region (1102b) of the second training image (1100b). At this time, the fine MPI generation module can train the inpainting model so as not to use information from the closed region (1101b) to restore the extended closed region (1103b).

[0116] By training the inpainting model to perform inpainting processing only using areas where it knows information about pixels, it can be trained to restore closed areas more accurately.

[0117] Since the inpainting model is trained to restore an extended closed region, the size of the region restored by the fine MPI generation module using the inpainting model can be related to the size of the extended closed region used to train the inpainting model.

[0118] FIG. 12 is a flowchart of a method for generating multi-planar images from a video according to one embodiment.

[0119] In step S1201, the electronic device can generate a rough multi-planar image of one of the video frames based on similarities between the video frames.

[0120] An electronic device can generate a coarse multi-planar image of a video frame by synthesizing information from the video frames based on similarities between one video frame and other video frames.

[0121] Electronic devices can generate multi-planar images with richer and more accurate information by utilizing spatiotemporal information of video frames.

[0122] In step S1202, the electronic device can generate a fine multi-planar image of one video frame by performing inpainting processing on the coarse multi-planar image.

[0123] The electronic device can perform inpainting processing using information of unoccluded areas of a coarse multi-planar image to generate a fine multi-planar image in which occluded areas are at least partially restored.

[0124] An electronic device can generate a multi-planar image with more accurate information by performing an inpainting process using information from an unenclosed area with known information.

[0125] Figure 13 is a diagram for comparing the performance of the proposed method and the conventional method according to one embodiment.

[0126] Figure 13 illustrates a first image (1301) that is a multi-planar image generated using a conventional method, represented on the time axis (t) and the position axis (x), and a second image (1302) that is a multi-planar image generated using the method proposed in the present disclosure, represented on the time axis (t) and the position axis (x). The same video frames were used to generate the first image (1301) and the second image (1302).

[0127] Conventional methods generate multi-planar images from a single 2D image, while the method proposed in this disclosure generates multi-planar images by synthesizing video frames. To compare the conventional method with the proposed method, the conventional method was used to generate multi-planar images using each video frame independently.

[0128] Looking at the part pointed by the arrow in the first image (1301), the positions of the pixels along the time axis are inconsistent. This is because the conventional method generates a multi-planar image using video frames independently, and thus fails to generate a multi-planar image by considering the similarity between the video frames. On the other hand, looking at the part pointed by the arrow in the second image (1302), the positions of the pixels along the time axis are consistent. Since the method proposed in the present disclosure utilizes spatiotemporal information of the video, it can generate a more accurate multi-planar image than the conventional method.

[0129] Figure 14 is a drawing for explaining the performance of the proposed method according to one embodiment.

[0130] FIG. 14 illustrates a video frame (1410), a depth map (1420) of the video frame (1410), and an image (1430) generated using a multi-planar image of the video frame (1410) generated according to the method proposed in the present disclosure. The image (1430) was generated to have a different view than the video frame (1410).

[0131] A video frame (1410) includes a first square (1411), a second square (1412) having a closed area covered by the first square, and a third square (1413) having a closed area covered by the second square. According to the depth map (1420), the first square (1411), the second square (1412), and the third square (1413) have different depth values.

[0132] Referring to image (1430), it can be confirmed that the closed areas of the second square (1412) and the third square (1413) have been restored. Also, referring to image (1430), it can be confirmed that the closed area of ​​the second square (1412) has been restored based on the non-closed area, i.e., the area not covered by the first square (1411). Also, referring to image (1430), it can be confirmed that the closed area of ​​the third square (1413) has been restored based on the non-closed area, i.e., the area not covered by the second square (1412).

[0133] FIG. 15A is a block diagram of an electronic device (1500) according to one embodiment.

[0134] The electronic device (1500) may include a processor (1510) and a memory (1520). The processor (1510) and the memory (1520) may communicate with each other via a bus.

[0135] The processor (1510) can control the overall operations of the electronic device (1500). For example, the processor (1510) can control the overall operations of the electronic device (1500) for generating a multi-plane image from a video by executing one or more instructions of a program stored in the memory (1520). For example, the processor (1510) can control the overall operations of the electronic device (1500) for generating a multi-plane image from a video and for generating a composite image from the multi-plane image by executing one or more instructions of a program stored in the memory (1520). There can be one or more processors (1510).

[0136] The processor (1510) may be configured as at least one of, but is not limited to, a CPU, a microprocessor, a GPU, an ASIC (Application Specific Integrated Circuits), a DSP (Digital Signal Processors), a DSPD (Digital Signal Processing Devices), a PLD (Programmable Logic Devices), an FPGA (Field Programmable Gate Arrays), an application processor, an NPU, or an artificial intelligence processor designed with a hardware structure specialized for processing artificial intelligence models.

[0137] The memory (1520) may store instructions, data structures, and program codes that can be read by the processor (1510). Operations performed by the processor (1510) may be implemented by executing instructions or codes of a program stored in the memory (1520).

[0138] The memory (1520) may include a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), and may include a non-volatile memory including at least one of a ROM (Read-Only Memory), an EEPROM (Electrically Erasable Programmable Read-Only Memory), a PROM (Programmable Read-Only Memory), a magnetic memory, a magnetic disk, or an optical disk, and a volatile memory such as a RAM (Random Access Memory) or an SRAM (Static Random Access Memory).

[0139] The memory (1520) may store one or more instructions and / or programs that cause the electronic device (1500) to generate a multi-planar image from a video. Additionally, the memory (1520) may store one or more instructions and / or programs that cause the electronic device (1500) to generate a composite image from the multi-planar images.

[0140] FIG. 15b is a block diagram of an electronic device (1500) according to one embodiment.

[0141] In explaining Fig. 15b, any content overlapping with that explained in Fig. 15a is omitted for brevity.

[0142] In one embodiment, the memory (1520) may store instructions and / or programs for implementing the functions of the coarse MPI generation module (1521) and the fine MPI generation module (1522). Additionally, the memory (1520) may further store an inpainting model.

[0143] The coarse MPI generation module (1521) and the fine MPI generation module (1522) can be executed by the processor (1510). Since the descriptions related to the operations of each of the aforementioned modules have already been described in the descriptions of the previous drawings, a repeated description will be omitted.

[0144] Meanwhile, embodiments of the present disclosure may also be implemented in the form of a recording medium containing computer-executable instructions, such as program modules, executed by a computer. Computer-readable media may be any available media that can be accessed by a computer, and include both volatile and nonvolatile media, removable and non-removable media. Furthermore, computer-readable media may include computer storage media and communication media. Computer storage media include both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, or other data. Communication media may typically include computer-readable instructions, data structures, or other data in a modulated data signal, such as program modules.

[0145] Additionally, a computer-readable storage medium may be provided in the form of a non-transitory storage medium. Here, the term "non-transitory storage medium" simply means a tangible device that does not contain signals (e.g., electromagnetic waves). This term does not distinguish between cases where data is permanently stored in the storage medium and cases where data is temporarily stored. For example, a "non-transitory storage medium" may include a buffer in which data is temporarily stored.

[0146] According to one embodiment, the method according to various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) through an application store or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable app) may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0147] The above description of the present disclosure is provided for illustrative purposes only, and those skilled in the art will readily appreciate that modifications to other specific forms can be made without altering the technical spirit or essential features of the present disclosure. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. For example, components described as being single may be implemented in a distributed manner, and similarly, components described as being distributed may be implemented in a combined manner.

[0148] The scope of the present disclosure is indicated by the claims described below rather than the detailed description above, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present disclosure.

Claims

1. A method for generating a multi-plane image (MPI) from a video, A step (S1201) of generating a coarse multi-planar image (coarse MPI) of one of the video frames based on the similarity between the video frames; and A method comprising a step (S1202) of generating a fine multi-planar image (fine MPI) of one of the video frames by performing an inpainting process on the rough multi-planar image.

2. In paragraph 1, The step of generating the above rough multi-planar image is: A step of generating spatiotemporal feature blocks by embedding pixel values and depth values of each of the above video frames; A step of calculating similarity between one of the above video frames and other video frames; A step of weighting the spatiotemporal feature blocks based on the similarity calculated above; and A method comprising the step of generating the rough multi-planar image by de-embedding the weighted spatiotemporal feature blocks.

3. In paragraph 2, The step of calculating the similarity between one of the above video frames and other video frames is: Comprising a step of calculating patch-wise similarity between a patch including a plurality of pixels of one of the video frames and corresponding patches of the other video frames, The step of weighting the above spatiotemporal feature blocks is: A method comprising a step of weighting corresponding patches of the spatiotemporal feature blocks based on the similarity of the patch units.

4. In any one of paragraphs 1 to 3, The step of generating the above rough multi-planar image is: A method comprising the step of generating a rough multi-planar image of one of the video frames based on similarity between pixel values and depth values of one of the video frames and pixel values and depth values of other video frames.

5. In any one of paragraphs 1 to 4, The step of generating the above rough multi-planar image is a method based on an attention mechanism.

6. In any one of paragraphs 1 to 5, The step of generating the above micro multi-planar image is: A method comprising the step of generating the fine multi-planar image by performing inpainting processing on a closed area of the rough multi-planar image using only pixels of an unclosed area of the rough multi-planar image.

7. In any one of paragraphs 1 to 6, The step of generating the above micro multi-planar image is: A method comprising the step of performing inpainting processing using a deep learning-based inpainting model.

8. In paragraph 7, The above deep learning-based inpainting model is A method for performing inpainting processing on a second learning image, the second learning image being generated by masking the first learning image using a mask generated based on the transmittance of the first learning image, the second learning image including an occluded region, an expanded occluded region extended from the occluded region, and a non-occluded region, excluding pixels of the occluded region, using pixels of the non-occluded region.

9. In paragraph 8, The closed region of the second learning image is a region having a transmittance less than or equal to a predetermined value in the first learning image, The unoccluded area of the second learning image is an area in the first learning image that is not masked by the mask, A method wherein the extended closed region of the above learning image has a transmittance greater than the predetermined value in the first learning image and is an area masked by the mask.

10. In any one of paragraphs 1 to 9, The above video frames are consecutive frames, method.

11. In an electronic device (1500) for generating a multi-plane image from a video, A memory (1520) storing one or more instructions; and comprising one or more processors (1510) for executing one or more instructions stored in the memory; The one or more processors (1510) execute the one or more instructions, Based on the similarity between the video frames, a rough multi-planar image of one of the video frames is generated, An electronic device configured to generate a fine multi-planar image of any one of the video frames by performing an inpainting process on the coarse multi-planar image.

12. In paragraph 11, The one or more processors (1510) execute the one or more instructions, By embedding the pixel values and depth values of each of the above video frames, spatiotemporal feature blocks are generated, Calculating the similarity between one of the above video frames and other video frames, Based on the similarity calculated above, the spatiotemporal feature blocks are weighted and added, An electronic device configured to generate the rough multi-planar image by de-embedding the weighted spatiotemporal feature blocks.

13. In paragraph 12, The one or more processors (1510) execute the one or more instructions, Compute the patch-by-patch similarity between a patch containing a plurality of pixels of one of the above video frames and corresponding patches of the other video frames, An electronic device configured to weight corresponding patches of the spatiotemporal feature blocks based on the similarity of the patch units.

14. In any one of paragraphs 11 to 13, The one or more processors (1510) execute the one or more instructions, An electronic device configured to generate a rough multi-planar image of one of the video frames based on similarity between pixel values and depth values of one of the video frames and pixel values and depth values of other video frames.

15. A computer-readable recording medium recording a program for executing the method of any one of clauses 1 to 10 on a computer.

Images