Method and apparatus for packing and unpacking of residual image using 3D warping

Abstract

An immersive video encoding method is provided, comprising the steps of: acquiring a plurality of input viewpoint images captured by an image capturing device at a plurality of input positions; generating a central viewpoint image by synthesizing the plurality of input viewpoint images based on a predetermined central viewpoint; generating a plurality of residual images showing the difference between the central viewpoint image and the plurality of input viewpoint images; acquiring a plurality of warped residual images by 3D warping the plurality of residual images based on the central viewpoint; and generating a packing image by synthesizing the plurality of warped residual images based on the central viewpoint.

Description

Technology Field

[0001] The present disclosure relates to a method, apparatus, and program for immersive video encoding / decoding. Specifically, the present disclosure provides a method, apparatus, and program for reducing data regarding a plurality of residual images generated according to the encoding of immersive video. Background Technology

[0002] The advent of digital broadcasting transformed the traditional broadcasting method, where viewers watched signals unilaterally transmitted by broadcasting stations, into a form that allows users to selectively watch only the content they want at their desired time. Furthermore, advancements in broadband transmission technology have overcome bandwidth limitations, making it possible to provide immersive broadcasting services capable of delivering high-definition, realistic media of 4K or higher resolution (e.g., UHDTV, 3DTV, etc.) to viewers.

[0003] 360-degree virtual reality (VR) media is a medium that allows viewers to select and view their desired view by providing omnidirectional video. Recently, efforts to provide 360-degree VR media services through integration with broadcasting networks have been accelerating. Currently, commonly applied 360-degree VR media can simultaneously provide full 4K or FHD-quality VR media. While this offers the advantage of providing viewers with their desired view or Region of Interest (RoI) without delay, there is a problem where the image quality of the playing view degrades depending on the viewer's actual movements. Additionally, another form of 360-degree VR service that streams the corresponding area based on the viewer's movements or the view selected by the viewer can provide relatively high-quality views, but there is a problem where the Motion to Photon (MTP) delay increases as the video is played in response to user movements.

[0004] Technologies related to 360-degree VR media include technologies that provide signaling for recognizing panoramic video broadcasting services and indicating related video characteristics, technologies that transmit thumbnail images first and transmit data for specific requested areas based on the thumbnails at the receiver, or technologies that classify panoramic videos into tiles of a certain area and transmit only the data for the tiles corresponding to the area selected by the user. While these existing technologies propose signaling, tiling transmission, and synchronization methods for providing panoramic broadcasting services, there is a problem in that it is difficult to provide 360-degree VR broadcasting services based on user movement or user selection in a broadcasting network environment for high-definition 360-degree VR videos. The problem to be solved

[0005] According to the present disclosure, an efficient method for packing and unpacking residual images using three-dimensional warping is provided. means of solving the problem

[0006] According to the present disclosure, an immersive video encoding method is provided, comprising the steps of: acquiring a plurality of input viewpoint images captured by an image capturing device at a plurality of input positions; generating a central viewpoint image by synthesizing the plurality of input viewpoint images based on a predetermined central viewpoint; generating a plurality of residual images showing the difference between the central viewpoint image and the plurality of input viewpoint images; acquiring a plurality of warped residual images by warping the plurality of residual images in three dimensions based on the central viewpoint; and generating a packing image by synthesizing the plurality of warped residual images based on the central viewpoint.

[0007] According to one embodiment, the immersive video encoding method may further include the step of converting the plurality of residual images by removing overlapping regions between the plurality of residual images.

[0008] According to one embodiment, the plurality of input viewpoint images include a texture image and a depth image, and the step of generating a central viewpoint image by synthesizing the plurality of input viewpoint images based on a predetermined central viewpoint may generate the central viewpoint image according to the texture image and the depth image of the plurality of input viewpoint images.

[0009] According to the present disclosure, an immersive video encoding device comprising one or more processors and one or more memory devices is provided, wherein the one or more processors perform one or more instructions of a program for residual image packing using dimensional warping stored in the one or more memory devices, and the one or more memory devices store a program for immersive video encoding, and the one or more instructions include: a step of acquiring a plurality of input viewpoint images captured by an image capturing device at a plurality of input positions; a step of generating a central viewpoint image by synthesizing the plurality of input viewpoint images based on a predetermined central viewpoint; a step of generating a plurality of residual images showing the difference between the central viewpoint image and the plurality of input viewpoint images; a step of acquiring a plurality of warped residual images by 3D warping the plurality of residual images based on the central viewpoint; and a step of generating a packed image by synthesizing the plurality of warped residual images based on the central viewpoint.

[0010] According to the present disclosure, an immersive video decoding method is provided, comprising the steps of: acquiring a packing image; generating a plurality of warping residual images from the packing image; generating a plurality of residual images by three-dimensionally warping the plurality of warping residual images according to a plurality of input time points; and generating a plurality of input time point images according to the plurality of residual images and a central time point image.

[0011] According to the present disclosure, an immersive video decoding device is provided, comprising one or more processors and one or more memory devices, wherein the one or more processors perform one or more instructions of a program for residual image packing using dimensional warping stored in the one or more memory devices, and the one or more memory devices store a program for immersive video decoding, and the one or more instructions include the steps of acquiring a packed image, generating a plurality of warped residual images synthesized from the packed image based on a central viewpoint, generating a plurality of residual images by 3D warping the plurality of warped residual images according to a plurality of input viewpoints, and generating a plurality of input viewpoint images according to the plurality of residual images and the central viewpoint image.

[0012] According to the present disclosure, a program is provided that includes instructions for performing each step of the immersive video encoding method or decoding method. Effects of the invention

[0013] According to the present disclosure, the encoding efficiency of immersive video can be improved through packing and unpacking of residual images using three-dimensional warping. Brief explanation of the drawing

[0014] FIG. 1 illustrates a conceptual diagram of an immersive video capable of supporting motion parallax according to one embodiment of the present disclosure. FIGS. 2a and 2b are conceptual diagrams of an immersive video generation method according to one embodiment of the present disclosure. FIG. 3 illustrates a method for generating RVD in the present disclosure. FIG. 4 is a conceptual diagram for providing an immersive video service according to one embodiment of the present disclosure. FIGS. 5a and 5b illustrate a method for synthesizing immersive video at a terminal using RVD in the present disclosure. FIG. 6 illustrates a concept for efficient transmission and reproduction of immersive media having 3DoF+ or 6DoF degrees of freedom. Figure 7 illustrates the concept of a global center. FIG. 8 is a block diagram illustrating a packing method for immersive media according to one embodiment of the present disclosure. FIG. 9 is an example of a packing image of a plurality of input images according to an embodiment of the present disclosure. FIG. 10 illustrates a method for packing a plurality of input images according to one embodiment of the present disclosure. FIG. 11 illustrates a central viewpoint image having a FoV that can include the entire image area for an omnidirectional image and an array-type perspective image. FIG. 12 illustrates an example in which a residual image is obtained based on the difference with the central viewpoint image for each of the 4x4 array perspective input images. FIG. 13 illustrates an example of a method for efficiently packing residual images of input time point images. FIG. 14 illustrates an example of a method for efficiently packing residual images of input time point images. FIG. 15 shows an example of a system structure to which the above-described three-dimensional warping-based packing method is applied. FIG. 16 is a flowchart of an immersive video encoding method according to one embodiment of the present disclosure. FIG. 17 is a flowchart of an immersive video decoding method according to one embodiment of the present disclosure. Specific details for implementing the invention

[0015] The present disclosure is subject to various modifications and may have various embodiments, and specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present disclosure to specific embodiments, and it should be understood that it includes all modifications, equivalents, and substitutions that fall within the spirit and scope of the present disclosure. Similar reference numerals in the drawings refer to the same or similar functions across various aspects. The shapes and sizes of elements in the drawings may be exaggerated for clearer explanation. The detailed description of exemplary embodiments described below refers to the accompanying drawings, which illustrate specific embodiments as examples. These embodiments are described in sufficient detail to enable those skilled in the art to practice the embodiments. It should be understood that various embodiments are different but need not be mutually exclusive. For example, specific shapes, structures, and characteristics described herein may be implemented in other embodiments without departing from the spirit and scope of the present disclosure in relation to one embodiment. It should also be understood that the location or arrangement of individual components within each disclosed embodiment may be changed without departing from the spirit and scope of the embodiment. Accordingly, the following detailed description is not intended to be taken in a limiting sense, and the scope of exemplary embodiments is limited only by the appended claims, together with all equivalents to those claimed therein, provided they are properly described.

[0016] In this disclosure, terms such as first, second, etc. may be used to describe various components, but said components should not be limited by said terms. Such terms are used solely for the purpose of distinguishing one component from another. For example, without departing from the scope of this disclosure, the first component may be named the second component, and similarly, the second component may be named the first component. The term "and / or" includes a combination of a plurality of related described items or any of a plurality of related described items.

[0017] Where it is stated that any component of the present disclosure is “connected” or “connected” to another component, it should be understood that it may be directly connected or connected to that other component, or that there may be other components in between. On the other hand, where it is stated that a component is “directly connected” or “directly connected” to another component, it should be understood that there are no other components in between.

[0018] The components shown in the embodiments of the present disclosure are depicted independently to represent different characteristic functions and do not imply that each component consists of separate hardware or a single software unit. That is, each component is listed and included as a separate component for convenience of explanation; however, at least two of the components may be combined to form a single component, or a single component may be divided into multiple components to perform a function, and such integrated and separated embodiments of each component are included within the scope of the rights of the present disclosure as long as they do not depart from the essence of the present disclosure.

[0019] The terms used in this disclosure are used merely to describe specific embodiments and are not intended to limit this disclosure. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this disclosure, terms such as “comprising” or “having” are intended to specify the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof. That is, the description in this disclosure that a specific configuration “comprising” does not exclude configurations other than that configuration, but means that additional configurations may be included within the scope of the practice or technical concept of this disclosure.

[0020] Some components of the present disclosure may not be essential components performing an essential function in the present disclosure, but may be optional components merely for enhancing performance. The present disclosure may be implemented by including only the components essential to embody the essence of the present disclosure, excluding components used merely for enhancing performance, and a structure including only the essential components, excluding optional components used merely for enhancing performance, is also included within the scope of the rights of the present disclosure.

[0021] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In describing the embodiments of this specification, if it is determined that a detailed description of related known configurations or functions may obscure the gist of this specification, such detailed description is omitted, and the same reference numerals are used for identical components in the drawings, and redundant descriptions of identical components are omitted.

[0022] Virtual reality services are evolving to maximize immersion and a sense of presence by generating omnidirectional 360-degree video in the form of live footage or CG and playing it on personal VR devices such as HMDs (Head Mounted Displays) and smartphones.

[0023] Currently, in order to play natural and highly immersive 360-degree video through an HMD, six degrees of freedom (DOF) must be reproduced. For example, video corresponding to the viewer's movement in six directions, such as (1) left-right movement, (2) up-down rotation, (3) up-down movement, and (4) left-right rotation, must be played through the HMD screen. Current omnidirectional videos, which play live-action footage acquired by a camera, have three degrees of freedom and provide only video for movements primarily focused on (2) up-down rotation and (4) left-right rotation. Furthermore, omnidirectional videos with three degrees of freedom cannot provide video for the viewer's left-right and up-down movements.

[0024] In the present disclosure, the immersive video described refers to spherical video or 360-degree video. Immersive video uses an omnidirectional camera or a collection of cameras to provide a view of all directions simultaneously. When immersive video is played on a standard flat display, the viewer can control the viewpoint as if it were a panorama.

[0025] Immersive video supports 3 degrees of freedom and can further reproduce motion parallax in the viewer's seated environment (3DoF+). Additionally, immersive video can provide motion parallax of 4 degrees of freedom or more. For example, 6-degree-of-freedom immersive video can provide motion parallax that changes according to the viewer's free movement (6DoF).

[0026] In the present disclosure, an immersive video generation format and a method and apparatus for playback are provided to effectively provide large-capacity immersive video supporting 6 degrees of freedom while maintaining compatibility with basic immersive video services such as UHD and 3DoF. Additionally, an immersive video generation format and a method and apparatus for playback are provided that enable the service of large-capacity immersive video supporting 6 degrees of freedom with only minimal additional video and metadata transmission.

[0027] FIG. 1 illustrates a conceptual diagram of an immersive video capable of supporting motion parallax according to one embodiment of the present disclosure.

[0028] In FIG. 1, O1 to 04 represent video regions within any scene, Vk represents the video acquired at the camera center position, Xk represents the viewpoint position (camera position), and Dk represents the depth information at the camera center position. As with the immersive video in FIG. 1, in order to support 6 degrees of freedom according to the viewer's movement, video must be generated in various directions at multiple positions.

[0029] Immersive video can consist of omnidirectional video and related spatial information (depth information, camera information). The immersive video is transmitted to the terminal side through video compression and packet multiplexing processes.

[0030] An immersive media system acquires, generates, transmits, and reproduces large volumes of immersive video composed of multiple views. Therefore, an immersive media system must be able to effectively store and compress large volumes of video data and must be compatible with existing immersive video (3DoF).

[0031] FIGS. 2a and 2b are conceptual diagrams of an immersive video generation method according to one embodiment of the present disclosure.

[0032] The primary viewpoint video (Xk) of the present disclosure is intended to maintain compatibility with the basic service. The primary viewpoint video (Vk) corresponds to a 2D video in Windowed-6DoF based on planar video, and corresponds to a 3DoF 360VR video in 3DoF+ and 6DoF based on omnidirectional video. Information indicating the primary viewpoint among multiple viewpoints may be included in the metadata. According to the embodiment, multiple primary viewpoints may exist. If multiple primary viewpoints exist, the process of selecting a primary viewpoint video to be used as a reference image for each non-primary viewpoint image among the multiple primary viewpoints may be further included.

[0033] A difference image obtained by differentiating a non-primary viewpoint video (Vk±n) and a primary viewpoint video (Vk) can be encoded / decoded. For example, in the present disclosure, RVD (Residual Video plus Depth) (RVk-2, RVk-1, …) is an additional video for supporting motion parallax with minimal additional data. RVD includes at least one of a texture image and a depth map generated by excluding the region overlapping with the primary viewpoint video from the immersive video acquired at all viewpoint positions (Xk-2, Xk-1 …). The primary viewpoint video (Vk) is a reference image and consists of a region acquired from the user's center position (red solid line in FIG. 2). Information indicating the primary viewpoint video among multiple viewpoint videos may be included in the metadata. For example, index information pointing to any one of the multiple viewpoints may be included in the metadata.

[0034] The non-primary viewpoint video Vk-1 includes an area not included in the primary viewpoint video Vk. RVk-1 can be obtained by subtracting the primary viewpoint video Vk from the non-primary viewpoint video Vk-1. As a result, the difference image RVk-1 may include an area within the non-primary viewpoint video Vk-1 that was not obtained by the primary video (Vk) (blue dashed arrow in FIG. 2).

[0035] The non-primary viewpoint video Vk-2 includes a region not included in the primary viewpoint video Vk. RVk-2 can be obtained by subtracting the primary viewpoint video Vk from the non-primary viewpoint video Vk-2. Alternatively, RVk-2 can be obtained by subtracting the difference image RVk-1 of the neighboring viewpoint video from the result of subtracting the primary viewpoint video Vk from the non-primary viewpoint video Vk-2. As a result, the difference image RVk-2 may include a region (green dashed arrow in Fig. 2) within the non-primary viewpoint video Vk-2 that is not obtained by the primary video Vk and the neighboring difference image RVk-1. Expressed differently from the perspective of image signal processing, RVk-1 is an additional video composed of a region (blue dashed arrow in Fig. 2) from which the video region overlapping with the primary viewpoint video (Vk) within the immersive video Vk-1 has been removed. RVk-2 is an additional video composed of the area (green dashed arrow in Fig. 2) from which the video area overlapping with the base video (Vk) and RVk-1 within the immersive video Vk-2 has been removed.

[0036] As a result, if the video area obtained at each point in time overlaps with the area obtained from other videos, only one video is generated instead of creating duplicate videos, thereby reducing the amount of data for transmission and compression.

[0037] The difference image of a non-default viewpoint video where the difference between the default viewpoint video and the viewpoint index is 1 can be obtained by differencing the default viewpoint video from the non-default viewpoint video. The difference image of a non-default viewpoint video where the difference between the default viewpoint video and the viewpoint index is greater than 1 can be obtained by differencing the difference image of the default viewpoint video and / or neighboring non-default viewpoint videos from the non-default viewpoint video. That is, the use of the difference image of neighboring non-default viewpoint videos may be determined differently for each non-default viewpoint video. Neighboring non-default viewpoint videos refer to non-default viewpoint videos where the difference between the viewpoint index and the non-default viewpoint video is 1 or one of predefined values.

[0038] Alternatively, information indicating another non-primary viewpoint for a non-primary viewpoint video may be included in the metadata. The difference image of the non-primary viewpoint video may be obtained by differentiating the primary viewpoint video and / or the non-primary viewpoint video indicated by the information from the non-primary viewpoint video.

[0039] Alternatively, a difference image for a non-primary viewpoint video can be obtained using multiple reference images. For example, a difference image of a non-primary viewpoint video (Vk-n) can be obtained based on a difference image of a primary viewpoint video (Vk) and non-primary viewpoint videos Vk-1 to Vk-n+1. Information indicating the number of reference images may be included in the metadata.

[0040] The RVD generated in the present disclosure is packed into a single video (one frame) for storage or transmission and converted into a Packed Video Plus Depth (PVD) format. The viewpoint position for each RVD within the packed video, the position and size within each RVD's area, etc., are described as metadata. This metadata is intended to describe information regarding all video areas within the RVD when packed into a PVD, and is included in a compressed stream or a transmission stream to be transmitted to a terminal.

[0041] The metadata in this disclosure may consist of a View number (the viewpoint where the corresponding video area is acquired with the highest quality (viewpoint number)), Shared views (the viewpoint where the corresponding video area is acquired commonly at other viewpoints (viewpoint number)), priority (used to determine transmission status, bit rate allocation, etc., when data capacity is limited, and priority can be determined according to the order of viewpoint numbers in Shared_views), location and size within RVD, location and size within PV (Packed video), and camera parameters per viewpoint. Additionally, priority may be used to assign lower importance to in-painted video areas (occluded areas that are not acquired by the actual camera but are interpolated by signal processing) compared to occluded areas, as their quality is relatively degraded.

[0042] FIG. 3 illustrates a method for generating RVD in the present disclosure.

[0043] In the present disclosure, the 3D warping step 3D warps all videos to be referenced to a target viewpoint location to eliminate redundancy. Here, 3D warping may follow a 3D warping algorithm that first warps the depth map of the reference viewpoint using camera parameter values ​​and then warps the texture region based on it. 3D warping undergoes the following two steps.

[0044] Step 1) Backprojection from the reference image coordinate system to the 3D spatial coordinate system

[0045] Step 2) Project from the 3D spatial coordinate system to the target virtual viewpoint image coordinate system

[0046] At this time, the following equation (1) is applied.

[0047] [Equation (1)]

[0048]

[0049] Equation (1) is a formula for back-projecting from a reference image Vk into 3D space. In this case, the projection matrix is ​​determined based on the intrinsic parameter K and extrinsic parameters R and T of each reference camera obtained through the camera calibration process. At this time, the following Equation (2) is applied.

[0050] .

[0051]

[0052] Here

[0053] Equation (2) is an equation for projecting to the target viewpoint position Vk-1 in the back-projected 3D space. In order to 3D warp the reference image, which is 2D array data, an additional depth value corresponding to the Z value is required.

[0054] In one embodiment of FIG. 3, the reference image refers to a video 3D-warped to match another viewpoint. For example, a reference viewpoint video Vk (300) can be 3D-warped to another reference viewpoint, Vk-1 (302), and used as a reference image for the non-primary viewpoint video Vk-1 (302). When the primary viewpoint video Vk (300) is 3D-warped to match the reference viewpoint Vk-1 (302), objects O3 and O2 of the primary viewpoint video Vk (300) move to match the reference viewpoint Vk-1 (302) according to their respective depth values. Additionally, areas that were not visible in the primary viewpoint video Vk (300) are determined as holes. In this disclosure, a hole represents a video area that is not subject to 3D-warping.

[0055] And the base view video 3D warped to the reference view (V ~ k Similarity between the two videos is determined by comparing texture values ​​and depth values ​​between (300) and the reference point video (Vk-1) (302). When the difference resulting from the comparison of texture values ​​and depth values ​​is within a threshold range, the two videos being compared are determined to have redundancy in three-dimensional space. Therefore, the redundant parts of the two videos are removed (e.g., the front of O3 and O2 of the reference point video (Vk-1) (302)). The non-redundant parts (the side of O4 and O2 of the reference point video (Vk-1) (302)) are included in the RVD video (RVk-1) (312). All pixels on the object can be compared, or samples can be compared.

[0056] When multiple reference images are used, the overlapping regions between each reference point are repeatedly removed based on the central point-of-view image, that is, the reference point-of-view image. For example, in generating RVk-2 (314), not only the image of the reference point-of-view image Vk (300) 3D warped to match point-of-view Vk-2 (304), but also the image of RVk-1 (312) 3D warped to match point-of-view Vk-2 (304) may be used as reference images. Accordingly, the residual image RVk-2 (314) of the non-reference point-of-view image Vk-2 (304) may be one in which the overlapping data with the reference point-of-view image Vk (300) and the overlapping parts with the residual image RVk-1 (312) have been removed from Vk-2 (304).

[0057] In order to determine which video regions visible in the viewpoint image Vk-2 (304) are also visible in other viewpoint images Vk and Vk-1 (300, 302), 3D warping is performed from the viewpoint images Vk and Vk-1 (300, 302) and the RV image RVk-1 (312) to the position of Vk-2, respectively. Then, regions determined to overlap with other viewpoint images in the viewpoint image Vk-2 (304) are removed from the reference viewpoint image Vk-2 (304). As a result, in FIG. 3, only parts of the sides of O1 and O4 remain in RVk-2 (314).

[0058] FIG. 4 is a conceptual diagram (400) for providing an immersive video service according to one embodiment of the present disclosure.

[0059] The base video (Vk and DK) can be encoded through a 2D video encoder (HEVC, VVC, etc.). The encoded base video is then transmitted to a terminal and decoded. Based on the decoded base video, basic services such as UHD and 3DoF 360VR are provided. On the other hand, the RVD is packed into a PVD after undergoing an additional information formatting step (see Fig. 2). The related metadata is then encoded and packetized, respectively, and transmitted to the terminal. At the immersive video terminal, the metadata is interpreted according to the additional information un-formatting step, and the RVD is separated from the PVD based on the metadata. In the immersive video synthesis step, a viewpoint image corresponding to the user viewport is dynamically synthesized by referencing the base viewpoint video according to the viewer's viewing position. The synthesized viewpoint image is then reproduced through the display.

[0060] In the supplementary information unformatting step for the immersive video terminal, information necessary for synthesizing the viewpoint video is interpreted from the metadata based on the viewing position. Then, using the interpreted information, the corresponding RVD is filtered and separated from the packed video. In the immersive video synthesis step, the viewpoint video corresponding to the viewport based on the viewer's viewing position is dynamically synthesized by referencing the base video. Finally, the synthesized viewpoint video is reproduced through the display.

[0061] FIGS. 5A and 5B illustrate a method for synthesizing immersive video at a terminal using an RVD in the present disclosure. In the metadata interpretation step of Additional Information Unformatting, a view number corresponding to the viewer's viewing position, shared views (indicated by an orange ellipse in FIGS. 5A and 5B), all video area location information in the PVD, and camera parameters are extracted from the metadata transmitted from the server. In the immersive video synthesis step, an RVD is generated by filtering the video area from the PVD using the extracted metadata, and a viewport image is synthesized by performing 3D warping with reference to the reference video (Vk). In the synthesis of the viewport image, the non-reference viewpoint image can be restored based on the RVD and the corresponding reference image. The reference image of the non-reference viewpoint image may include at least one of the reference viewpoint video (Vk) or the neighbor viewpoint residual image of the RVD image. For example, the reference image of a non-reference time point video (Vk-2) may include a reference time point video (Vk) and a neighboring time point residual image (RVk-1). Alternatively, information for determining the reference image of an RVD may be extracted from metadata. The information may be about at least one of the number of reference images or the disparity of the reference images.

[0062] Figure 5a illustrates the process of compositing an image at the reference point Vk-2 using the reference video (Vk) and RVD. First, using the reference video and its corresponding depth map, the object in the reference video (Vk) is 3D warped to the target position (Xk-2). Object O3 is mapped based on its position determined by depth. Object O2 is also warped, but it is not visible as it is not mapped within the viewport in Xk-2. Once the 3D warping step from the reference video is performed, the next step is to generate textures for areas that are not visible in the reference video but are visible in Xk-2. To this end, the most suitable reference point is selected for backward warping by referring to the 3D geometry to bring in texture information remaining as hole areas. In Figure 5, the candidates for the reference points are Vk-1 and Vk-2. Since the partial region (texture) of the image visible at each reference point is in the RVD and its definition is in the metadata (Vk-2 in the example of FIG. 5), Xk-2 is completed through a backward mapping process using this. As shown in the embodiment of FIG. 5, objects O4 and O1 are mapped to the target viewport image Xk-2.

[0063] The second embodiment of FIG. 5b shows an example of synthesis where the target viewpoint image Xk is located between the reference video (Vk) and the reference viewpoint Vk-1. O2 is mapped from the reference video (Vk), and the side portion of O2 is mapped from the RVD and becomes visible in the viewport image. On the other hand, O1 is not mapped within the viewport area and therefore becomes invisible. Additionally, since the target viewpoint image is located between Vk and Vk-1, the visible portions within the video area, such as O3 and O4, change.

[0064] In the present disclosure, the 360-degree video and playback method can provide a complete and natural stereoscopic image through a virtual reality device by playing a corresponding image when the viewer moves up, down, left, and right, as well as in addition to conventional rotational movement.

[0065] The present disclosure relates to a technique for efficiently packing and unpacking residual images based on virtual central viewpoint synthesis.

[0066] FIG. 6 illustrates a concept for efficient transmission and reproduction of immersive media having 3DoF+ or 6DoF degrees of freedom.

[0067] First, a full 360-degree image centered on the Global Center (hereinafter referred to as the central viewpoint) is synthesized into an ERP format from Partial Equi-rectangular Projection (ERP) images acquired from multiple locations with different camera centers. Figure 7 below illustrates the concept of the Global Center. The image generated around the Global Center includes all parallax images. Efficiency can be increased by reducing redundant data during the process of synthesizing partial images acquired from multiple cameras into a single Global Center.

[0068] Then, residual images are obtained by calculating information not included in the central viewpoint from each acquired input image (source view). Since the obtained residual images do not contain information about the entire image area but only about partial occlusion areas, transmission efficiency can be improved by efficiently packing and sending them.

[0069] Below, FIG. 8 is a block diagram illustrating a packing method for immersive media according to an embodiment of the present disclosure. A central viewpoint is synthesized from texture images and depth images of a plurality of input images (source views). A residual image is calculated from the synthesized central viewpoint. Then, by efficiently partitioning and packing it and then encoding it, the transmission efficiency can be improved.

[0070] FIG. 9 below is an example of a packing image of a plurality of input images according to an embodiment of the present disclosure. As shown in the figure, only the parts containing information from the residual image can be extracted and transformed, such as by rotation and translation, to be combined into a single image, thereby reducing the size of the transmitted data.

[0071] Packing as described above allows for a reduction in transmission bit count by consolidating multiple residual images into a single image, but additional metadata is required depending on the packing method. Unpacking of the packed image is possible only if there is mapping information indicating whether each pixel of the packed image is mapped to a specific image coordinate location within a specific input image. Therefore, metadata indicating the mapping information must also be transmitted and received. This corresponds to the "Additional packing infos" section in the block diagram of Fig. 8.

[0072] The size of the above mapping information may increase as the number of input images increases, the resolution of individual input images increases, or the amount of residual information within the input images increases. Consequently, if the size of the mapping information increases excessively, the total size of the data related to the images may increase. Therefore, a decrease in transmission and reception efficiency may occur due to the increase in the size of the metadata related to the image mapping information.

[0073] FIG. 10 illustrates a method for packing a plurality of input images according to an embodiment of the present disclosure. There are overlapping regions between residual images generated by removing redundancy from the input images. Therefore, by removing redundancy between residual images, the size of the images can be further compressed. According to FIG. 10, a method for packing input images is described that removes redundancy between a central viewpoint and each input viewpoint, as well as redundancy between residual images. However, according to the packing method, since detection of redundant regions based on 3D warping is required for all input image pairs, the amount of computation may be large.

[0074] Accordingly, the present disclosure provides a packing and unpacking method that can prevent a decrease in transmission and reception efficiency caused by an increase in the size of metadata while eliminating redundancy between residual images. The packing and unpacking method for residual images according to the present disclosure can efficiently eliminate redundancy between residual images based on 3D warping to a central viewpoint while minimizing metadata for mapping.

[0075] FIG. 11 illustrates a central viewpoint image having a FoV that can include the entire image area for an omnidirectional image and an array-type perspective image.

[0076] According to FIG. 11, regardless of the projection format of the image, a central viewpoint having a FoV that includes the entire image area can be synthesized based on input images obtained from multiple cameras.

[0077] FIG. 12 illustrates an example in which a residual image is obtained based on the difference with the central viewpoint image for each of the 4x4 array perspective input images.

[0078] According to FIG. 12, each of the input images (1200, 1202, 1204, 1206) is combined to form a central viewpoint image (1210) that includes the entire image area. Then, the central viewpoint image (1210) is compared with each of the input viewpoint images (1200, 1202, 1204, 1206) to obtain the residual images (1220, 1222, 1224, 1226) of each input viewpoint.

[0079] FIG. 13 illustrates an embodiment of a method for efficiently packing residual images of input viewpoint images. When residual images are packed intensively within a single image using operations such as rotation and translation, additional packing information regarding mapping relationships for unpacking must be additionally transmitted. Therefore, to avoid using metadata for the additional packing information or to reduce its size, a packing method is proposed that warps residual images three-dimensionally to a central viewpoint.

[0080] Object A is closest to the viewpoint. When there is an object B (or a background with a texture) obscured by object A, a central viewpoint image (1310) based on the central viewpoint is composited using viewpoint image 1 (1300) and viewpoint image 2 (1302). Then, the central viewpoint image (1310) is warped to the image position of the input viewpoint. Then, a residual image 1 (1320) and a residual image 2 (1322) are determined for the warped central viewpoint image (1310), viewpoint image 1 (1300), and viewpoint image 2 (1302).

[0081] The residual images of each input time point image can be combined by 3D warping them at the same time point, for example, to the central viewpoint. For example, residual image 1 (1320) of input time point image 1 (1300) and residual image 2 (1322) of input time point image 2 (1302) can be 3D warped to the central viewpoint. Then, the warped residual image 1 (1320) and the warped residual image 2 (1322) can be packed / combined. The packed / combined image can be 3D warped to the input time point position and unpacked to obtain the residual image (1330) at the input time point position.

[0082] Since object A overlaps in the central viewpoint image (1310) and viewpoint image 1 (1302), object A does not exist in residual image 1 (1320). Similarly, since object B overlaps in the central viewpoint image (1310) and viewpoint image 2 (1302), object A does not exist in residual image 2 (1322). Accordingly, the image formed by warping the residual images (1320, 1322) to the central viewpoint position and compositing them has no occlusion caused by object A. That is, an image can be obtained in which the information of object B is reconstructed at the central viewpoint position. According to the packing method above, intuitive packing and unpacking are possible, and additional mapping parameters for packing and unpacking alone are not required.

[0083] FIG. 14 illustrates an example of a method for efficiently packing residual images of input viewpoint images. In the packing process, overlapping data between residual images can be removed. Unlike FIG. 13, FIG. 14 further includes viewpoint image 3 (1404). Residual image 3 (1422) and residual image 2 (1420) for viewpoint image 3 (1404) have overlapping areas for object B. Accordingly, when residual image 2 (1420) and residual image 3 (1422) warped to the reference viewpoint are combined into one, the overlapping areas of residual image 2 (1420) and residual image 3 (1422) can be removed.

[0084] When removing duplicate regions by warping between residual images, there is a problem in that the amount of computation increases depending on the number of input images. However, according to the packing method described in FIG. 14, duplicate regions of objects are automatically removed without additional computation while warping all multiple residual images to the central viewpoint position. Therefore, according to the packing method described in FIG. 14, the amount of computation required to remove duplicate regions between residual images can be reduced.

[0085] FIG. 15 illustrates an embodiment of a system structure to which the above 3D warping-based packing method is applied. A central viewpoint image including the entire image area based on the global center is synthesized from multiple input viewpoint images, and a residual image is obtained by pruning and filtering the input viewpoint images using this. Then, 3D warping-based packing is performed, without using additional packing information. After decoding and transmitting / receiving using a codec such as HEVC, 3D warping-based unpacking is performed again, and a VoI viewpoint image is synthesized and rendered.

[0086] The present disclosure proposes a packing and unpacking method for efficient transmission and reproduction of immersive media. According to the configuration of the present disclosure, if a technique is used to pack residual images by 3D warping them to a central viewpoint, additional mapping parameters for packing and unpacking are not required, thereby reducing the size of the transmitted data and allowing redundant information between residual images to be automatically removed without additional computation.

[0087] FIG. 16 is a flowchart of an immersive video encoding method according to one embodiment of the present disclosure.

[0088] In step S1602, a plurality of input point-of-time images are obtained, which are captured by an image capturing device at a plurality of input positions.

[0089] In step S1604, a central viewpoint image is generated by synthesizing multiple input viewpoint images based on a predetermined central viewpoint.

[0090] In step S1606, multiple residual images are generated that represent the difference between the central viewpoint image and multiple input viewpoint images.

[0091] In step S1608, a plurality of warped residual images are obtained by three-dimensionally warping a plurality of residual images based on the central viewpoint.

[0092] In step S1610, a plurality of warping residual images are combined based on the central viewpoint to generate a packing image.

[0093] FIG. 17 is a flowchart of an immersive video decoding method according to one embodiment of the present disclosure.

[0094] In step S1702, a packing image is obtained.

[0095] In step S1704, a plurality of warping residual images are generated from the packing image.

[0096] In step S1706, multiple warping residual images are three-dimensionally warped according to multiple input times, thereby generating multiple residual images.

[0097] In step S1708, multiple input viewpoint images are generated based on multiple residual images and a central viewpoint image.

[0098] In the embodiments described above, methods are described based on flowcharts as a series of steps or units; however, the present disclosure is not limited to the order of steps, and some steps may occur in a different order or simultaneously with other steps as described above. Furthermore, those skilled in the art will understand that the steps shown in the flowcharts are not exclusive, other steps may be included, or one or more steps of the flowcharts may be omitted without affecting the scope of the present disclosure.

[0099] The embodiments described above include examples of various aspects. While it is not possible to describe all possible combinations for representing various aspects, those skilled in the art will recognize that other combinations are possible. Accordingly, the present disclosure shall be deemed to include all other substitutions, modifications, and changes that fall within the scope of the following claims.

[0100] The embodiments according to the present disclosure described above may be implemented in the form of program instructions that can be executed through various computer components and recorded on a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, etc., either alone or in combination. The program instructions recorded on the computer-readable recording medium may be those specifically designed and configured for the present disclosure, or they may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The hardware devices may be configured to operate as one or more software modules to perform processing according to the present disclosure, and vice versa.

[0101] Although the present disclosure has been described above with specific details such as specific components, limited embodiments, and drawings, this is provided only to aid in a more comprehensive understanding of the present disclosure and is not limited to the above embodiments, and a person skilled in the art to which the present disclosure belongs can make various modifications and variations from this description.

[0102] Accordingly, the scope of the present disclosure is not limited to the embodiments described above, and all things equivalent or equivalently modified to the claims set forth below, as well as the claims set forth below, shall be considered to be within the scope of the scope of the present disclosure.

Claims

Claim 1 An immersive video encoding method comprising: a step of acquiring a plurality of input point-time images captured by an image capturing device at a plurality of input positions; a step of generating a central point-time image by synthesizing the plurality of input point-time images based on a predetermined central point-time image; a step of generating a plurality of residual images showing the difference between the central point-time image and the plurality of input point-time images; a step of acquiring a plurality of warped residual images by 3D warping the plurality of residual images based on the central point-time image; and a step of generating a packing image by synthesizing the plurality of warped residual images based on the central point-time image. Claim 2 An immersive video encoding method according to claim 1, wherein the plurality of warping residual images are characterized in that the overlapping regions between the plurality of residual images are removed. Claim 3 An immersive video encoding method according to claim 1, wherein the plurality of residual images are generated by performing pruning and filtering on an input viewpoint image using the central viewpoint image. Claim 4 An immersive video encoding method according to claim 1, wherein the plurality of input time point images include a texture image and a depth image. Claim 5 In claim 4, the central viewpoint image is generated according to the texture image and depth image of the plurality of input viewpoint images, an immersive video encoding method. Claim 6 An immersive video decoding method comprising: a step of acquiring a packing image and a central viewpoint image; a step of generating a plurality of warping residual images from the packing image; a step of generating a plurality of residual images by three-dimensionally warping the plurality of warping residual images according to a plurality of input viewpoints; and a step of generating a plurality of input viewpoint images according to the plurality of residual images and the central viewpoint image. Claim 7 An immersive video decoding method according to claim 6, wherein the plurality of warping residual images are characterized in that the overlapping regions between the plurality of residual images are removed. Claim 8 An immersive video decoding method according to claim 6, wherein the plurality of residual images are generated as a result of performing pruning and filtering on an input time point image using the central time point image. Claim 9 An immersive video decoding method according to claim 6, characterized in that the plurality of input time point images include a texture image and a depth image. Claim 10 An immersive video decoding method according to claim 9, wherein the plurality of input viewpoint images are generated by generating a plurality of residual images by three-dimensional warping the plurality of warping residual images, and then summing the plurality of residual images and the central viewpoint image.

Images