Texture-Based Immersive Video Coding

Abstract

A method, apparatus, system, and product for texture-based immersive video coding are disclosed. The example apparatus includes a correspondence labeling unit for (i) identifying a first intrinsic pixel and a first corresponding pixel among a plurality of pixels of a first view and (ii) identifying a second intrinsic pixel and a second corresponding pixel among a plurality of pixels of a second view, a correspondence patch packing unit for (i) comparing neighboring pixels in the first view and (ii) identifying a first patch of intrinsic pixels and a second patch of corresponding pixels based on the comparison of the neighboring pixels and the correspondence, where the second patch of corresponding pixels is tagged with a correspondence list that identifies a correspondence patch in the second view, and an atlas generator for generating at least one atlas for inclusion in encoded video data (the encoded video data does not include a depth map).

Description

[Technical Field]

[0001] This patent claims the benefit of U.S. Provisional Patent Application No. 63 / 009,356, filed April 13, 2020. U.S. Provisional Patent Application No. 63 / 009,356 is incorporated herein by reference in its entirety. Priority to U.S. Patent Application No. 63 / 009,356 is claimed herein.

[0002] FIELD OF THE DISCLOSURE This disclosure relates generally to video coding, and more particularly to texture-based immersive video coding. [Background technology]

[0003] In compression / decompression (codec) systems, compression efficiency and video quality are important performance criteria. For example, visual quality is an important attribute of the user experience in many video applications, and compression efficiency affects the amount of memory storage needed to store a video file and / or the amount of bandwidth needed to transmit and / or stream the video content. A video encoder compresses video information so that more information can be transmitted over a given bandwidth or stored in a given memory space or the like. The compressed signal or data is then decoded by a decoder, which decodes or decompresses the signal or data for display to a user. In many implementations, higher visual quality is desired with greater compression.

[0004] Standards for immersive video coding are currently being developed, including Moving Picture Experts Group (MPEG) Immersive Video Coding, which aims to establish and improve compression efficiency and reconstruction quality in the context of immersive video and point cloud coding. [Brief explanation of the drawings]

[0005] [Figure 1]1 is a block diagram of an exemplary MPEG immersive video (MIV) encoder.

[0006] [Figure 2] 10 illustrates an example patch formation from selected input views.

[0007] [Figure 3] FIG. 2 is a block diagram of an exemplary MIV decoder.

[0008] [Figure 4] 10 illustrates an exemplary reconstruction of pruned views from atlas patches.

[0009] [Figure 5] FIG. 2 is a block diagram of an example texture-based MIV encoder in accordance with the teachings of this disclosure.

[0010] [Figure 6] 6 illustrates an exemplary atlas generation from input views in the texture-based MIV encoder of FIG. 5.

[0011] [Figure 7] 1 is a block diagram of an example texture-based MIV decoder in accordance with the teachings of this disclosure.

[0012] [Figure 8] 8 is a block diagram of an exemplary rendering process implemented by the texture-based MIV decoder of FIG. 7.

[0013] [Figure 9] 8 illustrates an exemplary atlas view reconstruction in the texture-based MIV decoder of FIG. 7.

[0014] [Figure 10] 1 illustrates an exemplary video-based volumetric video coding (V3C) sample streaming with MIV extension.

[0015] [Figure 11] 6 is a flowchart representing machine-readable instructions that may be executed to implement the texture-based MIV encoder of FIG. 5.

[0016] [Figure 12] 6 is a flowchart representing machine-readable instructions that may be executed to implement an example correspondence labeler included in the texture-based MIV encoder of FIG. 5.

[0017] [Figure 13] 6 is a flowchart representing machine-readable instructions that may be executed to implement an example correspondence pruning unit included in the texture-based MIV encoder of FIG. 5.

[0018] [Figure 14] 6 is a flowchart representing machine-readable instructions that may be executed to implement an example correspondence patch packer included in the texture-based MIV encoder of FIG. 5.

[0019] [Figure 15] 8 is a flowchart representing machine-readable instructions that may be executed to implement the texture-based MIV decoder of FIG. 7.

[0020] [Figure 16] FIG. 1 is an illustration of an example system in accordance with at least some implementations of the present disclosure.

[0021] [Figure 17] 1 illustrates an example device, all arranged in accordance with at least some implementations of the present disclosure.

[0022] [Figure 18] FIG. 15 is a block diagram of an exemplary processing platform configured to execute the instructions of FIGS. 11, 12, 13, and 14 to implement the texture-based MIV encoder of FIG. 5.

[0023] [Figure 19] 16 is a block diagram of an exemplary processing platform configured to execute the instructions of FIG. 15 to implement the texture-based MIV decoder of FIG. 7.

[0024] [Figure 20] FIG. 10 is a block diagram of an exemplary software distribution platform for distributing software (e.g., software corresponding to the exemplary computer-readable instructions of FIGS. 11, 12, 13, 14, and 15) to client devices, such as consumers (e.g., license, sell, and / or use), retailers (e.g., sell, resell, license, and / or sublicense), and / or original equipment manufacturers (OEMs) (e.g., for inclusion in products distributed to retailers and / or direct purchasing customers).

[0025] The figures are not to scale. Generally, the same reference numbers are used throughout the drawings and accompanying written description to refer to the same or similar parts.

[0026] Unless otherwise specified, descriptors such as “first,” “second,” and “third” are used herein without any denotation or otherwise indicating any priority, physical order, placement, and / or sequence in a list, but are used merely as labels and / or arbitrary names to distinguish elements to facilitate understanding of the disclosed examples. In some instances, the descriptor “first” may be used to refer to an element in the detailed description, while in the claims, the same element may be referred to by different descriptors such as “second” or “third.” In such cases, it should be understood that such descriptors may be used merely to clearly identify those elements (e.g., those elements would otherwise share the same name). As used herein, “approximately” and “about” refer to dimensions that may not be exact due to manufacturing tolerances and / or other practical imperfections. As used herein, “substantially real-time” refers to occurring nearly instantaneously, although it should be recognized that there are practical delays in computation time, transmission, and the like. Therefore, unless otherwise specified, "substantially real time" refers to real time plus or minus 1 second. DETAILED DESCRIPTION OF THE INVENTION

[0027] One or more embodiments or implementations are described herein with reference to the accompanying drawings. While particular configurations and arrangements are described, it should be understood that this is for illustrative purposes only. Those skilled in the art will recognize that other configurations and arrangements can be utilized without departing from the spirit and scope of the description. Those skilled in the art will appreciate that the techniques and / or arrangements described herein can also be utilized in a variety of other systems and applications other than those described herein.

[0028] Although the following description describes various implementations that may be manifested in architectures such as, for example, system-on-chip (SoC) architectures, implementation of the techniques and / or arrangements described herein is not limited to a particular architecture and / or computing system and may be implemented by any architecture and / or computing system of similar purpose. For example, various architectures utilizing, for example, multiple integrated circuit (IC) chips and / or packages, and / or various computing devices and / or consumer electronics (CE) devices such as set-top boxes and smartphones may implement the techniques and / or arrangements described herein. Furthermore, although the following description may set forth many specific details, such as the logic implementation, types and interrelationships of system components, and logical partitioning / integration options, claimed subject matter may be practiced without such specific details. In other instances, some material, such as, for example, control structures and full software instruction sequences, may not be shown in detail so as not to obscure the material disclosed herein.

[0029] References herein to "one implementation," "implementation," "exemplary implementation," etc. indicate that the described implementation may include a particular feature, structure, or characteristic, but that all embodiments may not necessarily include such a particular feature, structure, or characteristic. Moreover, such phrases do not necessarily refer to the same implementation. Furthermore, when a particular feature, structure, or characteristic is described in connection with an embodiment, it is indicated that it is within the knowledge of one skilled in the art to provide such feature, structure, or characteristic in connection with other implementations, whether or not explicitly described herein.

[0030] The terms "substantially," "near," "approximately," "nearby," and "about" generally refer to being within + / - 10% of a target value. For example, unless otherwise specified in the explicit context of use, the terms "substantially equal," "approximately equal," and "approximately equal" mean that there is only incidental variation between those so described. In the art, such variation is typically no more than ±10% of a predetermined target value.

[0031] Advances in digital media technology are enabling new media formats to deliver compelling immersive experiences. The Moving Picture Experts Group (MPEG) is one standards organization developing standards to support immersive media access and delivery. For example, the Coded Representation of Immersive Media (MPEG-1) is a set of immersive media industry standards for immersive media formats such as panoramic 360° video, volumetric point clouds, and immersive video.

[0032] MPEG is developing an immersive video coding standard called MPEG Immersive Video (MIV). See the MIV standard (J. Boyce, R. Dore, V. Kumar Malamal Vadakital (Eds.), "Working Draft 4 of Immersive Video," ISO / IEC JTC1 / SC29 / WG11 MPEG / N19001, January 2020, Brussels, Belgium) and the Test Model for Immersive Video (TMIV) (B. Salahieh, B. Kroon, J. Jong, M. Domanski (Eds.), "Test Model 4 for Immersive Video," ISO / IEC JTC1 / SC29 / WG11 MPEG / N19002, January 2020, Brussels, Belgium), which are referred to herein as the "Immersive Video Coding Standard." The MIV standard specifies a bitstream format and decoding process for immersive video. The Immersive Video Test Model describes the reference encoding and rendering processes, however, these processes are not normative for the MIV standard.

[0033] The MIV draft standard uses the High Efficiency Video Coding (HEVC) video codec to code texture and depth video for each of multiple input views at a specific position and orientation. The MIV standard does not specify a reference renderer, but rather specifies the necessary metadata and decoded streams. The intended output of the reference renderer is a texture perspective viewport, selected based on the viewer position and orientation and generated using the output of an immersive media decoder. The MIV standard allows the viewer to dynamically move with six degrees of freedom (6DoF) and adjust position (x, y, z) and orientation (yaw, pitch, roll) within limited ranges (e.g., as supported by a head-mounted display, such as a two-dimensional (2D) monitor with positional input).

[0034] In the MIV standard, depth information (also known as geometry) may be allocated as part of the bitstream for immersive video coding, which may require additional bandwidth if streamed along with texture content. Additionally, coding operations implemented in MIV test models, such as pruning, clustering, and packing, utilize only depth and do not consider texture information, so important immersive cues (e.g., specularity, transparency, varying lighting conditions for captured views from different cameras) are lost in the extracted patch (i.e., rectangular region from the view). Furthermore, estimating a depth map can be a costly operation if not directly captured / provided, and maintaining depth consistency across all cameras for natural content can be difficult due to the local nature of depth estimation algorithms. Therefore, some services choose not to use depth and are not currently supported by MIV. Currently, texture content from various capture cameras can be coded and transmitted separately using a standard video codec such as High Efficiency Video Coding (HEVC), or it can be predictively coded using a multiview extension of a video codec such as MV-HEVC. Next, view synthesis techniques are invoked to render the desired viewport based on the decoded texture content and viewport information (i.e., the target viewing position and orientation). Such view synthesis may require a depth estimation (or another type of disparity information) preprocessing stage on the decoded texture to properly perform the projection. However, such approaches do not fully exploit angular redundancy across different views because simultaneous encoding of different views introduces redundancy in the coding. In MV-HEVC, inter-view prediction is similar to traditional inter-frame temporal prediction, but only horizontal shifts in the block motion model are supported, which leads to inefficient bandwidth utilization compared to reprojection-based approaches that exploit inter-view correlations used in MIV.

[0035] Disclosed herein are methods, apparatuses, and systems for texture-based immersive video coding. Exemplary encoding and metadata signaling techniques disclosed herein have separately signaled depth options and occupancy maps, support correspondence coding operations, and enable excluding depth from the MIV bitstream. In some disclosed exemplary encoding techniques, only texture information per input view is utilized, and depth information per input view is not required. Camera parameters are also utilized, but may also be estimated / inferred from texture content. The disclosed exemplary encoding techniques include identifying correspondence patches in the texture content across available input views to remove redundant texture information. The disclosed exemplary encoding techniques transmit the texture content along with the identified correspondence patches in the coded bitstream. The disclosed exemplary techniques provide further bandwidth savings (since depth is not coded as part of the coded bitstream), thereby enabling more immersive cues to be provided and supporting a wider range of immersive applications.

[0036] An exemplary decoding technique disclosed herein receives an input bitstream including a coded texture video, a coded occupancy video, and metadata. The exemplary decoding technique disclosed infers depth information from the coded texture video. The exemplary decoding technique disclosed obtains correspondence information in the coded texture video. The exemplary decoding technique disclosed infers depth information in the video data using texture patches and corresponding texture patches from other views. The exemplary decoding technique disclosed further includes synthesizing a viewport based on the texture patches and the corresponding texture patches along with the metadata and a desired viewing position and orientation.

[0037] 1 is a block diagram of an exemplary immersive video (MIV) encoder 100. The exemplary MIV encoder 100 includes an exemplary view optimizer 102, an exemplary atlas builder 104, an exemplary depth / occupancy coding unit 120, an exemplary geometry scaling unit 122, an exemplary video encoder 132, and an exemplary video encoder 134. The exemplary MIV encoder 100 receives an exemplary source view 101. For example, the source view 101 includes texture data (e.g., a texture bitstream) and depth data (e.g., a depth bitstream) of a captured scene. As used herein, "texture" and "attributes" are used interchangeably and refer to the visible properties of a pixel, such as its color components (e.g., red, green, and blue (RGB) components). As used herein, "depth" and "geometry" are used interchangeably unless otherwise specified. This is because a pixel's geometry typically includes the pixel's depth, which refers to the pixel's distance from a reference plane. For example, the source view 101 may be a source view generated by a video capture unit, a virtual view generated by a computer, etc. In the illustrated example, the source view 101 is represented as a video sequence. The exemplary MIV encoder 100 further receives an exemplary source camera parameter list 103. For example, the source camera parameter list 103 may include a source camera position, a source camera angle, etc.

[0038] An exemplary view optimizer 102 selects views to encode. For example, the view optimizer 102 analyzes the source view 101 to select which views to encode. In some examples, the view optimizer 102 labels views of the source view 101 as base views or additional views. As used herein, a base view is an input view to be packed in the atlas as a single patch. As used herein, an additional view is an input view to be pruned and packed in one or more patches. For example, the view optimizer 102 may determine how many base views can be in the input view based on criteria such as pixel rate limitations, sample count per input view, etc.

[0039] The exemplary atlas builder 104 receives the base and / or additional views determined by the exemplary view optimizer 102. As shown in connection with FIG. 2 , the atlas builder 104 uses pruning and clustering to form patches from the selected views (e.g., the base and / or additional views), which are packed into one or more atlases, each of which includes an optional texture component and a required depth component. The exemplary atlas builder 104 includes an exemplary pruner 106, an exemplary mask aggregator 108, an exemplary patch packer 110, and an exemplary atlas generator 112. The exemplary pruner 106 prunes the additional views. For example, the pruner 106 projects depth and / or texture data of the base view onto the depth and / or texture data of each of the additional views and / or previously pruned views, and extracts non-redundant occlusion regions.

[0040] The exemplary mask aggregator 108 aggregates the pruning results generated by the exemplary pruner 106 to generate patches. For example, the mask aggregator 108 accumulates motion-accounting pruning results (e.g., pruning masks) over an intra-period (e.g., a predetermined set of frames). The exemplary patch packer 110 packs the patches into one or more exemplary atlases 116. For example, the patch packer 110 performs clustering (e.g., combining pixels in the pruning masks to form patches) and extracts and packs rectangular patches into the atlases. In some examples, the patch packer 110 updates the content per frame over the processed intra-period.

[0041] The patch packing unit 110 tags patches with patch identifiers (e.g., patch IDs). The patch IDs identify patch indexes. For example, the patch IDs may range from 0 to 1 less than the number of generated patches. The patch packing unit 110 may generate a block-to-patch map (e.g., a patch ID map). The block-to-patch map is a two-dimensional array (e.g., representing point positions / locations in an atlas) that indicates patch IDs associated with a given block of one or more pixels. The patch packing unit 110 provides example data 114 (including the block-to-patch map and patch ID data) to the depth / occupancy coding unit 120. The atlas generator 112 uses the patch data from the patch packing unit 110 to generate an atlas 116 having a depth / occupancy component 118 and an optional texture component. The atlas generator 112 provides the atlas 116 having the depth / occupancy component 118 to the depth / occupancy coding unit 120. The atlas generator 112 provides the atlas 116 with optional texture components as attribute (texture) video data 128 to an exemplary video encoder 134 .

[0042] The depth / occupancy coding unit 120 generates an occupancy map to indicate whether pixels of a patch are occupied (e.g., valid) or unoccupied (e.g., invalid). In some examples, the occupancy map is a binary map. In some such examples, the occupancy map has a value of 1 indicating that a pixel is occupied and a value of 0 indicating that a pixel is unoccupied. The depth / occupancy coding unit 120 generates the occupancy map based on the block-patch map and patch ID data included in the data 114 from the patch packing unit 110 and based on the data of the atlas having the depth / occupancy component 118 from the atlas generator 112. The depth / occupancy coding unit 120 includes the occupancy map, the block-patch map, and the patch ID data in an exemplary MIV data bitstream 130. In the example shown, the MIV data bitstream 130 includes patch parameters for mapping between views and atlases, a packing order for exposing overlapping patches required to later obtain a block-patch map, a video-based point cloud compression (V-PCC) parameter set, and an adaptation parameter set (including view parameters), and any other metadata from the video coding data.

[0043] The geometry scaling unit 122 obtains depth (geometry) data for the atlases from the depth / occupancy coding unit 120. The geometry scaling unit 122 performs quantization and scaling of the geometry video data for each atlas that includes depth data. The geometry scaling unit 122 provides the atlases with depth (geometry) components 124 as depth (geometry) video data 126 to the exemplary video encoder 132.

[0044] The exemplary video encoder 132 generates an encoded atlas from the depth (geometry) video data 126. The exemplary video encoder 134 generates an encoded atlas from the attribute (texture) video data 128. For example, the video encoder 132 receives the exemplary depth (geometry) video data 126 (including the atlas) and encodes the depth components into the exemplary HEVC bitstream 136 using an HEVC video encoder. Additionally, the video encoder 134 receives the exemplary attribute (texture) video data 128 (including the atlas) and encodes the texture components into the exemplary HEVC bitstream 138 using an HEVC video encoder. However, the exemplary video encoder 132 and the exemplary video encoder 134 may additionally or alternatively use an Advanced Video Coding (AVC) video encoder, etc. Exemplary HEVC bitstream 136 and exemplary HEVC bitstream 138 may include patch parameters for mapping between views and atlases, packing order for exposing overlapping patches necessary for later deriving block-patch maps, etc., related to depth (geometry) video data 126 and attribute (texture) video data 128, respectively.

[0045] 2 illustrates an exemplary patch formation from selected input views. The input views include an exemplary first view 202, an exemplary second view 204, and an exemplary third view 206. For example, the first view 202 is view 0, the second view 204 is view 1, and the third view 206 is view 2. For example, the views 202, 204, and 206 include view representations 208 (e.g., attribute (texture) maps, geometry (depth) maps, and entity (object) maps). In the illustrated example of FIG. 2, the view representations 208 of the views 202, 204, and 206 include a third person. That is, the views 202, 204, and 206 are captured from three different cameras at three different angles with respect to the third person.

[0046] The example pruner 106 ( FIG. 1 ) prunes the views 202, 204, and 206 to generate patches. For example, the first view 202 includes an example first patch 210 and an example second patch 212, the second view 204 includes an example third patch 214, and the third view 206 includes an example fourth patch 216 and an example fifth patch 218. In some examples, each patch corresponds to one respective entity (object). For example, the first patch 210 corresponds to the head of a first person, the second patch 212 corresponds to the head of a second person, the third patch 214 corresponds to the arm of the second person, the fourth patch 216 corresponds to the head of a third person, and the fifth patch 218 corresponds to the leg of the second person.

[0047] In some examples, the patch packing unit 110 (FIG. 1) tags patches with patch IDs. For example, the patch packing unit 110 tags the first patch 210 with a patch ID of 2, the second patch 212 with a patch ID of 5, the third patch 214 with a patch ID of 8, the fourth patch 216 with a patch ID of 3, and the fifth patch 218 with a patch ID of 7.

[0048] The example atlas generator 112 (FIG. 1) generates an example first atlas 220 and an example second atlas 222. The atlases 220, 222 include texture (attribute) maps and depth (geometry) maps. In the example shown, the first atlas 220 includes an example first patch 210, an example second patch 212, and an example third patch 214. In the example shown, the second atlas 222 includes an example fourth patch 216 and an example fifth patch 218.

[0049] 3 is a block diagram of an exemplary immersive video (MIV) decoder 300. The exemplary MIV decoder 300 includes an exemplary video decoder 308, an exemplary video decoder 312, an exemplary block-patch map decoder 318, an exemplary MIV decoder and parser 316, an exemplary geometry scaling unit 326, an exemplary cull unit 328, and an exemplary rendering unit 330. The exemplary rendering unit 330 includes an exemplary controller 332, an exemplary compositor 334, and an exemplary inpaint unit 336. In the illustrated example of FIG. 3, the MIV decoder 300 receives a coded video sequence (CVS) for each of the video substreams for the geometry component and, optionally, the texture attributes. In some examples, the MIV decoder 300 receives the exemplary MIV data bitstream 130, the exemplary HEVC bitstream 136, and the exemplary HEVC bitstream 138 from the exemplary MIV encoder 100 (FIG. 1).

[0050] In the illustrated example of FIG. 3 , the video decoder 308 receives an HEVC bitstream 138. The HEVC bitstream 138 includes patch parameters for attribute (texture) video data. Examples of such parameters include a map of patches between views and atlases, a packing order for exposing overlapping patches that are later used to obtain a block-patch map, etc. The video decoder 308 generates a series of decoded texture pictures 310. The video decoder 312 receives an HEVC bitstream 136. The HEVC bitstream 136 includes patch parameters for depth (geometry) video data. Examples of such parameters include a map of patches between views and atlases, a packing order for exposing overlapping patches that are later used to obtain a block-patch map, etc. The video decoder 308 generates a series of decoded depth pictures 314. The video decoder 308 and the video decoder 312 generate a series of decoded picture pairs: the decoded texture picture 310 and the decoded depth picture 314. In some examples, the decoded texture picture 310 and the decoded depth picture 314 may have the same or different resolutions. In the shown example, the exemplary decoded texture picture 310 and the exemplary decoded depth picture 314 represent an exemplary atlas. In some examples, the video decoder 308 and the video decoder 312 may be HEVC decoders. In the shown example of FIG. 3, the video decoder 308 provides the decoded texture picture 310 to the rendering unit 330. The video decoder 312 provides the decoded depth picture 314 to the geometry scaling unit 326 and to the block-patch map decoder 318.

[0051] The MIV decoder and parser 316 receives the MIV data bitstream 130. The exemplary MIV decoder and parser 316 parses the MIV data bitstream 130 and generates exemplary atlas data 320 and exemplary video-based point cloud compression (V-PCC) and viewpoint parameter sets 322. For example, the MIV decoder and parser 316 parses the encoded MIV data bitstream 130 for exemplary patch lists, camera parameter lists, etc. The MIV decoder and parser 316 provides the atlas data 320 and V-PCC and viewpoint parameter sets 322 to an exemplary block-to-patch map decoder 318, an exemplary cull unit 328, and an exemplary renderer 330.

[0052] The block-patch map decoder 318 receives the decoded depth picture 314 from the video decoder 312 and the atlas data 320 (including a patch list and a camera parameter list) and the V-PCC and viewpoint parameter set 322 from the exemplary MIV decoder and parser 316. The block-patch map decoder 318 decodes the block-patch map 324 to determine point positions / locations in the atlas of the decoded depth picture 314. In the illustrated example, the block-patch map 324 is a two-dimensional array indicating patch IDs (e.g., representing point positions / locations in the atlas) associated with a given block of one or more pixels in the decoded depth picture 314. The block-patch map decoder 318 provides the block-patch map 324 to an exemplary geometry scaling unit 326. In the illustrated example of FIG. 3, the geometry scaling unit 326 performs upscaling of the geometry video data for each atlas included in the decoded depth picture 314. The geometry scaling unit 326 provides upscaled geometry video data for each atlas included in the decoded depth picture 314 to an exemplary cull unit 328 and an exemplary renderer 330 .

[0053] The exemplary cull unit 328 receives the geometry-scaled block-to-patch map 324 from the geometry scaling unit 326 and also receives the atlas data 320 (including the patch list and camera parameter list) and the V-PCC and viewpoint parameter set 322 from the exemplary MIV decoder and parser 316. The cull unit 328 performs patch culling based on an exemplary viewing position and viewing orientation 340 (e.g., a target view). The cull unit 328 may receive the viewing position and viewing orientation 340 from a user. The cull unit 328 uses the block-to-patch map 324 to filter out blocks from the atlas data 320 that are not visible based on the target viewing position and viewing orientation 340.

[0054] The example renderer 330 generates an example viewport 338. For example, the renderer 330 accesses one or more of the decoded atlas from the decoded texture picture 310 and the decoded depth picture 314, the atlas data 320 (e.g., atlas parameter list, camera parameter list, atlas patch occupancy map), and the viewer position and orientation 342. That is, the example renderer 330 outputs a perspective viewport of a texture image selected based on the example viewer position and orientation 342. In the example disclosed herein, the viewer can dynamically move (e.g., adjust the viewer position and orientation 342) with six degrees of freedom (6DoF), adjusting their position (x, y, z) and orientation (yaw, pitch, roll) within limited ranges (e.g., supported by a head-mounted display, or a 2D monitor with positional input, or the like). The renderer 330 generates an exemplary viewport 338 based on both the required depth components (decoded depth pictures 314) and optional texture components (decoded texture pictures 310), whether they include patches, views, or atlases.

[0055] In the illustrated example of FIG. 3 , the rendering unit 330 includes a controller 332, a compositor 334, and an inpainter 336. The exemplary controller 332 accesses an atlas patch occupancy map from the atlas data 320, which indicates whether pixels of the atlas in the decoded depth picture 314 or the decoded texture picture 310 are occupied at the viewer position and orientation 342. The controller 332 reconstructs a pruned view from patches in the atlas of the decoded texture picture 310 and the decoded depth picture 314 using a block-to-patch map, which will be described in more detail below in connection with FIG. 4 . In the example disclosed herein, the pruned view is a partial view representation of the source (input) view on the encoder side (e.g., the MIV encoder 100 of FIG. 1 ) that is occupied by patches retained in the atlas of the decoded texture picture 310 and the decoded depth picture 314. The controller 332 reconstructs the pruned view that belongs to the viewer position and orientation 342. In some examples, the pruned view belonging to viewer position and orientation 342 may contain holes due to the presence of content in another pruned or global view. The compositor 334 performs view synthesis based on the atlas patch occupancy map and the reconstructed pruned view from the controller 332 and the viewer position and orientation 342. In the examples disclosed herein, the inpainter 336 fills any non-existent pixels in the reconstructed pruned view with matching values.

[0056] FIG. 4 illustrates an example reconstruction 400 of pruned views from atlas patches. As shown in FIG. 4, atlas 406 includes atlas 402 and atlas 404. In some examples, video decoder 308 (FIG. 3) and video decoder 312 (FIG. 3) decode atlases 402, 404. In the illustrated example of FIG. 4, atlases 402, 404 include texture maps and depth maps. Atlas 402 includes an example first patch 410, an example second patch 412, and an example third patch 414. Atlas 404 includes an example fourth patch 416 and an example fifth patch 418. In some examples, the patches include patch IDs determined by the example patch packing unit 110 (FIG. 1). For example, a first patch 410 includes a patch ID of 2, a second patch 412 includes a patch ID of 5, a third patch 414 includes a patch ID of 8, a fourth patch 416 includes a patch ID of 3, and a fifth patch 418 includes a patch ID of 7.

[0057] In some examples, the block-patch map decoder 318 ( FIG. 3 ) uses the block-patch map to match the first patch 410, the second patch 412, the third patch 414, the fourth patch 416, and the fifth patch 418 to the available views 420, 424, and 426 included in the view representation 408. The block-patch map decoder 318 matches the patches to the available views to at least partially reconstruct the available views 420, 424, and 426. For example, the block-patch map decoder 318 matches the first patch 410 and the second patch 412 to the first available view 420, the third patch 414 to the second available view 424, and the fourth patch 416 and the fifth patch 418 to the third available view 426.

[0058] In some examples, each patch corresponds to one respective entity (object) in the available views (e.g., first available view 420, second available view 424, and third available view 426). For example, in first available view 420, first patch 410 corresponds to the head of a first person, second patch 412 corresponds to the head of a second person, in second available view 424, third patch 414 corresponds to the arm of the second person, and in third available view 426, fourth patch 416 corresponds to the head of the third person and fifth patch 418 corresponds to the leg of the second person.

[0059] 5 is a block diagram of an exemplary texture-based immersive video (MIV) encoder 500 according to the teachings of this disclosure. In the illustrated example texture-based MIV encoder 500, only attributes (e.g., texture, entity, reflectance, etc.) are input per view, and per-view depth is not provided (or is optional). In some embodiments, camera parameters may be utilized (e.g., encoded). In some examples, such camera parameters may be estimated / inferred from texture content (e.g., at a decoder). In the illustrated example of FIG. 5, the exemplary texture-based MIV encoder 500 includes an exemplary view optimizer 502, an exemplary depth inferrer 504, an exemplary correspondence atlas builder 506, an exemplary occupancy packer 524, an exemplary video encoder 534, and an exemplary video encoder 536. The example correspondence atlas builder 506 includes an example correspondence labeler 508 , an example correspondence pruner 510 , an example mask aggregator 512 , an example correspondence patch packer 514 , and an example atlas generator 516 .

[0060] In the illustrated example, the exemplary texture-based MIV encoder 500 receives an exemplary source view 501. For example, the source view 501 includes texture data (e.g., a texture bitstream) and, in some cases, depth data (e.g., a depth bitstream) of a captured scene. For example, the source view 501 may be a source view generated by a video capture unit, a virtual view generated by a computer, etc. Additionally or alternatively, the source view 501 includes other attributes (e.g., entity, reflectance, etc.). In the illustrated example, the source view 501 is represented as a video sequence. The exemplary texture-based MIV encoder 500 further receives an exemplary source camera parameter list 503. For example, the source camera parameter list 503 may include where the source camera is located, the angle of the source camera, etc.

[0061] The exemplary view optimizer 502 uses camera parameters from the camera parameter list 503 to identify views to encode. For example, the view optimizer 502 analyzes the source view 501 and selects which views to encode. In some examples, the view optimizer 502 labels the views of the source view 501 as base views or additional views. As used herein, a base view is an input view that is to be packed in the atlas as a single patch. As used herein, an additional view is an input view that is to be pruned and packed in one or more patches. In the examples disclosed herein, the view optimizer labels the source view 501 with a view identification (ID).

[0062] The example depth inference unit 504 performs an optional depth inference process to obtain depth information from the source view 501. For example, the depth inference unit 504 estimates an actual depth map for the source view 501, computes a disparity representation of the source view 501 using techniques such as optical flow, uses structure from motion across views to discover a per-view point cloud representation, etc. In the example disclosed herein, the depth inference unit 504 determines whether there is any depth information included in the source view 501. If the depth inference unit 504 does not determine any depth information or if the depth inference unit 504 is omitted from the example implementation, the texture-based MIV encoder 500 proceeds using only the texture information included in the source view 501.

[0063] The correspondence atlas builder 506 receives the base and / or additional views determined by the exemplary view optimizer 102. The correspondence atlas builder 506 evaluates texture content / information in the base and / or additional views to identify unique texture information for each source view and extract the unique texture information as patches. Additionally or alternatively, the correspondence atlas builder 506 evaluates other attribute content / information (e.g., entities, reflectance, etc.) in the base and / or additional views to identify unique attribute information for each source view and extract the unique attribute information as patches. In some examples, the correspondence atlas builder 506 evaluates optional depth information (if available) from the depth inference unit 504. The correspondence atlas builder 506 forms patches from selected views (e.g., base and / or additional views) and determines the patches using labeling, pruning, and clusters. The correspondence atlas builder 506 includes a correspondence labeler 508 , a correspondence pruner 510 , a mask aggregator 512 , a correspondence patch packer 514 , and an atlas generator 516 .

[0064] The correspondence labeling unit 508 identifies corresponding pixels across all views (e.g., base and / or additional views) and labels them based on texture content (and / or other attribute content) and depth information, if available. Thus, the correspondence labeling unit 508 is an example of a means for identifying unique pixels and corresponding pixels. In the examples disclosed herein, if unprojection of a first pixel from a first view onto the three-dimensional world and subsequent reprojection of the first pixel onto a second view places the first pixel in the same location as the second pixel from the second view, the first pixel and the second pixel (the first pixel from the first view and the second pixel from the second view) are considered corresponding pixels. In the examples disclosed herein, the first view and the second view may be a base view, an additional view, and / or a combination of a base view and an additional view. In some examples, the correspondence labeling unit 508 uses multi-view feature extraction and pattern recognition techniques (e.g., traditional or AI-based) to identify and label matched pixels across all views after being trained on similar multi-view content.

[0065] In the illustrated example, the correspondence labeling unit 508 labels pixels as unique, or corresponding, thereby identifying corresponding pixels across all source views from the view optimization unit 502. The correspondence labeling unit 508 labels pixels as “unique” for those pixels from any view that do not have corresponding pixels in other available views. For example, pixels located in regions at the edge of the device camera or in occlusion regions that are visible only by a particular view are typically labeled as “unique.” The correspondence labeling unit 508 labels pixels as “corresponding” for those pixels from any view that have one or more corresponding pixels in other available views. For each group of two or more corresponding pixels, the correspondence labeling unit 508 determines whether the corresponding pixels in the group have similar texture content. For example, the correspondence labeling unit 508 compares the texture content between two corresponding pixels and compares the difference in texture content to a threshold. In some examples, the threshold may be a difference in color components (e.g., red, green, and blue (RGB) components). For example, the threshold may be a difference of 5 (or some other value) between any one of the color components. The correspondence labeling unit 508 determines that corresponding pixels have similar texture content when the difference in texture content is below the threshold. The correspondence labeling unit 508 labels the corresponding pixels as "similar texture" and stores the view IDs of the source views containing the corresponding pixels along with the coordinate (e.g., x, y) locations of the corresponding pixels in the correspondence list. For example, the correspondence list may include the entries {(4, 27, 33), (7, 450, 270)}. In this example, there are two corresponding similar-texture pixels (one located in the source view with view ID 4 at image coordinates (27, 33) and the other located in the source view with view ID 7 at image coordinates (450, 270)).

[0066] In some examples, the correspondence labeling unit 508 determines corresponding pixels having different texture content (but having corresponding projected locations and therefore classified as corresponding pixels) when the difference in texture content is above a threshold. In some examples, corresponding pixels may be determined to have different texture content based on different reflectance information, different illumination / lighting information, color mismatch due to different camera settings, etc. The correspondence labeling unit 508 labels the corresponding pixels as "different texture" and stores the view ID of the source view containing the corresponding pixels along with the coordinate (e.g., x, y) locations of the corresponding pixels in the correspondence list. In the examples disclosed herein, the correspondence labeling unit 508 includes additional labels "similar texture" and "different texture" for the corresponding pixels, allowing the correspondence pruning unit 510 to use such information to reduce redundancy and thereby output non-redundant information. In some examples, the correspondence labeler uses "similar texture" and "different texture" labels and / or other names / identifiers for additional labels for pixels to indicate differences between corresponding pixels in attributes other than texture, if available. For example, the additional labels may identify differences in transparency, reflectance, motion, etc. The correspondence labeler 508 outputs a correspondence labeling map and an indexed (e.g., corresponding_id) pixel-wise correspondence list for the view related to the correspondence case.

[0067] The correspondence pruning unit 510 determines the pruning order according to one or more criteria (e.g., overlapping information between views, distance between capturing cameras, etc.), which may be predetermined, configurable, etc. In the examples disclosed herein, the base views are first ordered during pruning. In some examples, the first of the base views (e.g., selected based on one or more criteria) is kept unpruned, and any other base views and additional views included in the source view 501 may be pruned. In the examples disclosed herein, the correspondence pruning unit 510 prunes the source views and outputs binary pruning masks (one per view) that indicate whether to keep (e.g., pruning mask pixels set to “1”) or remove (e.g., pruning mask pixels set to “0”) pixels in the view. The correspondence pruning unit 510 prunes the source views using the labeled pixels from the correspondence labeling unit 508.

[0068] In some examples, the correspondence pruning unit 510 initializes pixels in the pruning mask to 0 (or some other initial value). In some examples, the correspondence pruning unit sets all pixels in the pruning mask corresponding to the first (or only) base view to 1 (or some other value different from the initial value) to indicate that no pixels of the first base view are pruned. The correspondence pruning unit 510 identifies pixels labeled as “unique” and “corresponding different texture” by the correspondence labeling unit 508 for each of the other views and sets the pixels in the corresponding pruning mask for the view with a value of 1. In some examples, the correspondence pruning unit 510 can choose to set a weighting scheme for the “corresponding different texture” pixels so that those belonging to highly overlapping views are pruned and views that are far from each other are maintained. For example, views may be considered far from each other when the distance between the sources (e.g., cameras) of the views meets a threshold (e.g., 10 feet apart, 20 feet apart, etc.).

[0069] The correspondence pruning unit 510 searches the correspondence lists in the other views and identifies pixels labeled as “corresponding similar textures” by the correspondence labeling unit 508 for each of the other views. In some examples, if at least two of the pixels included in the correspondence list belong to two previously pruned views, the correspondence pruning unit 510 keeps the pixel in the associated pruning mask at 0 (e.g., an initial value to indicate that the pixel is to be pruned). Otherwise, the correspondence pruning unit 510 sets the pixel in the associated pruning mask to 1 (e.g., a value other than the initial value to indicate that the pixel is not to be pruned). In some examples, at least two pixels labeled as “corresponding similar textures” in the two pruned views are selected and set to 1 in the associated pruning mask used by the exemplary decoder to infer depth information.

[0070] In the example disclosed herein, the correspondence pruning unit 510 repeats pruning until all pixels across all views have been processed to identify a per-view pruning mask. Additionally or alternatively, the correspondence pruning unit 510 can complete pruning using other attribute content / information (e.g., entity, reflectance, etc.). Thus, the correspondence pruning unit 510 is an example of a means for generating a pruning mask. The correspondence pruning unit 510 retains corresponding pixels in the encoded bitstream if they have minimal correspondence information to provide significant information that can be used by an exemplary decoder to determine distinct texture information or depth information not present in the texture-based MIV encoder 500.

[0071] The exemplary mask aggregator 512 aggregates pruning results generated by the exemplary correspondence pruning unit 510 to generate patches. For example, the mask aggregator 108 accumulates motion-accounting pruning results (e.g., pruning masks) over an intra-period (e.g., a predetermined set of frames). Thus, the mask aggregator 512 is an example of a means for aggregating pruning masks. The exemplary correspondence patch packer 514 performs clustering on the aggregated pruning masks from the mask aggregator 512.

[0072] In some examples, the correspondence patch packing unit 514 extracts patches that can be tagged with a respective patch-wise correspondence list that includes a patch identification (patch_id) that identifies the correspondence patch. The correspondence patch packing unit 514 identifies adjacent pixels in a given aggregated pruned mask that do not have a correspondence (e.g., adjacent pixels are both labeled as “unique”) and groups the adjacent pixels into one patch. In the examples disclosed herein, the patch into which adjacent pixels are grouped is associated with an empty patch-wise correspondence list.

[0073] In some examples, the correspondence patch packing unit 514 identifies neighboring pixels in a given aggregated pruned mask that are labeled as “corresponding similar texture” or “corresponding different texture” with pixels in other aggregated pruned masks, and the correspondence patch packing unit 514 packs these neighboring pixels into one patch. The correspondence patch packing unit 514 also groups associated pixels in all other aggregated pruned masks into one patch per view, resulting in multiple correspondence patches. The correspondence patch packing unit 514 tags a patch in a given aggregated pruned mask and patches in all other aggregated pruned masks with a patch-wise correspondence list indicating the patch_id of the correspondence patch. In the examples disclosed herein, the correspondence patches need to belong to different views (e.g., each has a unique view_id). For example, two patches that belong to the same view (e.g., have the same view_id) cannot be correspondence patches. The correspondence patch packing unit 514 repeats this clustering until all pixels in the aggregated pruned mask are clustered and extracted into patches with associated correspondence information. Thus, the correspondence patch packing unit 514 is an example of a means for determining patches. In the example disclosed herein, the correspondence patch packing unit 514 clusters all information of the first base view into multiple patches (e.g., not a single entire patch) and establishes correspondences with patches in other available views.

[0074] In the examples disclosed herein, to save bits used in signaling the patch-wise correspondence list for each patch in each view, it is possible to indicate only one correspondence patch for each patch in a cyclical manner. For example, given three correspondence patches with patch_ids 2, 5, and 8, respectively, a pdu_corresponding_id may be introduced into each patch parameter. In such an example, the patch with patch_id=2 has pdu_corresponding_id=5 or a complete list pdu_corresponding_list=[5,8], the patch with patch_id=5 has pdu_corresponding_id=8 or a complete list pdu_corresponding_list=[2,8], and the patch with patch_id has pdu_corresponding_id=2 or a complete list pdu_corresponding_list=[2,5].

[0075] Once the patch clustering and extraction is complete, the correspondence patch packer 514 can generate a block-to-patch map (e.g., a patch ID map), which is a two-dimensional array indicating the patch IDs associated with a given block of one or more pixels (e.g., representing point positions / locations in the atlas).

[0076] The atlas generator 516 writes texture patches to a texture-only atlas 520 (because there is no depth component for atlases in the texture-based MIV encoder 500). The atlas generator 516 outputs the texture-only atlas 520 as exemplary attribute (texture) video data 530 to an exemplary video encoder 536. Thus, the atlas generator 516 is an example of a means for generating an atlas. The atlas generator 516 also outputs binary occupancy maps 522 (one per atlas) of the same size as the associated atlas to an exemplary occupancy packer 524. The atlas generator 516 generates metadata that conveys how patches are mapped between atlases, in addition to view parameters and other sequence information, and outputs metadata 518 to the exemplary occupancy packer 524 and the exemplary video encoder 536.

[0077] The occupancy packer 524 receives the occupancy map 522 from the atlas generator 516. In some examples, the occupancy packer 524 packs into one component every X binary pixels (bits) (e.g., X can be 8 bits, 10 bits, etc.), resulting in a reduced size for the explicitly signaled occupancy map and pixel rate savings. In some examples, the occupancy packer 524 uses length coding techniques to compress repeating patterns in a lossless manner. In some examples, the occupancy packer 524 includes the packed occupancy map 526, the block-patch map, and patch ID data in an exemplary MIV data bitstream 532. In the shown example, the MIV data bitstream 532 includes patch parameters for mapping between views and atlases, a packing order for exposing overlapping patches necessary for later deriving the block-patch map, a video-based point cloud compression (V-PCC) parameter set, and an adaptation parameter set (including view parameters), and any other metadata from the video coding data. In some examples, the MIV data bitstream 532 includes a flag (or bit) to identify that the coded video data does not include geometry (depth) information in the view / atlas.

[0078] The exemplary video encoder 534 generates an encoded atlas from the occupancy video data 528. The exemplary video encoder 536 generates an encoded atlas from the attribute (texture) video data 530. For example, the video encoder 534 receives the exemplary occupancy video data 528 and encodes the packed occupancy map 526 into an exemplary HEVC bitstream 538 using an HEVC video encoder. Additionally, the video encoder 536 receives the exemplary attribute (texture) video data 530 (including the atlas) and encodes the texture components into an exemplary HEVC bitstream 540 using an HEVC video encoder. However, the exemplary video encoder 534 and the exemplary video encoder 536 may additionally or alternatively use an Advanced Video Coding (AVC) video encoder, or the like. The exemplary HEVC bitstream 538 and the exemplary HEVC bitstream 540 may include patch parameters for mapping between views and atlases, packing order for exposing overlapping patches necessary for later obtaining block-patch maps, etc., associated with the occupancy video data 528 and the attribute (texture) video data 530, respectively. In the examples disclosed herein, the packed occupancy maps in the occupancy video data 528 and the attribute (texture) video data 530 are video coded and multiplexed with metadata included in the MIV data bitstream 532, which may be transmitted to a decoder (e.g., the exemplary texture-based MIV decoder 700 of FIG. 7) and / or a memory device for storage and eventual transmission to any number of decoders for decoding and presentation to a user.

[0079] 6 illustrates an exemplary atlas generation from input views in the texture-based MIV encoder 500 of FIG. 5. The input views include an exemplary first view 602, an exemplary second view 604, and an exemplary third view 606. In the illustrated example, the first view 602 is also labeled as view 0, the second view 604 is also labeled as view 1, and the third view 606 is also labeled as view 2. In the illustrated example, the views 602, 604, and 606 include view representations 608 (e.g., attribute (texture) maps). In the illustrated example of FIG. 6, the view representations 608 of the views 602, 604, and 606 include a third person. That is, the views 602, 604, and 606 are captured from three different cameras at three different angles with respect to the third person.

[0080] The example correspondence pruner 510 ( FIG. 5 ) prunes the views 602, 604, and 606 to generate patches. For example, the first view 602 includes an example first patch 610 and an example second patch 612, the second view 604 includes an example third patch 616, and the third view 606 includes an example fourth patch 624 and an example fifth patch 626. In some examples, each patch corresponds to one respective entity (object). For example, the first patch 210 corresponds to the head of a first person, the second patch 212 corresponds to the head of a second person, the third patch 214 corresponds to the arm of the second person, the fourth patch 216 corresponds to the head of a third person, and the fifth patch 218 corresponds to the leg of the second person.

[0081] The example correspondence pruning unit 510 prune the views 602, 604, and 606 to generate correspondence patches. For example, the first view 602 includes an example first correspondence patch 614, the second view 604 includes an example second correspondence patch 618, an example third correspondence patch 620, and an example fourth correspondence patch 622, and the third view 606 includes an example fifth correspondence patch 628. In the example disclosed herein, the correspondence pruning unit 510 identifies the correspondence patches 614, 618, 620, 622, and 628 using labeled pixels (e.g., corresponding similar textures, different textures, etc.) from the example correspondence labeling unit 508 ( FIG. 5 ). In the illustrated example, the first correspondence patch 614 of the first view 602 corresponds to the third patch 616 of the second view 604, the second correspondence patch 618 of the second view 604 corresponds to the first patch 610 of the first view 602, the third correspondence patch 620 of the second view 604 corresponds to the fourth patch 624 of the third view 606, the fourth correspondence patch 622 of the second view 604 corresponds to the fifth patch 626 of the third view 606, and the fifth correspondence patch 628 of the third view 606 corresponds to the second patch 612 of the first view 602. In some embodiments, the threshold for determining corresponding pixels may be adjusted (described above in connection with FIG. 5 ) so that more pixels are considered “different” rather than “similar,” thereby increasing the redundancy of transmitting sufficient information so that an exemplary decoder can obtain depth information from the patches. For example, the correspondence pruning unit 510 includes more corresponding pixels in the correspondence patches.

[0082] In some examples, the correspondence patch packing unit 514 (FIG. 5) tags patches with patch IDs and correspondence patches with correspondence patch IDs. For example, the correspondence patch packing unit 514 tags the first patch 610 with a patch ID of 2, the second patch 612 with a patch ID of 5, the third patch 616 with a patch ID of 8, the fourth patch 624 with a patch ID of 3, and the fifth patch 626 with a patch ID of 7. The correspondence patch packing unit 514 tags the first correspondence patch 614 with a patch ID of 18, the second correspondence patch 618 with a patch ID of 12, the third correspondence patch 620 with a patch ID of 13, the fourth correspondence patch 622 with a patch ID of 17, and the fifth correspondence patch 628 with a patch ID of 15.

[0083] The exemplary atlas generator 516 ( FIG. 5 ) generates an exemplary first atlas 630 and an exemplary second atlas 632. The atlases 630 and 632 include texture (attribute) maps (e.g., texture #0 and texture #1). The exemplary first atlas 630 includes a first patch 610, a second patch 612, a third patch 616, a third correspondence patch 620, and a fifth correspondence patch 628. The exemplary second atlas 632 includes a fourth patch 624, a fifth patch 626, the first correspondence patch 614, the second correspondence patch 618, and a fourth correspondence patch 622. In some examples, the correspondence patch packer 514 generates correspondence lists associated with the first atlas 630 and the second atlas 632. The correspondence list stores the coordinate (e.g., x, y) locations of the corresponding pixels along with the view IDs of the source views that contain the corresponding pixels. For example, the correspondence patch packer 514 identifies pixels included in the first correspondence patch 614 and stores the view IDs of the source views that contain the corresponding pixels (e.g., the first view 602) along with the coordinate (e.g., x, y) locations of the corresponding pixels in the first correspondence patch 614. The atlas generator 516 tags the first atlas 630 and the second atlas 632 with their respective correspondence lists and identifies the correspondence patches (e.g., the first correspondence patch 614, the second correspondence patch 618, the third correspondence patch 620, the fourth correspondence patch 622, and the fifth correspondence patch 628).

[0084] 7 is a block diagram of an example texture-based immersive video (MIV) decoder 700 according to the teachings of this disclosure. The example texture-based MIV decoder 700 includes an example video decoder 708, an example video decoder 712, an example occupation unpacking unit 714, an example MIV decoder and parser 718, an example block-patch map decoder 720, an example cull unit 728, and an example rendering unit 730. The example rendering unit 730 includes an example controller 732, an example depth inference unit 734, an example compositing unit 736, and an example inpainting unit 738. As shown in FIG. 7, the input bitstream of the texture-based MIV decoder 700 (e.g., received from the texture-based MIV encoder 500 of FIG. 5) includes coded texture video, coded occupation video, and metadata. In some examples, the texture-based MIV decoder 700 receives the exemplary HEVC bitstream 540, the exemplary HEVC bitstream 538, and the exemplary MIV data bitstream 532 from the exemplary texture-based MIV encoder 500 (FIG. 5).

[0085] In the illustrated example of FIG. 7, a video decoder 708 receives an HEVC bitstream 540. The HEVC bitstream 540 includes patch parameters related to attribute (texture) video data. Examples of such parameters include a map of patches between views and atlases, a packing order for exposing duplicated patches that are later used to obtain a block-patch map, etc. The video decoder 708 generates a sequence of decoded texture pictures 710. A video decoder 712 receives an HEVC bitstream 538. The HEVC bitstream 538 includes patch parameters related to decoded occupational video data. Examples of such parameters include a map of patches between views and atlases, a packing order for exposing duplicated patches that are later used to obtain a block-patch map, an occupation map, etc. The video decoder 712 provides the decoded occupational video data to an exemplary occupational unpacker 714. In the illustrated example, the exemplary decoded texture picture 710 represents an exemplary atlas. In some examples, the video decoder 708 and the video decoder 712 may be HEVC decoders. Thus, video decoder 708 and video decoder 712 are examples of means for decoding encoded video data (e.g., HEVC bitstream 538 and HEVC bitstream 540). In the illustrated example of Figure 7, video decoder 708 provides decoded texture picture 710 to renderer 730.

[0086] The exemplary occupancy unpacker 714 reverses the packing process and obtains an occupancy map 716 of the same size as the texture atlas of the decoded texture picture 710. In some examples, if the occupancy content is encoded using a run-length encoding technique, the video decoder for occupancy 712 and the occupancy unpacker 714 are replaced by a run-length decoder to reverse the encoding process and reconstruct the binary occupancy map. In the examples disclosed herein, the occupancy unpacker 714 provides the occupancy map 716 to the renderer 730 and the block-patch map decoder 720.

[0087] The MIV decoder and parser 718 receives the MIV data bitstream 532. The exemplary MIV decoder and parser 718 parses the MIV data bitstream 532 to generate exemplary atlas data 722 and exemplary video-based point cloud compression (V-PCC) and viewpoint parameter set 724. For example, the MIV decoder and parser 718 parses the encoded MIV data bitstream 532 for exemplary patch lists, camera parameter lists, etc. In the illustrated example, the MIV decoder and parser 718 parses the MIV data bitstream 532 to identify whether a flag (or bit) is set to indicate that the encoded video data does not contain geometry (depth) information. Thus, the MIV decoder and parser 718 is an example of a means for identifying a flag in encoded video data (e.g., the MIV data bitstream 532). The MIV decoder and parser 718 provides atlas data 722 and V-PCC and viewpoint parameter sets 724 to an exemplary block patch map decoder 720 , an exemplary cull unit 728 , and an exemplary renderer 730 .

[0088] The block-patch map decoder 720 receives the occupancy map 716 from the video decoder 712 and the atlas data 722 (including a patch list and a camera parameter list) and the V-PCC and viewpoint parameter set 724 from the exemplary MIV decoder and parser 718. The block-patch map decoder 720 decodes the block-patch map 726 to determine point positions / locations in the texture atlas and occupancy map 716. In the illustrated example, the block-patch map 726 is a two-dimensional array indicating patch IDs associated with a given block of one or more pixels in the texture atlas (e.g., representing point positions / locations in the atlas). The block-patch map decoder 720 provides the block-patch map 726 to the exemplary cull unit 728. Note that in the texture-based techniques described herein, there is no depth (geometry) coded in the bitstream, and therefore the geometry scaling unit is removed.

[0089] The exemplary cull unit 728 receives the block-patch map 726 from the block-patch map decoder 729, the atlas data 722 (including the patch list and camera parameter list) and the V-PCC and viewpoint parameter set 724 from the exemplary MIV decoder and parser 718. The cull unit 728 performs patch culling based on an exemplary viewing position and viewing orientation 742 (e.g., a target view). The cull unit 728 may receive the viewing position and viewing orientation 742 from a user. The cull unit 728 filters out blocks from the atlas data 722 based on the block-patch map 726 that are not visible based on the target viewing position and viewing orientation 742.

[0090] The exemplary renderer 730 generates an exemplary viewport 740. For example, the renderer 730 accesses the decoded atlas from the decoded texture picture 310, as well as one or more of the occupancy map 716, the atlas data 722 (e.g., atlas parameter list, camera parameter list, atlas patch occupancy map), the V-PCC and viewpoint parameter set 724, and the viewer position and orientation 744. That is, the exemplary renderer 730 outputs a perspective viewport of a selected texture image based on the exemplary viewer position and orientation 744. In the example disclosed herein, the viewer can dynamically move (e.g., adjust the viewer position and orientation 744) with six degrees of freedom (6DoF), adjusting their position (x, y, z) and orientation (yaw, pitch, roll) within limited ranges (e.g., supported by a head-mounted display, or a two-dimensional monitor with positional input, or the like). Thus, the renderer 730 is an example of a means for generating a target view. The renderer 730 generates an exemplary viewport 740 based on both the texture components (decoded texture picture 310) and the inferred depth components based on the corresponding texture components.

[0091] In the illustrated example of FIG. 7 , the rendering unit 730 includes a controller 732, a depth inference unit 734, a compositing unit 736, and an inpainting unit 738. The example controller 732 accesses an atlas patch occupancy map 716, which indicates whether pixels of the atlas in the decoded texture picture 710 are occupied at a viewer position and orientation 744. In the examples disclosed herein, the decoded atlas of the decoded texture picture 710 contains sufficient information to enable depth inference, if necessary. In some examples, the controller 732 uses the block-to-patch map to replace patches in the atlas in the decoded texture picture 710. The controller 732 uses the block-to-patch map to reconstruct a pruned view from the patches in the atlas of the decoded texture picture 710. In the examples disclosed herein, the pruned view is a partial view representation of the source (input) view at the encoder side (e.g., the texture-based MIV encoder 500 of FIG. 5) that is occupied by patches retained in the atlas of the decoded texture picture 710. The controller 732 reconstructs the pruned view belonging to the viewer position and orientation 744. In some examples, the pruned view belonging to the viewer position and orientation 744 may contain holes due to the presence of content in another pruned or full view.

[0092] In the illustrated example, the depth inference unit 734 infers depth information from the pruned views. Because each remapped patch in each view has at least one correspondence patch from a different pruned view (until the pixel is labeled as “unique” during the encoding stage), the depth inference unit 734 uses the correspondence patches to determine (e.g., infer / estimate) depth information from the atlas of the decoded texture picture 710. In some examples, the depth inference unit 734 identifies the correspondence patch using pdu_correspondence_id (or pdu_correspondence_list) available in the decoded texture picture 710 (or supplemented in an associated SEI message). The identification of the correspondence patch in pdu_correspondence_id (or pdu_correspondence_list) allows the depth inference unit 734 to efficiently search for corresponding pixels, for example, within a limited window (because the exact location of the correspondence patch is known), thereby eliminating the need for costly search and block-match operations typically deployed in conventional stereo-based depth estimation techniques. In some examples, the depth inference unit 734 may use more than two correspondence patches if available in the pruned views to improve estimation quality (e.g., with a tolerance that allows more than two correspondence patches for correspondence patches labeled as "corresponding different textures" or those labeled as "corresponding similar textures"). The depth inference unit 734 provides real-time performance that is interactive with viewer motion.

[0093] 7, the synthesizer 736 performs view synthesis based on the atlas patch occupancy map and the reconstructed pruned view from the controller 732 and the viewer position and orientation 744. In the example disclosed herein, the inpainter 738 fills any non-existent pixels in the reconstructed pruned view with matching values.

[0094] Figure 8 is a block diagram of an exemplary rendering process implemented by the texture-based MIV decoder 700 of Figure 7. The exemplary rendering process 800 includes an exemplary decoded geometry (depth) atlas 802, an exemplary decoded occupancy atlas 804, an exemplary decoded texture attribute atlas 806, exemplary atlas information 808, exemplary view information 810, an exemplary geometry video scaling process 812, an exemplary occupancy video reconstruction process 814, an exemplary entity filtering process 816, an exemplary source view reconstruction process 818, an exemplary depth decoding process 820, an exemplary depth estimation process 822, an exemplary reconstructed view-to-viewport projection process 824, and an exemplary blending and inpainting process 826. In the illustrated example, the atlas information 808 includes atlas parameters and patch lists, and the view information 810 includes view parameters. The exemplary rendering process 800 illustrates the rendering process of the texture-based MIV decoder 700 (Figure 7) when depth information is available or not available.

[0095] In the shown example, the entity filtering process 816 parses the atlas information 808 to identify a bitstream indicator (e.g., a flag, bit, etc.) that identifies whether a depth (geometry) atlas is included in the bitstream. For example, the atlas information 808 may include a flag (e.g., vps_geometry_video_present_flag[j], where j indicates a binary value of 0 or 1) for the atlas to identify whether depth information is included in the atlas. In some examples, if the atlas information 808 indicates that depth information is present, the exemplary geometry video scaling process 812 proceeds to decode the geometry (depth) information from the decoded geometry (depth) atlas 802. In the shown example, the occupancy video reconstruction process 814 reconstructs occupancy maps for the decoded occupancy atlas 804 and the decoded texture attribute atlas 806.

[0096] In some examples where the atlas information 808 indicates that depth information is present, the exemplary rendering process performs an exemplary source view reconstruction process 818, an exemplary depth decoding process 820, and an exemplary reconstructed view viewport projection process 824 to determine and output a perspective viewport of texture and geometry images using the decoded geometry (depth) atlas 802, the decoded occupancy atlas 804, the decoded texture attribute atlas 806, the atlas information 808, and the view information 810. In examples where the atlas information 808 indicates that depth information is not present, the exemplary rendering process 800 does not perform the exemplary depth decoding process 820, but instead performs a depth estimation process 822. In such examples, the depth estimation process 822 uses the decoded occupancy atlas 804, the decoded texture attribute atlas 806, the atlas information 808, and the view information 810 to determine (e.g., infer / estimate) depth values ​​from each of the texture attribute atlas 806. 8, the rendering process 800 may determine depth information from a provided depth atlas (e.g., a decoded geometry (depth) atlas 802) or from inferring depth from provided occupancy maps and texture atlases (e.g., a decoded occupancy atlas 804 and a decoded texture attribute atlas 806). In the example disclosed herein, an indicator included in the atlas information 808 determines which process (e.g., an example depth decoding process 820 or an example depth estimation process 822) is performed to determine depth information for the perspective viewport.

[0097] Figure 9 shows an example atlas view reconstruction 900 in the texture-based MIV decoder 700 of Figure 7. As shown in Figure 9, atlas 906 includes atlas 902 and atlas 904. In some examples, video decoder 708 (Figure 7) and video decoder 712 (Figure 7) decode atlases 902, 904. In the example shown in Figure 9, atlases 902, 904 include texture maps (e.g., texture #0 and texture #1). Atlas 902 includes an example first patch 910, an example second patch 912, an example third patch 914, an example first correspondence patch 916, and an example second correspondence patch 918. Exemplary atlas 904 includes an example fourth patch 920, an example fifth patch 922, an example third correspondence patch 924, an example fourth correspondence patch 926, and an example fifth correspondence patch 928. In some examples, the patches include patch IDs, and the correspondence patches include correspondence patch IDs determined by the example correspondence patch packing unit 514 (FIG. 5). For example, the first patch 910 includes a patch ID of 2, the second patch 912 includes a patch ID of 5, the third patch 914 includes a patch ID of 8, the fourth patch 920 includes a patch ID of 3, and the fifth patch 922 includes a patch ID of 7. The first correspondence patch 916 has a patch ID of 13, the second correspondence patch 918 has a patch ID of 15, the third correspondence patch 924 has a patch ID of 12, the fourth correspondence patch 922 has a patch ID of 17, and the fifth correspondence patch 928 has a patch ID of 18.

[0098] In some examples, the block-to-patch map decoder 720 ( FIG. 7 ) uses the block-to-patch map to match the first patch 910, the second patch 912, the third patch 914, the first correspondence patch 916, the second correspondence patch 918, the fourth patch 920, the fifth patch 922, the third correspondence patch 924, the fourth correspondence patch 926, and the fifth correspondence patch 928 with available views 930, 934, 936 included in the example view representation 908. The block-to-patch map decoder 720 matches the patches and correspondence patches with the available views to at least partially reconstruct the available views 930, 934, 936. For example, the block-patch map decoder 720 matches the first patch 910, the second patch 912, and the fifth correspondence patch 928 to the first available view 930, the third patch 914, the first correspondence patch 916, the third correspondence patch 924, and the fourth correspondence patch 926 to the second available view 934, and the fourth patch 920, the fifth patch 922, and the second correspondence patch 918 to the third available view 936.

[0099] In some examples, each patch and correspondence patch corresponds to one respective entity (object) in the available views (e.g., first available view 930, second available view 934, and third available view 936). For example, the first patch 910 in the first available view 930 and the third correspondence patch 924 in the second available view 934 correspond to the head of the first person, the second patch 912 in the first available view 930 and the second correspondence patch 918 correspond to the head of the third person, the third patch 914 in the second available view 934 and the fifth correspondence patch 928 in the first available view 930 correspond to the arm of the third person, the fourth patch 920 in the third available view 936 and the first correspondence patch 916 in the second available view 934 correspond to the head of the second person, and the fifth patch 922 and the fourth correspondence patch 926 correspond to the leg of the third person. In examples disclosed herein, patches (e.g., first patch 910, second patch 912, third patch 914, fourth patch 920, and fifth patch 922) are matched to correspondence patches (e.g., first correspondence patch 916, second correspondence patch 918, third correspondence patch 924, fourth correspondence patch 926, and fifth correspondence patch 928) in separate available views to provide additional texture information from texture maps and to determine depth information not provided in the atlases (e.g., atlas 902 and atlas 904). In some examples, the exemplary video decoder 708 (FIG. 7) decodes the exemplary HEVC bitstream 540 (FIG. 7) to identify atlases 902, 904 and their respective correspondence lists from the exemplary texture-based MIV encoder 500 of FIG. 5. The correspondence lists in the atlases 902, 904 identify, for each of the correspondence patches associated with the patch, the view ID of the source view and the coordinate (eg, x, y) location of the corresponding pixel.For example, the correspondence list for atlas 902 identifies the source view (first available view 930) of the first correspondence patch 916 and the coordinate (e.g., x, y) locations of the corresponding pixels contained in the first correspondence patch 916 as being associated with (e.g., matching) the fourth patch 920.

[0100] 10 shows an example V3C sample stream with an MIV extension (e.g., an example V3C sample stream 1000). For example, the texture-based MIV encoder 500 generates the V3C sample stream 1000. The example V3C sample stream 1000 includes an example V3C parameter set 1002, example common atlas data 1004, example atlas data 1006, example geometry video data 1008, example attribute video data 1010, and example occupancy video data 1011.

[0101] For example, the V3C parameter set 1002 includes exemplary IV access unit parameters and exemplary IV sequence parameters. The exemplary common atlas data 1004 includes an exemplary view parameter list 1012. For example, the common atlas data 1004 includes an atlas sub-bitstream, but the main Network Abstraction Layer (NAL) unit is a common atlas frame (CAF) that includes the view parameter list 1012 or an update thereof. The exemplary atlas data 1006 is an NAL sample stream that includes exemplary SEI messages. For example, the atlas data 1006 includes an atlas tile layer (ATL) that holds a list of patch data units (PDUs). In the example disclosed herein, each PDU describes the relationship between a patch in the atlas and the same patch in a hypothetical input view. The exemplary atlas data 1006 includes an exemplary atlas tile layer 1014 (e.g., patch data 1014). For example, patch data 1014 is transmitted, for example, only in intra periods 1016 (e.g., once per intra period 1016). In some examples, a frame order count NAL unit is used to skip all inter frames at once.

[0102] In the examples disclosed herein, the adaptive parameter set (contained in the common atlas data 1004) includes a texture-based signaling option. In some examples, the MIV operation mode is indicated by a dedicated metadata syntax element (e.g., aps_operation_mode) in the adaptive parameter set, such as in Table 1 below. [Table 1] Table 1

[0103] In some embodiments, aps_operation_mode specifies the MIV operation mode, as defined in Table 2 below. [Table 2] Table 2

[0104] In the example disclosed herein, the V3C parameter set 1002 includes an exemplary indicator (vps_geometry_video_present_flag[j]), where the index of the indicator (j) is set equal to 0, indicating that no geometry is present.

[0105] In some examples, vme_embedded_occupancy_flag is added to vps_miv_extension() to indicate that the occupancy is not embedded in the geometry, as defined below in connection with Table 3. [Table 3] Table 3

[0106] In some examples, an indicator (flag) (e.g., asme_patch_constant_depth_flag) is added to asps_miv_extension() to support constant depth patch per atlas functionality. In some examples, when the indicator is set, no geometry video is present for the atlas, and its value is determined directly by the pdu_depth_start[p] syntax element. [Table 4] Table 4

[0107] In the example of Table 4 above, asme_patch_constant_depth_flag set equal to 1 specifies that the decoded depth is derived as disclosed herein when vps_geometry_video_present_flag[atlasIdx] is equal to 0. In some examples, asme_patch_constant_depth_flag set equal to 0 specifies that the depth is determined by external means when vps_geometry_video_present_flag[atlasIdx] is equal to 0.

[0108] Whether implicitly embedded in the depth or explicitly present in the sub-bitstream (and how many bits are packed per component for occupancy sub-bitstream video), the occupancy map signaling can be determined for each atlas using the atlas sequence parameters set in the atlas data 1006, as shown in Table 5 below. [Table 5] Table 5

[0109] For example, the syntax element masp_occupancy_subbitstream_present_flag equal to 1 specifies that a VPCC_OVD vpcc unit may be present in the 3VC sequence. Masp_occupancy_subbitstream_present_flag equal to 0 specifies that a VPCC_OVD vpcc unit is not present in the 3VC sequence. When not present, the value of masp_occupancy_subbitstream_present_flag is inferred to be equal to 0.

[0110] The syntax element masp_occupancy_num_packed_bits specifies how many consecutive occupancy values ​​are packed into one sample. The size of the occupancy map is reduced to be (AspsFrameFrameWidth[a] / masp_occupancy_num_packed_bits) * AspsFrameFrameHeight[a].

[0111] Furthermore, the correspondence information for each patch can be defined in Table 6 as follows: [Table 6] Table 6

[0112] For example, the syntax element pdu_correspondence_id specifies the correspondence patch id associated with the patch with index equal to patchIdx. If pdu_correspondence_id is not present, it is inferred to be 0xFFFF (e.g., not valid), which means that there is no correspondence patch for the patch with index equal to patchIdx. In some embodiments, the correspondence information is additionally or alternatively signaled in a dedicated SEI message per patch per atlas.

[0113] Although an example manner of implementing the texture-based MIV encoder 500 is illustrated in Figure 5, one or more of the elements, processes, and / or devices illustrated in Figure 5 may be combined, divided, rearranged, omitted, removed, and / or implemented in any other manner. Furthermore, the example view optimizer 502, the example depth inferrer 504, the example correspondence atlas builder 506, the example correspondence labeler 508, the example correspondence pruner 510, the example mask aggregator 512, the example correspondence patch packer 514, the example atlas generator 516, the example occupancy packer 524, the example video encoder 534, the example video encoder 536, and / or more generally, the example texture-based MIV encoder 500 of Figure 5 may be implemented by hardware, software, firmware, and / or any combination of hardware, software, and / or firmware. Thus, for example, any of the exemplary view optimizer 502, the exemplary depth inferrer 504, the exemplary correspondence atlas builder 506, the exemplary correspondence labeler 508, the exemplary correspondence pruner 510, the exemplary mask aggregator 512, the exemplary correspondence patch packer 514, the exemplary video encoder 534, the exemplary atlas generator 516, the exemplary occupancy packer 524, the exemplary video encoder 536, and / or, more generally, the exemplary texture-based MIV encoder 500, may be implemented by one or more analog or digital circuits, logic circuits, programmable processors, programmable controllers, graphics processing units (GPUs), digital signal processors (DSPs), application specific integrated circuits (ASICs), programmable logic devices (PLDs), and / or field programmable logic devices (FPLDs).When reading any of the apparatus or system claims of this patent to cover solely software and / or firmware implementations, at least one of the example view optimizer 502, example depth inferrer 504, example correspondence atlas builder 506, example correspondence labeler 508, example correspondence pruner 510, example mask aggregator 512, example correspondence patch packer 514, example atlas generator 516, example occupancy packer 524, example video encoder 534, and / or example video encoder 536 is expressly defined herein to include a non-transitory computer-readable storage device or storage disk, such as a memory, a digital versatile disc (DVD), a compact disc (CD), a Blu-ray disc, etc., that contains such software and / or firmware. Still further, the example texture-based MIV encoder 500 of FIG. 5 may include one or more elements, processes, and / or devices in addition to or instead of those shown in FIG. 5, and / or may include more than one or all of the shown elements, processes, and devices. As used herein, the phrase "communicating with," including variations thereof, encompasses direct communication and / or indirect communication via one or more intermediary elements, and does not require direct physical (e.g., wired) communication and / or constant communication, but rather additionally includes selective communication at periodic intervals, scheduled intervals, aperiodic intervals, and / or one-time events.

[0114] Although an example manner of implementing the texture-based MIV decoder 700 is shown in Figure 7, one or more of the elements, processes, and / or devices shown in Figure 7 may be combined, divided, rearranged, omitted, removed, and / or implemented in any other manner. Furthermore, the example video decoder 708, the example video decoder 712, the example occupation unpacking unit 714, the example MIV decoder and parser unit 718, the example block-patch map decoder 720, the example cull unit 728, the example rendering unit 730, the example controller 732, the example depth inference unit 734, the example compositing unit 736, the example inpainting unit 738, and / or more generally, the example texture-based MIV decoder 700 of Figure 7 may be implemented in hardware, software, firmware, and / or any combination of hardware, software, and / or firmware. Thus, for example, any of the exemplary video decoder 708, the exemplary video decoder 712, the exemplary dedicated unpacking unit 714, the exemplary MIV decoder and parser unit 718, the exemplary block-patch map decoder 720, the exemplary cull unit 728, the exemplary rendering unit 730, the exemplary controller 732, the exemplary depth inference unit 734, the exemplary compositing unit 736, the exemplary inpainting unit 738, and / or, more generally, the exemplary texture-based MIV decoder 700 may be implemented by one or more analog or digital circuits, logic circuits, programmable processors, programmable controllers, graphics processing units (GPUs), digital signal processors (DSPs), application-specific integrated circuits (ASICs), programmable logic devices (PLDs), and / or field programmable logic devices (FPLDs).When any of the apparatus or system claims of this patent are read to cover solely software and / or firmware implementations, at least one of the example video decoder 708, the example video decoder 712, the example proprietary unpacking unit 714, the example MIV decoder and parser unit 718, the example block-patch map decoder 720, the example cull unit 728, the example rendering unit 730, the example controller 732, the example depth inference unit 734, the example compositor unit 736, and / or the example inpaint unit 738 is expressly defined herein to include a non-transitory computer-readable storage device or storage disk, such as a memory, a digital versatile disc (DVD), a compact disc (CD), a Blu-ray disc, etc., that contains such software and / or firmware. Still further, the example texture-based MIV decoder 700 of FIG. 7 may include one or more elements, processes, and / or devices in addition to or instead of those shown in FIG. 7, and / or may include more than one or all of the shown elements, processes, and devices. As used herein, the phrase "communicating with," including variations thereof, encompasses direct communication and / or indirect communication via one or more intermediary elements, and does not require direct physical (e.g., wired) communication and / or constant communication, but rather additionally includes selective communication at periodic intervals, scheduled intervals, aperiodic intervals, and / or one-time events.

[0115] Flowcharts representing example hardware logic, machine-readable instructions, hardware-implemented state machines, and / or any combination thereof for implementing the texture-based MIV encoder 500 of FIG. 5 are shown in FIGS. 11-14. The machine-readable instructions may be one or more executable programs or portions of executable programs for execution by a computer processor and / or processor circuitry, such as the processor 1812 shown in the example processor platform 1800 described below in connection with FIG. 18. The programs may be embodied in software stored on a non-transitory computer-readable storage medium, such as a CD-ROM, floppy disk, hard drive, DVD, Blu-ray disk, or memory associated with the processor 1812, although the entire program and / or portions thereof may alternatively be executed by a device other than the processor 1812 and / or embodied in firmware or dedicated hardware. Furthermore, although the example program is described with reference to the flowchart shown in FIG. 4, many other methods of implementing the example texture-based MIV encoder 500 may alternatively be used. For example, the order of execution of the blocks may be changed, and / or some of the described blocks may be changed, eliminated, or combined. Additionally or alternatively, any or all of these blocks may be implemented by one or more hardware circuits (e.g., discrete and / or integrated analog and / or digital circuits, FPGAs, ASICs, comparators, operational amplifiers (op-amps), logic circuits, etc.) constructed to perform the corresponding operations without executing software or firmware. Processor circuitry may be distributed locally across different network locations and / or on one or more devices (e.g., a multi-core processor on a single machine, multiple processors distributed across a server rack, etc.).

[0116] A flowchart representing example hardware logic, machine-readable instructions, hardware-implemented state machines, and / or any combination thereof for implementing the texture-based MIV decoder 700 of FIG. 7 is shown in FIG. 15. The machine-readable instructions may be one or more executable programs or portions of executable programs for execution by a computer processor and / or processor circuitry, such as the processor 1912 shown in the example processor platform 1900 described below in connection with FIG. 19. The program may be embodied in software stored on a non-transitory computer-readable storage medium, such as a CD-ROM, floppy disk, hard drive, DVD, Blu-ray disk, or memory associated with the processor 1912, although the entire program and / or portions thereof may alternatively be executed by a device other than the processor 1912 and / or embodied in firmware or dedicated hardware. Furthermore, although the example program is described with reference to the flowchart shown in FIG. 4, many other ways of implementing the example texture-based MIV decoder 700 may alternatively be used. For example, the order of execution of the blocks may be changed, and / or some of the described blocks may be changed, removed, or combined. Additionally or alternatively, any or all of these blocks may be implemented by one or more hardware circuits (e.g., discrete and / or integrated analog and / or digital circuits, FPGAs, ASICs, comparators, operational amplifiers (op-amps), logic circuits, etc.) configured to perform the corresponding operations without executing software or firmware. The processor circuits may be distributed locally across different network locations and / or on one or more devices (e.g., a multi-core processor on a single machine, multiple processors distributed across a server rack, etc.).

[0117] The machine-readable instructions described herein may be stored in one or more of a compressed format, an encrypted format, a fragmented format, a compiled format, an executable format, a packaged format, etc. The machine-readable instructions described herein may also be stored as data or data structures (e.g., portions of instructions, code, representations of code, etc.) that can be utilized to create, manufacture, and / or generate machine-executable instructions. For example, the machine-readable instructions may be fragmented and stored on one or more storage devices and / or computing devices (e.g., servers) located at the same or different locations of a network or collection of networks (e.g., in the cloud, in edge devices, etc.). The machine-readable instructions may require one or more of installing, modifying, adapting, updating, combining, supplementing, configuring, decrypting, decompressing, unpacking, distributing, reassigning, compiling, etc. to be directly readable, interpretable, and / or executable by the computing device and / or other machines. For example, the machine-readable instructions may be stored in multiple portions that are individually compressed, encrypted, and stored on separate computing devices. These portions, when decoded, decompressed, and combined, form a set of executable instructions that implement one or more functions that together may form a program such as those described herein.

[0118] In another example, machine-readable instructions may be stored in a state that can be read by a processor circuit, but may require the addition of a library (e.g., a dynamic link library (DLL)), a software development kit (SDK), an application programming interface (API), etc., to execute the instructions on a particular computing device or other device. In another example, machine-readable instructions may need to be configured (e.g., stored settings, data input, recorded network addresses, etc.) before the machine-readable instructions and / or corresponding program can be executed in whole or in part. Thus, as used herein, machine-readable media may include machine-readable instructions and / or programs, regardless of the particular format or state of the machine-readable instructions and / or programs when stored or at rest or in motion.

[0119] The machine-readable instructions described herein may be expressed in any past, present, or future command language, scripting language, programming language, etc. For example, the machine-readable instructions may be expressed using any of the following languages: C, C++, Java, C#, Perl, Python, JavaScript, Hypertext Markup Language (HTML), Structured Query Language (SQL), Swift, etc.

[0120] 11-15 may be implemented using executable instructions (e.g., computer- and / or machine-readable instructions) stored on a non-transitory computer- and / or machine-readable medium, such as a hard disk drive, flash memory, read-only memory, compact disk, digital versatile disk, cache, random access memory, and / or any other storage device or storage disk on which information is stored for any period of time (e.g., long term, permanent time, short momentary, temporary buffering, and / or caching of information). As used herein, the term non-transitory computer-readable medium is expressly defined to include any type of computer-readable storage device and / or storage disk, to exclude propagating signals, and to exclude transmission media.

[0121] The terms "including" and "comprising" (and all their forms and tenses) are used herein as open-ended terms. Thus, whenever a claim utilizes any form of "comprising" or "comprising" (e.g., comprises, includes, comprises, includes, has, etc.) as a preamble or in any type of claim recitation, it is understood that additional elements, terms, etc. may be present without departing from the scope of the corresponding claim or recitation. As used herein, the phrase "at least" is open-ended in the same way that the terms "comprising" and "comprising" are open-ended when used, for example, as a transitional phrase in a claim preamble. The term "and / or," when used in the form A, B, and / or C, means any combination of A, B, C, or any subset thereof, such as (1) A alone, (2) B alone, (3) C alone, (4) A and B, (5) A and C, (6) B and C, and (7) A, B, and C. As used herein in the context of describing a structure, element, item, object, and / or thing, the phrase “at least one of A and B” is intended to refer to an implementation that includes either (1) at least one A, (2) at least one B, or (3) at least one A and at least one B. Similarly, as used herein in the context of describing a structure, element, item, object, and / or thing, the phrase “at least one of A or B” is intended to refer to an implementation that includes either (1) at least one A, (2) at least one B, or (3) at least one A and at least one B. As used herein in the context of describing the execution or performance of a process, instruction, operation, activity, and / or step, the phrase “at least one of A and B” is intended to refer to an implementation that includes either (1) at least one A, (2) at least one B, or (3) at least one A and at least one B.Similarly, when the phrase "at least one of A or B" is used herein in the context of describing the implementation or performance of a process, instruction, action, activity, and / or step, it is intended to refer to implementations that include any of: (1) at least one A, (2) at least one B, and (3) at least one A and at least one B.

[0122] As used herein, a singular reference (e.g., "a," "an," "first," "second," etc.) does not exclude a plurality. As used herein, the term "a" or "an" entity refers to one or more of that entity. As used herein, the terms "a" (or "an"), "one or more," and "at least one" may be used interchangeably. Furthermore, multiple means, elements, or method actions, although individually listed, may be implemented by, for example, a single unit or processor. Additionally, although individual features may be included in different examples or claims, these may in some cases be combined, and inclusion in different examples or claims does not imply that a combination of features is not feasible and / or advantageous.

[0123] FIG. 11 is a flowchart representing machine-readable instructions that may be executed to implement the texture-based MIV encoder 500 of FIG. 5. The program 1100 of FIG. 11 begins execution at block 1102, where the exemplary view optimizer 502 (FIG. 5) receives video data for a scene. For example, the view optimizer 502 receives an input view (e.g., the source view 501 of FIG. 5) that includes a texture map. In block 1104, the exemplary correspondence labeler 508 (FIG. 5) identifies, synthesizes, and labels corresponding pixels of the available input views. For example, the correspondence labeler 508 identifies corresponding pixels across all views (e.g., base and / or additional views) and labels them based on texture content. As described in more detail below, the exemplary flowchart 1104 of FIG. 12 represents exemplary machine-readable instructions that may be implemented to identify, synthesize, and label corresponding pixels of the available input views.

[0124] At block 1106, the exemplary correspondence pruning unit 510 (FIG. 5) performs correspondence pruning on the base view and the additional views. For example, the correspondence pruning unit 510 prunes the base view and the additional views based on labeled pixels from the correspondence labeling unit 508. As described in more detail below, the exemplary flowchart 1106 of FIG. 13 represents exemplary machine-readable instructions that may be implemented to perform correspondence pruning on the base view and the additional views. At block 1108, the exemplary correspondence pruning unit 510 determines a correspondence pruning mask. For example, the correspondence pruning unit 510 outputs a binary pruning mask (one per view) that indicates whether pixels in the view are to be kept (e.g., pruning mask pixels are set to “1”) or removed (e.g., pruning mask pixels are set to “0”).

[0125] At block 1110, the exemplary correspondence patch packer 514 (FIG. 5) generates correspondence patches. For example, the correspondence patch packer 514 extracts patches that can be tagged with respective patch-wise correspondence lists that include patch identification information (patch_id) that identify the correspondence patches. As described in more detail below, the exemplary flowchart 1110 of FIG. 14 represents exemplary machine-readable instructions that may be implemented to generate correspondence patches. At block 1112, the exemplary atlas generator 516 (FIG. 5) generates an atlas. For example, the atlas generator 516 writes texture patches from the correspondence patch packer 514 into a texture-only atlas (because there is no depth component for atlases in the texture-based MIV encoder 500).

[0126] At block 1114, the exemplary texture-based MIV encoder 500 generates an encoded bitstream. For example, video encoder 534 and video encoder 536 generate encoded atlases from the occupancy video data 528 (FIG. 5) and the attribute (texture) video data 530 (FIG. 5), respectively. Video encoder 534 receives the exemplary occupancy video data 528 and encodes the occupancy map into an exemplary HEVC bitstream 538 (FIG. 5). Video encoder 536 receives the exemplary attribute (texture) video data 530 (including the atlas) and encodes the texture component into an exemplary HEVC bitstream 540 (FIG. 5). The exemplary HEVC bitstream 538 includes patch parameters for the decoded occupancy video data. Examples of such parameters include a map of patches between views and atlases, a packing order for exposing overlapping patches that are later utilized to obtain a block-patch map, an occupancy map, etc. The exemplary HEVC bitstream 540 includes patch parameters for the attribute (texture) video data. Examples of such parameters include a map of patches between views and atlases, a packing order for exposing overlapping patches that are later utilized to obtain a block-patch map, etc. In some examples, the MIV data bitstream 532 includes a flag (or bit) for identifying that the encoded video data (e.g., the HEVC bitstream 538 and the HEVC bitstream 540) does not include geometry (depth) information in the view / atlas. In the examples disclosed herein, the packed occupancy maps in the occupancy video data 528 and the attribute (texture) video data 530 are video coded and multiplexed with metadata included in the MIV data bitstream 532, which may be transmitted to a decoder (e.g., the exemplary texture-based MIV decoder 700 of FIG. 7) and / or a memory device for storage and eventual transmission to any number of decoders for decoding and presentation to a user. After block 1114, the program 1100 ends.

[0127] 12 is a flowchart representing machine-readable instructions that may be executed to implement an example correspondence labeler 508 (FIG. 5) included in the texture-based MIV encoder 500 of FIG. 5. The program 1104 of FIG. 12 begins execution at block 1202, where the example correspondence labeler 508 selects a first pixel from a first available input view. At block 1204, the example correspondence labeler 508 selects a second pixel from a second available input view. For example, the correspondence labeler 508 selects a first pixel and a second pixel (a first pixel from the first available view and a second pixel from the second available view) from any available input views (e.g., base and / or additional views) and labels them based on texture content and depth information, if available.

[0128] At block 1206, the exemplary correspondence labeling unit 508 determines a de-projection of the first pixel onto the three-dimensional world. At block 1208, the exemplary correspondence labeling unit 508 determines a re-projection of the first pixel onto a second available input view. At block 1210, the exemplary correspondence labeling unit 508 determines whether the first pixel is co-located with the second pixel. If the exemplary correspondence labeling unit 508 determines that the first pixel is co-located with the second pixel, at block 1212, the exemplary correspondence labeling unit 508 labels the first pixel and the second pixel as corresponding pixels. In the examples disclosed herein, if deprojection of a first pixel from a first available input view onto the three-dimensional world and subsequent reprojection of the first pixel onto a second available input view results in the first pixel being co-located with the second pixel from the second available input view, the first pixel and the second pixel are considered to be corresponding pixels. Returning to block 1210, if the exemplary correspondence labeling unit 508 determines that the first pixel is not co-located with the second pixel, then in block 1214, the exemplary correspondence labeling unit 508 labels the first pixel and the second pixel as unique pixels. If a pixel from any available input view does not have a corresponding pixel in another available input view, the correspondence labeling unit 508 labels the pixel as “unique.” For example, pixels located in regions at the edge of the device camera or in occlusion regions that are visible only by certain views are typically labeled as “unique.”

[0129] At block 1216, the example correspondence labeler 508 determines whether there are additional pixels. If the example correspondence labeler 508 determines that there are additional pixels, the program 1104 returns to block 1202 where the example correspondence labeler 508 selects a different pixel from the first available input view. Returning to block 1216, if the example correspondence labeler 508 determines that there are no additional pixels, then at block 1218, the example correspondence labeler 508 determines whether there are additional available input views. If the example correspondence labeler 508 determines that there are additional available input views, the program 1104 returns to block 1202 where the example correspondence labeler 508 selects a first pixel from a different available input view. Returning to block 1218, if the example correspondence labeler 508 determines that there are no additional available input views, then at block 1212, the example correspondence labeler 508 selects corresponding pixels from all available input views.

[0130] At block 1222, the exemplary correspondence labeling unit 508 compares texture content between two corresponding pixels. For example, for each of the corresponding pixels, the correspondence labeling unit 508 determines whether the corresponding pixels have similar texture content. At block 1224, the exemplary correspondence labeling unit 508 determines whether a difference in texture content meets a threshold. For example, the correspondence labeling unit 508 compares texture content between two corresponding pixels and compares the difference in texture content to a threshold. In some examples, the threshold may be a difference in color components (e.g., red, green, and blue (RGB) components). For example, the threshold may be a difference of 5 (or some other value) between any one of the color components. If the exemplary correspondence labeling unit 508 determines that the difference in texture content does not meet the threshold, then at block 1226, the exemplary correspondence labeling unit 508 labels the corresponding pixels as similar. The correspondence labeling unit 508 determines that corresponding pixels have similar texture content when the difference in texture content is below a threshold, and labels the corresponding pixels as "similar texture" and stores the view IDs of the source views containing the corresponding pixels along with the coordinate (e.g., x, y) locations of the corresponding pixels in a correspondence list.

[0131] Returning to block 1224, if the example correspondence labeler 508 determines that the difference in texture content does not meet a threshold, then in block 1228, the example correspondence labeler 508 labels the corresponding pixels as different. When the difference in texture content is above a threshold, the correspondence labeler 508 determines that two corresponding pixels have different texture content. In some examples, corresponding pixels may be determined to have different texture content based on different reflectance information, different illumination / light information, color mismatch due to different camera settings, etc. The correspondence labeler 508 labels the corresponding pixels as "different texture" and stores the view IDs of the source views containing the corresponding pixels along with the coordinate (e.g., x, y) locations of the corresponding pixels in a correspondence list.

[0132] At block 1230, the example correspondence labeler 508 determines whether there are additional corresponding pixels. If the example correspondence labeler 508 determines that there are additional corresponding pixels, the program 1104 returns to block 1220 where the example correspondence labeler 508 compares the texture content between two different corresponding pixels. Returning to block 1230, if the example correspondence labeler 508 determines that there are no additional corresponding pixels, the program 1104 is complete and returns to program 1100 of FIG. 11 .

[0133] FIG. 13 is a flowchart representing machine-readable instructions that may be executed to implement an example correspondence pruning unit 510 (FIG. 5) included in the texture-based MIV encoder 500 of FIG. 5. The program 1106 of FIG. 13 begins execution at block 1302, where the example correspondence pruning unit 510 sets all pixels to those in a pruning mask of a base view. The correspondence pruning unit 510 determines the pruning order according to a given criterion (e.g., overlapping information between views, distance between capture cameras, etc.), which may be predetermined, configurable, etc. In the example disclosed herein, the base views are ordered first during pruning. For example, the first of the base views (e.g., selected based on one or more criteria) may be kept unpruned, and any other base views and additional views included in the source view 501 (FIG. 5) may be pruned.

[0134] At block 1304, the exemplary correspondence pruning unit 510 selects an additional view. For example, the correspondence pruning unit 510 selects an additional view from the source view 501. At block 1306, the exemplary correspondence pruning unit 510 sets pixels in the pixel mask corresponding to view pixels labeled as unique to 1 (e.g., indicating that the pixels are not pruned). At block 1308, the exemplary correspondence pruning unit 510 sets pixels in the pixel mask corresponding to view pixels labeled as corresponding and different to 1 (e.g., indicating that the pixels are not pruned). The correspondence pruning unit 510 identifies pixels labeled as “unique” and “corresponding different texture” by the correspondence labeling unit 508 for each of the other views and sets the pixels in the corresponding pruning mask for the view with a value of 1. In some examples, the correspondence pruning unit 510 can choose to set a weighting scheme for "corresponding different texture" pixels so that those belonging to highly overlapping views are pruned and views that are far from each other are maintained. For example, views may be considered far from each other when the distance between the sources (e.g., cameras) of the views meets a threshold (e.g., 10 feet apart, 20 feet apart, etc.).

[0135] At block 1310, the exemplary correspondence pruning unit 510 determines whether pixels in the current pruned view labeled as corresponding and similar are included in two previously pruned views. The correspondence pruning unit 510 searches the correspondence lists in the other views and identifies pixels labeled as “corresponding similar textures” by the correspondence labeling unit 508 for each of the other views. If the exemplary correspondence pruning unit 510 determines whether pixels in the current pruned view labeled as corresponding and similar are included in two previously pruned views (e.g., the pixels in the two previously pruned views are set to 1), at block 1312, the exemplary correspondence pruning unit 510 sets the pixels labeled as corresponding and similar to 0 (e.g., indicating that the pixels are pruned). In some examples, if at least two of the pixels included in the correspondence list belong to a previously pruned view, the correspondence pruning unit 510 maintains the pixel in the associated pruning mask at 0. Returning to block 1310, if the example correspondence pruning unit 510 determines that pixels in the current pruned view labeled as corresponding and similar are not included in two previously pruned views (e.g., at least one of the two previously pruned views does not include the pixel, or the pixel in at least one of the two previously pruned views is set to 0), then in block 1314, the example correspondence pruning unit 510 sets the pixels labeled as corresponding and similar to 1 (e.g., indicating that the pixel will not be pruned). In some examples, at least two pixels labeled as “corresponding, similar texture” are selected and set to 1 in the associated pruning mask used by the example decoder to infer depth information.

[0136] At block 1316, the exemplary correspondence pruning unit 510 determines whether there are additional pixels in the current pruned view that are labeled as corresponding and similar. If the exemplary correspondence pruning unit 510 determines that there are additional pixels in the current pruned view that are labeled as corresponding and similar, the program 1106 returns to block 1310 where the exemplary correspondence pruning unit 510 determines whether pixels in the current pruned view that are labeled as corresponding and similar are included in two previously pruned views. Returning to block 1316, if the exemplary correspondence pruning unit 510 determines that there are no additional pixels in the current pruned view that are labeled as corresponding and similar, then at block 1318, the exemplary correspondence pruning unit 510 determines whether there are additional views. If the example correspondence pruning unit 510 determines that there are additional views, then the program 1106 returns to block 1310 where the example correspondence pruning unit 510 determines whether pixels in the current pruned view that correspond and are labeled as similar are included in two previously pruned views. Returning to block 1318, if the example correspondence pruning unit 510 determines that there are no additional views, then the program 1106 is complete and returns to the program 1100 of FIG. 11 .

[0137] Figure 14 is a flowchart representing machine-readable instructions that may be executed to implement the example correspondence patch packer 514 (Figure 5) included in the texture-based MIV encoder 500 of Figure 5. The program 1110 of Figure 14 begins execution at block 1402, where the example correspondence patch packer 514 obtains a pruned correspondence mask from the example correspondence pruner 510 (Figure 5).

[0138] At block 1404, the exemplary correspondence patch packing unit 514 compares adjacent pixels in the pruned correspondence mask. For example, the correspondence patch packing unit 514 compares the labeling of adjacent pixels in the pruned correspondence mask. At block 1406, the exemplary correspondence patch packing unit 514 determines whether the adjacent pixels are labeled as unique. For example, the correspondence patch packing unit 514 identifies adjacent pixels in a given aggregated pruned mask that do not have a correspondence (e.g., both adjacent pixels are labeled as “unique” in the pruned correspondence mask). If the exemplary correspondence patch packing unit 514 determines that the adjacent pixels are labeled as unique, at block 1408, the exemplary correspondence patch packing unit 514 groups the adjacent pixels into one patch that does not have a correspondence list. For example, the correspondence patch packing unit 514 groups adjacent pixels in a given aggregated pruned correspondence mask that do not have a correspondence (e.g., adjacent pixels are both labeled as “unique”) into one patch, and the patch is associated with an empty patch-wise correspondence list.

[0139] Returning to block 1406, if the exemplary correspondence patch packing unit 514 determines that the adjacent pixels are not labeled as unique, then at block 1410, the exemplary correspondence patch packing unit 514 determines whether the adjacent pixels are labeled as corresponding (either similar or different). For example, the correspondence patch packing unit 514 identifies adjacent pixels in a given aggregated pruned mask that are labeled as "corresponding similar texture" or "corresponding different texture." If the exemplary correspondence patch packing unit 514 determines that the adjacent pixels are labeled as corresponding (either similar or different), then at block 1412, the exemplary correspondence patch packing unit 514 groups the adjacent pixels into a second patch. Returning to block 1410, if the exemplary correspondence patch packing unit 514 determines that the adjacent pixels are not labeled as corresponding (similar or different), then in block 1416, the exemplary correspondence patch packing unit 514 determines whether there are any additional pixels in the pruned correspondence mask.

[0140] At block 1414, the exemplary correspondence patch packing unit 514 tags the second patch with a correspondence list that identifies the correspondence patch. For example, the correspondence patch packing unit 514 tags the second patch in a given aggregated pruned correspondence mask with a patch-wise correspondence list that indicates the patch_id of the correspondence patch. The correspondence patch packing unit 514 also groups associated pixels in all other aggregated pruned correspondence masks into one patch (e.g., the second patch) per view, resulting in multiple correspondence patches.

[0141] At block 1416, the example correspondence patch packing unit 514 determines whether there are additional pixels in the pruned correspondence mask. If the example correspondence patch packing unit 514 determines that there are additional pixels in the pruned correspondence mask, the program 1110 returns to block 1404 where the example correspondence patch packing unit 514 compares different adjacent pixels in the pruned correspondence mask. Returning to block 1416, if the example correspondence patch packing unit 514 determines that there are no additional pixels in the pruned correspondence mask, the program 1110 is complete and returns to program 1100 of FIG. 11 .

[0142] FIG. 15 is a flowchart representing machine-readable instructions that may be executed to implement the texture-based MIV decoder 700 of FIG. 7. The program 1500 of FIG. 15 begins execution at block 1502, where the exemplary texture-based MIV decoder 700 (FIG. 7) receives encoded data. For example, the texture-based MIV decoder 700 receives the exemplary HEVC bitstream 540, the exemplary HEVC bitstream 538, and the exemplary MIV data bitstream 532 generated by the exemplary texture-based MIV encoder 500 (FIG. 5). At block 1504, the exemplary texture-based MIV decoder 700 demultiplexes and decodes the encoded data. For example, the video decoder 708 (FIG. 7) and the video decoder 712 (FIG. 7) decode the encoded atlas to generate a decoded atlas including texture and occupancy data. In some examples, the HEVC bitstream 540 includes patch parameters for attribute (texture) video data. Examples of such parameters include a map of patches between views and atlases, a packing order for exposing duplicated patches that are later used to obtain a block-patch map, etc. The video decoder 708 generates a sequence of decoded texture atlases (e.g., decoded texture pictures 710). In some examples, the HEVC bitstream 538 includes patch parameters for the decoded occupancy video data. Examples of such parameters include a map of patches between views and atlases, a packing order for exposing duplicated patches that are later used to obtain a block-patch map, an occupancy map, etc. The video decoder 712 decodes the occupancy atlas.

[0143] In block 1506, the exemplary MIV decoder and parser 718 determines a parameter list. For example, the MIV decoder and parser 718 (FIG. 7) receives the MIV data bitstream 532. The exemplary MIV decoder and parser 718 parses the MIV data bitstream 532 to generate exemplary atlas data 722 (FIG. 7) and exemplary video-based point cloud compression (V-PCC) and viewpoint parameter set 724 (FIG. 7). For example, the MIV decoder and parser 718 parses the encoded MIV data bitstream 532 for an exemplary patch list, a camera parameter list, etc. In block 1508, the exemplary occupancy unpacker 714 (FIG. 7) unpacks the atlas patch occupancy map. For example, the occupancy unpacker 714 reverses the packing process to obtain an occupancy map 716 (FIG. 7) that is the same size as the texture atlas of the decoded texture picture 710 (FIG. 7).

[0144] At block 1510, the exemplary MIV decoder and parser 718 determines whether depth information is included in the encoded bitstream. For example, the MIV decoder and parser 718 determines whether the atlas data 722 (FIG. 7) includes an indicator (e.g., a flag, bit, etc.) indicating whether depth information is included in the encoded bitstream (e.g., HEVC bitstream 540, HEVC bitstream 538, and MIV data bitstream 532). If the exemplary MIV decoder and parser 718 determines that there is included depth information, then at block 1514, the exemplary renderer 730 synthesizes a view. For example, the synthesizer 736 (FIG. 7) synthesizes a scene based on the atlas (texture and depth), occupancy map, and / or parameters. After block 1514, the program 1500 ends.

[0145] Returning to block 1510, if the example MIV decoder and parser 718 determines that no depth information is included, then in block 1512, the example renderer 730 determines depth information from correspondence patches. For example, the depth inference unit 734 infers depth information using correspondence patches in a texture atlas. After block 1512, the program 1500 continues to block 1514, where the example renderer 730 composites the view. For example, the composite unit 736 composites the scene based on the texture atlas, occupancy map, inferred depth information, and / or parameters. After block 1514, the program 1500 ends.

[0146] 16-17 show example systems and devices, encoders (e.g., MIV encoder 100 of FIG. 1, texture-based MIV encoder 500 of FIG. 5, etc.), and decoders (e.g., MIV decoder 300 of FIG. 3, texture-based MIV decoder 700 of FIG. 7) for implementing the described techniques. For example, any encoder (encoder system), decoder (decoder system), or bitstream extractor described herein may be implemented via the system shown in FIG. 16 and / or the device implemented in FIG. 17. In some examples, in the described techniques, the encoder (e.g., MIV encoder 100 of FIG. 1, texture-based MIV encoder 500 of FIG. 5, etc.) and decoder (e.g., MIV decoder 300 of FIG. 3, texture-based MIV decoder 700 of FIG. 7) may be implemented via any suitable device or platform described herein, such as a personal computer, a laptop computer, a tablet, a phablet, a smartphone, a digital camera, a gaming console, a wearable device, a display device, an all-in-one device, a two-in-one device, or the like.

[0147] The various components of the systems described herein may be implemented in software, firmware, and / or hardware and / or any combination thereof. For example, the various components of the devices or systems described herein may be provided, at least in part, by the hardware of a computing system-on-chip (SoC), such as may be found in a computing system such as a smartphone. Those skilled in the art will appreciate that the systems described herein may include additional components not shown in the corresponding figures. For example, the systems described herein may include additional components not shown for clarity.

[0148] Although implementations of the example processes discussed herein may include the performance of all of the operations shown in the order shown, the disclosure is not limited in this respect, and in various examples, implementations of the example processes herein may include only a subset of the operations shown, operations performed in an order different from that shown, or additional operations.

[0149] Additionally, any one or more of the operations discussed herein may be performed in response to instructions provided by one or more computer program products. Such program products may include, for example, signal-bearing media that provide instructions that, when executed by a processor, may provide the functionality described herein. A computer program product may be provided in any form of one or more machine-readable media. Thus, for example, a processor including one or more graphics processing units or processor cores may execute one or more of the blocks of the example processes herein in response to program code and / or instructions or instruction sets transmitted to the processor by one or more machine-readable media. Generally, a machine-readable medium may carry software in the form of program code and / or instructions or instruction sets that may cause any of the devices and / or systems described herein to implement at least a portion of the device or system, or any other module or component described herein.

[0150] As used herein, the term "module" refers to any combination of software logic, firmware logic, hardware logic, and / or circuitry configured to provide the functionality described herein. Software may be embodied as a software package, code, and / or instruction set or instructions, and as used herein, "hardware" may include, for example, hardwired circuitry, programmable circuitry, state machine circuitry, fixed function circuitry, execution unit circuitry, and / or firmware that stores instructions executed by programmable circuitry, either singly or in any combination. Modules may collectively or individually be embodied as circuitry that forms part of a larger system, such as, for example, an integrated circuit (IC), system-on-chip (SoC), etc.

[0151] 16 is an illustrative diagram of an example system 1600 configured in accordance with at least some implementations of the present disclosure. In some examples, system 1600 may be a mobile device system, although system 1600 is not limited to this context. For example, system 1600 may be incorporated into a personal computer (PC), a laptop computer, an ultra-laptop computer, a tablet, a touchpad, a portable computer, a handheld computer, a palmtop computer, a personal digital assistant (PDA), a cellular phone, a cellular phone / PDA combination, a television, a smart device (such as a smartphone, a smart tablet, or a smart TV), a mobile internet device (MID), a messaging device, a data communication device, a camera (e.g., a full-frame camera, a superzoom camera, a digital single-lens reflex (DSLR) camera), a surveillance camera, a surveillance system including a camera, etc.

[0152] In some examples, system 1600 includes a platform 1602 coupled to a display 1620. Platform 1602 may receive content from content devices, such as content services device(s) 1630 or content delivery device(s) 1640, or other content sources, such as image sensor 1619. For example, platform 1602 may receive image data described herein from image sensor 1619 or any other content source. A navigation controller 1650, comprising one or more navigation functions, can be used to interact with platform 1602 and / or display 1620, for example. Each of these elements is described in more detail below.

[0153] In some examples, platform 1602 may include any combination of chipset 1605, processor 1610, memory 1612, antenna 1613, storage 1614, graphics subsystem 1615, applications 1616, image signal processor 1617, and / or radio 1618. Chipset 1605 may provide intercommunication between processor 1610, memory 1612, storage 1614, graphics subsystem 1615, applications 1616, image signal processor 1617, and / or radio 1618. For example, chipset 1605 may include a storage adapter (not shown) that can provide intercommunication with storage 1614.

[0154] The processor 1610 may be implemented as a complex instruction set computer (CISC) processor or a reduced instruction set computer (RISC) processor, multiple processors compatible with the x86 instruction set, a multi-core or any other microprocessor, or a central processing unit (CPU). In some examples, the processor 1610 may be a dual-core processor, a dual-core mobile processor, or the like.

[0155] The memory 1612 may be implemented as a volatile memory device such as, but not limited to, random access memory (RAM), dynamic random access memory (DRAM), or static RAM (SRAM).

[0156] Storage 1614 may be implemented as a non-volatile storage device such as, but not limited to, a magnetic disk drive, an optical disk drive, a tape drive, an internal storage device, an attached storage device, flash memory, a battery-backed SDRAM (synchronous DRAM), and / or a network-accessible storage device. In some examples, storage 1614 may be equipped with technology that increases storage performance-enhancing protection for valuable digital media, for example, when multiple hard drives are included.

[0157] The image signal processor 1617 may be implemented as a dedicated digital signal processor used for image processing or the like. In some examples, the image signal processor 1617 may be implemented based on a single-instruction-multiple-data or multiple-instruction-multiple-data architecture or the like. In some examples, the image signal processor 1617 may be characterized as a media processor. As described herein, the image signal processor 1617 may be implemented based on a system-on-chip architecture and / or based on a multi-core architecture.

[0158] The graphics subsystem 1615 may perform processing of multiple images, such as still images or moving images, for display. The graphics subsystem 1615 may be, for example, a graphics processing unit (GPU) or a visual processing unit (VPU). An analog or digital interface may be used to communicatively couple the graphics subsystem 1615 and the display 1620. For example, the interface may be a High-Definition Multimedia Interface, DisplayPort, Wireless HDMI, and / or Wireless HD compliant technology. The graphics subsystem 1615 may be integrated into the processor 1610 or the chipset 1605. In some implementations, the graphics subsystem 1615 may be a standalone device communicatively coupled to the chipset 1605.

[0159] The graphics and / or video processing techniques described herein may be implemented in a variety of hardware architectures. For example, graphics and / or video capabilities may be integrated within a chipset. Alternatively, a separate graphics and / or video processor may be used. In yet another implementation, graphics and / or video capabilities may be provided by a general-purpose processor, including a multi-core processor. In a further embodiment, these capabilities may be implemented in a consumer electronic device.

[0160] Radio unit 1618 may include one or more radios capable of transmitting and receiving signals using various suitable wireless communication techniques. Such techniques may include communications across one or more wireless networks. Exemplary wireless networks include, but are not limited to, wireless local area networks (WLANs), wireless personal area networks (WPANs), wireless metropolitan area networks (WMANs), cellular networks, and satellite networks. When communicating across such networks, radio unit 1618 may operate in accordance with one or more applicable standards in any version.

[0161] In some examples, the display 1620 may include any television-type monitor or display. The display 1620 may include, for example, a computer display screen, a touchscreen display, a video monitor, a television-like device, and / or a television. The display 1620 may be digital and / or analog. In some examples, the display 1620 may be a holographic display. The display 1620 may also be a transparent surface that can receive visual projections. Such projections may convey various forms of information, images, and / or objects. For example, such projections may be visual overlays for mobile augmented reality (MAR) applications. Under control of one or more software applications 1616, the platform 1602 may display a user interface 1622 on the display 1620.

[0162] In some examples, content services device(s) 1630 may be hosted by any national, international, and / or independent service and thus accessible to platform 1602, for example, via the Internet. Content services device(s) 1630 may be coupled to platform 1602 and / or display 1620. Platform 1602 and / or content services device(s) 1630 may be coupled to network 1660 to communicate (e.g., send and / or receive) media information to and from network 1660. Content delivery device(s) 1640 may also be coupled to platform 1602 and / or display 1620.

[0163] Image sensor 1619 may include any suitable image sensor capable of providing image data based on a scene. For example, image sensor 1619 may include a semiconductor charge-coupled device (CCD)-based sensor, a complementary metal-oxide semiconductor (CMOS)-based sensor, an N-type metal-oxide semiconductor (NMOS)-based sensor, or the like. For example, image sensor 1619 may include any device capable of detecting information of a scene to generate image data.

[0164] In some examples, content service device(s) 1630 may include a cable television box, a personal computer, a network, a telephone, an Internet-enabled device or appliance capable of delivering digital information and / or content, and any other similar device capable of unidirectionally or bidirectionally communicating content between a content provider and platform 1602 and / or display 1620 over network 1660 or directly. It will be understood that content can be communicated unidirectionally and / or bidirectionally to and from any one of the components and content providers in system 1600 over network 1660. Examples of content may include any media information including, for example, video, music, medical and gaming information, etc.

[0165] Content services device 1630 may receive content, such as cable television programs, including media information, digital information, and / or other content. Examples of content providers may include any cable or satellite television, radio, or Internet content provider. The examples provided are not intended to limit implementations according to this disclosure in any manner.

[0166] In some examples, platform 1602 can receive control signals from a navigation controller 1650 having one or more navigation functions. The navigation functions of navigation controller 1650 can be used, for example, to interact with user interface 1622. In various embodiments, navigation controller 1650 can be a pointing device, which can be a computer hardware component (particularly a human interface device) that allows a user to input spatial (e.g., continuous, multi-dimensional) data into a computer. Many systems, such as graphical user interfaces (GUIs), televisions, and monitors, can use physical gestures to allow a user to control and provide data to a computer or television.

[0167] Movement of navigation features on navigation controller 1650 may be replicated on a display (e.g., display 1620) by movement of a pointer, cursor, focus ring, or other visual indicator displayed on the display. For example, under the control of software application 1616, multiple navigation features located on navigation controller 1650 may be mapped to multiple virtual navigation features displayed on user interface 1622, for example. In various embodiments, navigation controller 1650 may be integrated into platform 1602 and / or display 1620 rather than being a separate element. However, this disclosure is not limited to the elements or in the context shown or described herein.

[0168] In some examples, the driver (not shown) may include technology that, if enabled, allows a user to instantly turn platform 1602 on and off, like a television, with the touch of a button after initial boot-up. Program logic may enable platform 1602 to stream content to a media adapter or other content service device 1630 or content delivery device 1640, even when the platform is “off.” Additionally, chipset 1605 may include hardware and / or software support for, for example, 5.1 surround sound audio and / or high-resolution 7.1 surround sound audio. Drivers may include graphics drivers for multiple integrated graphics platforms. In various embodiments, the graphics driver may comprise a peripheral component interconnect (PCI) express graphics card.

[0169] In some examples, any one or more of the elements shown in system 1600 may be integrated. For example, platform 1602 and content services device(s) 1630 may be integrated, or platform 1602 and content delivery device(s) 1640 may be integrated, or, for example, platform 1602, content services device(s) 1630, and content delivery device(s) 1640 may be integrated. In various embodiments, platform 1602 and display 1620 may be an integrated unit. For example, display 1620 and content services device(s) 1630 may be integrated, or display 1620 and content delivery device(s) 1640 may be integrated. These examples are not intended to limit the disclosure.

[0170] In various embodiments, system 1600 may be implemented as a wireless system, a wired system, or a combination of both. If implemented as a wireless system, system 1600 may include components and interfaces suitable for communicating over a wireless shared medium, such as one or more antennas, transmitters, receivers, transceivers, amplifiers, filters, control logic, etc. Examples of wireless shared media may include portions of the radio spectrum, such as the RF spectrum. If implemented as a wired system, system 1600 may include components and interfaces suitable for communicating over a wired communication medium, such as input / output (I / O) adapters, physical connectors connecting the I / O adapters to corresponding wired communication media, network interface cards (NICs), disk controllers, video controllers, audio controllers, etc. Examples of wired communication media may include wires, cables, metal leads, printed circuit boards (PCBs), backplanes, switch fabrics, semiconductor materials, twisted pair wires, coaxial cables, optical fibers, etc.

[0171] Platform 1602 may establish one or more logical or physical channels for communicating information. The information may include media information and control information. Media information may refer to any data representing content for a user. Examples of content may include, for example, data from a voice conversation, a video conference, streaming video, electronic mail ("email") messages, voicemail messages, alphanumeric symbols, graphics, images, video, text, etc. Data from a voice conversation may be, for example, speech information, periods of silence, background noise, comfort noise, tones, etc. Control information may refer to any data representing commands, instructions, or control words intended for an automated system. For example, control information may be used to route media information through a system or instruct a node to process the media information in a predetermined manner. However, embodiments are not limited to the elements or circumstances shown or described in FIG. 16 .

[0172] As noted above, system 1600 may be embodied in different physical styles or form factors. FIG. 17 illustrates an exemplary small form factor device 1700 arranged in accordance with at least some implementations of the present disclosure. In some examples, system 1600 may be implemented via device 1700. In other examples, other systems, components, or modules described herein or portions thereof may be implemented via device 1700. For example, in various embodiments, device 1700 may be implemented as a mobile computing device with wireless capabilities. A mobile computing device may refer to any device that includes a processing system and a mobile power source or power supply, such as, for example, one or more batteries.

[0173] Examples of mobile computing devices may include personal computers (PCs), laptop computers, ultra-laptop computers, tablets, touchpads, portable computers, handheld computers, palmtop computers, personal digital assistants (PDAs), cellular phones, combination cellular phones / PDAs, smart devices (e.g., smartphones, smart tablets, or smart mobile televisions), mobile internet devices (MIDs), messaging devices, data communication devices, cameras (e.g., full-frame cameras, superzoom cameras, digital single-lens reflex (DSLR) cameras), and the like.

[0174] Examples of mobile computing devices may also include computers implemented by motor vehicles or robots, or arranged to be worn by humans, such as wrist computers, finger computers, ring computers, eyeglass computers, belt clip computers, armband computers, shoe computers, clothing computers, and other wearable computers. For example, in various embodiments, a mobile computing device may be implemented as a smartphone capable of executing computer applications and voice and / or data communications. For illustrative purposes, some embodiments may be described with a mobile computing device implemented as a smartphone, although it may be understood that other embodiments may be implemented using other wireless mobile computing devices as well. The embodiments are not limited in this context.

[0175] As shown in FIG. 17 , device 1700 may include a housing having front side 1701 and back side 1702. Device 1700 includes display 1704, input / output (I / O) device 1706, color camera 1721, color camera 1722, and integrated antenna 1708. In some embodiments, color camera 1721 and color camera 1722 acquire planar images as described herein. In some embodiments, device 1700 does not include color cameras 1721 and 1722, and device 1700 acquires input image data (e.g., any input image data described herein) from another device. Device 1700 may also include navigation functionality 1712. I / O device 1706 may include any suitable I / O device for inputting information into a mobile computing device. Examples of I / O device 1706 may include an alphanumeric keyboard, a numeric keypad, a touchpad, input keys, buttons, switches, a microphone, a speaker, a voice recognition device, software, and the like. Information may also be entered into device 1700 through a microphone (not shown) or digitized by a voice recognition device. As shown, device 1700 may include color cameras 1721, 1722 and flash 1710 integrated into the backside 1702 (or elsewhere) of device 1700. In other examples, color cameras 1721, 1722 and flash 1710 may be integrated into the frontside 1701 of device 1700, or both front and back sets of cameras may be provided. Color cameras 1721, 1722 and flash 1710 may be components of a camera module for generating color image data with IR texture correction that may be output to display 1704 and / or processed into images or streaming video that is telecommunicated from device 1700, for example, via antenna 1708.

[0176] Exemplary embodiments may be implemented using hardware elements, software elements, or a combination of both. Examples of hardware elements may include a processor, a microprocessor, a circuit, a circuit element (e.g., a transistor, a resistor, a capacitor, an inductor, etc.), an integrated circuit, an application specific integrated circuit (ASIC), a programmable logic device (PLD), a digital signal processor (DSP), a field programmable gate array (FPGA), a logic gate, a register, a semiconductor device, a chip, a microchip, a chipset, etc. Examples of software may include a software component, a program, an application, a computer program, an application program, a system program, a machine program, an operating system software, a middleware, firmware, a software module, a routine, a subroutine, a function, a method, a procedure, a software interface, an application program interface (API), an instruction set, computing code, computer code, a code segment, a computer code segment, a word, a value, a symbol, or any combination thereof. The decision as to whether an embodiment is implemented using hardware and / or software elements may depend on any number of factors, such as desired computational speed, power level, heat tolerance, processing cycle budget, input data rate, output data rate, memory resources, data bus speed, and other design or performance constraints.

[0177] In some examples, one or more aspects of at least one embodiment may be implemented by representative instructions stored on a machine-readable medium that represent various logic within a processor and, when read by a machine, cause the machine to assemble logic that performs the techniques described herein. Such representations, known as "IP cores," may be stored on tangible machine-readable media and supplied to various customers or manufacturing facilities to be loaded into manufacturing machines or processors that actually produce the logic.

[0178] Figure 18 is a block diagram of an exemplary processor platform 1800 configured to execute the instructions of Figures 11-14 to implement the texture-based MIV encoder 500 of Figure 5. The processor platform 1800 may be, for example, a server, a personal computer, a workstation, a self-learning machine (e.g., a neural network), a mobile device (e.g., a mobile phone, a smartphone, a tablet such as an iPad®), a personal digital assistant (PDA), an Internet appliance, a DVD player, a CD player, a digital video recorder, a Blu-ray player, a gaming console, a personal video recorder, a set-top box, a headset or other wearable device, or any other type of computing device.

[0179] The processor platform 1800 of the illustrated example includes a processor 1812. The processor 1812 of the illustrated example is hardware. For example, the processor 1812 may be implemented with one or more integrated circuits, logic circuits, microprocessors, GPUs, DSPs, or controllers from any desired family or manufacturer. The hardware processor may be a semiconductor-based (e.g., silicon-based) device. In this example, the processor implements a view optimizer 502, an exemplary depth inferrer 504, an exemplary correspondence atlas builder 506, an exemplary correspondence labeler 508, an exemplary correspondence pruner 510, an exemplary mask aggregator 512, an exemplary correspondence patch packer 514, an exemplary atlas generator 516, an exemplary occupancy packer 524, an exemplary video encoder 534, and an exemplary video encoder 536.

[0180] The processor 1812 of the illustrated example includes a local memory 1813 (e.g., a cache). The processor 1812 of the illustrated example communicates with a main memory, including a volatile memory 1814 and a non-volatile memory 1816, via a bus 1818. The volatile memory 1814 may be implemented with synchronous dynamic random access memory (SDRAM), dynamic random access memory (DRAM), RAMBUS® dynamic random access memory (RDRAM®), and / or any other type of random access memory device. The non-volatile memory 1816 may be implemented with flash memory and / or any other desired type of memory device. Access to the main memory 1814, 1816 is controlled by a memory controller.

[0181] The processor platform 1800 of the illustrated example also includes an interface circuit 1820. The interface circuit 1820 may be implemented with any type of interface standard, such as an Ethernet interface, a Universal Serial Bus (USB), a Bluetooth interface, a Near Field Communication (NFC) interface, and / or a PCI Express interface.

[0182] In the illustrated example, one or more input devices 1822 are connected to the interface circuit 1820. The input devices 1822 allow a user to enter data and / or commands into the processor 1812. The input devices may be implemented by, for example, audio sensors, microphones, cameras (still or video), keyboards, buttons, mice, touchscreens, trackpads, trackballs, isopoints, and / or voice recognition systems.

[0183] One or more output devices 1824 are also connected to the interface circuit 1820 of the illustrated example. The output device(s) 1824 may be implemented with, for example, a display device (e.g., a light-emitting diode (LED), an organic light-emitting diode (OLED), a liquid crystal display (LCD), a cathode ray tube display (CRT), an in-place switching (IPS) display, a touch screen, etc.), a tactile output device, a printer, and / or a speaker. Accordingly, the interface circuit 1820 of the illustrated example typically includes a graphics driver card, a graphics driver chip, and / or a graphics driver processor.

[0184] The interface circuit 1820 of the illustrated example also includes communications devices such as transmitters, receivers, transceivers, modems, residential gateways, wireless access points, and / or network interfaces to facilitate data exchange with external machines (e.g., any type of computing device) over a network 1826. Communications may be via, for example, an Ethernet connection, a digital subscriber line (DSL) connection, a telephone line connection, a coaxial cable system, a satellite system, a line-of-sight wireless system, a cellular telephone system, etc.

[0185] The processor platform 1800 of the illustrated example also includes one or more mass storage devices 1828 for storing software and / or data. Examples of such mass storage devices 1828 include floppy disk drives, hard drive disks, compact disk drives, Blu-ray disk drives, redundant array of independent disks (RAID) systems, and digital versatile disk (DVD) drives.

[0186] The machine-executable instructions 1832 of Figures 11-14 may be stored on the mass storage device 1828, the volatile memory 1814, the non-volatile memory 1816, and / or a removable non-transitory computer-readable storage medium such as a CD or DVD.

[0187] Figure 19 is a block diagram of an exemplary processor platform 1900 configured to execute the instructions of Figure 15 to implement the texture-based MIV decoder 700 of Figure 7. The processor platform 1900 may be, for example, a server, a personal computer, a workstation, a self-learning machine (e.g., a neural network), a mobile device (e.g., a mobile phone, a smartphone, a tablet such as an iPad®), a personal digital assistant (PDA), an Internet appliance, a DVD player, a CD player, a digital video recorder, a Blu-ray player, a gaming console, a personal video recorder, a set-top box, a headset or other wearable device, or any other type of computing device.

[0188] The processor platform 1900 of the illustrated example includes a processor 1912. The processor 1912 of the illustrated example is hardware. For example, the processor 1912 may be implemented with one or more integrated circuits, logic circuits, microprocessors, GPUs, DSPs, or controllers from any desired family or manufacturer. The hardware processor may be a semiconductor-based (e.g., silicon-based) device. In this example, the processor implements a video decoder 708, an example video decoder 712, an example dedicated unpacking unit 714, an example MIV decoder and parser unit 718, an example block-patch map decoder 720, an example cull unit 728, an example renderer unit 730, an example controller 732, an example depth inference unit 734, an example compositor unit 736, and an example inpaint unit 738.

[0189] The processor 1912 of the illustrated example includes local memory 1913 (e.g., cache). The processor 1912 of the illustrated example communicates with main memory, including volatile memory 1914 and nonvolatile memory 1916, via a bus 1918. The volatile memory 1914 may be implemented with synchronous dynamic random access memory (SDRAM), dynamic random access memory (DRAM), RAMBUS® dynamic random access memory (RDRAM®), and / or any other type of random access memory device. The nonvolatile memory 1916 may be implemented with flash memory and / or any other desired type of memory device. Access to the main memory 1914, 1916 is controlled by a memory controller.

[0190] The processor platform 1900 of the illustrated example also includes an interface circuit 1920. The interface circuit 1920 may be implemented with any type of interface standard, such as an Ethernet interface, a Universal Serial Bus (USB), a Bluetooth interface, a Near Field Communication (NFC) interface, and / or a PCI Express interface.

[0191] In the illustrated example, one or more input devices 1922 are connected to the interface circuit 1920. The input devices 1922 allow a user to enter data and / or commands into the processor 1912. The input devices may be implemented by, for example, audio sensors, microphones, cameras (still or video), keyboards, buttons, mice, touchscreens, trackpads, trackballs, isopoints, and / or voice recognition systems.

[0192] One or more output devices 1924 are also connected to the interface circuit 1920 of the illustrated example. The output device(s) 1924 may be implemented with, for example, a display device (e.g., a light-emitting diode (LED), an organic light-emitting diode (OLED), a liquid crystal display (LCD), a cathode ray tube display (CRT), an in-place switching (IPS) display, a touch screen, etc.), a tactile output device, a printer, and / or a speaker. Accordingly, the interface circuit 1920 of the illustrated example typically includes a graphics driver card, a graphics driver chip, and / or a graphics driver processor.

[0193] The interface circuitry 1920 of the illustrated example also includes communications devices such as transmitters, receivers, transceivers, modems, residential gateways, wireless access points, and / or network interfaces to facilitate data exchange with external machines (e.g., any type of computing device) over a network 1926. Communications may be via, for example, an Ethernet connection, a digital subscriber line (DSL) connection, a telephone line connection, a coaxial cable system, a satellite system, a line-of-sight wireless system, a cellular telephone system, etc.

[0194] The processor platform 1900 of the illustrated example also includes one or more mass storage devices 1928 for storing software and / or data. Examples of such mass storage devices 1928 include floppy disk drives, hard drive disks, compact disk drives, Blu-ray disk drives, redundant array of independent disks (RAID) systems, and digital versatile disk (DVD) drives.

[0195] The machine-executable instructions 1932 of FIG. 15 may be stored in mass storage device 1928, in volatile memory 1914, in non-volatile memory 1916, and / or on a removable non-transitory computer-readable storage medium such as a CD or DVD.

[0196] A block diagram illustrating an example software distribution platform 2005 for distributing software, such as the example computer-readable instructions 1832 of FIG. 18 and / or the example computer-readable instructions 1932 of FIG. 19, to third parties is shown in FIG. 20. The example software distribution platform 2005 may be implemented by any computer server, data facility, cloud service, etc. capable of storing and transmitting software to other computing devices. The third parties may be customers of the entity that owns and / or operates the software distribution platform. For example, the entity that owns and / or operates the software distribution platform may be a developer, distributor, and / or licensor of software, such as the example computer-readable instructions 1832 of FIG. 18 and / or the example computer-readable instructions 1932 of FIG. 19. The third parties may be consumers, users, retailers, OEMs, etc. that purchase and / or license the software for use and / or resale and / or sublicense. In the illustrated example, the software distribution platform 2005 includes one or more servers and one or more storage devices. The storage device stores computer-readable instructions 1832 and / or 1932, which may correspond to the example computer-readable instructions 1832 of FIG. 18 and / or the example computer-readable instructions 1932 of FIG. 19, as described above. One or more servers of the example software distribution platform 2005 communicate with the Internet and / or a network 2010, which may correspond to any one or more of either the example networks 1826, 1926 described above. In some examples, the one or more servers respond to requests to send software to a requester as part of a commercial transaction. Payment for distribution, sale, and / or licensing of the software may be processed by the one or more servers of the software distribution platform and / or via a third-party payment entity.The server allows purchasers and / or licensors to download the computer-readable instructions 1832 and / or 1932 from the software distribution platform 2005. For example, software that may correspond to the example computer-readable instructions 1832 of FIG. 18 may be downloaded to the example processor platform 1800 that executes the computer-readable instructions 1832 to implement the texture-based MIV encoder 500 of FIG. 5. For example, software that may correspond to the example computer-readable instructions 1932 of FIG. 19 may be downloaded to the example processor platform 1900 that executes the computer-readable instructions 1932 to implement the texture-based MIV decoder 700 of FIG. 7. In some examples, one or more servers of the software distribution platform 2005 periodically provide, transmit, and / or force update software (e.g., the example computer-readable instructions 1832 of FIG. 18 and / or the example computer-readable instructions 1932 of FIG. 19) to ensure that improvements, patches, updates, etc. are distributed and applied to the software on end-user devices.

[0197] From the foregoing, it can be seen that exemplary methods, apparatus, and products are disclosed that enable texture-based immersive video coding. The disclosed methods, apparatus, and products improve the efficiency of using a computing device by determining and allocating correspondence patches in a correspondence list. For example, a texture-based encoder stores a list of correspondence patches in an atlas. The texture-based encoder identifies correspondence patches to infer depth information from different texture information between views. The texture-based encoder requires less bandwidth to transmit the encoded bitstream because the texture-based encoder does not transmit any depth atlas. The texture-based decoder requires relatively less bandwidth to identify correspondence patches of interest due to the list of correspondence patches included in the atlas from the texture-based encoder. The disclosed methods, apparatus, and products are therefore directed to one or more improvements in computer functionality.

[0198] Disclosed herein are exemplary methods, apparatus, systems, and articles of manufacture for texture-based immersive video coding, further examples and combinations of which include:

[0199] Example 1 is a video encoder, a correspondence labeling unit for (i) identifying a first intrinsic pixel and a first corresponding pixel included in a plurality of pixels of a first view, and (ii) identifying a second intrinsic pixel and a second corresponding pixel included in a plurality of pixels of a second view, wherein the first plurality of corresponding pixels have respective correspondences with the second plurality of corresponding pixels, and a first one of the first corresponding pixels and a second one of the second corresponding pixels are classified as having at least one of similar texture information or different texture information; a correspondence patch packing unit for (i) comparing adjacent pixels in the first view; and (ii) identifying a first patch of unique pixels and a second patch of corresponding pixels based on the comparison of the adjacent pixels and the correspondences, the second patches being tagged with a correspondence list identifying correspondence patches in the second view; an atlas generator configured to generate at least one atlas for inclusion in encoded video data, the at least one atlas including the first patch of unique pixels and the second patch of corresponding pixels, and the encoded video data not including a depth map; The video encoder includes:

[0200] Example 2 includes the video encoder of Example 1, further comprising a correspondence pruning unit for generating a first pruning mask for the first view, the first pruning mask being based on texture information from the first unique pixels and the first corresponding pixels.

[0201] Example 3 includes the video encoder of example 2, wherein the first corresponding pixels do not include corresponding pixels included in a previously pruned view.

[0202] Example 4 includes the video encoder of any one of Examples 2 or 3, wherein the correspondence pruning unit generates a second pruning mask for the second view, the second pruning mask being based on texture information from the second intrinsic pixels and the second corresponding pixels.

[0203] Example 5 includes the video encoder of example 4, wherein the second corresponding pixels include corresponding pixels included in the first corresponding pixels of the first view.

[0204] Example 6 includes the video encoder of any one of Examples 4 or 5, wherein the correspondence pruning unit generates a third pruning mask for a third view, the third pruning mask being based on texture information from third intrinsic pixels and third corresponding pixels, the third corresponding pixels including corresponding pixels included in the first corresponding pixel of the first view and the second corresponding pixel of the second view.

[0205] Example 7 includes the video encoder of Example 6, wherein the correspondence pruning unit generates the first pruning mask, the second pruning mask, and the third pruning mask to include a first value for identifying a first plurality of pixels, a second plurality of pixels, and a third plurality of pixels to be retained, and a second value for identifying a fourth plurality of pixels to be pruned, wherein the fourth plurality of pixels includes the first corresponding pixel of the first view and the third corresponding pixel included in the second corresponding pixel of the second view.

[0206] Example 8 includes the video encoder of example 7, further comprising a mask aggregator for aggregating the first pruning mask, the second pruning mask, and the third pruning mask.

[0207] Example 9 includes the video encoder of Example 1, wherein the correspondence labeling unit identifies the first unique pixel as one of the plurality of pixels of the first view that does not project to a location of one of the plurality of pixels of the second view when reprojected onto the second view.

[0208] Example 10 includes the video encoder of Example 1, wherein the correspondence labeling unit identifies the first corresponding pixels as being ones of the plurality of pixels of the first view that, when reprojected onto the second view, project to locations of respective ones of the second corresponding pixels of the second view.

[0209] Example 11 includes the video encoder of Example 10, wherein the correspondence labeling unit identifies the first one of the first corresponding pixels and the second one of the second corresponding pixels as similar when a difference in texture content between the first view and the second view satisfies a threshold.

[0210] Example 12 includes the video encoder of example 1, wherein the at least one atlas includes at least one of an attribute map or an occupancy map.

[0211] Example 13 includes the video encoder of example 1, wherein the input view includes an attribute map.

[0212] Example 14 includes the video encoder of example 1, wherein the encoded video data includes a flag that identifies the at least one atlas as not including a depth map.

[0213] Example 15 is at least one non-transitory computer-readable medium comprising instructions that, when executed, cause one or more processors to at least: identifying a first unique pixel and a first corresponding pixel included in a plurality of pixels of a first view; identifying a second unique pixel and a second corresponding pixel included in a plurality of pixels of a second view, wherein the first plurality of corresponding pixels have a respective correspondence with the second plurality of corresponding pixels, and wherein a first one of the first corresponding pixels and a second one of the second corresponding pixels are classified as having at least one of similar texture information or different texture information; comparing adjacent pixels in the first view; identifying a first patch of unique pixels and a second patch of corresponding pixels based on the comparison of the neighboring pixels and the correspondences, the second patch being tagged with a correspondence list identifying correspondence patches in the second view; and generating at least one atlas for inclusion in the encoded video data, the at least one atlas including the first patch of unique pixels and the second patch of corresponding pixels, and the encoded video data not including a depth map; The method includes at least one non-transitory computer-readable medium that causes the

[0214] Example 16 includes at least one non-transitory computer-readable medium of Example 15, wherein the instructions cause the one or more processors to generate a first pruning mask for the first view, the first pruning mask being based on texture information from the first intrinsic pixels and the first corresponding pixels.

[0215] Example 17 includes at least one non-transitory computer-readable medium of Example 16, wherein the first corresponding pixels do not include corresponding pixels included in a previously pruned view.

[0216] Example 18 includes at least one non-transitory computer-readable medium of any one of Example 16 or Example 17, wherein the instructions cause the one or more processors to generate a second pruning mask for the second view, the second pruning mask being based on texture information from the second intrinsic pixels and the second corresponding pixels.

[0217] Example 19 includes at least one non-transitory computer-readable medium of Example 18, wherein the second corresponding pixels include corresponding pixels included in the first corresponding pixels of the first view.

[0218] Example 20 includes at least one non-transitory computer-readable medium of any one of Example 18 or Example 19, wherein the instructions cause the one or more processors to generate a third pruning mask for a third view, the third pruning mask based on texture information from the third unique pixels and third corresponding pixels, the third corresponding pixels including corresponding pixels included in the first corresponding pixel of the first view and the second corresponding pixel of the second view.

[0219] Example 21 includes at least one non-transitory computer-readable medium of Example 20, wherein the instructions cause the one or more processors to: generate the first pruning mask, the second pruning mask, and the third pruning mask to include a first value for identifying a first plurality of pixels, a second plurality of pixels, and a third plurality of pixels to be maintained, and a second value for identifying a fourth plurality of pixels to be pruned, wherein the fourth plurality of pixels includes the first corresponding pixel of the first view and the third corresponding pixel included in a second corresponding pixel of the second view.

[0220] Example 22 includes at least one non-transitory computer-readable medium of Example 21, wherein the instructions cause the one or more processors to aggregate the first pruning mask, the second pruning mask, and the third pruning mask.

[0221] Example 23 includes at least one non-transitory computer-readable medium of Example 15, wherein the instructions cause the one or more processors to identify the first unique pixel as one of the plurality of pixels of the first view that does not project to a location of one of the plurality of pixels of the second view when reprojected onto the second view.

[0222] Example 24 includes at least one non-transitory computer-readable medium of Example 15, wherein the instructions cause the one or more processors to identify the first corresponding pixel as a plurality of the plurality of pixels of the first view that, when reprojected onto the second view, project to a location of a respective one of the second corresponding pixels of the second view.

[0223] Example 25 includes at least one non-transitory computer-readable medium of Example 24, wherein the instructions cause the one or more processors to identify a first one of the first corresponding pixels and a second one of the second corresponding pixels as similar when a difference between respective texture content in the first view and the second view meets a threshold.

[0224] Example 26 includes at least one non-transitory computer-readable medium of Example 15, wherein the at least one atlas includes at least one of an attribute map or an occupancy map.

[0225] Example 27 includes at least one non-transitory computer-readable medium of Example 15, wherein the encoded video data includes a flag that identifies that the at least one atlas does not include a depth map.

[0226] Example 28 is a method comprising: identifying a first unique pixel and a first corresponding pixel included in a plurality of pixels of a first view; identifying a second unique pixel and a second corresponding pixel included in a plurality of pixels of a second view, wherein the first plurality of corresponding pixels have a respective correspondence with the second plurality of corresponding pixels, and a first one of the first corresponding pixels and a second one of the second corresponding pixels are classified as having at least one of similar texture information or different texture information; comparing adjacent pixels in the first view; identifying a first patch of unique pixels and a second patch of corresponding pixels based on the comparison of the neighboring pixels and the correspondences, the second patches being tagged with a correspondence list identifying correspondence patches in the second view; generating at least one atlas for inclusion in the encoded video data, the at least one atlas including the first patch of unique pixels and the second patch of corresponding pixels, the encoded video data not including a depth map; The method includes:

[0227] Example 29 includes the method of Example 28, further comprising generating a first pruning mask for the first view, the first pruning mask being based on texture information from the first unique pixels and the first corresponding pixels.

[0228] Example 30 includes the method of example 29, wherein the first corresponding pixels do not include corresponding pixels included in a previously pruned view.

[0229] Example 31 includes the method of any one of Example 29 or Example 30, further comprising: generating a second pruning mask for the second view, the second pruning mask being based on texture information from the second intrinsic pixels and the second corresponding pixels.

[0230] Example 32 includes the method of example 31, in which the second corresponding pixel includes a corresponding pixel included in the first corresponding pixel of the first view.

[0231] Example 33 includes the method of any one of Examples 31 or 32, further comprising: generating a third pruning mask for a third view, the third pruning mask based on texture information from third unique pixels and third corresponding pixels, the third corresponding pixels including corresponding pixels included in the first corresponding pixels of the first view and the second corresponding pixels of the second view.

[0232] Example 34 includes the method of Example 33, further including generating the first pruning mask, the second pruning mask, and the third pruning mask to include a first value for identifying a first plurality of pixels, a second plurality of pixels, and a third plurality of pixels to be kept and a second value for identifying a fourth plurality of pixels to be pruned, wherein the fourth plurality of pixels includes the third corresponding pixels included in the first corresponding pixel of the first view and the second corresponding pixel of the second view.

[0233] Example 35 includes the method of example 34, further comprising aggregating the first pruning mask, the second pruning mask, and the third pruning mask.

[0234] Example 36 includes the method of Example 28, further comprising identifying the first unique pixel as one of the plurality of pixels of the first view that does not project to a location of one of the plurality of pixels of the second view when reprojected onto the second view.

[0235] Example 37 includes the method of Example 28, further comprising identifying the first corresponding pixels as those of the plurality of pixels of the first view that, when reprojected onto the second view, project to locations of respective ones of the second corresponding pixels of the second view.

[0236] Example 38 includes the method of Example 37, further comprising identifying the first one of the first corresponding pixels and the second one of the second corresponding pixels as similar when a difference between respective texture contents in the first view and the second view meets a threshold.

[0237] Example 39 includes the method of example 28, wherein the at least one atlas includes at least one of an attribute map or an occupancy map.

[0238] Example 40 includes the method of example 28, wherein the encoded video data includes a flag to identify that the at least one atlas does not include a depth map.

[0239] Example 41 is a video decoder, 1. A bitstream decoder comprising: identifying a flag in the encoded video data, the flag identifying a first atlas and a second atlas that do not include a depth map; and decoding the encoded video data to obtain (i) first metadata of a first view and (ii) second metadata of a second view, wherein patches from the first view and the second view belong to the same atlas or are distributed across different atlases, the first view including a first patch in the first atlas, and the second view including a corresponding second patch in the first atlas or a second atlas tagged with a correspondence list; a bitstream decoder that performs a rendering unit, determining depth information based on a comparison of pixels in the first patch with corresponding pixels in the corresponding second patch; and synthesizing a target view from the determined depth information; A rendering part that performs The video decoder includes:

[0240] Example 42 includes the video decoder of Example 41, wherein the first metadata includes a first map that maps blocks of pixels in the first view to the first patches included in the first view, and the second metadata includes a second map that maps blocks of pixels in the second view to the corresponding second patches included in the second view.

[0241] Example 43 includes the video decoder of Example 42, wherein the rendering unit reconstructs the corresponding second patch based on the second map.

[0242] Example 44 includes the video decoder of example 41, wherein the renderer identifies the corresponding pixels in the corresponding second patch based on the correspondence list from the encoded video data.

[0243] Example 45 includes a video decoder according to any one of Examples 41 or 44, wherein the rendering unit determines the depth information based on a first one of the first patches from the first view and each of the corresponding second patches from the second view that are determined to correspond to the first one of the first patches, and each of the corresponding second patches has texture information that is different from the first one of the first patches or similar to the first one of the first patches.

[0244] Example 46 is at least one non-transitory computer-readable medium comprising instructions that, when executed, cause one or more processors to at least: identifying a flag in the encoded video data, the flag identifying a first atlas and a second atlas that do not include a depth map; decoding the encoded video data to obtain (i) first metadata of a first view and (ii) second metadata of a second view, wherein patches from the first view and the second view belong to the same atlas or are distributed across different atlases, the first view including a first patch in the first atlas, and the second view including a corresponding second patch in the first atlas or a second atlas tagged with a correspondence list; determining depth information based on a comparison of pixels in the first patch with corresponding second pixels in the corresponding second patch; and synthesizing a target view from the determined depth information; The method includes at least one non-transitory computer-readable medium that causes the

[0245] Example 47 includes at least one non-transitory computer-readable medium of Example 46, wherein the first metadata includes a first map that maps blocks of pixels in the first view to the first patches included in the first view, and the second metadata includes a second map that maps blocks of pixels in the second view to the corresponding second patches included in the second view.

[0246] Example 48 includes at least one non-transitory computer-readable medium as described in Example 47, wherein the instructions cause the one or more processors to reconstruct the corresponding second patch based on the second map.

[0247] Example 49 includes at least one non-transitory computer-readable medium of Example 46, wherein the instructions cause the one or more processors to identify the corresponding second pixels in the corresponding second patches based on the correspondence list from the encoded video data.

[0248] Example 50 includes at least one non-transitory computer-readable medium of any one of Examples 46 or 49, wherein the instructions cause the one or more processors to determine the depth information based on a first one of the first patches from the first view and each of the corresponding second patches from the second view determined to correspond to the first one of the first patches, wherein each of the corresponding second patches has texture information that is different from the first one of the first patches or similar to the first one of the first patches.

[0249] Example 51 is a method comprising: identifying a flag in the encoded video data, the flag identifying a first atlas and a second atlas that do not include a depth map; decoding the encoded video data to obtain (i) first metadata of a first view and (ii) second metadata of a second view, wherein patches from the first view and the second view belong to the same atlas or are distributed across different atlases, the first view including a first patch in the first atlas and the second view including a corresponding second patch in the first atlas or a second atlas tagged with a correspondence list; determining depth information based on a comparison of pixels in the first patch with corresponding second pixels in the corresponding second patch; synthesizing a target view from the determined depth information; The method includes:

[0250] Example 52 includes a method as in Example 51, wherein the first metadata includes a first map that maps blocks of pixels in the first view to the first patches included in the first view, and the second metadata includes a second map that maps blocks of pixels in the second view to the corresponding second patches included in the second view.

[0251] Example 53 includes the method of example 52, further comprising reconstructing the corresponding second patch based on the second map.

[0252] Example 54 includes the method of example 51, further comprising identifying the corresponding second pixels in the corresponding second patches based on the correspondence list from the encoded video data.

[0253] Example 55 includes the method of any one of Examples 51 or 54, further comprising: determining the depth information based on a first one of the first patches from the first view and each of the corresponding second patches from the second view determined to correspond to the first one of the first patches, wherein each of the corresponding second patches has texture information different from the first one of the first patches or texture information similar to the first one of the first patches.

[0254] Although certain exemplary methods, apparatus, and articles of manufacture are disclosed herein, the scope of coverage of this patent is not limited thereto. Rather, this patent covers all methods, apparatus, and articles of manufacture fairly falling within the scope of the claims of this patent.

[0255] The following claims are hereby incorporated by reference, with each claim standing on its own as a separate embodiment of this disclosure. (Item 1) 1. A video encoder comprising: a correspondence labeling unit for (i) identifying a first intrinsic pixel and a first corresponding pixel included in a plurality of pixels of a first view, and (ii) identifying a second intrinsic pixel and a second corresponding pixel included in a plurality of pixels of a second view, wherein the first plurality of corresponding pixels have respective correspondences with the second plurality of corresponding pixels, and a first one of the first corresponding pixels and a second one of the second corresponding pixels are classified as having at least one of similar texture information or different texture information; a correspondence patch packing unit for (i) comparing adjacent pixels in the first view; and (ii) identifying a first patch of unique pixels and a second patch of corresponding pixels based on the comparison of the adjacent pixels and the correspondences, the second patches being tagged with a correspondence list identifying correspondence patches in the second view; an atlas generator configured to generate at least one atlas for inclusion in encoded video data, the at least one atlas including the first patch of unique pixels and the second patch of corresponding pixels, and the encoded video data not including a depth map; A video encoder comprising: (Item 2) 2. The video encoder of claim 1, further comprising: a correspondence pruning unit for generating a first pruning mask for the first view, the first pruning mask being based on texture information from the first intrinsic pixels and the first corresponding pixels. (Item 3) 3. The video encoder of claim 2, wherein the first corresponding pixels do not include corresponding pixels included in a previously pruned view. (Item 4) 3. The video encoder of claim 2, wherein the correspondence pruning unit generates a second pruning mask for the second view, the second pruning mask being based on texture information from the second intrinsic pixels and the second corresponding pixels. (Item 5) Item 5. The video encoder of item 4, wherein the second corresponding pixels include corresponding pixels included in the first corresponding pixels of the first view. (Item 6) 5. The video encoder of claim 4, wherein the correspondence pruning unit generates a third pruning mask for a third view, the third pruning mask being based on texture information from third intrinsic pixels and third corresponding pixels, the third corresponding pixels including corresponding pixels included in the first corresponding pixels of the first view and second corresponding pixels of the second view. (Item 7) 7. The video encoder of claim 6, wherein the correspondence pruning unit generates the first pruning mask, the second pruning mask, and the third pruning mask to include a first value for identifying a first plurality of pixels, a second plurality of pixels, and a third plurality of pixels to be retained, and a second value for identifying a fourth plurality of pixels to be pruned, wherein the fourth plurality of pixels includes the first corresponding pixel in the first view and the third corresponding pixel included in the second corresponding pixel in the second view. (Item 8) Item 8. The video encoder of item 7, further comprising a mask aggregator for aggregating the first pruning mask, the second pruning mask, and the third pruning mask. (Item 9) 2. The video encoder of claim 1, wherein the correspondence labeling unit identifies the first unique pixels as those of the plurality of pixels of the first view that are not projected to locations of those of the plurality of pixels of the second view when reprojected onto the second view. (Item 10) 2. The video encoder of claim 1, wherein the correspondence labeling unit identifies the first corresponding pixels as being ones of the plurality of pixels of the first view that project to locations of respective ones of the second corresponding pixels of the second view when reprojected onto the second view. (Item 11) Item 11. The video encoder of item 10, wherein the correspondence labeling unit identifies the first one of the first corresponding pixels and the second one of the second corresponding pixels as similar when a difference in texture content between the first view and the second view satisfies a threshold. (Item 12) Item 1. The video encoder of item 1, wherein the at least one atlas includes at least one of an attribute map or an occupancy map. (Item 13) Item 1. The video encoder of item 1, wherein the input view includes an attribute map. (Item 14) Item 1. The video encoder of item 1, wherein the encoded video data includes a flag that identifies the at least one atlas as not including a depth map. (Item 15) At least one non-transitory computer-readable medium comprising instructions that, when executed, cause one or more processors to at least: identifying a first unique pixel and a first corresponding pixel included in a plurality of pixels of a first view; identifying a second unique pixel and a second corresponding pixel included in a plurality of pixels of a second view, wherein the first plurality of corresponding pixels have a respective correspondence with the second plurality of corresponding pixels, and wherein a first one of the first corresponding pixels and a second one of the second corresponding pixels are classified as having at least one of similar texture information or different texture information; comparing adjacent pixels in the first view; identifying a first patch of unique pixels and a second patch of corresponding pixels based on the comparison of the neighboring pixels and the correspondences, the second patch being tagged with a correspondence list identifying correspondence patches in the second view; and generating at least one atlas for inclusion in the encoded video data, the at least one atlas including the first patch of unique pixels and the second patch of corresponding pixels, and the encoded video data not including a depth map; At least one non-transitory computer-readable medium that causes (Item 16) Item 16. The at least one non-transitory computer-readable medium of item 15, wherein the instructions cause the one or more processors to generate a first pruning mask for the first view, the first pruning mask being based on texture information from the first intrinsic pixels and the first corresponding pixels. (Item 17) Item 17. At least one non-transitory computer-readable medium according to item 16, wherein the first corresponding pixels do not include corresponding pixels included in a previously pruned view. (Item 18) Item 17. The at least one non-transitory computer-readable medium of item 16, wherein the instructions cause the one or more processors to generate a second pruning mask for the second view, the second pruning mask being based on texture information from the second intrinsic pixels and the second corresponding pixels. (Item 19) Item 19. At least one non-transitory computer-readable medium according to item 18, wherein the second corresponding pixels include corresponding pixels included in the first corresponding pixels of the first view. (Item 20) Item 19. The at least one non-transitory computer-readable medium of item 18, wherein the instructions cause the one or more processors to generate a third pruning mask for a third view, the third pruning mask being based on texture information from the third unique pixels and third corresponding pixels, the third corresponding pixels including corresponding pixels included in the first corresponding pixels of the first view and second corresponding pixels of the second view. (Item 21) 21. The at least one non-transitory computer-readable medium of claim 20, wherein the instructions cause the one or more processors to: generate the first pruning mask, the second pruning mask, and the third pruning mask to include a first value for identifying a first plurality of pixels, a second plurality of pixels, and a third plurality of pixels to be maintained, and a second value for identifying a fourth plurality of pixels to be pruned, wherein the fourth plurality of pixels includes the first corresponding pixel of the first view and the third corresponding pixel included in the second corresponding pixel of the second view. (Item 22) 22. The at least one non-transitory computer-readable medium of claim 21, wherein the instructions cause the one or more processors to aggregate the first pruning mask, the second pruning mask, and the third pruning mask. (Item 23) Item 16. The at least one non-transitory computer-readable medium of item 15, wherein the instructions cause the one or more processors to identify the first unique pixel as one of the plurality of pixels of the first view that does not project to a location of one of the plurality of pixels of the second view when reprojected onto the second view. (Item 24) Item 16. The at least one non-transitory computer-readable medium of Item 15, wherein the instructions cause the one or more processors to identify the first corresponding pixel as a plurality of the plurality of pixels of the first view that, when reprojected onto the second view, project to a location of a respective one of the second corresponding pixels of the second view. (Item 25) 25. The at least one non-transitory computer-readable medium of claim 24, wherein the instructions cause the one or more processors to identify a first one of the first corresponding pixels and a second one of the second corresponding pixels as similar when a difference between respective texture contents in the first view and the second view meets a threshold. (Item 26) Item 16. The at least one non-transitory computer-readable medium of item 15, wherein the at least one atlas includes at least one of an attribute map or an occupancy map. (Item 27) Item 16. At least one non-transitory computer-readable medium according to item 15, wherein the encoded video data includes a flag that identifies that at least one atlas does not include a depth map. (Item 28) 1. A method comprising: identifying a first unique pixel and a first corresponding pixel included in a plurality of pixels of a first view; identifying a second unique pixel and a second corresponding pixel included in a plurality of pixels of a second view, wherein the first plurality of corresponding pixels have a respective correspondence with the second plurality of corresponding pixels, and a first one of the first corresponding pixels and a second one of the second corresponding pixels are classified as having at least one of similar texture information or different texture information; comparing adjacent pixels in the first view; identifying a first patch of unique pixels and a second patch of corresponding pixels based on the comparison of the neighboring pixels and the correspondences, the second patches being tagged with a correspondence list identifying correspondence patches in the second view; generating at least one atlas for inclusion in the encoded video data, the at least one atlas including the first patch of unique pixels and the second patch of corresponding pixels, the encoded video data not including a depth map; A method for providing the above. (Item 29) Item 29. The method of item 28, further comprising: generating a first pruning mask for the first view, the first pruning mask being based on texture information from the first intrinsic pixels and the first corresponding pixels. (Item 30) Item 30. The method of item 29, wherein the first corresponding pixels do not include corresponding pixels included in a previously pruned view. (Item 31) 30. The method of claim 29, further comprising: generating a second pruning mask for the second view, the second pruning mask being based on texture information from the second intrinsic pixels and the second corresponding pixels. (Item 32) Item 32. The method according to item 31, wherein the second corresponding pixels include corresponding pixels included in the first corresponding pixels of the first view. (Item 33) Item 32. The method of item 31, further comprising: generating a third pruning mask for a third view, the third pruning mask being based on texture information from third intrinsic pixels and third corresponding pixels, the third corresponding pixels including corresponding pixels included in the first corresponding pixels of the first view and the second corresponding pixels of the second view. (Item 34) Item 34. The method of item 33, further comprising generating the first pruning mask, the second pruning mask, and the third pruning mask to include a first value for identifying a first plurality of pixels, a second plurality of pixels, and a third plurality of pixels to be kept, and a second value for identifying a fourth plurality of pixels to be pruned, wherein the fourth plurality of pixels includes the third corresponding pixels included in the first corresponding pixel of the first view and the second corresponding pixel of the second view. (Item 35) Item 35. The method of item 34, further comprising aggregating the first pruning mask, the second pruning mask, and the third pruning mask. (Item 36) Item 29. The method of item 28, further comprising identifying the first unique pixel as a plurality of the plurality of pixels of the first view that, when reprojected onto the second view, do not project to a location of a plurality of the plurality of pixels of the second view. (Item 37) Item 29. The method of item 28, further comprising identifying a plurality of the plurality of pixels of the first view that, when reprojected onto the second view, project to the location of a respective one of the second corresponding pixels of the second view. (Item 38) Item 38. The method of item 37, further comprising identifying the first one of the first corresponding pixels and the second one of the second corresponding pixels as similar when a difference between the respective texture contents in the first view and the second view meets a threshold. (Item 39) 29. The method of claim 28, wherein the at least one atlas includes at least one of an attribute map or an occupancy map. (Item 40) 29. The method of claim 28, wherein the encoded video data includes a flag to identify that the at least one atlas does not include a depth map. (Item 41) 1. A video decoder comprising: 1. A bitstream decoder comprising: identifying a flag in the encoded video data, the flag identifying a first atlas and a second atlas that do not include a depth map; and decoding the encoded video data to obtain (i) first metadata of a first view and (ii) second metadata of a second view, wherein patches from the first view and the second view belong to the same atlas or are distributed across different atlases, the first view including a first patch in the first atlas, and the second view including a corresponding second patch in the first atlas or a second atlas tagged with a correspondence list; a bitstream decoder that performs a rendering unit, determining depth information based on a comparison of pixels in the first patch with corresponding pixels in the corresponding second patch; and synthesizing a target view from the determined depth information; A rendering part that performs A video decoder comprising: (Item 42) Item 42. The video decoder of item 41, wherein the first metadata includes a first map that maps blocks of pixels in the first view to the first patches included in the first view, and the second metadata includes a second map that maps blocks of pixels in the second view to the corresponding second patches included in the second view. (Item 43) Item 43. The video decoder of item 42, wherein the rendering unit reconstructs the corresponding second patch based on the second map. (Item 44) Item 42. The video decoder of item 41, wherein the rendering unit identifies the corresponding pixels in the corresponding second patch based on the correspondence list from the encoded video data. (Item 45) Item 42. The video decoder of item 41, wherein the rendering unit determines the depth information based on a first one of the first patches from the first view and each of the corresponding second patches from the second view that are determined to correspond to the first one of the first patches, and each of the corresponding second patches has texture information that is different from the first one of the first patches or texture information that is similar to the first one of the first patches. (Item 46) At least one non-transitory computer-readable medium comprising instructions that, when executed, cause one or more processors to at least: identifying a flag in the encoded video data, the flag identifying a first atlas and a second atlas that do not include a depth map; decoding the encoded video data to obtain (i) first metadata of a first view and (ii) second metadata of a second view, wherein patches from the first view and the second view belong to the same atlas or are distributed across different atlases, the first view including a first patch in the first atlas, and the second view including a corresponding second patch in the first atlas or a second atlas tagged with a correspondence list; determining depth information based on a comparison of pixels in the first patch with corresponding second pixels in the corresponding second patch; and synthesizing a target view from the determined depth information; At least one non-transitory computer-readable medium that causes (Item 47) Item 47. At least one non-transitory computer-readable medium according to Item 46, wherein the first metadata includes a first map that maps blocks of pixels in the first view to the first patches included in the first view, and the second metadata includes a second map that maps blocks of pixels in the second view to the corresponding second patches included in the second view. (Item 48) Item 48. The at least one non-transitory computer-readable medium of item 47, wherein the instructions cause the one or more processors to reconstruct the corresponding second patch based on the second map. (Item 49) Item 47. The at least one non-transitory computer-readable medium of item 46, wherein the instructions cause the one or more processors to identify the corresponding second pixels in the corresponding second patches based on the correspondence list from the encoded video data. (Item 50) Item 47. The at least one non-transitory computer-readable medium of Item 46, wherein the instructions cause the one or more processors to determine the depth information based on a first one of the first patches from the first view and each of the corresponding second patches from the second view determined to correspond to the first one of the first patches, wherein each of the corresponding second patches has texture information different from the first one of the first patches or texture information similar to the first one of the first patches. (Item 51) 1. A method comprising: identifying a flag in the encoded video data, the flag identifying a first atlas and a second atlas that do not include a depth map; decoding the encoded video data to obtain (i) first metadata of a first view and (ii) second metadata of a second view, wherein patches from the first view and the second view belong to the same atlas or are distributed across different atlases, the first view including a first patch in the first atlas and the second view including a corresponding second patch in the first atlas or a second atlas tagged with a correspondence list; determining depth information based on a comparison of pixels in the first patch with corresponding second pixels in the corresponding second patch; synthesizing a target view from the determined depth information; A method for providing the above. (Item 52) Item 52. The method of item 51, wherein the first metadata includes a first map that maps blocks of pixels in the first view to the first patches included in the first view, and the second metadata includes a second map that maps blocks of pixels in the second view to the corresponding second patches included in the second view. (Item 53) Item 53. The method of item 52, further comprising reconstructing the corresponding second patch based on the second map. (Item 54) Item 52. The method of item 51, further comprising identifying the corresponding second pixel in the corresponding second patch based on the correspondence list from the encoded video data. (Item 55) Item 52. The method of item 51, further comprising a step of determining the depth information based on a first one of the first patches from the first view and each of the corresponding second patches from the second view determined to correspond to the first one of the first patches, wherein each of the corresponding second patches has texture information different from the first one of the first patches or texture information similar to the first one of the first patches.

Claims

1. 1. A method comprising: identifying a first unique pixel and a first corresponding pixel included in a plurality of pixels of a first view; identifying a second unique pixel and a second corresponding pixel included in a plurality of pixels of a second view, wherein the first plurality of corresponding pixels have a respective correspondence with the second plurality of corresponding pixels, and a first one of the first corresponding pixels and a second one of the second corresponding pixels are classified as having at least one of similar texture information or different texture information; comparing adjacent pixels in the first view; identifying a first patch of unique pixels and a second patch of corresponding pixels based on the comparison of the adjacent pixels and the correspondences, the second patches being tagged with a correspondence list identifying correspondence patches in the second view; generating at least one atlas for inclusion in encoded video data, the at least one atlas including the first patch of unique pixels and the second patch of corresponding pixels, the encoded video data not including a depth map; A method for providing the above.

2. 2. The method of claim 1, further comprising: generating a first pruning mask for the first view, the first pruning mask being based on texture information from the first intrinsic pixels and the first corresponding pixels.

3. The method of claim 2 , wherein the first corresponding pixels do not include corresponding pixels included in a previously pruned view.

4. 4. The method of claim 2 or 3, further comprising: generating a second pruning mask for the second view, the second pruning mask being based on texture information from the second intrinsic pixels and the second corresponding pixels.

5. The method of claim 4 , wherein the second corresponding pixels include pixels that correspond to the first corresponding pixels of the first view.

6. 6. The method of claim 4, further comprising: generating a third pruning mask for a third view, the third pruning mask being based on texture information from third intrinsic pixels and third corresponding pixels, the third corresponding pixels including pixels corresponding to the first corresponding pixels of the first view and pixels corresponding to the second corresponding pixels of the second view.

7. 7. The method of claim 6, further comprising: generating the first pruning mask, the second pruning mask, and the third pruning mask to include a first value for identifying a first plurality of pixels, a second plurality of pixels, and a third plurality of pixels to be kept and a second value for identifying a fourth plurality of pixels to be pruned, wherein the fourth plurality of pixels includes the third corresponding pixels included in the first corresponding pixel of the first view and the second corresponding pixel of the second view.

8. The method of claim 7 , further comprising aggregating the first pruning mask, the second pruning mask, and the third pruning mask.

9. 9. The method of claim 1, further comprising identifying the first unique pixel as being one of the plurality of pixels of the first view that does not project to a location of one of the plurality of pixels of the second view when reprojected onto the second view.

10. 9. The method of claim 1, further comprising identifying a plurality of the plurality of pixels of the first view that, when reprojected onto the second view, project to a location of a respective one of the second corresponding pixels of the second view.

11. 11. The method of claim 10, further comprising identifying the first one of the first corresponding pixels and the second one of the second corresponding pixels as similar when a difference between respective texture contents in the first view and the second view meets a threshold.

12. The method of claim 1 , wherein the at least one atlas comprises at least one of an attribute map or an occupancy map.

13. Apparatus comprising means for carrying out the method according to any one of claims 1 to 12.

14. A computer program comprising machine-readable instructions that, when executed by an apparatus, cause the apparatus to perform a method according to any one of claims 1 to 12.

15. At least one memory; machine-readable instructions in the memory; and a processor circuit for at least one of instantiating and executing the machine-readable instructions to perform the method of any one of claims 1 to 12; A system comprising:

16. 1. A method comprising: identifying a flag in the encoded video data, the flag identifying a first atlas and a second atlas that do not include a depth map; decoding the encoded video data to obtain (i) first metadata for a first view and (ii) second metadata for a second view, where patches from the first view and the second view belong to the same atlas or are distributed across different atlases, the first view including a first patch in a first atlas and the second view including a corresponding second patch in the first atlas or a second atlas tagged with a correspondence list; determining depth information based on a comparison of pixels in the first patch and corresponding second pixels in the corresponding second patch; synthesizing a target view from the determined depth information; A method for providing the above.

17. 17. The method of claim 16, wherein the first metadata includes a first map that maps blocks of pixels in the first view to the first patches included in the first view, and the second metadata includes a second map that maps blocks of pixels in the second view to the corresponding second patches included in the second view.

18. The method of claim 17 , further comprising reconstructing the corresponding second patch based on the second map.

19. 19. The method of claim 16, further comprising identifying the corresponding second pixels in the corresponding second patches based on the correspondence list from the encoded video data.

20. 19. The method of claim 16, further comprising: determining the depth information based on a first one of the first patches from the first view and each of the corresponding second patches from the second view determined to correspond to the first one of the first patches, wherein each of the corresponding second patches has texture information different from the first one of the first patches or texture information similar to the first one of the first patches.

21. Apparatus comprising means for carrying out the method according to any one of claims 16 to 20.

22. A computer program comprising machine-readable instructions that, when executed by an apparatus, cause the apparatus to perform a method according to any one of claims 16 to 20.

23. At least one memory; machine-readable instructions in the memory; and A processor circuit for at least one of instantiating or executing the machine-readable instructions to perform the method of any one of claims 16 to 20. A system comprising:

24. 23. A computer readable storage medium storing a computer program according to claim 14 or 22.

Images