Multidimensional Video Transcoding

Abstract

To provide an apparatus, method and system for transcoding multi-dimensional video.SOLUTION: A system 100 for transcoding multi-dimensional video includes: a multi-dimensional video decoder to decode a first bitstream of multi-dimensional video; a transcoder to transcode the decoded first bitstream to generate transcoded multi-dimensional data; and a multi-dimensional video encoder to generate a second bitstream based on the transcoded multi-dimensional data.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] Video transcoding can be used to convert a first bitstream into a second bitstream using a different resolution or bitrate. For example, the second bitstream may be streamed using a network using bandwidth that may not support the first bitstream, or may be played using a device that supports the lower resolution of the second bitstream. [Brief description of the drawings]

[0002] [Figure 1] 1 is a block diagram illustrating an example system for transcoding multidimensional video. [Diagram 2] 1 is a block diagram illustrating another example system for transcoding multidimensional video. [Diagram 3] 1 is a flow chart illustrating a method for transcoding multidimensional video. [Figure 4] 1 is a block diagram illustrating an example computing device capable of transcoding multidimensional video. [Diagram 5] 1 is a block diagram illustrating a computer-readable medium having code stored thereon for transcoding multidimensional video.

[0003] The same numbers are used throughout this disclosure and the figures to refer to similar components and features, with 100 series numbers referring to features depicted in Figure 1, 200 series numbers referring to features depicted in Figure 2, and so on for the other figures. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0004] Multidimensional video as used herein refers to video that includes three or more dimensions. For example, volumetric video may include three dimensions, light field video may include four to eight dimensions, and immersive video may have six degrees of freedom for a total of four to five dimensions. Multidimensional video may include stereoscopic video, immersive video, and light field video. Stereoscopic video includes video recordings in which a view in every direction is recorded simultaneously. For example, stereoscopic video may be captured using an omnidirectional camera or a collection of cameras. During playback on a normal flat display, the viewer controls the viewing direction as in a panoramic image.

[0005] Multi-dimensional video, including immersive video, point-cloud based video, and light-field video, may be captured and generated using a variety of techniques. For example, immersive video codecs allow multiple views of video to be captured, encoded, and rendered by a decoder into a particular viewport. A viewport, as used herein, is a projection of an omnidirectional or three-dimensional image or texture onto a planar surface in a field of view of video suitable for display and viewing by a user from a particular viewing position and orientation. For example, a viewport may be a two-dimensional view at a particular angle, which is displayed by a head mounted display (HMD) worn by a user to provide a number of different viewing positions and a number of different viewing directions to modify the particular view, e.g., the view can be rotated, translated, etc. based on the movement of the user wearing the HMD device.

[0006] Multiple bright fields describe the incident light from all directions at a given sample point and may be used in post-processing to create effects such as light field depth and allowing the user to move their head slightly.

[0007] Various standards are being developed to encode various types of multi-dimensional video, such as stereoscopic video or point cloud compressed video. As one example, the Metadata for Immersive Video (MIV) MPEG-1 Part 12 standard, a working draft 3 published in October 2019, by MPEG, encodes texture and depth video for each of multiple source views at a specific position and orientation using an existing HEVC video codec. As another example, the Video-based Point Cloud Compression (V-PCC) Algorithm by MPEG, released on November 22, 2019, encodes a point cloud sequence by projecting the point cloud onto multiple planar surfaces and then encoding the planar projections of texture and depth using an existing video codec standard, such as AVC or HEVC.

[0008] For efficient distribution of video, video transcoding using standard two-dimensional video codecs such as AVC / H.264 or HEVC is used. For example, a contribution bitstream may be provided at a very high bitrate and very high video quality, and transcoded to a lower bitrate and quality used to distribute the video over a constrained bandwidth network. On the other hand, existing two-dimensional video transcoders may not be directly applicable to multidimensional video codec standards. For example, existing video codec standards use the same format for the input to a video encoder and the output of a video decoder, which is an uncompressed (raw) video sequence consisting of a sequence of pictures files. These video codec standards define normative output conformance points, and the output of any conforming video decoder must exactly match the output of the specified decoder for the normative output conformance points. On the other hand, the normatively defined output of various multidimensional codec standards is not necessarily in the same format as the input to a multidimensional encoder.

[0009] As one example, the intended output of the reference renderer in the MIV standard is a texture perspective viewport that is selected based on the viewer's position and orientation, which is generated using the output of an immersive media decoder. The MIV standard allows the viewer to move dynamically with six degrees of freedom (6DoF), positional (x,y,z), and orientation (yaw, pitch, roll), within a limited range. For example, the range may be limited to that supported by a head-mounted display or a 2D monitor using multiple positional inputs. On the other hand, the multiple conformance point outputs of the output of the MIV reference renderer are not in the same format as the test model for immersive video (TMIV) encoder input, and therefore cannot be directly applied to a transcoding system.

[0010] This disclosure generally relates to techniques for transcoding multidimensional video. In particular, the techniques described herein include an apparatus, method, and system for transcoding multidimensional video. One example apparatus includes a multidimensional video decoder for decoding a first bitstream of multidimensional video. The apparatus also includes a transcoder for transcoding the decoded first bitstream to generate transcoded volumetric data. The apparatus further includes a multidimensional video encoder for generating a second bitstream based on the transcoded multidimensional data. In some examples, the transcoder is capable of rendering multiple views based on texture atlases, depth atlases, atlas patch occupancy map sequence, atlas parameters, and camera parameters of the decoded first bitstream. In various examples, the apparatus includes an atlas resampler for resampling a texture map and a depth map of the decoded first bitstream.

[0011] In this way, the techniques described herein allow for a reduction in the bit rate of the multidimensional video bit stream. For example, the bit rate of the bit stream may be reduced to accommodate the available network bandwidth used to deliver the bit stream. In addition, the techniques described herein allow for the adaptation of the characteristics of the transcoded content to accommodate the physical constraints of the consuming display devices. For example, the characteristics of the bit stream may be adapted when the display devices support only a limited depth, field of view, or angle, spatial, or depth resolution of the field of view or viewing range. Furthermore, the techniques may allow for a reduction in the complexity of the transcoding operation. For example, a transcoder may be used to resample and select a limited number of views, thereby encoding the second bit stream to support a wide range of consuming devices or to accommodate network conditions such as limited bandwidth.

[0012] 1 is a block diagram illustrating an example system 100 for transcoding multidimensional video. The example system 100 may be implemented by a computing device 400 of FIG. 4 using the method 300 of FIG. 3.

[0013] The example system 100 includes a multidimensional video decoder 102. The system 100 also includes a transcoding renderer 104, which is communicatively coupled to the multidimensional video decoder 102. The system 100 further includes a multidimensional video encoder 106, which is communicatively coupled to the transcoding renderer 104. The multidimensional video decoder 102 includes a texture video decoder 108, a depth video decoder 110, and a metadata parser 112. The multidimensional video decoder 102 also includes an atlas patch occupancy map generator 114, which is communicatively coupled to the depth video decoder 110 and the metadata parser 112.

[0014] The multidimensional video encoder 106 includes a view selector 116. The multidimensional video encoder 106 also includes an atlas constructor 118. The multidimensional video encoder 106 further includes a video texture encoder 120, a depth video encoder 122, and a metadata encoder 124.

[0015] In the example of FIG. 1, the system 100 may receive a first bitstream 126 and transcode the first bitstream 126 to generate a second bitstream 128. For example, the first bitstream 126 may be a high-quality multidimensional video bitstream, such as an immersive video bitstream. The second bitstream 128 may be a compressed or lower-quality immersive video bitstream. For example, the multidimensional video decoder 102 may receive the first bitstream 126 and generate corresponding texture component 130 and depth component 132 diagrams. The texture video decoder 108 may generate multiple diagrams of texture component 130 from the first bitstream 126. For example, the texture component 130 may include multiple decoded texture pictures. In various examples, the texture video decoder 108 can generate the texture chart 130 by detecting the coded video sequence (CVS) for each texture of a layer pair in the first bitstream 126 and decoding the texture into a decoded texture image file.

[0016] In various examples, the depth video decoder 110 can generate multiple map depth components 132 from the first bitstream 126. For example, the depth video decoder 110 can receive a CVS for each depth layer of a layer pair in the bitstream 126. The depth video decoder 110 can generate a decoded depth image file based on each depth layer. In this manner, the depth map 132 may include a decoded depth image file. In some examples, the depth video decoder 110 can generate the depth map 132 from associated portions of the compressed bitstream. In various examples, the depth map 132 and the texture map 130 may be output as a sequence of decoded image file pairs of synchronized decoded texture image files and decoded depth image files. Each pair of texture map 130 and synchronized depth map 132 is referred to herein as a map.

[0017] The graph partition occupancy map generator 114 may generate a graph partition occupancy map 134 based on the depth map 132 from the depth video decoder 110 and the graph parameters 136 from the metadata parser 112. For example, the graph partition occupancy map 134 may include a sequence of maps associated with the depth map 132 and the texture map 130. Each of the multiple maps may indicate, for each sample location in the texture and depth videos, whether the sample location is occupied or not. For example, a sample location may be occupied if a valid value exists for the same pixel location in the map that will later be used by a renderer.

[0018] In various examples, the metadata parser 112 can parse the metadata in the bitstream 126 to detect and output view parameters 136 and camera parameters 138. For example, the view parameters 136 may include patches' parameters that specify how each patch extracted from a given view is packed into a given view, as well as grouping parameters and depth occupancy coding parameters, and specify the entity to which each patch belongs. For example, the entity may be an object. The camera parameters may include extrinsic and intrinsic parameters for each view streamed, along with the quality of depth maps and other parameters for the associated entity. For example, the extrinsic parameters may include position and rotation parameters. In various examples, the intrinsic parameters may include field of view, focal length, and projection type.

[0019] The transcoding renderer 104 can receive the texture map 130, the depth map 132, the map partition occupancy map 134, the map parameters 136, the camera parameters 138, and the view selection information 140 and generate the texture view 142, the depth view 144, and the camera parameters 146. For example, the texture view 142 and the depth view 144 can be texture and depth video sequences for multiple views. In various examples, each view can correspond to a particular 6DoF position and direction. The texture view 142 and the depth view 144 can be generated based on the specifications of the client device using the bitstream 128. In some of the examples, the transcoding renderer 104 can render the output texture view 142 and the depth view 144 to any desired resolution, and therefore it is not possible to avoid using a separate resampler. In various examples, it is possible to reduce the complexity of the rendering operation when the two-dimensional video encoder of the transcoder encodes at a lower resolution. For example, the bitstream 128 output from the transcoder 104 may be consumed by multiple user side devices, some of which may have limited computing capabilities. As one example, when using reduced resolution, the decoder complexity may also be reduced in the user side devices. In some of the examples, the transcoding renderer 104 may also adjust the position and orientation, field of view, or depth of view of the output texture view 142 and depth view 144.For example, if the system 100 is provided with information about a particular display used by a client viewer or a client multi-dimensional video decoder capability level, the resolution, field of view, field of view depth value, and region of interest of the output view may be adapted to match the resolution, field of view, field of view depth value, and region of interest of the client display or decoder. In various examples, the target display device may support views of 1024×720 pixels (720P), 1920×1080 pixels (1080P), or 3840×2160 pixels (4K), among other possible resolutions. The multi-view texture and depth video sequences are used as inputs to the multi-dimensional video encoder 106.

[0020] In some examples, the transcoding renderer 104 may also receive input view selection information 148. For example, the view selection information input 148 may be used to input different criteria that the transcoding renderer 104 may use for selecting the characteristics of the view along with the selection of the view positions and view directions that the renderer outputs. The selected view may correspond to some or all of the camera parameters 138 included in the bitstream 126, or may be different. The camera parameters 146 for the selected view are output by the transcoding renderer 104 and input to the multidimensional video encoder 106. In some examples, the camera parameters 146 of the selected view may be copied from the camera parameters 138 in response to detecting that the camera parameters are already present in the bitstream 126. If the camera parameters are not already present in the bitstream, the transcoding renderer 104 may modify or set new camera parameters 146 when the new view is rendered. For example, the transcoding renderer 104 can filter some views or define a new set of new views based on different bandwidth and display requirements, and can generate camera parameters 146 accordingly.

[0021] In some of the examples, the transcoding renderer 104 may reduce the number of views in the bitstream 128 in response to detecting that the transcoder's target output bitrate for the bitstream 128 is significantly lower than the input bitrate of the bitstream 126. For example, the target output bitrate may differ from the input bitrate of the bitstream 126 by more than a threshold difference. In various examples, the bit depth of the views may also be reduced from the bit depth used in the original bitstream 126. Rendering multidimensional video may be a computationally complex process. Reducing the number of rendered output views may significantly reduce the implementation complexity of the system 100. Reducing the number or resolution of the transcoded multidimensional video may also reduce the complexity required by the client's immersive video decoder, player, and display device.

[0022] In various examples, the view selection information 148 may be determined by a multidimensional video service (not shown). For example, the multidimensional video service may choose to limit the range of views that the bitstream supports. As one example, the multidimensional video service may choose to limit the range of views in the bitstream 128 from the original 180 degree field of view in the bitstream 126 to a 90 degree field of view. In various examples, the range of views may be limited for creative purposes as selected by the content creator. In some of the examples, the transcoding may be targeted to a particular viewer. For example, the view selection information 148 may be used to limit the range of views that are close to the current view position of that viewer. In this way, the view selection information 148 may also enable lower latency adaptation of the bitstream for constrained networks or constrained decoders or constrained client capabilities. In some of the examples, the view selection information 148 may be used to reduce the number of views to improve rendering fidelity when processing natural content. For example, the overall rendering fidelity may be improved by utilizing more coherent data from closer viewpoints.

[0023] In some examples, the transcoding renderer 104 can choose to perform more accurate and more complex rendering in a region of interest of the multidimensional video. For example, the transcoding renderer 104 can perform more accurate and higher complexity rendering for objects that are near or closer to the viewer's current view position. As one example, the transcoding renderer 104 can render objects more accurately using smaller depth values ​​and using higher complexity.

[0024] The multidimensional video encoder 106 may receive the texture view 142, the depth view 144, and the camera parameters 146 and generate the second bitstream 128. In some examples, a view selector 116 of the multidimensional video encoder 106 may be used to select a view for the bitstream 128. For example, the view selector 116 may specify a particular view based on a target display device. In some examples, the multidimensional video encoder 106 may use the view selector 116 to provide guidance to the transcoding renderer 104 on which views to render based on the availability of the views. For example, the multidimensional video encoder 106 may instruct the transcoding renderer 104 to avoid rendering views that the view selector 116 of the immersive video encoder 106 selects to exclude. In various examples, the transcoding renderer 104 may render a larger set of views for a subset of the image files of an encoded video sequence for use by the encoder's view selector 116, and then use the view selector information 148 to reduce the number of views rendered for other image files in the same encoded video sequence. For example, the transcoding renderer 104 may render a larger set of views for I-frames, and use the view selector information 148 for I-frames to reduce the number of views rendered for other image files in the same encoded video sequence.

[0025] In various examples, output bitstream 128 may be sent to another device and decoded into a viewport that may be displayed on a display device, such as an HMD device. For example, output bitstream 128 may have a lower bitrate than bitstream 126 and therefore may be transmitted over a network using fewer resources. In some examples, the resolution of output bitstream 128 may match the supported resolution of the display device.

[0026] The diagram of Figure 1 is not intended to indicate that the exemplary system 100 is configured to include all of the components shown in Figure 1. Rather, it is possible to implement the exemplary system 100 using fewer components or additional components not shown in Figure 1 (e.g., additional bitstreams, transcoding renderers, decoders, encoders, etc.) For example, an immersive video encoder may output multiple bitstreams using multiple different sizes and compressions.

[0027] 2 is a block diagram illustrating another example system for transcoding multidimensional video. The example system 200 can be implemented by a computing device 400 of FIG. 4 using the method 300 of FIG.

[0028] The exemplary system 200 includes several similarly numbered elements of Figure 1. In addition, the system 200 includes an immersive video encoder 202. The system 200 includes resamplers 204A and 204B that are communicatively coupled to the immersive video decoder 102 and the immersive video encoder 202. The immersive video encoder 202 of Figure 2 includes a metadata streamer 206. For example, the metadata streamer 206 may encode metadata or may pass through the metadata from the multidimensional video decoder 102.

[0029] In the example of Fig. 2, by avoiding the need to render the views altogether, it is possible in some specific cases to simplify the transcoding process. In this example, the same views, charts, and partitions that are present in bitstream 126 are used in bitstream 128. For example, all of the chart component video sequences, including texture chart 130 and depth chart 132, are simply decoded and re-encoded using video encoding, selective resampling by resamplers 204A and 204B, and decoding. For example, resamplers 204A and 204B can perform upscaling or downscaling of the texture and depth components, respectively. In various examples, metadata, including chart parameters 136 and camera parameters 138, are passed through and multiplexed into the bitstream by metadata streamer 206. In some examples, if the atlas patch occupancy map sequence 134 is coded separately rather than embedded in the depth component 132, the layers of the bitstream 126 used to code the atlas patch occupancy map sequence 134 video sequence may be passed to the immersive video encoder 202 without re-encoding, or occupancy map decoded pictures may be output from the immersive video decoder 102 and re-encoded in the immersive video encoder 202. For example, the MPEG Immersive Video standard embeds the occupancy map in the depth, while the V-PCC standard streams multiple occupancy maps in separate channels.

[0030] As one example, the V-PCC draft standard specifies a decoding process whose outputs are a set of decoded video streams corresponding to the decoded partition data information, as well as information about the point cloud sequence and the texture, geometry, occupancy map, and any attributes. Although not required for standard conformance, these outputs can be used to perform the specified 3D point cloud reconstruction process. On the other hand, when transcoding is applied to a V-PCC bitstream, it is not necessary to generate a point cloud as part of the decoding process. Rather, it is possible to input the decoded texture video 130 and depth planar projection videos 132 to the V-PCC encoder 202, bypassing the stage that forms projections from the point cloud. In the V-PCC draft specification, the occupancy map video sequence 134 is encoded to indicate, for each sample location in the texture video and the depth video, whether the sample location is occupied or not. In various examples, the V-PCC encoder used in the transcoder system 200 can be modified to utilize as an input the occupancy map output 134 from the V-PCC decoder.

[0031] The diagram of Figure 2 is not intended to indicate that the example system 200 is configured to include all of the components shown in Figure 2. Rather, it is possible to implement the example system 200 using fewer components or additional components not shown in Figure 2 (e.g., additional bitstreams, resamplers, decoders, encoders, etc.). For example, the system 200 may not include resamplers 204A and 204B in some implementations.

[0032] 3 is a flow chart illustrating a method for transcoding multidimensional video. An exemplary method 300 may be implemented by the systems 100 or 200 of FIGs. 1 and 2, the computing device 400 of FIG. 4, or the computer-readable medium 500 of FIG. 5.

[0033] At block 302, a processor receives a first bitstream of multi-dimensional video. For example, the multi-dimensional video may be a compressed immersive video, a point-cloud based video, or a light field video.

[0034] At block 304, the processor decodes the first bitstream. For example, the processor may decode the first bitstream into texture atlases, depth atlases, atlas patch occupancy map sequences, atlas parameters, and camera parameters, or any combination thereof.

[0035] At block 306, the processor transcodes the decoded first bitstream to generate transcoded multi-dimensional data. In some of the examples, the processor can render multiple views based on the texture map, the depth map, the map partition occupancy map sequence, the map parameters, and the camera parameters. The processor can then resample the multiple views. For example, the processor can upscale or downscale the multiple views. In various examples, the multiple rendered views may be a subset of the views in the first bitstream. For example, the processor can render the multiple views based on received view selector information. In some of the examples, the processor can reduce the range or number of views from the first bitstream included in the second bitstream. For example, the processor can limit the range of views to views in a particular field of view associated with the display device. In various examples, the processor may render an identified region of interest in the first bitstream with greater precision or complexity than other regions in the second bitstream. For example, the region of interest may be an object closer to the viewer. In some of the examples, the processor may render a larger set of views for a subset of the image files in the second bitstream, generate view selector information based on the subset of image files, and reduce the number of views rendered for other images in the second bitstream.For example, the processor may render a larger set of views for an I-frame in the second bitstream while reducing the number of views for other frames in the second bitstream. In various examples, the views may be used to generate the updated chart. In some of the examples, the processor may generate the resampled texture chart and the resampled depth chart directly from the decoded texture chart and the decoded depth chart. For example, the processor may generate the resampled texture chart and the resampled depth chart from the decoded texture chart and the decoded depth chart without rendering any views in response to detecting that the second bitstream is configured to include the same view with a lower resolution or a lower bitrate. In various examples, the processor may also generate metadata included in the second bitstream. In some of the examples, the processor may copy metadata from the first bitstream. In other examples, the processor may generate metadata included in the second bitstream based on the transcoded multidimensional data. For example, metadata may be generated based on the multiple rendered views used to generate an updated diagram, and if a renderer in the transcoder discards any views or synthesizes new views, the metadata may be updated accordingly.

[0036] At block 308, the processor generates a second bitstream based on the transcoded multi-dimensional data. For example, the second bitstream can then be transmitted to a second processor for display. As one example, the second processor may reside in a head-mounted display device. In various examples, the second bitstream may have a lower bitrate than the first bitstream. In this manner, the second bitstream may be transmitted to the second processor more efficiently. For example, the second bitstream may be transmitted to the second processor via any suitable network connection.

[0037] This process flow diagram is not intended to imply that the blocks of the example method 300 are configured to be performed in any particular order, or that all of the blocks are configured to be included in every instance. Moreover, any number of additional blocks not shown may be included in the example method 300, depending on the details of its particular implementation.

[0038] Referring now to FIG. 4, a block diagram illustrating an exemplary computing device capable of transcoding multidimensional video is shown. The computing device 400 may be, for example, a laptop computer, a desktop computer, a tablet computer, a mobile device, or a wearable device, among others. In some of the examples, the computing device 400 may be a cloud computing node. The computing device 400 may include a central processing unit (CPU) 402 configured to execute stored instructions, and may include a memory device 404 storing instructions executable by the CPU 402. The CPU 402 may be coupled to the memory device 404 by a bus 406. Additionally, the CPU 402 may be a single-core processor, a multi-core processor, a computing cluster, or any number of other configurations. Furthermore, the computing device 400 may include more than one CPU 402. In some of the examples, the CPU 402 may be a system-on-chip (SoC) having a multi-core processor architecture. In some of the examples, the CPU 402 may be a specialized digital signal processor (DSP) used for image processing. The memory device 404 may include random access memory (RAM), read only memory (ROM), flash memory, or any other suitable memory system. For example, the memory device 404 may include dynamic random access memory (DRAM).

[0039] The memory device 404 may include random access memory (RAM), read only memory (ROM), flash memory, or any other suitable memory system. For example, the memory device 404 may include dynamic random access memory (DRAM).

[0040] Computing device 400 may also include a graphics processing unit (GPU) 408. As shown, CPU 402 may be coupled to GPU 408 via bus 406. GPU 408 may be configured to perform any number of graphics operations within computing device 400. For example, GPU 408 may be configured to render or manipulate graphics images, graphics frames, videos, or the like that are displayed to a user of computing device 400.

[0041] The memory device 404 may include a random access memory (RAM), a read only memory (ROM), a flash memory, or any other suitable memory system. For example, the memory device 404 may include a dynamic random access memory (DRAM). The memory device 404 may include a device driver 410 configured to execute instructions for training a plurality of convolutional neural networks to perform sequence-independent processing. The device driver 410 may be software, an application program, application code, or the like.

[0042] The CPU 402 may also be connected via the bus 406 to an input / output (I / O) device interface 412 configured to connect the computing device 400 to one or more I / O devices 414. The I / O devices 414 may include, for example, a keyboard and a pointing device, which may include a touchpad or a touch screen, among others. The I / O devices 414 may be built-in components of the computing device 400 or may be devices externally connected to the computing device 400. In some of the examples, the memory 404 may be communicatively coupled to the I / O devices 414 by direct memory access (DMA).

[0043] CPU 402 may also be linked via bus 406 to a display interface 416, which is configured to connect computing device 400 to a display device 418. Display device 418 may include a display screen that is an integral component of computing device 400. Display device 418 may also include a computer monitor, television, or projector, among others, either internal to computing device 400 or externally connected.

[0044] The computing device 400 also includes a storage device 420, which may be a physical memory such as a hard drive, an optical drive, a thumb drive, an array of drives, a solid-state drive, or any combination thereof. The storage device 420 may also include a remote storage drive.

[0045] The computing device 400 may also include a network interface controller (NIC) 422. The NIC 422 may be configured to connect the computing device 400 to a network 424 via the bus 406. The network 424 may be a wide area network (WAN), a local area network (LAN), or the Internet, among others. In some of the examples, the computing device may communicate with other devices by wireless technology. For example, the computing device may communicate with other devices by a wireless local area network connection. In some of the examples, the computing device may connect to and communicate with other devices by Bluetooth or similar technology.

[0046] The computing device 400 further includes a camera 426. For example, the camera 426 may include one or more imaging sensors. In some of the examples, the camera 426 may include a processor that generates multi-dimensional video, such as an immersive video, a point cloud-based video, or a bright field video.

[0047] The computing device 400 further includes a multidimensional video transcoder 428. The multidimensional video transcoder 428 may be used to transcode the multidimensional video. For example, the multidimensional video may be an immersive video, a point cloud video, or a light field video. The multidimensional video transcoder 428 may include a multidimensional video decoder 430, a transcoder 432, and a multidimensional video encoder 434. In some of the examples, each of the components 430-434 of the immersive video transcoder 428 may be a microcontroller, an embedded processor, or a software module. The multidimensional video decoder 430 may decode a first bitstream of the multidimensional video. For example, the decoded first bitstream may include a texture map, a depth map, a map partition occupancy map sequence, map parameters, and camera parameters, or any combination thereof. The transcoder 432 may transcode the decoded first bitstream to generate transcoded multidimensional data. For example, the transcoded multidimensional data may include a texture view, a depth view, and a set of camera parameters. The set of camera parameters of the transcoded multidimensional data may represent source views or views that the transcoder renders based on bandwidth specifications or display specifications. In some of the examples, the transcoder 432 may render views based on the decoded first bitstream. For example, the transcoder 432 may generate multiple views based on the transcoded multidimensional data, resample the generated multiple views, and generate a resampled texture map and a resampled depth map based on the multiple resampled views.In some of the examples, the transcoder 432 can reduce the range or number of views from the first bitstream included in the second bitstream. For example, the transcoder 432 can render a subset of views based on the generated view selection information from the multidimensional video encoder 434. In various examples, the transcoder 432 can directly resample the texture and depth charts of the decoded first bitstream. The multidimensional video encoder 434 can generate the second bitstream based on the transcoded multidimensional data. In some of the examples, the multidimensional video encoder 434 can generate the view selection information. For example, the multidimensional video encoder 434 can include a view selector (not shown) that generates the view selection information. In various examples, the multidimensional video encoder 434 can include a chart constructor that generates a chart based on a subset of views from the transcoder and a set of camera parameters. For example, the multidimensional video encoder 434 can generate the second bitstream based on the charts. In some examples, the multidimensional video encoder 434 may include a metadata streamer that copies metadata from the decoded first bitstream and multiplexes the copied metadata into the second bitstream. In this manner, in various examples, if the multidimensional video decoder 430 or transcoder 432 outputs multiple views, the multidimensional video encoder 434 may encode the multiple views into multiple views. In some examples, if the multidimensional video decoder 430 or transcoder 432 outputs multiple modified views, the multidimensional video encoder 434 in the immersive video transcoder 428 may be bypassed.

[0048] The block diagram of FIG. 4 is not intended to indicate that the computing device 400 is configured to include all of the components shown in FIG. 4. Rather, the computing device 400 may include fewer components or additional components not shown in FIG. 4, such as additional buffers and additional processors. For example, in various embodiments, the functionality of the transcoder 432 may be included in the multidimensional video decoder 430 or the multidimensional video encoder 434. The computing device 400 may include any number of additional components not shown in FIG. 4, depending on the details of the particular implementation. Furthermore, any of the functions of the multidimensional video decoder 430, the transcoder 432, and the multidimensional video encoder 434 may be implemented, in part or in whole, by hardware and / or by the processor 402. For example, the functions may be implemented using an application specific integrated circuit, by logic implemented by the processor 402, or by any other device. Additionally, any of those functions of the CPU 402 may be implemented, in part or in whole, by hardware and / or by a processor. For example, the functionality of the multidimensional video transcoder 428 may be implemented using an application specific integrated circuit, by logic implemented by a processor, by logic implemented by a specialized graphics processing unit such as GPU 408, or by any other device.

[0049] 5 is a block diagram illustrating a computer-readable medium 500 having stored thereon code for transcoding multidimensional video. The computer-readable medium 500 may be accessed by a processor 502 via a computer bus 504. Additionally, the computer-readable medium 500 may include code configured to instruct the processor 502 to perform the methods described herein. In some of the embodiments, the computer-readable medium 500 may be a non-transitory computer-readable medium. In some of the examples, the computer-readable medium 500 may be a storage medium.

[0050] Various software components described herein may be stored on one or more computer-readable media 500 illustrated in FIG. 5. For example, a volumetric video decoder module 506 may be configured to decode a first bitstream of volumetric video. A transcoder module 508 may be configured to transcode the decoded first bitstream to generate transcoded stereoscopic data. In some of the examples, the transcoder module 508 may be configured to render views based on the decoded first bitstream. In some of the examples, the transcoder module 508 may be configured to render a subset of views based on view selection information. For example, the transcoder module 508 may receive the view selection information from the stereoscopic video encoder module 510. In various examples, the transcoder module 508 may be configured to reduce a range or number of views from the first bitstream included in the second bitstream. In some of the examples, the transcoder module 508 may be configured to copy metadata used in the second bitstream from the first bitstream. In various examples, the transcoder module 508 may be configured to generate metadata for the second bitstream based on the transcoded multi-dimensional data. The stereoscopic video encoder module 510 may be configured to generate the second bitstream based on the transcoded stereoscopic data. For example, the stereoscopic video encoder module 510 may be configured to generate multiple views based on a subset of views and a set of camera parameters. The stereoscopic video encoder module 510 may be configured to generate the second bitstream based on the multiple views.In some of the examples, the stereoscopic video encoder module 510 may be configured to generate view selection information. In some of the examples, the stereoscopic video encoder module 510 may be configured to render an identified region of interest in the first bitstream with greater precision or greater complexity than other regions in the second bitstream. In various examples, the stereoscopic video encoder module 510 may be configured to render a larger set of views for a subset of image files in the second bitstream, generate view selector information based on the subset of image files, and reduce the number of views rendered for other image files in the second bitstream. In some of the examples, the stereoscopic video encoder module 510 may be configured to send the second bitstream to a second processor for display.

[0051] The block diagram of Figure 5 is not intended to indicate that computer-readable medium 500 is configured to include all of the components shown in Figure 5. Moreover, computer-readable medium 500 may include any number of additional components not shown in Figure 5, depending on the details of a particular implementation.

[0052] example

[0053] Example 1 is an apparatus for transcoding multidimensional video. The apparatus includes a multidimensional video decoder for decoding a first bitstream of the multidimensional video. The apparatus also includes a transcoder for transcoding the decoded first bitstream to generate transcoded multidimensional data. The apparatus further includes a multidimensional video encoder for generating a second bitstream based on the transcoded multidimensional data.

[0054] Example 2 includes the apparatus of example 1 with or without the optional feature. In this example, the decoded first bitstream includes a texture map, a depth map, a map partition occupancy map sequence, map parameters, and camera parameters. The transcoder is configured to render a view based on the decoded first bitstream.

[0055] Example 3 includes the apparatus of any one of examples 1 and 2, with or without the selective feature. In this example, the multidimensional video encoder includes a view selector that generates view selection information. The transcoder is configured to render a subset of the views based on the view selection information.

[0056] Example 4 includes the apparatus of any one of Examples 1 to 3, with or without the optional feature. In this example, the multidimensional video encoder includes a view constructor that generates a view based on a subset of views from the transcoder and a set of camera parameters. The multidimensional video encoder is configured to generate the second bitstream based on the view.

[0057] Example 5 includes the apparatus of any one of examples 1 to 4 with or without the optional feature, in which the transcoded multi-dimensional data includes a texture view, a depth view, and a set of camera parameters.

[0058] Example 6 includes the apparatus of any one of examples 1 to 5 with or without the optional feature, in which the transcoder is configured to reduce a range or number of views from the first bitstream that are included in the second bitstream.

[0059] Example 7 includes the apparatus of any one of Examples 1 to 6 with or without the optional feature, in which the transcoder is configured to directly resample the texture and depth maps of the decoded first bitstream.

[0060] Example 8 includes the apparatus of any one of Examples 1 to 7 with or without the optional feature, in which the transcoder is configured to generate a view based on the transcoded multidimensional data, resample the generated view, and generate a resampled texture map and a resampled depth map based on the resampled view.

[0061] Example 9 includes the apparatus of any one of Examples 1 to 8, with or without the optional feature. In this example, the multidimensional video encoder includes a metadata streamer that copies metadata from the decoded first bitstream and multiplexes the copied metadata into the second bitstream.

[0062] Example 10 includes the apparatus of any one of examples 1 to 9, with or without the optional feature. In this example, the multidimensional video decoder or multidimensional video encoder includes a transcoder.

[0063] Example 11 is a method for transcoding multidimensional video. The method includes receiving, by a processor, a first bitstream of the multidimensional video. The method also includes decoding, by the processor, the first bitstream. The method further includes transcoding, by the processor, the decoded first bitstream to generate transcoded multidimensional data, and generating, by the processor, a second bitstream based on the transcoded multidimensional data.

[0064] Example 12 includes the method of example 11 with or without the optional feature. In this example, decoding the first bitstream includes generating a texture map, a depth map, a map segment occupancy map sequence, map parameters, and camera parameters.

[0065] Example 13 includes the method of any one of Examples 11-12 with or without the optional feature, in which the method includes resampling, by a processor, a texture map and a depth map of the decoded first bitstream to generate a resampled texture map and a resampled depth map.

[0066] Example 14 includes the method of any one of Examples 11-13 with or without the selective feature, in which the method includes rendering, by a processor, a view based on the texture map, the depth map, the map section occupancy map sequence, the map parameters, and the camera parameters.

[0067] Example 15 includes the method of any one of Examples 11 to 14, with or without the selective feature, in which the method includes reducing a range or number of views from the first bitstream that are included in the second bitstream.

[0068] Example 16 includes the method of any one of Examples 11-15, with or without the selective feature, including rendering the identified region of interest in the first bitstream with greater precision or greater complexity than other regions in the second bitstream.

[0069] Example 17 includes the method of any one of Examples 11-16 with or without the selective feature, in which the method includes rendering a larger set of views for a subset of image files in the second bitstream, generating view selector information based on the subset of image files, and reducing the number of views rendered for other image files in the second bitstream.

[0070] Example 18 includes the method of any one of Examples 11-17, with or without the optional feature, in which the method includes copying metadata from the first bitstream that is used in the second bitstream.

[0071] Example 19 includes the method of any one of Examples 11 to 18, with or without the optional feature, in which the method includes generating metadata for the second bitstream based on the transcoded multidimensional data.

[0072] Example 20 includes the method of any one of Examples 11-19, with or without the optional feature. In this example, the method includes transmitting the second bitstream to a second processor for display.

[0073] Example 21 is at least one computer-readable medium storing instructions for transcoding multidimensional video, the instructions directing a processor to decode a first bitstream of the multidimensional video. The computer-readable medium includes instructions directing the processor to transcode the decoded first bitstream to generate transcoded multidimensional data. The computer-readable medium includes instructions directing the processor to generate a second bitstream based on the transcoded multidimensional data.

[0074] Example 22 includes the computer-readable medium of example 21 with or without the optional feature. In this example, the computer-readable medium includes instructions for rendering a view based on the decoded first bitstream.

[0075] Example 23 includes the computer-readable medium of any one of Examples 21-22, with or without the selective feature. In this example, the computer-readable medium includes instructions for generating view selection information. The processor is configured to render a subset of views based on the view selection information.

[0076] Example 24 includes the computer-readable medium of any one of Examples 21-23, with or without the optional feature, in which the computer-readable medium includes instructions for generating a diagram based on the subset of views and the set of camera parameters, and the processor is configured to generate a second bitstream based on the diagram.

[0077] Example 25 includes the computer-readable medium of any one of Examples 21-24 with or without the optional feature, in which the computer-readable medium includes instructions for reducing a range or number of views from the first bitstream that are included in the second bitstream.

[0078] Example 26 includes the computer-readable medium of any one of Examples 21-25, with or without the optional feature, in which the computer-readable medium includes instructions for rendering identified regions of interest in the first bitstream with greater precision or greater complexity than other regions in the second bitstream.

[0079] Example 27 includes the computer-readable medium of any one of Examples 21-26 with or without the optional feature, where the computer-readable medium includes instructions for rendering a larger set of views for a subset of the image files in the second bitstream, generating view selector information based on the subset of the image files, and reducing the number of views rendered for other image files in the second bitstream.

[0080] Example 28 includes the computer-readable medium of any one of Examples 21-27 with or without the optional feature. In this example, the computer-readable medium includes instructions for copying metadata from the first bitstream for use in the second bitstream.

[0081] Example 29 includes the computer-readable medium of any one of Examples 21-28 with or without the optional feature, in which the computer-readable medium includes instructions for generating metadata for the second bitstream based on the transcoded multidimensional data.

[0082] Example 30 includes the computer-readable medium of any one of Examples 21-29 with or without the optional feature, in which the computer-readable medium includes instructions for transmitting the second bitstream to a second processor for display.

[0083] Example 31 is a system for transcoding multidimensional video. The system includes a multidimensional video decoder for decoding a first bitstream of the multidimensional video. The system also includes a transcoder for transcoding the decoded first bitstream to generate transcoded multidimensional data. The system further includes a multidimensional video encoder for generating a second bitstream based on the transcoded multidimensional data.

[0084] Example 32 includes the system of example 31 with or without the optional feature. In this example, the decoded first bitstream includes a texture map, a depth map, a map partition occupancy map sequence, map parameters, and camera parameters. The transcoder is configured to render a view based on the decoded first bitstream.

[0085] Example 33 includes the system of any one of Examples 31 to 32, with or without the selective feature. In this example, the multidimensional video encoder includes a view selector that generates view selection information. The transcoder is configured to render a subset of the views based on the view selection information.

[0086] Example 34 includes the system of any one of Examples 31 to 33, with or without the optional feature. In this example, the multidimensional video encoder includes a view constructor that generates a view based on a subset of views from the transcoder and a set of camera parameters. The multidimensional video encoder is configured to generate the second bitstream based on the view.

[0087] Example 35 includes the system of any one of Examples 31 to 34 with or without the optional feature, in which the transcoded multi-dimensional data includes a texture view, a depth view, and a set of camera parameters.

[0088] Example 36 includes the system of any one of Examples 31 to 35 with or without the selective feature, in which the transcoder is configured to reduce a range or number of views from the first bitstream that are included in the second bitstream.

[0089] Example 37 includes the system of any one of Examples 31 to 36 with or without the optional feature, in which the transcoder is configured to directly resample the texture and depth maps of the decoded first bitstream.

[0090] Example 38 includes the system of any one of Examples 31 to 37 with or without the selective feature, where the transcoder is configured to generate a view based on the transcoded multidimensional data, resample the generated view, and generate a resampled texture map and a resampled depth map based on the resampled view.

[0091] Example 39 includes the system of any one of Examples 31 to 38 with or without the optional feature, in which the multidimensional video encoder includes a metadata streamer that copies metadata from the decoded first bitstream and multiplexes the copied metadata into the second bitstream.

[0092] Example 40 includes the system of any one of examples 31 to 39 with or without the optional feature. In this example, the multidimensional video decoder or multidimensional video encoder includes a transcoder.

[0093] Example 41 is a system for transcoding multidimensional video. The system includes means for decoding a first bitstream of the multidimensional video. The system also includes means for transcoding the decoded first bitstream to generate transcoded multidimensional data. The system further includes means for generating a second bitstream based on the transcoded multidimensional data.

[0094] Example 42 includes the system of example 41 with or without the optional feature. In this example, the decoded first bitstream includes a texture map, a depth map, a map partition occupancy map sequence, map parameters, and camera parameters. The means for transcoding the decoded first bitstream is configured to render a view based on the decoded first bitstream.

[0095] Example 43 includes the system of any one of Examples 41 to 42, with or without the selective feature. In this example, the multidimensional video encoder includes a view selector that generates view selection information. The means for transcoding the decoded first bitstream is configured to render a subset of views based on the view selection information.

[0096] Example 44 includes the system of any one of Examples 41 to 43 with or without the optional feature, including means for generating a chart based on a subset of views and a set of camera parameters from the means for transcoding the decoded first bitstream, and means for generating a second bitstream configured to generate the second bitstream based on the chart.

[0097] Example 45 includes the system of any one of Examples 41 to 44 with or without the optional feature, in which the transcoded multi-dimensional data includes a texture view, a depth view, and a set of camera parameters.

[0098] Example 46 includes the system of any one of Examples 41 to 45 with or without the selective feature, in which the means for transcoding the decoded first bitstream is configured to reduce a range or number of views from the first bitstream that are included in the second bitstream.

[0099] Example 47 includes the system of any one of Examples 41 to 46 with or without the optional feature, in which the means for transcoding the decoded first bitstream is configured to directly resample a texture map and a depth map of the decoded first bitstream.

[0100] Example 48 includes the system of any one of Examples 41 to 47 with or without the optional feature, in which the means for transcoding the decoded first bitstream is configured to generate a view based on the transcoded multi-dimensional data, resample the generated view, and generate a resampled texture map and a resampled depth map based on the resampled view.

[0101] Example 49 includes the system of any one of Examples 41 to 48, with or without the optional feature, in which the means for generating the second bitstream includes a metadata streamer, the metadata streamer copying metadata from the decoded first bitstream and multiplexing the copied metadata into the second bitstream.

[0102] Example 50 includes the system of any one of examples 41 to 49 with or without the optional feature, in which the means for transcoding the decoded first bitstream includes a multidimensional video decoder or a multidimensional video encoder.

[0103] Not all of the components, features, configurations, characteristics, etc. described and illustrated herein need be present in a particular embodiment or embodiments. For example, if the specification states that a component, feature, configuration, or characteristic "may," "may be," "can be," or "could be" included, the particular component, feature, configuration, or characteristic need not be included. When the specification or claims refer to "a" element, the reference does not exclude that more than one additional element is present. When the specification or claims refer to "additional" elements, the reference does not exclude that more than one additional element is present.

[0104] Although aspects have been described with reference to specific implementations, it should be noted that other implementations are possible in accordance with some of the aspects. Additionally, the arrangement and / or order of circuit elements or other features illustrated in the figures and / or described herein need not be arranged in the particular manner shown and described. Many other configurations are possible in accordance with some of the aspects.

[0105] In each system shown in the figures, multiple elements may in some cases each have the same reference number or different reference numbers to indicate that the elements represented by those numbers may be different and / or similar elements. On the other hand, an element may be flexible enough to have different implementations and to operate with some or all of the systems shown or described herein. The various elements shown in the figures may be the same or different. It is arbitrary which element is referred to as a first element and which element is referred to as a second element.

[0106] It should be understood that details in the above examples may be used anywhere in one or more aspects. For example, all of the optional features of the computing device described above may also be implemented with respect to any of the methods or computer-readable media described herein. Furthermore, although flow diagrams and / or state diagrams may be used herein to describe aspects, the techniques are not limited to those diagrams or the corresponding descriptions herein. For example, the flow need not move through each illustrated box or state, or in the exact same order as illustrated and described herein.

[0107] The present technology is not limited to the specific details recited herein.Indeed, those skilled in the art who have the benefit of this disclosure will understand that many other variations from the above description and drawings may be made within the scope of the present technology.Thus, it is the following claims, including any amendments thereto, that define the scope of the present technology.

Claims

1. 1. An apparatus for transcoding multidimensional video, the apparatus comprising: an interface circuit configured to receive an input bitstream of multi-dimensional video; A processor circuit, the processor circuit comprising: Decoding the input bitstream of multi-dimensional video; Detecting when an output bit rate of an output bit stream is lower than an input bit rate of said input bit stream by at least a threshold difference; in response to detecting that the output bitrate is lower than the input bitrate by at least the threshold difference, transcoding the decoded input bitstream to generate transcoded multidimensional data having a reduced number of views compared to the input bitstream; generating the output bitstream based on the multi-dimensional data being transcoded; and a processor circuit configured to: Device.

2. 2. The apparatus of claim 1, wherein the input bitstream being decoded includes a texture map, a depth map, a map partition occupancy map sequence, map parameters, and camera parameters, and the processor circuitry is configured to render a view based on the input bitstream being decoded.

3. The apparatus of claim 1 , wherein the processor circuitry is configured to render a subset of views based on view selection information.

4. The apparatus of claim 1 , wherein the processor circuitry is configured to generate a representation based on a subset of views and a set of camera parameters, and the processor circuitry is configured to generate the output bitstream based on the representation.

5. The apparatus of claim 1 , wherein the multi-dimensional data being transcoded includes a texture view, a depth view, and a set of camera parameters.

6. 6. Apparatus according to any preceding claim, wherein the processor circuitry is configured to reduce a range of views included in the output bitstream compared to the input bitstream.

7. 6. An apparatus according to claim 1, wherein the processor circuitry is configured to directly resample texture and depth maps of the input bitstream being decoded.

8. The processor circuit includes: generating a view based on the transcoded multi-dimensional data; resampling the generated views; and The apparatus of claim 1 , configured to generate a resampled texture map and a resampled depth map based on the resampled views.

9. The apparatus includes a metadata streamer, the metadata streamer comprising: Parsing metadata from the decoded input bitstream that will be used in the output bitstream; and 6. An apparatus according to claim 1, further comprising: a processor for multiplexing the parsed metadata into the output bitstream.

10. 1. A method for transcoding multi-dimensional video, the method comprising: receiving an input bitstream of a multi-dimensional video; decoding the input bitstream of multi-dimensional video; detecting, using one or more processor circuits, when an output bit rate of an output bit stream is lower than an input bit rate of said input bit stream by at least a threshold difference; using the one or more processor circuits, in response to detecting that the output bitrate is lower than the input bitrate by at least the threshold difference, transcoding the decoded input bitstream to generate transcoded multidimensional data having a reduced number of views compared to the input bitstream; generating the output bitstream based on the multi-dimensional data being transcoded. method.

11. The method of claim 10, wherein the step of decoding the input bitstream includes generating a texture map, a depth map, a map partition occupancy map sequence, map parameters, and camera parameters.

12. The method of claim 10, further comprising a step of resampling a texture map and a depth map of the input bitstream being decoded using the one or more processor circuits to generate a resampled texture map and a resampled depth map.

13. The method of claim 10, further comprising a step of rendering a view based on the texture map, the depth map, the map section occupancy map sequence, the map parameters, and the camera parameters using the one or more processor circuits.

14. The method of claim 10, further comprising the step of reducing the range of views included in the output bitstream compared to the input bitstream.

15. 15. The method of claim 10, further comprising the step of rendering identified regions of interest in the input bitstream with greater precision or greater complexity than other regions in the output bitstream.

16. Rendering a larger set of views for a subset of image files in the output bitstream; generating view selector information based on said subset of image files; and reducing the number of views rendered for other image files in the output bitstream.

17. 15. The method of any one of claims 10 to 14, further comprising the step of parsing metadata from the input bitstream that is used in the output bitstream.

18. The method of any one of claims 10 to 14, further comprising the step of generating metadata for the output bitstream based on the transcoded multi-dimensional data.

19. 15. The method of claim 10, further comprising the step of transmitting the output bitstream to a second processor for display.

20. A computer program comprising instructions which, when executed by a computing device, cause the computing device to perform at least: receiving an input bitstream of a multi-dimensional video; Decoding the input bitstream of multi-dimensional video; Detecting when an output bit rate of an output bit stream is lower than an input bit rate of said input bit stream by at least a threshold difference; in response to detecting that the output bitrate is lower than the input bitrate by at least the threshold difference, transcoding the decoded input bitstream to generate transcoded multidimensional data having a reduced number of views compared to the input bitstream; generating the output bitstream based on the multi-dimensional data being transcoded; To make it so, Computer program.

21. 21. The computer program product of claim 20, wherein the instructions cause the computing device to render a view based on the input bitstream being decoded.

22. The instructions further include causing the computing device to: Generate view selection information; and 21. The computer program product of claim 20, further comprising: rendering a subset of views based on the view selection information.

23. The instructions further include causing the computing device to: generating a diagram based on the subset of views and the set of camera parameters; and 23. A computer program product as claimed in any one of claims 20 to 22, adapted to generate the output bitstream on the basis of the diagram.

24. 23. A computer program product as claimed in any one of claims 20 to 22, wherein the instructions cause the computing device to reduce a range of views included in the output bitstream compared to the input bitstream.

25. A computer readable storage medium storing a computer program according to any one of claims 20 to 24.

Images