Error suppression in view-dependent video coding based on sub-picture bitstream

Abstract

A video coding mechanism for view-dependent video coding is disclosed. The mechanism includes receiving a spherical video signal spliced from a multi-directional video stream. The spherical video signal is mapped into a plurality of sub-picture video signals, each of which includes a sequence of sub-pictures. The plurality of sub-picture video signals are coded into a plurality of sub-picture bitstreams such that, when decoded in a decoder, each pixel in each sub-picture has a value equal to a corresponding pixel in a whole picture composed of the sub-pictures after decoding. The plurality of sub-picture bitstreams form a plurality of sub-picture tracks, and it is indicated that the sub-picture tracks conform to a specific video profile. The sub-picture bitstreams are sent to a decoder to support decoding and displaying a virtual reality video view.

Description

TECHNICAL FIELD

[0001] The present disclosure generally relates to video coding for virtual reality (VR), and in particular to error prevention when encoding a VR image bitstream into multiple sub-picture bitstreams to support view-dependent video coding. BACKGROUND

[0002] When data is to be sent or otherwise communicated in a communication network with limited bandwidth capacity, it can be difficult because a large amount of video data can need to be described even for a relatively short video. Therefore, video data is typically compressed before being communicated in modern telecommunication networks. The size of a video can also be an issue when the video is stored on a storage device, as memory resources can be limited. Video compression devices typically use software and / or hardware at the source to code the video data before transmission or storage, thus reducing the amount of data needed to represent digital video images. The compressed data is then received at a destination by a video decompression device. With limited network resources and ever increasing demands of higher video quality, there is a need to improve compression and decompression techniques that improve compression ratios with little to no sacrifice in picture quality. SUMMARY

[0003] In one embodiment, the present disclosure includes a method implemented in a video encoding device. The method includes receiving, by a processor of the video encoding device, a spherical video signal stitched from multiple directional video signals. The method also includes mapping, by the processor, the spherical video signal into a plurality of sub-picture video signals, each sub-picture video signal comprising a sequence of sub-pictures. The method also includes encoding, by the processor, the plurality of sub-picture video signals into a plurality of sub-picture bitstreams such that, upon decoding, each pixel in each sub-picture has a value equal to a corresponding pixel in an entire picture composed of the sub-pictures after decoding. The method also includes grouping, by the processor, the plurality of sub-picture bitstreams into a plurality of sub-picture tracks and indicating that the sub-picture tracks conform to a particular video profile. The method also includes transmitting, by a transmitter of the video encoding device, the set of the plurality of sub-picture tracks to a decoder to support decoding and displaying a virtual reality video view. For example, an encoder can receive a virtual reality (VR) video sequence in a spherical shape and divide the sequence into sub-picture streams. Each sub-picture stream can describe a sub-portion of the entire video sequence and can be encoded individually. This is beneficial for encoding video portions that are not visible to a user or are actually background at a lower resolution and / or not at all. Video coding processes related to inter-prediction can be problematic when a sub-picture stream uses pixels in a neighboring sub-picture stream. For example, the pixels can not exist or can not include expected data due to out-of-order decoding, resolution differences, missing certain sub-picture streams, and the like. The present embodiment ensures that each pixel in each sub-picture has a value equal to a corresponding pixel in an entire picture composed of the sub-pictures after decoding.

[0004] Optionally, in any of the preceding aspects, in a further implementation thereof, encoding the plurality of sub-picture video signals into the plurality of sub-picture bitstreams includes partially encoding the sub-picture bitstreams using inter-prediction by selecting, for each current sub-picture, only motion vectors that refer to pixels in the same sub-picture video signal as the current sub-picture. This embodiment ensures that each sub-picture stream only refers to its own pixels during inter-prediction. This ensures that each sub-picture stream can be decoded without referring to neighboring sub-picture streams, thereby avoiding inter-prediction related artifacts between sub-picture streams during rendering.

[0005] Optionally, in any of the preceding aspects, in a further implementation thereof, the particular video profile is based on a High Efficiency Video Coding (HEVC) view-dependent Omnidirectional MediA Format (OMAF) video profile.

[0006] Optionally, in any of the preceding aspects, in a further implementation thereof, the selecting the motion vector of the pixel point in the same sub-picture video signal comprises: selecting the motion vector of an integer pixel position in the same sub-picture video signal; and selecting the motion vector of a fractional pixel position, the fractional pixel position being interpolated using only integer pixel positions in the same sub-picture video signal. The fractional pixel position can be used as long as it can be reconstructed by interpolation from the common sub-picture stream pixel positions (e.g., at the edges of the sub-picture stream).

[0007] Optionally, in any of the preceding aspects, in a further implementation thereof, the encoding the plurality of sub-picture video signals by inter prediction further comprises: for each current sub-picture, selecting only candidate motion vectors from blocks in the same sub-picture video signal as the current sub-picture. Another error source when generating a merge sub-picture stream when applying inter prediction based on merge mode. In merge mode, a list of candidate motion vectors is created and the motion vector is encoded as an index into the candidate list. When selecting the candidate index from a neighboring sub-picture stream, an error can occur because the selected candidate index can not be available at decoding time. This embodiment restricts the candidate motion vectors to be selected only from the same sub-picture stream, thereby avoiding this problem.

[0008] Optionally, in any of the preceding aspects, in a further implementation thereof, the encoding the plurality of sub-picture video signals by inter prediction further comprises disabling wavefront parallel processing (WPP). HEVC uses coding blocks and WPP, but they cannot be used together because WPP interferes with the merge being done in the decoder. Coding blocks are used to divide the video sequence into sub-picture streams, so WPP can be disabled to comply with HEVC.

[0009] Optionally, in any of the preceding aspects, in a further implementation thereof, the disabling WPP comprises setting entropy_coding_sync_enabled_flag to 0, wherein the entropy_coding_sync_enabled_flag is included in a picture parameter set (PPS) for the spherical video signal.

[0010] Optionally, in any above aspect, in a further implementation form of the aspect, the constraint on the sub-picture tracks can be that each sub-picture track comprises the same number of media samples. Each sub-picture stream is included in a different track. To ensure that the sub-picture streams can be merged in a decoder, the media samples are encoded / decoded at the same rate in each track. This facilitates track merging and no temporal mismatch occurs.

[0011] Optionally, in any above aspect, in a further implementation form of the aspect, the constraint on the sub-picture tracks can be that the sub-picture tracks comprise a common presentation time.

[0012] Optionally, in any above aspect, in a further implementation form of the aspect, the constraint on the sub-picture tracks can be that the sub-picture tracks are constrained such that the media samples with the common presentation time are assigned a common picture order count value on the sub-picture tracks.

[0013] Optionally, in any above aspect, in a further implementation form of the aspect, mapping the spherical video signal to the plurality of sub-picture video signals comprises setting a width and a height for each sub-picture video signal, wherein the width is an integer multiple of 64 pixels, the width is greater than or equal to 256 pixels, and the height is greater than or equal to 256 pixels.

[0014] In an embodiment, the present application comprises a non-transitory computer readable medium comprising a computer program product for use by a video encoding device, the computer program product comprising computer executable instructions stored in the non-transitory computer readable medium such that when executed by a processor, the video encoding device performs the method according to any of the above aspects.

[0015] In one embodiment, the present disclosure includes a video encoding device comprising a processor configured to receive a spherical video signal stitched from a plurality of directional video signals. The processor is further configured to map the spherical video signal into a plurality of sub-picture video signals, each sub-picture video signal comprising a sequence of sub-pictures. The processor is further configured to encode the plurality of sub-picture video signals into a plurality of sub-picture bitstreams such that, when decoded in a decoder, each pixel in each sub-picture has a value equal to a value of a corresponding pixel in an entire picture composed of the sub-picture after decoding. The processor is further configured to compose the plurality of sub-picture bitstreams into a plurality of sub-picture tracks and indicate that the sub-picture tracks conform to a particular video profile. The video encoding device further comprises a transmitter coupled to the processor, the transmitter configured to transmit the set of sub-picture tracks to a decoder to support decoding and displaying a virtual reality video perspective. For example, an encoder can receive a virtual reality (VR) video sequence in a spherical shape and divide the sequence into sub-picture streams. Each sub-picture stream can describe a sub-portion of the entire video sequence and can be encoded individually. This is advantageous for encoding video portions that are not visible to a user or are effectively background at a lower resolution and / or not at all. Video encoding processes related to inter-prediction can cause problems when a sub-picture stream uses pixels in a neighboring sub-picture stream. For example, the pixels can not exist or can not include expected data due to out-of-order decoding, resolution differences, omission of certain sub-picture streams, etc. The present embodiment ensures that each pixel in each sub-picture has a value equal to a value of a corresponding pixel in an entire picture composed of the sub-picture after decoding.

[0016] Optionally, in any of the preceding aspects, in a further implementation thereof, encoding the plurality of sub-picture video signals into the plurality of sub-picture bitstreams comprises partially encoding the sub-picture bitstreams using inter-prediction by selecting, for each current sub-picture, only motion vectors that refer to pixels in the same sub-picture video signal as the current sub-picture. This embodiment ensures that each sub-picture stream only refers to its own pixels during inter-prediction. This ensures that each sub-picture stream can be decoded without referring to neighboring sub-picture streams, thereby avoiding inter-prediction related artifacts between sub-picture streams during rendering.

[0017] Optionally, in any of the preceding aspects, in a further implementation thereof, the particular video profile is a HEVC perspective-related OMAF video profile.

[0018] Optionally, in any of the preceding aspects, in a further implementation thereof, the selecting the motion vector of the pixel in the same sub-picture video signal as the current sub-picture comprises: selecting the motion vector of an integer pixel position in the same sub-picture video signal; and selecting the motion vector of a fractional pixel position, the fractional pixel position being interpolated using only integer pixel positions in the same sub-picture video signal. The fractional pixel position can be used as long as it can be reconstructed by interpolation from the common sub-picture stream pixel positions (e.g., at the edges of the sub-picture stream).

[0019] Optionally, in any of the preceding aspects, in a further implementation thereof, the encoding the plurality of sub-picture video signals by inter prediction further comprises: for each current sub-picture, selecting only candidate motion vectors from blocks in the same sub-picture video signal as the current sub-picture. When selecting candidate indices from neighboring sub-picture streams, an error can occur because the selected candidate indices can not be available at the time of decoding. This embodiment restricts the candidate motion vectors to be selected only from the same sub-picture stream, thereby avoiding this problem.

[0020] Optionally, in any of the preceding aspects, in a further implementation thereof, the encoding the plurality of sub-picture video signals by inter prediction further comprises disabling WPP. HEVC uses coding blocks and WPP, but they cannot be used together because WPP interferes with the merging that is done in the decoder. Coding blocks are used to divide the video sequence into sub-picture streams, so WPP can be disabled to comply with HEVC.

[0021] Optionally, in any of the preceding aspects, in a further implementation thereof, the disabling WPP comprises setting entropy_coding_sync_enabled_flag to 0, wherein the entropy_coding_sync_enabled_flag is included in a PPS for the spherical video signal.

[0022] Optionally, in any of the preceding aspects, in a further implementation thereof, the constraining the sub-picture tracks can be such that each sub-picture track includes the same number of media samples. Each sub-picture stream is included in a different track. To ensure that the sub-picture streams can be merged in the decoder, the media samples are encoded / decoded at the same rate in each track. This facilitates track merging and no temporal mismatch occurs.

[0023] Optionally, in any of the preceding aspects, in a further implementation thereof, the constraining the sub-picture tracks can be such that the sub-picture tracks include a common presentation time.

[0024] Optionally, in any above aspect, in a further implementation thereof, the constraining the sub-picture track can further include constraining the sub-picture track such that a media pixel with the common presentation time is designated to have a common picture order count value on the sub-picture track.

[0025] Optionally, in any above aspect, in a further implementation thereof, the mapping the spherical video signal into the plurality of sub-picture video signals can include setting a width and a height for each sub-picture video signal, wherein the width is an integer multiple of 64 pixels, the width is greater than or equal to 256 pixels, and the height is greater than or equal to 256 pixels.

[0026] In one embodiment, the present application includes a video encoding device, comprising a video signal receiving module configured to receive a spherical video signal stitched from a multi-directional video signal. The video encoding device further comprises a mapping module configured to map the spherical video signal into a plurality of sub-picture video signals, each sub-picture video signal comprising a sequence of sub-pictures. The video encoding device further comprises an encoding module configured to encode the plurality of sub-picture video signals into a plurality of sub-picture bitstreams such that, when decoded in a decoder, each pixel in each sub-picture has a value equal to a value of a corresponding pixel in an entire picture composed of the sub-pictures after decoding. The video encoding device further comprises an assembling module configured to assemble the plurality of sub-picture bitstreams into a plurality of sub-picture tracks and indicate that the sub-picture tracks conform to a specific video profile. The video encoding device further comprises a sending module configured to send the set of the plurality of sub-picture tracks to a decoder to support decoding and displaying a virtual reality video perspective.

[0027] Optionally, in any above aspect, in a further implementation thereof, the video signal receiving module, the mapping module, the encoding module, and the sending module are further configured to perform the method in any of the above aspects.

[0028] Optionally, in any above aspect, in a further implementation thereof, the restriction further includes that the picture order count values of the sub-pictures carried by the media pixels with the same number of pixels on the tracks are equal.

[0029] For clarity, any of the above embodiments can be combined with any one or more of the other embodiments above to create new embodiments within the scope of the present application.

[0030] These and other features will be more readily understood from the following detailed description, taken in conjunction with the accompanying drawings and claims. BRIEF DESCRIPTION OF DRAWINGS

[0031] For a more complete understanding of the present application, reference is now made to the following description taken in connection with the accompanying drawings and specific implementations, wherein like numerals represent like parts.

[0032] Figure 1 Diagram of an example system for video coding based on VR.

[0033] Figure 2 Flowchart of an example method for encoding a VR image bitstream into a plurality of sub-image bitstreams.

[0034] Figure 3 Flowchart of an example method for encoding a video signal including sub-image video signals.

[0035] Figure 4 Diagram of an example codec system for video coding.

[0036] Figure 5 Diagram of an example video encoder that can encode sub-image bitstreams.

[0037] Figure 6 Diagram of an example video decoder that can decode sub-image bitstreams.

[0038] Figure 7 Diagram of a plurality of sub-image video signals partitioned from a VR image video stream.

[0039] Figure 8 Diagram of an example of unidirectional inter prediction.

[0040] Figure 9 Diagram of an example of bidirectional inter prediction.

[0041] Figure 10 Diagram of an example of a block-based video coding mechanism.

[0042] Figure 11 Diagram of an example International Standard Organization (ISO) base media file format (ISOBMFF) for transmission of video coding.

[0043] Figure 12 Flowchart of an example method for encoding sub-image bitstreams.

[0044] Figure 13 Diagram of an example video encoding device.

[0045] Figure 14A schematic diagram of an exemplary video encoding device that can encode sub-picture streams.

[0046] Figure 15 A flowchart of another exemplary method of encoding sub-picture streams. DETAILED DESCRIPTION

[0047] It should be understood at the outset that although illustrative implementations of one or more embodiments are provided below, the disclosed systems and / or methods can be implemented using any number of techniques, whether currently known or in existence. The disclosure should in no way be limited to the illustrative implementations, drawings, and techniques illustrated below, including exemplary design and implementation set forth herein, but can be modified in any manner by one of skill in the art with the scope and spirit of the disclosure.

[0048] The present disclosure relates to VR, which can also be referred to as omnidirectional media, immersive media, and 360-degree media. VR is an interactive recording and / or computer-generated experience in a simulated environment that employs visual, audio, and / or tactile feedback. For visual angles, VR provides for the imaging of a sphere (or a sub-portion of a sphere) with the user located at the center of the sphere. The sphere imaging can be presented by a head mounted display (HMD). Specifically, the HMD provides the user with a field of view (FOV) of the sphere, also referred to as a perspective. Thus, the user is able to view a sub-portion of the sphere through the perspective and move the user's head to dynamically change the location of the perspective. This gives the effect of being in the environment defined by the sphere. What each eye of the user sees is the sphere presented in an offset manner, providing the user with a sense of three-dimensional space. Each image (also referred to as a frame) includes a sphere region within the user's FOV and a sphere region outside the user's FOV. Thus, a VR frame includes significantly more data than a non-VR video image.

[0049] To generate a VR video stream, a multi-dimensional camera array records the sphere from all relevant angles simultaneously. Multi-directional video signals are produced that combine to form a spherical video signal that is used as the VR video stream. To compress the VR video stream, each frame of the stream can be divided into sub-pictures of a predefined shape, such as a rectangle, square, or other shape. These sub-pictures produce sub-picture video signals that can then be encoded and transmitted to users for decoding, reconstruction into the VR video stream, and display to the users based on the perspective. One mechanism to ensure proper reconstruction encodes each sub-picture video signal at the same resolution. However, this still results in very large files.

[0050] View-dependent coding is an example compression technique that can further reduce the file size of a VR video stream (e.g., to support VR-based video streaming). View-dependent coding uses the concept that a user is only looking at a spherical portion of the view at any time. Thus, areas outside of the user’s FOV can be transmitted at a lower resolution (e.g., resolution is down compared to the current FOV). The resolution of the sub-image video signal can be dynamically increased or decreased as the FOV moves, e.g., by dynamically selecting different sub-image video signal representations of different resolutions. This approach reduces the overall file size of the VR video stream by reducing the video quality of the spherical areas that the user cannot see (e.g., and / or that are considered background and unimportant at the time of encoding).

[0051] In certain cases, view-dependent encoding can result in video encoding errors. For example, a video stream (e.g., a subpicture video signal) can be encoded using inter-prediction. In inter-prediction, blocks of pixels (e.g., representing an object) in multiple frames can be encoded as a motion vector pointing from a current block in a different frame to another block. Thus, one block in multiple frames can be encoded once (e.g., by intra-prediction), and a series of motion vectors all reference this block in order to reduce redundancy and reduce file size. There are multiple modes of inter-prediction. Modes such as merge mode and advanced motion vector prediction (AMVP) mode generate a list of candidate motion vectors for a current block from neighboring blocks. In these modes, a selected motion vector is indicated by an index in the candidate list. In this way, an entire block of pixels can be encoded as an index value. When applied to neighboring subpicture video signals that include different resolutions and / or frame rates, inter-prediction using candidate lists can cause problems in view-dependent encoding. Specifically, a candidate list for a block at an edge of a first subpicture video signal can depend on motion vectors used at a corresponding edge of a neighboring second subpicture video signal. When the first and second subpicture video signals are transmitted at different resolutions / frame rates, resolution-dependent data mismatches can result in undesirable motion vector values being included in the candidate list (or desirable values being missing). This results in the decoder generating an incorrect candidate list, which leads to an incorrect interpretation of the candidate list index. The incorrect interpretation of the candidate list index results in errors in the decoded subpicture video signal, which results in errors in the VR video stream displayed to the user. Similar problems can occur if a motion vector in one subpicture video signal references a pixel in another subpicture video signal, as that pixel can be unavailable and / or resolution-reduced. Furthermore, a video codec can use wavefront parallel processing (WPP) to encode / decode blocks of a video image in parallel. However, WPP can affect block-level encoding mechanisms, and thus decoding different portions of an image at different resolutions according to WPP can result in other errors.

[0052] A mechanism is disclosed herein to reduce the error caused by using a candidate list in inter prediction when applying inter prediction to sub-picture video signals of a VR video stream encoded according to a view-dependent coding mechanism. For example, when encoding a sub-picture video signal, a current block in a sub-picture of the sub-picture video signal can only refer to reference samples in a reference block at integer pixel positions in the same sub-picture video signal, and thus cannot refer to reference samples at integer pixel positions or fractional pixel positions in other sub-picture video signals. This applies to both intra prediction and inter prediction. Furthermore, a fractional pixel position in a sub-picture video signal can only be referred to when the pixel at the fractional pixel position can be recreated by interpolation from only integer pixel positions and / or fractional pixel positions in the same sub-picture stream. Furthermore, when performing inter prediction, the motion vectors of blocks in neighboring sub-picture streams are not included in the candidate motion vector list. These restrictions ensure that each sub-picture stream can be decoded without referring to neighboring sub-picture streams, thus avoiding resolution mismatch. Furthermore, WPP cannot be used with sub-picture streams. For example, entropy_coding_sync_enabled_flag in each picture parameter set (PPS) can be set to 0 to disable WPP for sub-picture video signals. Furthermore, by ensuring that the tracks including the sub-picture video signals have the same presentation time, and by ensuring that each pixel in a common VR picture uses the same picture order count value, even if these pixels are divided into different sub-picture streams or carried in different tracks, timing mismatch between sub-picture streams can be avoided.

[0053] Figure 1FIG. 1 is a schematic diagram of an exemplary system 100 for VR video encoding. The system 100 includes a multi-directional camera 101, a VR encoding device 104 including an encoder 103, a decoder 107, and a rendering device 109. The multi-directional camera 101 includes an array of camera devices. Each camera device is pointed at a different angle, enabling the multi-directional camera 101 to acquire a multi-directional video signal of the surrounding environment from multiple angles. For example, the multi-directional camera 101 can take a video of the environment as a sphere with the multi-directional camera 101 at the center of the sphere. As used herein, a sphere and / or spherical video refers to both a geometric sphere and a sub-portion of a geometric sphere, such as a spherical cap, a spherical dome, a spherical segment, etc. For example, the multi-directional camera 101 can take a 180-degree video to cover half of the environment, enabling a producer to stay behind the multi-directional camera 101. The multi-directional camera 101 can also take a video within 360 degrees (or any sub-portion thereof). However, a portion of the ground under the multi-directional camera 101 can be omitted, resulting in a video smaller than a perfect sphere. Thus, the term "sphere" used herein is a general term used for clarity of description and should not be construed as limiting from a geometric standpoint.

[0054] The video of the multi-directional camera 101 is forwarded to the VR encoding device 104. The VR encoding device 104 can be a computing system including specialized VR encoding software. The VR encoding device 104 can include the encoder 103. In some examples, the encoder 103 can also be included in a computer system separate from the VR encoding device 104. The VR encoding device 104 is used to convert the multi-directional video signal into one multi-directional video signal that includes the entire recorded area of all relevant angles. This conversion can be referred to as image stitching. For example, frames of each video stream captured simultaneously can be stitched together to create one spherical image. A spherical video signal can then be created from the spherical image. For clarity of description, it should be noted that the terms "frame" and "picture / image" are interchangeable herein unless specifically stated otherwise.

[0055] The spherical video signal can then be forwarded to an encoder 103 for compression. An encoder 103 is a device and / or program that is capable of converting information from one format to another format in order to standardize, speed up, and / or compress. A standardization encoder 103 is used to encode rectangular and / or square images. Thus, the encoder 103 is used to map each spherical image of the spherical video signal into a plurality of sub-images of a predefined shape, such as a rectangle, square, or other two-dimensional geometric shape. While the spherical images can be mapped into sub-images of any shape, for clarity and simplicity of description, the sub-images can be referred to hereinafter as rectangles. However, this rectangular limitation should not be construed as having the opposite meaning of no context indication. After the mapping is complete, the sub-images can be placed in separate sub-image video signals. Thus, each sub-image video signal displays a stream of images recorded from a sub-portion of the spherical video signal over time. The encoder 103 can then encode each sub-image video signal to compress the video stream into a manageable file size. The encoding process will be discussed in more detail below. Generally, the encoder 103 divides each frame of each sub-image video signal into blocks of pixels, compresses the blocks of pixels through inter-prediction and / or intra-prediction to create encoded blocks comprising predicted blocks and residual blocks, applies a transform to the residual blocks for further compression, and applies various filters on the blocks. The compressed blocks and corresponding syntax are stored in a bitstream in formats such as International Standardization Organization base media file format (ISOBMFF) and / or omnidirectional media format (OMAF).

[0056] The VR encoding device 104 can store the encoded bitstream in a local and / or server storage for communication with a decoder 107 on demand. The data can be forwarded through a network 105, which can include the Internet, a mobile communication network (e.g., a long term evolution (LTE) based data network), or other data communication data system.

[0057] The decoder 107 is a device located at the user's location that performs the inverse of the encoding process to reconstruct the sub-picture video signal from the encoded bitstream. The decoder 107 also combines the sub-picture video signals to reconstruct the spherical video signal. The spherical video signal or its sub-portions can then be forwarded to a presentation device 109. The presentation device 109 is a device for displaying the spherical video signal to the user. For example, the presentation device 109 can include an HMD that is fixed to the user's head and occludes the user's eyes. The presentation device 109 can include a screen for each eye, a camera, motion sensors, speakers, etc., and can communicate with the decoder 107 through a wireless and / or wired connection. The presentation device 109 can display a sub-portion of the spherical video signal to the user. The sub-portion shown is based on the FOV and / or perspective of the presentation device. For example, the presentation device 109 can change the location of the FOV according to the user's head movement by employing motion tracking sensors. The user can see different portions of the spherical video signal according to the head movement. In addition, the presentation device 109 can adjust the FOV for each eye according to the user's interpupillary distance (IPD) to create a three-dimensional spatial effect.

[0058] Figure 2 A flowchart of an exemplary method 200 for encoding a VR image bitstream into a plurality of sub-picture bitstreams, for example, by employing components of the system 100. In step 201, a multi-directional video signal is captured using a multi-directional camera set, such as the multi-directional camera 101. The multi-directional video signal includes views of the environment at different angles. For example, the multi-directional video signal can capture video in a plane (horizontal plane) around the camera at 360 degrees, 180 degrees, 240 degrees, etc. The multi-directional video signal can also capture video in a vertical plane around the camera at 360 degrees, 180 degrees, 240 degrees, etc., resulting in a signal that includes enough information to cover a spherical region around the camera over a period of time.

[0059] In step 203, the multi-directional video signal is synchronized in the time domain. Specifically, each directional video stream includes a series of images taken at a corresponding angle. The multi-directional video signal is synchronized by ensuring that frames captured at the same time domain position from each directional video stream are processed together. The frames in the directional video streams can then be stitched together in the spatial domain to produce the spherical video signal. Thus, each frame of the spherical video signal includes data taken from frames of all of the directional video streams that were produced at a common time position.

[0060] In step 205, the spherical video signal is mapped into sub-picture video signals (e.g., rectangular, square, etc.). This process can also be referred to as projecting the spherical video signal into rectangular sub-picture video signals. As mentioned above, encoders and decoders are typically designed to encode rectangular frames and / or square frames. Therefore, mapping the spherical video signal into rectangular sub-picture video signals results in a video stream that can be encoded and decoded by non-VR-specific encoders and decoders, respectively. It should be noted that steps 203 and 205 are for VR video processing, and therefore can be performed by specialized VR hardware, software, or a combination thereof.

[0061] In step 207, the rectangular sub-picture video signals can be forwarded to an encoder 103 or the like. The encoder then encodes the sub-picture video signals into sub-picture bitstreams in corresponding media file formats. Specifically, the encoder can treat each sub-picture video signal as a video signal. The encoder can encode each frame of each sub-picture video signal by inter-prediction, intra-prediction, or the like. Details are discussed below with respect to Figures 3 to 14 Encoding and corresponding decoding, as well as encoders and decoders, are discussed in detail. In terms of file formats, the sub-picture video signals can be stored in ISOBMFF. For example, the sub-picture video signals are captured at a specified resolution. The sub-picture video signals can then be down-sampled to various lower resolutions for encoding. Each resolution can be referred to as a representation. The lower the quality, the less clear the image, but the smaller the file size. Therefore, a lower quality representation can be sent to a user using less network resources (e.g., time, bandwidth) than a higher quality representation with a loss in visual quality. Each representation can be stored in a corresponding set of tracks. Therefore, a track can be sent to a user, where the track includes sub-picture bitstreams at various resolutions (e.g., visual quality).

[0062] In step 209, the sub-picture code streams can be sent to the decoder as tracks. In some examples, all sub-picture code streams are transmitted at the same quality by transmitting the tracks from the same representation. A drawback of this approach is that areas of the final VR video stream that are less interesting are transmitted at the same resolution as all other areas. View-dependent coding can be used to improve compression with this approach. In view-dependent coding, the tracks of sub-picture code streams that include data in the user's FOV are transmitted at a higher resolution by selecting a higher quality representation. The tracks of sub-picture code streams that include areas outside the user's FOV can be transmitted at progressively lower resolutions by selecting lower quality representations. In some examples, certain areas can even be omitted entirely. For example, areas adjacent to the FOV can be transmitted at slightly reduced quality in the event that the user decides to change the FOV to include these areas. Areas further away from the FOV can be transmitted at progressively lower quality as the likelihood of such areas entering the FOV and thus being presented to the user progressively decreases. The tracks can include relatively short video segments (e.g., approximately 3 seconds), so the representation selected for a particular video area can change over time in response to changes in the FOV. The quality can change as the user's FOV changes. View-dependent coding can significantly reduce the file size of the tracks sent to the user, while the visual quality is not significantly impaired as the areas of reduced quality are less likely to be seen by the user.

[0063] In step 211, a decoder 107 or the like receives the tracks including the sub-picture code streams. The decoder can then decode the sub-picture code streams into sub- picture video signals for display. The decoding process involves the inverse of the encoding process (e.g., using inter- and intra-frame prediction), and is discussed in more detail below with respect to Figures 3 to 14 In more detail.

[0064] In step 213, the decoder can merge the sub-picture video signals into a spherical video signal for presentation to the user. In particular, the decoder can use a lightweight merging algorithm that selects frames from each sub-picture video signal produced at the same presentation time and merges them together according to the location and / or angle associated with the corresponding sub-picture video signal. The decoder can also use filters to smooth edges between sub-picture video signals, remove artifacts, and the like. The decoder can then forward the spherical video signal to a presentation device, such as presentation device 109.

[0065] In step 215, the presentation device presents the view of the spherical video signal to the user. As noted above, areas of the spherical video signal outside the FOV at each point in time are not presented. Thus, in view-dependent coding, the lower quality representations are effectively omitted, so the impact of reduced viewing quality on the user experience can be negligible while reducing the file size.

[0066] The view-dependent encoding applied to system 100 and method 200 reduces file size without a noticeable loss of visual quality to the user. However, in some cases, the method can produce video encoding errors. Specifically, when the sub-picture video signals have different representations, inter-frame prediction and intra-frame prediction that extend onto the boundaries of the sub-picture video signals can produce artifacts. This is because the lower quality representations are lossy and omit data to reduce file size. The high quality sub-picture video signals can be encoded based on data removed from the adjacent lower quality sub-picture video signals. Thus, the high quality sub-picture video signals can be decoded incorrectly when this data has been removed from the adjacent lower quality sub-picture video signals. The present invention includes mechanisms to restrict encoding of the sub-picture video signals to ensure that each sub-picture video signal is self-contained, so that this error does not occur when a lightweight merge algorithm is applied in the decoder. Figures 3 to 10 A method of encoding each sub-picture video signal and a system are described. Figures 11 to 13 Further constraints are described that can be applied to avoid such artifacts.

[0067] Figure 3 A flowchart of an exemplary method 300 of encoding a video signal including sub-picture video signals is described. For example, method 300 can receive a plurality of sub-picture video signals from step 205 of method 200. Method 300 treats each sub-picture video signal as a video signal input. Method 300 applies steps 301-317 to each sub-picture video signal to implement steps 207-211 of method 200. Thus, the output video signals in method 300 include decoded sub-picture video signals that can be merged and displayed according to steps 213 and 215 of method 200.

[0068] Method 300 encodes video signals (e.g., including sub-picture video signals) in an encoder. The encoding process compresses the video signals using various mechanisms to reduce the size of the video file. Smaller file sizes facilitate transmission of the compressed video file to the user while reducing the associated bandwidth overhead. A decoder then decodes the compressed video file to reconstruct the original video signal for display to the end user. The decoding process is typically identical to the encoding process to facilitate consistent reconstruction of the video signal by the decoder.

[0069] In step 301, a video signal is input into an encoder. For example, the video signal can be an uncompressed video file stored in memory. As another example, the video file can be captured by a video capture device (e.g., a video camera) and encoded to support live streaming of the video. The video file can include both an audio component and a video component. The video component includes a series of image frames that, when viewed in order, give the visual impression of motion. The frames include pixels that represent light (referred to herein as luma components (or luma pixels)) and color (referred to as chroma components (or chroma pixels)).

[0070] In step 303, the video is partitioned into blocks. The partitioning includes subdividing the pixels in each frame into square and / or rectangular blocks for compression. For example, in High Efficiency Video Coding (HEVC) (also known as H.265 and MPEG-H Part 2), a frame can first be divided into coding tree units (CTUs), which are blocks of a predefined size (e.g., 64 pixels x 64 pixels). The CTUs include luma pixels and chroma pixels. As used herein, a pixel (also referred to as a pixel point) is a value of an image at a corresponding location (e.g., a luma / light value or a chroma / color value). The CTUs can be divided into blocks using a coding tree, which then recursively subdivides the blocks until a configuration is obtained that supports further encoding. For example, the luma component of a frame can be subdivided until individual blocks include relatively uniform lighting values. Further, the chroma component of a frame can be subdivided until individual blocks include relatively uniform color values. Thus, the content of a video frame varies, and the partitioning mechanism varies.

[0071] In step 305, the image blocks partitioned in step 303 are compressed using various compression mechanisms. For example, inter-frame prediction and / or intra-frame prediction can be used. Inter-frame prediction aims to exploit the fact that objects tend to appear in successive frames in a common scene. Thus, a block depicting an object in a reference frame need not be repeated in a neighboring frame. Specifically, an object, such as a table, can remain in a constant position across multiple frames. Thus, the table is described only once, and neighboring frames can refer back to the reference frame. Pattern matching mechanisms can be employed to match objects across multiple frames. Further, a moving object can be represented across multiple frames due to movement of the object or movement of the camera, among other reasons. In a particular example, a video can show a car moving across a screen through multiple frames. Motion vectors can be used to describe this movement (or lack thereof). A motion vector is a two-dimensional vector that provides an offset from a coordinate of an object in a frame to a coordinate of the object in a reference frame. Thus, inter-frame prediction can encode an image block in a current frame as a set of motion vectors that indicate an offset from a corresponding block in a reference frame.

[0072] Intra prediction encodes blocks in a common frame. Intra prediction takes advantage of the fact that luminance components and chrominance components tend to cluster in frames. For example, a patch of green in a portion of a tree tends to be adjacent to several other patches of green. Intra prediction uses multi-directional prediction modes (e.g., 33 modes in HEVC), planar modes, and direct current (DC) modes. Directional modes indicate that the current block is similar / same to pixels in neighboring blocks in a corresponding direction. Planar modes indicate that a series of blocks on a row / column (e.g., a plane) can be interpolated from the neighboring blocks on the row edge. In effect, planar modes indicate a smooth transition of light / color between rows / columns by using a relatively constant slope in the varying values. DC modes are used for boundary smoothing, indicating that the current block is similar / same to an average value of pixels associated with all neighboring blocks associated with the corner direction of the directional prediction mode. Thus, an intra prediction block can represent an image block as various relational prediction mode values rather than actual values. Furthermore, an inter prediction block can represent an image block as motion vector values rather than actual values. In both cases, the prediction block can not completely represent the image block in some cases. Any differences are stored in a residual block. The residual block can be transformed to further compress the file.

[0073] In step 307, various filtering techniques can be applied. In HEVC, filters are applied according to an in-loop filtering scheme. The block-based prediction discussed above can create a blocky image in the decoder. Furthermore, the block-based prediction scheme can encode the blocks and then reconstruct the encoded blocks for later use as reference blocks. The in-loop filtering scheme iteratively applies a noise suppression filter, a deblocking filter, a sample adaptive offset (SAO) filter, and an adaptive loop filter to the blocks / frames. These filters reduce such block artifacts so that the encoded file can be accurately reconstructed. Furthermore, these filters reduce reconstructed reference block artifacts so that artifacts are less likely to create other artifacts in subsequent blocks encoded based on the reconstructed reference blocks.

[0074] In step 309, once the video signal is partitioned, compressed, and filtered, the resulting data is encoded into a bitstream. The bitstream includes the above-described data as well as any signaling data (e.g., syntax) desired to support proper video signal reconstruction in a decoder. For example, such data can include partitioning data, prediction data, residual blocks, and various flags that provide encoding instructions to the decoder. The bitstream can be stored in memory for transmission to a decoder upon request, e.g., as a track and / or track segment in an ISB BMFF. The bitstream can also be broadcast and / or multicast to multiple decoders. The creation of the bitstream is an iterative process. Thus, steps 301, 303, 305, 307, and 309 can occur consecutively and / or simultaneously over multiple frames and blocks. Figure 3The order shown is presented for clarity of description and conciseness of discussion and is not intended to limit the video encoding process to a particular order.

[0075] In step 311, the decoder receives the bitstream and begins the decoding process. For example, the decoder converts the bitstream to corresponding syntax and video data using an entropy decoding scheme. In step 311, the decoder uses the syntax data in the bitstream to determine the partitioning of the frame. The partitioning should match the results of the block partitioning in step 303. Entropy encoding / decoding that can be used in step 311 is now described. The encoder makes many choices during the compression process, such as choosing a block partitioning scheme from a plurality of possible choices based on the spatial location of values in the input image. Indicating the exact choice can use a large number of bits. A bit as used herein is a binary value that varies as a variable (e.g., a bit value that can vary based on a context). Entropy coding facilitates the encoder discarding any options that are clearly not a good fit for the particular situation, leaving a set of usable options. Then, a code word is assigned to each of the usable options. The length of the code word depends on the number of usable options (e.g., one bit for two options, two bits for three to four options). Then, the encoder encodes the code word for the selected option. This scheme reduces the size of the code word because the size of the code word is as large as is needed to uniquely indicate one option from a small subset of usable options, rather than uniquely indicating the option from a potentially large set of all possible options. Then, the decoder decodes the option by determining the set of usable options in a similar manner as the encoder. By determining the set of usable options, the decoder can read the code word and determine the choice made by the encoder.

[0076] In step 313, the decoder performs block decoding. Specifically, the decoder performs an inverse transform to generate a residual block. Then, the decoder uses the residual block and corresponding prediction blocks to reconstruct the image block according to the partitioning. The prediction blocks can include the intra prediction blocks and the inter prediction blocks generated in the encoder in step 305. Then, the reconstructed image block is placed in the frame of the reconstructed video signal according to the partitioning data determined in step 311. The syntax of step 313 can also be indicated in the bitstream through entropy coding discussed above.

[0077] In step 315, filtering is performed on the frames of the reconstructed video signal in a similar manner as the encoder in step 307. For example, noise suppression filters, deblocking filters, adaptive loop filters, and SAO filters can be used on the frames to eliminate block artifacts. Once the frames are filtered, the video signal can be forwarded for merging in step 317 and output to a display such as an HDM for viewing by an end user.

[0078] The present invention relates to modifications that prevent artifacts when sub-picture video signals in a VR video stream are encoded at a particular resolution and then decoded at a different resolution according to view-dependent coding. These artifacts are caused by the block compression mechanism in step 305. Therefore, the present invention improves the functionality of the block compression in step 305 by the encoder and the block decoding in step 313 by the decoder. This reduces the file size, thereby improving the encoding efficiency, and / or improves the visual quality of the resulting decoded VR video stream in the decoder.

[0079] Figure 4 is a schematic diagram of an exemplary codec (coder-decoder) system 400 for video coding. In particular, the codec system 400 provides functionality to support coding of sub-picture video signals according to the methods 200 and 300. Moreover, the codec system 400 can be used to implement the encoder 103 and / or the decoder 107 of the system 100.

[0080] Broadly, the codec system 400 is used to describe components used in encoders and decoders. As discussed with respect to steps 301 and 303 in the method of operation 300, the codec system 400 receives a frame and partitions the frame to form a video signal (e.g., including sub-picture video signals), resulting in a partitioned video signal 401. As discussed with respect to steps 305, 307, and 309 in the method 300, when acting as an encoder, the codec system 400 compresses the partitioned video signal 401 into an encoded bitstream. As discussed with respect to steps 311, 313, 315, and 317 in the method 300, when acting as a decoder, the codec system 400 generates an output video signal from the bitstream. The codec system 400 includes a general encoder control component 411, a transform scaling and quantization component 413, an intra estimation component 415, an intra prediction component 417, a motion compensation component 419, a motion estimation component 421, a scaling and inverse transform component 429, a filter control analysis component 427, an in-loop filter component 425, a decoded picture buffer component 423, a header formatting and context adaptive binary arithmetic coding (CABAC) component 431. These components are coupled as shown. In Figure 4 In general, the solid lines represent motion of data to be encoded / decoded, while the dashed lines represent motion of control data that controls operation of the other components. The components in the codec system 400 can all be present in an encoder. A decoder can include a subset of the components of the codec system 400. For example, a decoder can include the intra prediction component 417, the motion compensation component 419, the scaling and inverse transform component 429, the in-loop filter component 425, and the decoded picture buffer component 423. These components are now discussed.

[0081] The partitioned video signal 401 is a captured video sequence that has been partitioned into blocks of pixels by a coding tree. The coding tree uses various splitting modes to subdivide the blocks of pixels into smaller blocks of pixels. These blocks can then be further subdivided into smaller blocks. The blocks can be referred to as nodes on the coding tree. Larger parent nodes are split into smaller child nodes. The number of times a node is subdivided is referred to as the depth of the node / coding tree. In some cases, the split blocks can be included in a coding unit (CU). For example, a CU can be a sub-portion of a CTU including a luma block, a red-difference chroma (Cr) block, and a blue-difference chroma (Cb) block, and corresponding syntax instructions for the CU. The splitting modes can include binary trees (BT), triple trees (TT), and quad trees (QT) for splitting nodes into two, three, or four child nodes of different shapes, respectively, depending on the splitting mode used. The partitioned video signal 401 is forwarded to the general encoder control component 411, the transform, scaling, and quantization component 413, the intra-estimation component 415, the filter control analysis component 427, and the motion estimation component 421 for compression.

[0082] The general encoder control component 411 is used to make decisions related to encoding the images of a video sequence into a bitstream according to application constraints. For example, the general encoder control component 411 manages the optimization of the bit rate / bitstream size versus the reconstructed quality. Such decisions can be made according to storage space / bandwidth availability and image resolution requests. The general encoder control component 411 also manages the utilization of the buffers according to the transmission speed to mitigate buffer underflow and overflow problems. To manage these problems, the general encoder control component 411 manages the partitioning, prediction, and filtering by the other components. For example, the general encoder control component 411 can dynamically increase the compression complexity to increase the resolution and bandwidth utilization, or decrease the compression complexity to decrease the resolution and bandwidth utilization. Thus, the general encoder control component 411 controls the other components of the codec system 400 to balance the video signal reconstructed quality versus the bit rate problem. The general encoder control component 411 creates control data that controls the operation of the other components. The control data is also forwarded to the header formatting and CABAC component 431 for encoding in the bitstream to indicate parameters for decoding at the decoder.

[0083] The partitioned video signal 401 is also sent to the motion estimation component 421 and the motion compensation component 419 for inter-prediction. Frames or slices of the partitioned video signal 401 can be divided into video blocks. The motion estimation component 421 and the motion compensation component 419 perform inter-predictive encoding on a received video block according to one or more blocks in one or more reference frames to provide temporal prediction. The codec system 400 can perform a number of encoding processes in order to select an appropriate encoding mode for each block of video data, etc.

[0084] Motion estimation component 421 and motion compensation component 419 can be highly integrated, but are illustrated separately for conceptual purposes. Motion estimation performed by motion estimation component 421 is a process of generating motion vectors that are used to estimate the motion of a video block. For example, a motion vector can indicate a displacement of a coding object relative to a prediction block. A prediction block is a block that is found to closely match the block to be coded in terms of pixel difference. A prediction block can also be referred to as a reference block. Such pixel difference can be determined by sum of absolute difference (SAD), sum of square difference (SSD), or other difference metrics. HEVC uses several coding objects, including a CTU, a coding tree block (CTB), and a CU. For example, a CTU can be divided into multiple CTBs, which can then be partitioned into multiple CBs included in a CU. A CU can be coded as a prediction unit (PU) that includes prediction data and / or a transform unit (TU) that includes transform residual data of the CU. Motion estimation component 421 generates motion vectors, PUs, and TUs using rate-distortion analysis as part of a rate-distortion optimization process. For example, motion estimation component 421 can determine multiple reference blocks, multiple motion vectors, etc. for a current block / frame, and can select a reference block, a motion vector, etc. that has the best rate-distortion characteristics. The best rate-distortion characteristics balance the quality of video reconstruction (e.g., amount of data loss due to compression) and coding efficiency (e.g., size of the final encoding).

[0085] In some examples, video coding system 400 can calculate values for sub-integer pixel positions of reference pictures stored in decoded picture buffer component 423. For example, video coding system 400 can interpolate values for quarter-pel positions, eighth-pel positions, or other fractional-pel positions of reference pictures. Thus, motion estimation component 421 can perform a motion search with respect to integer-pel positions and fractional-pel positions, and output motion vectors with fractional-pel precision. Motion estimation component 421 calculates motion vectors for PUs of a video block in an inter-coded slice by comparing the position of the PU to the position of a prediction block of a reference picture. Motion estimation component 421 outputs the calculated motion vectors as motion data to header formatting and CABAC component 431 for encoding, and to motion compensation component 419.

[0086] Motion compensation performed by motion compensation component 419 can involve fetching or generating a prediction block according to a motion vector determined by motion estimation component 421. Also, in some examples, motion estimation component 421 and motion compensation component 419 can be functionally integrated. After receiving a motion vector for a PU of a current video block, motion compensation component 419 can locate the prediction block to which the motion vector points. Then, by subtracting pixel values of the prediction block from pixel values of the current video block being encoded, pixel difference values are produced, thereby forming a residual video block. Typically, motion estimation component 421 performs motion estimation on luma components, and motion compensation component 419 uses motion vectors calculated from luma components for both luma and chroma components. The prediction block and the residual block are forwarded to transform scaling and quantization component 413.

[0087] The partitioned video signal 401 is also sent to intra-estimation component 415 and intra-prediction component 417. Like motion estimation component 421 and motion compensation component 419, intra-estimation component 415 and intra-prediction component 417 can be highly integrated, but are illustrated separately for conceptual purposes. Intra-estimation component 415 and intra-prediction component 417 perform intra-prediction on blocks in the current frame, as an alternative to inter-prediction performed by motion estimation component 421 and motion compensation component 419 between frames, as described above. Specifically, intra-estimation component 415 determines an intra-prediction mode for encoding a current block. In some examples, intra-estimation component 415 selects an appropriate intra-prediction mode from among a plurality of tested intra-prediction modes to encode the current block. The selected intra-prediction mode is then forwarded to header formatting and CABAC component 431 for encoding.

[0088] For example, intra-estimation component 415 uses rate-distortion analysis of various tested intra-prediction modes to calculate rate-distortion values, and selects an intra-prediction mode from among the tested modes that has the best rate-distortion characteristics. Rate-distortion analysis generally determines an amount of distortion (or error) between an encoded block and an original, unencoded block that was encoded to produce the encoded block, as well as a code rate (e.g., number of bits) used to produce the encoded block. Intra-estimation component 415 calculates a ratio of rate to distortion for various encoded blocks, and determines which intra-prediction mode results in the best rate-distortion values for the block. Additionally, intra-estimation component 415 can be used to encode depth blocks of a depth map according to a depth modeling mode (DMM) according to rate-distortion optimization (RDO).

[0089] When implemented on an encoder, the intra prediction component 417 can generate a residual block from the prediction block according to the selected intra prediction mode determined by the intra estimation component 415, or read the residual block from the bitstream when implemented on a decoder. The residual block includes the value differences between the prediction block and the original block, denoted as a matrix. The residual block is then forwarded to the transform scaling and quantization component 413. The intra estimation component 415 and the intra prediction component 417 can perform operations on luma and chroma components.

[0090] The transform scaling and quantization component 413 functions to further compress the residual block. The transform scaling and quantization component 413 applies a transform, such as a discrete cosine transform (DCT), a discrete sine transform (DST), or a conceptually similar transform, to the residual block, producing a video block comprising residual transform coefficient values. Wavelet transforms, integer transforms, sub-band transforms or other types of transforms could also be used. The transform can convert the residual information from a pixel value domain to a transform domain, e.g., a frequency domain. The transform scaling and quantization component 413 also functions to scale the transformed residual information, e.g., according to frequency. This scaling involves applying a scale factor to the residual information so as to quantize different frequency information at different granularities, which can affect the ultimate visual quality of the reconstructed video. The transform scaling and quantization component 413 also functions to quantize the transform coefficients to further reduce the bit rate. The quantization process can reduce the bit depth associated with some or all of the coefficients. The degree of quantization can be modified by adjusting a quantization parameter. In some examples, the transform scaling and quantization component 413 can then scan the matrix including the quantized transform coefficients. The quantized transform coefficients are forwarded to the header formatting and CABAC component 431 to be encoded into the bitstream.

[0091] The scaling and inverse transform component 429 performs inverse operations of the transform scaling and quantization component 413 to support motion estimation. The scaling and inverse transform component 429 performs inverse scaling, inverse transform, and / or inverse quantization to reconstruct a residual block in the pixel domain, e.g., for subsequent use as a reference block, which can become a prediction block for another current block. The motion estimation component 421 and / or the motion compensation component 419 can calculate the reference block by adding the residual block to the corresponding prediction block for use in motion estimation of a subsequent block / frame. Filters are applied to the reconstructed reference block to reduce artifacts created in the scaling, quantization, and transform processes. These artifacts can lead to inaccurate predictions (and create further artifacts) when predicting a subsequent block.

[0092] The filter control analysis component 427 and the in-loop filter component 425 apply filters to the residual blocks and / or the reconstructed image blocks. For example, a transformed residual block in the scaling and inverse transform component 429 can be combined with a corresponding prediction block in the intra-prediction component 417 and / or the motion compensation component 419 to reconstruct an original image block. Filters can then be applied to the reconstructed image block. In some examples, filters can be applied to the residual blocks. As with the other components in FIG. 4, the filter control analysis component 427 and the in-loop filter component 425 are highly integrated and can be implemented together, but are described separately for conceptual purposes. Figure 4 The filter control analysis component 427 and the in-loop filter component 425, like the other components in FIG. 4, are highly integrated and can be implemented together, but are described separately for conceptual purposes. Filters applied to reconstructed reference blocks are applied to particular spatial regions, and include multiple parameters to adjust how such filters are applied. The filter control analysis component 427 analyzes the reconstructed reference blocks to determine where these filters should be applied and to set the corresponding parameters. These data are forwarded to the header formatting and CABAC component 431 as filter control data for encoding. The in-loop filter component 425 applies these filters according to the filter control data. The filters can include a deblocking filter, a noise suppression filter, a SAO filter, and an adaptive loop filter. These filters can be applied in the spatial / pixel domain (e.g., on reconstructed pixel blocks) or in the frequency domain, according to examples.

[0093] When operating as an encoder, the filtered reconstructed image blocks, residual blocks, and / or prediction blocks are stored in the decoded picture buffer component 423 for later use in motion estimation as described above. When operating as a decoder, the decoded picture buffer component 423 stores reconstructed and filtered blocks and forwards the reconstructed and filtered blocks to a display as part of an output video signal. The decoded picture buffer component 423 can be any memory device capable of storing prediction blocks, residual blocks, and / or reconstructed image blocks.

[0094] The header formatting and CABAC component 431 receives data from various components of the codec system 400 and encodes this data into an encoded bitstream for transmission to a decoder. In particular, the header formatting and CABAC component 431 generates various headers to encode control data, such as overall control data and filter control data. In addition, prediction data, including intra prediction and motion data, and residual data in the form of quantized transform coefficient data are both encoded into the bitstream. The final bitstream includes all the information that a decoder wants to reconstruct the original partitioned video signal 401. Such information can also include an intra prediction mode index table (also referred to as a codeword mapping table), a definition of the encoding context of various blocks, an indication of the most probable intra prediction mode, an indication of partitioning information, and so on. This data can be encoded through entropy encoding techniques. For example, the information can be encoded by using context adaptive variable length coding (CAVLC), CABAC, syntax-based context-adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding, or other entropy encoding techniques. After the entropy encoding, the encoded bitstream can be transmitted to another device (e.g., a video decoder) or archived for later transmission or retrieval.

[0095] The present disclosure includes modifications that prevent artifacts when sub-picture video signals in a VR video stream are encoded at a particular resolution and then decoded at a different resolution according to view-dependent encoding. The mechanisms for intra prediction according to the intra estimation component 415 and / or the intra prediction component 417 and inter prediction according to the motion compensation component 419 and / or the motion estimation component 421 produce such artifacts. Accordingly, the present disclosure improves the functionality of the intra estimation component 415, the intra prediction component 417, the motion compensation component 419, and / or the motion estimation component 421 in the encoder and / or the decoder. File size is reduced, and thus encoding efficiency is improved, and / or the visual quality of the resulting decoded VR video stream in the decoder is improved.

[0096] Figure 5A block diagram of an example video encoder 500 that can encode sub-picture bitstreams. The video encoder 500 can be used to implement the encoding functionality of the codec system 400 and / or implement steps 301, 303, 305, 307, and / or 309 of the method 300. Further, the encoder 500 can be used to implement steps 205-209 of the method 200 and the encoder 103. The encoder 500 partitions an input video signal (e.g., a sub-picture video signal) to produce a partitioned video signal 501 that is substantially similar to the partitioned video signal 401. The partitioned video signal 501 is then compressed and encoded into a bitstream by components of the encoder 500.

[0097] In particular, the partitioned video signal 501 is forwarded to an intra prediction component 517 for intra prediction. The intra prediction component 517 can be substantially similar to the intra estimation component 415 and the intra prediction component 417. The partitioned video signal 501 is also forwarded to a motion compensation component 521 for inter prediction from reference blocks in a decoded picture buffer 523. The motion compensation component 521 can be substantially similar to the motion estimation component 421 and the motion compensation component 419. The predicted and residual blocks from the intra prediction component 517 and the motion compensation component 521 are forwarded to a transform and quantization component 513 to transform and quantize the residual blocks. The transform and quantization component 513 can be substantially similar to the transform scaling and quantization component 413. The transformed and quantized residual blocks and corresponding predicted blocks (and related control data) are forwarded to an entropy encoding component 531 for encoding into the bitstream. The entropy encoding component 531 can be substantially similar to the header formatting and CABAC component 431.

[0098] The transformed and quantized residual blocks and / or corresponding predicted blocks are also forwarded from the transform and quantization component 513 to an inverse transform and quantization component 529 for reconstruction into reference blocks for use by the motion compensation component 521. The inverse transform and quantization component 529 can be substantially similar to the scaling and inverse transform component 429. In accordance with examples, in-loop filters in an in-loop filter component 525 are also applied to the residual blocks and / or the reconstructed reference blocks. The in-loop filter component 525 can be substantially similar to the filter control analysis component 427 and the in-loop filter component 425. In accordance with the in-loop filter component 425, the in-loop filter component 525 can include multiple filters. The filtered blocks are then stored in the decoded picture buffer component 523 for use by the motion compensation component 521 as reference blocks. The decoded picture buffer component 523 can be substantially similar to the decoded picture buffer component 423.

[0099] The encoder 500 receives sub-picture video signals partitioned from a spherical video signal, for use with a VR system that uses view-dependent coding. As described above, when sub-picture video signals are transmitted to a decoder at different resolutions, artifacts can be generated due to data loss in the process of reducing the resolution for lower quality sub-picture video signals. This is because both intra-frame prediction and inter-frame prediction encode a block based on the pixels (picture elements) of neighboring blocks. When a reference block exceeds the boundary of a sub-picture video signal, the reference block can become inaccurate due to the loss of data in neighboring sub-picture video signals. To address these issues, the motion compensation component 521 and the intra-frame prediction component 517 of the encoder 500 encode each sub-picture video signal to be included. Specifically, the motion compensation component 521 and the intra-frame prediction component 517 are configured to only reference integer pixel positions in the same sub-picture video signal when encoding. Specifically, when encoding a first sub-picture video signal, the encoder 500 is prevented from referencing integer pixel positions from other sub-picture video signals. This applies to both intra-frame prediction modes and inter-frame prediction motion vectors.

[0100] As described above, a picture element, also referred to as a pixel, is a value at a corresponding location, where the value can include a luminance / light value and / or one or more chrominance / color values. A recording device takes a picture (and a display displays) by representing integer location values on a sampling grid. An integer location on the sampling grid is referred to as an integer pixel position, because the picture element value is represented at the integer pixel position. A fractional pixel position is at a non-integer location on the sampling grid. Fractional pixel positions can be used when the distance of movement of an object between frames is non-integer, and when filtering is performed. The value of a fractional pixel position can be determined as a weighted value (e.g., a weighted average) based on the values of integer pixel positions. The motion compensation component 521 and the intra-frame prediction component 517 can reference fractional pixel positions in the first sub-picture video signal only when the valid picture element values of the reference fractional pixel positions can be reconstructed by interpolation based on the pixel positions in the first sub-picture bitstream (e.g., without referencing any other sub-picture bitstream).

[0101] Furthermore, the motion compensation component 521 can generate a motion vector candidate list for the first sub-picture bitstream when performing inter-frame prediction. However, the motion compensation component 521 can not include a motion vector in the candidate list when the motion vector is from a block in another sub-picture bitstream. These restrictions ensure that each sub-picture bitstream can be decoded without referencing neighboring sub-picture bitstreams, thus avoiding resolution mismatch situations.

[0102] Furthermore, video encoding can use parallelization (e.g., WPP) to speed up the video encoding process. WPP helps encode a current block (e.g., CTU) as long as the blocks above the current block and the blocks above and to the right of the current block have been decoded. WPP can create waves, where the top row of blocks are encoded two blocks before the second row of blocks, the second row of two blocks are two blocks before the third row, and so on. In an encoder, subpicture coded frame can be treated as a coding block and the coding blocks can be merged to reconstruct the spherical video signal. WPP does not work when there are coding blocks because WPP operates on the entire frame (e.g., a frame in the spherical video signal) at once, not in coding blocks. Accordingly, the encoder 500 can disable WPP when encoding the subpicture coded stream. For example, WPP uses entropy_coding_sync_enabled_flag. The flag is included in the PPS syntax of each picture. The encoder 500 can set entropy_coding_sync_enabled_flag to 0 to disable WPP for the subpicture video signal.

[0103] Furthermore, the encoder 500 can avoid timing mismatch issues between subpicture coded streams by encoding the subpicture video signal in a track and ensuring that the track has the same presentation time. Furthermore, the encoder 500 can ensure that each pixel in a common VR picture (e.g., a frame in the spherical video signal) uses the same picture order count value, even if the pixels are divided into different subpicture coded streams and / or carried in different tracks.

[0104] Figure 6 A block diagram of an example video decoder 600 that can decode subpicture coded streams. The video decoder 600 can be used to implement the decoding functionality of the codec system 400 and / or implement steps 311, 313, 315, and / or 317 of the operating method 300. Furthermore, the decoder 600 can be used to implement steps 211-213 of the method 200 and the decoder 107. For example, the decoder 600 receives a plurality of subpicture coded streams from the encoder 500, generates a reconstructed output video signal that includes a subpicture video signal, merges the subpicture video signal into a spherical video signal, and forwards the spherical video signal for display to a user by a presentation device.

[0105] The bitstream is received by an entropy decoding component 633. The entropy decoding component 633 is configured to implement an entropy decoding scheme, such as CAVLC, CABAC, SBAC, PIPE coding, or other entropy coding techniques. For example, the entropy decoding component 633 can use header information to provide context to interpret other data encoded as codewords in the bitstream. The decoded information includes any information needed to decode the video signal, such as general control data, filter control data, partitioning information, motion data, prediction data, and quantized transform coefficients of a residual block. The quantized transform coefficients are forwarded to the inverse transform and quantization component 629 to be reconstructed into a residual block. The inverse transform and quantization component 629 can be substantially similar to the inverse transform and quantization component 529.

[0106] The reconstructed residual blocks and / or predicted blocks are forwarded to an intra prediction component 617 to be reconstructed into image blocks according to intra prediction operations. The intra prediction component 617 can be substantially similar to the intra estimation component 415 and the intra prediction component 417. Specifically, the intra prediction component 617 uses a prediction mode to locate a reference block in the frame and applies a residual block to the result to reconstruct an intra predicted image block. The reconstructed intra predicted image blocks and / or residual blocks, along with corresponding inter predicted data, are forwarded through an in-loop filter component 625 to a decoded picture buffer component 623, which and the in-loop filter component 625 can be substantially similar to the decoded picture buffer component 423 and the in-loop filter component 425, respectively. The in-loop filter component 625 filters the reconstructed image blocks, residual blocks, and / or predicted blocks, and such information is stored in the decoded picture buffer component 623. The reconstructed image blocks in the decoded picture buffer component 623 are forwarded to a motion compensation component 621 for inter prediction. The motion compensation component 621 can be substantially similar to the motion estimation component 421 and / or the motion compensation component 419. Specifically, the motion compensation component 621 uses motion vectors in a reference block to generate a predicted block and applies a residual block to the result to reconstruct an image block. The resulting reconstructed block can also be forwarded through the in-loop filter component 625 to the decoded picture buffer component 623. The decoded picture buffer component 623 continues to store other reconstructed image blocks, which can be reconstructed into frames through partitioning information. The frames can also be placed in order. The order is output as a reconstructed output video signal to a display.

[0107] As described above, the decoder 600 decodes the sub-picture code streams to create sub-picture video signals. The decoder 600 can then merge the sub-picture video signals into a spherical video signal for display using a lightweight merging algorithm. The lightweight merging algorithm includes selecting a sub-picture from each sub-picture video signal that was produced at a common presentation time and thus is part of the same spherical frame. The sub-pictures are then merged by positioning each sub-picture according to the location associated with the corresponding sub-picture video signal. The merging algorithm can be lightweight because this approach can not require the decoder 600 to consider correlations between the sub-picture video signals. Instead, each sub-picture video signal can be decoded relatively independently and the results can be merged (e.g., a series of filters are applied at the sub-picture boundaries). This lightweight merging algorithm uses less processing resources than an approach that considers correlations between the sub-picture code streams during decoding. Moreover, the lightweight merging algorithm can be performed in real-time during display to the user, rather than in pre-processing before display. Thus, the lightweight merging algorithm can make it possible to perform view-dependent encoding because the view-dependent encoding is performed in real-time during display to the user. Moreover, the lightweight merging algorithm can be performed without artifacts because of the encoding constraints used in the encoder 500 as described above. In particular, the lightweight merging algorithm can be performed because each sub-picture code stream is encoded in a self-contained manner that does not depend on the other sub-picture code streams.

[0108] The foregoing figures describe various components and methods for encoding and decoding sub-picture video signals for VR systems. Figure 7 Exemplary spatial positioning of sub-picture frames mapped from a spherical frame is illustrated. Figures 8 to 11 Mechanisms for performing inter-prediction using motion vectors and candidate list generation are described. These figures illustrate Figure 7 Problems that can arise when motion vectors exceed sub-picture video signal boundaries as described in the Background section, and constraints that can be used to solve these problems are illustrated.

[0109] Figure 7A diagram of a plurality of sub-picture video signals 701 and 703 partitioned from a VR image video stream 700. The VR image video stream 700 can be mapped to the sub-picture video signals 701 and 703 by the encoder 103 and / or the VR encoding device 104. The sub-picture video signals 701 and 703 can be merged back into the VR image video stream 700 by the decoder 107. Thus, the VR image video stream 700 and the sub-picture video signals 701 and 703 illustrate steps 205 and 213 of the method 200. For example, the sub-picture video signals 701 and 703 can be encoded and decoded by inter prediction by the motion estimation component 421, the motion compensation component 419, the motion compensation component 521, and / or the motion compensation component 621. For example, the sub-picture video signals 701 and 703 can also be encoded and decoded by intra prediction by the intra estimation component 415, the intra prediction component 417, the intra prediction component 517, and / or the intra prediction component 617. Thus, the sub-picture video signals 701 and 703 can each be encoded and decoded according to the method 300.

[0110] The VR image video stream 700 includes a plurality of frames presented over time. The frames of the VR image video stream 700 can each be subdivided into sub-frames, also referred to as sub-pictures. Thus, each frame and corresponding sub-pictures include a temporal position (e.g., picture order) as part of the temporal representation. The subdivision is accompanied by a temporal streaming growth in the process of generating the sub-picture video signals 701 and 703. This consistent subdivision results in the sub-picture video signals 701 and 703, where each stream includes a set of sub-pictures having a predetermined size, shape, and spatial position relative to the corresponding frames in the VR image video stream 700. Furthermore, the temporal position of the set of sub-pictures in the sub-picture video signals 701 and / or 703 changes over the presentation time. In this way, the sub-pictures of the sub-picture video signals 701 and 703 can be aligned in the time domain according to the temporal position. The sub-pictures in the sub-picture video signals 701 and 703 at each temporal position can then be merged in the spatial domain according to the predefined spatial position to reconstruct the VR image video stream 700 for display.

[0111] As used herein, the sub-picture video signal 701 can be referred to as a first sub-picture video signal (and / or a first sub-picture bitstream if in an encoding context). The sub-picture video signal 703 is presented to illustrate an adjacent sub-picture video signal (and / or an adjacent sub-picture bitstream) adjacent to the first sub-picture video signal 701. Thus, for the sake of discussing the various embodiments disclosed herein, the sub-picture video signals 701 and 703 are differentiated, otherwise they are substantially similar.

[0112] Figure 8An illustration of an example of a uni-directional inter prediction 800 performed in the block compression step 305, the block decoding step 313, the motion estimation component 421, the motion compensation component 419, the motion compensation component 521, and / or the motion compensation component 621 to determine a motion vector (MV). For example, the uni-directional inter prediction 800 can be used to determine a motion vector for a block of a sub-picture in the sub-picture video signal 701 and / or 703, and thus can be used by the encoder 103 and the decoder 107 when performing steps 207 and 211, respectively, of the method 200.

[0113] The uni-directional inter prediction 800 uses a reference frame 830 having a reference block 831 to predict a current block 811 in a current frame 810. As shown, the reference frame 830 can be positioned in time after the current frame 810 (e.g., as a subsequent reference frame), but in some examples can also be positioned in time before the current frame 810 (e.g., as an earlier reference frame). The current frame 810 is an example frame / image that is being encoded / decoded at a particular time. The current frame 810 includes an object in the current block 811 that matches an object in the reference block 831 of the reference frame 830. The reference frame 830 is a frame used as a reference for encoding the current frame 810, and the reference block 831 is a block in the reference frame 830 that includes an object that is also included in the current block 811 of the current frame 810.

[0114] The current block 811 is any coding unit being encoded / decoded at a given point in the encoding process. The current block 811 can be an entire partitioned block, or a sub-block in the case of affine inter prediction. The current frame 810 is separated from the reference frame 830 by a time distance (TD) 833. The TD 833 indicates an amount of time between the current frame 810 and the reference frame 830 in a video sequence, and can be measured in units of frames. Prediction information for the current block 811 can reference the reference frame 830 and / or the reference block 831 by a reference index that indicates a direction and time distance between the frames. Within the time period represented by the TD 833, an object in the current block 811 moves from one location in the current frame 810 to another location in the reference frame 830 (e.g., the location of the reference block 831). For example, the object can move along a motion trajectory 813, which is a direction in which the object moves over time. A motion vector 835 describes a direction and magnitude of the object’s movement along the motion trajectory 813 within the TD 833. Thus, the encoded motion vector 835 and the reference block 831 provide information sufficient to reconstruct the current block 811 and position the current block 811 in the current frame 810.

[0115] Figure 9An illustration of an example of a bi-directional inter-frame prediction 900 performed in the block compression step 305, the block decoding step 313, the motion estimation component 421, the motion compensation component 419, the motion compensation component 521, and / or the motion compensation component 621 to determine a motion vector (MV). For example, the uni-directional inter-frame prediction 800 can be used to determine a motion vector for a block of a sub-picture in the sub-picture video signal 701 and / or 703, and thus can be used by the encoder 103 and the decoder 107 when performing steps 207 and 211, respectively, of the method 200. It is noted that the encoder can choose to encode a block according to the bi-directional inter-frame prediction 900 or the uni-directional inter-frame prediction 800 (e.g., as part of RDO) depending on which method provides the best encoding efficiency and / or reconstruction quality for the specified block.

[0116] The bi-directional inter-frame prediction 900 is similar to the uni-directional inter-frame prediction 800, but uses a pair of reference frames to predict a current block 911 in a current frame 910. Thus, the current frame 910 and the current block 911 are substantially similar to the current frame 810 and the current block 811, respectively. The current frame 910 is temporally between a preceding reference frame 920 that occurs before the current frame 910 in the video sequence and a subsequent reference frame 930 that occurs after the current frame 910 in the video sequence. The preceding reference frame 920 and the subsequent reference frame 930 are otherwise substantially similar to the reference frame 830.

[0117] The current block 911 is matched to a preceding reference block 921 in the preceding reference frame 920 and to a subsequent reference block 931 in the subsequent reference frame 930. This matching indicates that, over the course of the video sequence, an object moves along a motion trajectory 913 and from a location of the preceding reference block 921 to a location of the subsequent reference block 931 via the current block 911. The current frame 910 is separated from the preceding reference frame 920 by a preceding temporal distance (TD0) 923 and from the subsequent reference frame 930 by a subsequent temporal distance (TD1) 933. The TD0 923 indicates, in units of frames, an amount of time between the preceding reference frame 920 and the current frame 910 in the video sequence. The TD1 933 indicates, in units of frames, an amount of time between the current frame 910 and the subsequent reference frame 930 in the video sequence. Thus, the object moves along the motion trajectory 913 from the preceding reference block 921 to the current block 911 over a time period indicated by the TD0 923. The object also moves along the motion trajectory 913 from the current block 911 to the subsequent reference block 931 over a time period indicated by the TD1 933. Prediction information for the current block 911 can reference the preceding reference frame 920 and / or the preceding reference block 921 and the subsequent reference frame 930 and / or the subsequent reference block 931 through a pair of reference indices that indicate a direction and a temporal distance between the respective frames.

[0118] A previous motion vector (MV0) 925 describes the direction and magnitude of the object's movement along the motion trajectory 913 (e.g., between the previous reference frame 920 and the current frame 910) within the TD0 923. A subsequent motion vector (MV1) 935 describes the direction and magnitude of the object's movement along the motion trajectory 913 (e.g., between the current frame 910 and the subsequent reference frame 930) within the TD1 933. Thus, in bi-directional inter prediction 900, the current block 911 can be encoded and reconstructed by using the previous reference block 921 and / or the subsequent reference block 931, the MV0 925, and the MV1 935.

[0119] As mentioned above, blocks in a sub-picture in the sub-picture video signals 701 and / or 703 can be encoded according to the uni-directional inter prediction 800 and / or the bi-directional inter prediction 900. For example, an object can move between the sub-picture video signal 701 and the neighboring sub-picture video signal 703 over time. In this case, inter prediction can match a pixel (e.g., a pixel) in a current block in the sub-picture video signal 701 with a pixel in a reference block in the neighboring sub-picture video signal 703. The current block can then be encoded as a motion vector that references the pixel in the neighboring sub-picture video signal 703. This approach works fine as long as the sub-picture video signal 701 and the sub-picture video signal 703 are decoded at the same resolution. However, in view-dependent coding, the resolution of the sub-picture video signal 703 can be reduced by a lossy process of removing data to further compress the sub-pictures in the sub-picture video signal 703. Accordingly, the reference block that includes the reference pixel can be changed or even removed from the sub-picture in the sub-picture video signal 703. In this case, the encoded motion vector can still point to the correct reference block, but the pixel included in the reference block can no longer be accurate for the resolution used by the sub-picture video signal 701. Moreover, in some cases, the sub-picture video signal 703 can even be omitted completely, which would result in a motion vector that references empty data. In either of the above cases, the current block decoded from the sub-picture video signal 701 is replaced by a low-quality pixel when available, or even represented in an undefined, erroneous manner depending on the implementation of the decoder. It should be noted that while this problem is discussed from the perspective of inter prediction, the same error can occur in intra prediction as well. The only difference is that in the intra prediction case, the reference pixel is produced at the same temporal location as the current block, whereas in the inter prediction case, the reference pixel is produced at a different temporal location than the current block.

[0120] To avoid these problems, the encoders disclosed herein can be constrained when encoding each sub-picture video signal (e.g., including sub-picture video signals 701 and 703). For example, a block in the first sub-picture video signal 701 can only be encoded with reference to integer pixel positions in the same first sub-picture video signal 701 by motion vectors or intra-prediction modes, and thus cannot reference pixel positions in the other (e.g., neighboring) sub-picture video signal 703. Furthermore, a block in the first sub-picture video signal 701 can only be encoded with reference to fractional pixel positions in the sub-picture video signal 701 if the pixels in the fractional pixel positions are reconstructed by interpolation from pixel positions in the same first sub-picture video signal 701 (e.g., and do not rely in any way on data in the neighboring sub-picture video signal 703). In this way, the pixels in each sub-picture video signal 701 / 703 can be independently coded, and changes in resolution of the neighboring sub-picture video signal 703 do not affect the first sub-picture video signal 701, and vice versa.

[0121] Figure 10 An example of a block-based video encoding mechanism 1000. The mechanism 1000 can be used, for example, for intra-prediction, uni-directional inter-prediction 800, and / or bi-directional inter-prediction 900. Thus, the block-based video encoding mechanism 1000 can be used to determine an intra-prediction mode or a motion vector in the intra-estimation component 415, the intra-prediction component 417, the motion estimation component 421, the motion compensation component 419, the intra-prediction component 517, the motion compensation component 521, the intra-prediction component 617, and / or the motion compensation component 621. For example, the mechanism 1000 can be used when encoding a block of a sub-picture in the sub-picture video signals 701 and / or 703, and thus can be used by the encoder 103 and the decoder 107 when performing steps 207 and 211, respectively, of the method 200.

[0122] The mechanism 1000 encodes a current block 1001 from a previously encoded block 1002 in the same frame. In the case of intra-prediction, the current block 1001 matches the pixels in the encoded block 1002. The current block 1001 can then be encoded by an intra-prediction mode that indicates the pixels in the encoded block 1002 and predicts the pixels in the current block 1001. If the current block 1001 is in the first sub-picture video signal 701 and the encoded block 1002 is in the neighboring sub-picture video signal 703, then view-dependent encoding can produce encoding errors due to the difference in resolution. Thus, the encoders described herein can be limited to using intra-prediction modes in the first sub-picture video signal 701 that do not rely on pixels in the neighboring sub-picture video signal 703.

[0123] The mechanism 1000 can also be used for certain types of inter-prediction. Specifically, inter-prediction can use various modes, such as merge mode and AMVP mode. In both modes, a motion vector for the current block 1001 can be selected based on motion vectors in the coded blocks 1002. In merge mode, the motion vectors selected from the coded blocks 1002 can be included in a candidate list. The candidate list can also include a motion vector indicating no movement, a motion vector for a temporally neighboring block in a previous frame and / or a subsequent frame, a motion vector repeated for a previous coded block, etc. A motion vector for the current block 1001 is then selected from the candidate list. The motion vector for the current block 1001 can then be indicated as a candidate list index. This allows the current block 1001 to be encoded by an index value. A decoder can generate a candidate list for the current block 1001 using the same process as the encoder, and can then determine the motion vector for the current block 1001 based on the candidate list index. AMVP mode is similar to merge mode, but can be used when none of the motion vectors in the candidate list represent the correct motion vector for the current block 1001. In this case, the AMVP mode can select the motion vector from the candidate list that best matches the correct motion vector for the current block 1001. The AMVP mode can then encode the candidate list index for the selected candidate motion vector and a difference value between the selected candidate motion vector and the correct motion vector for the current block.

[0124] Both merge mode and AMVP mode can cause view-dependent encoding errors when the current block 1001 is in the first sub-picture video signal 701 and the coded blocks 1002 are in the neighboring sub-picture video signal 703. For example, when the resolution of the neighboring sub-picture video signal 703 is reduced, the motion vectors for the coded blocks 1002 in the neighboring sub-picture video signal 703 can change. A candidate list will be generated in the decoder for the current block 1001 that references the motion vectors that changed due to the resolution change. However, the candidate list generated in the encoder was generated when the current block 1001 and the coded blocks 1002 used the same resolution. Thus, in this case, the candidate list no longer matches. When the candidate list does not match, the index can point to the wrong candidate motion vector in the decoder. The decoder can be caused to select the wrong reference block for the current block 1001 and thus display the wrong pixel point values for the current block 1001.

[0125] To address these issues, the encoder disclosed herein can be limited such that the candidate motion vector list of the current block 1001 in the first sub-picture video signal 701 does not include the motion vector of the coded block 1002 in the neighboring sub-picture bitstream 703. For example, during the generation of the candidate list, such motion vectors of other neighboring sub-picture bitstreams 703 are not added to the candidate list, and thus cannot be selected by the AMVP mode or the merge mode when performing inter prediction. This limitation, together with the limitation of preventing reference to the pixels in other sub-picture bitstreams 703, ensures that each sub-picture bitstream 701 can be decoded without reference to the neighboring sub-picture bitstreams 703, thus avoiding the resolution mismatch situation.

[0126] Figure 11 An exemplary ISOBMFF 1100 for storing video encodings. For example, the video encodings generated by the encoder 103, the codec system 400, and / or the encoder 500 can be stored in the ISOBMFF 1100 for decoding by the decoder 107, the codec system 400, and / or the decoder 600. Thus, in the methods 200 and 300, the ISOBMFF 1100 can be used to store and transmit data. Moreover, the results of the unidirectional inter prediction 800 and / or the bidirectional inter prediction 900 applied to the sub-pictures in the sub-picture video signals 701 and / or 703, as well as the mechanism 1000, can be stored in the ISOBMFF 1100.

[0127] The ISOBMFF 1100 includes a series of boxes that can include other boxes. A box is an object-oriented building block defined by a unique type identifier and a length. A box is a basic syntax structure in the ISOBMFF 1100, which can include a four-character coded box type, a number of bytes of the box, and a payload. For example, a box can be a syntax structure that includes a coded box type, a number of bytes of the box, and a payload. In some cases, a box can include all data in a file conforming to the High Efficiency Video Coding (HEVC) file format, and no data in the file can be outside of a box. Thus, an ISOBMFF file can include a series of boxes, and a box can include other boxes. For example, the payload of a box can include one or more other boxes.

[0128] Data in the encoded bitstream, including syntax and encoded media samples, can be stored in predetermined boxes in the ISOBMFF 1100. This facilitates a decoder to find relevant data by opening the corresponding box. At the top level, the ISOBMFF 1100 includes a file type (ftyp) box 1101, a movie (moov) box 1103, movie fragment (moof) boxes 1105, and a media data (mdat) box 1107. The moov box 1103 includes a plurality of track (trak) boxes 1113. The moof boxes 1105 include one or more track fragment (traf) boxes 1115. The mdat box 1107 includes one or more media samples 1117. The boxes described above can include other sub-boxes. Note that the ISOBMFF 1100 includes a large number of available boxes. Accordingly, Figure 11 The boxes shown in FIG. 11 are exemplary and are included to illustrate some examples. Examples disclosed herein can also use other ISOBMFF 1100 boxes and / or configurations.

[0129] The ftyp box 1101 includes file compatibility information for the encoding. Typically, the ftyp box 1101 and any sub-boxes include data indicating decoding devices capable of displaying the included video data.

[0130] The moov box 1103 includes functional data related to the encoding. For example, the moov box 1103 can include digital right management (DRM) data for the file. The moov box 1103 can also include overall data indicating how the video should be presented. For example, the moov box 1103 includes a plurality of track boxes 1113. Each trak box 1113 includes data describing a corresponding track. For example, a track can include a subtitle, an audio track, and various representations of a video. For example, a video encoded at multiple resolutions can include a trak box 1113 for each resolution. Accordingly, the trak box 1113 can include data indicating to a decoder how the corresponding track should be played and / or displayed. Furthermore, an encoder and / or associated storage device can send only relevant tracks to a decoder. For example, a decoder can request certain tracks but not others, facilitating the decoder to receive only a particular resolution. A decoder can determine and obtain an appropriate resolution based on the capabilities of a client device and / or network, such as based on user presentation device capabilities, available network bandwidth, etc. In a VR context, different sub-image bitstreams can be stored in different tracks. In this case, the moov box 1103 can include a trak box 1113 for each sub-image bitstream of each representation (e.g., resolution).

[0131] Each track can be divided into a plurality of media segments 1118, also referred to as track segments. A media segment 1118 is a predetermined length of media sequence, such as a three second video segment. Encoding a track as a series of media segments 1118 facilitates a decoder to fetch data from different representations based on user input and / or changing network requirements. In VR-based view-dependent encoding, media segments 1118 facilitate a decoder to fetch different spatial portions of a VR video at different resolutions based on the direction in which a user turns the view of a VR presentation device. Information describing the media segments 1118 is stored in moof boxes 1105 and corresponding traf boxes 1115. For example, a moof box 1105 can include data indicating the start and end points of a media segment 1118 and the location of the media segment 1118 in memory. A traf box 1115 can include timing data related to a media segment 1118. For example, a traf box 1115 can include picture order counts 1116 for each frame in a media segment 1118. The picture order counts 1116 indicate the order of frames in a media segment 1118, so a decoder can use the picture order counts 1116 to position the frames in a media segment 1118 in the proper order for display. A traf box 1115 can also include media presentation time data 1119. The media presentation time data 1119 indicates the length of time a media segment 1118 should be displayed according to the picture order counts 1116, frame rate, etc.

[0132] The mdat box 1107 is a container for the media pixels 1117 referenced by the traf boxes 1115. Thus, the media pixels 1117 are organized into track segments according to time, type (audio, caption, video, etc.), representation (e.g., resolution), and, in the case of view-dependent encoding, spatial location. Thus, the media pixels 1117 include sub-image bitstreams encoded from the sub-image video signals 701 and 703 at various resolutions. These media pixels 1117 can be forwarded to a decoder for presentation and display according to user device (e.g., decoder) requests.

[0133] Another reason for errors in view-dependent encoding is that the presentation time data 1119 and / or the picture order counts 1116 can vary according to resolution. Since this data is for sub-image video signals, such resolution-dependent variations can cause spatial portions of a VR video to be displayed out of order. For example, different sub-image portions of a spherical video frame can be displayed at different rates, causing a user to view portions of a VR video out of order with other portions of the VR video in the same VR frame.

[0134] To address this timing mismatch issue, the encoders herein can be constrained to ensure that each track included in the media pixel points 1117 and described by the traf boxes 1115 and trak boxes 1113 have the same presentation time 1119 regardless of resolution. Further, the media pixel points 1117 of each track are constrained to include the same number of media pixel points 1117 (e.g., the same number of total frames). Further, the media pixel points 1117 of each track segment associated with the same VR frame can use the same picture order count 1116 value. This state can be maintained even when such sub-pictures are partitioned into different sub-picture streams and / or carried in different tracks. When these constraints are enforced, the sub-pictures are displayed in the same order and rate regardless of resolution. Thus, different resolutions can be mixed into one spherical video signal without creating timing errors.

[0135] Further, as described above, WPP can change the way blocks are decoded by changing the decoding order. Errors can arise when applied to the multiple sub-pictures that make up one frame of a VR-based spherical video signal from multiple tracks. The encoders disclosed herein can disable WPP by changing the data in the PPS of each sub-picture. For example, the entropy_coding_sync_enabled_flag in each PPS can be set to 0 to disable WPP for the sub-picture video signal. In some examples, the PPS data is stored in an in-band manner by including the PPS in the media pixel points 1117. In other examples, the PPS data can be stored in the moof 1105, traf boxes 1115, etc.

[0136] Figure 12 A flowchart of an example method 1200 for encoding sub-picture video signals (e.g., sub-picture video signals 701 and 703 partitioned from a spherical video signal for use as VR video displayed through a perspective). The method 1200 can encode these sub-picture video signals in the ISOBMFF 1100. The method 1200 can be used by an encoder (e.g., the encoder 103, the codec system 400, and / or the encoder 500) to encode sub-picture video signals for perspective-dependent encoding by a decoder (e.g., the decoder 107, the codec system 400, and / or the decoder 600). Further, to support the methods 200 and 300, the perspective-dependent encoding can be implemented using the method 1200. Further, the method 1200 can use the uni-directional inter-prediction 800, the bi-directional inter-prediction 900, and / or the mechanism 1000 in the perspective-dependent encoding without creating visual artifacts due to sub-picture video signal boundaries.

[0137] In step 1201, a spherical video signal is received. The spherical video signal can be stitched together from multi-directional video streams acquired from a multi-directional camera array, such as multi-directional camera 101.

[0138] In step 1203, the spherical video signal is mapped into a plurality of sub- picture video signals, such as sub-picture video signals 701 and 703. Each sub-picture video signal includes a sequence of sub-pictures of the spherical video signal. For example, each sub-picture video signal can include a specified spatial portion of each frame of the spherical video signal, and thus can include a constant spatial portion of the spherical video signal over the length of the video stream (e.g., in the time domain). It is noted that mapping the spherical video signal into the plurality of sub-picture video signals can include setting a width and a height for each sub-picture video signal in the spatial domain. In some examples, the setting of the width is constrained such that the width is an integer multiple of 64 pixels and the width is greater than or equal to 256 pixels. Further, in some examples, the setting of the height is constrained such that the height is an integer multiple of 64 pixels and the height is greater than or equal to 256 pixels.

[0139] In steps 1205-1213, the plurality of sub-picture video signals is encoded into a plurality of sub-picture bitstreams according to several constraints. In step 1205, the plurality of sub-picture video signals can be encoded into a plurality of sub-picture bitstreams. The encoding operation can be performed such that, when decoded in a decoder, the value of each pixel in each sub-picture is equal to the value of the corresponding pixel in the entire picture composed of the sub-pictures after decoding. For example, the encoding operation can be accomplished by steps 1207-1213. Further, as part of the encoding process, WPP can be selectively disabled. An example mechanism to disable WPP includes setting entropy_coding_sync_enabled_flag to 0. entropy_coding_sync_enabled_flag is included in the PPS for the spherical video signal and / or in the PPS for the sub-picture bitstreams.

[0140] In step 1207, the plurality of sub-picture video signals is encoded, at least in part, through inter-prediction. In inter-predicting each current sub-picture of each sub-picture video signal, a motion vector is selected. However, the selection is constrained to only select motion vectors that refer to pixel points in the same sub-picture video signal as the current sub-picture. Thus, the selected motion vectors do not refer to pixel points in sub-pictures that are not included in the same sub-picture video signal as the current sub-picture.

[0141] In step 1209, the selection of motion vectors referring to the same sub-picture video signal as the current sub-picture is further restricted to include the selection of motion vectors referring to integer pixel positions in the same sub-picture video signal. It is possible to select motion vectors referring to fractional pixel positions in the same sub-picture video signal. However, this selection is restricted to only select fractional pixel positions that can be reconstructed by using only integer pixel positions in the same sub-picture video signal for interpolation.

[0142] In step 1211, the encoding of the plurality of sub-picture video signals by inter prediction is further restricted. In particular, a motion vector can be selected for each current frame sub-picture using inter prediction. However, when using a list of candidate motion vectors, only candidate motion vectors of blocks in the same sub-picture video signal as the current sub-picture are added to the candidate list. Thus, motion vectors of blocks in other sub-picture video signals are not added to the candidate list and are not selected for the current sub-picture.

[0143] In step 1213, the sub-picture streams created by encoding the sub-picture video signals further constitute a plurality of sub-picture tracks, e.g. in ISOBMFF. When using view dependent encoding, the sub-picture tracks are further restricted to prevent timing mismatches. In particular, the sub-picture tracks can be restricted to each include the same number of media pixels. This can be achieved by ensuring that all sub-picture video signals have the same number of sub-pictures, regardless of the resolution, even if the sub-picture video signals are included in different sub-picture tracks. Furthermore, it can be restricted that all tracks include a common presentation time. This can be achieved by ensuring that all tracks have the same frame rate, the same number of frames, and / or all tracks play over the same duration, regardless of the resolution. It can be restricted that the sub-picture tracks use consistent picture order count values. In particular, a picture order count value can be assigned to each sub-picture in each sub-picture video signal. Furthermore, the same picture order count value can be assigned to sub-pictures in the spherical video signal that are associated with the same frame, regardless of the resolution. This ensures that the media pixels have a common picture order count value across all sub-picture tracks. Due to the various timing mismatches related to inter prediction and cross errors based on the sub-picture video signals that are avoided by steps 1205 to 1213, it is possible to include an indication in the sub-picture tracks that the sub-picture tracks comply with a certain video profile, e.g. the OMAF video profile.

[0144] In step 1215, the set of sub-picture tracks is sent to a decoder to support decoding and display as a perspective-dependent virtual reality video stream through a virtual reality perspective. As described above, the sub-picture tracks can be selected and transmitted for different spatial portions of the spherical video signal of different resolutions, and even some spatial portions can be omitted. Advantageously, spatial portions in and around the user's perspective in the VR video stream / signal can be transmitted at high resolution, while spatial portions outside the user's perspective (less likely to be watched) are omitted and / or transmitted at lower resolution. This approach results in a significant reduction in file size sent to the user in the context of a video stream. Furthermore, the constraints adopted above ensure that each sub-picture video signal / stream is encoded in a self-contained manner and can be decoded without artifacts regardless of the resolution difference between the sub-picture video signals.

[0145] It should be noted that perspective-dependent encoding can be implemented according to the omnidirectional media format (OMAF) controlled by the motion picture experts group (MPEG). The method 1200 can be implemented by including the following in the OMAF-related standard document.

[0146] When the untransformed pixel entry type is "hvc2", the track shall include one or more "scal" reference tracks. The referenced tracks shall conform to the HEVC-based perspective-independent OMAF video profile or the HEVC-based perspective-dependent OMAF video profile, and for each sub-picture stream carried in the referenced tracks, the following constraints apply. (1) Inter prediction shall not reference pixel values outside the picture. (2) For the last prediction unit located outside the right picture boundary to the right of the bottom-right corner of the picture, when CuPredMode[xPb][yPb] is equal to MODE INTER, the following applies, where (xPb, yPb) specifies the relationship of the top-left pixel of the corresponding luma prediction block relative to the top-left pixel of the current picture. The derivation of the number of candidate spatial merge candidates numSpatialMergeCand is as follows:

[0147] numSpatialMergeCand = availableFlagA0 + availableFlagA1 + availableFlagB0 + availableFlagB1 + availableFlagB2

[0148] availableFlagB1 + availableFlagB2

[0149] When availableFlagA0, availableFlagA1, availableFlagB0, availableFlagB1 and availableFlagB2 are the outputs of the derivation process of candidate spatial merge as specified in ISO / IEC 23008-2 clause 8.5.3.2.3, the following applies. If numSpatialMergeCand is equal to 0, merge_flag[ xPb ][ yPb ] shall be equal to 0. Otherwise (numSpatialMergeCand is greater than 0), merge_idx[ xPb ][ yPb ] shall be in the range of 0 to numSpatialMergeCand - 1, inclusive. The derivation of the number of candidate spatial motion vector predictors numSpatialMvpCand is as follows:

[0150]

[0151] where availableFlagLXA, availableFlagLXB, mvLXA and mvLXB are the outputs from the derivation process of candidate motion vector predictors from neighboring prediction units as specified in ISO / IEC 23008-2 clause 8.5.3.2.7, the following applies. If numSpatialMvpCand is equal to 0, mvp_l0_flag[ xPb ][ yPb ] and mvp_l1_flag[ xPb ][ yPb ] shall be equal to 1. Otherwise (numSpatialMvpCand is greater than 0), mvp_l0_flag[ xPb ][ yPb ] and mvp_l1_flag[ xPb ][ yPb ] shall be in the range of 0 to numSpatialMvpCand - 1, inclusive.

[0152] NOTE 1: The first constraint restricts the motion vector to point to integer pixel positions in the picture and to fractional pixel positions that only require integer pixel positions in the picture for interpolation. It should be noted that in the above constraint and in the previous sentence, the subpicture becomes the picture in the context of a subpicture bitstream. The second constraint restricts the decoding of the entire reconstructed bitstream when the subpicture bitstream of the reference track is reconstructed together with other subpicture bitstreams carried on other reference tracks into a conforming bitstream. For blocks of the subpicture of this subpicture bitstream, there is no candidate motion vector for temporal motion vector prediction derived from blocks outside the "subpicture". (3) The value of entropy_coding_sync_enabled_flag in the primary PPS shall be equal to 0.

[0153] In addition, these constraints also apply to the referenced tracks. (1) The tracks should include the same number of media samples. (2) The presentation time of the samples with the same sample number on the tracks should be the same. (3) The picture order count value, e.g., PicOrderCntVal, carried in the samples with the same sample value on the tracks should be the same. In some examples, the following constraints also apply. (4) The width and height of each sub-picture must be an integer multiple of 64, and the width is not less than 256 and the height is not less than 64.

[0154] Figure 13 FIG. 13 shows a schematic diagram of an example video encoding device 1300. The video encoding device 1300 is suitable for implementing the disclosed examples / embodiments described herein. The video encoding device 1300 includes a downstream port 1320, an upstream port 1350, and / or a transceiver unit (Tx / Rx) 1310 including a transmitter and / or a receiver for data communication upstream and / or downstream over a network. The video encoding device 1300 also includes a processor 1330 including a logic unit and / or a central processing unit (CPU) for processing data, and a memory 1332 for storing data. The video encoding device 1300 can also include an optical-to-electrical (OE) component, an electrical-to-optical (EO) component, a wireless communication component coupled to the upstream port 1350 and / or the downstream port 1320 for data communication over an optical or wireless communication network. The video encoding device 1300 can also include an input and / or output (I / O) device 1360 for transmitting data to and from a user. The I / O device 1360 can include an output device such as a display for displaying video data, a speaker for outputting audio data. The I / O device 1360 can also include an input device such as a keyboard, a mouse, a trackball, and / or a corresponding interface for interacting with such output devices.

[0155] The processor 1330 is implemented by hardware and software. The processor 1330 can be implemented as one or more CPU chips, cores (e.g., as a multi-core processor), field-programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), and digital signal processors (DSPs). The processor 1330 is in communication with the downstream port 1320, the Tx / Rx 1310, the upstream port 1350, and the memory 1332. The processor 1330 includes an encoding module 1314. The encoding module 1314 implements the above-disclosed embodiments, such as the methods 200, 300, 1200, and / or combinations thereof. For example, the encoding module 1314 can implement an encoder, such as the encoder 103, the codec system 400, and / or the encoder 500, to encode sub-picture video signals, such as the sub-picture video signals 701 and 703, with view-dependent coding. As another example, the encoding module 1314 can implement a decoder, such as the decoder 107, the codec system 400, and / or the decoder 600, to decode sub-picture video signals. The encoding module 1314 can encode / decode these sub-picture video signals in the ISOBMFF 1100. Further, the encoding module 1314 can use the uni-directional inter prediction 800, the bi-directional inter prediction 900, and / or the mechanism 1000 in view-dependent coding without producing visual artifacts due to sub-picture video signal boundaries.

[0156] In particular, the encoding module 1314 can encode and / or decode the sub-picture video signal according to the constraints to prevent visual artifacts. For example, the encoding module 1314 can encode the sub-picture video signal such that a block only references integer pixel positions in the same sub-picture video signal and thus cannot reference pixel positions in other sub-picture video signals. The encoding module 1314 can use fractional pixel positions in a sub-picture video signal to encode a block only if the pixel in the fractional pixel position can be recreated by interpolation from only the fractional pixel positions in the same sub-picture bitstream. In addition, the encoding module 1314 can ensure that the motion vectors of blocks in neighboring sub-picture bitstreams are not included in the list of candidate motion vectors. The encoding module 1314 can apply these constraints to ensure that each sub-picture bitstream can be decoded without reference to neighboring sub-picture bitstreams, thus avoiding resolution mismatch. For example, the encoding module 1314 can also stop WPP by setting the entropy_coding_sync_enabled_flag in each PPS to 0. In addition, the encoding module 1314 can prevent temporal mismatch between sub-picture bitstreams by ensuring that the presentation times of tracks including sub-picture video signals are the same, ensuring that the tracks have the same number of media pixels (e.g., frames), and ensuring that the picture order count values of each pixel in the common VR image are equal. Thus, the encoding module 1314 enables the video encoding device 1300 to operate with higher encoding efficiency and / or use less network resources by using view-dependent encoding without producing visual artifacts and / or degrading the visual quality of the VR video stream displayed to users by a presentation device. Accordingly, the encoding module 1314 improves the functionality of the video encoding device 1300 and solves problems with video encoding techniques. Moreover, the encoding module 1314 transforms the video encoding device 1300 to a different state. Alternatively, the encoding module 1314 can be implemented as instructions stored in memory 1332 and executed by the processor 1330 (e.g., a computer program product stored on a non-transitory medium).

[0157] The memory 1332 includes one or more memory types such as disk, tape drive, solid state drive, read only memory (ROM), random access memory (RAM), flash memory, ternary content-addressable memory (TCAM), static random-access memory (SRAM), etc. The memory 1332 can be used as an over-flow data storage device, to store programs when such programs are selected for execution, and to store instructions and data read from a program as such program is being executed.

[0158] Figure 14 FIG. 13 is a schematic diagram of an exemplary video encoding device 1400 that can encode sub-picture bitstreams. The video encoding device 1400 implements the embodiments disclosed above, such as the encoding module 1314, the methods 200, 300, 1200, and / or combinations thereof. For example, the encoding module 1314 of the video encoding device 1400 can implement an encoder, such as the encoder 103, the codec system 400, and / or the encoder 500, to encode sub-picture video signals, such as the sub-picture video signals 701 and 703, using view-dependent encoding. The video encoding device 1400 can encode these sub-picture video signals in the ISOBMFF 1100. Moreover, the video encoding device 1400 can use the uni-directional inter prediction 800, the bi-directional inter prediction 900, and / or the mechanism 1000 in the view-dependent encoding without producing visual artifacts at the boundaries of the sub-picture video signals.

[0159] The video encoding device 1400 includes a video signal receiving module 1401 configured to receive a spherical video signal that is stitched from a multi-directional video signal. The video encoding device 1400 further includes a mapping module 1403 configured to map the spherical video signal into a plurality of sub-picture video signals, each of which includes a sequence of sub-pictures. The video encoding device 1400 further includes an encoding module 1405 configured to encode the plurality of sub-picture video signals into a plurality of sub-picture bitstreams such that, when decoded in a decoder, each pixel in each sub-picture has a value equal to a value of a corresponding pixel in an entire picture composed of the sub-picture after decoding. The video encoding device 1400 further includes a

[0160] The video encoding device 1400 further includes a composing module 1406 configured to compose the plurality of sub-picture bitstreams into a plurality of sub-picture tracks and indicate that the sub-picture tracks conform to a particular video profile. The video encoding device 1400 further includes a sending module 1407 configured to send a set of the plurality of sub-picture tracks to a decoder to support decoding and displaying a virtual reality video view.

[0161] Figure 15A flowchart of another example method 1500 of encoding subpicture code streams. According to method 1500, in step 1501, an encoder encodes pictures in subpicture code streams in a track by ensuring that inter prediction does not reference pixel point values outside the subpicture. Further, in step 1503, the encoder encodes pictures in subpicture code streams in a track by limiting motion vectors to point to integer pixel point positions within the subpicture and using only integer pixel point positions within the subpicture for interpolation of fractional pixel point positions. Further, in step 1505, the encoder encodes pictures in subpicture code streams in a track by limiting inter prediction such that candidate motion vectors for temporal motion vector prediction are not derived from blocks outside the subpicture. It should be noted that steps 1501, 1503, and 1505 are described for clarity of discussion. However, steps 1501, 1503, and 1505 can be performed in any order, including simultaneously and / or substantially simultaneously. Also, encoding pictures in subpicture code streams in a track can further include that when there are multiple coded slices in a first subpicture code stream, the entropy_coding_sync_enabled_flag value in the PPS is 0 for each subpicture code stream carried on the corresponding track. Further, encoding pictures in subpicture code streams in a track can further include requiring that the track include the same number of media pixel points. Further, encoding pictures in subpicture code streams in a track can further include requiring that the presentation time of pixel points with the same pixel point number be the same across multiple tracks. Further, encoding pictures in subpicture code streams in a track can further include that the value of the picture order count of subpictures carried in media pixel points with the same pixel point number on the track are equal.

[0162] A first component is directly coupled to a second component when the first component and the second component are connected to each other without any intervening components between the first component and the second component. A first component is indirectly coupled to a second component when there are one or more intervening components between the first component and the second component. The term "coupled" and variations thereof are used herein to express a relationship between or among such components as directly and / or indirectly connected to each other. The term "about" is used in connection with a numerical value to mean plus or minus 10% of the numerical value.

[0163] While the present application has been presented with a number of embodiments, it is to be understood that the disclosed system and method can be embodied through other specific forms without departing from the spirit or essential characteristics thereof. The examples given are intended to be illustrative and not restrictive, and the application is not to be limited to the details given herein. For example, various elements or components can be combined or integrated within another system, or certain features can be omitted or not implemented.

[0164] In addition, techniques, systems, subsystems, and methods described and illustrated in the various embodiments as discrete or separate can be combined or integrated with other systems, components, techniques, or methods without departing from the scope of the present disclosure. Other changes, modifications, substitutions, and alternatives will occur to readers of this document and it is understood that equivalents will be considered as within the spirit and scope of the disclosure.

Claims

1. A method implemented in a video encoding device, the method comprising: The method comprises: The processor of the video coding device receives a spherical video signal spliced by a multi-directional video signal; The processor maps the spherical video signal into a plurality of sub-picture video signals, each of which comprises a sub-picture sequence; The processor encodes the plurality of sub-picture video signals into a plurality of sub-picture code streams, so that when decoded in a decoder, the value of each pixel point in each sub-picture is equal to the value of the corresponding pixel point in the entire image composed of the sub-pictures after decoding, and in the case of encoding the plurality of sub-picture video signals by inter-prediction, wavefront parallel processing (WPP) is disabled; The processor groups the plurality of sub-picture code streams into a plurality of sub-picture tracks, and indicates that the sub-picture tracks conform to a specific video profile; The transmitter of the video coding device transmits the set of the plurality of sub-picture tracks to a decoder to support decoding and displaying a virtual reality video view.

2. The method of claim 1, wherein, Encoding the plurality of sub-picture video signals into the plurality of sub-picture code streams comprises partially encoding the sub-picture code streams by inter-prediction by selecting only motion vectors referring to pixel points in the same sub-picture video signal as the current sub-picture for each current sub-picture.

3. The method according to claim 1 or 2, characterized in that, The specific video profile is based on a High Efficiency Video Coding (HEVC) view-dependent Omnidirectional MediA Format (OMAF) video profile.

4. The method according to claim 1 or 2, characterized in that, Selecting motion vectors referring to pixel points in the same sub-picture video signal as the current sub-picture comprises: Selecting motion vectors referring to integer pixel point positions in the same sub-picture video signal as the current sub-picture; Selecting motion vectors referring to fractional pixel point positions, which are only interpolated using integer pixel point positions in the same sub-picture video signal as the current sub-picture.

5. The method according to claim 1 or 2, characterized in that, Encoding the plurality of sub-picture video signals by inter-prediction further comprises, for each current sub-picture, selecting only candidate motion vectors from blocks in the same sub-picture video signal as the current sub-picture.

6. The method of claim 1 or 2, wherein, Disabling WPP comprises setting entropy_coding_sync_enabled_flag to 0, wherein the entropy_coding_sync_enabled_flag is included in a picture parameter set (PPS) for the spherical video signal.

7. The method of claim 1 or 2, wherein, Restricting the sub-picture tracks can be that each sub-picture track comprises the same number of media pixels.

8. The method of claim 1 or 2, wherein, Restricting the sub-picture tracks can also be that the sub-picture tracks comprise a common presentation time.

9. The method of claim 1 or 2, wherein, Restricting the sub-picture tracks can also be that the sub-picture tracks are restricted such that media pixels with the common presentation time are designated to have a normal picture order count value on the sub-picture tracks.

10. The method of claim 1 or 2, wherein, Mapping the spherical video signal to the plurality of sub-picture video signals comprises setting a width and a height for each sub-picture video signal, wherein the width is an integer multiple of 64 pixels, the width is greater than or equal to 256 pixels, and the height is greater than or equal to 256 pixels.

11. A video encoding apparatus, comprising: Comprise: A processor configured to: receive a spherical video signal stitched from a multi-directional video signal; map the spherical video signal into a plurality of sub-picture video signals, each sub-picture video signal comprising a sequence of sub-pictures; encode the plurality of sub-picture video signals into a plurality of sub-picture bitstreams such that, when decoded in a decoder, a value of each pixel in each sub-picture is equal to a value of a corresponding pixel in an entire picture composed of the sub-pictures after decoding, and, in a case that the plurality of sub-picture video signals are encoded by inter prediction, wavefront parallel processing (WPP) is disabled; compose the plurality of sub-picture bitstreams into a plurality of sub-picture tracks, and indicate that the sub-picture tracks conform to a specific video profile; a transmitter coupled to the processor, the transmitter configured to: transmit the set of the plurality of sub-picture tracks to a decoder to support decoding and displaying a virtual reality video view.

12. The video coding device of claim 11, wherein, Encoding the plurality of sub-picture video signals into the plurality of sub-picture bitstreams comprises partially encoding the sub-picture bitstreams by selecting, for each current sub-picture, only motion vectors that refer to pixels in a same sub-picture video signal as the current sub-picture.

13. The video coding device of any of claims 11 or 12, wherein, The specific video profile is based on a High Efficiency Video Coding (HEVC) view-dependent Omnidirectional MediA Format (OMAF) video profile.

14. The video coding device of any of claims 11 or 12, wherein, Selecting, for each current sub-picture, only motion vectors that refer to pixels in a same sub-picture video signal as the current sub-picture comprises: selecting motion vectors that refer to integer pixel positions in the same sub-picture video signal as the current sub-picture; selecting motion vectors that refer to fractional pixel positions, the fractional pixel positions being interpolated using only integer pixel positions in the same sub-picture video signal as the current sub-picture.

15. The video coding device of any of claims 11 or 12, wherein, Encoding the plurality of sub-picture video signals by inter prediction further comprises, for each current sub-picture, selecting only candidate motion vectors from blocks in a same sub-picture video signal as the current sub-picture.

16. The video coding device of any of claims 11 or 12, wherein, Disabling WPP includes setting entropy_coding_sync_enabled_flag to 0, wherein the entropy_coding_sync_enabled_flag is included in a picture parameter set (PPS) for the spherical video signal; the constraint on the sub-picture tracks can be that each sub-picture track includes the same number of media samples; the constraint on the sub-picture tracks can also be that the sub-picture tracks include a common presentation time; the constraint on the sub-picture tracks can also be that the sub-picture tracks are constrained such that media samples having the common presentation time are designated to have a normal picture order count value on the sub-picture tracks; and mapping the spherical video signal to the plurality of sub-picture video signals includes setting a width and a height for each sub-picture video signal, wherein the width is an integer multiple of 64 pixels, the width is greater than or equal to 256 pixels, and the height is greater than or equal to 256 pixels.

17. A video encoding apparatus, comprising: The method comprises: receiving a spherical video signal spliced from a multi-directional video signal; mapping the spherical video signal into a plurality of sub-picture video signals, each of which includes a sub-picture sequence; encoding the plurality of sub-picture video signals into a plurality of sub-picture bitstreams, such that when decoded in a decoder, the value of each pixel in each sub-picture is equal to the value of the corresponding pixel in the entire image composed of the sub-pictures after decoding, and in the case of encoding the plurality of sub-picture video signals through inter-prediction, wavefront parallel processing (WPP) is disabled; composing the plurality of sub-picture bitstreams into a plurality of sub-picture tracks, and indicating that the sub-picture tracks conform to a specific video profile; sending the set of the plurality of sub-picture tracks to a decoder to support decoding and displaying a virtual reality video perspective.

18. The video coding device of claim 17, wherein, The video signal receiving module, the mapping module, the encoding module, and the sending module are further configured to perform the method of any one of claims 2 to 10.

19. A non-transitory computer readable medium, characterized in that, A computer program product used by a video encoding device, wherein the computer program product includes computer executable instructions stored in the non-transitory computer readable medium, and when a processor executes the computer executable instructions, the video encoding device performs the method of any one of claims 1 to 10.

Images