Method and apparatus for encoding and decoding 360-degree virtual reality video sequences

Abstract

A method and apparatus for encoding and decoding a VR360 video sequence are disclosed, wherein surround motion compensation is included as a coding and decoding tool. According to the method, a bitstream corresponding to the encoded data of the VR360 video sequence is generated at the encoder side or received at the decoder side, wherein the bitstream includes one or more PPS syntaxes related to surround motion compensation information in a picture parameter set (PPS). Based on the surround motion compensation information, the VR360 video sequence is encoded at the encoder side or decoded at the decoder side.

Description

[0001] Related references

[0002] This application claims priority to U.S. Provisional Application No. 62 / 935,665 filed on November 15, 2019 and U.S. Provisional Application No. 62 / 941,934 filed on November 29, 2019, respectively. The entire contents of the above are incorporated herein by reference. Technical Field

[0003] The present invention relates to image processing of 360-degree virtual reality (VR360) images, and more particularly, to a method and apparatus for encoding and decoding a 360-degree virtual reality video sequence. Background Art

[0004] 360-degree video (also known as immersive video) is an emerging technology that provides a "feeling of being present". Immersion can be achieved by surrounding the user with a surround scene that covers a panoramic view (especially a 360-degree field of view). Stereoscopic presentation can further improve the "feeling of being present". Therefore, panoramic video is widely used in virtual reality (VR) applications.

[0005] Immersive video involves the use of multiple cameras to capture a scene to cover a panoramic view, such as a 360-degree field of view. Immersive video typically uses a panoramic camera or a set of cameras to capture a 360-degree field of view. Typically, immersive video uses two or more cameras. All videos must be shot simultaneously, and separate segments of the scene (also called separate perspectives) are recorded. In addition, the camera set is usually arranged to capture the view horizontally, but other arrangements of the cameras are also possible.

[0006] 360-degree virtual reality (VR) images can be acquired using a 360-degree spherical panoramic camera or arranged as multiple images covering all 360-degree fields of view. Three-dimensional (3D) spherical images are difficult to process or store using conventional image / video processing equipment. Therefore, 360-degree VR images are usually converted into two-dimensional (2D) formats using 3D to 2D projection methods (e.g., EquiRectangular Projection (ERP) and CubMapProjection (CMP)). In addition to the ERP and CMP projection formats, there are many other VR projection formats, such as Octahedron Projection (OHP), icosahedron (ISP), SegmentedSphere Projection (SSP), and RotatedSphere Projection (RSP), which have been widely used in the field.

[0007] VR360 video sequences usually require more storage space than conventional 2D video sequences. Therefore, video compression is usually applied to VR360 video sequences to reduce the storage space for storage or the bit rate for streaming / transmission.

[0008] The High Efficiency Video Coding (HEVC) standard was developed under a joint video project of the ITU-T Video Coding Experts Group (VCEG) and the ISO / IEC Moving Picture Experts Group (MPEG) standardization organizations, especially in collaboration with a group called the Joint Collaborative Team on Video Coding (JCT-VC). VR360 video sequences can be encoded and decoded using HEVC. An emerging video coding standard development called "Versatile Video Coding (VVC)" also includes coding and decoding technology for VR360 video sequences. As described below, VVC supports reference image resampling.

[0009] Reference image resampling

[0010] During the development of VVC, according to the "Requirements for future video codec standards", "the standard should support fast representation switching (such as spatial resolution or sampling bit depth) in the case of adaptive streaming services that provide multiple representations of the same content, each with different properties." In real-time video communications, changing the resolution in a coded video sequence without inserting I pictures can not only enable the video data to seamlessly adapt to dynamic channel conditions or user preferences, but also eliminate the jitter effect caused by I pictures. Figure 1 A hypothetical example of Adaptive Resolution Change (ARC) with Reference Picture Resampling (RPR) is shown in FIG, where a current image (110) is predicted based on reference images (Ref0 120 and Ref1 130) of different sizes. Figure 1 As shown, the reference image Ref0 (120) has a lower resolution than the current image (110). In order to use the reference image Ref0 as a reference, Ref0 must be scaled up to the same resolution as the current image. The reference image Ref1 (130) has a higher resolution than the current image (110). In order to use the reference image Ref1 as a reference, Ref1 must be scaled down to the same resolution as the current image.

[0011] To support spatial scalability, the image size of the reference image can be different from the current image, which is useful for streaming applications. Methods for supporting Reference Picture Resampling (RPR), also known as Adaptive Resolution Change (ARC), have been studied for inclusion in the VVC specification. At the 14th JVET meeting in Geneva, some papers on RPR were presented and discussed at the meeting.

[0012] Horizontal surround motion compensation

[0013] Horizontal surround motion compensation has been proposed for inclusion in VTM7 (J. Chen, et al., Algorithm description for Versatile Video Coding and Test Model 7 (VTM 7), Joint Video Experts Team (JVET) of ITU-T SG 16WP 3 and ISO / IEC JTC 1 / SC 29 / WG 11 16th Meeting: Geneva, CH, 1–11 Oct. 2019, Document: JVET-P2002). Horizontal surround motion compensation is a 360-degree specific codec tool designed to improve the visual quality of reconstructed 360-degree videos in equirectangular projection (ERP) format. In conventional motion compensation, when a motion vector refers to a sample outside the image boundary of a reference image, repeated padding is applied to derive the value of the out-of-bounds sample by copying the nearest neighbor sample corresponding to the image boundary. For 360-degree video, this repeated padding method is not suitable and may cause visual artifacts known as "seam artifacts" in the reconstructed viewport video. Because 360-degree video is acquired on a sphere and inherently has no "borders", reference samples outside the borders of the reference image in the projection domain can always be obtained from neighboring samples in the spherical domain. For general projection formats, it may be difficult to derive corresponding neighboring samples in the spherical domain because it involves 2D to 3D and 3D to 2D coordinate conversions, as well as sample interpolation of fractional sample positions. For the left and right borders of the ERP projection format, this problem is much simpler because the spherical neighboring samples outside the left image border can be obtained from samples within the right image border, and vice versa. In view of the widespread use of the ERP projection format and its relatively easy implementation, horizontal surround motion compensation is adopted in VTM7 to improve the visual quality of 360 videos encoded and decoded in the ERP projection format.

[0014] exist Figure 2A In FIG. 2 , an example of a VR360 frame 210 corresponding to a world map in ERP format is shown. If the frame is viewed as a regular 2D image, the reference block 220 covers an area 222 outside the frame boundary 224. This external reference area will be considered unusable. Figure 2A As shown in area 230 of FIG. 1 , according to conventional motion compensation, unavailable reference data may be generated using repeated padding, which may result in seam artifacts. Figure 2BAs shown in the area 250 of , the surround motion compensation adopted in VTM7 can use horizontal surround to generate unavailable reference data. Therefore, the unavailable area of ​​the reference block 220 now uses the surround reference data 242 to achieve appropriate motion compensation.

[0015] An example of the horizontal surround motion compensation process is shown in Figure 3 When a portion of the reference block is outside the left (or right) border of the reference image in the projection domain, instead of repeated padding, the "out-of-bounds" portion is taken from the corresponding spherical neighboring area, which is located inside the reference image towards the right (or left) border in the projection domain. Repeat padding is only used for top and bottom image boundaries. Figure 3 As shown, the current image 310 is filled on the left boundary (314) and the right boundary (312) of the ERP image (the area between the left boundary (314) and the right boundary (312)). Similarly, the reference image 320 is filled on the left boundary (324) and the right boundary (322) of the ERP image (the area between the left boundary (324) and the right boundary (322)). Block 330 corresponds to the current CU of the current image 310 and block 340 corresponds to the co-located CU of the reference image 320. A motion vector (MV) is used to locate the reference block 342, wherein a portion of the reference block (i.e., the area 344 filled with diagonal lines) is outside the reference image. The out-of-bounds area 344 can be generated from the surrounding reference block 346, wherein the surrounding reference block 346 is located by horizontally moving the out-of-bounds area 344 by the ERP width.

[0016] like Figure 3 As shown, horizontal surround motion compensation can be combined with non-standard padding methods often used in 360-degree video encoding and decoding. In VVC, this is achieved by sending high-level syntax elements to indicate the surround offset, which should be set to the ERP image width before padding. The syntax is used to adjust the position of the horizontal surround accordingly. This syntax is not affected by the specific amount of padding on the left and right image boundaries. Therefore, the syntax naturally supports asymmetric padding of ERP images to allow different left and right padding. When the reference sample is outside the left or right boundary of the reference image, horizontal surround motion compensation provides more meaningful information for motion compensation. Under the common test condition (CTC) of 360 videos, this tool not only improves compression performance in terms of rate distortion performance, but also reduces seam artifacts and improves the subjective quality of reconstructed 360-degree videos. Improved compression performance. Horizontal surround motion compensation can also be used for other single-sided projection formats with constant sampling density in the horizontal direction, such as the adjusted equal-area projection in 360Lib.

[0017] The present invention solves the problems associated with sending surround motion compensation information. Summary of the invention

[0018] A method and apparatus for encoding a VR360 video sequence is disclosed, wherein surround motion compensation is included as a coding and decoding tool. According to the method, a bitstream corresponding to the encoded data of the VR360 video sequence is generated at the encoder side or received at the decoder side, wherein the bitstream includes one or more PPS syntaxes related to surround motion compensation information in a picture parameter set (PPS). Based on the surround motion compensation information, the VR360 video sequence is encoded at the encoder side or decoded at the decoder side.

[0019] In one embodiment, the PPS syntax includes a first PPS syntax corresponding to a PPS flag indicating whether wraparound motion compensation is used for a target image. For example, the first PPS syntax may be specified as pps_ref_wraparound_enabled_flag. In another embodiment, when the first PPS syntax indicates that wraparound motion compensation is used for a target image, a second PPS syntax is included in the bitstream, wherein the second PPS syntax is related to a value of a wraparound offset. The second PPS syntax may represent a value of a wraparound motion compensation offset minus 1. For example, the second PPS syntax may be specified as pps_ref_wraparound_offset_minus1.

[0020] In one embodiment, the bitstream includes one or more SPS syntaxes related to wraparound motion compensation information in a sequence parameter set (SPS). The SPS syntax may include a first SPS syntax corresponding to an SPS flag that indicates whether wraparound motion compensation is used for a target sequence. For example, the first SPS syntax may be specified as sps_ref_wraparound_enabled_flag. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A hypothetical example of Adaptive Resolution Change (ARC) with Reference Picture Resampling (RPR) is shown, where the current picture is predicted from reference pictures (Ref0 and Ref1) of different sizes.

[0022] Figure 2A An example showing repeated padding of unavailable reference data for a VR360 frame.

[0023] Figure 2B Horizontal surround example showing unavailable reference data for a VR360 frame.

[0024] Figure 3 An example of horizontal surround motion compensation processing is shown.

[0025] Figure 4 An exemplary block diagram of a system for transmitting surround motion compensation information according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0026] The following description is the best mode of implementing the present invention. This description is for the purpose of illustrating the general principles of the present invention and should not be considered as limiting. The scope of the present invention is best determined by reference to the attached claims.

[0027] It is readily understood that the elements of the present invention as generally described and illustrated in the drawings herein may be arranged and designed in a variety of different configurations. Therefore, as shown in the drawings, the following more detailed description of the embodiments of the systems and methods of the present invention is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention.

[0028] References in this specification to "an embodiment", "some embodiments" or similar language mean that a particular feature, structure or characteristic described in conjunction with the embodiment may be included in at least one embodiment of the present invention. Therefore, the phrases "in an embodiment" or "in some embodiments" appearing in various places throughout this specification do not necessarily refer to the same embodiment.

[0029] Furthermore, the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. However, those skilled in the art will recognize that the invention may be practiced without one or more of the specific details or with other methods, elements, etc. In other cases, well-known structures or operations are not shown or described in detail to avoid obscuring aspects of the invention.

[0030] The illustrated embodiments of the present invention will be best understood by referring to the drawings, wherein like parts are represented by like numerals throughout. The following description is intended to be exemplary only and simply illustrates certain selected embodiments of apparatus and methods consistent with the invention claimed herein.

[0031] In the specification, like reference numerals throughout the drawings and the specification designate corresponding or similar elements between the different views.

[0032] As mentioned above, VVC Draft 7 uses surround motion compensation to handle reference image areas outside the reference image boundaries. In addition, according to VVC Draft 7, a high-level syntax element is sent to indicate a surround offset, which is set to the ERP image width before padding. However, when RPR is enabled, the surround offset of a group of images with different resolutions may be different. Another problem is that as long as there is an image referencing SPS that violates the consistency regulations, we cannot enable horizontal surround motion compensation. In order to solve the above two problems, the sending of horizontal surround motion compensation information is modified according to an embodiment of the present invention.

[0033] Method 1: Send horizontal surround motion compensation information in PPS

[0034] In VVC draft 7, the transmission of horizontal surround motion compensation information is in the sequence parameter set (SequenceParameter Set, referred to as SPS). Since adaptive resolution change (ARC) / reference picture resampling (RPR) allows reference images to have different resolutions, the transmission of horizontal surround motion compensation in SPS may cause incorrect reference data. Therefore, one way to solve this problem is to move the transmission of horizontal surround motion compensation information from SPS to the picture parameter set (PictureParameter Set, referred to as PPS).

[0035] Method 2: Conditionally send horizontal surround motion compensation information in PPS

[0036] In another method of the present invention, when the horizontal surround motion compensation information is not in the SPS, it is allowed to send the horizontal surround motion compensation information in the PPS. If the horizontal surround motion compensation information is neither in the PPS nor in the SPS, it should be in the picture header.

[0037] In another embodiment, when RPR is disabled, surround offset information is sent in SPS. In addition, when RPR is enabled, surround compensation information is sent in PPS.

[0038] Method 3: Support horizontal surround motion compensation regardless of whether RPP is enabled or disabled

[0039] According to method 3, whether RPR is enabled or disabled, when there is no horizontal scaling and the image size between the current image and the reference image is the same, only horizontal surround motion compensation is supported. Specifically, no horizontal scaling means that PicOutputWidthL is the same between the current image and the reference image, where PicOutputWidthL represents the width of the current image or the reference image after applying the scaling window, PicOutputHeightL represents the height of the current image or the reference image after applying the scaling window, and the image size is pic_width_in_luma_samples and pic_height_in_luma_samples as described above.

[0040] Four embodiments of method 3 are listed below:

[0041] 1. In one embodiment, surround motion compensation is supported only when PicOutputWidthL and picture size are the same between the current picture and the reference picture, respectively.

[0042] 2. In one embodiment, surround motion compensation is supported only when PicOutputWidthL is the same between the current picture and the reference picture.

[0043] 3. In one embodiment, surround motion compensation is supported only when PicOutputWidthL, PicOutputHeightL and image size are the same between the current image and the reference image, respectively.

[0044] 4. In one embodiment, surround motion compensation is supported only when PicOutputWidthL and PicOutputHeightL are the same between the current picture and the reference picture, respectively.

[0045] Method 4: Mutually exclusive RPR and horizontal surround motion compensation

[0046] According to this method, reference picture resampling (RPR) and horizontal surround motion compensation are mutually exclusive. Method 4 can have two embodiments:

[0047] 1. When RPR is enabled or inter_layer_ref_pics_present_flag is equal to 1, horizontal surround motion compensation is disabled in SPS.

[0048] 2. When RPR is enabled, horizontal surround motion compensation is disabled in SPS.

[0049] As shown below, some examples of implementing the present invention are described based on the working draft.

[0050] According to method 1, the working sketch can be modified as shown in the following table.

[0051] Table 1. Example sequence parameter set RBSP syntax according to an embodiment of method 1

[0052] / / sps_ref_wraparound_enabled_flag / / / / u(1) / / / / if(sps_ref_wraparound_enabled_flag) / / / / sps_ref_wraparound_offset_minus1 / / / / ue(v) / /

[0053] In the above table, the text between a pair of double slashes indicates deleted text. As shown in the above table, the wraparound flag (i.e., sps_ref_wraparound_enabled_flag) and the wraparound offset information (i.e., sps_ref_wraparound_offset_minus1) are deleted in the SPS. As shown in the following table, according to an embodiment of the present invention, this information is sent in the PPS.

[0054] Table 2. Example Picture Parameter Set RBSP Syntax according to an embodiment of Method 1

[0055]

[0056] In the above table, italic text indicates inserted text. As shown in the above table, a wraparound flag (ie, pps_ref_wraparound_enabled_flag) and wraparound offset information (ie, pps_ref_wraparound_offset_minus1) are inserted in the PPS.

[0057] The semantic description of the image parameter set RBSP is as follows:

[0058] pps_ref_wraparound_enabled_flag equal to 1 indicates that horizontal wraparound motion compensation is applied in inter prediction. pps_ref_wraparound_enabled_flag equal to 0 indicates that horizontal wraparound motion compensation is not applied. The value of pps_ref_wraparound_enabled_flag shall be equal to 0 when the value of (CtbSizeY / MinCbSizeY+1) is greater than or equal to (pic_width_in_luma_samples / MinCbSizeY-1).

[0059] pps_ref_wraparound_offset_minus1 plus 1 represents the offset used to calculate the horizontal wrapping position in units of MinCbSizeY luma samples. The value of ref_wraparound_offset_minus1 should be in the range of (CtbSizeY / MinCbSizeY)+1 to (pic_width_in_luma_samples / MinCbSizeY)-1 (inclusive).

[0060] Under Method 2, the working draft of the SPS may be modified as shown in the following table:

[0061] Table 3a. Example sequence parameter set RBSP syntax according to an embodiment of method 2

[0062]

[0063]

[0064] According to another embodiment, an example syntax design for wraparound information in SPS is shown in the following table. In the following table, SPS syntax sps_ref_wraparound_enabled_flag is sent. If ref_pic_resampling_enabled_flag is not set and sps_ref_wraparound_enabled_flag is set, then syntax sps_ref_wraparound_offset_minus1 is sent.

[0065] Table 3b. Example sequence parameter set RBSP syntax according to an embodiment of method 2

[0066]

[0067] As shown in Table 3a, the syntax sps_ref_wraparound_enabled_present_flag is introduced. The syntax sps_ref_wraparound_enabled_flag is sent only when the value of sps_ref_wraparound_enabled_present_flag is 1. The modification to PPS is the same as that in Method 1. In other words, as shown in the following table, the transmission of wraparound information can also be done in PPS.

[0068] Table 4a. Example Picture Parameter Set RBSP Syntax according to an embodiment of Method 2

[0069] pps_ref_wraparound_enabled_flag u(1) if(pps_ref_wraparound_enabled_flag) pps_ref_wraparound_offset_minus1 ue(v)

[0070] According to another embodiment, a syntax design similar to Table 3b is proposed. Sending surround information in PPS is shown in the following table.

[0071] Table 4b. Example Picture Parameter Set RBSP Syntax according to an embodiment of Method 2

[0072] pps_ref_wraparound_present_flag u(1) if(pps_ref_wraparound_present_flag) pps_ref_wraparound_offset_minus1 ue(v)

[0073] Following the syntax design of Table 3a and Table 4a, the surround information can be sent in the picture header (Picture Header, PH for short), as shown in the following table.

[0074] Table 5. Example picture header RBSP syntax according to an embodiment of method 2

[0075]

[0076] The semantics of the sequence parameter set RBSP in the above table are described as follows.

[0077] sps_ref_wraparound_enabled_present_flag is equal to 1, indicating that sps_ref_wraparound_enabled_flag is present in the SPS. sps_ref_wraparound_enabled_present_flag is equal to 0, indicating that sps_ref_wraparound_enabled_flag is not present in the SPS.

[0078] sps_ref_wraparound_enabled_flag equal to 1 indicates that horizontal wraparound motion compensation is applied in inter prediction. sps_ref_wraparound_enabled_flag equal to 0 indicates that horizontal wraparound motion compensation is not applied. When the value of (CtbSizeY / MinCbSizeY+1) is less than or equal to (pic_width_in_luma_samples / MinCbSizeY-1), where pic_width_in_luma_samples is the value of pic_width_in_luma_samples in any PPS that references the SPS, and the value of sps_ref_wrap shall be equal to 0.

[0079] ref_pic_resampling_enabled_flag equal to 1 indicates that reference picture resampling can be applied when decoding coded pictures that refer to SPS in a coded layer video sequence (CLVS). ref_pic_resampling_enabled_flag equal to 0 indicates that reference picture resampling is not applied when decoding pictures that refer to SPS in a CLVS.

[0080] sps_ref_wraparound_offset_minus1 plus 1 represents the offset used to calculate the horizontal wrapping position in units of MinCbSizeY luma samples. The value of ref_wraparound_offset_minus1 shall be in the range of (CtbSizeY / MinCbSizeY)+1 to (pic_width_in_luma_samples / MinCbSizeY)-1 (inclusive), where pic_width_in_luma_samples is the value of pic_width_in_luma_samples in any PPS that references the SPS.

[0081] The semantics of the picture parameter set RBSP in the above table are described as follows.

[0082] pps_ref_wraparound_enabled_flag equal to 1 indicates that horizontal wraparound motion compensation is applied to inter prediction of all pictures that reference the PPS. pps_ref_wraparound_enabled_flag equal to 0 indicates that horizontal wraparound motion compensation is not applied. The value of pps_ref_wraparound_enabled_flag shall be equal to 0 when the value of (CtbSizeY / MinCbSizeY+1) is greater than or equal to (pic_width_in_luma_samples / MinCbSizeY-1).

[0083] When sps_ref_wraparound_enabled_flag is equal to 0 or sps_ref_wraparound_enabled_flag is equal to 1, pps_ref_wraparound_enabled_flag shall be equal to 0.

[0084] pps_ref_wraparound_present_flag equal to 1 indicates that horizontal wraparound motion compensation is applied to inter prediction of all pictures that reference the PPS. pps_ref_wraparound_present_flag equal to 0 indicates that horizontal wraparound motion compensation is not applied. The value of pps_ref_wraparound_present_flag shall be equal to 0 when the value of (CtbSizeY / MinCbSizeY+1) is greater than or equal to (pic_width_in_luma_samples / MinCbSizeY - 1). When ref_pic_resampling_enabled_flag is equal to 0, pps_ref_wraparound_present_flag shall be equal to 0.

[0085] pps_ref_wraparound_offset_minus1 plus 1 represents the offset used to calculate the horizontal wrapping position in units of MinCbSizeY luma samples. The value of ref_wraparound_offset_minus1 should be in the range of (CtbSizeY / MinCbSizeY)+1 to (pic_width_in_luma_samples / MinCbSizeY)-1 (inclusive).

[0086] The semantics of the picture header RBSP is described as follows.

[0087] ph_ref_wraparound_enabled_flag equal to 1 indicates that horizontal wraparound motion compensation is applied to inter prediction of pictures referring to the PH. ph_ref_wraparound_enabled_flag equal to 0 indicates that horizontal wraparound motion compensation is not applied to pictures referring to the PH.

[0088] ph_ref_wraparound_offset_minus1 plus 1 represents the offset used to calculate the horizontal wrapping position in units of MinCbSizeY luma samples. The value of ref_wraparound_offset_minus1 should be in the range of (CtbSizeY / MinCbSizeY)+1 to (pic_width_in_luma_samples / MinCbSizeY)-1 (inclusive).

[0089] The implementation according to method 4 can be realized by using a new syntax design or by modifying the syntax design of the existing VVC working draft. Some examples based on the working draft are shown below.

[0090] In one embodiment, the sequence parameter set RBSP syntax may be modified as shown in the following table.

[0091] Table 6a. Example SPS RBSP syntax according to an embodiment of method 4

[0092]

[0093] According to another embodiment, the value of inter_layer_ref_pics_present_flag is ignored to send surround information. The syntax design based on the VVC working draft is as follows.

[0094] Table 6b. Example SPS RBSP syntax according to another embodiment of method 4

[0095]

[0096]

[0097] The semantics of sequence parameter set RBSP are described as follows. These semantics have the same meaning as the existing working draft.

[0098] sps_ref_wraparound_enabled_flag equal to 1 indicates that horizontal wraparound motion compensation is applied in inter prediction. sps_ref_wraparound_enabled_flag equal to 0 indicates that horizontal wraparound motion compensation is not applied. When the value of (CtbSizeY / MinCbSizeY+1) is less than or equal to (pic_width_in_luma_samples / MinCbSizeY-1), where pic_width_in_luma_samples is the value of pic_width_in_luma_samples in any PPS that references the SPS. If sps_ref_wraparound_enabled_flag is not present, the value of sps_ref_wraparound_enabled_flag shall be equal to 0.

[0099] ref_pic_resampling_enabled_flag equal to 1 indicates that reference picture resampling may be applied when decoding coded pictures that refer to the SPS in the CLVS. ref_pic_resampling_enabled_flag equal to 0 indicates that reference picture resampling is not applied when decoding pictures that refer to the SPS in the CLVS.

[0100] sps_ref_wraparound_offset_minus1 plus 1 represents the offset used to calculate the horizontal wrapping position in units of MinCbSizeY luma samples. The value of ref_wraparound_offset_minus1 shall be in the range of (CtbSizeY / MinCbSizeY)+1 to (pic_width_in_luma_samples / MinCbSizeY)-1 (inclusive), where pic_width_in_luma_samples is the value of pic_width_in_luma_samples in any PPS that references the SPS.

[0101] inter_layer_ref_pics_present_flag equal to 0 indicates that no ILRP is used for inter prediction of any coded picture in the CLVS. inter_layer_ref_pics_flag equal to 1 indicates that ILRP may be used for inter prediction of one or more coded pictures in the CLVS. When sps_video_parameter_set_id is equal to 0, the value of inter_layer_ref_pics_present_flag is inferred to be equal to 0. When vps_independent_layer_flag[GeneralLayerIdx[nuh_layer_id]] is equal to 1, the value of inter_layer_ref_pics_present_flag shall be equal to 0

[0102] According to another embodiment, some constraints are imposed on the value of sps_ref_wraparound_enabled_flag instead of modifying the syntax table of SPS.

[0103] For example, the semantics of sps_ref_wraparound_enabled_flag may be modified as follows.

[0104] sps_ref_wraparound_enabled_flag is equal to 1 to indicate that horizontal wraparound motion compensation is applied in inter prediction. sps_ref_wraparound_enabled_flag is equal to 0 to indicate that horizontal wraparound motion compensation is not applied. When the value of (CtbSizeY / MinCbSizeY+1) is less than or equal to (pic_width_in_luma_samples / MinCbSizeY-1), where pic_width_in_luma_samples is the value of pic_width_in_luma_samples in any PPS that references the SPS. According to an embodiment of the present invention, when ref_pic_resampling_enabled_flag is equal to 1 or inter_layer_ref_pics_present_flag is equal to 1, the value of sps_ref_wraparound_enabled_flag should be equal to 0.

[0105] When sps_ref_wraparound_enabled_flag is not present, the value of sps_ref_wraparound_enabled_flag shall be equal to 0.

[0106] The video encoder must follow the above syntax design to generate a legal bitstream, and the video decoder can correctly decode the bitstream only if the parsing process complies with the above syntax design. When the syntax is skipped in the bitstream, the encoder and decoder should set the syntax value to the inferred value to ensure that the encoding and decoding results match.

[0107] Figure 4 An exemplary block diagram of a system for introducing sending surround motion compensation information according to an embodiment of the present invention is shown. The steps shown in the flowchart and other subsequent flowcharts in the present disclosure can be implemented as program codes that can be executed on one or more processors (e.g., one or more CPUs) on the encoder side and / or the decoder side. The steps shown in the flowchart can also be implemented based on hardware, such as one or more electronic devices or processors arranged to execute the steps in the flowchart. According to the method, in step 410, a bitstream corresponding to the encoded data of a VR360 video sequence is generated on the encoder side or received on the decoder side, wherein the bitstream includes one or more PPS syntaxes related to the surround motion compensation information in the picture parameter set (Picture Parameter Set, PPS for short). In step 420, based on the surround motion compensation information, the VR360 video sequence is encoded on the encoder side or decoded on the decoder side.

[0108] The flowchart shown is intended to illustrate an example of an embodiment according to the present invention. Those skilled in the art may modify each step, rearrange the steps, split the steps or combine the steps to practice the present invention without departing from the spirit of the present invention.

[0109] The above description is given to enable those skilled in the art to practice the present invention provided in the context of specific applications and requirements thereof. Various modifications to the described embodiments will be apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments. Therefore, the present invention is not intended to be limited to the specific embodiments shown and described, but is consistent with the widest range of principles and novel features disclosed herein. In the above detailed description, various specific details are shown in order to provide a thorough understanding of the present invention. However, those skilled in the art will appreciate that the present invention can be implemented.

[0110] The embodiments of the present invention as described above can be implemented in various hardware, software codes or a combination of the two. For example, an embodiment of the present invention can be a program code integrated into one or more circuits in a video compression chip or integrated into video compression software to perform the processing described herein. An embodiment of the present invention can also be a program code executed on a digital signal processor (Digital Signal Processor, DSP) to perform the processing described herein. The present invention may also be related to many functions performed by a computer processor, a digital signal processor, a microprocessor or a field programmable gate array (fieldprogrammable gate arragy, referred to as FPGA). These processors can be configured to perform specific tasks according to the present invention by executing machine-readable software code or firmware code that defines the specific method embodied in the present invention. The software code or firmware code can be developed in different programming languages ​​and different formats or styles. The software code can also be compiled for different target platforms. However, different code formats, styles and languages ​​of software codes and other means of configuring codes to perform tasks according to the present invention will not depart from the spirit and scope of the present invention.

[0111] The present invention may be implemented in other specific forms without departing from the spirit or essential features of the present invention. The described examples should be considered in all respects only as illustrative and not restrictive. Therefore, the scope of the present invention is indicated by the appended claims rather than the foregoing description. All changes falling within the equivalent meaning and scope of the claims should be included within their scope.

Claims

1. A method for encoding and decoding a 360-degree virtual reality video sequence, wherein surround motion compensation is included as a coding tool, The method comprises: Generate at an encoder side or receive at a decoder side a bitstream corresponding to the encoded data of the 360-degree virtual reality video sequence, wherein the bitstream includes one or more picture parameter set syntax associated with surround motion compensation information in a picture parameter set; and encoding the 360-degree virtual reality video sequence at the encoder side or decoding the 360-degree virtual reality video sequence at the decoder side using the surround motion compensation information, Among them, when PicOutputWidthL and the image size between the current image and the reference image are respectively the same, only horizontal surround motion compensation is supported in the surround motion compensation information; or when PicOutputWidthL between the current image and the reference image are the same, only horizontal surround motion compensation is supported in the surround motion compensation information; or when PicOutputWidthL, PicOutputHeightL and the image size between the current image and the reference image are respectively the same, only horizontal surround motion compensation is supported in the surround motion compensation information; or when PicOutputWidthL and PicOutputHeightL between the current image and the reference image are respectively the same, only horizontal surround motion compensation is supported in the surround motion compensation information, wherein PicOutputWidthL represents the width of the current image or the reference image after applying the scaling window, and PicOutputHeightL represents the height of the current image or the reference image after applying the scaling window.

2. The method for encoding and decoding a 360-degree virtual reality video sequence according to claim 1, wherein: The one or more picture parameter set syntaxes include a first picture parameter set syntax corresponding to a picture parameter set flag, the picture parameter set flag indicating whether the surround motion compensation is used for a target picture.

3. The method for encoding and decoding a 360-degree virtual reality video sequence according to claim 2, wherein: The first picture parameter set syntax is specified as pps_ref_wraparound_enabled_flag.

4. The method for encoding and decoding a 360-degree virtual reality video sequence according to claim 2, wherein: When the first picture parameter set syntax indicates that the surround motion compensation is used for the target picture, a second picture parameter set syntax is included in the bitstream, wherein the second picture parameter set syntax is associated with a value of a surround offset.

5. The method for encoding and decoding a 360-degree virtual reality video sequence according to claim 4, wherein: The second picture parameter set syntax indicates a value of the surround motion compensation offset minus 1.

6. The method for encoding and decoding a 360-degree virtual reality video sequence according to claim 4, wherein: The second picture parameter set syntax is specified as pps_ref_wraparound_offset_minus1.

7. The method for encoding and decoding a 360-degree virtual reality video sequence according to claim 1, wherein: The bitstream includes one or more sequence parameter set syntax associated with the surround motion compensation information in a sequence parameter set.

8. The method for encoding and decoding a 360-degree virtual reality video sequence according to claim 7, wherein: The one or more sequence parameter set syntaxes include a first sequence parameter set syntax corresponding to a sequence parameter set flag, the sequence parameter set flag indicating whether the surround motion compensation is used for a target sequence.

9. The method for encoding and decoding a 360-degree virtual reality video sequence according to claim 8, wherein: The first sequence parameter set syntax is specified as sps_ref_wraparound_enabled_flag.

10. A device for encoding and decoding a 360-degree virtual reality video sequence, wherein surround motion compensation is included as a coding and decoding tool, The apparatus comprises one or more electronic circuits or processors configured to: Generate at an encoder side or receive at a decoder side a bitstream corresponding to the encoded data of the 360-degree virtual reality video sequence, wherein the bitstream includes one or more picture parameter set syntax associated with surround motion compensation information in a picture parameter set; and encoding the 360-degree virtual reality video sequence at the encoder side or decoding the 360-degree virtual reality video sequence at the decoder side using the surround motion compensation information, Among them, when PicOutputWidthL and the image size between the current image and the reference image are respectively the same, only horizontal surround motion compensation is supported in the surround motion compensation information; or when PicOutputWidthL between the current image and the reference image are the same, only horizontal surround motion compensation is supported in the surround motion compensation information; or when PicOutputWidthL, PicOutputHeightL and the image size between the current image and the reference image are respectively the same, only horizontal surround motion compensation is supported in the surround motion compensation information; or when PicOutputWidthL and PicOutputHeightL between the current image and the reference image are respectively the same, only horizontal surround motion compensation is supported in the surround motion compensation information, wherein PicOutputWidthL represents the width of the current image or the reference image after applying the scaling window, and PicOutputHeightL represents the height of the current image or the reference image after applying the scaling window.

11. The apparatus for encoding and decoding a 360-degree virtual reality video sequence as described in claim 10, wherein the one or more picture parameter set syntaxes include a first picture parameter set syntax corresponding to a picture parameter set flag, wherein the picture parameter set flag indicates whether the surround motion compensation is used for a target image.

12. The apparatus for encoding and decoding a 360-degree virtual reality video sequence according to claim 11, wherein: The first picture parameter set syntax is specified as pps_ref_wraparound_enabled_flag.

13. The apparatus for encoding and decoding a 360-degree virtual reality video sequence according to claim 11, wherein: When the first picture parameter set syntax indicates that the surround motion compensation is used for the target picture, a second picture parameter set syntax is included in the bitstream, wherein the second picture parameter set syntax is associated with a value of a surround offset.

14. The apparatus for encoding and decoding a 360-degree virtual reality video sequence according to claim 13, wherein: The second picture parameter set syntax indicates a value of the surround motion compensation offset minus 1.

15. The device for encoding and decoding a 360-degree virtual reality video sequence according to claim 13, characterized in that: The second picture parameter set syntax is specified as pps_ref_wraparound_offset_minus1.

16. The apparatus for encoding and decoding a 360-degree virtual reality video sequence according to claim 10, wherein: The bitstream includes one or more sequence parameter set syntax associated with the surround motion compensation information in a sequence parameter set.

17. The apparatus for encoding and decoding a 360-degree virtual reality video sequence according to claim 16, wherein: The one or more sequence parameter set syntaxes include a first sequence parameter set syntax corresponding to a sequence parameter set flag, the sequence parameter set flag indicating whether the surround motion compensation is used for a target sequence.

18. The apparatus for encoding and decoding a 360-degree virtual reality video sequence according to claim 17, wherein: The first sequence parameter set syntax is specified as sps_ref_wraparound_enabled_flag.

Images