Video processing methods and apparatus

Abstract

A video processing method includes the steps of receiving a bitstream and decoding a portion of the bitstream to generate a decoded frame, the step including parsing multiple syntax elements from the bitstream. The decoded frame is a projection-based frame comprising multiple projection planes, said multiple projection planes being packed at multiple face locations with different position indices in a hemispherical cubemap projection layout. A portion of the 360-degree content of the sphere is mapped onto the multiple projection planes via hemispherical cubemap projection. The values ​​of the multiple syntax elements respectively indicate the multiple face indices of the multiple projection planes packed at the multiple face locations, and are constrained to meet bitstream consistency requirements.

Description

[0001] Related cross-references

[0002] This application claims priority to U.S. Provisional Application No. 63 / 037,064, filed on June 10, 2020, which is incorporated herein by reference. Technical Field

[0003] The present invention relates to video processing, and more specifically, to a video processing method for processing projection-based frames having been packed in a video processing apparatus having surface packing constraints (e.g., surface index constraints and / or surface rotation) and associated features. Background Technology

[0004] Virtual reality (VR) with a head-mounted display (HMD) is associated with a variety of applications. The ability to display wide-field-of-view content to users can be used to provide an immersive visual experience. The real-world environment must be captured from all directions to produce omnidirectional video corresponding to the field of view. With advancements in camera equipment and HMDs, the delivery of VR content may soon become a bottleneck due to the high bitrates required to represent this 360-degree content. When the resolution of omnidirectional video is 4K or higher, data compression / encoding is crucial for reducing the bitrate.

[0005] Generally, omnidirectional video corresponding to a sphere is converted into frames with 360-degree image content. These frames are represented by one or more projection surfaces arranged in a 360-degree virtual reality (360VR) projection layout. The generated frames are then encoded into a bitstream for transmission. The bitstream generated from the encoder side is received and decoded by the decoder side. Furthermore, the configuration of the adopted 360VR projection layout can also be sent from the encoder side to the decoder side. When the adopted 360VR projection layout is a hemispherical cubemap projection layout, the packaged projection surfaces in the projection-based frame include a full face and four half faces, where the full face is square, and each half face is not square. To ensure the image quality of projection-based frames using a hemispherical cubemap projection layout, an innovative design is needed to apply face packing constraints to the hemispherical cubemap projection layout. Summary of the Invention

[0006] One object of the claimed invention is to provide a video processing method for processing projection-based frames having projection surfaces packed in a hemispherical cubemap projection layout with face packing constraints (e.g., face index constraints and / or face rotation), and to provide associated video processing apparatus.

[0007] According to a first aspect of the present invention, an exemplary video processing method is disclosed. This exemplary video processing method includes the steps of receiving a bitstream and decoding a portion of the bitstream by a decoding circuit to generate a decoded frame, the step including parsing a plurality of syntax elements from the bitstream. The decoded frame is a projection-based frame comprising a plurality of projection surfaces, said plurality of projection surfaces being packed at multiple face locations with different position indices in a hemispherical cubemap projection layout. A portion of the 360-degree content of the sphere is mapped onto the plurality of projection surfaces via hemispherical cubemap projection. The values ​​of the plurality of syntax elements respectively indicate the face indices of the plurality of projection surfaces packed at the multiple face locations, and are constrained to meet bitstream consistency requirements.

[0008] According to a second aspect of the present invention, an exemplary video processing method is disclosed. This exemplary video processing method includes the steps of receiving a bitstream and decoding a portion of the bitstream by a decoding circuit to generate a decoded frame, the step including parsing a plurality of first syntax elements from the bitstream. The decoded frame is a projection-based frame comprising a plurality of projection planes, the plurality of projection planes being packed at multiple face positions with different position indices in a hemispherical cube map projection layout. A portion of the 360-degree content of the sphere is mapped onto the plurality of projection planes via hemispherical cube map projection. The values ​​of the plurality of first syntax elements respectively indicate the rotation angle of the plurality of projection planes packed at the multiple face positions. A portion of the values ​​of the plurality of first syntax elements is constrained to meet bitstream consistency requirements.

[0009] According to a third aspect of the present invention, an exemplary video processing apparatus is disclosed. The exemplary video processing apparatus includes a decoding circuit arranged to receive a bitstream and decode a portion of the bitstream to generate a decoded frame. The decoding circuit parses a plurality of first syntax elements and a plurality of second syntax elements from the bitstream, wherein the decoded frame is a projection-based frame comprising a plurality of projection surfaces, the plurality of projection surfaces being packaged at multiple face positions with different position indices in a hemispherical cube map projection layout; a portion of 360-degree content of a sphere is mapped onto the plurality of projection surfaces via hemispherical cube map projection; the values ​​of the plurality of second syntax elements respectively indicate the face indices of the plurality of projection surfaces packaged at the multiple face positions, and are constrained to meet bitstream consistency requirements; the values ​​of the plurality of first syntax elements respectively indicate the rotation angles of the plurality of projection surfaces packaged at the multiple face positions; a portion of the values ​​of the plurality of first syntax elements is constrained to meet bitstream consistency requirements.

[0010] These and other objects of the invention will undoubtedly become apparent to those skilled in the art after reading the following detailed description of the preferred embodiments shown in the various accompanying drawings. Attached Figure Description

[0011] Figure 1 A diagram illustrating a 360-degree virtual reality (360VR) system according to an embodiment of the present invention is shown.

[0012] Figure 2 A diagram illustrating a hemispherical cube mapping projection according to an embodiment of the present invention is shown.

[0013] Figure 3 A diagram illustrating a horizontally packaged hemispherical cube map projection layout according to an embodiment of the present invention.

[0014] Figure 4 A diagram illustrating a vertically packaged hemispherical cube texture projection layout according to an embodiment of the present invention.

[0015] Figure 5 A diagram illustrating a specification of the syntax element gcmp_face_rotation[i] according to an embodiment of the present invention is shown.

[0016] Figure 6 A diagram illustrating another specification of the syntax element gcmp_face_rotation[i] according to an embodiment of the present invention is shown.

[0017] Figure 7 A diagram illustrating the coordinate definition according to an embodiment of the present invention is shown.

[0018] Figure 8 The diagram shows an example of a hemispherical cube map projection layout where the front face is selected as full.

[0019] Figure 9 The diagram shows an example of a hemispherical cube map projection layout where the back face is selected as full.

[0020] Figure 10 The diagram shows an example of four hemispheres selected for a hemispherical cubemap projection layout when the left side is selected as full.

[0021] Figure 11 The diagram shows an example of four hemispheres selected for a hemispherical cubemap projection layout when the right side is selected as full.

[0022] Figure 12 The diagram shows an example of four hemispheres selected for a hemispherical cubemap projection layout when the top face is selected as full.

[0023] Figure 13 The diagram shows an example of four hemispheres selected for a hemispherical cube map projection layout when the bottom face is selected as full.

[0024] Figure 14A diagram illustrating a rotation angle constraint of a half-face packed in a horizontally packed hemispherical cube map projection layout according to an embodiment of the present invention.

[0025] Figure 15 A diagram illustrating another rotation angle constraint of a half-face packed in a horizontally packed hemispherical cube map projection layout according to an embodiment of the present invention.

[0026] Figure 16 A diagram illustrating a rotation angle constraint of a half-face packed in a vertically packed hemispherical cube map projection layout according to an embodiment of the present invention.

[0027] Figure 17 The diagram illustrates another rotation angle constraint of a half-face packed in a vertically packed hemispherical cube map projection layout according to an embodiment of the present invention. Detailed Implementation

[0028] Specific terms are used in the following description and claims to refer to specific elements. As those skilled in the art will understand, electronic device manufacturers may use different names to refer to an element. This document is not intended to distinguish between elements with different names but the same function. In the following description and claims, the terms "comprising" and "including" are used in an open-ended manner and should therefore be interpreted as meaning "including but not limited to...". Furthermore, the term "coupled" is intended to indicate an indirect or direct electrical connection. Thus, if one device is coupled to another device, the connection can be through a direct electrical connection or through an indirect electrical connection via other devices and connections.

[0029] Figure 1 A diagram illustrating a 360-degree virtual reality (360VR) system according to an embodiment of the present invention is shown. The 360VR system 100 includes two video processing devices (e.g., a source electronics device 102 and a target electronics device 104). The source electronics device 102 includes a video acquisition device 112, a conversion circuit 114, and a video encoder 116. For example, the video acquisition device 112 may be an omnidirectional camera. The conversion circuit 114 generates a projection-based frame IMG with a 360-degree virtual reality (360VR) projection layout L_VR based on an omnidirectional video frame S_IN corresponding to a sphere, wherein the omnidirectional video frame S_IN contains 360-degree content of the sphere. The video encoder 116 is an encoding circuit for encoding the projection-based frame IMG (which has a projection surface packed in the 360VR projection layout L_VR) to generate a portion of a bitstream BS, and outputting the bitstream BS to the target electronics device 104 via a transmission device 103 such as a wired / wireless communication link or storage medium.

[0030] The target electronic device 104 can be a head-mounted display (HMD) device. For example... Figure 1 As shown, the target electronic device 104 includes a video decoder 122, a graphics rendering circuit 124, and a display device 126. The video decoder 122 is a decoding circuit that receives a bitstream BS from a transmission device 103 (e.g., a wired / wireless communication link or storage medium) and decodes a portion of the received bitstream BS to generate a decoded frame IMG'. In this embodiment, the projection-based frame IMG to be encoded by the video encoder 116 has a 360VR projection layout L_VR. Therefore, after a portion of the bitstream BS is decoded by the video decoder 122, the decoded frame (i.e., the reconstructed frame) IMG' has the same 360VR projection layout L_VR. In other words, the decoded frame IMG' is also a projection-based frame, having one or more projection surfaces packed within the 360VR projection layout L_VR. Hereinafter, the terms "decoded frame" and "projection-based frame" are used interchangeably. The graphics rendering circuit 124 drives the display device 126 to display image content of a viewport area selected by the user.

[0031] In this embodiment, the 360VR projection layout L_VR can be a hemispherical cube map projection layout. Specifically, a cube-based projection with five projection planes (including one full face and four half faces) representing a 180°x180° omnidirectional video (i.e., only a portion of the 360-degree content of the sphere) can be employed. Regarding the conversion circuitry 114 of the source electronics 102, the hemispherical cube map projection is used to generate one full face and four half faces of a cube in three-dimensional (3D) space. Figure 2 A diagram illustrating a hemispherical cubemap projection according to an embodiment of the present invention is shown. Only half of the 360-degree content of the sphere 200 is projected onto multiple faces of the cube 201, including the top half (labeled "Top_H"), bottom half (labeled "Bottom_H"), left half (labeled "Left_H"), front half (labeled "Front"), and right half (labeled "Right_H"). In this example, half of the cube 201 is used for the hemispherical cubemap projection, wherein the hemisphere (e.g., half of the sphere 200) is inscribed within half of the cube 201. See also... Figure 2 The image content of half of the North Pole region of sphere 200 is projected onto the top half "Top_H" (i.e., half of the top face of cube 201), the image content of half of the South Pole region of sphere 200 is projected onto the bottom half "Bottom_H" (i.e., half of the bottom face of cube 201), and the image content of half of the equatorial region of sphere 200 is projected onto the left half "Left_H" (half of the left face of cube 201), the front half "Front" (the front face of cube 201), and the right half "Right_H" (half of the right face of cube 201).

[0032] In a 3D space defined by the x-axis, y-axis, and z-axis, each point on the five projection planes lies at (x, y, z), where x, y, z ∈ [-1, 1]. Figure 2 In the example shown, the front half is on the x-plane at x=1, the top half "Top_H" is on the z-plane at z=1, the bottom half "Bottom_H" is on the z-plane at z=-1, the left half "Left_H" is on the y-plane at y=1, and the right half "Right_H" is on the y-plane at y=-1. In another design, the front half "Front" can be on the x-plane at x=1, the top half "Top_H" can be on the y-plane at y=1, the bottom half "Bottom_H" can be on the y-plane at y=-1, the right half "Right_H" can be on the z-plane at z=1, and the left half "Left_H" can be on the z-plane at z=-1.

[0033] Forward transformations can be used to transform from 3D space (x,y,z) to a 2D plane (u,v). Therefore, the top half "Top_H", bottom half "Bottom_H", left half "Left_H", front half "Front", and right half "Right_H" of cube 201 in 3D space can be transformed into the top half (labeled "2"), bottom half (labeled "3"), left half (labeled "5"), front half (labeled "0"), and right half (labeled "4") in a 2D plane. Each face lies on a 2D plane defined by the horizontal u-axis and the vertical v-axis, with each point located at (u,v). Furthermore, the front half (labeled "0") is twice the size of each of the top half (labeled "2"), bottom half (labeled "3"), left half (labeled "5"), and right half (labeled "4").

[0034] The inverse transformation can be used to transform from a 2D plane (u,v) to a 3D space (x,y,z). Therefore, the top half (labeled "2"), bottom half (labeled "3"), left half (labeled "5"), front half (labeled "0"), and right half (labeled "4") on the 2D plane can be transformed into the top half "Top_H", bottom half "Bottom_H", left half "Left_H", front half "Front", and right half "Right_H" of cube 201 in 3D space.

[0035] The conversion circuit 114 of the source electronics device 102 can employ an inverse transformation to generate a top half-face "2", a bottom half-face "3", a left half-face "5", a full front half-face "0", and a right half-face "4". The top half-face "2", bottom half-face "3", left half-face "5", full front half-face "0", and right half-face "4" on the two-dimensional plane are packaged to form a projection-based frame IMG, which will be encoded by the video encoder 116. The video decoder 122 receives a bitstream BS from the transmission device 103 and decodes a portion of the received bitstream BS to generate a decoded frame IMG' using the same projection layout L_VR (e.g., a hemispherical cube map layout) employed at the encoder end.

[0036] As described above, the top half "2", bottom half "3", left half "5", full face "0", and right half "4" are packaged to form a projection-based frame IMG. For example, the conversion circuit 114 can select a packaging type such that the projection-based frame IMG can have projected image data arranged in a cube-based projection layout (hemispherical cube mapping layout) 202. As another example, the conversion circuit 114 can select another packaging type such that the projection-based frame IMG can have projected image data arranged in another cube-based projection layout (hemispherical cube mapping layout) 204, which is different from the cube-based projection layout (hemispherical cube mapping layout) 202. In this embodiment, the front face is selected as the full face packaged in the cube-based projection layouts 202 / 204. In fact, the full face packed in the cube-based projection layout 202 / 204 can be any of the top face, bottom face, front face, back face, left face, and right face, and the four half faces packed in the cube-based projection layout 202 / 204 depend on the choice of full face.

[0037] Cube-based projection layout 202 is also known as a horizontally packed hemispherical cubemap projection layout, in which all projection faces are horizontally packed. Cube-based projection layout 204 is also known as a vertically packed hemispherical cubemap projection layout, in which all projection faces are vertically packed. Figure 3 A diagram illustrating a horizontally packed hemispherical cube map projection layout according to an embodiment of the present invention is shown. The horizontally packed hemispherical cube map projection layout 300 includes five face positions with different position indices “0”, “1”, “2”, “3”, and “4”, wherein the entire face is packed at the face position with position index “2”, and the four half faces are packed at the face positions with position indices “0”, “1”, “3”, and “4”, respectively. Figure 4A diagram illustrating a vertically packed hemispherical cube map projection layout according to an embodiment of the present invention is shown. The vertically packed hemispherical cube map projection layout 400 includes five face positions with different position indices “0”, “1”, “2”, “3”, and “4”, wherein the entire face is packed at the face position with position index “2”, and the four half-faces are packed at the face positions with position indices “0”, “1”, “3”, and “4”, respectively.

[0038] The conversion circuit 114 can determine the configuration of the projection-based frame IMG, which is generated from a cube-based projection (e.g., Figure 2 The hemispherical cube map projection shown in the diagram comprises multiple projection surfaces, and the video encoder 116 can send one or more syntax elements SE via the bitstream BS. For example, one or more syntax elements SE can be associated with a packing of projection surfaces in the projection-based frame IMG. Therefore, the video decoder 122 can parse one or more syntax elements SE' associated with the packing configuration of projection surfaces in the projection-based frame IMG from the bitstream BS, and can provide the parsed one or more syntax elements SE' to the graphics rendering circuit 124 so that the graphics rendering circuit 124 is informed of the surface packing configuration information. It should be noted that the projection-based frame IMG and the decoded frame IMG' have the same surface packing configuration. In this way, when determining the image content of the viewport region selected by the user, the graphics rendering circuit 124 can refer to the surface packing configuration information to correctly determine the pixel values ​​of the selected viewport region. Ideally, one or more syntax elements SE encoded by the video encoder 116 into the bitstream BS are the same as one or more syntax elements SE' parsed by the video decoder 122 from the bitstream BS. For example, one or more syntax elements SE / SE' represent face packing configurations that may include gcmp_packing_type, gcmp_face_rotation, gcmp_face_index, etc.

[0039] The syntax element `gcmp_packing_type` specifies the packing type of the projected faces in a cube-based projection layout, and further specifies a predefined arrangement of position indices assigned to the face positions under the selected packing type. When the value of `gcmp_packing_type` is 4 or 5, a hemispherical cube map packing with one full face and four half faces is used, where each packing type is associated with five face positions respectively assigned position indices {0, 1, 2, 3, 4}. Specifically, when... Figure 3 The horizontally packed hemispherical cube map projection layout 300 shown is adopted, with the value of gcmp_packing_type equal to 4; and when... Figure 4The vertically packed hemispherical cube map projection layout 400 shown is used, and the value of gcmp_packing_type is equal to 5.

[0040] The syntax element `gcmp_face_index[i]` can specify the face index for position index `i` under the packing type specified by the syntax element `gcmp_packing_type`. Taking a hemispherical cube map projection as an example, the allocatable face indexes for the front full / half face are `gcmp_face_index[i]` = 0, for the back full / half face `gcmp_face_index[i]` = 1, for the top full / half face `gcmp_face_index[i]` = 2, for the bottom full / half face `gcmp_face_index[i]` = 3, for the right full / half face `gcmp_face_index[i]` = 4, and for the left full / half face `gcmp_face_index[i]` = 5. Therefore, when the syntax element `gcmp_packing_type` is set to 4 or 5, the syntax element `gcmp_face_index[i]` specifies the face index of the projected face (e.g., ...). Figure 2 The front face “0”, back face “1”, top face “2”, bottom face “3”, right face “4” or left face “5” shown, wherein the projected face with the face index specified by the syntax element gcmp_face_index[i] is packed at the face position with position index i under the selected packing type.

[0041] The syntax element gcmp_face_rotation[i] specifies a rotation with a rotation angle applied to the projected face packed at the face position with position index i under the packing type specified by the syntax element gcmp_packing_type. Figure 5A diagram illustrating a specification of the syntax element gcmp_face_rotation[i] according to an embodiment of the present invention is shown. Therefore, when the rotation angle applied to the rotation of the projection surface to be packaged at the face position with position index i is 0°, the syntax element gcmp_face_rotation[i] is set to 0; when the rotation angle applied to the rotation of the projection surface to be packaged at the face position with position index i is 90° counterclockwise, the syntax element gcmp_face_rotation[i] is set to 1; when the rotation angle applied to the rotation of the projection surface to be packaged at the face position with position index i is 180° counterclockwise, the syntax element gcmp_face_rotation[i] is set to 2; and when the rotation angle applied to the rotation of the projection surface to be packaged at the face position with position index i is 270° counterclockwise, the syntax element gcmp_face_rotation[i] is set to 3.

[0042] In some embodiments of the present invention, the input to the rotation process may include the width (faceWidth) and height (faceHeight) of the projection plane (full surface), and may also include the sample positions (hPosFace, vPosFace) within the projection plane on the 2D plane. The width (faceWidth) may be the same as the height (faceHeight). The output of the rotation process may include the rotated sample positions (hPosRot, vPosRot) within the projection plane on the 2D plane. The output can be derived as follows:

[0043]

[0044] In the examples above, the rotation applied to a projection surface is defined by a rotation angle in the counterclockwise direction. However, this is for illustrative purposes only and is not intended to limit the invention. In another design, the rotation applied to a projection surface can be defined by a rotation angle in the clockwise direction. Figure 6 A diagram illustrating another specification of the syntax element gcmp_face_rotation[i] according to an embodiment of the present invention is shown.

[0045] As described above, a full face and four half faces are packed in a hemispherical cube map projection layout, such as a horizontally packed hemispherical cube map projection layout 300 (gcmp_packing_type == 4) or a vertically packed hemispherical cube map projection layout 400 (gcmp_packing_type == 5). To represent continuous 180° x 180° omnidirectional video content, the four half faces packed in the hemispherical cube map projection layout on the 2D plane should originate from half faces connected (adjacent) to the full face in 3D space. Embodiments of the present invention propose applying constraints to the selection of half faces packed in the hemispherical cube map projection layout. That is, the conversion circuit 114 is configured to impose constraints on the face indices of the half faces packed at face positions with position indices {0, 1, 3, 4} to meet bitstream consistency requirements. The syntax element gcmp_face_index[i] indicating the face index of the half face packed in the hemispherical cube map projection layout is emitted via the bitstream BS. By applying the constraints of the proposed face index, a bitstream BS satisfying the bitstream consistency requirements is generated and transmitted from source electronics 102 to target electronics 104. At target electronics 104, the bitstream BS satisfying the bitstream consistency requirements does not contain invalid syntax values ​​and can be decoded at video decoder 122. Video decoder 122 can parse one or more syntax elements SE' associated with a packing configuration of the projection planes in the projection-based frame IMG from the bitstream BS, and can provide the parsed one or more syntax elements SE' to graphics rendering circuitry 124, allowing graphics rendering circuitry 124 to correctly process the post-processing of decoded frame IMGs with the same hemispherical cube map projection layout.

[0046] Regarding the application of constraints to the face indices of half-faces packaged within a hemispherical cube map projection layout, the coordinates used are defined in... Figure 7 As shown, the viewing angle is from the origin of sphere 200 outwards into the interior of sphere 200. Each projection plane derived from the square face of the cube in 3D space lies on a 2D plane defined by the u-axis and v-axis, and has a local sample position at (u, v) within the projection plane, with its coordinate origin as shown. Figure 7 As shown.

[0047] As mentioned above, the full face to be packed in a hemispherical cube map projection layout can be any of the top, bottom, front, back, left, and right faces, and the four halves packed in the hemispherical cube map projection layout depend on the choice of the full face. The half to be packed in the hemispherical cube map projection layout can be half of the front face, half of the back face, half of the left face, half of the right face, half of the top face, or half of the bottom face, depending on the choice of the full face to be filled in the hemispherical cube map projection layout and the spatial relationship between the full face and the half faces.

[0048] Figure 8 This diagram shows an example of four hemispheres selected for a hemispherical cubemap projection layout when the front face is selected as full. According to... Figure 7 The coordinates shown indicate that the four halves of the hemispherical cube map projection layout include the right half of the left face, the left half of the right face, the right half of the top face, and the left half of the bottom face.

[0049] Figure 9 This diagram shows an example of four hemispheres selected for a hemispherical cubemap projection layout when the back face is selected as full. According to... Figure 7 The coordinates shown indicate that the four halves of the hemispherical cube map projection layout include the left half of the left face, the right half of the right face, the left half of the top face, and the right half of the bottom face.

[0050] Figure 10 This diagram shows an example of four hemispheres selected for a hemispherical cubemap projection layout when the left side is chosen as the full side. According to... Figure 7 As shown in the coordinate definition, the four halves of the hemispherical cube map projection layout include the lower half of the back face, the left half of the front face, the lower half of the top face, and the lower half of the bottom face.

[0051] Figure 11 This diagram shows an example of four hemispheres selected for a hemispherical cubemap projection layout when the right side is selected as the full side. According to... Figure 7 As shown in the coordinate definition, the four halves of the hemispherical cube map projection layout include the upper half of the back face, the right half of the front face, the upper half of the top face, and the upper half of the bottom face.

[0052] Figure 12 This diagram shows an example of four hemispheres selected for a hemispherical cubemap projection layout when the top face is selected as full. According to... Figure 7 As shown in the coordinate definition, the four halves of the hemispherical cube map projection layout include the upper half of the left side, the upper half of the front side, the right half of the back side, and the upper half of the right side.

[0053] Figure 13 This diagram shows an example of four hemispheres selected for a hemispherical cube map projection layout when the bottom face is selected as the full face. According to... Figure 7 The coordinates shown indicate that the four halves of the hemispherical cube map projection layout include the lower half of the left face, the lower half of the front face, the left half of the back face, and the lower half of the right face.

[0054] When the value of gcmp_packing_type is set to 4 or 5, the constraint on the face index value can guarantee that the projected frame IMG / IMG' under the selected packing type contains continuous 180° x 180° omnidirectional video content of sphere 200. Depending on the full selection specified by gcmp_face_index[2], the four hemispheres can be determined and specified by gcmp_face_index[i] of i = {0, 1, 3, 4} to achieve the goal of having continuous 180° x 180° omnidirectional video content in the projected frame IMG / IMG'. For example, when gcmp_packing_type is equal to 4 or 5, the following constraints apply to the bitstream consistency requirements:

[0055] - If gcmp_face_index[2] is equal to 0 or 1, then for i equal to 0, 1, 3 or 4, the value of gcmp_face_index[i] should be in the range of 2 to 5, including 2 and 5.

[0056] Otherwise, if gcmp_face_index[2] is equal to 2 or 3, then for i equal to 0, 1, 3 or 4, the value of gcmp_face_index[i] should be 0, 1, 4 or 5.

[0057] Otherwise, when i equals 0, 1, 3 or 4, the value of gcmp_face_index[i] should be in the range of 0 to 3, inclusive.

[0058] In the first case, the entire face packed at position index 2 is set by either the front or back face. The four faces packed at positions indexes 0, 1, 3, and 4 are constrained to be the left half, the right half, the top half, and the bottom half, as shown below. Figure 8-9 As shown. In the second case, the entire face packed at position index 2 is set by either the top or bottom face, while the four faces packed at positions indexes 0, 1, 3, and 4 are constrained to be half of the front face, half of the back face, half of the right face, and half of the left face, as shown. Figure 12-13As shown. In the third case, the entire face packed at position index 2 is set by either the right or left face, while the four faces packed at positions indexes 0, 1, 3, and 4 are constrained to be half of the front face, half of the back face, half of the top face, and half of the bottom face, as shown. Figure 10-11 As shown.

[0059] As described above, a rotation with an angle is applied to the projected face packed at the face position with a position index under the selected packing type. That is, the rotated face is packed in a hemispherical cubemap projection layout, where the rotation information is specified by the syntax element `gcmp_face_rotation`. The rotation angle of the half-face should be constrained to ensure that the rotated half-face can be accommodated in the hemispherical cubemap projection layout specified by the syntax element `gcmp_packing_type`.

[0060] For example, when a horizontally packed hemispherical cube map projection layout 300 (gcmp_packing_type=4) is used, the rotation of the half-face is constrained. If the half-face packed at a face position where position index i is not equal to 2 (i.e., i = 0, 1, 3, or 4) is derived from the upper or lower half of a square face in 3D space, the value of the syntax element gcmp_face_rotation[i] assigned to position index i indicates the rotation angle selected from a combination of 90° and 270°. That is, the rotation angle should be 90° clockwise / counterclockwise or 270° to ensure that the rotated half-face can be accommodated within the horizontally packed hemispherical cube map projection layout 300, such as... Figure 14 As shown. If a half-face packed at a position i not equal to 2 (i.e., i = 0, 1, 3, or 4) is derived from the left or right half of a square face of a cube in 3D space, the value of the syntax element gcmp_face_rotation[i] assigned to position index i indicates the rotation angle selected from a combination of 0° and 180°. That is, the rotation angle should be 0° clockwise / 180° counterclockwise to ensure that the rotated half-face can be accommodated in the horizontally packed hemispherical cube map projection layout 300, as shown. Figure 15 As shown.

[0061] For another example, when using a vertically packed hemispherical cubemap projection layout 400 (gcmp_packing_type = 5), the rotation of the half-face is constrained. If a half-face packed at a face position where position index i is not equal to 2 (i.e., i = 0, 1, 3, or 4) is derived from the upper or lower half of a cube square face in 3D space, the value of the syntax element gcmp_face_rotation[i] assigned to position index i indicates the rotation angle selected from a combination of 0° and 180°. That is, the rotation angle should be either 0° clockwise or 180° counterclockwise to ensure that the rotated half-face can be accommodated within the vertically packed hemispherical cubemap projection layout 400, such as... Figure 16 As shown. If a half-face packed at a face position where position index i is not equal to 2 (i.e., i = 0, 1, 3, or 4) is derived from the left or right half of a square face of a cube in 3D space, the value of the syntax element gcmp_face_rotation[i] assigned to position index i indicates the rotation angle selected from a combination of 90° and 270°. That is, the rotation angle should be 90° clockwise / 270° counterclockwise to ensure that the rotated half-face can be accommodated in the vertically packed hemispherical cube map projection layout 400, as... Figure 17 As shown.

[0062] In summary, when gcmp_packing_type equals 4, the bitstream consistency requirement applies the following constraints:

[0063] —If gcmp_face_index[2] is equal to 0 or 1, then for i equal to 0, 1, 3 or 4, the value of gcmp_face_rotation[i] should be 0 or 2.

[0064] Otherwise, if gcmp_face_index[2] equals 2 or 3, when gcmp_face_index[i] equals 1, the value of gcmp_face_rotation[i] should be 0 or 2, and when gcmp_face_index[i] equals 0, 4 or 5, the value of gcmp_face_rotation[i] should be 1 or 3.

[0065] Otherwise, when gcmp_face_index[i] equals 0, the value of gcmp_face_rotation[i] should be 0 or 2, and when gcmp_face_index[i] equals 1, 2 or 3, the value of gcmp_face_rotation[i] should be 1 or 3.

[0066] When gcmp_packing_type equals 5, the bitstream consistency requirement applies the following constraints:

[0067] – If gcmp_face_index[2] is equal to 0 or 1, then the value of gcmp_face_rotation[i] should be 1 or 3 for i equal to 0, 1, 3 or 4.

[0068] Otherwise, if gcmp_face_index[2] equals 2 or 3, when gcmp_face_index[i] equals 1, the value of gcmp_face_rotation[i] should be 1 or 3, and when gcmp_face_index[i] equals 0, 4 or 5, the value of gcmp_face_rotation[i] should be 0 or 2.

[0069] Otherwise, when gcmp_face_index[i] equals 0, the value of gcmp_face_rotation[i] should be 1 or 3, and when gcmp_face_index[i] equals 1, 2 or 3, the value of gcmp_face_rotation[i] should be 0 or 2.

[0070] Those skilled in the art will readily observe that many modifications and alterations can be made to the apparatus and method while retaining the teachings of the present invention. Therefore, the above disclosure should be construed as being limited only by the scope and limitations of the appended claims.

Claims

1. A video processing method, comprising: Receive bit stream; as well as Decoding a portion of the bitstream to generate a decoded frame includes: Parse multiple syntax elements from the bitstream; The decoded frame is a projection-based frame, comprising multiple projection surfaces packed at multiple face locations with different position indices in a hemispherical cube map projection layout; a portion of the 360-degree content of the sphere is mapped onto the multiple projection surfaces via hemispherical cube map projection; multiple values ​​of the multiple syntax elements respectively represent multiple face indices of the multiple projection surfaces packed at the multiple face locations, and are constrained to meet bitstream consistency requirements; the multiple projection surfaces comprise a full surface packed at face locations with position indices, the value of the syntax element assigned to the position index indicating a face index of one of the top, bottom, front, back, left, and right faces, and the value of the syntax element assigned to any other position index imposing a face index different from the value of the syntax element of the position index.

2. The video processing method as described in claim 1, characterized in that, The plurality of projection surfaces include a full surface packed at a surface location with a position index, wherein the value of the syntax element assigned to the position index indicates the face index of the front face, and the value of the syntax element assigned to any other position index indicates a face index selected from the group consisting of the face index of the left face, the face index of the right face, the face index of the top face, and the face index of the bottom face.

3. The video processing method as described in claim 1, characterized in that, The plurality of projection surfaces include a full surface packed at a surface location with a position index, the value of the syntax element assigned to the position index indicating the face index of the back face, and the value of the syntax element assigned to any other position index indicating a face index selected from the group consisting of the face index of the left face, the face index of the right face, the face index of the top face, and the face index of the bottom face.

4. The video processing method as described in claim 1, characterized in that, The plurality of projection surfaces include a full surface packed at a surface location with a position index, wherein the value of the syntax element assigned to the position index indicates the surface index of the top surface, and the value of the syntax element assigned to any other position index indicates a surface index selected from the group consisting of the surface index of the left surface, the surface index of the front surface, the surface index of the back surface, and the surface index of the right surface.

5. The video processing method as described in claim 1, characterized in that, The plurality of projection surfaces include a full surface packed at a face location with a position index, the value of the syntax element assigned to the position index indicating the face index of the bottom face, and the value of the syntax element assigned to any other position index indicating a face index selected from the group consisting of the face index of the left face, the face index of the front face, the face index of the back face, and the face index of the right face.

6. The video processing method as described in claim 1, characterized in that, The plurality of projection surfaces include a full surface packed at a surface location with a position index, the value of the syntax element assigned to the position index indicating the face index of the right side, and the value of the syntax element assigned to any other position index indicating a face index selected from the group consisting of the face index of the back side, the face index of the front side, the face index of the top side, and the face index of the bottom side.

7. The video processing method as described in claim 1, characterized in that, The plurality of projection surfaces include a full surface packed at a surface location with a position index, wherein the value of the syntax element assigned to the position index indicates the face index of the left side, and the value of the syntax element assigned to any other position index indicates a face index selected from the group consisting of the face index of the back side, the face index of the front side, the face index of the top side, and the face index of the bottom side.

8. The video processing method as described in claim 1, characterized in that, The portion of the 360-degree content of the sphere is projected onto a square face and half of four other square faces of a cube. The plurality of projection faces packaged in the hemispherical cube mapping projection layout include a full face and four half faces, the full face being derived from the square face and the four half faces being derived from half of the other four square faces connected to the square face, respectively.

9. A video processing method, comprising: Receive bit stream; as well as Decoding a portion of the bitstream to generate a decoded frame includes: Multiple first syntax elements are parsed from the bitstream; The decoded frame is a projection-based frame, comprising multiple projection surfaces packed at multiple face locations with different position indices in a hemispherical cube map projection layout; a portion of the 360-degree content of the sphere is mapped onto the multiple projection surfaces via hemispherical cube map projection; multiple values ​​of the multiple first syntax elements respectively indicate multiple rotation angles of the multiple projection surfaces packed at the multiple face locations; and a portion of the multiple values ​​of the multiple first syntax elements is constrained to meet bitstream consistency requirements; wherein the multiple projection surfaces located in the hemispherical cube map projection layout comprise four half-faces, and a portion of the multiple values ​​of the multiple first syntax elements includes values ​​indicating the rotation angles of the four half-faces.

10. The video processing method as described in claim 9, characterized in that, The hemispherical cube map projection layout is a horizontally packed hemispherical cube map layout. The plurality of projection faces include half-faces packed at face positions with position indices. These half-faces are derived from the upper or lower half of a square face of the cube, and the value of the first syntax element assigned to the position index indicates the origin of the half-face. and The rotation angle selected in the group.

11. The video processing method as described in claim 9, characterized in that, The hemispherical cube map projection layout is a horizontally packed hemispherical cube map layout. The plurality of projection faces include half-faces packed at face positions with position indices. These half-faces are derived from the left or right half of a square face of the cube, and the value of the first syntax element assigned to the position index indicates the origin of the half-face. and The rotation angle selected in the group.

12. The video processing method as described in claim 9, characterized in that, The hemispherical cube map projection layout is a vertically packed hemispherical cube map layout. The plurality of projection faces include half-faces packed at face positions with position indices. These half-faces are derived from the left or right half of a square face of the cube, and the value of the first syntax element assigned to the position index indicates the origin of the half-face. and The rotation angle selected in the group.

13. The video processing method as described in claim 9, characterized in that, The hemispherical cube map projection layout is a vertically packed hemispherical cube map layout. The plurality of projection faces include half-faces packed at face positions with position indices. These half-faces are derived from the upper or lower half of a square face of the cube, and the value of the first syntax element assigned to the position index indicates the origin of the half-face. and The rotation angle selected in the group.

14. The video processing method as described in claim 9, characterized in that, The hemispherical cube map projection layout is a horizontally packed hemispherical cube map layout; decoding this portion of the bitstream to generate the decoded frame further includes: Multiple second syntax elements are parsed from the bitstream; Wherein, the values ​​of the plurality of second syntax elements respectively indicate the face indices of the plurality of projected faces packed at the plurality of face locations; the plurality of projected faces include full faces packed at face locations with first position indices and plurality of half faces packed at face locations with a plurality of different second position indices; the value of the second syntax element assigned to the first position index indicates a face index selected from a group consisting of face indices of the front face and face indices of the back face; and the value of the first syntax element assigned to any of the second position indices .... and The rotation angle selected in the group.

15. The video processing method as described in claim 9, characterized in that, The hemispherical cube map projection layout is a horizontally packed hemispherical cube map layout; decoding the portion of the bitstream to generate the decoded frame further includes: Multiple second syntax elements are parsed from the bitstream; Wherein, the values ​​of the plurality of second syntax elements respectively indicate the face indices of the plurality of projection faces packed at the plurality of face positions; the plurality of projection faces include: a full face packed at a face position having a first position index, a half face packed at a face position having a second position index, and a plurality of other half faces packed at a plurality of face positions having a plurality of different third position indices; the value of the second syntax element assigned to the first position index indicates a face index selected from the group consisting of the face index of the top face and the face index of the bottom face; the value of the second syntax element assigned to the second position index indicates the face index of the back face; the value of the first syntax element assigned to the second position index indicates a face index selected from the group consisting of the face index of the top face and the face index of the bottom face; and The rotation angle selected in the group; and the value of the first syntax element assigned to any third position index indicates the rotation angle selected from the group. and The rotation angle selected in the group.

16. The video processing method as described in claim 9, characterized in that, The hemispherical cube map projection layout is a horizontally packed hemispherical cube map layout; decoding the portion of the bitstream to generate the decoded frame further includes: Multiple second syntax elements are parsed from the bitstream; Wherein, the values ​​of the plurality of second syntax elements respectively indicate the face indices of the plurality of projection faces packed at the plurality of face positions; the plurality of projection faces include: a full face packed at a face position having a first position index, a half face packed at a face position having a second position index, and a plurality of other half faces packed at a plurality of face positions having a plurality of different third position indices; the value of the second syntax element assigned to the first position index indicates the face index selected from the group consisting of the face indices of the right face and the face indices of the left face; the value of the second syntax element assigned to the second position index indicates the face index of the front face; the value of the first syntax element assigned to the second position index indicates the face index selected from the group consisting of the face indices of the right face and the face index of the left face; and The rotation angle selected in the group; and the value of the first syntax element assigned to any third position index indicates the rotation angle selected from the group. and The rotation angle selected in the group.

17. The video processing method as described in claim 9, characterized in that, The hemispherical cube map projection layout is a vertically packed hemispherical cube map layout; decoding the portion of the bitstream to generate the decoded frame further includes: Multiple second syntax elements are parsed from the bitstream; Wherein, the values ​​of the plurality of second syntax elements respectively indicate the face indices of the plurality of projected faces packed at the plurality of face locations; the plurality of projected faces include: a full face packed at a face location having a first position index, and a plurality of half faces packed at face locations having a plurality of different second position indices; the value of the second syntax element assigned to the first position index indicates a face index selected from a group consisting of face indices of the front face and face indices of the back face; and the value of the first syntax element assigned to any of the second position indices indicates a face index selected from a group consisting of face indices of the front face and face indices of the back face; and The rotation angle selected in the group.

18. The video processing method as described in claim 9, characterized in that, The hemispherical cube map projection layout is a vertically packed hemispherical cube map layout; decoding the portion of the bitstream to generate the decoded frame further includes: Multiple second syntax elements are parsed from the bitstream; Wherein, the values ​​of the plurality of second syntax elements respectively indicate the face indices of the plurality of projection faces packed at the plurality of face positions; the plurality of projection faces include: a full face packed at a face position having a first position index, a half face packed at a face position having a second position index, and a plurality of other half faces packed at a plurality of face positions having a plurality of different third position indices; the value of the second syntax element assigned to the first position index indicates a face index selected from the group consisting of the face index of the top face and the face index of the bottom face; the value of the second syntax element assigned to the second position index indicates the face index of the back face; the value of the first syntax element assigned to the second position index indicates a face index selected from the group consisting of the face index of the top face and the face index of the bottom face; and The rotation angle selected in the group; and the value of the first syntax element assigned to any third position index indicates the rotation angle selected from the group. and The rotation angle selected in the group.

19. The video processing method as described in claim 9, characterized in that, The hemispherical cube map projection layout is a vertically packed hemispherical cube map layout; decoding the portion of the bitstream to generate the decoded frame further includes: Multiple second syntax elements are parsed from the bitstream; Wherein, the values ​​of the plurality of second syntax elements respectively indicate the face indices of the plurality of projected faces packed at the plurality of face positions; the plurality of projected faces include: a full face packed at a face position with a first position index, a half face packed at a face position with a second position index, and a plurality of other half faces packed at multiple face positions with multiple different third position indices; the value of the second syntax element assigned to the first position index indicates the face index selected from the group consisting of the face indices of the right face and the face index of the left face; the value of the second syntax element assigned to the second position index indicates the face index of the front face; the value of the first syntax element assigned to the second position index indicates the face index selected from the group consisting of the face indices of the right face and the face index of the left face; and The rotation angle selected in the group; and the value of the first syntax element assigned to any third position index indicates the rotation angle selected from the group. and The rotation angle selected in the group.

20. A video processing apparatus, comprising: A decoding circuit is configured to receive a bit stream and decode a portion of the bit stream to generate a decoded frame, wherein the decoding circuit parses a plurality of first syntax elements and a plurality of second syntax elements from the bit stream; The decoded frame is a projection-based frame, comprising multiple projection surfaces packed at multiple face locations with multiple different position indices in a hemispherical cube map projection layout; a portion of the 360-degree content of the sphere is mapped onto the multiple projection surfaces via the hemispherical cube map projection; the values ​​of the multiple second syntax elements respectively indicate the face indices of the multiple projection surfaces packed at the multiple face locations, and are constrained to meet bitstream consistency requirements; the multiple projection surfaces comprise full faces packed at face locations with position indices, the values ​​of the syntax elements assigned to the position indices indicate the face index of one of the top, bottom, front, back, left, and right faces, and the values ​​of the syntax elements assigned to any other position indices impose a face index different from the value of the syntax element of the position indices; the values ​​of the multiple first syntax elements respectively indicate the rotation angle of the multiple projection surfaces packed at the multiple face locations; and a portion of the multiple first syntax elements is constrained to meet bitstream consistency requirements; wherein the multiple projection surfaces located in the hemispherical cube map projection layout comprise four halves, and a portion of the multiple values ​​of the multiple first syntax elements includes values ​​indicating the rotation angles of the four halves.

Images