Video processing methods and apparatus
By introducing the syntax elements gcmp_face_index and gcmp_face_rotation into the video processing method, the problem of insufficient face packing constraints and rotation in the hemispherical cube map projection layout is solved. This enables effective packing and rotation of the projection face in the hemispherical cube map projection layout, ensuring image quality and bitstream consistency, and supporting efficient video transmission and display.
Patent Information
- Application Number
- CN202611214310.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2021-02-16
- Filing Date
- 2021-03-05
- Publication Date
- 2026-11-17
AI Technical Summary
Existing technologies struggle to ensure image quality and bitstream consistency when processing videos based on hemispherical cube map projection layouts, especially given the deficiencies in the application of face packing constraints and face rotation in hemispherical cube map projection layouts.
By introducing the syntax elements gcmp_face_index and gcmp_face_rotation into the video processing method to indicate the face index and rotation angle respectively, and constraining them to meet the requirements of bitstream consistency, the correct packing and rotation of the projected faces in the hemispherical cube map projection layout is ensured.
It achieves efficient packing and rotation of projection surfaces in a hemispherical cube map projection layout, ensuring consistency of image quality and bitstream, and supporting efficient video transmission and display.
Smart Images

Figure CN123123554A_ABST
Abstract
Description
[0001] Related cross-references This application claims priority to U.S. Provisional Application No. 63 / 037,064, filed on June 10, 2020, which is incorporated herein by reference. Technical Field
[0002] The present invention relates to video processing, and more specifically, to a video processing method for processing projection-based frames having been packed in a video processing apparatus having surface packing constraints (e.g., surface index constraints and / or surface rotation) and associated features. Background Technology
[0003] Virtual reality (VR) with a head-mounted display (HMD) is associated with a variety of applications. The ability to display wide-field-of-view content to users can be used to provide an immersive visual experience. The real-world environment must be captured from all directions to produce omnidirectional video corresponding to the field of view. With advancements in camera equipment and HMDs, the delivery of VR content may soon become a bottleneck due to the high bitrates required to represent this 360-degree content. When the resolution of omnidirectional video is 4K or higher, data compression / encoding is crucial for reducing the bitrate.
[0004] Generally, the omnidirectional video corresponding to a sphere is converted into frames with 360-degree image content. These frames are represented by one or more projection surfaces arranged in a 360-degree virtual reality (360VR) projection layout. The generated frames are then encoded into a bitstream for transmission. The bitstream generated from the encoder side is received and decoded by the decoder side. Furthermore, the configuration of the adopted 360VR projection layout can also be sent from the encoder side to the decoder side. When the adopted 360VR projection layout is a hemispherical cubemap projection layout, the packaged projection surfaces in the projection-based frame include a full face and four half faces, where the full face is square, and each half face is not square. To ensure the image quality of projection-based frames using a hemispherical cubemap projection layout, an innovative design is needed to apply face packing constraints to the hemispherical cubemap projection layout. Summary of the Invention
[0005] One object of the claimed invention is to provide a video processing method for processing projection-based frames having projection surfaces packed in a hemispherical cubemap projection layout with face packing constraints (e.g., face index constraints and / or face rotation), and to provide associated video processing apparatus.
[0006] According to a first aspect of the present invention, an exemplary video processing method is disclosed. This exemplary video processing method includes the steps of receiving a bitstream and decoding a portion of the bitstream by a decoding circuit to generate a decoded frame, the step including parsing a plurality of syntax elements from the bitstream. The decoded frame is a projection-based frame comprising a plurality of projection surfaces, said plurality of projection surfaces being packed at multiple face locations with different position indices in a hemispherical cubemap projection layout. A portion of the 360-degree content of the sphere is mapped onto the plurality of projection surfaces via hemispherical cubemap projection. The values of the plurality of syntax elements respectively indicate the face indices of the plurality of projection surfaces packed at the multiple face locations, and are constrained to meet bitstream consistency requirements.
[0007] According to a second aspect of the present invention, an exemplary video processing method is disclosed. This exemplary video processing method includes the steps of receiving a bitstream and decoding a portion of the bitstream by a decoding circuit to generate a decoded frame, the step including parsing a plurality of first syntax elements from the bitstream. The decoded frame is a projection-based frame comprising a plurality of projection planes, the plurality of projection planes being packed at multiple face positions with different position indices in a hemispherical cube map projection layout. A portion of the 360-degree content of the sphere is mapped onto the plurality of projection planes via hemispherical cube map projection. The values of the plurality of first syntax elements respectively indicate the rotation angle of the plurality of projection planes packed at the multiple face positions. A portion of the values of the plurality of first syntax elements is constrained to meet bitstream consistency requirements.
[0008] According to a third aspect of the present invention, an exemplary video processing apparatus is disclosed. The exemplary video processing apparatus includes a decoding circuit arranged to receive a bitstream and decode a portion of the bitstream to generate a decoded frame. The decoding circuit parses a plurality of first syntax elements and a plurality of second syntax elements from the bitstream, wherein the decoded frame is a projection-based frame comprising a plurality of projection surfaces, the plurality of projection surfaces being packaged at multiple face positions with different position indices in a hemispherical cube map projection layout; a portion of 360-degree content of a sphere is mapped onto the plurality of projection surfaces via hemispherical cube map projection; the values of the plurality of second syntax elements respectively indicate the face indices of the plurality of projection surfaces packaged at the multiple face positions, and are constrained to meet bitstream consistency requirements; the values of the plurality of first syntax elements respectively indicate the rotation angles of the plurality of projection surfaces packaged at the multiple face positions; a portion of the values of the plurality of first syntax elements is constrained to meet bitstream consistency requirements.
[0009] These and other objects of the invention will undoubtedly become apparent to those skilled in the art after reading the following detailed description of the preferred embodiments shown in the various accompanying drawings. Attached Figure Description
[0010] Figure 1 A diagram illustrating a 360-degree virtual reality (360 VR) system according to an embodiment of the present invention.
[0011] Figure 2 A diagram illustrating a hemispherical cube mapping projection according to an embodiment of the present invention is shown.
[0012] Figure 3 A diagram illustrating a horizontally packaged hemispherical cube map projection layout according to an embodiment of the present invention.
[0013] Figure 4 A diagram illustrating a vertically packaged hemispherical cube texture projection layout according to an embodiment of the present invention.
[0014] Figure 5 A diagram illustrating a specification of the syntax element gcmp_face_rotation[i] according to an embodiment of the present invention is shown.
[0015] Figure 6 A diagram illustrating another specification of the syntax element gcmp_face_rotation[i] according to an embodiment of the present invention is shown.
[0016] Figure 7 A diagram illustrating the coordinate definition according to an embodiment of the present invention is shown.
[0017] Figure 8 The diagram shows an example of a hemispherical cube map projection layout where the front face is selected as full.
[0018] Figure 9 The diagram shows an example of a hemispherical cube map projection layout where the back face is selected as full.
[0019] Figure 10 The diagram shows an example of four hemispheres selected for a hemispherical cubemap projection layout when the left side is selected as full.
[0020] Figure 11 The diagram shows an example of four hemispheres selected for a hemispherical cubemap projection layout when the right side is selected as full.
[0021] Figure 12 The diagram shows an example of four hemispheres selected for a hemispherical cubemap projection layout when the top face is selected as full.
[0022] Figure 13 The diagram shows an example of four hemispheres selected for a hemispherical cube map projection layout when the bottom face is selected as full.
[0023] Figure 14A diagram illustrating a rotation angle constraint of a half-face packed in a horizontally packed hemispherical cube map projection layout according to an embodiment of the present invention.
[0024] Figure 15 A diagram illustrating another rotation angle constraint of a half-face packed in a horizontally packed hemispherical cube map projection layout according to an embodiment of the present invention.
[0025] Figure 16 A diagram illustrating a rotation angle constraint of a half-face packed in a vertically packed hemispherical cube map projection layout according to an embodiment of the present invention.
[0026] Figure 17 The diagram illustrates another rotation angle constraint of a half-face packed in a vertically packed hemispherical cube map projection layout according to an embodiment of the present invention. Detailed Implementation
[0027] Specific terms are used in the following description and claims to refer to specific elements. As those skilled in the art will understand, electronic device manufacturers may use different names to refer to an element. This document is not intended to distinguish between elements with different names but the same function. In the following description and claims, the terms "comprising" and "including" are used in an open-ended manner and should therefore be interpreted as meaning "including but not limited to...". Furthermore, the term "coupled" is intended to indicate an indirect or direct electrical connection. Thus, if one device is coupled to another device, the connection can be through a direct electrical connection or through an indirect electrical connection via other devices and connections.
[0028] Figure 1 A diagram illustrating a 360-degree virtual reality (360 VR) system according to an embodiment of the present invention is shown. The 360 VR system 100 includes two video processing devices (e.g., a source electronics device 102 and a target electronics device 104). The source electronics device 102 includes a video acquisition device 112, a conversion circuit 114, and a video encoder 116. For example, the video acquisition device 112 may be an omnidirectional camera. The conversion circuit 114 generates a projection-based frame IMG with a 360-degree virtual reality (360 VR) projection layout L_VR based on an omnidirectional video frame S_IN corresponding to a sphere, wherein the omnidirectional video frame S_IN contains 360-degree content of the sphere. The video encoder 116 is an encoding circuit for encoding the projection-based frame IMG (which has a projection surface packed in the 360 VR projection layout L_VR) to generate a portion of a bitstream BS, and outputting the bitstream BS to the target electronics device 104 via a transmission device 103 such as a wired / wireless communication link or storage medium.
[0029] The target electronic device 104 can be a head-mounted display (HMD) device. For example... Figure 1 As shown, the target electronic device 104 includes a video decoder 122, a graphics rendering circuit 124, and a display device 126. The video decoder 122 is a decoding circuit that receives a bitstream BS from a transmission device 103 (e.g., a wired / wireless communication link or storage medium) and decodes a portion of the received bitstream BS to generate a decoded frame IMG'. In this embodiment, the projection-based frame IMG to be encoded by the video encoder 116 has a 360VR projection layout L_VR. Therefore, after a portion of the bitstream BS is decoded by the video decoder 122, the decoded frame (i.e., the reconstructed frame) IMG' has the same 360VR projection layout L_VR. In other words, the decoded frame IMG' is also a projection-based frame, having one or more projection surfaces packed within the 360VR projection layout L_VR. Hereinafter, the terms "decoded frame" and "projection-based frame" are used interchangeably. The graphics rendering circuit 124 drives the display device 126 to display image content of a viewport area selected by the user.
[0030] In this embodiment, the 360VR projection layout L_VR can be a hemispherical cube map projection layout. Specifically, a cube-based projection with five projection planes (including one full face and four half faces) representing 180°x180° omnidirectional video (i.e., only a portion of the 360-degree content of the sphere) can be employed. Regarding the conversion circuitry 114 of the source electronics 102, the hemispherical cube map projection is used to generate one full face and four half faces of a cube in three-dimensional (3D) space. Figure 2 A diagram illustrating a hemispherical cubemap projection according to an embodiment of the present invention is shown. Only half of the 360-degree content of the sphere 200 is projected onto multiple faces of the cube 201, including the top half (labeled "Top_H"), bottom half (labeled "Bottom_H"), left half (labeled "Left_H"), front half (labeled "Front"), and right half (labeled "Right_H"). In this example, half of the cube 201 is used for the hemispherical cubemap projection, wherein the hemisphere (e.g., half of the sphere 200) is inscribed within half of the cube 201. See also... Figure 2The image content of half of the North Pole region of sphere 200 is projected onto the top half "Top_H" (i.e., half of the top face of cube 201), the image content of half of the South Pole region of sphere 200 is projected onto the bottom half "Bottom_H" (i.e., half of the bottom face of cube 201), and the image content of half of the equatorial region of sphere 200 is projected onto the left half "Left_H" (half of the left face of cube 201), the front half "Front" (the front face of cube 201), and the right half "Right_H" (half of the right face of cube 201).
[0031] In a 3D space defined by the x-axis, y-axis, and z-axis, each point on the five projection planes lies at (x, y, z), where In the example shown in Figure 2, the entire "Front" face lies on the x-plane at x=1, the top half "Top_H" face lies on the z-plane at z=1, the bottom half "Bottom_H" face lies on the z-plane at z=-1, the left half "Left_H" face lies on the y-plane at y=1, and the right half "Right_H" face lies on the y-plane at y=-1. In another design, the entire "Front" face can lie on the x-plane at x=1, the top half "Top_H" face can lie on the y-plane at y=1, the bottom half "Bottom_H" face can lie on the y-plane at y=-1, the right half "Right_H" face can lie on the z-plane at z=1, and the left half "Left_H" face can lie on the z-plane at z=-1.
[0032] Forward transformations can be used to transform from 3D space (x, y, z) to a 2D plane (u, v). Therefore, the top half "Top_H", bottom half "Bottom_H", left half "Left_H", front half "Front", and right half "Right_H" of cube 201 in 3D space can be transformed into the top half (labeled "2"), bottom half (labeled "3"), left half (labeled "5"), front half (labeled "0"), and right half (labeled "4") on a 2D plane. Each face lies on a 2D plane defined by the horizontal u-axis and the vertical v-axis, with each point located at (u, v). Furthermore, the front half (labeled "0") is twice the size of each of the top half (labeled "2"), bottom half (labeled "3"), left half (labeled "5"), and right half (labeled "4").
[0033] The inverse transformation can be used to transform from a 2D plane (u, v) to a 3D space (x, y, z). Therefore, the top half (labeled "2"), bottom half (labeled "3"), left half (labeled "5"), front half (labeled "0"), and right half (labeled "4") on the 2D plane can be transformed into the top half "Top_H", bottom half "Bottom_H", left half "Left_H", front half "Front", and right half "Right_H" of cube 201 in 3D space.
[0034] The conversion circuit 114 of the source electronics device 102 can employ an inverse transformation to generate a top half-face "2", a bottom half-face "3", a left half-face "5", a full front half-face "0", and a right half-face "4". The top half-face "2", bottom half-face "3", left half-face "5", full front half-face "0", and right half-face "4" on the two-dimensional plane are packaged to form a projection-based frame IMG, which will be encoded by the video encoder 116. The video decoder 122 receives a bitstream BS from the transmission device 103 and decodes a portion of the received bitstream BS to generate a decoded frame IMG' using the same projection layout L_VR (e.g., a hemispherical cube map layout) employed at the encoder end.
[0035] As described above, the top half "2", bottom half "3", left half "5", full face "0", and right half "4" are packaged to form a projection-based frame IMG. For example, the conversion circuit 114 can select a packaging type such that the projection-based frame IMG can have projected image data arranged in a cube-based projection layout (hemispherical cube mapping layout) 202. As another example, the conversion circuit 114 can select another packaging type such that the projection-based frame IMG can have projected image data arranged in another cube-based projection layout (hemispherical cube mapping layout) 204, which is different from the cube-based projection layout (hemispherical cube mapping layout) 202. In this embodiment, the front face is selected as the full face packaged in the cube-based projection layouts 202 / 204. In fact, the full face packed in the cube-based projection layout 202 / 204 can be any of the top face, bottom face, front face, back face, left face, and right face, and the four half faces packed in the cube-based projection layout 202 / 204 depend on the choice of full face.
[0036] Cube-based projection layout 202 is also known as a horizontally packed hemispherical cubemap projection layout, in which all projection faces are horizontally packed. Cube-based projection layout 204 is also known as a vertically packed hemispherical cubemap projection layout, in which all projection faces are vertically packed. Figure 3 A diagram illustrating a horizontally packed hemispherical cube map projection layout according to an embodiment of the present invention is shown. The horizontally packed hemispherical cube map projection layout 300 includes five face positions with different position indices “0”, “1”, “2”, “3”, and “4”, wherein the entire face is packed at the face position with position index “2”, and the four half faces are packed at the face positions with position indices “0”, “1”, “3”, and “4”, respectively. Figure 4 A diagram illustrating a vertically packed hemispherical cube map projection layout according to an embodiment of the present invention is shown. The vertically packed hemispherical cube map projection layout 400 includes five face positions with different position indices “0”, “1”, “2”, “3”, and “4”, wherein the entire face is packed at the face position with position index “2”, and the four half-faces are packed at the face positions with position indices “0”, “1”, “3”, and “4”, respectively.
[0037] The conversion circuit 114 can determine the configuration of the projection-based frame IMG, which is generated from a cube-based projection (e.g., Figure 2 The hemispherical cube map projection shown in the diagram comprises multiple projection surfaces, and the video encoder 116 can send one or more syntax elements SE via the bitstream BS. For example, one or more syntax elements SE can be associated with a packing of projection surfaces in the projection-based frame IMG. Therefore, the video decoder 122 can parse one or more syntax elements SE' associated with the packing configuration of projection surfaces in the projection-based frame IMG from the bitstream BS, and can provide the parsed one or more syntax elements SE' to the graphics rendering circuit 124 so that the graphics rendering circuit 124 is informed of the surface packing configuration information. It should be noted that the projection-based frame IMG and the decoded frame IMG' have the same surface packing configuration. In this way, when determining the image content of the viewport region selected by the user, the graphics rendering circuit 124 can refer to the surface packing configuration information to correctly determine the pixel values of the selected viewport region. Ideally, one or more syntax elements SE encoded by the video encoder 116 into the bitstream BS are the same as one or more syntax elements SE' parsed by the video decoder 122 from the bitstream BS. For example, one or more syntax elements SE / SE' represent face packing configurations that may include gcmp_packing_type, gcmp_face_rotation, gcmp_face_index, etc.
[0038] The syntax element `gcmp_packing_type` specifies the packing type of the projected faces in a cube-based projection layout, and further specifies a predefined arrangement of position indices assigned to the face positions under the selected packing type. When the value of `gcmp_packing_type` is 4 or 5, a hemispherical cube map packing with one full face and four half faces is used, where each packing type is associated with five face positions respectively assigned position indices {0, 1, 2, 3, 4}. Specifically, when... Figure 3 The horizontally packed hemispherical cube map projection layout 300 shown is adopted, with the value of gcmp_packing_type equal to 4; and when... Figure 4 The vertically packed hemispherical cube map projection layout 400 shown is used, and the value of gcmp_packing_type is equal to 5.
[0039] The syntax element `gcmp_face_index[i]` can specify the face index for position index `i` under the packing type specified by the syntax element `gcmp_packing_type`. Taking a hemispherical cube map projection as an example, the assignable face index for the front full / half face is `gcmp_face_index[i] = 0`, the assignable face index for the back full / half face is `gcmp_face_index[i] = 1`, the assignable face index for the top full / half face is `gcmp_face_index[i] = 2`, the assignable face index for the bottom full / half face is `gcmp_face_index[i] = 3`, the assignable face index for the right full / half face is `gcmp_face_index[i] = 4`, and the assignable face index for the left full / half face is `gcmp_face_index[i] = 5`. Therefore, when the syntax element gcmp_packing_type is set to 4 or 5, the syntax element gcmp_face_index[i] specifies the face index of the projected face (e.g., front face "0", back face "1", top face "2", bottom face "3", right face "4" or left face "5" as shown in Figure 2), where the projected face with the face index specified by the syntax element gcmp_face_index[i] is packed at the face position with position index i under the selected packing type.
[0040] The syntax element gcmp_face_rotation[i] specifies a rotation with a rotation angle applied to the projected face packed at the face position with position index i under the packing type specified by the syntax element gcmp_packing_type. Figure 5A diagram illustrating a specification of the syntax element gcmp_face_rotation[i] according to an embodiment of the present invention is shown. Therefore, when applied to the rotation of the projected surface to be packaged at the face position with position index i, the rotation angle is... When the syntax element gcmp_face_rotation[i] is set to 0, the rotation angle applied to the rotation of the projection surface to be packed at position index i is counterclockwise. When the syntax element gcmp_face_rotation[i] is set to 1, the rotation angle applied to the rotation of the projection plane to be packed at position index i is counterclockwise. When the syntax element gcmp_face_rotation[i] is set to 2, the rotation angle applied to the rotation of the projection plane to be packed at position index i is counterclockwise. At that time, the syntax element gcmp_face_rotation[i] is set to 3.
[0041] In some embodiments of the present invention, the input to the rotation process may include the width (faceWidth) and height (faceHeight) of the projection plane (full surface), and may also include the sample positions (hPosFace, vPosFace) within the projection plane on the 2D plane. The width (faceWidth) may be the same as the height (faceHeight). The output of the rotation process may include the rotated sample positions (hPosRot, vPosRot) within the projection plane on the 2D plane. The output can be derived as follows: If (gcmp_face_rotation[i] = = 0 ) { hPosRot = hPosFace vPosRot = vPosFace } else if (gcmp_face_rotation[i] == 1 ) { hPosRot = vPosFace vPosRot = faceWidth - hPosFace } else if (gcmp_face_rotation[i] == 2 ) { hPosRot = faceWidth - hPosFace vPosRot = faceHeight – vPosFace } else if (gcmp_face_rotation[i] == 3 ) { hPosRot = faceHeight – vPosFace vPosRot = hPosFace } In the examples above, the rotation applied to a projection surface is defined by a rotation angle in the counterclockwise direction. However, this is for illustrative purposes only and is not intended to limit the invention. In another design, the rotation applied to a projection surface can be defined by a rotation angle in the clockwise direction. Figure 6 A diagram illustrating another specification of the syntax element gcmp_face_rotation[i] according to an embodiment of the present invention is shown.
[0042] As described above, a full face and four halves are packed within a hemispherical cubemap projection layout, such as a horizontally packed hemispherical cubemap projection layout 300 (gcmp_packing_type == 4) or a vertically packed hemispherical cubemap projection layout 400 (gcmp_packing_type == 5). This is to represent continuous... x For omnidirectional video content, the four halves packed in a hemispherical cube map projection layout on the 2D plane should originate from the halves that are fully connected (adjacent) to the 3D face. This embodiment proposes applying constraints to the selection of the halves packed in the hemispherical cube map projection layout. Specifically, the conversion circuit 114 is configured to impose constraints on the face indices of the halves packed at face positions with position indices {0, 1, 3, 4} to satisfy bitstream consistency requirements. The syntax element `gcmp_face_index[i]` indicating the face index of the halves packed in the hemispherical cube map projection layout is emitted via the bitstream BS. Through the proposed constraints on the face indices, a bitstream BS satisfying the bitstream consistency requirements is generated and transmitted from the source electronic device 102 to the target electronic device 104. At the target electronic device 104, the bitstream BS satisfying the bitstream consistency requirements does not contain invalid syntax values and can be decoded at the video decoder 122. The video decoder 122 can parse one or more syntax elements SE' associated with a packing configuration of the projection plane in the projection-based frame IMG from the bitstream BS, and can provide the parsed one or more syntax elements SE' to the graphics rendering circuit 124, allowing the graphics rendering circuit 124 to correctly process the post-processing of the decoded frame IMG' with the same hemispherical cube map projection layout.
[0043] Regarding the application of constraints to the face indices of half-faces packaged within a hemispherical cube map projection layout, the coordinates used are defined in... Figure 7 As shown, the viewing angle is from the origin of sphere 200 outwards into the interior of sphere 200. Each projection plane derived from the square face of the cube in 3D space lies on a 2D plane defined by the u-axis and v-axis, and has a local sample position at (u, v) within the projection plane, with its coordinate origin as shown. Figure 7 As shown.
[0044] As mentioned above, the full face to be packed in a hemispherical cube map projection layout can be any of the top, bottom, front, back, left, and right faces, and the four halves packed in the hemispherical cube map projection layout depend on the choice of the full face. The half to be packed in the hemispherical cube map projection layout can be half of the front face, half of the back face, half of the left face, half of the right face, half of the top face, or half of the bottom face, depending on the choice of the full face to be filled in the hemispherical cube map projection layout and the spatial relationship between the full face and the half faces.
[0045] Figure 8 This diagram shows an example of four hemispheres selected for a hemispherical cubemap projection layout when the front face is selected as full. According to... Figure 7 The coordinates shown indicate that the four halves of the hemispherical cube map projection layout include the right half of the left face, the left half of the right face, the right half of the top face, and the left half of the bottom face.
[0046] Figure 9 This diagram shows an example of four hemispheres selected for a hemispherical cubemap projection layout when the back face is selected as full. According to... Figure 7 The coordinates shown indicate that the four halves of the hemispherical cube map projection layout include the left half of the left face, the right half of the right face, the left half of the top face, and the right half of the bottom face.
[0047] Figure 10 This diagram shows an example of four hemispheres selected for a hemispherical cubemap projection layout when the left side is chosen as the full side. According to... Figure 7 As shown in the coordinate definition, the four halves of the hemispherical cube map projection layout include the lower half of the back face, the left half of the front face, the lower half of the top face, and the lower half of the bottom face.
[0048] Figure 11 This diagram shows an example of four hemispheres selected for a hemispherical cubemap projection layout when the right side is selected as the full side. According to... Figure 7As shown in the coordinate definition, the four halves of the hemispherical cube map projection layout include the upper half of the back face, the right half of the front face, the upper half of the top face, and the upper half of the bottom face.
[0049] Figure 12 This diagram shows an example of four hemispheres selected for a hemispherical cubemap projection layout when the top face is selected as full. According to... Figure 7 As shown in the coordinate definition, the four halves of the hemispherical cube map projection layout include the upper half of the left face, the upper half of the front face, the right half of the back face, and the upper half of the right face.
[0050] Figure 13 This diagram shows an example of four hemispheres selected for a hemispherical cubemap projection layout when the bottom face is selected as the full face. According to... Figure 7 As shown in the coordinate definition, the four halves of the hemispherical cube map projection layout include the lower half of the left face, the lower half of the front face, the left half of the back face, and the lower half of the right face.
[0051] When the value of gcmp_packing_type is set to 4 or 5, the constraint on the opposite index value can guarantee that the projection-based frame IMG / IMG' under the selected packing type contains a contiguous sphere 200. x Omnidirectional video content. Based on the overall selection specified by gcmp_face_index[2], the four halves can be determined and specified by gcmp_face_index[i] of i = {0, 1, 3, 4} to achieve continuous projection-based frame IMG / IMG'. x The target of omnidirectional video content. For example, when gcmp_packing_type equals 4 or 5, the following constraints apply to bitstream consistency requirements: - If gcmp_face_index[2] is equal to 0 or 1, then for i equal to 0, 1, 3 or 4, the value of gcmp_face_index[i] should be in the range of 2 to 5, inclusive.
[0052] - Otherwise, if gcmp_face_index[2] is equal to 2 or 3, then for i equal to 0, 1, 3 or 4, the value of gcmp_face_index[i] should be 0, 1, 4 or 5.
[0053] Otherwise, when i equals 0, 1, 3 or 4, the value of gcmp_face_index[i] should be in the range of 0 to 3, inclusive.
[0054] In the first case, the entire face packed at position index 2 is set by either the front or back face. The four faces packed at positions indexes 0, 1, 3, and 4 are constrained to be the left half, the right half, the top half, and the bottom half, as shown below. Figure 8-9 As shown. In the second case, the entire face packed at position index 2 is set by either the top or bottom face, while the four faces packed at positions indexes 0, 1, 3, and 4 are constrained to be half of the front face, half of the back face, half of the right face, and half of the left face, as shown. Figure 12-13 As shown. In the third case, the entire face packed at position index 2 is set by either the right or left face, while the four faces packed at positions indexes 0, 1, 3, and 4 are constrained to be half of the front face, half of the back face, half of the top face, and half of the bottom face, as shown. Figure 10-11 As shown.
[0055] As described above, a rotation with an angle is applied to the projected face packed at the face position with a position index under the selected packing type. That is, the rotated face is packed in a hemispherical cubemap projection layout, where the rotation information is specified by the syntax element `gcmp_face_rotation`. The rotation angle of the half-face should be constrained to ensure that the rotated half-face can be accommodated in the hemispherical cubemap projection layout specified by the syntax element `gcmp_packing_type`.
[0056] For example, when a horizontally packed hemispherical cube map projection layout 300 (gcmp_packing_type = 4) is used, the rotation of the half-face is constrained. If the half-face packed at a face position where position index i is not equal to 2 (i.e., i = 0, 1, 3, or 4) is derived from the upper or lower half of a square face in 3D space, the value of the syntax element gcmp_face_rotation[i] assigned to position index i indicates the rotation from the upper or lower half of the face. and The rotation angle selected from the combinations formed. That is, the rotation angle should be clockwise / counterclockwise. or To ensure that the rotated half-face can be accommodated within the horizontally packed hemispherical cube map projection layout 300, such as... Figure 14 As shown. If a half-face packed at a position index i not equal to 2 (i.e., i = 0, 1, 3, or 4) is derived from the left or right half of a square face of a cube in 3D space, the value of the syntax element gcmp_face_rotation[i] assigned to position index i indicates the origin from the left or right half of the square face of a cube in 3D space. and The rotation angle selected from the combinations formed. That is, the rotation angle should be clockwise / counterclockwise. or To ensure that the rotated half-face can be accommodated within the horizontally packed hemispherical cube map projection layout 300, such as... Figure 15 As shown.
[0057] For another example, when using a vertically packed hemispherical cubemap projection layout 400 (gcmp_packing_type = 5), the rotation of the half-face is constrained. If the half-face packed at a face position where position index i is not equal to 2 (i.e., i = 0, 1, 3, or 4) is derived from the upper or lower half of a cube square face in 3D space, the value of the syntax element gcmp_face_rotation[i] assigned to position index i indicates the rotation from the upper or lower half of the face. and The rotation angle selected from the combinations formed. That is, the rotation angle should be clockwise / counterclockwise. or To ensure that the rotated half-face can be accommodated in the vertically packed hemispherical cube texture projection layout 400, such as... Figure 16 As shown. If a half-face packed at a face position where position index i is not equal to 2 (i.e., i = 0, 1, 3, or 4) is derived from the left or right half of a square face of a cube in 3D space, the value of the syntax element gcmp_face_rotation[i] assigned to position index i indicates the origin from the left or right half of the square face of a cube in 3D space. and The rotation angle selected from the combinations formed. That is, the rotation angle should be clockwise / counterclockwise. or To ensure that the rotated half-face can be accommodated in the vertically packed hemispherical cube texture projection layout 400, such as... Figure 17 As shown.
[0058] In summary, when gcmp_packing_type equals 4, the bitstream consistency requirement applies the following constraints: — If gcmp_face_index[2] is equal to 0 or 1, then for i equal to 0, 1, 3 or 4, the value of gcmp_face_rotation[i] should be 0 or 2.
[0059] Otherwise, if gcmp_face_index[2] equals 2 or 3, when gcmp_face_index[i] equals 1, the value of gcmp_face_rotation[i] should be 0 or 2, and when gcmp_face_index[i] equals 0, 4 or 5, the value of gcmp_face_rotation[i] should be 1 or 3.
[0060] Otherwise, when gcmp_face_index[i] equals 0, the value of gcmp_face_rotation[i] should be 0 or 2, and when gcmp_face_index[i] equals 1, 2 or 3, the value of gcmp_face_rotation[i] should be 1 or 3.
[0061] When gcmp_packing_type equals 5, the bitstream consistency requirement applies the following constraints: – If gcmp_face_index[2] is equal to 0 or 1, then the value of gcmp_face_rotation[i] should be 1 or 3 for i equal to 0, 1, 3 or 4.
[0062] Otherwise, if gcmp_face_index[2] is equal to 2 or 3, the value of gcmp_face_rotation[i] should be 1 or 3 when gcmp_face_index[i] is equal to 1, and the value of gcmp_face_rotation[i] should be 0 or 2 when gcmp_face_index[i] is equal to 0, 4 or 5.
[0063] Otherwise, when gcmp_face_index[i] equals 0, the value of gcmp_face_rotation[i] should be 1 or 3, and when gcmp_face_index[i] equals 1, 2 or 3, the value of gcmp_face_rotation[i] should be 0 or 2.
[0064] Those skilled in the art will readily observe that many modifications and alterations can be made to the apparatus and method while retaining the teachings of the present invention. Therefore, the above disclosure should be construed as being limited only by the scope and limitations of the appended claims.
Claims
1. A video processing method, comprising: Receive bit stream; as well as Decoding a portion of the bitstream to generate a decoded frame includes: Multiple syntax elements are parsed from the bitstream; wherein the decoded frame is a projection-based frame, the projection-based frame comprising multiple projection faces packed at multiple face locations with different position indices in a hemispherical cube map projection layout; the multiple projection faces packed at the multiple face locations in the hemispherical cube map projection layout comprise a full face; a portion of the 360-degree content of the sphere is mapped onto the multiple projection faces via hemispherical cube map projection; the values of the multiple syntax elements respectively indicate the face indices of the multiple projection faces packed at the multiple face locations, and are constrained to meet bitstream consistency requirements; and the constraints applied to the values of the multiple syntax elements when the full face is set by a first face that is one of the top face, bottom face, front face, back face, left face, and right face are different from the constraints applied to the values of the multiple syntax elements when the full face is set by a second face adjacent to the first face.
2. The video processing method as described in claim 1, characterized in that, The plurality of projection surfaces include a full surface packed at a surface location with a position index, wherein the value of the syntax element assigned to the position index indicates the face index of the front face, and the value of the syntax element assigned to any other position index indicates a face index selected from the group consisting of the face indexes of the left face, the right face, the top face, and the bottom face.
3. The video processing method as described in claim 1, characterized in that, The plurality of projection surfaces include a full surface packed at a surface location with a position index, the value of the syntax element assigned to the position index indicating the face index of the back face, and the value of the syntax element assigned to any other position index indicating a face index selected from the group consisting of the face index of the left face, the face index of the right face, the face index of the top face, and the face index of the bottom face.
4. The video processing method as described in claim 1, characterized in that, The plurality of projection surfaces include a full surface packed at a surface location with a position index, wherein the value of the syntax element assigned to the position index indicates the surface index of the top surface, and the value of the syntax element assigned to any other position index indicates a surface index selected from the group consisting of the surface index of the left surface, the surface index of the front surface, the surface index of the back surface, and the surface index of the right surface.
5. The video processing method as described in claim 1, characterized in that, The plurality of projection surfaces include a full surface packed at a surface location with a position index, the value of the syntax element assigned to the position index indicating the surface index of the bottom surface, and the value of the syntax element assigned to any other position index indicating a surface index selected from the group consisting of the surface index of the left surface, the surface index of the front surface, the surface index of the back surface, and the surface index of the right surface.
6. The video processing method as described in claim 1, characterized in that, The plurality of projection surfaces include a full surface packed at a surface location with a position index, the value of the syntax element assigned to the position index indicating the face index of the right side, and the value of the syntax element assigned to any other position index indicating a face index selected from the group consisting of the face index of the back side, the face index of the front side, the face index of the top side, and the face index of the bottom side.
7. The video processing method as described in claim 1, characterized in that, The plurality of projection surfaces include a full surface packed at a surface location with a position index, the value of the syntax element assigned to the position index indicating the face index of the left side, and the value of the syntax element assigned to any other position index indicating a face index selected from the group consisting of the face index of the back side, the face index of the front side, the face index of the top side, and the face index of the bottom side.
8. The video processing method as described in claim 1, characterized in that, The portion of the 360-degree content of the sphere is projected onto one square face and half of the other four square faces of a cube. The plurality of projection faces packaged in the hemispherical cube mapping projection layout include a full face and four half faces, the full face being derived from the square face and the four half faces being derived from the half faces of the other four square faces connected to the square face, respectively.
9. A video processing method, comprising: Receive bit stream; as well as Decoding a portion of the bitstream to generate a decoded frame includes: Multiple first syntax elements are parsed from the bitstream; wherein the decoded frame is a projection-based frame, the projection-based frame comprising multiple projection surfaces packed at multiple face positions with different position indices in a hemispherical cube map projection layout; a portion of the 360-degree content of the sphere is mapped onto the multiple projection surfaces via hemispherical cube map projection; the values of the multiple first syntax elements respectively indicate the rotation angle of the multiple projection surfaces packed at the multiple face positions; and a portion of the values of the multiple first syntax elements is constrained to meet bitstream consistency requirements, wherein the multiple projection surfaces packed at the multiple face positions in the hemispherical cube map projection layout comprise four half-faces, and the portion of the values of the multiple first syntax elements includes values indicating the rotation angles of the four half-faces.
10. The video processing method as described in claim 9, characterized in that, The hemispherical cube map projection layout is a horizontally packed hemispherical cube map layout, wherein the plurality of projection faces include half-faces packed at face positions with position indices, the half-faces being derived from the upper or lower half of the square face of the cube, and the value of the first syntax element assigned to the position index indicates a rotation angle selected from the group consisting of 90° and 270°.
11. The video processing method as described in claim 9, characterized in that, The hemispherical cube map projection layout is a horizontally packed hemispherical cube map layout, the plurality of projection faces including half faces packed at face positions with position indices, the half faces being derived from the left or right half of the square face of the cube, and the value of the first syntax element assigned to the position index indicating a rotation angle selected from the group consisting of 0° and 180°.
12. The video processing method as described in claim 9, characterized in that, The hemispherical cube map projection layout is a vertically packed hemispherical cube map layout, wherein the plurality of projection faces include half-faces packed at face positions with position indices, the half-faces being derived from the left or right half of the square face of the cube, and the value of the first syntax element assigned to the position index indicates a rotation angle selected from the group consisting of 90° and 270°.
13. The video processing method as described in claim 9, characterized in that, The hemispherical cube map projection layout is a vertically packed hemispherical cube map layout, wherein the plurality of projection faces include half-faces packed at face positions with position indices, the half-faces being derived from the upper or lower half of the square face of the cube, and the value of the first syntax element assigned to the position index indicates a rotation angle selected from the group consisting of 0° and 180°.
14. The video processing method as described in claim 9, characterized in that, The hemispherical cube map projection layout is a horizontally packed hemispherical cube map layout; decoding the portion of the bitstream to generate the decoded frame further includes: parsing a plurality of second syntax elements from the bitstream; wherein the values of the plurality of second syntax elements respectively indicate the face indices of the plurality of projection faces packed at the plurality of face positions; the plurality of projection faces includes full faces packed at face positions with a first position index and a plurality of half faces packed at face positions with a plurality of different second position indices; the value of the second syntax element assigned to the first position index indicates a face index selected from a group consisting of the face indices of the front face and the face indices of the back face; and the value of the first syntax element assigned to any of the second position indices indicates a rotation angle selected from a group consisting of 0° and 180°.
15. The video processing method as described in claim 9, characterized in that, The hemispherical cube map projection layout is a horizontally packed hemispherical cube map layout; decoding the portion of the bitstream to generate the decoded frame further includes: parsing a plurality of second syntax elements from the bitstream; wherein the values of the plurality of second syntax elements respectively indicate the face indices of the plurality of projection faces packed at the plurality of face positions; the plurality of projection faces include: a full face packed at a face position having a first position index, a half face packed at a face position having a second position index, and a remaining half face packed at a face position having a plurality of different third position indices; the value of the second syntax element assigned to the first position index indicates a face index selected from the group consisting of the face index of the top face and the face index of the bottom face; the value of the second syntax element assigned to the second position index indicates the face index of the back face; the value of the first syntax element assigned to the second position index indicates a rotation angle selected from the group consisting of 0° and 180°; and the value of the first syntax element assigned to any of the third position indices indicates a rotation angle selected from the group consisting of 90° and 270°.
16. The video processing method as described in claim 9, characterized in that, The hemispherical cube map projection layout is a horizontally packed hemispherical cube map layout; decoding the portion of the bitstream to generate the decoded frame further includes: parsing a plurality of second syntax elements from the bitstream; wherein the values of the plurality of second syntax elements respectively indicate the face indices of the plurality of projection faces packed at the plurality of face positions; the plurality of projection faces include: a full face packed at a face position having a first position index, a half face packed at a face position having a second position index, and a remaining half face packed at a face position having a plurality of different third position indices; the value of the second syntax element assigned to the first position index indicates a face index selected from the group consisting of the face indices of the right and the left; the value of the second syntax element assigned to the second position index indicates the face index of the front face; the value of the first syntax element assigned to the second position index indicates a rotation angle selected from the group consisting of 0° and 180°; and the value of the first syntax element assigned to any of the third position indices indicates a rotation angle selected from the group consisting of 90° and 270°.
17. The video processing method as described in claim 9, characterized in that, The hemispherical cube map projection layout is a vertically packed hemispherical cube map layout; decoding the portion of the bitstream to generate the decoded frame further includes: parsing a plurality of second syntax elements from the bitstream; wherein the values of the plurality of second syntax elements respectively indicate the face indices of the plurality of projection faces packed at the plurality of face positions; the plurality of projection faces includes: a full face packed at a face position having a first position index, and a plurality of half faces packed at face positions having a plurality of different second position indices; the value of the second syntax element assigned to the first position index indicates a face index selected from a group consisting of the face indices of the front face and the face indices of the back face; and the value of the first syntax element assigned to any of the second position indices indicates a rotation angle selected from a group consisting of 90° and 270°.
18. The video processing method as described in claim 9, characterized in that, The hemispherical cube map projection layout is a vertically packed hemispherical cube map layout; decoding the portion of the bitstream to generate the decoded frame further includes: Parse a plurality of second syntax elements from the bitstream; wherein the values of the plurality of second syntax elements respectively indicate the face indices of the plurality of projection faces packed at the plurality of face positions; the plurality of projection faces include: a full face packed at a face position having a first position index, a half face packed at a face position having a second position index, and a remaining half face packed at a face position having a plurality of different third position indices; the value of the second syntax element assigned to the first position index indicates a face index selected from the group consisting of the face index of the top face and the face index of the bottom face; the value of the second syntax element assigned to the second position index indicates the face index of the back face; the value of the first syntax element assigned to the second position index indicates a rotation angle selected from the group consisting of 90° and 270°; and the value of the first syntax element assigned to any of the third position indices indicates a rotation angle selected from the group consisting of 0° and 180°.
19. The video processing method as described in claim 9, characterized in that, The hemispherical cube map projection layout is a vertically packed hemispherical cube map layout; decoding the portion of the bitstream to generate the decoded frame further includes: Parse a plurality of second syntax elements from the bitstream; wherein the values of the plurality of second syntax elements respectively indicate the face indices of the plurality of projection faces packed at the plurality of face positions; the plurality of projection faces include: a full face packed at a face position having a first position index, a half face packed at a face position having a second position index, and a remaining half face packed at a face position having a plurality of different third position indices; the value of the second syntax element assigned to the first position index indicates a face index selected from the group consisting of the face indices of the right face and the face indices of the left face; the value of the second syntax element assigned to the second position index indicates the face index of the front face; the value of the first syntax element assigned to the second position index indicates a rotation angle selected from the group consisting of 90° and 270°; and the value of the first syntax element assigned to any of the third position indices indicates a rotation angle selected from the group consisting of 0° and 180°.
20. A video processing apparatus, comprising: A decoding circuit is configured to receive a bitstream and decode a portion of the bitstream to generate a decoded frame, wherein the decoding circuit parses a plurality of first syntax elements and a plurality of second syntax elements from the bitstream; The decoded frame is a projection-based frame, comprising multiple projection surfaces packed at multiple face locations with different position indices in a hemispherical cube map projection layout; a portion of the 360-degree content of the sphere is mapped onto the multiple projection surfaces via hemispherical cube map projection; the multiple projection surfaces packed at the multiple face locations in the hemispherical cube map projection layout include full faces and four half faces; the values of the multiple second syntax elements respectively indicate the face indices of the multiple projection surfaces packed at the multiple face locations, and are constrained to meet bitstream consistency requirements; the constraints applied to the values of the multiple syntax elements when the full face is set by a first face that is one of the top, bottom, front, back, left, and right faces are different from the constraints applied to the values of the multiple syntax elements when the full face is set by a second face adjacent to the first face; the values of the multiple first syntax elements respectively indicate the rotation angles of the multiple projection surfaces packed at the multiple face locations; and a portion of the values of the multiple first syntax elements is constrained to meet bitstream consistency requirements, wherein the portion of the values of the multiple first syntax elements includes values indicating the rotation angles of the four half faces.
