360-degree video encoding using geometry projection

Abstract

To provide projection geometry for coding a 360-video content.SOLUTION: Processing of video data can include capturing the video data with multiple cameras, and stitching the video data together to obtain a 360-degree video. A frame-packed picture can be provided based on the captured and stitched video data. A current sample location can be identified in the frame-packed picture. Whether a neighboring sample location is located outside a content boundary of the frame-packed picture can be determined. When the neighboring sample location is located outside the content boundary, a padding sample location can be derived based on at least one circular characteristic of the 360-degree video content and the projection geometry. The 360-degree video content can be processed based on the padding sample location.SELECTED DRAWING: Figure 28

Description

[Technical field]

[0001] The present invention relates to 360-degree video encoding using geometry projection. [Background technology]

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of U.S. Provisional Application No. 62 / 360,112, filed July 8, 2016, U.S. Provisional Application No. 62 / 404,017, filed October 4, 2016, U.S. Provisional Application No. 62 / 463,242, filed February 24, 2017, and U.S. Provisional Application No. 62 / 500,605, filed May 3, 2017, the contents of which are incorporated herein by reference, and this application claims the benefit of the filing dates of these priority applications.

[0003] Virtual reality (VR) is being applied in many application areas, including, but not limited to, healthcare, education, social networking, industrial design / training, gaming, movies, shopping, and / or entertainment. VR can enhance the viewer's experience, for example, by creating a virtual environment that surrounds the viewer, creating the feeling that the viewer is truly "there." VR systems can support interaction through posture, gesture, gaze, voice, etc. The system can provide haptic feedback to the user, allowing the user to interact with objects in the VR environment in a natural way. Summary of the Invention [Problem to be solved by the invention]

[0004] Provides projection geometry for encoding 360-degree video content. [Means for solving the problem]

[0005] Processing the video data may include capturing video data using multiple cameras and stitching the video data together to obtain a 360-degree video. As used herein, a 360-degree video may include a spherical video, an omnidirectional video, a virtual reality (VR) video, a panoramic video, an immersive video (e.g., a light field video that may include six degrees of freedom), and / or a point cloud video, etc. Geometry padding may be applied during encoding and / or decoding of the 360-degree video.

[0006] The 360-degree video content can be encoded. A frame picture can be provided based on video data captured using one or more cameras and stitching together the captured video data. A current sample location can be identified within the frame-packed picture. The frame-packed picture can include one or more faces associated with a first projection geometry. The current sample location can be associated with the first projection geometry. The first projection geometry can include at least one of, for example, an equirectangular projection, an equal-area projection, a cube-map projection, an octahedral projection, or a cylindrical projection, etc. The projection geometries described herein for encoding the 360-degree video content are used as exemplary methods, and those skilled in the art will understand that other projection formats can be used interchangeably.

[0007] It may be determined whether the neighboring sample location is located outside the content boundary of the frame packed picture. The content boundary may include the boundary of the frame packed picture, and / or a surface boundary, etc. If the neighboring sample location is located outside the content boundary of the frame packed picture, a padding sample location may be derived based on one or more circular characteristics of the 360-degree video content and the first projection geometry. A 3D position of the neighboring sample location may be calculated. The neighboring sample location may be calculated in association with a surface that includes the current sample location. A second surface that includes the padding sample location may be determined based on the 3D position of the neighboring sample location. The second surface may be different from the first surface that includes the current sample location. A geometric projection may be applied using the 3D position of the neighboring sample location to derive a 2D planar position within the second surface. The 2D planar position of the neighboring sample location within the second surface may be derived as a padding sample location based on a geometric projection of the 3D position of the neighboring sample location.

[0008] If the neighboring sample location is located inside the content boundary of the frame-packed picture, the derivation of the padding sample location can be skipped. 360-degree video content can be processed using the neighboring sample location (e.g., located inside the content boundary).

[0009] To derive the padding sample locations, an intermediate projection can be used. For example, coordinates in a first projection geometry can be transformed to intermediate coordinates associated with a second projection geometry. The first projection geometry can be a unicube projection and the second projection geometry can be a cubemap projection. 3D positions of neighboring sample locations can be calculated in the intermediate coordinates and 2D planar positions of neighboring sample locations within the second surface can be identified in the intermediate coordinates. The identified 2D planar positions of neighboring sample locations associated with the second projection geometry can be transformed back to coordinates associated with the first projection geometry.

[0010] The 360-degree video content can be processed based on the derived padding sample location. For example, padding sample values ​​can be derived at the padding sample location. The padding sample values ​​at the padding sample location can be derived using an interpolation filter. For example, a bilinear filter, a bicubic filter, a Lanczos filter, a spline interpolation filter, and / or a discrete cosine transform-based interpolation filter, etc. can be used. For example, inter prediction, intra prediction, deblocking, loop filtering, sample adaptive offsetting, and / or adaptive loop filtering, etc. can be performed to derive the padding sample value. When the current sample location is located on a content boundary, a deblocking filter based on the padding sample location can be applied.

[0011] According to the circular characteristic of the 360-degree video content, when the current sample location is located near the right-most content boundary of the frame packed picture and the determined neighboring sample location is located outside the frame packed picture beyond the right content boundary, the padding sample location can be located near the left-most content boundary of the frame packed picture that is circularly connected to the right-most content boundary of the frame packed picture. When the current sample location is located near the left-most content boundary of the frame packed picture and the determined neighboring sample location is located outside the frame packed picture beyond the left content boundary, the padding sample location can be located near the right-most content boundary of the frame packed picture that is circularly connected to the left-most content boundary of the frame packed picture. When the current sample location is located near a first area of ​​a top content boundary of the frame packed picture and the determined neighboring sample location is located outside the frame packed picture beyond the first area of ​​the top content boundary, the padding sample location can be located near a second area of ​​the top content boundary that is located on the opposite annular side of the first area of ​​the top content boundary.When the current sample location is located near a first area of ​​a bottom content boundary of the frame packed picture and the determined neighboring sample location is located outside the frame packed picture beyond the first area of ​​the bottom content boundary, the padding sample location can be located near a second area of ​​the bottom content boundary that is located on the opposite annular side of the first area of ​​the bottom content boundary.

[0012] To encode the 360-degree video content, a current block may be predicted based on the derived or mapped reference block. It may be determined whether an initial reference block associated with the current block is located outside the content boundary of the frame-packed picture. If the initial reference block is outside the content boundary, a mapped reference block may be derived based on one or more circular characteristics of the 360-degree video content and the first projection geometry. A 3D position of a location associated with the initial reference block may be identified. The location associated with the initial reference block may be calculated in association with a surface that includes the current block. A second surface that includes the mapped reference block may be identified based on the 3D position of the location associated with the initial reference block. The first surface may be different from the second surface. A geometric projection may be applied based on the 3D position of the location associated with the initial reference block. Based on the geometric projection, a 2D planar position of the initial reference block within the second surface may be derived for the mapped reference block.

[0013] If the initial reference block is inside the content boundary, the derivation of the mapped reference block may be skipped, and the current block may be predicted based on the initial reference block (e.g., positioned inside the content boundary).

[0014] An intermediate projection can be used to derive the mapped reference block. For example, coordinates in the first projection geometry can be transformed to intermediate coordinates associated with the second projection geometry. The first projection geometry can be a unicube projection and the second projection geometry can be a cubemap projection. A 3D position of a location associated with the initial reference block location can be calculated in the intermediate coordinates. A 2D planar position of the initial reference block within the second facet for deriving the mapped reference block can be identified in the intermediate coordinates. The identified 2D planar position of the initial reference block associated with the second projection geometry can be transformed back to coordinates associated with the first projection geometry.

[0015] When the current block is located near the right-most content boundary of the frame packed picture and the determined initial reference block is located outside the frame packed picture beyond the right content boundary, the mapped reference block may be located near the left-most content boundary of the frame packed picture that is circularly connected to the right-most content boundary of the frame packed picture. When the current block is located near the left-most content boundary of the frame packed picture and the determined initial reference block is located outside the frame packed picture beyond the left content boundary, the mapped reference block may be located near the right-most content boundary of the frame packed picture that is circularly connected to the left-most content boundary of the frame packed picture. When the current block is located near a first area of ​​the top content boundary of the frame packed picture and the determined initial reference block is located outside the frame packed picture beyond the first area of ​​the top content boundary, the mapped reference block can be located near a second area of ​​the top content boundary that is located on the opposite circular side of the first area of ​​the top content boundary.When the current block is located near a first area of ​​the bottom content boundary of the frame packed picture and the determined initial reference block is located outside the frame packed picture beyond the first area of ​​the bottom content boundary, the mapped reference block can be located near a second area of ​​the bottom content boundary that is located on the opposite circular side of the first area of ​​the bottom content boundary.

[0016] The sample values ​​associated with the mapped reference block may be derived using an interpolation filter, such as a bilinear filter, a bicubic filter, a Lanczos filter, a spline interpolation filter, and / or a discrete cosine transform-based interpolation filter.

[0017] To predict a current block, one or more attributes associated with the mapped reference block may be derived. For example, the attributes may include intra mode, inter mode, merge mode, and / or motion information of the location of the mapped block, etc. The current block may be predicted based on the attributes. To process 360-degree video content, sample values ​​in the mapped reference block may be derived. For example, inter prediction, intra prediction, deblocking, loop filtering, sample adaptive offsetting, and / or adaptive loop filtering, etc. may be used to derive sample values ​​in the mapped reference block for video processing.

[0018] The mapped reference block can be derived by unfolding the 3D geometry associated with the first projection geometry. When the initial reference block is not in a diagonal position with respect to the content boundary, the current block can be predicted by duplicating the nearest available block that is positioned perpendicular or diagonal to the content boundary.

[0019] The current block may be predicted by determining a valid padded area of ​​the surface based on the size of the surface. It may be determined whether the initial reference block is located within the valid padded area. When the initial reference block is located outside the valid padded area, the current block may be predicted based on the nearest available block located within the valid padded area. When the initial reference block is located inside the valid padded area, the current block may be predicted based on the mapped reference block.

[0020] The initial reference block may be partially located outside the valid padded area. For example, the initial reference block may include a first sub-block located within the valid padded area and a second sub-block located outside the valid padded area. The samples of the second sub-block located outside the valid padded area may then be padded using respective neighboring sample values ​​located within the valid padded area. The current block may be predicted based on the samples in the first sub-block and the padded samples in the second sub-block.

[0021] The extent of the valid padded area can be determined based on at least a position and / or a size of the surface area associated with the first projection geometry. It can be determined whether the initial reference block is located at least partially outside the valid padded area. When the initial reference block is located at least partially outside the valid padded area, the initial reference block can be excluded from the prediction of the current block.

[0022] A more detailed understanding of the embodiments disclosed herein may be had from the following description, given by way of example in conjunction with the accompanying drawings, in which: [Brief description of the drawings]

[0023] [Figure 1A] FIG. 1 illustrates an exemplary spherical sampling along the longitude and latitude directions in an equirectangular projection (ERP). [Figure 1B] FIG. 1 illustrates an exemplary 2D plane in ERP. [Figure 1C] FIG. 2 illustrates an example picture generated using ERP. [Figure 2A] FIG. 1 illustrates an example 3D geometric structure in cube-map projection (CMP). [Figure 2B] FIG. 1 illustrates an example 2D plane with 6 faces using 4×3 frame packing. [Figure 2C] FIG. 2 is a diagram of an example picture produced using cube map projection. [Figure 3A] FIG. 1 illustrates an exemplary sphere sampling in equal area projection (EAP). [Figure 3B] FIG. 2 illustrates an exemplary 2D plane in EAP. [Figure 3C] FIG. 2 illustrates an example picture generated using EAP. [Figure 4A] FIG. 1 illustrates an example 3D geometric structure in an octahedral projection. [Figure 4B] FIG. 13 is a diagram of an example 2D planar frame packing in an octahedral projection. [Figure 4C] FIG. 2 illustrates an example picture produced using an octahedral projection. [Diagram 5] FIG. 1 illustrates an exemplary 360-degree video processing. [Figure 6] FIG. 2 is a block diagram of an example block-based encoder. [Figure 7] FIG. 2 is a block diagram of an exemplary block-based decoder. [Figure 8] FIG. 2 is a diagram showing an example of using reference samples in HEVC intra prediction. [Figure 9] FIG. 2 illustrates an exemplary representation of intra-prediction direction in HEVC. [Figure 10] FIG. 1 illustrates an exemplary inter prediction using one motion vector. [Figure 11] A diagram showing padding for reference samples outside picture boundaries in HEVC / H.264. [Figure 12] FIG. 1 illustrates how spatial neighboring blocks can be used to derive spatial merging candidates in the HEVC merging process. [Figure 13]FIG. 1 illustrates discontinuities and geometry padding for a surface picture projected using a cube map. [Figure 14] FIG. 1A shows an example 3D representation of padding for a cube map; FIG. 1B shows an example flat representation of padding for a cube map with faces expanded around the front faces. [Figure 15] (A) is a diagram showing an example extrapolation of missing samples in corner areas using a nearest neighbor technique, and (B) is a diagram showing an example extrapolation of missing samples in corner areas using a diagonal technique on a surface boundary. [Figure 16] FIG. 1 illustrates an example motion compensated prediction for a cube map 4×3 frame packed picture. [Figure 17A] FIG. 13 is a diagram of an exemplary padding for a picture projected using equirectangular projection to show continuity on a sphere. [Figure 17B] FIG. 2 is another diagram of exemplary padding for a picture projected using equirectangular projection. [Figure 18] FIG. 13 illustrates how to derive a sine function using a lookup table. [Figure 19A] FIG. 13 is a diagram of a padded surface buffer for motion estimation. [Figure 19B] FIG. 13 illustrates motion estimation in a padded surface buffer. [Figure 20A] FIG. 13 illustrates padding in the horizontal direction when the source picture size is not a multiple of the smallest coding unit (CU) size. [Figure 20B] A diagram showing padding in the vertical direction when the source picture size is not a multiple of the minimum CU size. [Figure 20C] A diagram showing padding in the horizontal and vertical directions when the source picture size is not a multiple of the minimum CU size. [Figure 21]FIG. 2 illustrates neighboring blocks used when deblocking a current block. [Figure 22] FIG. 1 illustrates pixels that can be filtered by edge offset sample adaptive offset (SAO). [Figure 23A] FIG. 1 is a diagram of an exemplary decoded picture buffer (DPB) management performed. [Figure 23B] FIG. 2 is another diagram of exemplary DPB management performed. [Figure 24A] FIG. 1 illustrates uniform sampling of cube faces for a cube map. [Figure 24B] FIG. 1 is an example diagram of non-uniform spherical sampling for a cube map. [Figure 24C] FIG. 13 is an illustration of non-uniform sampling of cube faces for a unicube. [Figure 24D] FIG. 13 is an exemplary diagram of uniform spherical sampling for a unicube. [Diagram 25] FIG. 1A shows an exemplary mapping from a non-uniform partitioning grid of cube faces to a uniform partitioning grid of unicube faces; FIG. 1B shows an exemplary mapping from a uniform partitioning grid of unicube faces to a non-uniform partitioning grid of cube faces. [Figure 26A] FIG. 1 illustrates an exemplary non-uniform partitioning on a cube face. [Figure 26B] FIG. 2 illustrates an exemplary uniform partitioning on a sphere. [Figure 26C] FIG. 13 illustrates an example mapping between cube face coordinates β and unicube face coordinates β′. [Figure 27] FIG. 13 is an example diagram of the valid area of ​​padded samples for one unicube facet. [Figure 28] FIG. 13 is an exemplary diagram of clipping MV to the valid region of padded samples for the unicube format. [Figure 29] FIG. 13 is an exemplary diagram of hybrid padding for a unicube projection. [Diagram 30] FIG. 1 shows an example picture of hybrid padding for a unicube projection. [Diagram 31] FIG. 1 illustrates an exemplary comparison of cubemap, unicube, ACP, and EAC mapping functions. [Diagram 32] FIG. 13 is an example diagram using lookups to six faces for geometry padding for a unicube. [Figure 33A] FIG. 1 illustrates an example communication system in which one or more disclosed embodiments may be implemented. [Figure 33B] FIG. 33B is a system diagram of an example wireless transmit / receive unit (WTRU) that may be used within the communications system shown in FIG. [Figure 33C] A system diagram of an example radio access network and core network that can be used within the communication system shown in FIG. 33A. [Figure 33D] A system diagram of another example radio access network and core network that can be used within the communication system shown in FIG. 33A. [Figure 33E] A system diagram of another example radio access network and core network that can be used within the communication system shown in FIG. 33A. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0024] A detailed description of illustrative embodiments will now be given with reference to various figures. While this description provides detailed examples of possible implementations, it should be noted that the details are intended to be illustrative and in no way limit the scope of the present application.

[0025] A VR system can use one or more 360-degree videos. The 360-degree videos can be viewed from a 360-degree angle horizontally and from a 180-degree angle vertically. The VR system and the 360-degree videos can be used for media consumption beyond ultra-high definition (UHD) services. Freeview TV (FTV) can test the performance of the solution. For example, FTV can test the performance of a 360-degree video (or, e.g., omnidirectional video) based system and / or a multi-view based system.

[0026] The VR system may include a processing chain. The processing chain may include capturing, processing, displaying, and / or applying. With regard to capturing, the VR system may use one or more cameras to capture a scene from different and diverse views (e.g., 6 to 12 views). The views may be stitched together to form a 360-degree video (e.g., at a high resolution such as 4K or 8K). The client and / or user side of the VR system may include a computing platform, a head-mounted display (HMD), and / or a head tracking sensor. The computing platform may receive (e.g., a 360-degree video), decode, and / or generate a viewport for display. Two pictures (e.g., one for each eye) may be rendered for display. The two pictures may be displayed in the HMD for stereoscopic vision. Lenses may be used to magnify the image displayed in the HMD. For example, lenses may be used to magnify the image displayed in the HMD for better viewing. The head tracking sensor may track (e.g., track at all times) the orientation of the viewer's head. The orientation information can be provided to the VR system to display a viewport picture for that orientation. The VR system can provide a dedicated touch device. For example, the dedicated touch device can enable the viewer to interact with objects in the virtual world. The VR system can be driven by a workstation with graphical processing unit (GPU) support. The VR system can use a mobile device (e.g., a smartphone) as the computing platform, as the HMD display, and / or as the head tracking sensor. The spatial HMD resolution can be, for example, 2160×1200. The refresh rate can be, for example, 90 Hz. The field of view (FOV) can be, for example, 110 degrees. The sampling rate of the head tracking sensor can be, for example, 1000 Hz to capture fast movements.The VR system may include lenses and a cardboard and may be powered by a smartphone. The VR system may be used for gaming. One or more 360-degree video streaming services may be provided.

[0027] The VR system may be capable of providing interactivity and / or haptic feedback. Large HMDs may not be convenient for a person to wear. A resolution of 2160x1200 for stereoscopic viewing (e.g., as provided by some HMDs) may not be sufficient and may cause dizziness and discomfort to some users. An increase in resolution may be desirable. The VR experience may be enhanced by combining the visual effects of the VR system with real-world force feedback. A VR roller coaster application may be an example of combining the visual effects of a VR system with real-world force feedback.

[0028] The 360-degree video can be compressed and / or delivered using, for example, Dynamic Adaptive Streaming over HTTP (DASH) based video streaming techniques. The 360-degree video content can be represented using a spherical geometry structure. For example, multiple synchronized views can be captured by multiple cameras and stitched together on a sphere. The multiple synchronized views can be stitched together on a sphere as a monolithic structure. The spherical information can be projected onto a 2D plane through a geometric transformation process. For example, the spherical information can be projected onto a 2D plane through a geometric transformation process by using Equirectangular Projection (ERP). FIG. 1A shows an exemplary spherical sampling in longitude (φ) and latitude (θ). FIG. 1B shows an exemplary sphere projected onto a 2D plane using ERP. In aeronautics, a longitude φ in the range [-π,π] may be called yaw, a latitude θ in the range [-π / 2,π / 2] may be called pitch, π may be the ratio of a circle's circumference to its diameter, (x,y,z) may represent the coordinates of a point in 3D space, and (ue,ve) may represent the coordinates of a point in a 2D plane after the ERP. The ERP may be expressed mathematically, for example, as shown in equations (1) and / or (2): ue = (φ / 2 × π) + 0.5) × W (1) ve = (0.5 - θ / π) × H (2) W and H can be the width and height of the 2D plane picture. As shown in FIG. 1A, point P, which is the intersection between longitude L4 and latitude A1 on the sphere, can be mapped to a unique point q in the 2D plane in FIG. 1B using equations (1) and / or (2). Point q in the 2D plane shown in FIG. 1B can be projected back to point P on the sphere shown in FIG. 1A, for example, via back projection. The field of view (FOV) in FIG. 1B shows an example where the FOV on the sphere is mapped to the 2D plane with a viewing angle of about 110 degrees in the X-axis direction.

[0029] One or more 360-degree videos can be mapped to 2D videos. For example, 360-degree videos can be mapped to 2D videos using ERP. The 360-degree videos can be encoded and delivered to the client using a video codec such as H.264 or High Efficiency Video Coding (HEVC). At the client side, the videos can be decoded and rendered (e.g., in equirectangular format) based on the user's viewport, for example, by projecting and displaying the part belonging to the FOV in the equirectangular picture on the HMD. The characteristics of an equirectangular 2D picture can be different from a 2D picture (e.g., a rectilinear video picture). FIG. 1C illustrates an exemplary equirectangular picture. The top of the picture can correspond to the north pole and the bottom can correspond to the south pole. As shown in FIG. 1C, the top and / or bottom can be stretched. For example, the top and / or bottom can be stretched compared to the center of the picture, which corresponds to the equator. The stretching at the top and / or bottom can indicate that the equirectangular sampling in the 2D spatial domain may be non-uniform. The motion field in a 2D equirectangular picture may be complex. For example, the motion field in a 2D equirectangular picture in the time direction may be complex.

[0030] Some video codecs (e.g., MPEG-2, H.264, and / or HEVC) that use a translation model to describe the motion field may not be able to represent shape-changing motion in an equirectangular projected 2D planar picture. With ERP, areas closer to the poles may be less interesting to the viewer and / or content provider than areas closer to the equator, and the viewer may not focus on the top and bottom for a long duration. The top and bottom may be stretched (e.g., due to distortion effects) to become a large part of the 2D plane after ERP. Video compression of the stretched parts may require a large number of bits. Certain processing techniques may be applied to equirectangular picture coding. For example, preprocessing such as smoothing may be applied to the polar areas to reduce the bit cost of coding those areas.

[0031] 360-degree video can be represented using one or more different geometric projections, including, but not limited to, cube maps, equal area, cylinders, pyramids, and / or octahedra. A cube map can utilize six faces. A face in a cube map can be a planar square. FIG. 2A illustrates an exemplary cube map projection (CMP). A cube map can include six faces. For example, a cube map can include six square faces. Assuming that the radius of the illustrated inscribed sphere is 1, the width of a face (e.g., a square face) of a cube map can be 2. FIG. 2B illustrates an exemplary packing for placing six faces into a rectangular area that can be used for encoding and distribution. FIG. 2C illustrates an exemplary picture generated using a CMP. The gray portions 201, 203 without pictures can represent one or more padded regions to fill the rectangular picture. For faces in a CMP, the picture can look the same as a 2D picture. The boundaries of the faces may not be continuous. For example, a line that crosses two neighboring faces may be bent at the boundary between those two faces. The motion at the face boundary may be discontinuous.

[0032] FIG. 3A shows an exemplary geometric structure of an equal area projection (EAP). The vertical sampling on the sphere of the equal area projection may not be based on a uniformly spaced pitch. The projection of the sampled latitudes (e.g., each sampled latitude) onto the Y axis may be uniformly distributed so that the samples on the sphere occupy the same area. The sampling in the vertical direction may be sparse for regions close to the pole regions. As shown in FIG. 3A, more samples may be used near the equator. A user may view regions close to the equator more frequently than regions close to the poles. FIG. 3B shows an exemplary 2D plane in the EAP. FIG. 3C shows an exemplary picture using the EAP. As shown in FIG. 3C, the regions near the equator (e.g., 301) may be expanded, while the regions near the poles (e.g., 303) may be deflated.

[0033] 4A shows an example geometric structure of an octahedral projection. An octahedron can include eight equal side triangular faces. If the radius of the inscribed sphere is 1, then the side of each triangle is

[0034]

number

[0035] 4B shows an example packing for placing eight triangles in a rectangular area. FIG. 4C shows an example picture using an octahedral projection. At the corners (e.g., 401) of the shared boundary of two adjacent triangles, a warping distortion can be observed.

[0036] The coding efficiency of different geometric projections can be compared. For example, latitude-based peak signal-to-noise ratio (L-PSNR) can be used for comparison. One or more (e.g., two) factors can be considered, i.e., uniform sampling on a sphere and / or the viewing behavior of the viewer. A specific number of samples that are uniformly distributed on the sphere can be defined. Sample weights can be defined based on latitude. The distortion can be measured using a weighted mean square error (MSE) by considering uniformly distributed samples. The weights can be derived by tracking the view angle of the viewer. For example, the weights can be derived by tracking the view angle of the viewer using a training sequence.

[0037] A sample can be given more weight if it is viewed more frequently. For example, samples around the equator can be given more weight than samples closer to the poles, since interesting content may be located around the equator. By using samples that are uniformly distributed on a sphere, the performance of different projections can be compared. These spherical samples can be predefined and may not be projected to integer sampling positions when different projections are applied. If an interpolation filter-based resampling is applied, additional interpolation errors may be introduced. If nearest neighbor sampling is applied, uniform sampling may not be guaranteed.

[0038] With one or more 360-degree cameras and stitching, an equirectangular format can be supported. During encoding of 360-degree video with cube-map geometry, the equirectangular format can be converted to a cube-map format. There can be a relationship between the equirectangular and the cube-map. In FIG. 2A, the faces can be referenced with each of the three axes extending from the center of the sphere to the center of the face. As shown in FIG. 2A, P can represent positive and N can represent negative. PX can represent the direction along the positive x-axis from the center of the sphere, and NX can represent the opposite direction of PX. A similar notation can be used for PY, NY, PZ, and / or NZ. The six faces (e.g., PX, NX, PY, NY, PZ, and / or NZ) can correspond to the front, back, top, bottom, left, and / or right faces, respectively. The faces can be indexed from 0 to 5. For example, the surface segments can be indexed as follows: PX(0), NX(1), PY(2), NY(3), PZ(4), and / or NZ(5). Ps(X_s, Y_s, Z_s) can be a point on a sphere of radius 1. The point Ps can be expressed in terms of yaw φ and pitch θ as follows: X_s=cos(θ)cos(φ) (3) Y_s=sin(θ) (4) Z_s=-cos(θ)sin(φ) (5)

[0039] Point Pf can be a point on the cube when a line is extended from the center of the sphere to Ps. Pf can be on the face segment NZ. The coordinates (X_f, Y_f, Z_f) of Pf can be calculated as follows: X_f=X_s / |Z_s| (6) Y_f=Y_s / |Z_s| (7) Z_f=-1 (8) Here, |x| can be the absolute value of the variable x. The coordinates (uc, vc) of Pf in the 2D plane of the surface segment NZ can be calculated as follows: uc=W×(1-X_f) / 2 (9) vc = H × (1 - Y_f) / 2 (10)

[0040] Using one or more of equations (3) through (10), the relationship between the coordinates (uc, vc) in the cube map on a particular surface and the point (φ, θ) on the sphere can be derived. The relationship between the equirectangular point (ue, ve) and the coordinates (φ, θ) on the sphere can be known from equations (1) and / or (2). There can be a relationship between the equirectangular geometry and the cube map geometry.

[0041] The geometry mapping from the cubemap to the equirectangular can be expressed as follows: Given a point (uc,vc) on one surface on the cubemap, the output (ue,ve) on the equirectangular plane can be calculated as one or more of the following: The coordinates of a 3D point P_f on the surface using (uc,vc) can be calculated according to the relationships in equations (9) and / or (10). The coordinates of a 3D point P_s on the sphere using P_f can be calculated according to the relationships in equations (6), (7), and / or (8). The coordinates of (φ,θ) on the sphere using P_s can be calculated according to the relationships in equations (3), (4), and / or (5). The coordinates of a point (ue,ve) on the equirectangular picture from (φ,θ) can be calculated according to the relationships in equations (1) and / or (2).

[0042] To use a cube map to represent 360-degree video in a 2D picture, six faces of the cube map can be packed into a rectangular area. This is sometimes called frame packing. A frame-packed picture can be treated (e.g., encoded) as a 2D picture. Different frame packing configurations (e.g., 3×2 and / or 4×3) can be used. In a 3×2 configuration, six cube map faces can be packed into two rows with three faces in one row. In a 4×3 configuration, four faces (e.g., PX, NZ, NX, and PZ) can be packed into one row (e.g., the center row) and the other two faces (e.g., PY and NY) can be packed separately into two different rows (e.g., the top row and the bottom row). Figure 2C shows an example of 4×3 frame packing, which corresponds to the equirectangular picture in Figure 1C.

[0043] A 360-degree video in equirectangular format can be converted to a cubemap format. For example, a 360-degree video in equirectangular format can be converted to a cubemap format using the 360-degree video in equirectangular format as input. For a sample point (uc, vc) in the cubemap format, a corresponding coordinate (ue, ve) in the equirectangular format can be calculated. If the calculated coordinate (ue, ve) in the equirectangular is not at an integer sample position, an interpolation filter can be used. For example, an interpolation filter can be used to obtain a sample value at this fractional position using samples at neighboring integer positions.

[0044] The cube map format can be a representation format for 360-degree video. Samples on the sphere can be non-uniformly sampled by the cube map format. For example, samples on the sphere can be non-uniformly sampled by the cube map format with a higher sampling rate near the surface boundary and / or a lower sampling rate near the surface center. Samples on the sphere may be non-uniformly sampled by the cube map format due to limitations of rectilinear projection. There can be certain spherical sampling characteristics in the cube map format. For example, there can be a spherical sampling characteristic in the cube map format with a higher sampling rate at the surface boundary and / or a lower sampling rate at the surface center. The cube map projection can expand areas near the surface boundary and / or shrink areas near the surface center.

[0045] The motion model may include a translational motion model and / or an affine motion model, etc. Cube map projection may reduce the regularity of the resulting motion field in the projection plane.

[0046] A unicubemap projection format can be provided. The cubemap sampling grid can be transformed into a uniform sampling grid on a sphere. Mapping can be used to change the coordinates of the samples on the 2D plane before the actual cube faces are generated.

[0047] 24A-24D show examples of planar and spherical sampling patterns between a cube map and a unicube. For example, FIG. 24A shows an example uniform sampling of cube faces for a cube map. As shown in FIG. 24A, the sampling grid of a cube map face can include two sets of parallel lines, one horizontal and the other vertical. The sets of parallel partitioning lines can be uniformly spaced apart. When a cube map face is projected onto a sphere, the sampling grid can be distorted. For example, as shown in FIG. 24B, the lines in a planar face can become curved. FIG. 24B shows an example non-uniform spherical sampling for a cube map. The sampling grid on a sphere can be non-uniform. For example, as shown in FIG. 24B, the rectilinear projection can not be a distance-preserving projection, so the sampling grid on a sphere can be non-uniform. To maintain a similar sampling structure to a cube map, faces in the unicube format can be sampled based on one or more (e.g., two) sets of parallel lines. The parallel lines in the set can be non-uniformly distributed, such that the corresponding sampling grid on the sphere can be uniform. Figure 24C shows an exemplary non-uniform sampling of cube faces for a unicube. For example, as shown in Figure 24C, the parallel lines in the set can be non-uniformly distributed. Figure 24D shows an exemplary uniform sphere sampling for a unicube. As shown in Figure 24D, the corresponding sampling grid on the sphere can be uniform.

[0048] Non-uniform partitioning on the cube faces, as shown in the example of FIG. 24C, can be associated with varying distances between neighboring samples. A mapping can be used to convert a non-uniform planar sampling grid to a uniform planar sampling grid. FIG. 25 shows an example mapping between cube faces and unicube faces. For example, FIG. 25A shows an example mapping from a non-uniform partitioning grid of cube faces to a uniform partitioning grid of unicube faces. FIG. 25B shows an example mapping from a uniform partitioning grid of unicube faces to a non-uniform partitioning grid of cube faces. If the horizontal and vertical transformations are separable, the mapping from (x,y) to (x',y') can include one or more (e.g., two) separate transformations, e.g., x'=f(x), y'=f(y), and the transformations can be applied to x and y. For example, the transformations can be applied independently to x and y. An inverse transformation can be calculated that can map (x',y') to (x,y), e.g., x=g(x'), y=g(y'). The two transformations of x and y can be identical. If the two transformation functions of x and y are identical, the derivation of the transformation function of y can be performed. For example, if β∈[-1,1] is the y coordinate of the pattern area on the cube, FIG. 26 shows an example of calculating the mapping function between the coordinates of the cube facets and the coordinates of the unicube facets. FIG. 26A shows an example mapping function of a unicube using one non-uniform partitioning on the cube facets. FIG. 26B shows an example mapping function of a unicube using a corresponding uniform partitioning on the sphere. FIG. 26C shows the mapping function between the coordinates of the cube facets β and the coordinates of the unicube facets β'. The mapping function β'=f(β) can be aimed at transforming β to β' using an equal rectilinear structure partitioning on a sphere (e.g., as shown in FIG. 24B), where f(β) can be proportional to the area of ​​the sphere region corresponding to β. For example, as shown in FIG. 26B, the value of f(β) can be equal to the ratio between the area of ​​the pattern sphere region and a quarter of the sphere.For example, the mapping function f(β) is:

[0049]

number

[0050]

number

[0051] where β′∈[−1,1]. FIG. 26C shows the corresponding mapping relationship between β and β′.

[0052] A cubemap-like projection format can provide one or more spherical sampling characteristics by adjusting the cubemap coordinates. For example, the cubemap coordinates can be adjusted by using one or more mapping functions. An adjusted cubemap projection (ACP) can provide spherical sampling uniformity. Coordinates in the cube domain can be adjusted based on one or more of the following mapping functions: β'=f(β)=sgn(β)(-0.36·β 2 +1.36·|β|) (13)

[0053]

number

[0054] Here, sgn(·) can return the sign of the input.

[0055] The conformal cube map (EAC) projection can be performed by transforming coordinates between the cube domain and the EAC domain. For example, the coordinates between the cube domain and the EAC domain can be transformed based on the tangent of the angle of the sphere sample on the cube map sampling grid. The mapping function for the EAC projection is

[0056]

number

[0057]

number

[0058] It can be calculated as follows:

[0059] FIG. 31 shows an exemplary comparison of mapping functions g(β′) for cubemap, unicube, ACP, and EAC.

[0060] FIG. 5 illustrates an exemplary workflow for 360-degree video processing. 360-degree video capture can be captured using one or more cameras. For example, one or more cameras can be used to capture 360-degree video covering a spherical space. The videos can be stitched together. For example, the videos can be stitched together using an equirectangular geometry structure. The equirectangular geometry structure can be converted to other geometry structures, such as cube-mapped geometry, for encoding (e.g., encoding with a video codec). The encoded video can be distributed to a client, for example, via dynamic streaming and / or broadcasting. The video can be decoded, for example, at a receiver. The decompressed frames can be unpacked into a display, such as a display geometry. For example, the display geometry can be an equirectangular geometry. The geometry can be used for rendering. For example, the geometry can be used for rendering via viewport projection according to the user's viewing angle.

[0061] The chroma components can be subsampled to a smaller resolution than that of the luma component. Chroma subsampling can reduce the amount of video data used for encoding, can save bandwidth and / or computing power, and can do so without affecting (e.g., significantly affecting) the video quality. When using a 4:2:0 chroma format, both chroma components can be subsampled to 1 / 4 of the luma resolution. For example, the chroma components can be subsampled to 1 / 2 of the luma resolution in the horizontal direction and 1 / 2 in the vertical direction. After chroma subsampling, the chroma sampling grid can be different from the luma sampling grid. In FIG. 5, throughout the processing flow, the 360-degree video processed in one or more stages can be in a chroma format with the chroma components subsampled.

[0062] FIG. 6 illustrates a block diagram of an exemplary block-based hybrid video encoding system 600. An input video signal 602 may be processed block by block. To compress high resolution (e.g., 1080p and / or higher) video signals, an extended block size (e.g., called a coding unit or CU) may be used (e.g., in HEVC). A CU may have up to 64×64 pixels (e.g., in HEVC). A CU may be partitioned into prediction units or PUs, on which separate predictions may be applied. Spatial prediction 660 or temporal prediction 662 may be performed on input video blocks (e.g., macroblocks (MBs) or CUs). Spatial prediction (or, e.g., intra prediction) may predict a current video block using pixels belonging to already coded neighboring blocks in the same video picture and / or slice. Spatial prediction may reduce spatial redundancy inherent in a video signal. Temporal prediction (e.g., called inter prediction or motion compensated prediction) may predict a current video block using pixels belonging to already coded video pictures. Temporal prediction can reduce temporal redundancy inherent in a video signal. A temporal prediction signal for a given video block can be conveyed by a motion vector, which indicates the amount and / or direction of motion between a current block and its reference block. If multiple reference pictures are supported (e.g., in H.264 / AVC or HEVC), a reference picture index for the video block can be conveyed to the decoder. The reference index can be used to identify which reference picture in the reference picture store 664 the temporal prediction signal can originate from.

[0063] After spatial and / or temporal prediction, a mode decision 680 in the encoder may select a prediction mode based, for example, on rate-distortion optimization. At 616, the prediction block may be subtracted from the current video block. The prediction residual may be decorrelated using the transform module 604 and the quantization module 606 to achieve a target bit rate. The quantized residual coefficients may be inverse quantized at 610 and inverse transformed at 612 to form a reconstructed residual. At 626, the reconstructed residual may be added back to the prediction block to form a reconstructed video block. At 666, an in-loop filter, such as a deblocking filter and / or an adaptive loop filter, may be applied to the reconstructed video block, which is then placed into a reference picture store 664. The reference pictures in the reference picture store 664 may be used to encode future video blocks. An output video bitstream 620 may be formed. The coding mode (e.g., inter or intra), prediction mode information, motion information, and / or quantized residual coefficients may be sent to an entropy coding unit 608 for compression and packing to form a bitstream 620.

[0064] FIG. 7 shows a general block diagram of an exemplary block-based video decoder. In the entropy decoding unit 208, the video bitstream 202 may be received, unpacked, and / or entropy decoded. Coding mode and / or prediction information may be sent to the spatial prediction unit 260 (e.g., if intra-coded) and / or to the temporal prediction unit 262 (e.g., if inter-coded). In the spatial prediction unit 260 and / or the temporal prediction unit 262, a prediction block may be formed. Residual transform coefficients may be sent to the inverse quantization unit 210 and the inverse transform unit 212 to reconstruct a residual block. In 226, the prediction block and the residual block may be added. The reconstructed block may pass through in-loop filtering 266 and may be stored in a reference picture store 264. The reconstructed video in the reference picture store 264 may be used to drive a display device and / or to predict future video blocks.

[0065] To encode 2D planar rectilinear video, video codecs such as H.264 and / or HEVC can be used. Video encoding can utilize spatial and / or temporal correlation to remove information redundancy. During video encoding, one or more prediction techniques can be applied, such as intra prediction and / or inter prediction. Intra prediction can predict sample values ​​using neighboring reconstructed samples. Figure 8 shows exemplary reference samples that can be used to intra predict a current transform unit (TU). The reference samples can include reconstructed samples located above and / or to the left of the current TU.

[0066] There may be multiple intra prediction modes to choose from. Figure 9 shows an example display of intra prediction directions in HEVC. For example, HEVC may specify 35 intra prediction modes, including planar (0), DC (1), and / or angular prediction (2-34), as shown in Figure 9. An appropriate intra prediction mode may be selected. For example, at the encoder side, an appropriate intra prediction mode may be selected. Predictions generated by multiple candidate intra prediction modes may be compared. A candidate intra prediction mode that produces the least distortion between predicted samples and original samples may be selected. The selected intra prediction mode may be coded and included in the bitstream.

[0067] Angular prediction can be used to predict directional texture. FIG. 10 shows an example inter prediction using one motion vector (MV). Blocks B0′ and B1′ in a reference picture can be reference blocks for blocks B0 and B1 of a current picture, respectively. Reference block B0′ can be partially outside the boundary of the reference picture. A padding process can be used to fill unknown samples outside the picture boundary. FIG. 11 shows an example padding for reference samples outside the picture boundary. For example, a padding example for block B0′ can have four portions P0, P1, P2, and P3. Portions P0, P1, and P2 can be outside the picture boundary and can be filled, for example, via a padding process. For example, portion P0 can be filled with the top-left sample of the reference picture. Portion P1 can be filled with vertical padding using the top row of the reference picture. Portion P2 can be filled with horizontal padding using the left-most column of the picture.

[0068] For inter-coding, motion vector prediction and / or merge modes can be used.

[0069] In motion vector prediction, motion vectors from neighboring PUs and / or temporally co-located PUs can be used as predictors of the current MV. An index of the selected MV predictor can be coded and / or communicated to the decoder. The decoder can construct an MV predictor list with multiple entries. An entry with the communicated index can be used to predict the MV of the current PU.

[0070] When using the merge mode, MV information for spatial and / or temporal neighboring PUs can be reused and motion vector coding for the current PU can be skipped. FIG. 12 shows exemplary spatial neighboring blocks that can be used for merge candidate derivation. For example, bottom left (A0), left (A1), top right (B0), top (B1), and / or top left (B2) can be used for merge candidate derivation. Temporally co-located blocks can be used as merge candidates. An index of the selected merge candidate can be coded. At the decoder side, a merge candidate list can be constructed. For example, the merge candidate list constructed at the decoder side can be similar to the construction of the MV predictor list at the encoder side. An entry with the transferred merge candidate index can be used as the MV of the current PU.

[0071] 360-degree video coding and 2D planar video coding can differ in various aspects. For example, 360-degree video can be frame packed and can include 360-degree information of the environment surrounding the viewer. Therefore, 360-degree video can have one or more inherent circular characteristics that 2D video may not have. Some codecs, such as MPEG-2, H.264, and / or HEVC, which can be suitable for encoding 2D video, may not fully take into account the circular characteristic features of 360-degree video. For example, if a pixel in a 360-degree video is outside the boundary of the picture, repetitive padding can be applied.

[0072] Frame packing can be used to place one or more surfaces of a 360-degree video in a rectangular picture for encoding. Sample values ​​along surface boundaries may not be continuous. The discontinuity may reduce the efficiency of spatial prediction. As shown in FIG. 8, neighboring reconstructed samples can be used to derive reference samples for intra-predicting the current TU. The neighboring reconstructed samples can be located in a surface different from the surface to which the current TU belongs. For example, if the current TU is close to or at the boundary of a surface in a cube map, the nearest neighboring sample in the corresponding frame-packed 2D picture can be in the neighboring surface, and the samples may not be continuous in the frame-packed picture.

[0073] FIG. 13 shows an example of discontinuous samples and padding in a cube map surface picture. The circle can represent the equator of the sphere. The four sides / edges of the square can represent the four lines on the four edge surface of the cube map onto which the equator is projected. P can represent a surface of the cube map, and Q can represent a neighboring surface of surface P. Px(x∈[0,3]) can represent a sampling point on surface P. Qx(x∈[0,3]) can represent a sampling point on neighboring surface Q. For example, if neighboring samples P4, P5, and P6 are samples outside the surface boundary, and if P4, P5, and P6 should be padded on surface P (e.g., using a uniform sampling interval), padding samples K0, K1, and K2 can represent the projections of P4, P5, and P6 from the center of the sphere onto surface Q, respectively. The point locations of P4, P5, and P6 may be referred to as neighboring sample locations, or initial sample locations associated with the initial reference block. As shown in FIG. 13, K0 and Q0 may not overlap with each other. K1 and Q1, and K2 and Q2 may not overlap with each other. The further the extended sample is to the surface boundary, the greater the mismatch may be. The padding samples K0, K1, and K2 may be used to fill the samples in P4, P5, and P6, respectively. K0, K1, and K2 may be referred to as reference samples, derived sample locations, padding sample locations, or sample values ​​associated with the mapped reference block located at the respective reference sample locations. As shown in FIG. 13, K0 and Q0 may not overlap with each other, and the same may be true for K1 and Q1, and K2 and Q2. The padding samples K0, K1, and / or K2 may be used to fill the samples in P4, P5, and / or P6.

[0074] Inter motion compensation prediction may have discontinuous samples. For example, when cube map 4×3 frame packing (e.g., as shown in FIG. 2B) is applied during encoding, pixels of a reference block outside a plane of a current block may be discontinuous. FIG. 16 shows an example motion compensation prediction for a cube map 4×3 frame packed picture. As shown in FIG. 16, plane V may be a virtual plane for frame packing. If a current block B0 belongs to plane 0 of a current picture and a temporal reference block B0′ pointed to by a motion vector is partially located outside plane 0 of a reference picture, a sub-block 1402 on plane 4 (e.g., a sub-block of block B0′) may have a different texture direction. The current block B0 may be divided into two prediction units.

[0075] Motion along a plane boundary in a frame-packed picture may not be continuous. For example, the current PU in FIG. 12 may be at a plane boundary. Neighboring blocks A0, A1, B0, B1, and / or B2 may be located in one or more different planes. The motion of the neighboring blocks (e.g., A0, A1, B0, B1, and / or B2) may have one or more directions different from the motion direction of the current block.

[0076] 360-degree video coding may take into account the geometric structure and / or characteristics of the video being coded. For intra and / or inter prediction, when a reference sample is outside the frame-packed picture, geometry padding may be applied by taking into account one or more circular characteristics of the picture. The geometry padding may increase prediction efficiency for reference samples outside the picture. For example, the geometry padding may increase prediction efficiency for reference samples outside the picture in the case of inter motion compensation prediction. For example, the value of a point p in sub-blocks P0, P1, and / or P2 as shown in FIG. 11 may be derived via geometry mapping and / or interpolation as described herein.

[0077] For intra prediction and / or inter prediction, if a reference sample in a frame packed picture is inside the picture but outside the plane to which the current block belongs, the reference sample value can be derived using geometry padding. For example, geometry padding can be used to handle discontinuities. A reference sample location, which may be referred to as a neighboring sample location and / or one or more initial sample locations associated with an initial reference block, which may be outside the plane of the current block, can be mapped from its current plane to other planes, for example, by using geometry mapping. For example, the reference sample location P4 in FIG. 13 can be mapped to K0, and the sample value at location K0 can be derived from the sample values ​​on the neighboring planes (e.g., Q0 and / or Q1, etc.). For example, the sub-block 1402 in FIG. 16, which can be used for inter prediction, can be derived via geometry padding. For example, in FIG. 16, the sample value on plane 4 may not be used (e.g., directly) for inter prediction.

[0078] In motion vector prediction and / or inter-merge modes, spatial neighboring blocks, or temporally co-located blocks for temporal motion vector prediction, may be outside the surface of the current block. For example, the temporally co-located blocks for temporal motion vector prediction may be defined by HEVC. The neighboring block positions may be derived via a geometry mapping process. For example, sample positions may be mapped. Sample values ​​may be derived based on the mapped positions.

[0079] 360-degree videos can be encoded using hybrid projection techniques. For example, the hybrid projection techniques can be based on the characteristics of the video and / or the projection technique. For example, intra-pictures can be encoded using one projection, while inter-pictures can be encoded with another projection. For example, intra-pictures can be encoded using ERP or equal-area projection. Inter-pictures can be encoded with cube-map projection.

[0080] Geometry padding can be applied in 360-degree video coding. Figures 17A and 17B show exemplary geometry padding for equirectangular pictures. As shown in Figures 17A and 17B, the right and left boundaries can be spherically continuous. For example, the arrows shown in Figure 17A can indicate continuity on the sphere. For example, arrows with the same reference numbers, such as 02 and 02', 32 and 32', 42 and 42', 04 and 04', 34 and 34', and 44 and 44' in Figure 17A can indicate continuity of the right and left boundaries on the sphere. The top boundary can represent the north pole of the sphere, and the bottom boundary can represent the south pole of the sphere. As described herein, the top and bottom boundaries can be spherically disconnected and can be furthest apart on the sphere. The upper boundaries 20 and 20', 22 and 22', 26 and 26', and / or 28 and 28' may indicate spherical continuity at the North Pole. The lower boundaries 06 and 06', 08 and 08', 10 and 10', and / or 12 and 12' may indicate spherical continuity at the South Pole.

[0081] Using the sphere properties as described herein and shown in Figures 17A and 17B, if the neighboring sample locations are beyond the right boundary of the picture (e.g., 02, 32, 42, D, E, and / or F), padding can be performed using padding sample locations by rewinding to the left boundary of the picture (e.g., 02', 32', 42', D', E', and / or F', respectively). Padding can be performed by rewinding based on one or more of the circle properties of the picture. If the neighboring sample locations are inside the right boundary of the picture, a repetition from the rightmost line inside the picture boundary can be performed for padding. If a neighboring sample location falls beyond the bottom boundary of the picture (e.g., 06, 08, 10, 12, K, L, M, and / or N), the padding may follow the line of longitude and proceed to the opposite hemisphere (e.g., 06', 08', 10', 12', K', L', M', and / or N', respectively). The arrows and corresponding alphabetical labels in FIG. 17B may indicate a correspondence between the padded samples and padding samples that may be used to pad the neighboring sample locations. For example, the padding filled at arrow A may be obtained along arrow A'. The padding filled at arrow B may be obtained along arrow B', and so on.

[0082] An initial reference sample point (u,v) may be provided based on a current location in the picture. For example, the point (u,v) may be outside the equirectangular picture. The initial reference sample location (u,v) may be used to derive a padding sample location (up,vp). The point (up,vp) may be referred to as a padding sample, or one or more sample values ​​associated with a mapped reference block. The derived reference sample location (up,vp) for the equirectangular picture may be calculated as follows: When u < 0, or u ≥ W, and 0 ≤ v < H, up = u % W, vp = v (17) When v < 0, vp = -v - 1, up = (u + W / 2) % W (18) When v ≥ H, vp = 2 × H - 1 - v, up = (u + W / 2) % W (19) Here, W and H can be the width and height of the equirectangular picture. For example, the padding sample location (up, vp) can be calculated based on the location of the point (u, v). For example, when the point (u, v) crosses the right or left boundary, the padding sample (up, vp) can be calculated based on Equation (17). The padding samples (up, vp) of the point (u, v) that crosses the right or left boundary can respectively correspond to points located near the left or right boundary of the equirectangular picture. Other padding samples can be calculated based on the location of the point (u, v) corresponding to Equation (18) and / or Equation (19).

[0083] As shown in FIG. 13, padding for CMP can be provided. The padding can be applied to the point (u, v). The point p(u, v) can be positioned within the video content in the geometry structure. The point p(u, v) can be a neighboring sample of the current sample. The sample value can be derived using one or more of the following.

[0084] The point p(u,v) can be identified based on the current sample at the current sample location. For example, the current sample at the current sample location can be at the current surface location. The current sample can be associated with a padding sample. The point p(u,v) can be located outside the current surface boundary. The point p(u,v) can be located within a different surface. Depending on the surface to which the point p(u,v) belongs, the 3D position Pf(X,Y,Z) of the point can be calculated. For example, if the surface of the point p is NZ, the 3D position (X,Y,Z) of the point p can be calculated using equation (6), equation (7), and / or equation (8). The new surface to which the sample used for padding belongs can be determined based on the 3D position (X,Y,Z). The sample can be used for padding using (X,Y,Z) as follows: If |X|≧|Y|, and |X|≧|Z|, and X>0, the new surface segment can be PX. If |X|≧|Y|, and |X|≧|Z|, and X<0, the new surface can be NX. If |Y|≧|X|, |Y|≧|Z|, and Y>0), the new surface segment can be PY. If |Y|≧|X|, and |Y|≧|Z|, and Y<0), the new surface segment can be NY. If |Z|≧|X|, |Z|≧|Y|, and Z>0), the new surface can be PZ. If |Z|≧|X|, |Z|≧|Y|, and Z<0), the new surface can be NZ.

[0085] The 2D position (u',v') of point p can be calculated using Ps(X',Y',Z'). For example, the 2D position (u',v') of point p can be calculated based on the location of the new surface. If the new surface is located in NZ, the padding sample location (u',v') can be calculated using (X',Y') using equation (9) and / or equation (10). The sample value at (u',v') can be determined using an interpolation filter and / or based on integer position samples on the new surface to which (u',v') belongs.

[0086] Padding for a point (u,v) in the geometry structure may include one or more of the following: A current sample at a current sample location associated with a padding sample may be determined. A 3D position Pf(X,Y,Z) on the surface to be padded may be determined based on the point (u,v). The point (u,v) may represent a point located outside a picture boundary or surface. The padding sample location may be determined based on the location of the sample to be padded. For example, a 2D plane position (u',v') may be identified by applying a geometry projection of the 3D position of the padding sample to the geometry structure. A padding sample location, a padding sample value at the 2D plane position (u',v'), may be derived. The sample value at the 2D plane position (u',v') may be determined using an interpolation filter and / or based on samples at integer positions. A mapped sample position and / or a padding sample value may be derived.

[0087] One or more interpolation filters may be used. For example, one or more interpolation filters may be used when determining sample values ​​at (u',v') on the new surface. For example, bilinear filters, bicubic filters, Lanczos filters, spline interpolation filters, and / or discrete cosine transform-based interpolation filters (DCT-IF) may be used. Nearest neighbor interpolation may be used. As shown in FIG. 13, if nearest neighbor interpolation is used, the value of Q0 may be used to pad P4 and / or P5. For example, Q0 may be the nearest neighbor to K0 and K1, so the value of Q0 may be used to pad P4 and / or P5. The value of Q1 may be used to pad P6. For example, Q1 may be the nearest neighbor to K2, so the value of Q1 may be used to pad P6. In FIG. 13, the approximation may be coarse. For example, the sampling may be sparse, so the approximation in FIG. 13 may be coarse. It can provide an approximation for denser sampling, taking into account that cube map geometry may oversample spheres near the cube boundary.

[0088] Nearest neighbor interpolation can derive the mapped sample position. For example, the mapped sample position can include one or more of the following: Based on the point (u,v), a 3D position Pf(X,Y,Z) can be calculated on the padded surface. By projecting Pf(X,Y,Z) onto the geometric structure, a position Ps(X',Y',Z') can be calculated. On the new surface, a 2D planar position (u',v') can be calculated. For example, nearest neighbor interpolation can derive the mapped sample position, which may include a nonlinear function. Padding can be used. For example, facet-based padding can be used. With facet-based padding, one or more neighboring faces can be used to fill the surrounding padded area, which may correspond to the unfolding of the cube. For example, in the case of facet-based padding, four neighboring faces can be used (e.g., directly used) to fill the surrounding padded area. FIG. 14 shows an example of a padding process for a cube map. FIG. 14A shows an example 3D representation of the padding process for a cube map. FIG. 14B shows an example flat representation of the padding process for a cube map with faces unfolded around the front face. As shown in FIG. 14, the padded area surrounded by the dashed line can be filled with samples from neighboring faces. For example, the padded area surrounded by the dashed line shown in FIG. 14 can be filled directly with samples from neighboring faces. As shown in FIG. 13, if face-based padding is used, Q0, Q1, and / or Q2 can be used to pad P4, P5, and / or P6, respectively. Q2 may belong to the opposite hemisphere with respect to the hemisphere defined by the face containing the P sample, and / or may not make sense to pad the face containing the P sample. Face-based padding may not make sense for padding with a size larger than half the face size on the face.If the size of the padded area is small (e.g., a few samples wide), the projection angle can be close to the 45 degree angle that connects the cube boundary to its center, and / or facet-based padding can provide a sufficient approximation. As shown in FIG. 14B, the four corners of the padded area can be extrapolated. The four corners of the padded area shown in FIG. 14B can have samples that are not directly available from neighboring faces. Repetitive padding can be used in one or more corners. For example, repetitive padding can be used in the four corners. One or more of the following extrapolations can be used: mirroring the available samples, duplicating the nearest available samples (e.g., perpendicular to the facet boundary, as shown in FIG. 15A), and / or duplicating the samples diagonally to the facet boundary, as shown in FIG. 15B. For example, the sample values ​​can be derived as follows: If |x| ≥ |y|, then v(x,y) = v(sign(x)(|x| + |y|),0). If |x|<|y|, then v(x,y)=v(0,sign(y)(|x|+|y|)

[0089] For an extrapolated sample whose location is equidistant from two existing samples, the average of the two existing samples can be used. For example, as shown in Figures 15A and 15B, along a diagonal (e.g., shown as a dashed line), the extrapolated sample can be equidistant from two existing samples, and a weighted sum based on the distance to the available samples can be used. A sample value can be derived. For example, according to the exemplary principles shown in Figure 15A, a sample value can be derived as follows: If |x|≧|y|, v(x,y)=a×v(x,0)+b×v(0,x), a=(2×|x|-|y|) / (2×|x|), b=|y| / (2×|x|). If |x|<|y|, v(x,y)=a×v(y,0)+b×v(0,y), a=|x| / (2×|y|), b=(2×|y|-|x|) / (2×|y|). And / or, in accordance with the exemplary principles illustrated in FIG. 15B, the sample values ​​can be derived as follows: v(x,y)=a×v(sign(x)(|x|+|y|),0)+b×v(0,sign(y)(|x|+|y|), a=|x| / (|x|+|y|), b=|y| / (|x|+|y|)

[0090] The process (e.g., a similar process) can be used for one or more projected geometries that include one or more facets. For example, projected geometries that are composed of one or more facets can include, but are not limited to, an octahedron and / or an icosahedron.

[0091] Geometry padding may be applied in one or more of the following cases. For example, in intra prediction, geometry padding may be performed. For example, geometry padding may be performed when a neighboring sample is outside of a picture boundary. For example, geometry padding may be performed when a neighboring sample is not in the same plane as the current sample to be predicted. For example, geometry padding may be performed in inter prediction. For example, geometry padding may be performed when a reference sample is outside of a reference picture boundary. For example, geometry padding may be performed when a reference sample is not in the same plane as the current sample to be predicted. The derivation of the mapped sample position may include one or more of the following: A 3D position Pf(X,Y,Z) on the padded plane may be calculated based on the point (u,v). A position Ps(X',Y',Z') may be calculated by projecting Pf(X,Y,Z) onto the geometry structure. A 2D plane position (u',v') on the new plane may be calculated. The derivation of the mapped sample position can be used to derive spatial neighboring blocks that may be outside the picture boundary. The derivation of the mapped sample position can be used to derive spatial neighboring blocks that may not be in the same plane as the predicted block. The attributes of the mapped block position can be used as a predictor for a block (e.g., a current block). For example, the attributes of the mapped block position can include one or more of the following: intra mode, inter mode, merge mode, and / or motion information (which may include motion vectors and / or reference picture indexes), etc. The mapped block position used as a predictor for a block can include one or more of the following, derived using techniques such as computing the 3D position Pf(X,Y,Z), computing Ps(X',Y',Z'), and / or computing the 2D position (u',v') of p(u,v).For example, for motion vector prediction and / or motion vector derivation in inter-merge mode, the motion vector of the mapping block position derived as described herein can be used as a motion vector predictor and / or as a motion vector of the merge candidate. The position of the neighboring block can be derived and / or the attribute from the mapped block can be inferred. The position of the neighboring block can be calculated, for example, as the location p(u,v) closest to the current block, considering which one of the neighboring blocks (e.g., left, top, etc.). For example, if (uc,vc) is the location of the top left corner of the current block, then (uc-1,vc) can be used to calculate the location of the left neighboring block, (uc,vc-1) can be used to calculate the location of the top neighboring block, etc. The position of the neighboring block can be calculated as the center of the neighboring block. For example, if (uc,vc) is the location of the top left corner of the current block, then the location of the left neighboring block can be calculated as the center location of the block containing the position (uc-1,vc), the location of the top neighboring block can be calculated as the center location of the block containing the position (uc,vc-1), etc. For example, the positions of neighboring blocks can be determined by unfolding the 3D geometry, which can determine the mapping block positions on the neighboring surfaces.

[0092] As shown in FIG. 14B, the mapping position for one or more samples in the padded four corner regions can be the nearest corner sample position inside the facet when using repetitive padding. If the position of the neighboring block is on the diagonal (e.g., represented using dashed lines in FIG. 15A and / or FIG. 15B), it can be considered unavailable because it may be mapped to a cube facet boundary that may not belong to the facet. If the position of the neighboring block is not on the diagonal, the neighboring block position can be projected to the nearest available position with respect to the facet boundary, as shown in FIG. 15A and / or FIG. 15B, respectively. For example, the neighboring block position can be projected to the nearest available position in the vertical direction with respect to the facet boundary, as shown in FIG. 15A. For example, the neighboring block position can be projected to the nearest available position in the diagonal direction with respect to the facet boundary, as shown in FIG. 15B. The geometry padding technique can be applied to one or more filtering processes during video encoding. The filtering process may include, for example, loop filtering, sample adaptive offset (SAO), and / or adaptive loop filtering (ALF), and may use neighboring samples. Geometry padding may be used to derive neighboring sample values ​​when the neighboring samples are outside the picture boundary or not within the same plane as the sample being filtered.

[0093] Equation (3), Equation (4), and / or Equation (5) use sine and / or cosine functions. Hardware implementation of nonlinear functions, such as sine and / or cosine functions in Equation (3), Equation (4), and / or Equation (5), can use look-up tables (LUTs) and / or linear interpolation. Since they are circular functions, the LUTs can be defined in the range [0,90]. The approximation accuracy can be determined based on application requirements and / or memory size constraints. FIG. 18 shows an example derivation of a sine function using a LUT. sin(x) can be a sine function of a variable x. One or more solid black circles 1602 can define a LUT for a function sin(x) uniformly sampled in the range [0,90]. x can be an input variable whose value may not be defined in the LUT. P and Q can be the first and second nearest neighbors (or, for example, sampling points) of the input x. One or more different interpolations may be applied to derive the value of sin(x). For example, if nearest neighbor interpolation is used, sin(x) may be approximated as sin(x)=sin(P). If linear interpolation is used, sin(x) may be approximated as sin(x)=sin(P)+(sin(Q)-sin(P))×(Qx) / (QP).

[0094] A LUT can be defined with non-uniform sampling across a range, e.g., for sin(x), more sampling points can be used in the range closer to 0 degrees (e.g., because the slope near 0 degrees is larger than that near 90 degrees) and fewer sampling points can be used in the range closer to 90 degrees.

[0095] The faces of a frame-packed picture can be padded with geometry padding and / or stored in a decoded picture buffer. For example, as shown in FIG. 19A, one or more (e.g., all) faces can be padded with geometry padding and / or stored in a decoded picture buffer for motion estimation of future picture encoding. The faces can be padded with geometry padding and / or stored in a decoded picture buffer at the encoder side and / or at the decoder side. The faces can be padded with geometry padding and / or stored in a decoded picture after a reference picture is reconstructed. One or more faces can be padded with geometry padding and / or stored in a decoded picture buffer for motion estimation of future picture encoding. The motion estimation process can determine the faces and / or positions where the current block can be located according to the frame packing information.

[0096] A block (e.g., the entire block) may be skipped without encoding. For example, if the block (e.g., the entire block) is inside a null surface portion marked with "V" in FIG. 19A, the block (e.g., the entire block) may be skipped without encoding. If the block (e.g., the entire block) is not inside a null surface portion marked with "V" in FIG. 19A, the motion estimation process may determine a search window for the block based on a starting position and / or a predefined search range. For example, as shown in FIG. 19A and FIG. 19B, block "B" may be located in surface portion "0". The starting position may be a motion vector predictor derived from its spatial and / or temporal neighboring blocks. The search window may be limited to be within a padded surface portion buffer. For example, the search window may be limited to be within a completely padded surface portion buffer. A motion search may be used. For example, the motion search may include a full search and / or a fast search, etc. For example, the motion search may be used when determining the search window.

[0097] The size of the padded region can affect performance. For example, the size of the padded region can affect performance because the search window can be limited to be within the padded surface buffer. Geometry padding can be performed to obtain padded surfaces for the reference picture. Geometry padding can be performed at the decoder side and / or at the encoder side. The padded surface buffer can be used for motion compensation. Geometry padding can generate a prediction block. For example, geometry padding can generate a prediction block on the fly. The generation of the prediction block via geometry padding can perform padding on a block-by-block basis. For example, the generation of the prediction block via geometry padding can perform padding on an as-needed basis.

[0098] To decode an inter block, the decoder may determine whether the predictive block (or a portion of the predictive block) can be outside or inside the surface segment. For example, the decoder may determine whether the predictive block can be outside the current surface segment. The decoder may determine whether a portion of the predictive block can be outside the current surface segment. The decoder may determine whether the predictive block (or, e.g., a portion of the predictive block) can be outside the current surface segment based on the position of the decoding block and / or its motion vector. The decoder may not perform any padding. For example, if the predictive block is inside the surface segment, the decoder may skip padding. The decoder may generate (e.g., only generate) a particular block (or, e.g., a portion of the predictive block) using a geometry padding technique. The decoder may not pad surface segments boundaries (e.g., the entire surface segment boundary). For example, if a predictive block (and / or a portion of a predictive block) is outside of a surface, the decoder may use geometry padding techniques to generate that particular block (and / or, for example, a portion of the predictive block).

[0099] A lookup table may be used that stores corresponding sample position information. The lookup table may store corresponding sample position information for each padded position of each facet. One or more constraints on the size of the padded region may be defined. For example, a constraint on the maximum size and / or size of the padded region may be defined to reduce requirements on the size of the LUT used for geometry padding. One or more constraints on the size (e.g., maximum size) of the padded region may be defined depending on the picture size, which may be profile / level defined to reduce implementation resource requirements. For example, a video with a larger picture size may use a larger padded region size. The size of the padded region may be conveyed at a sequence level, such as a sequence parameter set (SPS) and / or a picture parameter set (PPS).

[0100] The decoder can be initialized (e.g., faster) since a LUT can be constructed by, for example, communicating the size of the region to be padded at the sequence level. The LUT can be constructed based on the communicated information.

[0101] For motion compensation prediction, when a geometry padding technique is used and when the temporal reference sample is outside the plane to which the current block belongs, the motion vector of the current block and / or the motion vector of the neighboring block of the current block may be different. For example, when the current block and its neighboring blocks belong to different projection planes, the motion vector of the current block and / or the motion vector of the neighboring block of the current block may be completely different. Since two neighboring planes in a frame packed picture may not be adjacent (e.g., directly adjacent) in 3D space, the motion vector of the current block and / or the motion vector of the neighboring block of the current block may be completely different. When two adjacent (e.g., directly adjacent) planes in 3D space are placed in a 2D frame packed picture, there may be a distortion effect, which may cause discontinuity of sample values ​​and / or motion vectors at the plane boundary, so the motion vector of the current block and / or the motion vector of the neighboring block of the current block may be completely different. Whether a motion vector of a neighboring block for a motion vector predictor can be used to predict a current block can be determined based on whether the neighboring block belongs to a different surface. The encoder and / or decoder can check whether one or more (e.g., two) adjacent surface boundary can be connected in 3D space. One surface boundary can be a surface boundary to which the neighboring block can belong. The other surface boundary can be a surface boundary to which the current block can belong. If one or more (e.g., two) surface boundary are connected in 3D space, the motion vector of the neighboring block can be used as a motion vector predictor for the current block. If one or more (e.g., two) surface boundary are not connected in 3D space, the motion vector of the neighboring block may not be used for motion vector prediction of the current block. One or more (e.g., two) motion vector prediction constraints can be applicable to the inter-merge mode.For example, if neighboring blocks belong to different faces, the neighboring blocks may not be used as merge candidates for the inter merge mode.

[0102] It is possible to perform geometry padding for unicube, ACP, and EAC projections.

[0103] As described herein, Equation (11) and / or Equation (12) can provide a relationship between coordinates (x',y') in the unicube domain and coordinates (x,y) in the cube domain. A geometric relationship between coordinates in the cube domain and corresponding 3D points Ps on the sphere can be known. To derive reference samples in the padded region for the unicube projection format, the cubemap geometry padding can be extended using a cubemap as an intermediate stage. For example, to derive reference samples in coordinates in the padded region of a unicube face, one or more of the following can be performed: Coordinates associated with an input projection of a frame packed picture can be mapped to intermediate coordinates. For example, coordinates associated with a unicube projection can be mapped to intermediate coordinates as defined in a cube face, for example, based on Equation (12). The intermediate cube coordinates can be projected onto one or more other cube coordinates in neighboring faces. For example, the intermediate cube coordinates can be projected onto one or more other cube coordinates in neighboring faces based on a geometry projection function of the cubemap format. The cube coordinates can be mapped back to the reference coordinates defined in the unicube domain, for example, based on equation (11). Samples at the reference coordinates within the unicube faces can be derived, for example, using an interpolation filter.

[0104] Coordinates (u,v) within the padded region of the facet can be provided. Geometry padding for unicube format can be performed. One or more coordinates can be converted from unicube format / geometry to cube format / geometry. For example, input coordinates (u,v) can be defined within the unicube facet. Intermediate coordinates (uc,vc) defined within the cube facet can be calculated according to equation (12). Reference coordinates (uc',vc') can be derived by projecting (uc,vc) onto neighboring cube faces via one or more of the following: 2D to 3D mapping can be performed from the cube to 3D space. Depending on which facet (uc,vc) belongs to, the corresponding 3D position (X,Y,Z) can be calculated. For example, if the facet is NZ, the 3D position (X,Y,Z) can be calculated using equations (6), (7), and / or (8). The new facet to which the reference sample belongs can be determined, for example, based on the 3D coordinates (X,Y,Z). If |X|≧|Y|, and |X|≧|Z|, and X>0, the new surface segment can be PX. If |X|≧|Y|, and |X|≧|Z|, and X<0, the new surface can be NX. If |Y|≧|X|, and |Y|≧|Z|, and Y>0, the new surface segment can be PY. If |Y|≧|X|, and |Y|≧|Z|, and Y<0, the new surface segment may be NY. If |Z|≧|X|, and |Z|≧|Y|, and Z>0, the new surface can be PZ. If |Z|≧|X|, and |Z|≧|Y|, and Z<0, the new surface can be NZ. A 3D to 2D mapping can be performed from 3D space to a cube. Using (X,Y,Z), the projected 2D coordinates (uc',vc') can be calculated at the new facet. For example, if the new facet is NZ, (uc',vc') can be derived using (X,Y,Z) using equation (9) and / or equation (10). One or more coordinates can be converted from a cube format / geometry to a unicube format / geometry. For example, given a reference coordinate (uc',vc') defined within a cube facet, the corresponding reference coordinate (u',v') defined within the unicube facet can be calculated. The sample at reference coordinate (u',v') within the unicube facet used for padding can be derived by applying an interpolation filter to samples at integer positions around (u',v') within the unicube facet.

[0105] As shown in equation (12), the transformation function β=g(β') can map coordinates from the unicube faces to the cube faces. The transformation function β=g(β') in equation 12 can include fractions, including square roots, in the denominator. To prevent the denominator from becoming zero, the following can be done:

[0106]

number

[0107] β'∈[-1,1] may define the range of coordinates of samples inside the unicube facet. If β'∈[-1,1] may define the range of coordinates of samples inside the unicube facet, then equation (20) may provide that when any 3D point is projected onto the unicube facet, the projection onto the facet may be centered at the facet center and / or may be within a region that may be 1.5 times as large as the facet size in the horizontal and / or vertical directions. When applying geometry padding to the unicube format, there may be a limit on the maximum size of the padded region for a unicube facet. For example, the maximum size of the padded region for a unicube facet may not exceed 1.5 times the facet size. FIG. 27 shows an example of a valid region of padded samples within a facet for the unicube projection format.

[0108] When motion compensation prediction refers to samples that are outside picture and / or surface boundaries, the geometry padding described herein can be applied to pad the corresponding reference samples, taking into account the 3D geometric structure represented by the coded projection format. For example, a prediction block (or a portion of a prediction block) can be outside the valid area of ​​the padded samples within the unicube surface.

[0109] Geometry padding can be performed in 360-degree video encoding based on unicube projection.

[0110] FIG. 28 illustrates an exemplary MV clipping to a valid region of padded samples for the unicube format. As shown in FIG. 28, the MV clipping operation can be performed before motion compensation prediction in the encoder and / or decoder. For example, the temporal prediction can use a reference sample that is from the valid region of padded samples in the unicube facet. If the MV points to a reference block that can be outside or partially outside the valid padded region of the unicube facet, the MV can be clipped to the closest value so that the reference sample of the prediction block can be within the valid region for padding. As shown in FIG. 28, the reference block A is partially outside the valid padded region. The MV associated with the reference block A can be clipped to generate an MV' that points to the reference block A'. The reference block A' can be a neighboring block that is within the valid padded region.

[0111] Bitstream conformance may be provided to ensure that the coded MV has a proper value. For example, bitstream conformance may be provided to ensure that the coded MV has a proper value where one or more (e.g., all) of the reference samples of the prediction block are from the valid padded area of ​​the unicube facet. Given the position and / or size of the coding block, a range of valid MVs may be determined. For example, the range of valid MVs may correspond to a motion compensated prediction signal that does not use reference samples outside the valid padded area. The value of the MV in the adapted bitstream may be within a predetermined range.

[0112] Hybrid padding can be performed to pad reference samples for motion compensation prediction for a unicube. Hybrid padding can jointly apply geometry padding and iterative padding. If the position of the reference sample is within the valid region of padded samples in the unicube facet, geometry padding can be applied to derive corresponding sample values ​​based on the 3D geometric structure represented in the 360-degree video. If the position of the reference sample is not within the valid region of padded samples in the unicube facet (e.g., the position of the reference sample is outside the valid padded region in the unicube facet), iterative padding can be applied to pad the sample by clipping the sample position to the closest position that is within the valid padded region in the horizontal and / or vertical directions. Figure 29 illustrates an exemplary hybrid padding for a unicube projection format. As shown in Figure 29, reference block B can be outside (e.g., partially outside) the valid padded region in the unicube facet. Samples in sub-block B3 can be inside the valid padded region, and sample values ​​can be derived based on geometry padding. For sub-blocks B0, B1, and B2, corresponding samples can be padded by copying (e.g., repeatedly copying) sample values ​​from the boundaries of the valid padded region. For example, B0 can be filled with the top-left sample of B3. B1 can be filled with the top row of B3. B2 can be filled with the left column of B3. Figure 30 shows an example of a picture padded using hybrid padding. Repeated padding can be used as an example in hybrid padding.To pad samples that may be outside the valid area of ​​padded samples for the unicube projection format, one or more other padding methods (e.g., facet-based padding in FIG. 14 and / or vertical and / or diagonal extrapolation-based padding in FIG. 15) can be applied in the hybrid padding.

[0113] At the sequence level, when the unicube facet size is known, one or more of the following can be pre-calculated and / or stored as a LUT. For example, a padding sample position LUT can indicate the correlation of calculated intermediate coordinates (uc, vc) defined within the cube facet, derived reference coordinates (uc', vc'), and / or calculated reference coordinates (u', v') defined within the unicube facet. The LUT can store derived sample positions that can be used to pad sample values ​​at padded sample positions. Derived mapped samples at reference coordinates (u', v') within the unicube facet can be calculated (e.g., dynamically calculated) for a picture. The projection from 3D space onto the six unicube faces can be symmetric. A reference LUT can be stored for a facet that can be reused by one or more of the other five faces. For example, a single reference LUT can be stored for a facet that can be reused by one or more of the other five faces. FIG. 32 shows an example diagram of using LUTs for six faces for geometry padding for a unicube. As shown in FIG. 32, a 3×2 frame packing arrangement can be applied. In FIG. 32, a blank block 3201 can represent a face area. A patterned block 3203 can represent a padded area. As shown in FIG. 32, sample A and sample B can be two samples that can be located at the same position in the padded area of ​​face #0 and face #5, respectively. Sample A can be projected onto 2D coordinates (u,v) in face #4. Based on the symmetric 3D projection structure as presented in FIG. 32, sample B can be projected to the same coordinates (e.g., (u,v)) in a neighboring face of face #5, e.g., face #0. For example, for the same sample position in the padded area of ​​different faces, the corresponding coordinates in the projected face can be identical.A LUT containing projected coordinates for padded sample locations can be stored for a surface that can be reused to calculate projected coordinates of padded samples in other surfaces. When a LUT is reused across one or more (e.g., all) of the surfaces, the coordinates of the mapped samples can be mapped into the projected surface. For example, when a LUT (e.g., the same LUT) is reused across one or more (e.g., all) of the surfaces based on a rotation during a frame packing process, the coordinates of the mapped samples can be mapped into the projected surface.

[0114] When geometry padding is applied for 360-degree video encoding, a unicube projection may be used herein to describe the limited padded area, but a limited padded area may be applied as exists for other cube-like projection formats (e.g., ACP and / or EAC, etc.) that can achieve different spherical sampling grids by changing the coordinates of samples on the 2D planar surface before the actual cube map surface is generated. When geometry padding is used together with one or more other projection formats, one or more of the geometry padding techniques described herein (e.g., MV clipping, bitstream conformance, and / or hybrid padding) may be applicable. The mapping functions used by different projection formats may vary. The size of the effective padded area may be adjusted based on the projection format. For example, when ACP is used as the encoding projection format, -1.28≦β′≦1.28 may be used to ensure that the inside of the square root of Equation (14) is not less than zero. For example, the maximum size of the padded region for an ACP surface area can be 1.28 times the surface area size. When EAC is used as the coding projection format, the maximum size of the padded region can be derived from equation (16) as -2<β'<2.

[0115] Geometry padding for reference pictures using conformance windows may be provided. The size of the picture to be coded may be a multiple of the minimum CU size in the horizontal and / or vertical directions. For example, a picture may be partitioned into coding units for coding, so that the size of the picture to be coded may be a multiple of the minimum CU size in the horizontal and / or vertical directions. A conformance window may be used and / or a window size may be signaled in the sequence parameter set to make the size of the padded picture a multiple of the minimum CU size. For example, if the source picture size is not a multiple of the minimum CU size, a window size may be signaled in the sequence parameter set to make the size of the padded picture a multiple of the minimum CU size. The parameters W and H may indicate the width and height of the source picture. The parameters mCU_width and mCU_height may indicate the width and height of the minimum CU. The parameters dW and dH may indicate the padded size in the horizontal and vertical directions for the conformance window. Ceil(x) may be a ceiling function for computing the smallest integer value greater than or equal to a variable x. dW=Ceil(W / mCU_width)×mCU_width-W dH=Ceil(H / mCU_height)×mCU_height-H

[0116] Figures 20A, 20B, and 20C show three examples of padding when the source picture size is not a multiple of the minimum CU size in the horizontal and / or vertical directions. Figure 20A shows an example padding in the horizontal direction when the source picture size is not a multiple of the minimum CU size. Figure 20B shows an example padding in the vertical direction when the source picture size is not a multiple of the minimum CU size. Figure 20C shows an example padding in the horizontal and vertical directions when the source picture size is not a multiple of the minimum CU size. When geometry padding is performed on a surface of a reference picture, the padding can start from a surface boundary and can move outward, which may not include the conformance window, to make the padded surface continuous. For example, when geometry padding is performed on a surface of a reference picture, the padding can start from a surface boundary and can move outward, as padding within the conformance window can be indicated by the encoder. For example, the reference picture surface may not include a conformance window to make the padded surface contiguous. For example, the reference picture surface may include a conformance window to make the padded surface contiguous. The conformance window may be kept unchanged and / or when geometry padding is performed, the padding may start from the conformance window.

[0117] Deblocking with geometry padding can be performed. A deblocking filter can be applied to remove blocking artifacts. For example, a deblocking filter can be applied on the boundary between one or more (e.g., two) neighboring prediction units and / or on the boundary between one or more (e.g., two) transform units to remove blocking artifacts introduced by the encoding process. For the boundary, one or more samples in a neighboring block can be used for filtering those samples in the current block. For example, in FIG. 21, block "p" can be a neighboring block of the current block "q". One or more reference samples (p0, p1, p2, and / or p3) can be used to filter one or more reference samples (q0, q1, q2, and / or q3). If the boundary is a segment boundary, block "p" and / or block "q" can be in one or more (e.g., two) different segments. If the boundary is a surface segment boundary, there may be a discontinuity between the reference sample {p0, p1, p2, p3} and the reference sample {q0, q1, q2, q3}. For example, if the boundary is a surface segment boundary, there may be a discontinuity between the sample {p0, p1, p2, p3} and the sample {q0, q1, q2, q3} due to geometry distortion. Application of a deblocking filter on a surface segment boundary may not remove blocking artifacts. Application of a deblocking filter on a surface segment boundary may degrade subjective visual quality. Surface segment boundaries may be treated as picture boundaries. For example, surface segment boundaries may be treated as picture boundaries such that a deblocking filter may not be performed on the surface segment boundary. The derived neighboring sample values ​​may be used according to a geometry padding technique for neighboring samples used as inputs to a deblocking filter instead of directly using those neighboring samples in another surface segment.For example, when filtering samples q0, q1, q2, and / or q3, the derived reference samples p0, p1, p2, and / or p3 can be derived using a geometry padding technique as needed. The surface boundary can be determined according to the frame packing information.

[0118] Sample Adaptive Offset (SAO) with Geometry Padding can be used. SAO can be used to remove high frequency coding artifacts. For example, SAO can be used to remove high frequency coding artifacts in HEVC. SAO can include band offset and / or edge offset. SAO parameters can be conveyed for a coding tree unit (CTU). For example, a CTU can have a block size of 64x64. For edge offset, the SAO process can determine whether the pixel is on an edge boundary or not along a given direction. FIG. 22 shows an example in which a pixel can be processed by edge offset SAO. S(x) can be the sample value at position x. P can be the current sample location, n0 can be its left / top neighboring location, and n1 can be its right / bottom neighboring location. In the four examples shown in FIG. 22, the current sample can be on an edge boundary. The edge boundary can be processed by edge offset SAO. The edge offset SAO process can refer to its neighboring sample values. If the current sample is located on a surface segment boundary, the neighboring samples of the current sample may be located in a different surface segment. An example may use derived samples. For example, the derived samples may replace one or more neighboring samples. If the current sample is on a surface segment boundary, one or more neighboring samples may be replaced. Surface segment boundaries may be treated as picture boundaries, and / or SAO filtering for those surface segment boundary samples may be skipped.

[0119] Whether to signal the SAO parameters may be determined based on the location of the CTU. For example, for 360-degree video encoding, if the CTU (e.g., the entire CTU) is inside a null surface area (marked with "V" in FIG. 19A), the SAO parameters may not be signaled because the samples may be used for frame packing and may not be provided to the user.

[0120] Different projection geometries can have different characteristics. For example, ERP and / or equal area projections may have one facet, and the facet boundary problem may be a trivial matter even when the picture is stretched. A cube map may have six faces and many facet boundaries in a frame packed picture. Thus, a picture may be coded with different projection geometries. A projection geometry index (e.g., as listed in Table 1) may be conveyed for a slice (e.g., for each slice). Table 1 provides an example projection geometry index.

[0121] [Table 1]

[0122] The projection format can be signaled at the slice level. The projection format signaling can be performed at the picture level. For example, the projection format signaling can be performed at the picture level via an index of the projection geometry. At the sequence level, the total number of projection formats used in the entire sequence and / or the format itself can be signaled. Table 2 provides an example projection format signaling at the sequence level.

[0123] [Table 2]

[0124] The parameter num_projection_format may specify the total number of projection formats used in the sequence. The parameter project_geometry_idx[i] may specify the mapping index (e.g., as shown in Table 1) indicated by the i-th index (e.g., as signaled at the picture / slice level). For example, if equirectangular and equal-area projections are used in the sequence, and indices "0" and "1" are used to represent equal-area and equirectangular, respectively, in the bitstream, project_geometiy_idx may be customized and signaled according to Table 2 as follows: project_geometry_idx[0]=2 project_geometry_idx[1]=0

[0125] The projection format for a picture may be determined using rate-distortion optimization. The encoder may encode the current picture using one or more available projection formats. For example, the encoder may encode the current picture using one or more available projection formats. The encoder may measure the rate-distortion cost of the projected format. The projection format with the minimum rate-distortion cost may be selected. The distortion may be measured in the original format domain. For example, O may represent the original projection format, X may represent the measured projection format, I may represent the original picture, and I' may represent the reconstructed picture. The original picture I may be converted from the original projection format O to the coded projection format X. The picture may be coded and reconstructed. Projection(I,F) may represent a projection function used to convert picture I to format F. The distortion between I and the picture converted from I' may be determined as follows: ||t|| may be the sum of squared errors (SSE) of variable t. Distortion = ||I-Projection(I',O)||

[0126] Decoded Picture Buffer (DPB) management can be performed as follows: An on-the-fly conversion mode can be used. The reconstructed picture can be stored in the DPB. If the reference picture and the current picture have different projection formats, the reference picture can be converted to the format of the current picture. Figures 23A and 23B show examples of how DPB management can be performed. The shaded and blank blocks can indicate two projection formats, respectively. The dashed blocks can indicate virtual pictures, and the solid blocks can indicate physical pictures. The reference picture R0 can be in one projection format. The reference pictures R1 and R2 can be in another projection format (e.g., a projection format different from the projection format for R0). The current picture P can have the same projection format as the pictures R1 and R2. The block pictures with dashed borders (e.g., R1 and R2 belonging to the reference picture list shown in Figure 23A) can represent virtual pictures. A block with a solid border (e.g., R0 belonging to the reference picture list shown in FIG. 23A) may represent a physical picture stored in the buffer. As shown in FIG. 23A, reference picture R0 may be converted to the projection format of picture P before it is decoded. The converted version may be placed on the reference picture list. Reference pictures R1 and R2 may be placed in the reference picture buffer. For example, reference pictures R1 and R2 have the same projection format as picture P, so they may be placed directly in the reference picture buffer.

[0127] While the reference pictures are stored in the DPB, the reference pictures can be converted to different projection formats. As shown in Figure 23B, the reference pictures can be selected from the DPB based on the projection format of the current picture P. The reference pictures can be used to construct a reference picture list.

[0128] One or more (e.g., all) intra slices and one or more (e.g., all) inter slices may use the same projection geometry coding. One or more (e.g., all) intra slices may use one projection geometry coding, while one or more (e.g., all) inter slices may use another projection geometry (e.g., different from the projection geometry coding used in the intra slice). The reconstructed intra picture may be converted to the projection geometry used by the inter slice. For example, the reconstructed intra picture, which may be present in the decoded picture buffer, may be converted to the projection geometry used by the inter slice so that motion compensation can be directly applied. For example, the intra slice may use ERP for coding, while the inter slice may use cube map projection for coding. The resolution for the two projections may be different. For example, the intra slice may have a larger resolution than the inter slice. If the resolutions are different, the reconstructed intra picture (e.g., the one stored in the decoded picture buffer) may be converted to the geometry projection used by the inter slice. The reconstructed intra picture can be resampled to the same resolution as the inter slice.

[0129] FIG. 33A illustrates a diagram of an example communication system 100 in which one or more disclosed embodiments may be implemented and / or used. The communication system 100 may be a multiple access system that provides content, such as voice, data, video, messaging, broadcast, etc., to multiple wireless users. The communication system 100 may enable the multiple wireless users to access such content through sharing of system resources, including wireless bandwidth. For example, the communication system 100 may utilize one or more channel access methods, such as Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Frequency Division Multiple Access (FDMA), Orthogonal FDMA (OFDMA), and Single Carrier FDMA (SC-FDMA).

[0130] 33A, the communications system 100 may include wireless transmit / receive units (WTRUs) 102a, 102b, 102c, and / or 102d (which may be referred to collectively or collectively as WTRUs 102), radio access networks (RANs) 103 / 104 / 105, core networks 106 / 107 / 109, public switched telephone network (PSTN) 108, the Internet 110, and other networks 112, although it will be understood that the disclosed embodiments contemplate any number of WTRUs, base stations, networks, and / or network elements. Each of the WTRUs 102a, 102b, 102c, and / or 102d may be any type of device configured to operate and / or communicate in a wireless environment. As examples, WTRUs 102a, 102b, 102c, and / or 102d may be configured to transmit and / or receive wireless signals and may include user equipment (UE), mobile stations, fixed or mobile subscriber units, pagers, cellular phones, personal digital assistants (PDAs), smartphones, laptops, netbooks, personal computers, wireless sensors, and home appliances.

[0131] The communications system 100 may also include a base station 114a and a base station 114b. Each of the base stations 114a, 114b may be any type of device configured to wirelessly interface with at least one of the WTRUs 102a, 102b, 102c, and / or 102d to facilitate access to one or more communications networks, such as the core network 106 / 107 / 109, the Internet 110, and / or the network 112. By way of example, the base stations 114a and / or 114b may be a base transceiver station (BTS), a Node B, an eNode B, a Home Node B, a Home eNode B, a site controller, an access point (AP), a wireless router, and the like. Although the base stations 114a, 114b are each shown as a single element, it will be understood that the base stations 114a, 114b may include any number of interconnected base stations and / or network elements.

[0132] The base station 114a may be part of the RAN 103 / 104 / 105, which may also include other base stations and / or network elements (not shown), such as a base station controller (BSC), a radio network controller (RNC), relay nodes, etc. The base station 114a and / or the base station 114b may be configured to transmit and / or receive wireless signals within a particular geographical area, sometimes referred to as a cell (not shown). The cell may be further divided into cell sectors. For example, the cell associated with the base station 114a may be divided into three sectors. Thus, in one embodiment, the base station 114a may include three transceivers, one for each sector of the cell. In another embodiment, the base station 114a may utilize multiple-input multiple-output (MIMO) technology and thus may utilize multiple transceivers for each sector of the cell.

[0133] The base stations 114a and / or 114b may communicate with one or more of the WTRUs 102a, 102b, 102c, and / or 102d over an air interface 115 / 116 / 117, which may be any suitable wireless communication link (e.g., radio frequency (RF), microwave, infrared (IR), ultraviolet (UV), visible light, etc.). The air interface 115 / 116 / 117 may be established using any suitable radio access technology (RAT).

[0134] More specifically, as mentioned above, the communication system 100 may be a multiple access system and may utilize one or more channel access schemes, such as CDMA, TDMA, FDMA, OFDMA, and SC-FDMA. For example, the base station 114a in the RAN 103 / 104 / 105 and the WTRUs 102a, 102b, and / or 102c may implement a radio technology such as Universal Mobile Telecommunications System (UMTS) Terrestrial Radio Access (UTRA), which may establish the air interface 115 / 116 / 117 using Wideband CDMA (WCDMA). WCDMA may include communication protocols such as High Speed ​​Packet Access (HSPA) and / or Evolved HSPA (HSPA+). HSPA may include High Speed ​​Downlink Packet Access (HSDPA) and / or High Speed ​​Uplink Packet Access (HSUPA).

[0135] In another embodiment, the base station 114a and the WTRUs 102a, 102b, and / or 102c may implement a radio technology such as Evolved UMTS Terrestrial Radio Access (E-UTRA), which may establish the air interface 115 / 116 / 117 using Long Term Evolution (LTE) and / or LTE-Advanced (LTE-A).

[0136] In other embodiments, the base station 114a, and the WTRUs 102a, 102b, and / or 102c may implement a radio technology such as IEEE 802.16 (i.e., Worldwide Interoperability for Microwave Access (WiMAX)), CDMA2000, CDMA2000 1X, CDMA2000 EV-DO, Interim Standard 2000 (IS-2000), Interim Standard 95 (IS-95), Interim Standard 856 (IS-856), Global System for Mobile Communications (GSM), Enhanced Data Rates for GSM Evolution (EDGE), and GSM EDGE (GERAN).

[0137] The base station 114b of FIG. 33A may be, for example, a wireless router, a Home NodeB, a Home eNodeB, or an access point, and may utilize any suitable RAT to facilitate wireless connectivity in a local area, such as a workplace, a home, a vehicle, and a campus. In one embodiment, the base station 114b and the WTRUs 102c, 102d may implement a radio technology, such as IEEE 802.11, to establish a wireless local area network (WLAN). In another embodiment, the base station 114b and the WTRUs 102c, 102d may implement a radio technology, such as IEEE 802.15, to establish a wireless personal area network (WPAN). In yet another embodiment, the base station 114b and the WTRUs 102c, 102d may utilize a cellular-based RAT (e.g., WCDMA, CDMA2000, GSM, LTE, LTE-A, etc.) to establish a picocell or femtocell. 33A, the base station 114b may have a direct connection to the Internet 110. Thus, the base station 114b may not need to access the Internet 110 through the core network 106 / 107 / 109.

[0138] The RAN 103 / 104 / 105 may communicate with the core network 106 / 107 / 109, which may be any type of network configured to provide voice, data, application, and / or Voice over Internet Protocol (VoIP) services to one or more of the WTRUs 102a, 102b, 102c, and / or 102d. For example, the core network 106 / 107 / 109 may provide call control, billing services, mobile location-based services, prepaid calling, Internet connectivity, video distribution, etc., and / or perform high level security functions, such as user authentication. Although not shown in FIG. 33A, it will be appreciated that the RAN 103 / 104 / 105 and / or the core network 106 / 107 / 109 may communicate directly or indirectly with other RANs that utilize the same RAT as the RAN 103 / 104 / 105 or a different RAT. For example, in addition to connecting to the RANs 103 / 104 / 105 that may utilize E-UTRA radio technology, the core networks 106 / 107 / 109 may also communicate with another RAN (not shown) that utilizes GSM radio technology.

[0139] The core network 106 / 107 / 109 may also serve as a gateway for the WTRUs 102a, 102b, 102c, and / or 102d to access the PSTN 108, the Internet 110, and / or other networks 112. The PSTN 108 may include a circuit-switched telephone network providing plain old telephone service (POTS). The Internet 110 may include a global system of interconnected computer networks and devices that use common communication protocols, such as Transmission Control Protocol (TCP), User Datagram Protocol (UDP), and Internet Protocol (IP) in the TCP / IP Internet protocol suite. The network 112 may include wired or wireless communication networks owned and / or operated by other service providers. For example, the network 112 may include another core network connected to one or more RANs that may utilize the same RAT as the RANs 103 / 104 / 105 or a different RAT.

[0140] Some or all of the WTRUs 102a, 102b, 102c, and / or 102d in the communications system 100 may include multi-mode capabilities, i.e., the WTRUs 102a, 102b, 102c, and / or 102d may include multiple transceivers for communicating with different wireless networks over different wireless links. For example, the WTRU 102c shown in FIG. 33A may be configured to communicate with a base station 114a that may utilize a cellular-based wireless technology and with a base station 114b that may utilize an IEEE 802 wireless technology.

[0141] Figure 33B shows a system diagram of an example WTRU 102. As shown in Figure 33B, the WTRU 102 may include a processor 118, a transceiver 120, a transmit / receive element 122, a speaker / microphone 124, a keypad 126, a display / touchpad 128, non-removable memory 130, removable memory 132, a power source 134, a Global Positioning System (GPS) chipset 136, and other peripherals 138. It will be understood that the WTRU 102 may include any subcombination of the above elements while remaining consistent with an embodiment. Also, embodiments contemplate that the base stations 114a, 114b and / or nodes that they may represent, such as, but not limited to, a Transmitter-Transmitter Station (BTS), a Node B, a site controller, an Access Point (AP), a Home Node B, an Evolved Node B (eNodeB), a Home Evolved Node B (HeNB), a Home Evolved Node B Gateway, and a proxy node, among others, may include some or all of the elements shown in FIG. 33B and described herein.

[0142] The processor 118 may be a general purpose processor, a special purpose processor, a conventional processor, a digital signal processor (DSP), multiple microprocessors, one or more microprocessors in conjunction with a DSP core, a controller, a microcontroller, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) circuit, any other type of integrated circuit (IC), a state machine, etc. The processor 118 may perform signal coding, data processing, power control, input / output processing, and / or any other functionality that enables the WTRU 102 to operate in a wireless environment. The processor 118 may be coupled to the transceiver 120, which may be coupled to the transmit / receive element 122. Although FIG. 33B depicts the processor 118 and the transceiver 120 as separate components, it will be appreciated that the processor 118 and the transceiver 120 may be integrated together in an electronic package or chip.

[0143] The transmit / receive element 122 may be configured to transmit signals to or receive signals from a base station (e.g., base station 114a) over the air interface 115 / 116 / 117. For example, in one embodiment, the transmit / receive element 122 may be an antenna configured to transmit and / or receive RF signals. In another embodiment, the transmit / receive element 122 may be an emitter / detector configured to transmit and / or receive IR, UV, or visible light signals, for example. In yet another embodiment, the transmit / receive element 122 may be configured to transmit and receive both RF and light signals. It will be appreciated that the transmit / receive element 122 may be configured to transmit and / or receive any combination of wireless signals.

[0144] Additionally, although the transmit / receive element 122 is shown as a single element in Figure 33B, the WTRU 102 may include any number of transmit / receive elements 122. More specifically, the WTRU 102 may utilize MIMO technology. Thus, in one embodiment, the WTRU 102 may include two or more transmit / receive elements 122 (e.g., multiple antennas) for transmitting and receiving wireless signals over the air interface 115 / 116 / 117.

[0145] The transceiver 120 may be configured to modulate signals to be transmitted by the transmit / receive element 122 and demodulate signals received by the transmit / receive element 122. As mentioned above, the WTRU 102 may have multi-mode capabilities. Thus, the transceiver 120 may include multiple transceivers to enable the WTRU 102 to communicate via multiple RATs, such as, for example, UTRA and IEEE 802.11.

[0146] The processor 118 of the WTRU 102 may be coupled to and may receive user input data from a speaker / microphone 124, a keypad 126, and / or a display / touchpad 128 (e.g., a liquid crystal display (LCD) display unit or an organic light emitting diode (OLED) display unit). The processor 118 may also output user data to the speaker / microphone 124, the keypad 126, and / or the display / touchpad 128. In addition, the processor 118 may obtain information from and store data in any type of suitable memory, such as a non-removable memory 130 and / or a removable memory 132. The non-removable memory 130 may include a random access memory (RAM), a read only memory (ROM), a hard disk, or any other type of memory storage device. The removable memory 132 may include a subscriber identity module (SIM) card, a memory stick, a secure digital (SD) memory card, and the like. In other embodiments, the processor 118 may access information from, and store data in, memory that is not physically located on the WTRU 102, such as located on a server or home computer (not shown).

[0147] The processor 118 may receive power from the power source 134 and may be configured to distribute and / or control the power to other components within the WTRU 102. The power source 134 may be any suitable device for powering the WTRU 102. For example, the power source 134 may include one or more dry batteries (e.g., nickel-cadmium (NiCd), nickel-zinc (NiZn), nickel-metal hydride (NiMH), lithium-ion (Li-ion), etc.), solar cells, fuel cells, etc.

[0148] The processor 118 may also be coupled to a GPS chipset 136, which may be configured to provide location information (e.g., longitude and latitude) regarding the current location of the WTRU 102. In addition to or instead of information from the GPS chipset 136, the WTRU 102 may receive location information over the air interface 115 / 116 / 117 from a base station (e.g., base stations 114a, 114b) and / or may determine its location based on the timing of signals received from two or more nearby base stations. It will be appreciated that the WTRU 102 may acquire location information using any suitable location determination method while remaining consistent with an embodiment.

[0149] The processor 118 may further be coupled to other peripherals 138, which may include one or more software and / or hardware modules that provide additional features, functionality, and / or wired or wireless connectivity. For example, the peripherals 138 may include an accelerometer, an e-compass, a satellite transceiver, a digital camera (for photos or videos), a Universal Serial Bus (USB) port, a vibration device, a television transceiver, a hands-free headset, a Bluetooth module, a frequency modulation (FM) radio unit, a digital music player, a media player, a video game player module, an Internet browser, and the like.

[0150] FIG. 33C illustrates a system diagram of the RAN 103 and the core network 106, according to an embodiment. As mentioned above, the RAN 103 may communicate with the WTRUs 102a, 102b, and / or 102c over the air interface 115 utilizing UTRA radio technology. The RAN 103 may also communicate with the core network 106. As shown in FIG. 33C, the RAN 103 may include Node Bs 140a, 140b, and / or 140c, each of which may include one or more transceivers for communicating with the WTRUs 102a, 102b, and / or 102c over the air interface 115. The Node Bs 140a, 140b, and / or 140c may each be associated with a particular cell (not shown) within the RAN 103. The RAN 103 may also include RNCs 142a and / or 142b. It will be appreciated that the RAN 103 may include any number of Node Bs and RNCs while remaining consistent with an embodiment.

[0151] As shown in FIG. 33C, Node B 140a and / or 140b can communicate with RNC 142a. Additionally, Node B 140c can communicate with RNC 142b. Node B 140a, 140b, and / or 140c can communicate with their respective RNCs 142a, 142b via an Iub interface. The RNCs 142a, 142b can communicate with each other via an Iur interface. Each of the RNCs 142a, 142b can be configured to control the respective Node B 140a, 140b, and / or 140c to which it is connected. Additionally, each of the RNCs 142a, 142b can be configured to implement or support other functionalities, such as outer loop power control, load control, admission control, packet scheduling, handover control, macro diversity, security functions, and data encryption.

[0152] The core network 106 shown in Figure 33C may include a media gateway (MGW) 144, a mobile switching center (MSC) 146, a serving GPRS support node (SGSN) 148, and / or a gateway GPRS support node (GGSN) 150. Although each of the above elements is shown as part of the core network 106, it will be understood that any one of these elements may be owned and / or operated by an entity different from the core network operator.

[0153] The RNC 142a in the RAN 103 may be connected to an MSC 146 in the core network 106 via an IuCS interface. The MSC 146 may be connected to the MGW 144. The MSC 146 and MGW 144 may provide the WTRUs 102a, 102b, and / or 102c with access to circuit-switched networks, such as the PSTN 108, to facilitate communications between the WTRUs 102a, 102b, and / or 102c and traditional landline communication devices.

[0154] The RNC 142a in the RAN 103 may also be connected to an SGSN 148 in the core network 106 via an IuPS interface. The SGSN 148 may be connected to a GGSN 150. The SGSN 148 and GGSN 150 may provide the WTRUs 102a, 102b, and / or 102c with access to packet-switched networks, such as the Internet 110, to facilitate communications between the WTRUs 102a, 102b, and / or 102c and IP-enabled devices.

[0155] As mentioned above, the core network 106 may also connect to the networks 112, which may include other wired or wireless networks owned and / or operated by other service providers.

[0156] 33D shows a system diagram of the RAN 104 and the core network 107, according to an embodiment. As mentioned above, the RAN 104 may utilize E-UTRA radio technology to communicate with the WTRUs 102a, 102b, and / or 102c over the air interface 116. The RAN 104 may also communicate with the core network 107.

[0157] The RAN 104 may include eNodeBs 160a, 160b, and / or 160c, although it will be understood that the RAN 104 may include any number of eNodeBs while remaining consistent with an embodiment. The eNodeBs 160a, 160b, and / or 160c may each include one or more transceivers for communicating with the WTRUs 102a, 102b, and / or 102c over the air interface 116. In one embodiment, the eNodeBs 160a, 160b, and / or 160c may implement MIMO techniques. Thus, the eNodeB 160a, for example, may use multiple antennas to transmit wireless signals to and receive wireless signals from the WTRU 102a.

[0158] Each of the eNodeBs 160a, 160b, and / or 160c may be associated with a particular cell (not shown) and may be configured to handle radio resource management decisions, handover decisions, scheduling of users on the uplink and / or downlink, etc. As shown in FIG. 33D, the eNodeBs 160a, 160b, and / or 160c may communicate with one another over an X2 interface.

[0159] The core network 107 shown in Figure 33D may include a mobility management gateway (MME) 162, a serving gateway 164, and a packet data network (PDN) gateway 166. Although each of the above elements is shown as part of the core network 107, it will be understood that any one of these elements may be owned and / or operated by an entity different from the core network operator.

[0160] The MME 162 may be connected to each of the eNodeBs 160a, 160b, and / or 160c in the RAN 104 via an S1 interface and may act as a control node. For example, the MME 162 may be responsible for authentication of users of the WTRUs 102a, 102b, and / or 102c, bearer activation / deactivation, selection of a particular serving gateway during initial attachment of the WTRUs 102a, 102b, and / or 102c, etc. The MME 162 may also provide a control plane function for switching between the RAN 104 and other RANs (not shown) that employ other radio technologies, such as GSM or WCDMA.

[0161] The serving gateway 164 may be connected to each of the eNodeBs 160a, 160b, and / or 160c in the RAN 104 via an S1 interface. The serving gateway 164 may generally route and forward user data packets to / from the WTRUs 102a, 102b, and / or 102c. The serving gateway 164 may also perform other functions, such as anchoring the user plane during inter-eNodeB handovers, triggering paging when downlink data is available to the WTRUs 102a, 102b, and / or 102c, and managing and storing the context of the WTRUs 102a, 102b, and / or 102c.

[0162] The serving gateway 164 may also be connected to a PDN gateway 166, which may provide the WTRUs 102a, 102b, and / or 102c with access to packet-switched networks, such as the Internet 110, to facilitate communications between the WTRUs 102a, 102b, and / or 102c and IP-enabled devices.

[0163] The core network 107 may facilitate communications with other networks. For example, the core network 107 may provide the WTRUs 102a, 102b, and / or 102c with access to circuit-switched networks, such as the PSTN 108, to facilitate communications between the WTRUs 102a, 102b, and / or 102c and traditional landline communication devices. For example, the core network 107 may include or communicate with an IP gateway (e.g., an IP Multimedia Subsystem (IMS) server) that serves as an interface between the core network 107 and the PSTN 108. In addition, the core network 107 may provide the WTRUs 102a, 102b, and / or 102c with access to the network 112, which may include other wired or wireless networks owned and / or operated by other service providers.

[0164] 33E shows a system diagram of the RAN 105 and the core network 109, according to an embodiment. The RAN 105 may be an Access Service Network (ASN) that utilizes IEEE 802.16 wireless technology to communicate with the WTRUs 102a, 102b, and / or 102c over the air interface 117. As described further below, communication links between different functional entities of the WTRUs 102a, 102b, and / or 102c, the RAN 105, and the core network 109 may be defined as reference points.

[0165] As shown in FIG. 33E, the RAN 105 may include base stations 180a, 180b, and / or 180c and an ASN gateway 182, although it will be understood that the RAN 105 may include any number of base stations and ASN gateways while remaining consistent with an embodiment. The base stations 180a, 180b, and / or 180c may each be associated with a particular cell (not shown) in the RAN 105 and each may include one or more transceivers for communicating with the WTRUs 102a, 102b, and / or 102c over the air interface 117. In one embodiment, the base stations 180a, 180b, and / or 180c may implement MIMO technology. Thus, the base station 180a, for example, may use multiple antennas to transmit wireless signals to and receive wireless signals from the WTRUs 102a. The base stations 180a, 180b, and / or 180c may also provide mobility management functions such as handoff triggering, tunnel establishment, radio resource management, traffic classification, and quality of service (QoS) policy enforcement. The ASN gateway 182 may act as a traffic aggregation point and may be responsible for paging, caching of subscriber profiles, routing to the core network 109, etc.

[0166] The air interface 117 between the WTRUs 102a, 102b, and / or 102c and the RAN 105 may be defined as an R1 reference point, which implements the IEEE 802.16 specification. In addition, each of the WTRUs 102a, 102b, and / or 102c may establish a logical interface (not shown) with the core network 109. The logical interface between the WTRUs 102a, 102b, and / or 102c and the core network 109 may be defined as an R2 reference point, which may be used for authentication, authorization, IP host configuration management, and / or mobility management.

[0167] The communication link between each of the base stations 180a, 180b, and / or 180c may be defined as an R8 reference point, which includes protocols for facilitating WTRU handovers and the transfer of data between base stations. The communication link between the base stations 180a, 180b, and / or 180c and the ASN gateway 182 may be defined as an R6 reference point, which may include protocols for facilitating mobility management based on mobility events associated with each of the WTRUs 102a, 102b, and / or 102c.

[0168] As shown in Figure 33E, the RAN 105 may be connected to a core network 109. The communication link between the RAN 105 and the core network 109 may be defined as an R3 reference point, which includes protocols for facilitating, for example, data forwarding and mobility management functions. The core network 109 may include a Mobile IP Home Agent (MIP-HA) 184, an Authentication, Authorization and Accounting (AAA) server 186, and a gateway 188. Although each of the above elements is shown as part of the core network 109, it will be understood that any one of these elements may be owned and / or operated by an entity different from the core network operator.

[0169] The MIP-HA may be responsible for IP address management and may enable the WTRUs 102a, 102b, and / or 102c to roam between different ASNs and / or different core networks. The MIP-HA 184 may provide the WTRUs 102a, 102b, and / or 102c with access to packet-switched networks, such as the Internet 110, to facilitate communications between the WTRUs 102a, 102b, and / or 102c and IP-enabled devices. The AAA server 186 may be responsible for user authentication and support of user services. The gateway 188 may facilitate interworking with other networks. For example, the gateway 188 may provide the WTRUs 102a, 102b, and / or 102c with access to circuit-switched networks, such as the PSTN 108, to facilitate communications between the WTRUs 102a, 102b, and / or 102c and traditional landline communication devices. In addition, the gateway 188 may provide the WTRUs 102a, 102b, and / or 102c with access to the network 112, which may include other wired or wireless networks owned and / or operated by other service providers.

[0170] Although not shown in Figure 33E, it should be understood, can be understood, and / or will be understood that the RAN 105 can be connected to other ASNs and the core network 109 can be connected to other core networks. The communication link between the RAN 105 and the other ASNs can be defined as an R4 reference point, which can include protocols for coordinating mobility of the WTRUs 102a, 102b, and / or 102c between the RAN 105 and the other ASNs. The communication link between the core network 109 and the other core networks can be defined as an R5 reference point, which can include protocols for facilitating interworking between a home core network and a visited core network.

[0171] Although features and elements have been described above in certain combinations, one skilled in the art will understand that each feature or element can be used alone or in any combination with other features and elements. In addition, the methods described herein can be implemented in a computer program, software, or firmware contained in a computer-readable medium executed by a computer or processor. Examples of computer-readable media include electronic signals (transmitted over a wired or wireless connection) and computer-readable storage media. Examples of computer-readable storage media include, but are not limited to, read-only memory (ROM), random access memory (RAM), registers, cache memory, semiconductor memory devices, magnetic media such as internal hard disks and removable disks, magneto-optical media, and optical media such as CD-ROM disks and digital versatile disks (DVDs). A processor in conjunction with software can be used to implement a radio frequency transceiver for use in a WTRU, UE, terminal, base station, RNC, or any host computer. [Industrial Applicability]

[0172] The present invention can be used in communication video coding.

Claims

1. 1. A method of video decoding, comprising: obtaining a current block location in video content, the current block location being associated with a current block of the video content; obtaining spatial neighboring block locations, the current block being configured to be decoded based on the spatial neighboring block locations; determining that the spatial neighboring block locations are located outside a content boundary of the video content; based on the determination that the spatial neighboring block location is located outside the content boundary of the video content, obtaining a mapped block location associated with the spatial neighboring block location; obtaining a prediction mode attribute associated with the mapped block location; decoding the current block based on the obtained prediction mode attribute associated with the mapped block location; A method for providing the above.

2. The video content includes a plurality of surfaces, and the step of obtaining mapped block locations associated with the spatial neighboring block locations includes: projecting the spatial neighboring block locations perpendicular to a diagonal extending out of a corner of a facet of the plurality of facets to obtain an intermediate location; The method of claim 1 , comprising projecting the intermediate location perpendicular to a second content boundary to obtain the mapped block location associated with the spatially neighboring block location.

3. The video content includes a plurality of surfaces, and the step of obtaining mapped block locations associated with the spatial neighboring block locations includes: projecting the spatial neighboring block locations obliquely onto a diagonal extending out of a corner of a facet of the plurality of facets to obtain an intermediate location; The method of claim 1 , comprising: projecting the intermediate location obliquely onto a second content boundary to obtain the mapped block location associated with the spatially neighboring block location.

4. The prediction mode attributes associated with the mapped block location include a merge mode; obtaining indices of candidate block locations associated with the spatial neighboring block locations, the indices of the candidate block locations including the mapped block locations; obtaining the mapped block location based on the obtained index of the candidate block location; obtaining one or more motion vectors for predictive motion information of the current block based on the obtained mapped block location; The method of claim 1 further comprising:

5. the prediction mode attribute associated with the mapped block location includes an intra-mode; reconstructing the current block based on the intra mode of a block associated with the mapped block location. The method of claim 1 further comprising:

6. the video content includes a plurality of views; 2. The method of claim 1, wherein the spatially neighboring block locations are located within a first surface and the mapped block locations are located within a second surface, the first surface being different from the second surface.

7. the video content includes a plurality of views; The step of obtaining the mapped block location includes: calculating a 3D position associated with the spatial neighboring block location, the spatial neighboring block location being associated with a first facet; identifying a second surface associated with the mapped block location based on the calculated 3D positions associated with the spatially neighboring block locations; applying a geometric projection to the 3D positions of the spatially neighboring block locations to derive 2D planar positions of the spatially neighboring block locations within the second surface as the mapped block locations; 2. The method of claim 1, comprising:

8. the video content includes a plurality of surfaces associated with a first geometric projection; transforming coordinates associated with the first geometric projection into intermediate coordinates, the intermediate coordinates being associated with a second geometric projection, the 3D positions of the spatial neighboring block locations being obtained in the intermediate coordinates, and the 2D planar positions of the spatial neighboring block locations within the second surface being obtained in the intermediate coordinates; Transforming the derived 2D plane positions of spatial neighboring block locations associated with the second geometric projection back to the coordinates associated with the first geometric projection. The method of claim 7 further comprising:

9. 2. The method of claim 1 , wherein the content boundary is at least one of a frame-packed picture boundary or a surface boundary, the video content includes 360-degree video content, and the mapped block locations are obtained based on a circular characteristic of the 360-degree video content.

10. 1. An apparatus for video decoding, comprising: obtaining a current block location in the video content, the current block location being associated with a current block of the video content; obtaining spatial neighboring block locations, the current block being configured to be decoded based on the spatial neighboring block locations; determining that the spatial neighboring block location is located outside a content boundary of the video content; based on the determination that the spatial neighboring block location is located outside the content boundary of the video content, obtain a mapped block location associated with the spatial neighboring block location; obtaining a prediction mode attribute associated with the mapped block location; Decoding the current block based on the obtained prediction mode attribute associated with the mapped block location. Processor configured for An apparatus comprising:

11. The video content includes a plurality of surfaces, and obtaining the mapped block locations associated with the spatial neighboring block locations includes the processor: projecting the spatial neighboring block locations perpendicular to a diagonal extending out of a corner of a facet of the plurality of facets to obtain an intermediate location; Projecting the intermediate location perpendicular to a second content boundary to obtain the mapped block location associated with the spatially neighboring block location.

11. The apparatus of claim 10 configured to:

12. The video content includes a plurality of surfaces, and obtaining the mapped block locations associated with the spatial neighboring block locations includes the processor: projecting the spatial neighboring block locations obliquely onto a diagonal extending out of a corner of a facet of the plurality of facets to obtain an intermediate location; and projecting the intermediate location obliquely onto a second content boundary to obtain the mapped block location associated with the spatially neighboring block location.

11. The apparatus of claim 10 configured to:

13. The prediction mode attribute associated with the mapped block location includes a merge mode, and the processor: obtaining an index of a candidate block location associated with the spatial neighboring block location, the index of the candidate block location including the mapped block location; Obtaining the mapped block location based on the obtained index of the candidate block location; Obtaining one or more motion vectors for predictive motion information of the current block based on the obtained mapped block location.

11. The apparatus of claim 10 configured to:

14. The prediction mode attribute associated with the mapped block location includes an intra-mode, and the processor: The apparatus of claim 10 , configured to reconstruct the current block based on the intra mode of a block associated with the mapped block location.

15. 11. The apparatus of claim 10, wherein the video content comprises a plurality of surfaces, the spatially neighboring block locations are located within a first surface and the mapped block locations are located within a second surface, the first surface is different from the second surface, the content boundary is at least one of a frame packed picture boundary or a surface boundary, the video content comprises 360 degree video content, and the mapped block locations are obtained based on a circular characteristic of the 360 ​​degree video content.

16. the video content includes a plurality of views; Obtaining the mapped block location comprises: calculating a 3D position associated with the spatial neighboring block location, the spatial neighboring block location being associated with a first facet; identifying a second surface associated with the mapped block location based on the calculated 3D positions associated with the spatially neighboring block locations; Applying a geometric projection to the 3D positions of the spatially neighboring block locations to derive 2D planar positions of the spatially neighboring block locations within the second surface as the mapped block locations.

11. The apparatus of claim 10 configured to:

17. The video content includes a plurality of surfaces associated with a first geometric projection, the processor comprising: transforming coordinates associated with the first geometric projection into intermediate coordinates, the intermediate coordinates being associated with a second geometric projection, the 3D positions of the spatial neighboring block locations being obtained in the intermediate coordinates, and the 2D planar positions of the spatial neighboring block locations within the second surface being obtained in the intermediate coordinates; Transforming the derived 2D plane positions of spatial neighboring block locations associated with the second geometric projection back to the coordinates associated with the first geometric projection.

20. The apparatus of claim 16, further configured to:

18. 1. A method of video de-encoding, comprising: obtaining video content; obtaining a current block location in the video content, the current block location being associated with a current block of the video content; obtaining spatial neighboring block locations, the current block being configured to be encoded based on the spatial neighboring block locations; determining that the spatial neighboring block locations are located outside a content boundary of the video content; based on the determination that the spatial neighboring block location is located outside the content boundary of the video content, obtaining a mapped block location associated with the spatial neighboring block location; obtaining a prediction mode attribute associated with the mapped block location; encoding the current block based on the obtained prediction mode attribute associated with the mapped block location; A method for providing the above.

19. The prediction mode attributes associated with the mapped block location include a merge mode; obtaining indices of candidate block locations associated with the spatial neighboring block locations, the indices of the candidate block locations including the mapped block locations; obtaining the mapped block location based on the obtained index of the candidate block location; obtaining one or more motion vectors for predictive motion information of the current block based on the obtained mapped block location; 20. The method of claim 18, comprising:

20. 20. The method of claim 18, wherein the content boundary is at least one of a frame packed picture boundary or a surface boundary, the video content includes 360 degree video content, and the mapped block locations are obtained based on a circular characteristic of the 360 ​​degree video content.

21. An apparatus for video de-encoding, comprising: Get video content, obtaining a current block location in the video content, the current block location being associated with a current block of the video content; obtaining spatial neighboring block locations, the current block being configured to be encoded based on the spatial neighboring block locations; determining that the spatial neighboring block location is located outside a content boundary of the video content; based on the determination that the spatial neighboring block location is located outside the content boundary of the video content, obtain a mapped block location associated with the spatial neighboring block location; obtaining a prediction mode attribute associated with the mapped block location; encoding the current block based on the obtained prediction mode attribute associated with the mapped block location. Processor configured for An apparatus comprising:

22. The prediction mode attribute associated with the mapped block location includes a merge mode, and the processor: obtaining an index of a candidate block location associated with the spatial neighboring block location, the index of the candidate block location including the mapped block location; Obtaining the mapped block location based on the obtained index of the candidate block location; Obtaining one or more motion vectors for predictive motion information of the current block based on the obtained mapped block location.

22. The apparatus of claim 21 configured to:

23. 22. The apparatus of claim 21, wherein the content boundary is at least one of a frame packed picture boundary or a surface boundary, the video content includes 360 degree video content, and the mapped block locations are obtained based on a circular characteristic of the 360 ​​degree video content.

24. A non-transitory computer readable medium comprising instructions for video decoding, causing a processor to perform the method of any of claims 1 to 9.

25. A non-transitory computer readable medium comprising instructions for video encoding, causing a processor to perform the method of any of claims 18 to 20.

26. The method of claim 1, wherein the prediction mode attributes include intra mode, inter mode, merge mode, or motion information of a block at the mapped block location.

27. The method of claim 18, wherein the prediction mode attributes include intra mode, inter mode, merge mode, or motion information of a block at the mapped block location.

28. The apparatus of claim 10, wherein the prediction mode attributes include intra mode, inter mode, merge mode, or motion information of a block at the mapped block location.

29. The apparatus of claim 21, wherein the prediction mode attributes include intra mode, inter mode, merge mode, or motion information of a block at the mapped block location.

Images