Spherical projection motion estimation / compensation and mode decision

Abstract

Embodiments of the present disclosure relate to spherical projection motion estimation / compensation and mode decision. Techniques are disclosed for predictively encoding video data based on a prediction made from a spherical domain projection of an input picture to be encoded and a reference picture as a prediction candidate. A spherical projection of the input picture and the candidate reference picture can be generated. A search can then be performed for a match between a spherical domain representation of a pixel block to be encoded and a spherical domain representation of the reference picture. Upon a match, an offset between the spherical domain representation of the pixel block and the matching portion of the reference picture in the spherical domain representation can be determined. The spherical domain offset can be converted to a motion vector in a source domain representation of the input picture, and the pixel block can be predictively encoded with reference to the matching portion of the reference picture in the source domain representation.

Description

[0001] This application is a divisional application of the invention patent application filed on September 14, 2017, which entered the Chinese national phase on June 21, 2019, with Chinese national application number 201780079476.0 and entitled "Spherical Projection Motion Estimation / Compensation and Pattern Decision". Technical Field

[0002] This disclosure relates to encoding 360° video to achieve bandwidth compression, and more specifically, to techniques for processing 360° video using compression algorithms designed to process two-dimensional video data. Background Technology

[0003] The term "360° video" refers to video recording that simultaneously captures views from multiple (and sometimes all) directions from a camera. Omnidirectional cameras or camera arrays can be used to capture 360° video, capturing image data from different but overlapping fields of view, and their outputs are stitched together. Viewers of 360° video can be given control over the viewing direction during playback, allowing them to navigate within the video's field of view.

[0004] Although 360° video captures image information from three-dimensional space, the video data itself is typically represented by image data in a two-dimensional format. Image data is represented by an array of pixels arranged at predetermined spatial locations (e.g., x, y positions) in two dimensions. Furthermore, while objects at different depths within the field of view will be represented in image data whose size corresponds not only to the object's physical dimensions but also to its distance from the camera, the pixel data representing the object does not change pixel positions with depth.

[0005] The two-dimensional representation of three-dimensional space can lead to distortion of image data at different locations within the field of view. For example, a straight line in three-dimensional space may not appear as a straight line in two-dimensional image data. Furthermore, when different objects move within the field of view of a 360° image, their size and shape may become distorted. These variations pose challenges to processing systems that manipulate such image data.

[0006] For example, distortion can cause problems in video coding. Video encoders typically reduce the bandwidth of image signals by utilizing spatial and temporal redundancy in image data. However, video encoders operating on two-dimensional representations of three-dimensional images do not always detect such redundancy due to the distortion that can occur between frames. When such video encoders fail to detect redundancy in the content, they often generate coded representations of image data that are not as bandwidth-efficient as they are capable of achieving.

[0007] Therefore, the inventors believe that there is a need in the art for video coding systems that can better identify redundancy in two-dimensional representations of three-dimensional image content (such as 360° video). Attached Figure Description

[0008] Figure 1 A system for exchanging 360° video is shown.

[0009] Figure 2 This is a functional block diagram of a coding system according to one embodiment of the present disclosure.

[0010] Figure 3 An exemplary transformation that can be performed for a spherical projection is shown in one embodiment of this disclosure.

[0011] Figure 4 An exemplary transformation that can be performed for a spherical projection is shown in another embodiment of this disclosure.

[0012] Figure 5 An encoding method according to one embodiment of the present disclosure is shown.

[0013] Figure 6 An exemplary spherical projection that can be performed by one embodiment of this disclosure is shown.

[0014] Figure 7 This is a functional block diagram of a coding system according to one embodiment of the present disclosure.

[0015] Figure 8 This is a functional block diagram of a decoding system according to one embodiment of the present disclosure.

[0016] Figure 9 An encoding method according to one embodiment of the present disclosure is shown.

[0017] Figure 10 An exemplary spherical projection that can be performed by another embodiment of this disclosure is shown.

[0018] Figure 11 An exemplary spherical projection that can be performed by another embodiment of this disclosure is shown.

[0019] Figure 12 This is a functional block diagram of a decoding system according to another embodiment of this disclosure.

[0020] Figure 13 This is a functional block diagram of a decoding system according to another embodiment of this disclosure. Detailed Implementation

[0021] The embodiments of this disclosure provide a technique for predictively encoding video data based on predictions made from spherical domain projections of an input frame to be encoded and a reference frame serving as prediction candidates. Spherical projections of the input frame and candidate reference frames can be generated. A search can then be performed to find a match between the spherical domain representation of the pixel block to be encoded and the spherical domain representation of the reference frame. During matching, an offset between the spherical domain representation of the pixel block and the matching portion of the reference frame in the spherical domain representation can be determined. The spherical domain offset can be transformed into a motion vector in the source domain representation of the input frame, and the pixel block can be predictively encoded with reference to the source domain representation of the matching portion of the reference frame.

[0022] Figure 1 A system 100 employing embodiments of the present disclosure is illustrated. System 100 may include at least two terminals 110-120 interconnected via a network 130. A first terminal 110 may have a camera system 112 for capturing 360° video. Terminal 110 may also include an encoding system and a transmission system (not shown) to transmit an encoded representation of the 360° video to a second terminal 120, where the encoded representation can be consumed. For example, the second terminal 120 may display the 360° video on a local display, execute a video editing program to modify the 360° video, integrate the 360° video into an application (e.g., a virtual reality program), exist in a head-mounted display (e.g., a virtual reality application), or store the 360° video for later use.

[0023] Figure 1 Components suitable for unidirectional transmission of 360° video from a first terminal 110 to a second terminal 120 are shown. In some applications, it may be suitable to provide bidirectional exchange of video data, in which case the second terminal 120 may include its own camera system, video encoder, and transmitter (not shown), and the first terminal 100 may include its own receiver and display (also not shown). If bidirectional exchange of 360° video is desired, the techniques discussed below can be replicated to generate a pair of independent 360° video unidirectional exchanges. In other applications, it is permissible to transmit 360° video in one direction (e.g., from the first terminal 110 to the second terminal 120) and transmit “flat” video (e.g., video from a limited field of view) in the reverse direction.

[0024] exist Figure 1In this disclosure, the second terminal 120 is shown as a computer display, but the principles of this disclosure are not limited thereto. Embodiments of this disclosure can be applied to laptop computers, tablet computers, smartphones, servers, media players, virtual reality headsets, augmented reality displays, holographic displays, and / or dedicated video conferencing equipment. Network 130 represents any number of networks that transmit encoded video data between terminals 110-120, including, for example, wired communication networks and / or wireless communication networks. Communication network 130 can exchange data in circuit-switched channels and / or packet-switched channels. Representative networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For the purposes of this discussion, unless explained below, the architecture and topology of network 130 are irrelevant to the operation of this disclosure.

[0025] Figure 2 This is a functional block diagram of an encoding system 200 according to one embodiment of the present disclosure. System 200 may include a camera system 210, an image processing system 220, a video encoder 230, a video decoder 240, a reference image repository 250, a predictor 260, and a pair of spherical transformation units 270, 280. Camera system 210 can generate image data representing a local environment as a so-called “360° image,” which contains image data of the field of view extending in all directions around camera system 210. Image processing system 220 can transform image data from camera system 210 as needed to meet the requirements of video encoder 230. Video encoder 230 typically generates an encoded representation of its input image data by utilizing spatial and / or temporal redundancy in the image data. Video encoder 230 can output an encoded representation of the input data that consumes less bandwidth than the input data during transmission and / or storage.

[0026] Video decoder 240 can reverse the encoding operations performed by video encoder 230 to obtain a reconstructed image from the encoded video data. Typically, the encoding process applied by video encoder 230 is a lossy process, which results in various errors in the reconstructed image when compared to the original image. Video decoder 240 can reconstruct the image of a selected encoded image designated as a "reference image" and store the decoded reference image in reference image repository 250. In the absence of transmission errors, the decoded reference image will be copied from the decoder (…). Figure 2 The decoded reference image obtained (not shown in the image).

[0027] Predictor 260 can select a prediction reference for encoding a new input frame. For each portion of the input frame to be encoded (referred to as a "pixel block" for convenience), predictor 260 can select an encoding mode and identify a portion of the prediction reference search in the reference frame that can serve as the pixel block to be encoded. The encoding mode can be an intra-frame encoding mode, in which case the prediction reference can be drawn from a previously encoded (and decoded) portion of the input frame. Alternatively, the encoding mode can be an inter-frame encoding mode, in which case the prediction reference can be drawn from another previously encoded and decoded frame. In one embodiment, predictor 260 can search for a prediction reference for the encoded frame that has been operated on by the input frame and the reference frame, which have been transformed into spherical projection representations. Spherical transformation units 270, 280 can transform the input frame and the reference frame into spherical projection representations.

[0028] When an appropriate prediction reference is identified, predictor 260 can provide prediction data to video encoder 230 in a representation acceptable to video encoder 230. Typically, one or more reference frames stored in a reference frame repository will be in a format acceptable to the video encoder.

[0029] As noted, the encoded video data output by video encoder 230 should consume less bandwidth than the input data during transmission and / or storage. Encoding system 200 can output encoded video data to output device 290, such as a transmitter (not shown), which can cross communication network 130 ( Figure 1 Encoded video data may be transmitted via storage devices (not shown) such as electronic storage media, magnetic storage media, and / or optical storage media.

[0030] Figure 3 The first embodiment shows that it can be made by Figure 2 The exemplary transformations performed by the spherical transformation units 270 and 280. In this embodiment, the camera system 110 ( Figure 2 The device can perform a 360° capture operation 310 and output an equal-length rectangular frame 320 with a size of M×N pixels. The equal-length rectangular frame 320 can represent a 360° field of view divided along slices 312 that divide the cylindrical field of view into data. In the equal-length rectangular frame 320, pixels on either side of slice 312 represent adjacent image content, even if they appear on different edges of the equal-length rectangular frame 320.

[0031] The spherical transformation unit 270 can transform pixel data at position (x, y) within the equal-length rectangular frame 320 into position along the spherical projection 330 according to transformations such as the following.

[0032] θ = x + θ0 and (Equation 1)

[0033]

[0034] θ and Let θ0 and θ2 represent the longitude and latitude of a location in the spherical projection 330, respectively. x represents the origin of the spherical projection 330, and x and y represent the horizontal and vertical coordinates of the source data in the equal-length rectangular image 320.

[0035] When this transformation is applied, the spherical transformation unit 270 can transform each pixel position along a predetermined row of the equal-length rectangular image 320 to have a unique position at the equatorial latitude in the spherical projection 330. In such regions, each position in the spherical projection 330 can be assigned a pixel value from a corresponding position in the equal-length rectangular image 320. At other positions, particularly towards the poles of the spherical projection 330, the spherical projection unit 270 can map several source positions from the equal-length rectangular image 320 to a common position in the spherical projection 330. In such cases, the spherical projection unit 270 can derive the pixel value for a position in the spherical projection 330 from a mixture of corresponding pixel values ​​in the equal-length rectangular image 320 (e.g., by averaging the pixel values ​​at corresponding positions in the equal-length rectangular image 320).

[0036] Figure 4 It is shown that in another embodiment, it can be derived from Figure 2 The exemplary transformations performed by the spherical transformation units 270 and 280. In this embodiment, the camera system 110 ( Figure 2 The image capture 410 can perform a 360° capture operation and output an image 420 with a size of M×N pixels, wherein the image content is arranged according to a cube map. The image capture 410 can capture image data in each of a predetermined number of directions (typically six), which are stitched together according to the cube map layout. Figure 4 In the illustrated embodiment, six sub-images corresponding to the left view 411, front view 412, right view 413, rear view 414, top view 415, and bottom view 416 can be captured and arranged within the cube map screen 420 according to the "seams" of the image content between the corresponding views. Therefore, as... Figure 4 As shown, pixels from the front image that are adjacent to pixels from each of the top, left, right, and bottom images represent image content that is adjacent to the content of the adjacent sub-image, respectively. Similarly, pixels from the right and rear images that are adjacent to each other represent adjacent image content. Furthermore, the content of the terminal edge 422 from the rear image is adjacent to the content of the opposite terminal edge 424 from the left image. The cube map screen 420 may also have regions 426.1-426.4 that do not belong to any image.

[0037] The spherical transformation unit 270 can transform the pixel data at position (x,y) within the cube map frame 420 to the position along the spherical projection 330 based on the transformation derived from each sub-image in the cube map. Figure 4 The image capture 410, with six faces 411-416 superimposed on the spherical projection 430 to be generated, is shown. Each sub-image of the image capture corresponds to a predetermined angular region of the surface of the spherical projection 430. Thus, image data of the front face 412 can be projected onto a predetermined portion of the surface of the spherical projection, and image data of the left, right, back, top, and bottom sub-images can be projected onto corresponding portions of the surface of the spherical projection.

[0038] In a cube map with square sub-images (i.e., sub-images 411-416 have equal height and width), each sub-image is projected onto a 90°×90° region of the projection surface. Therefore, based on the form... and θ=g k The sinusoidal projection of (x,y) maps each position x,y of the sub-image to θ on the spherical projection 430. Position, where x, y represent the displacements of the top, bottom, front, right, left, and right sides from the center of the cube face k, and θ, This represents the angular deviation within the sphere.

[0039] When this transformation is applied, some pixel locations in the cube map screen 420 can be mapped to unique locations in the spherical projection 430. In such areas, each location in the spherical projection 430 can be assigned a pixel value from the corresponding location in the cube map screen 420. At other locations, particularly towards the edges of the corresponding sub-image, the spherical projection unit 270 can map image data from several source locations in the cube map screen 420 to common locations in the spherical projection 430. In such cases, the spherical projection unit 270 can derive pixel values ​​for locations in the spherical projection 430 from a mixture of corresponding pixel values ​​in the cube map screen 420 (e.g., by a weighted average of pixel values ​​at corresponding locations in the cube map screen 420).

[0040] The techniques disclosed herein can be applied to other types of image capture techniques. For example, image capture techniques based on truncated pyramids, tetrahedrons, octahedrons, dodecahedrons, and icosahedrons can be employed. Images obtained from these techniques can be mapped to spherical projections using similar techniques.

[0041] return Figure 2Predictor 260 can perform a prediction search using image data obtained through spherical projection. Therefore, spherical transformation unit 270 can transform image data captured by the imaging system into a spherical projection, and second spherical transformation unit 280 can transform candidate reference images into other spherical projections. Predictor 260 can search for prediction reference data to be used by video encoder 230 within the spherical domain. Once a suitable prediction match is identified, predictor 260 provides prediction data from reference image repository 250 in a format acceptable to video encoder 230 (i.e., without transforming it into a spherical projection).

[0042] Figure 5 An encoding method 500 according to one embodiment of the present disclosure is illustrated. Method 500 can operate on a pixel-by-pixel block basis to encode a new input image to be encoded. Method 500 can begin by transforming the data of the input pixel block into a spherical representation (box 510). Method 500 can also transform candidate reference images into spherical representations (box 520). Method 500 can then perform a prediction search by comparing the transformed pixel block data with the transformed reference image data (box 530). When a suitable prediction reference is found, method 500 can perform differential encoding on the input pixel block using the matching reference image data (referred to as the "reference block" for convenience) as the basis for prediction (box 540). Typically, this differential encoding includes calculating the pixel residual by pixel-by-pixel subtraction of the prediction block data from the input pixel block data (box 542) and transformation, quantization, and entropy encoding of the resulting pixel residual (box 544). In this respect, method 500 may follow coding protocols defined by mainstream coding standards, such as ITU H.265 (also known as “HEVC”), H.264 (also known as “AVC”), or previous coding standards. These standards define the protocols used to define pixel blocks, define the search window for the prediction reference, and perform differential coding of pixel blocks with reference to the reference block. Method 500 may also transform the spherical domain representation of motion vectors into an encoder domain representation used by the video coding standard (box 546). Method 500 may output coded pixel residuals, motion vectors, and other metadata associated with the prediction (typically a coding mode indicator and a reference frame ID) (box 548).

[0043] A predictive search (box 530) can be performed to maximize bandwidth savings and minimize information loss. Method 500 can perform operations to estimate when a suitable predictive benchmark is found. In one implementation, for each input pixel block, method 500 can rotate the spherical projection of the benchmark frame around a plurality of candidate rotations relative to the transformed input pixel block (box 532). At each candidate rotation, the method can estimate the predictive residual that will be obtained when using the candidate rotation (box 534). These calculations can be performed by comparing the spherically projected input pixel block with the portion of the rotated candidate benchmark frame that is aligned with the position of the input pixel block. Typically, when comparisons generate pixel residuals with high magnitude and high variance, it will result in reduced coding efficiency compared to comparisons of other candidate pixel blocks that generate pixel residuals with lower magnitude and lower variance. Method 500 can also estimate the coding distortion that will occur when using the candidate benchmark block (box 536). These calculations can be performed again in the spherical projection domain by estimating the loss of the pixel residual based on the level of quantization parameters to be applied to the input pixel block. Once estimates of all candidate reference blocks under consideration are obtained, method 500 selects the reference pixel block (box 538) that minimizes the total coding cost.

[0044] For example, when the reference frame is rotated by angles α, β, γ, the candidate "reference block" BLK that is aligned with the position of the input pixel block is referenced. α,β,γ The encoding cost J of the input pixel block can be given as:

[0045] J = Bits(BLK) α,β,γ )+k*DIST(BLK α,β,γ ), where (Equation 3)

[0046] Bits(BLK α,β,γ ) indicates the estimated reference block BLK α,β,γ The number of bits required to encode the input pixel block, and DIST(BLK) α,β,γ ) indicates the reference block BLK α,β,γ The distortion obtained by encoding the input pixel block, k can be a scalar chosen by the operator to balance the contributions of these factors. As explained, method 500 can be performed to select the reference pixel block that minimizes the value J.

[0047] Figure 6 Symbolically shown according to Figure 5 The implementation scheme involves a prediction operation between the spherical protection of input frame 610 and the spherical projection of reference frame 620. As discussed, the two-dimensional representation of the input frame can be parsed into multiple pixel blocks to be encoded. These pixel blocks can have corresponding projections in the spherical projection of input frame 610, such as... Figure 6As shown. The size and shape of the blocks can vary depending on their position in the spherical projection. For example, in the two-dimensional representation, pixel blocks positioned toward the top and bottom edges of the image can be projected into the shape of triangles in the spherical projection of the input frame 610, and the polar regions toward the spherical projection can be positioned. Pixel blocks positioned along the rows in the middle of the two-dimensional representation can be projected into a generally rectangular shape, and the equatorial region toward the spherical projection can be positioned. Pixel blocks from the two-dimensional representation mapped to the intermediate position between the equatorial and polar regions can have a generally trapezoidal shape. This can be determined according to... Figure 5 Method 500 considers all such representations of pixel blocks projected onto a spherical surface, even if their shapes vary between pixel blocks.

[0048] Figure 6 An exemplary rotation of the spherical projection of reference frame 620 is also shown. Here, the axis 622 of the spherical projection of reference frame 620 is shown as a rotation angle α, β, γ from its initial position. When encoding pixel blocks 614, 616 of the input frame, Figure 5 Method 500 performs a pixel-by-pixel comparison between pixels in the spherical projection of the input frame and pixels in a reference block (shown as blocks 624 and 626, respectively) coexisting in the spherical projection of the reference frame 620. Here, again, the reference block can vary in size and shape at different positions in the spherical projection of the reference frame 620.

[0049] When selecting a prediction reference for an input pixel block, the rotation angles α, β, γ corresponding to the selected prediction reference can be converted into motion vectors in the two-dimensional space of the reference image. These motion vectors can be transmitted as part of the encoded video data of the input pixel block.

[0050] Motion vectors used in many encoding protocols such as HEVC and AVC are limited to describing spatial displacements (e.g., in the x and y directions) in a two-dimensional domain of the input and reference frames. It is possible that a given rotation angle α, β, γ for the prediction search is mapped to a spatial location in the two-dimensional domain that is both shifted in the x and y directions and rotated relative to the source pixel block. In one embodiment, if it is desired to apply the principles of this disclosure to such video encoders, the input pixel block can be encoded using a prediction reference uniquely identified by the x and y displacements obtained by transforming the motion vectors from the spherical domain to the two-dimensional domain. In other words, if the rotation angles α, β, γ in the spherical domain are transformed into motion vectors of the form Δx, Δy, and λ (where λ represents the rotation of the pixel block in two-dimensional space), the video encoder can perform prediction using a prediction reference selected by the motion vectors of the form Δx, Δy, where λ is ignored. However, better performance is expected where the motion vectors used for prediction can be fully represented, for example, in the format Δx, Δy, λ or α, β, γ; these alternative embodiments are discussed below.

[0051] Many coding applications perform motion estimation at a granularity smaller than a single pixel in a reference image. For example, in HEVC and H.264 protocols, video encoders perform motion estimation at quarter-pixel and / or half-pixel increments. In one implementation, the video encoder may perform spatial interpolation to develop image data sufficient to perform motion estimation at these smaller granularities. For example, the video encoder may perform interpolation to find a reference pixel that matches the source pixel at a finer rotation angle. In this way, the rotation data of a spherical projection can contain enough information to perform predictive searches at such a granularity.

[0052] The aforementioned process can also be applied to intra-frame coding. When encoding an input pixel block, it can be encoded by referring to previously encoded image data from the same frame containing the pixel block. Therefore, referencing... Figure 6 Before encoding pixel block 616, the encoded data of several other pixel blocks (including block 614) will be encoded and transmitted to the decoder. Both the encoder and decoder will have the decoded data of those pixel blocks, which can be used as a prediction source for pixel block 616. According to one embodiment, the video encoder can identify prediction references based on intra-frame coding by rotating the encoded image data in various arrangements of α, β, γ, thereby searching for appropriate prediction data within the previously encoded and decoded image data of the current frame. Furthermore, the intra-frame prediction direction in 2D can be reflected in the sphere of 610 because block 616 is transformed into a wedge shape. For example, the vertical direction predicted in a 2D frame can be mapped accordingly in 610 using interpolation. The video encoder can estimate the coding cost J of the candidate prediction references according to the technique of Equation 3.

[0053] Figure 7This is a functional block diagram of an encoding system 700 according to one embodiment of the present disclosure. System 700 may include a pixel block encoder 710, a pixel block decoder 720, a loop filter system 730, a reference image repository 740, a pair of spherical transformation units 750 and 760, a predictor 770, a controller 780, and a syntax unit 790. The pixel block encoder and decoder 710, 720 and the predictor 770 operate iteratively on individual pixel blocks of an image. The predictor 770 predicts data to be used during the encoding of newly presented input pixel blocks. The pixel block encoder 710 encodes new pixel blocks using predictive coding techniques and presents the encoded pixel block data to the syntax unit 790. The pixel block decoder 720 decodes the encoded pixel block data, thereby producing decoded pixel block data. The loop filter 730 can perform various filtering operations on the decoded image obtained from the combination of decoded pixel blocks obtained by the pixel block decoder 720. The filtered image can be stored in the reference image repository 740, which can be used as a source for predicting pixel blocks received later. Syntax unit 790 can combine data streams from coded pixel block data conforming to a management encoding protocol.

[0054] The pixel block encoder 710 may include a subtractor 712, a transform unit 714, a quantizer 716, and an entropy encoder 718. The pixel block encoder 710 may accept pixel blocks of input data at the subtractor 712. The subtractor 712 may receive predicted pixel blocks from the predictor 770 and thereby generate a pixel residual array representing the difference between the input pixel blocks and the predicted pixel blocks. The transform unit 714 may apply a transform to the sample data output from the subtractor 712 to convert the data from the pixel domain to the transform coefficient domain. The quantizer 716 may perform quantization on the transform coefficients output by the transform unit 714. The quantizer 716 may be a uniform or non-uniform quantizer. The entropy encoder 718 may reduce the output bandwidth of the coefficient quantizer by, for example, encoding the output with variable-length codewords.

[0055] Transform unit 714 can operate in various transform modes determined by controller 780. For example, transform unit 714 can apply Discrete Cosine Transform (DCT), Discrete Sine Transform (DST), Walsh-Hadamard Transform, Haar Transform, Daubechies Wavelet Transform, etc. In one embodiment, controller 780 can select the encoding mode M to be applied by transform unit 715, configure transform unit 715 accordingly, and explicitly or implicitly signal the encoding mode M in the encoded video data.

[0056] Quantizer 716 can quantize according to the quantization parameter Q provided by controller 780. P To operate. In one implementation, the quantization parameter Q PIt can be applied as a multi-valued quantization parameter to the transform coefficients, and the multi-valued quantization parameter can be changed at different coefficient positions within a pixel block in the transform domain. Therefore, the quantization parameter Q P It can be provided as an array of quantized parameters.

[0057] The pixel block decoder 720 can invert the encoding operation of the pixel block encoder 710. For example, the pixel block decoder 720 may include a dequantizer 722, an inverse transform unit 724, and an adder 726. The pixel block decoder 720 can obtain its input data from the output of the quantizer 716. Although permissible, the pixel block decoder 720 does not need to perform entropy decoding of the entropy-encoded data, because entropy encoding is a lossless process. The dequantizer 722 can invert the operation of the quantizer 716 of the pixel block encoder 710. The dequantizer 722 can perform actions such as decoding the signal Q. P The specified uniform or non-uniform dequantization. Similarly, the inverse transform unit 724 can reverse the operation of the transform unit 714. The dequantizer 722 and the inverse transform unit 724 can use the same quantization parameter Q as their corresponding components in the pixel block encoder 710. P And transform mode M. Quantization operations may truncate data in various aspects, and therefore, the data recovered by dequantizer 722 may have encoding errors when compared with the data presented to quantizer 716 in pixel block encoder 710.

[0058] Adder 726 can reverse the operation performed by subtractor 712. It can receive the same predicted pixel block from predictor 770 used by subtractor 712 in generating the residual signal. Adder 726 can add the predicted pixel block to the reconstructed residual value output by inverse transform unit 724 and can output the reconstructed pixel block data.

[0059] The loop filter 730 can perform various filtering operations on the recovered pixel block data. For example, the loop filter 730 may include a deblocking filter 732 and a Sample Adaptive Offset (“SAO”) filter 733. The deblocking filter 732 can filter the data at the seams between the reconstructed pixel blocks to reduce discontinuities between pixel blocks due to encoding. The SAO filter can, for example, add an offset to the pixel value based on the edge direction / shape and / or pixel / color component level, according to the SAO “type”. The loop filter 730 can operate according to parameters selected by the controller 780.

[0060] The reference frame repository 740 can store filtered pixel data for use in subsequent predictions of other pixel blocks. This makes different types of prediction data available to the predictor 770 for different prediction modes. For example, for an input pixel block, intra-frame prediction obtains a prediction reference from decoded data of the same frame containing the input pixel block. Therefore, the reference frame repository 740 can store decoded pixel block data for each frame during encoding. For the same input pixel block, inter-frame prediction can obtain a prediction reference from previously encoded and decoded frames designated as reference frames. Therefore, the reference frame repository 740 can store these decoded reference frames.

[0061] Spherical transformation units 750 and 760 perform transformations of image data to a spherical projection representation. The first spherical transformation unit 750 performs its transformation on candidate prediction reference data from a reference frame repository, regardless of whether it is used for intra-frame or inter-frame prediction. The second spherical transformation unit 760 performs its transformation on the input video data as it is presented to the pixel block encoder 710. Spherical transformation units 750 and 760 each output their transformed data to the predictor 770.

[0062] As discussed, predictor 770 can provide prediction data to pixel block encoder 710 for use in generating residuals. Predictor 770 may include inter-frame predictor 772, intra-frame predictor 773, and mode decision unit 774. Inter-frame predictor 772 can receive spherical projection pixel block data representing new pixel blocks to be encoded, and can search a repository 740 for spherical projections of reference frame data for pixel block data from one or more reference frames to encode the input pixel blocks. Inter-frame predictor 772 can support multiple prediction modes, such as P-mode encoding and B-mode encoding. Inter-frame predictor 772 can select an inter-frame prediction mode and provide identifiers of candidate prediction reference data that best matches the input pixel block to be encoded. Inter-frame predictor 772 can generate prediction reference metadata, such as motion vectors, to identify which parts(s) of which reference frames are selected as prediction sources for the input pixel blocks.

[0063] The intra predictor 773 can support intra (I) mode coding. The intra predictor 773 can search from spherically projected pixel block data from the same frame as the pixel block being encoded, providing the closest match to the spherically projected input pixel block. The intra predictor 773 can also generate a prediction reference indicator to identify which portion of the frame is selected as the prediction source for the input pixel block.

[0064] The mode decision unit 774 can select the final encoding mode to be applied to the input pixel block. Typically, as described above, given a target bit rate, the mode decision unit 774 selects the prediction mode that will achieve the lowest distortion when decoding the video. Exceptions may occur when the encoding mode is selected to satisfy other strategies followed by the encoding system 700, such as satisfying specific channel behavior or supporting random access or data refresh strategies. When selecting the final encoding mode, the mode decision unit 774 can output a reference block of aspherical projection from the repository 740 to the pixel block encoder and decoder 710, 720, and can provide the controller 780 with an identifier of the selected prediction mode and a prediction reference indicator corresponding to the selected mode.

[0065] The controller 780 controls the entire operation of the encoding system 700. The controller 780 can select the operating parameters of the pixel block encoder 710 and predictor 770 based on analysis of the input pixel blocks and external constraints (such as encoding bit rate targets and other operating parameters). Related to this discussion, in selecting the quantization parameter Q... P When using a uniform or non-uniform quantizer and / or transform mode M, those parameters can be provided to the syntax unit 790, which can include the data representing those parameters in the data stream of the encoded video data output by the system 700.

[0066] During operation, the controller 780 can modify the operating parameters of the quantizer 716 and the transform unit 715 at different granularities of the image data, either on a per-pixel block basis or at a larger granularity (e.g., per frame, per slice, per maximum coding unit (“LCU”), or another region). In one embodiment, the quantization parameters can be modified on a per-pixel basis within the encoded frame.

[0067] Additionally, as discussed, controller 780 can control the operation of loop filter 730 and prediction unit 770. For prediction unit 770, such control may include mode selection (λ, test mode, search window, distortion strategy, etc.), and for loop filter 730, it may include selecting filter parameters, reordering parameters, weighted prediction, etc.

[0068] Figure 8 This is a functional block diagram of a decoding system 800 according to one embodiment of the present disclosure. The decoding system 800 may include a syntax unit 810, a pixel block decoder 820, a loop filter 830, a reference frame repository 840, a predictor 850, and a controller 860. The syntax unit 810 can receive encoded video data streams and can parse the encoded data into its components. Data representing encoding parameters can be provided to the controller 860, while data representing encoding residuals (…) Figure 2The data output by the pixel block encoder 210 can be provided to the pixel block decoder 820. The pixel block decoder 820 can reverse the encoding operation provided by the pixel block encoder. Figure 2 The loop filter 830 filters the reconstructed pixel block data. The reconstructed pixel block data can be combined into a picture for display and output from the decoding system 800 as output video. The picture can also be stored in the prediction buffer 840 for use in the prediction operation. The predictor 850 can provide prediction data to the pixel block decoder 820, the prediction data being determined by the encoded data received in the encoded video data stream.

[0069] The pixel block decoder 820 may include an entropy decoder 822, a dequantizer 824, an inverse transform unit 826, and an adder 828. The entropy decoder 822 can perform entropy decoding to reverse the process performed by the entropy encoder 718. Figure 8 The operation of quantizer 716 in dequantizer 824 of invertible pixel block encoder 710 ( Figure 7 Similarly, the inverse transform unit 826 can reverse the operation of the transform unit 714. Figure 7 They can use the quantization parameter Q provided in the encoded video data stream. P and transform mode M. Since quantization may truncate the data, the data recovered by dequantizer 824 is compared with the input data of its corresponding quantizer 716 presented to pixel block encoder 210. Figure 2 There may be encoding errors when making comparisons.

[0070] The adder 828 can reverse the operation performed by the subtractor 712. Figure 7 It can receive predicted pixel blocks from predictor 850, which are determined by a prediction reference in the encoded video data stream. Adder 828 can add the predicted pixel blocks to the reconstruction residual value output by inverse transform unit 826, and can output the reconstructed pixel block data.

[0071] The loop filter 830 can perform various filtering operations on the reconstructed pixel block data. As shown, the loop filter 830 may include a deblocking filter 832 and a SAO filter 834. The deblocking filter 832 can filter the data at the seams between reconstructed pixel blocks to reduce discontinuities between pixel blocks due to encoding. The SAO filter 834 can add an offset to the pixel value based on the SAO type, such as based on edge direction / shape and / or pixel level. Other types of loop filters can also be used in a similar manner. The operation of the deblocking filter 832 and the SAO filter 834 ideally replicates the operation of their corresponding parts in the analog encoding system 700. Figure 7Therefore, in the absence of transmission errors or other anomalies, the decoded image obtained from the loop filter 830 of the decoding system 800 will be identical to the decoded image obtained from the loop filter 730 of the encoding system 700. Figure 7 The encoding system 700 and the decoding system 800 should store the same set of reference images in their respective reference image storage libraries 740 and 840.

[0072] The reference frame repository 840 can store filtered pixel data for use in subsequent predictions of other pixel blocks. The reference frame repository 840 can also store decoded pixel block data from each frame during encoding for use in intra-frame prediction. The reference frame repository 840 can also store decoded reference frames.

[0073] As discussed, predictor 850 can provide prediction data to pixel block decoder 820. Predictor 850 can provide predicted pixel block data determined by prediction reference indicators provided in the encoded video data stream.

[0074] The controller 860 can control the entire operation of the encoding system 800. The controller 860 can set the operating parameters of the pixel block decoder 820 and the predictor 850 based on parameters received in the encoded video data stream. For the purposes of this discussion, these operating parameters may include the quantization parameter Q of the dequantizer 824. P The transformation mode M of the inverse transformation unit 815. As discussed, the received parameters can be set according to various granularities of the image data, such as per pixel block, per frame, per slice, per LCU, or based on other types of regions defined for the input image.

[0075] In one implementation, the encoder uses a spherical transform during encoding to select a prediction benchmark, eliminating the need for the decoder to use such a transform. Figure 2 and Figure 7 In the illustrated implementation, encoders 200 and 700 may perform predictive search with reference to spherical projection input data and reference data, but differential video encoders 230 and 710 may themselves perform the search using aspherical projection data. In this way, the implementation of this disclosure can be used in conjunction with decoders that do not perform spherical projection.

[0076] Figure 9An encoding method 900 according to one embodiment of the present disclosure is illustrated. Method 900 can operate on pixel-by-pixel blocks to encode a new input image to be encoded. Method 900 can begin by transforming the data of the input pixel blocks into a spherical representation (box 910). Method 900 can also transform candidate reference image data into a spherical representation (box 920). Then, method 900 can perform a prediction search by comparing the transformed pixel block data with the transformed reference image data (box 930). When a suitable prediction reference is found, method 900 can perform differential encoding on the input pixel blocks using the matching reference block as the basis for prediction (box 940). Typically, this differential encoding includes pixel-by-pixel subtraction from the prediction block data of the input pixel block data (box 942) and transformation, quantization, and entropy encoding of the resulting pixel residuals (box 944). Method 900 can represent motion vector data in α, β, γ format (box 946), where α, β, and γ represent the rotation of the spherical projection reference image from its initial axis, respectively. Figure 6 Method 900 can output motion vectors as well as encoded residuals and other metadata of the prediction (typically encoding mode indicators, reference frame IDs) (box 948).

[0077] In another implementation, method 900 may represent the α, β, γ format differently relative to the global rotation value estimated for the reference frame. In such an implementation, method 900 may estimate the global rotation difference between the reference frame and the input frame (box 940). For example, method 900 may execute input frame 610 ( Figure 6 The method involves an overall comparison between the spherical projection of the input frame 610 and the spherical projection of the reference frame 620. This method generates a first rotation parameter, called "global rotation," which reflects the detected rotation between the input frame 610 and the reference frame 620. The global rotation can be encoded as a first vector α,β,γ. During the encoding of the individual spherical projection pixel blocks 614, 616, the reference frame can be further rotated in an attempt to find a rotation with a lower encoding cost J than that obtained when selecting the reference block using only the global rotation value. If such a rotation is identified, the motion vectors of the pixel blocks 614, 616 can be differentially encoded relative to the global rotation vectors α,β,γ in the format Δa, Δβ, and Δγ. In one embodiment, the global vectors can be included in the encoded video data within syntax elements that appear at a higher level than the encoded pixel block data, such as in a frame title or slice header. Furthermore, when encoding a single input frame with reference to multiple reference frames, such headers can include fields for identifying the reference frames (e.g., by frame ID) and global rotation vectors α,β,γ applied to each of them.

[0078] Figure 9 The encoding method 900 can be provided by a video encoder, such as Figure 7 The video encoder shown is used for execution. However, in this embodiment, when the predictor 770 outputs a spherical domain motion vector to the controller 780, the controller does not need to convert the motion vector to an encoder domain representation. The controller 780 can perform estimation of the rotation difference between the input frame and the reference frame (input not shown), and can derive the appropriate motion vector representation as described above. The controller 780 can also output its estimated rotation difference to the channel.

[0079] Similarly, decoding can be performed by a video decoder, such as Figure 8 The video decoder shown performs this operation. However, in this embodiment, the controller 860 may perform a conversion operation to convert the motion vector from its differential representation (ΔMV) to its absolute representation (MV), and further convert the absolute representation of the motion vector from a spherical domain representation to an encoder domain representation. The predictor 850 may then retrieve appropriate reference blocks from the reference image repository 840 and provide them to the pixel block decoder 820.

[0080] In another implementation, spherical projection can assign a variable radius to portions of image content from an input or reference frame. This variable radius can accommodate object movement or camera movement that causes the image content to resize between the input and reference frames. For example, as an object moves toward or away from the camera, the size of the object within the camera's field of view can increase or decrease accordingly. Similarly, as the camera moves, some objects move closer to the camera, in which case their size increases, and other objects move away from the camera, in which case their size decreases. In either case, the video encoder can compensate for such changes using different radii between spherical projections.

[0081] Figure 10 An exemplary use case is illustrated, in which the input screen 1010 to be encoded is transformed into a spherical projection 1012 on a sphere having a radius R. The input screen 1010 can be predictively encoded using a reference screen 1020 as a prediction reference, which has its own spherical projection 1022. For convenience, Figure 10 Only a portion of the spherical projection used for the input and reference images is shown. Figure 10 In this context, the spherical projection of the reference frame is also shown as having a radius R. In one embodiment, the video encoder can change the radii of the spherical projections 1012 and 1022 of the input and reference frames based on the relative size differences detected between objects therein, in order to detect the correlation between them. When a correlation is found, the video encoder can derive the relative ratio of the radii in the spherical projections 1012 and 1022 (shown as R). I / R PThis reflects the size adjustment between elements of the image content in the two projections 1012 and 1022. In this way, the spherical projection (shown as projection 1024) of one of the images (here, the reference image 1020) can be changed to accommodate alternating radii R. P Then, the size ratio R can be adjusted. I / R P Along with other components of the motion vector derived from the previous implementation, the reference frame is mapped to a reference block 1026, which can be resized and used as the basis for predicting pixel blocks 1014 in the input frame 1010.

[0082] The radius variation between spherical projections can be re-executed for each encoded pixel block or at other encoding granularities. Therefore, if needed, the video encoder can change the radius of the spherical projection of the reference frame frame by frame, by patch, or by slice.

[0083] In another implementation, the spherical projection can assign a variable spatial position to the origin of the spherical projection of the image content from the input and / or reference frames. This variable origin can accommodate object movement or camera movement that causes the image content to resize between the input and reference frames. As discussed, when the camera moves, some objects move closer to the camera, in which case their size increases, while other objects move further away from the camera, in which case their size decreases. In this case, the video encoder can compensate for such changes by assigning a different position to the origin of the spherical projection.

[0084] Figure 11 An exemplary use case is illustrated, in which the input screen 1110 to be encoded is transformed into a spherical projection 1112 on a sphere having a radius R. The input screen 1110 can be predictively encoded using a reference screen 1120 as a prediction reference, which has its own spherical projection 1122. For convenience, Figure 11 Only a portion of the spherical projection used for the input and reference images is shown. Figure 11In this context, the spherical projection of the reference frame is also shown as having a radius R. In one embodiment, the video encoder can change the origin position of the radius R of the spherical projections 1112, 1122 of the input and reference frames based on the relative size differences detected between objects therein, in order to detect the correlation between them. When a correlation is found, the video encoder can derive the relative offset between the spherical projections 1112, 1122, which is shown as Δx, Δy, Δz, reflecting the shift between the two projections 1112, 1122. The origin offset Δx, Δy, Δz, along with other components of the motion vector derived from the previous embodiment, can then be mapped onto the reference frame, which identifies the reference block 1124 in the reference frame that can be resized and used as the basis for predicting pixel block 1114 in the input frame 1110.

[0085] The origin offsets Δx, Δy, and Δz between spherical projections can be re-executed for each encoded pixel block or at other encoding granularities. Therefore, if needed, the video encoder can change the origin offset of the reference frame's spherical projection frame-by-frame, tile-by-tile, or slice-by-slice.

[0086] In another implementation, it can be combined Figure 9 , Figure 10 and Figure 11 The technology provides multiple degrees of freedom for video encoders and decoders to identify differences between the spherical projections of the input and reference images. Therefore, the video encoder can provide motion vectors that reflect, for example, offsets relative to a global rotation vector (which can be represented by angular vectors α, β, γ) and radial ratio R. I / R P It is represented by and / or the offset of the origin Δx, Δy, Δz.

[0087] Figure 12 This is a functional block diagram of a decoding system 1200 according to another embodiment of this disclosure. Figure 12 In the implementation scheme, the decoding system 1200 can operate in the domain of spherical projection, such as Figures 3-4 , Figure 6 and Figures 10-11 As symbolically shown, the decoded image data is output from the decoding system 1200. The decoded image data can be converted to a two-dimensional domain when it is to be output from the decoding system 1200.

[0088] The decoding system 1200 may include a syntax unit 1210, a pixel block decoder 1220, a loop filter 1230, a reference image storage unit 1240, a predictor 1250, a controller 1260, an inverse spherical transform unit 1270, and an inverse spherical transform unit 1280. The syntax unit 1210 can receive encoded video data streams and can parse the encoded data into its components. Data representing encoding parameters can be provided to the controller 1260, while data representing encoded residuals (…) Figure 2 The data output by the pixel block encoder 210 can be provided to the pixel block decoder 1220. The pixel block decoder 1220 can reverse the encoding operation provided by the pixel block encoder. Figure 2 The loop filter 1230 filters the reconstructed pixel block data. The reconstructed pixel block data can be combined into a picture for display and output from the decoding system 1200 as output video. The picture can also be stored in the prediction buffer 1240 for use in the prediction operation. The predictor 1250 can provide prediction data to the pixel block decoder 1220, the prediction data being determined by the encoded data received in the encoded video data stream. The spherical transformation unit 1270 can transform data from the reference picture storage 1240 to the spherical domain and provide the transformed data to the predictor 1250. The inverse spherical transformation unit 1280 can transform the prediction data from the spherical domain back to the domain of the pixel block encoder 1220 and provide the transformed data to the adder 1228 therein.

[0089] The pixel block decoder 1220 may include an entropy decoder 1222, a dequantizer 1224, an inverse transform unit 1226, and an adder 12212. The entropy decoder 1222 can perform entropy decoding to reverse the process performed by the entropy encoder 7112. Figure 12 The operation of quantizer 716 in dequantizer 1224 of invertible pixel block encoder 710 ( Figure 7 Similarly, the inverse transform unit 1226 can reverse the operation of the transform unit 714. Figure 7 They can use the quantization parameter Q provided in the encoded video data stream. P and transform mode M. Since quantization may truncate the data, the data recovered by dequantizer 1224 is compared with the input data of its corresponding quantizer 716 presented to pixel block encoder 210. Figure 2 There may be encoding errors when making comparisons.

[0090] The adder 1228 can reverse the operation performed by the subtractor 712. Figure 7It can receive predicted pixel blocks from predictor 1250, which are determined by a prediction reference in the encoded video data stream. Adder 1228 can add the predicted pixel blocks to the reconstruction residual value output by inverse transform unit 1226, and can output the reconstructed pixel block data.

[0091] The loop filter 1230 can perform various filtering operations on the reconstructed pixel block data. As shown, the loop filter 1230 may include a deblocking filter 1232 and a SAO filter 1234. The deblocking filter 1232 can filter the data at the seams between reconstructed pixel blocks to reduce discontinuities between pixel blocks due to encoding. The SAO filter 1234 can add an offset to the pixel value based on the SAO type, such as based on edge direction / shape and / or pixel level. Other types of loop filters can also be used in a similar manner. The operation of the deblocking filter 1232 and the SAO filter 1234 ideally replicates the operation of their corresponding parts in the analog encoding system 700. Figure 7 Therefore, in the absence of transmission errors or other anomalies, the decoded image obtained from the loop filter 1230 of the decoding system 1200 will be identical to the decoded image obtained from the loop filter 730 of the encoding system 700. Figure 7 The encoding system 700 and the decoding system 1200 should store the same set of reference images in their respective reference image storage libraries 740 and 1240.

[0092] The reference frame repository 1240 can store filtered pixel data for use in subsequent predictions of other pixel blocks. The reference frame repository 1240 can also store decoded pixel block data from each frame during encoding for use in intra-frame prediction. The reference frame repository 1240 can also store decoded reference frames.

[0093] As discussed, predictor 1250 can provide prediction data to pixel block decoder 1220. Predictor 1250 can provide predicted pixel block data determined by prediction reference indicators provided in the encoded video data stream.

[0094] The controller 1260 can control the entire operation of the encoding system 1200. The controller 1260 can set the operating parameters of the pixel block decoder 1220 and the predictor 1250 based on parameters received in the encoded video data stream. For the purposes of this discussion, these operating parameters may include the quantization parameter Q of the dequantizer 1224. PThe transformation mode M of the inverse transformation unit 1215. As discussed, the received parameters can be set according to various granularities of the image data, for example, on a per-pixel block basis, on a per-frame basis, on a per-slice basis, on a per-LCU basis, or based on other types of regions defined for the input image.

[0095] exist Figure 12 In the implementation scheme, the pixel block decoder 1220, the loop filter 1230, and the reference image repository 1240 can operate on image data in the two-dimensional domain. However, the predictor 1250 can be based on the above reference. Figure 6 , Figure 10 and / or Figure 11 The described technique operates on the motion vectors of the identified reference blocks, thereby identifying the reference pixel blocks via rotation vectors (α, β, γ), the ratio of projection radii (Ri / Rp), and / or the offset (Δx, Δy, Δz) between the origins of the spherical projection. In response, predictor 1250 may retrieve a reference image from reference image repository 1240, and spherical transformation 1270 may transform the retrieved image to a spherical domain. Predictor 1250 may align the spherical projection of the reference image with the input pixel blocks based on the received motion vectors (e.g., by rotating the spherical projection of the reference image according to rotation vectors (α, β, γ), by adjusting the size of the spherical projection of the reference image according to the ratio of projection radii (Ri / Rp), and / or by shifting the origin of the spherical projection of the reference image according to the origin offset (Δx, Δy, Δz)). After this processing, a portion of the reference image aligned with the position of the encoded pixel block being decoded can be output as a reference block to adder 1228.

[0096] The inverse spherical inverse transform unit 1280 can transform the data of the reference block to the two-dimensional domain. The inverse transform unit can reverse the operations described in paragraphs 29 to 36 above. The transformed reference block can be output as a prediction block to the adder 1228 of the pixel block decoder 1220.

[0097] Figure 13 This is a functional block diagram of a decoding system 1300 according to another embodiment of this disclosure. Figure 13 In the implementation scheme, the decoding system 1300 can operate in the domain of a spherical projection, such as Figures 3-4 , Figure 6 and Figures 10-11 As symbolically shown, the decoded image data is output from the decoding system 1300. The decoded image data can be converted to a two-dimensional domain when it is to be output from the decoding system 1300.

[0098] The decoding system 1300 may include a syntax unit 1310, a pixel block decoder 1320, a loop filter 1330, a reference frame repository 1340, a predictor 1350, a controller 1360, and an inverse spherical transform unit 1370. The syntax unit 1310 can receive encoded video data streams and can parse the encoded data into its components. Data representing encoding parameters can be provided to the controller 1360, while data representing encoding residuals (…) Figure 2 The data output by the pixel block encoder 210 can be provided to the pixel block decoder 1320. The pixel block decoder 1320 can reverse the encoding operation provided by the pixel block encoder. Figure 2 The loop filter 1330 filters the reconstructed pixel block data. The reconstructed pixel block data can be combined into a picture for display and output from the decoding system 1300 as output video. The picture can also be stored in the prediction buffer 1340 for use in the prediction operation. The predictor 1350 can provide prediction data to the pixel block decoder 1320, the prediction data being determined by the encoded data received in the encoded video data stream.

[0099] The pixel block decoder 1320 may include an entropy decoder 1322, a dequantizer 1324, an inverse transform unit 1326, and an adder 13213. The entropy decoder 1322 can perform entropy decoding to reverse the process performed by the entropy encoder 7113. Figure 13 The operation of quantizer 716 in dequantizer 1324 of invertible pixel block encoder 710 ( Figure 7 Similarly, the inverse transform unit 1326 can reverse the operation of the transform unit 714. Figure 7 They can use the quantization parameter Q provided in the encoded video data stream. P and transform mode M. Since quantization may truncate the data, the data recovered by dequantizer 1324 is compared with the input data of its corresponding quantizer 716 presented to pixel block encoder 210. Figure 2 There may be encoding errors when making comparisons.

[0100] Adder 1328 can reverse the operation performed by subtractor 713. Figure 7 It can receive predicted pixel blocks from predictor 1350, which are determined by a prediction reference in the encoded video data stream. Adder 1328 can add the predicted pixel blocks to the reconstruction residual value output by inverse transform unit 1326, and can output the reconstructed pixel block data.

[0101] The loop filter 1330 can perform various filtering operations on the reconstructed pixel block data. As shown, the loop filter 1330 may include a deblocking filter 1332 and a SAO filter 1334. The deblocking filter 1332 can filter the data at the seams between reconstructed pixel blocks to reduce discontinuities between pixel blocks due to encoding. The SAO filter 1334 can add an offset to the pixel value based on the SAO type, such as based on edge direction / shape and / or pixel level. Other types of loop filters can also be used in a similar manner. The operation of the deblocking filter 1332 and the SAO filter 1334 ideally replicates the operation of their corresponding parts in the analog encoding system 700. Figure 7 Therefore, in the absence of transmission errors or other anomalies, the decoded image obtained from the loop filter 1330 of the decoding system 1300 will be identical to the decoded image obtained from the loop filter 730 of the encoding system 700. Figure 7 The encoding system 700 and the decoding system 1300 should store the same set of reference images in their respective reference image storage libraries 740 and 1340.

[0102] The reference frame repository 1340 can store filtered pixel data for use in subsequent prediction of other pixel blocks. The reference frame repository 1340 can also store decoded pixel block data for each frame during encoding for use in intra-frame prediction. The reference frame repository 1340 can also store decoded reference frames.

[0103] As discussed, predictor 1350 can provide prediction data to pixel block decoder 1320. Predictor 1350 can provide predicted pixel block data determined by prediction reference indicators provided in the encoded video data stream.

[0104] The controller 1360 can control the entire operation of the encoding system 1300. The controller 1360 can set the operating parameters of the pixel block decoder 1320 and the predictor 1350 based on parameters received in the encoded video data stream. For the purposes of this discussion, these operating parameters may include the quantization parameter Q of the dequantizer 1324. P The transformation mode M of the inverse transformation unit 1315. As discussed, the received parameters can be set according to various granularities of the image data, such as per pixel block, per frame, per slice, per LCU, or based on other types of regions defined for the input image.

[0105] As shown in the figure, the pixel block decoder 1320, loop filter 1330, reference image storage 1340, predictor 1350, and controller 1360 can operate on image data in the spherical projection domain. Therefore, the decoding system 1300 can decode coded pixel blocks of different sizes and shapes, such as... Figure 6 As shown. Predictor 1350 can be referenced above. Figure 6 , Figure 10 and / or Figure 11 The described technique operates on the motion vectors of the identified reference blocks, thereby identifying the reference pixel blocks via rotation vectors (α, β, γ), the ratio of projection radii (Ri / Rp), and / or the offset (Δx, Δy, Δz) between the origins of the spherical projection. In response, predictor 1350 may retrieve reference pixel blocks from a reference image repository and align them with the input pixel blocks according to the received motion vectors (e.g., by rotating the spherical projection of the reference image according to rotation vectors (α, β, γ), by adjusting the size of the spherical projection of the reference image according to the ratio of projection radii (Ri / Rp), and / or by shifting the origin of the spherical projection of the reference image according to the origin offset (Δx, Δy, Δz)). After this processing, a portion of the reference image aligned with the position of the encoded pixel block being decoded can be output as a reference block to adder 1328.

[0106] The inverse spherical transform unit 1370 transforms the reconstructed image output from the loop filter 1330 to the output domain. Typically, the output domain will be a two-dimensional domain, making the reconstructed image suitable for display on a display device or for use by applications that consume such data. The inverse transform unit reverses the operation described above. In one embodiment, the inverse spherical transform 1370 can be omitted when outputting the reconstructed image to an application that processes graphic data in the spherical domain.

[0107] The foregoing embodiments describe the video encoding and decoding process for operating on 360° video obtained from a camera system, but the principles of this disclosure are not limited thereto. The techniques described herein can be applied to 360° video, regardless of the techniques used to generate such video. For example, 360° video can be used in computer applications such as video games and 3D rendering applications. Thus, as needed, 360° video can represent computer-generated virtual world models or computer-rendered video data representing human-written content. The principles of this disclosure can also be applied to augmented reality systems, where camera-generated image data and computer-generated graphics data are combined into an encoded 360° video frame. In this respect, the source of the 360° video is irrelevant to this discussion.

[0108] In one implementation, the video encoder and decoder can exchange signaling to identify parameters of the spherical projection. In one such implementation, this signaling can be performed according to the following syntax.

[0109] The video encoder and decoder can exchange a projection format field, which identifies the type of projection format used by the video encoder in the encoder domain. For example, the projection format field can contain a projection format id value that can take the following values:

[0110]

[0111] Table 1

[0112] For example, projection format fields can be provided in the sequence parameter set of an encoding protocol such as H.265.

[0113] The video encoder and decoder can exchange other signaling, such as the use_rotation_estimation field and the rotation_estimation_mode field. use_rotation_estimate can indicate, for example, whether to perform actions as shown in box 840 (…). Figure 8 The rotation estimation is described above. The `rotation_estimation_mode` field can identify the parameters of the rotation estimation, such as:

[0114]

[0115] Table 2

[0116] The `use_rotation_estimation` and `rotation_estimation_mode` fields can be signaled at different levels of coding granularity within the encoding protocol. For example, the `use_rotation_estimation` field can be signaled at the pixel block level during encoding, while the `rotation_estimation_mode` field can be signaled at a higher level of granularity, such as a sequence parameter set or a picture parameter set. Conversely, the `use_rotation_estimation` field can be signaled at the sequence parameter set level during encoding, while the `rotation_estimation_mode` field can be signaled at a lower level of granularity, such as at the pixel block level. Furthermore, the `use_rotation_estimation` and `rotation_estimation_mode` fields can be predicted as needed, either picture-to-picture or pixel-to-pixel. Additionally, context-adaptive binary arithmetic coding (CABAC) of the parameter data can be performed.

[0117] Similarly, the video encoder and decoder can exchange other signaling that may indicate the type of global rotation estimation being performed, such as the `global_rotation_estimation_mode` field. The `global_rotation_estimation_mode` field can identify parameters of the global rotation estimation, such as:

[0118]

[0119] The global_rotation_estimation_mode field can be conveniently signaled in the sequence parameter set, screen parameter set, or slice segment header.

[0120] The foregoing discussion has described the operation of various embodiments of this disclosure within the context of video encoders and decoders. These components are often provided as electronic devices. Video decoders and / or controllers can be embedded in integrated circuits, such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), and / or digital signal processors (DSPs). Alternatively, they can be embedded in computer programs that execute in camera devices, personal computers, laptops, tablets, smartphones, or computer servers. Such computer programs are typically stored in physical storage media such as electronic, magnetic, and / or optical-based storage devices, where they are read by a processor and executed. Decoders are often packaged in consumer electronic devices, such as smartphones, tablets, gaming systems, DVD players, portable media players, etc.; and they can also be packaged in consumer software applications, such as video games, media players, media editors, etc. Furthermore, these components can, of course, be provided as hybrid systems that distribute functionality as needed across dedicated hardware components and programmed general-purpose processors.

[0121] The foregoing description has been presented for purposes of illustration and description. It is not exhaustive and does not limit the embodiments of this disclosure to the precise form disclosed. Modifications and variations are possible in light of the foregoing teachings, or can be derived from embodiments consistent with this disclosure in practice. Any combination may be employed to implement any method unless otherwise described herein.

Claims

1. A video decoding method, comprising: Decode the reference frame from the encoded video sequence in the source domain; Decoding spherical motion vectors for pixel blocks of the current frame from the encoded video sequence, the motion vectors indicating the change in radius between a first radius of a first spherical projection of the reference frame and a second radius of a second spherical projection of the current frame, wherein the first radius and the second radius are not the same; Transform the reference frame from the source domain to the spherical domain; The spherical domain prediction of the pixel block is generated from the transformed spherical domain reference frame and the spherical domain motion vector, the generation being based on the change in the radius of the spherical domain projection; The spherical domain prediction is transformed into the source domain prediction of the pixel block; as well as Based on the source domain prediction of the pixel blocks, a decoded current frame is generated in the source domain.

2. The video decoding method according to claim 1, wherein the spherical domain motion vector specifies the rotation of the reference frame in three dimensions.

3. The video decoding method according to claim 1 further includes: Decode the global rotation value for the reference frame from the encoded video sequence; Decoding the differential motion vectors for the pixel blocks from the encoded video sequence; The spherical domain motion vector for the pixel block is determined based on the global rotation value and the differential motion vector.

4. The video decoding method of claim 1, wherein the spherical domain motion vector includes a relative radius ratio, the relative radius ratio specifying the ratio of the current frame radius used in the spherical domain projection of the current frame to the reference frame radius used in the spherical projection of the reference frame, and the spherical domain prediction is based on the relative radius ratio.

5. The video decoding method according to claim 1, further comprising: Decoding the reference space origin for the reference frame from the encoded video sequence; Decoding is performed from the encoded video sequence for the current spatial origin of the current frame; Furthermore, the spherical domain prediction is based on the reference space origin and the current space origin.

6. A video decoding system, comprising: processor; as well as The memory stores instructions that, when executed by the processor, cause: Decode the reference frame from the encoded video sequence in the source domain; Decoding spherical motion vectors for pixel blocks of the current frame from the encoded video sequence, the motion vectors indicating the change in radius between a first radius of a first spherical projection of the reference frame and a second radius of a second spherical projection of the current frame, wherein the first radius and the second radius are not the same; Transform the reference frame from the source domain to the spherical domain; Based on the change in the radius of the spherical domain projection, a spherical domain prediction of the pixel block is generated from the transformed spherical domain reference frame and the spherical domain motion vector; The spherical domain prediction is transformed into the source domain prediction of the pixel block; as well as Based on the source domain prediction of the pixel blocks, a decoded current frame is generated in the source domain.

7. The video decoding system of claim 6, wherein the spherical domain motion vector specifies the rotation of the reference frame in three dimensions.

8. The video decoding system according to claim 6, wherein the instruction further causes: Decode the global rotation value for the reference frame from the encoded video sequence; Decoding the differential motion vectors for the pixel blocks from the encoded video sequence; The spherical domain motion vector for the pixel block is determined based on the global rotation value and the differential motion vector.

9. The video decoding system of claim 6, wherein the spherical domain motion vector includes a relative radius ratio, the relative radius ratio specifying the ratio of the current frame radius used in the spherical domain projection of the current frame to the reference frame radius used in the spherical projection of the reference frame, and the spherical domain prediction is based on the relative radius ratio.

10. The video decoding system according to claim 6, wherein the instruction further causes: Decoding the reference space origin for the reference frame from the encoded video sequence; Decoding is performed from the encoded video sequence for the current spatial origin of the current frame; Furthermore, the spherical domain prediction is based on the reference space origin and the current space origin.

11. A video decoding method, comprising: Decode the reference frame from the encoded video sequence in the spherical domain; Decoding spherical motion vectors for pixel blocks of the current frame from the encoded video sequence, the motion vectors indicating the change in radius between a first radius of a first spherical projection of the reference frame and a second radius of a second spherical projection of the current frame, wherein the first radius and the second radius are not the same; Based on the change in the radius of the spherical domain projection, a spherical domain prediction of the pixel block is generated from the transformed spherical domain reference frame and the spherical domain motion vector; Based on the spherical domain prediction of the pixel blocks, a decoded current frame is generated in the spherical domain; The decoded current frame in the spherical domain will be transformed to the source domain; as well as Output the decoded current frame in the source domain.

12. The video decoding method of claim 11, wherein the spherical domain motion vector specifies the rotation of the reference frame in three dimensions.

13. The video decoding method according to claim 11, further comprising: Decode the global rotation value for the reference frame from the encoded video sequence; Decoding the differential motion vectors for the pixel blocks from the encoded video sequence; The spherical domain motion vector for the pixel block is determined based on the global rotation value and the differential motion vector.

14. The video decoding method of claim 11, wherein the spherical domain motion vector includes a relative radius ratio, the relative radius ratio specifying the ratio of the current frame radius used in the spherical domain projection of the current frame to the reference frame radius used in the spherical projection of the reference frame, and the spherical domain prediction is based on the relative radius ratio.

15. The video decoding method according to claim 11, further comprising: Decoding the reference space origin for the reference frame from the encoded video sequence; Decoding is performed from the encoded video sequence for the current spatial origin of the current frame; Furthermore, the spherical domain prediction is based on the reference space origin and the current space origin.

16. A non-transitory computer-readable medium storing program instructions that, when executed by a processing device, cause the device to: Decode the reference frame from the encoded video sequence in the spherical domain; Decoding spherical motion vectors for pixel blocks of the current frame from the encoded video sequence, the motion vectors indicating the change in radius between a first radius of a first spherical projection of the reference frame and a second radius of a second spherical projection of the current frame, wherein the first radius and the second radius are not the same; Based on the change in the radius of the spherical domain projection, a spherical domain prediction of the pixel block is generated from the transformed spherical domain reference frame and the spherical domain motion vector; Based on the spherical domain prediction of the pixel blocks, a decoded current frame is generated in the spherical domain; The decoded current frame in the spherical domain will be transformed to the source domain; as well as Output the decoded current frame in the source domain.

17. The computer-readable medium of claim 16, wherein the spherical domain motion vector specifies the rotation of the reference frame in three dimensions.

18. The computer-readable medium of claim 16, further comprising: Decode the global rotation value for the reference frame from the encoded video sequence; Decoding the differential motion vectors for the pixel blocks from the encoded video sequence; The spherical domain motion vector for the pixel block is determined based on the global rotation value and the differential motion vector.

19. The computer-readable medium of claim 16, wherein the spherical domain motion vector includes a relative radius ratio, the relative radius ratio specifying the ratio of the current frame radius used in the spherical domain projection of the current frame to the reference frame radius used in the spherical projection of the reference frame. Furthermore, the spherical domain prediction is based on the relative radius ratio.

20. The computer-readable medium of claim 16, further comprising: Decoding the reference space origin for the reference frame from the encoded video sequence; Decoding is performed from the encoded video sequence for the current spatial origin of the current frame; Furthermore, the spherical domain prediction is based on the reference space origin and the current space origin.

Images