Encoding device, decoding device, and program

Abstract

The present invention provides an encoding device, a decoding device, and a program capable of efficiently processing curved images. [Solution] An encoding device according to one embodiment is an encoding device used in a video distribution system. The encoding device has: an input image of a curved image converted into a rectangular image, an extraction unit that extracts a part of the input image; a geometric transformation unit that converts a part of the input image into a two-dimensional image by perspective projection; a base layer encoding unit that encodes the two-dimensional image as a base layer; an inverse geometric transformation unit that converts the decoded image obtained by decoding the encoded two-dimensional image into the rectangular image by inverse perspective projection transformation; and an enhancement layer encoding unit that uses the rectangular image output from the inverse geometric transformation unit as a prediction image and encodes the input image as an enhancement layer.

Description

[Technical Field]

[0001] This disclosure relates to an encoding device, a decoding device, and a program. [Background technology]

[0002] In recent years, viewing 360° video, a wide-angle video (or omnidirectional video) using head-mounted displays (HMDs) and omnidirectional displays, has begun to become widespread. Such 360° video can provide viewers with an unprecedentedly immersive and realistic experience.

[0003] Regarding 360° video, the ITU report (Non-Patent Literature 1 below) recommends a resolution of 30720 (30K) × 15360 (15K). In this case, approximately 16 times the amount of data is required compared to the maximum resolution currently available for commercial services (7680 (8K) × 4320 (4K)). Therefore, when implementing 360° video as a broadcast service, encoding technology with higher transmission efficiency is required.

[0004] Furthermore, regarding 360° video, the ITU report (Non-Patent Document 1 below) also recommends that instead of using 360° video as is, it be converted to rectangular video (or a rectangular image) using equirectangular projection (ERP).

[0005] On the other hand, there is a conventional coding technique called scalable coding. Scalable coding is a coding technique that performs coding in a hierarchical manner, for example. In scalable coding, the base layer is used as the basic stream, and the difference video is transmitted as the enhancement layer. Scalable coding is sometimes called multi-layer coding or hierarchical coding. The receiving side can decode not only the base layer alone, but also a combination of the base layer and the enhancement layer.

[0006] One such scalable coding method is spatially scalable coding. In spatially scalable coding, an enhancement layer (high-resolution video) is transmitted to improve the resolution of a base layer (low-resolution video). Spatially scalable coding makes it possible to efficiently transmit services with different resolutions simultaneously, such as sets of 2K and 4K video, or sets of 4K and 8K video.

[0007] Regarding scalable coding, the latest coding method, VVC (Versatile Video Coding; Non-Patent Document 2), is defined as the Multilayer Main10 profile, which enables the transmission of video across multiple layers. [Prior art documents] [Non-patent literature]

[0008] [Non-Patent Document 1] Recommendation ITU-R BT.2123-0 (01 / 2019) "Video parameter values ​​for advanced immersive audio-visual systems for production and international program exchange in broadcasting" [Non-Patent Document 2] Recommendation ITU-T H.266 (09 / 2023) “Versatile video coding” [Overview of the project] [Problems that the invention aims to solve]

[0009] This disclosure aims to provide an encoding device, a decoding device, and a program capable of efficiently processing curved images. [Means for solving the problem]

[0010] The encoding device according to the first embodiment is an encoding device used in a video distribution system. The encoding device has: an extraction unit that takes a curved image converted into a rectangular image as an input image and extracts a part of the input image; a geometric transformation unit that converts a part of the input image into a two-dimensional image by perspective projection; a base layer encoding unit that encodes the two-dimensional image as a base layer; an inverse geometric transformation unit that converts the decoded image obtained by decoding the encoded two-dimensional image into the rectangular image by inverse perspective projection transformation; and an enhancement layer encoding unit that uses the rectangular image output from the inverse geometric transformation unit as a prediction image and encodes the input image as an enhancement layer.

[0011] The decoding device according to the second embodiment is a decoding device used in a video distribution system. The decoding device includes a base layer decoding unit that decodes a first decoded image of the base layer from a first encoded image encoded as the base layer; an inverse geometric transformation unit that converts the first decoded image into a curved image converted into a rectangular image by inverse perspective projection transformation; and an enhancement layer decoding unit that decodes the difference image between the enhancement layer and the base layer from a second encoded image encoded as the enhancement layer, and outputs a decoded image of the enhancement layer based on the difference image and the rectangular image output from the inverse geometric transformation unit.

[0012] The program according to the third embodiment is a program that causes a computer to function as an encoding device according to the first embodiment or a decoding device according to the second embodiment. [Effects of the Invention]

[0013] According to this disclosure, it is possible to provide an encoding device, a decoding device, and a program that can efficiently process curved images. [Brief explanation of the drawing]

[0014] [Figure 1] Figure 1 is a diagram showing an example configuration of a video distribution system according to the first embodiment. [Figure 2]FIG. 2 is a diagram showing an example of a coordinate system of a rectangular video according to the first embodiment. [Figure 3] FIG. 3 is a diagram showing a configuration example of an encoding device according to the first embodiment. [Figure 4] FIG. 4 is a diagram showing an example of an encoding target by multi-layer encoding according to the first embodiment. [Figure 5] FIG. 5 is a diagram showing a configuration example of an EL encoding unit and a BL encoding unit according to the first embodiment. [Figure 6] FIG. 6 is a diagram showing an example of a two-dimensional video of an encoding target according to the first embodiment. [Figure 7] FIGS. 7(A) to 7(D) are diagrams showing examples of vertices as starting points according to the first embodiment. [Figure 8] FIG. 8 is a diagram showing an example of a cut-out area according to the first embodiment. [Figure 9] FIG. 9 is a diagram showing an example of a reference vector according to the first embodiment. [Figure 10] FIG. 10 is a diagram showing a configuration example of a decoding device according to the first embodiment.

MODE FOR CARRYING OUT THE INVENTION

[0015] [First Embodiment] A video distribution system according to an embodiment will be described while referring to the drawings. In the description of the drawings, the same or similar parts are denoted by the same or similar reference numerals.

[0016] (Video Distribution System According to the First Embodiment) First, a video distribution system according to the first embodiment will be described.

[0017] Figure 1 is a diagram showing an example configuration of the video distribution system 10 according to the first embodiment. The video distribution system 10 according to the first embodiment may be used in a broadcasting system in which video is transmitted from a broadcasting station to television receivers in each home via a broadcast propagation path. Alternatively, the video distribution system according to the first embodiment may be used for video distribution using a communication line in which the transmitting and receiving sides communicate with each other, such as the Internet (registered trademark; hereinafter the same).

[0018] As shown in Figure 1, the video distribution system 10 includes an encoding device 100 and a decoding device 200.

[0019] The encoding device 100 according to the first embodiment does not use 360° video as input video, but rather uses 360° video converted to rectangular video in accordance with the ITU report (Non-Patent Literature 1) described above. For this reason, a conversion unit that converts 360° video to rectangular video using equirectangular projection (ERP) may be provided in front of the encoding device 100. Alternatively, such a conversion unit may be provided inside the encoding device 100.

[0020] In the first embodiment, ERP is used as an example of a projection format for converting (projecting) a 360° image into a rectangular image, but other projection formats may be used. Such projection formats may include, for example, CMP (Cubemap Projection format), ACP (Adjusted Cubemap Projection format), EAC (Equi-angular cubemap projection), HEC (Hybrid equi-angular cubemap projection), GCMP (Generalized cubemap projection format), AEP (Adjusted equal-area projection format), OHP (Octahedron projection format), ISP (Icosahedron projection format), SSP (Segmented sphere projection format), RSP (Rotated sphere projection format), ECP (Equatorial cylindrical projection format), TSP (Truncated square pyramid projection format), HCMP (Hemisphere cubemap projection), or HEAC (Hemisphere Equi-angular cubemap projection), as specified in "JVET (Joint Video Experts Team)-S2004-v1".

[0021] Here, we will explain the 360° video that serves as the source for ERP conversion. In the first embodiment, the 360° video is an example of a video attached to the surface of a unit sphere, as per Non-Patent Literature 1. The video attached to the surface of the unit sphere may be less than 360° or even a half-circle video. Hereinafter, a video attached to the surface of a unit sphere will be referred to as a curved video. The coordinate system of the curved video is represented by three-dimensional coordinates (X,Y,Z). The three-dimensional coordinates (X,Y,Z) of the curved video may be represented as follows, according to "JVET-S2004-v" 1.

[0022]

number

[0023]

number

[0024]

number

[0025]

number

[0026]

number

[0027] Figure 2 shows an example of the coordinate system of a rectangular image according to the first embodiment. In ERP, θ and φ can be calculated using equations (1) to (3), and substituted into equations (4) and (5) to project (transform) the coordinates (X,Y,Z) of the curved image onto the coordinates (i,j) of the rectangular image. Figure 4(B) shows an example of a rectangular image converted using ERP. The rectangular image shown in Figure 4(B) can be represented in the coordinate system shown in Figure 2. As shown in Figure 4(B), the rectangular image is simply a curved image converted into a rectangular image, and therefore contains distorted parts compared to the actual image. Conversely, if the rectangular image is pasted onto a unit sphere, an undistorted image will be displayed on the surface of the unit sphere.

[0028] Returning to Figure 1, the input video input to the encoding device 100 is a rectangular video converted from a curved video.

[0029] The encoding device 100 according to the first embodiment performs encoding processing using spatially scalable encoding (hereinafter sometimes referred to as "multilayer encoding"). In multilayer encoding, the base layer (hereinafter sometimes referred to as "BL") is used as the basic stream, and the difference video is transmitted as the enhancement layer (hereinafter sometimes referred to as "EL"). In Figure 1, an example is shown where the EL is transmitted as a single bitstream, but the EL may be transmitted as multiple bitstreams.

[0030] The multilayer coding according to the first embodiment may use the Scalable Main 10 profile in HEVC (High Efficiency Video Coding). The Scalable Main 10 profile is an coding mechanism that allows for the transmission of video equivalent to the Main 10 profile as the BL while adding an EL and superimposing it on the BL video. Alternatively, the multilayer coding according to the first embodiment may use the Multilayer Main 10 profile in VVC (Versatile Video Coding). The Multilayer Main 10 profile, like the Scalable Main 10 profile described above, is a profile that enables the transmission of multiple layers relative to the Main 10 profile, and allows for the transmission of video of multiple layers. Alternatively, the multilayer coding according to the first embodiment may use a method based on LCEVC (Low Complexity Enhancement Video Coding) or scalable coding using AV1 (AOMedia Video 1). In the following, multilayer coding using VVC will be explained as an example.

[0031] The decoding device 200 can decode not only the bitstream (BL) alone, but also a combination of the BL and EL. With multi-layer coding, for example, high-quality video can be reproduced by decoding using the bitstreams of multiple layers, while low-quality video can be reproduced by decoding only the bitstream of some layers (BL). Details of the decoding device 200 will be described later.

[0032] (Example of encoding device configuration) Next, an example of the configuration of the encoding device 100 according to the first embodiment will be described.

[0033] Figure 3 is a diagram showing an example configuration of the encoding device 100.

[0034] As shown in Figure 3, the encoding device 100 has a cutting unit 110, a geometric transformation unit 120, and a BL encoding unit 130. The BL encoding block may be formed by the cutting unit 110, the geometric transformation unit 120, and the BL encoding unit 130.

[0035] Furthermore, the encoding device 100 includes a local decoding unit 140, an inverse geometric transformation unit 150, and an EL encoding unit 160. The local decoding unit 140, the inverse geometric transformation unit 150, and the EL encoding unit 160 may constitute an EL encoding block.

[0036] Furthermore, the encoding device 100 has a transmitting unit 170.

[0037] The cropping unit 110 crops a portion of the input video (rectangular video). In a scenario where a curved video is actually used, not the entire curved video is used; rather, only a portion of it is used according to the director's intentions. In the first embodiment, cropping is performed assuming such a real-world scenario. Figure 4(B) shows an example of a cropped video obtained by cropping a rectangular video. The cropping unit 110 outputs the cropped portion of the rectangular video to the geometric transformation unit 120.

[0038] Returning to Figure 3, the geometric transformation unit 120 performs a geometric transformation using perspective projection on a portion of the rectangular image, converting it into a distortion-free two-dimensional image. Because the rectangular image contains distortion, even if a portion of the rectangular image is cut out, the resulting image is still distorted. In the first embodiment, the geometric transformation unit 120 converts the distorted rectangular image into a distortion-free two-dimensional image. Perspective projection is used as the conversion method. Perspective projection is a projection method in which all projection lines converge to a single viewpoint. In perspective projection, because the projection lines converge to a single viewpoint, objects of the same size appear larger the closer they are to the viewpoint and smaller the further they are from the viewpoint. Perspective projection makes it possible to display the image using perspective, and it is possible to obtain a distortion-free image that can be perceived by human vision. Specifically, the geometric transformation unit 120 converts the coordinates (i,j) of the rectangular image output from the cutting unit 110 to a three-dimensional coordinate system (i1,j1,h1) such as the world coordinate system, and then performs a geometric transformation using perspective projection with the following formula.

[0039]

number

[0040]

number

[0041] Furthermore, the perspective projection may be performed using a one-point perspective projection method with one vanishing point where parallel lines intersect, a two-point perspective projection method with two vanishing points, or a three-point perspective projection method with three vanishing points. In addition, the coordinate system transformation performed before perspective projection may include a transformation to the camera coordinate system. The perspective projection itself may also be performed using a known method other than those given by equations (6) and (7).

[0042] The BL encoding unit 130 encodes the two-dimensional video output from the geometric transformation unit 120. The BL encoding unit 130 can output a lower-resolution encoded video compared to the encoded video encoded by the EL encoding unit 160.

[0043] Figure 5 shows an example of the configuration of the BL encoding unit 130.

[0044] As shown in Figure 5, the BL encoding unit 130 includes a sorting unit 131, an encoding processing unit 132, a decoding processing unit 133, a DPB (Decode Picture Buffer) 134, and an entropy encoding unit 135. Although the example configuration of the BL encoding unit 130 shown in Figure 5 is explained using VVC as an example, other encoding schemes such as HEVC or AV1 may also be used.

[0045] The sorting unit 131 rearranges the encoding order of the 2D video and outputs it to the encoding processing unit 132.

[0046] In the encoding processing unit 132, for intra prediction, for each prediction block, it performs in-screen prediction (spatial prediction) (prediction unit) using decoded pixels (prediction reference pixels) near the prediction block, converts the predicted difference signal to an integer (conversion unit), and quantizes it (quantization unit). In addition, for inter prediction, the encoding processing unit 132 generates a predicted image by performing motion compensation using multiple reference images stored in the DPB 134 for each prediction block (prediction unit), and performs integer conversion (conversion unit) and quantization (quantization unit) on the predicted difference signal between the predicted image and the input image.

[0047] The decoding processing unit 133 performs inverse quantization and inverse integer transformation (inverse quantization unit and inverse transformation unit) on the encoded image output from the encoding processing unit 132 to generate a reconstructed image. Loop filtering (loop filtering processing unit) is then applied to this image to reduce block boundary distortion and generate a decoded image. The decoding processing unit 133 outputs the decoded image to the DPB 134.

[0048] DPB135 is a buffer that stores the decoded image.

[0049] The entropy coding unit 135 performs variable-length coding by applying CABAC (Context-based Adaptive Binary Arithmetic Coding) coding to the coded video output from the coding processing unit 132. The video after entropy coding is output as BL coded video.

[0050] Here, we will provide a summary of the entire BL encoding block shown in Figure 3. Figure 6 is a diagram showing an example of a 2D video to be encoded in the BL encoding unit 130. As shown in Figure 6, the video extracted from the rectangular video by the extraction unit 110 is distorted. This is because the extracted video is only a part of the rectangular video. By performing a geometric transformation (perspective projection transformation) on this video using the geometric transformation unit 120, a distortion-free 2D video can be obtained. By using such a 2D video, the BL encoding unit 130 can encode the BL input video as the same video as conventional input videos such as HD (High Definition), 4K, or 8K. Furthermore, the decoding device 200 can decode the encoded video of the BL, making the decoded HD, 4K, or 8K video available for viewing as is.

[0051] Returning to Figure 3, the local decoding unit 140 performs decoding on the encoded video output from the BL encoding unit 130. However, if the decoded video is output from the BL encoding unit 130, the local decoding unit 140 is not required. In other words, in Figure 5, if the decoded video from DBP 134 is output from the BL encoding unit 130, the local decoding unit 140 is not required, but if the output from the encoding processing unit 132 is output from the BL encoding unit 130, the local decoding unit 140 may be present. The local decoding unit 140 may perform the same processing as the decoding processing unit 133 in the BL encoding unit 130. The decoded video output from the local decoding unit 140 is a reconstruction of the 2D video input to the BL encoding unit 130, and unlike the rectangular video, it is a distortion-free video. The local decoding unit 140 outputs the decoded video (2D video) to the inverse geometric transformation unit 150.

[0052] The inverse geometric transformation unit 150 performs an inverse geometric transformation on the decoded image. Specifically, the inverse geometric transformation unit 150 performs an inverse perspective projection, the opposite of perspective projection, on the decoded image to obtain a rectangular image. While the geometric transformation unit 120 used equations (6) and (7) to transform the rectangular image (i,j) into a two-dimensional image (x,y), the inverse geometric transformation unit 150 can use equations (6) and (7) to do the opposite, obtaining a rectangular image (i,j) from a two-dimensional image (x,y). The transformed image output from the inverse geometric transformation unit 150 is a reconstruction of the cropped image (a part of the rectangular image) output from the cropping unit 110 in the encoding device 100. The inverse geometric transformation unit 150 outputs a part of the transformed rectangular image to the EL encoding unit 160. This part of the rectangular image is used as a prediction image in the EL encoding unit 160.

[0053] The EL encoding unit 160 encodes the difference between the EL and BL. The EL encoding unit 160 uses interlayer prediction, which references the BL, during encoding. Specifically, the EL encoding unit 160 uses a portion of the rectangular image output from the inverse geometric transformation unit 150 as a predicted image to encode the difference with respect to the input image (rectangular image) (EL). Interlayer prediction allows the difference image (encoded image) with respect to the BL to be transmitted as EL. The EL encoding unit 160 can output encoded images with higher resolution compared to the BL encoding unit 130.

[0054] Figure 5 shows an example of the configuration of the EL coding unit 160. The EL coding unit 160 includes a sorting unit 161, an coding processing unit 162, a decoding processing unit 163, a DPB 164, and an entropy coding unit 165. In the EL coding unit 160, the sorting unit 161, the decoding processing unit 163, the DPB 164, and the entropy coding unit 165 may perform basically the same processing as the sorting unit 131, the decoding processing unit 133, the DPB 134, and the entropy coding unit 135 in the BL coding unit 130, except that the processing target is the input video (rectangular video).

[0055] The encoding processing unit 162 generates a difference image between the rectangular image output from the rearrangement unit 161 and a portion of the rectangular image (BL side) output from the inverse geometry transformation unit 150 (not shown in Figure 5). The encoding processing unit 162 then performs integer transformation and quantization on the difference image. The encoding processing unit 162 may restrict the region where the difference with BL is zero to the region used for interlayer prediction. In this case, for regions where the difference is not zero, the encoding processing unit 162 may read the decoded image decoded by the decoding processing unit 163 from the DPB 164 and use the decoded image as the prediction image. The entropy encoding unit 165 outputs the EL encoded image (i.e., the image in which the difference image between EL and BL is encoded).

[0056] Returning to Figure 3, the transmitting unit 170 transmits the BL-encoded video output from the BL encoding unit 130 as a BL bitstream to the decoding device 200, and transmits the EL-encoded video output from the EL encoding unit 160 as an EL bitstream to the decoding device 200. The transmitting unit 170 may also multiplex the two bitstreams and transmit them to the decoding device 200 as a multilayer bitstream. The transmitting unit 170 may also be called a multiplexing unit.

[0057] As described above, in the encoding device 100 according to the first embodiment, a rectangular image converted from a curved image is used as the input image, and the BL encoding unit 130 has a cropping unit 110 and a geometric transformation unit 120 in front of it. This makes it possible to use a distortion-free two-dimensional image as the target of BL encoding, and the BL encoding unit 130 can perform the same processing as conventional encoding targets. Furthermore, the EL encoding block has an inverse geometric transformation unit 150 in front of the EL encoding unit 160. This allows the EL encoding unit 160 to perform multi-layer encoding processing using the same type of rectangular image for both the BL image and the input image. Therefore, the encoding device 100 as a whole can efficiently encode the input image (rectangular image, or the curved image that is the source of the rectangular image).

[0058] Specifically, the encoding device 100 has the following configuration: an input image of a curved image converted into a rectangular image, an extraction unit 110 that extracts a portion of the input image, a geometric transformation unit 120 that converts a portion of the input image into a two-dimensional image by perspective projection, a BL encoding unit 130 that encodes the two-dimensional image as BL, an inverse geometric transformation unit 150 that converts the decoded image obtained by decoding the encoded two-dimensional image into a rectangular image by inverse perspective projection transformation, and an EL encoding unit 160 that uses the rectangular image output from the inverse geometric transformation unit 150 as a predicted image and encodes the input image as EL.

[0059] (Interlayer prediction) In the EL encoding unit 160 (specifically the encoding processing unit 162), the image converted into a part of the rectangular image by the inverse geometric transformation unit 150 (hereinafter referred to as "partial image") is used as the prediction image to perform encoding on the rectangular image on the EL side. Specifically, the EL encoding unit 160 uses the corresponding area of ​​the partial image (BL) corresponding to the encoding target area of ​​the rectangular image (EL) as a reference area, and generates the prediction image (BL) based on the decoded image of the reference area.

[0060] In a typical multilayer encoding, the EL encoding unit 160 generates a predicted image by upsampling the reference region on the BL side based on the resolution ratio of the BL and EL, making the reference block on the BL side the same resolution as the encoding target region on the EL side. The EL encoding unit 160 then generates a difference image between the image of the encoding target region on the EL side and the decoded image of the reference region on the BL side, so that the difference image approaches zero. This is because the encoding target image on the EL side and the decoded image on the BL side have the same field of view (content) (Figure 4(A)), and by assuming that the encoding target region on the EL side and the reference region on the BL side exist in the same region, the difference image approaches zero, which means that an improvement in encoding efficiency can be expected.

[0061] However, in the multi-layer encoding according to the first embodiment, the BL side is part of the rectangular image on the EL side, so the EL side and the BL side do not have the same field of view (content) (Figure 4(B)). Even if the EL encoding unit 160 upsamples a portion of the image as is, the reference area on the BL side is significantly different from the encoding target area on the EL side. Therefore, it is difficult to improve the encoding efficiency by generating a difference image in this state.

[0062] Therefore, in the first embodiment, the reference region on the BL side can be referenced even outside the region of the encoding target region on the EL side. To achieve this, in the first embodiment, (1) syntax for extraction and (2) syntax for the reference vector that references the BL are defined, and these are encoded in the EL encoding unit 160 and transmitted to the decoding device 200. A specific example of the syntax will be described below.

[0063] (1) Syntax regarding the cutting position In the first embodiment, the syntax for the cutting position in the cutting section 110 is defined. Specifically, it is as follows:

[0064] Firstly, information representing the conversion type used when converting a curved image to a rectangular image (input image) may be defined as syntax. This syntax may also be called the conversion type syntax. Specifically, the conversion type syntax may be represented by information such as ERP or CMP.

[0065] Secondly, information indicating the extraction position of the input video by the extraction unit 110 may be defined as syntax. This syntax may be called the extraction position syntax.

[0066] The cropping position syntax may include syntax for identifying the starting vertex among the four vertices in the rectangular image. Figures 7(A) to 7(D) show examples of starting vertices according to the first embodiment. The syntax only needs to be able to identify the four vertices, and for example, it may be "0" when the upper left corner of the rectangular image is the starting point (Figure 7(A)), "1" when the upper right corner is the starting point (Figure 7(B)), "2" when the lower left corner is the starting point (Figure 7(C)), and "3" when the lower right corner is the starting point (Figure 7(D)), but is not limited to these.

[0067] Furthermore, the cropping position syntax may include syntax indicating the relative coordinates (x1, y1) from the starting vertex to the vertex closest to the starting vertex among the four vertices of the cropping area. Figure 8 shows an example of a cropping area according to the first embodiment. In Figure 8, since the upper left corner is used as the starting point among the four vertices of the rectangular image, the coordinates of the upper left corner among the four vertices of the cropping area become the relative coordinates (x1, y1). If the upper right corner is used as the starting point among the four vertices of the rectangular image, the coordinates of the upper right corner of the cropping area may become the relative coordinates.

[0068] Furthermore, the cropping position syntax may include syntax indicating the width and height (dx, dy) of the cropping region, starting from the relative coordinates (x1, y1).

[0069] Furthermore, the cropping position syntax may include syntax indicating the relative position of the upper-left corner and the relative position of the lower-right corner of the cropped area. Figure 8 shows an example where the upper-left corner of the rectangular image is used as the starting point, and (x1, y1) is shown as the relative position of the upper-left corner of the cropped area, and (x2, y2) is shown as the relative position of the lower-right corner. The starting point can be any of the four vertices of the rectangular image, and this starting point may be indicated by the "syntax for identifying the starting vertex" described above.

[0070] The EL encoding unit 160 encodes the extraction position syntax. The EL encoding unit 160 may output the encoded extraction position syntax in the EL bitstream.

[0071] (2) Syntax for reference vectors that refer to BL In order to allow the BL-side reference block to be referenced even outside the same region as the EL-side encoding target region, the first embodiment defines the following syntax for the reference vector. However, if the BL-side reference region is the same region as the EL-side encoding target region, the BL reference vector in the multi-layer encoding scheme defined in VVC may be used as is.

[0072] First, the syntax regarding the reference vector includes information indicating whether to reference the BL side in a region outside the encoding target region on the EL side. This information may be referred to as referenceable syntax.

[0073] Second, the syntax regarding the reference vector includes the difference value between the reference vector used when referencing the BL side in the adjacent encoding target region adjacent to the encoding target region on the EL side and the reference vector for referencing the BL side in the encoding target region. FIG. 9 is a diagram showing an example of a reference vector according to the first embodiment. In FIG. 9, the encoding target region is the region indicated by the dotted line, and the reference vector for referencing the BL side in the encoding target region is (mv x , mv y ). Also, in FIG. 9, the adjacent encoding target region adjacent to the encoding target region is the region indicated by the solid line, and the reference vector used when referencing the BL side in the adjacent encoding target region is (mv’ x , mv’ y ). The difference value (dmv x , dmv y ) can be (mv x - mv’ x , mv y - mv’ y ). The difference value means that the coordinate information in the reference image (decoded image) of the BL (i.e., the relative position from the upper left corner of the BL reference image) is represented as the syntax regarding the reference vector. By using the difference value instead of using the reference vector as it is, the amount of information in the syntax can be reduced. Note that the reference region on the BL side may have the same size as the encoding region on the EL side, or its size may be enlarged or reduced. In this case, the EL encoding unit 160 may use the Scaling window defined in VVC to enlarge or reduce the reference region on the BL side.

[0074] The EL encoding unit 160 encodes the syntax regarding the reference vector. The EL encoding unit 160 may include the encoded syntax regarding the reference vector in the EL bitstream and output it.

[0075] (Example of a decoding device configuration) Next, an example of the configuration of the decoding device 200 according to the first embodiment will be described.

[0076] Figure 10 is a diagram showing an example configuration of the decoding device 200.

[0077] As shown in Figure 10, the decoding device 200 includes a receiving unit 210, a BL decoding unit 220, an inverse geometric transformation unit 230, and an EL decoding unit 240.

[0078] The receiving unit 210 receives the BL bitstream and EL bitstream transmitted from the encoding device 100. The receiving unit 210 may also receive a multilayer bitstream in which the two bitstreams are multiplexed. The receiving unit 210 outputs the BL encoded video contained in the BL bitstream to the BL decoding unit 220 and the EL encoded video contained in the EL bitstream to the EL decoding unit 240.

[0079] The BL decoding unit 220 decodes the BL-coded video from the BL-coded video. Unlike the rectangular video, the decoded video is basically a distortion-free two-dimensional video. That is, the decoded two-dimensional video is a video in which the two-dimensional video (partially the video) before BL encoding in the encoding device 100 (Figure 3) has been restored. The BL decoding unit 220 may also include a decoding processing unit 133 (Figure 5), and may decode the two-dimensional video by the same processing as the decoding processing unit 133. The BL decoding unit 220 outputs the decoded two-dimensional video to the inverse geometry transformation unit 230.

[0080] The inverse geometric transformation unit 230 performs an inverse geometric transformation (inverse perspective projection) on the decoded 2D image to convert it into a rectangular image. The inverse geometric transformation unit 230 may perform the same processing as the inverse geometric transformation unit 150 in the encoding device 100 (Figure 3). Specifically, the inverse geometric transformation unit 230 can use equations (6) and (7) to obtain a rectangular image (i,j) from a 2D image (x,y). The rectangular image after the inverse geometric transformation is a reconstruction of the cropped image (part of the rectangular image) before input to the geometric transformation unit 120 in the encoding device 100 (Figure 3). The inverse geometric transformation unit 230 outputs (part of) the rectangular image after the inverse geometric transformation to the EL decoding unit 240.

[0081] The EL decoding unit 240 decodes the difference between EL and BL from the EL-encoded video encoded as EL. Then, the EL decoding unit 240 outputs the decoded EL video based on the difference video and the rectangular video (BL side) output from the inverse geometric transformation unit 230. The EL decoding unit 240 may have a decoding processing unit 163 of the EL encoding unit 160, and the decoding processing in the decoding processing unit 163 may be the same decoding process.

[0082] Specifically, the EL decoding unit 240 reads the syntax related to the extraction position contained in the EL bitstream. The EL decoding unit 240 confirms the extraction position and, based on the extraction position, reads the rectangular image output from the inverse geometric transformation unit 230 as the predicted image of the region to be decoded. Then, the EL decoding unit 240 obtains the decoded image of the region to be decoded by adding the predicted image to the difference image decoded from the EL encoded image.

[0083] Furthermore, if the EL bitstream contains referential syntax indicating a reference to the BL side in a region outside the EL-side encoding target region, the EL decoding unit 240 decodes the difference value contained in the subsequently received bitstream and decodes a reference vector obtained by adding the difference value to the reference vector used immediately before. The EL decoding unit 240 uses the derived reference vector for prediction and reads the region indicated by this reference vector from the rectangular image output from the inverse geometric transformation unit 230 as the predicted image of the decoding target region. Then, the EL decoding unit 240 obtains the decoded image of the decoding target region by adding the predicted image to the difference image decoded from the EL-encoded image.

[0084] The EL decoding unit 240 may transmit the decoded EL image as an output image to an external device. The decoded image is a rectangular image, and by displaying it on a panoramic display, which is an example of an external device, a curved image without distortion (basically) can be displayed on a spherical surface.

[0085] On the other hand, the decoding device 200 may transmit the decoded video decoded by the BL decoding unit 220 to an external device as the decoded video of the BL. The decoded video on the BL side is a planar video, and unlike a rectangular video, it is basically a distortion-free video, and can be displayed on a flat display such as a television as the decoded video of the BL side.

[0086] Thus, the decoding device 200 according to the first embodiment has an inverse geometric transformation unit 230 between the BL decoding unit 220 and the EL decoding unit 240.

[0087] Specifically, the decoding device 200 has a configuration comprising a BL decoding unit 220 that decodes a two-dimensional image (first decoded image) from an encoded image (first encoded image) encoded as BL, an inverse geometric transformation unit 230 that converts the two-dimensional image into a rectangular image by inverse perspective projection transformation, and an EL decoding unit 240 that decodes the difference image between EL and BL from an encoded image (second encoded image) encoded as EL, and outputs an EL decoded image based on the difference image and the rectangular image output from the inverse geometric transformation unit 230.

[0088] As a result, even if the BL encoded video is a two-dimensional video (a cropped image) and the EL encoded video is a rectangular video, the inverse geometry transformation unit 230 can convert the two-dimensional planar video on the BL side into a rectangular video. Therefore, the EL decoding unit 240 can process both the BL and EL sides as the same type of rectangular video. Consequently, the decoding device 200 can efficiently decode even encoded videos that are rectangular videos (or curved videos that are the source of rectangular videos).

[0089] As explained above, the encoding device 100 can efficiently encode curved images, and the decoding device 200 can efficiently decode curved images. Therefore, the entire video distribution system 10 can also efficiently process curved images.

[0090] [Other embodiments] A program may be provided that causes a computer to perform each of the processes carried out by the above-described devices (encoding device 100 and decoding device 200). The program may be recorded on a computer-readable medium. Using a computer-readable medium, it is possible to install the program on a computer. Here, the computer-readable medium on which the program is recorded may be a non-transient recording medium. The non-transient recording medium is not particularly limited, but may be a recording medium such as a CD-ROM or DVD-ROM. Furthermore, the circuits that perform each of the processes carried out by the above-described devices (encoding device 100 and decoding device 200) may be integrated, and the device may be configured using a semiconductor integrated circuit (chipset, SoC).

[0091] The terms "based on" and "depending on / in response to" used in this disclosure do not mean "based solely on" or "depending solely on" unless otherwise specified. The term "based on" means both "based solely on" and "at least partially on." Similarly, the term "depending on" means both "at least partially on" and "at least partially on." The terms "include," "comprise," and their variations do not mean that only the listed items are included, but that they may include only the listed items or include additional items in addition to the listed items. Furthermore, the term "or" used in this disclosure is not intended to mean exclusive OR. Moreover, any reference to elements using designations such as "first," "second," etc., used in this disclosure does not generally limit the quantity or order of those elements. These designations may be used herein as a convenient way to distinguish between two or more elements. Therefore, references to the first and second elements do not imply that only two elements may be adopted therein, or that the first element must precede the second element in any way. In this disclosure, where articles are added by translation, such as a, an, and the in English, these articles shall be plural unless it is clearly indicated by the context that they are not.

[0092] Although the embodiments have been described in detail above with reference to the drawings, the specific configuration is not limited to those described above, and various design changes can be made without departing from the gist of the invention. Furthermore, it is possible to combine the various operational examples within a range that does not contradict each other.

[0093] (Note) In summary, the above points are as noted in the appendix, but the appendix does not limit the embodiments.

[0094] (Note 1) An encoding device used in a video distribution system, A curved image converted into a rectangular image is used as the input image, and a cropping unit extracts a portion of the input image, A geometric transformation unit that converts a portion of the aforementioned input video into a two-dimensional image by perspective projection, A base layer encoding unit that encodes the aforementioned two-dimensional image as a base layer, An inverse geometric transformation unit converts the decoded image obtained by decoding the encoded two-dimensional image into the rectangular image by inverse perspective projection transformation, The system includes an enhancement layer encoding unit that uses the rectangular image output from the inverse geometric transformation unit as a predicted image and encodes the input image as an enhancement layer. Encoding device.

[0095] (Note 2) The enhancement layer encoding unit uses the following syntax for the extraction position by the extraction unit: The syntax representing the conversion type used when converting the curved image to the rectangular image, A syntax indicating the position of the input video cut out by the aforementioned cutting unit, Encode The encoding device described in Appendix 1.

[0096] (Note 3) The enhancement layer encoding unit uses the following syntax for the reference vector that refers to the base layer: Syntax indicating whether or not the base layer is referenced in an area outside the encoding area of ​​the enhancement layer, The difference between the reference vector used to reference the base layer in an adjacent encoding target region adjacent to the encoding target region of the enhancement layer, and the reference vector used to reference the base layer in that encoding target region, Encode The encoding device described in Appendix 1 or Appendix 2.

[0097] (Note 4) A decoding device used in a video distribution system, A base layer decoding unit that decodes the first decoded video of the base layer from the first encoded video encoded as the base layer, An inverse geometric transformation unit converts the first decoded image into a curved image that has been converted into a rectangular image by inverse perspective projection transformation, The system includes an enhancement layer decoding unit that decodes the difference between the enhancement layer and the base layer from a second encoded image encoded as an enhancement layer, and outputs the decoded image of the enhancement layer based on the difference image and the rectangular image output from the inverse geometric transformation unit. Decoding device.

[0098] (Note 5) A program that causes a computer to function as the encoding device described in Appendix 1 or the decoding device described in Appendix 4. [Explanation of symbols]

[0099] 10: Video distribution system 100: Encoding device 110: Cutting section 120: Geometric transformation section 130: BL encoding unit 140: Local decoding unit 150: Inverse geometric transformation unit 160: EL encoding unit 200: Decoder 220: BL Decoder 230: Inverse geometric transformation unit 240: EL decoding unit

Claims

1. An encoding device used in a video distribution system, A curved image converted into a rectangular image is used as the input image, and a cropping unit extracts a portion of the input image, A geometric transformation unit that converts a portion of the aforementioned input video into a two-dimensional image by perspective projection, A base layer encoding unit that encodes the aforementioned two-dimensional image as a base layer, An inverse geometric transformation unit converts the decoded image obtained by decoding the encoded two-dimensional image into the rectangular image by inverse perspective projection transformation, The system includes an enhancement layer encoding unit that uses the rectangular image output from the inverse geometric transformation unit as a predicted image and encodes the input image as an enhancement layer. Encoding device.

2. The enhancement layer encoding unit uses the following syntax for the extraction position by the extraction unit: The syntax representing the conversion type used when converting the curved image to the rectangular image, A syntax indicating the position of the input video cut out by the aforementioned cutting unit, Encode The encoding device according to claim 1.

3. The enhancement layer encoding unit uses the following syntax for the reference vector that refers to the base layer: Syntax indicating whether or not the base layer is referenced in an area outside the encoding area of ​​the enhancement layer, The difference between the reference vector used to reference the base layer in an adjacent encoding target region adjacent to the encoding target region of the enhancement layer, and the reference vector used to reference the base layer in that encoding target region, Encode The encoding device according to claim 1.

4. A decoding device used in a video distribution system, A base layer decoding unit that decodes the first decoded video of the base layer from the first encoded video encoded as the base layer, An inverse geometric transformation unit converts the first decoded image into a curved image that has been converted into a rectangular image by inverse perspective projection transformation, The system includes an enhancement layer decoding unit that decodes the difference between the enhancement layer and the base layer from a second encoded image encoded as an enhancement layer, and outputs the decoded image of the enhancement layer based on the difference image and the rectangular image output from the inverse geometric transformation unit. Decoding device.

5. A program that causes a computer to function as the encoding device described in claim 1 or the decoding device described in claim 4.

Images