Image encoding / decoding method and computer-readable recording medium

Abstract

Disclosed are an image encoding / decoding method and a computer-readable recording medium. The method for decoding an image using a decoding device includes: receiving a bitstream in which an image is encoded; obtaining index information for specifying a block division type of a current block in the image from the bitstream; determining a block division type of the current block from a candidate group predefined in the decoding device on the basis of the index information, and dividing the current block into a plurality of sub-blocks on the basis of the determined block division type; and decoding each sub-block with reference to syntax information obtained from the bitstream.

Description

[0001] This application is a divisional application of the invention patent application filed on October 10, 2017, with application number 202210624259.8 and entitled "Image Data Encoding / Decoding Method and Computer-Readable Recording Medium". Technical Field

[0002] This invention relates to image data encoding and decoding technology, and more specifically, to methods and apparatus for encoding and decoding 360-degree images of real-world media services. Background Technology

[0003] With the widespread adoption of the internet and mobile devices, and the development of information and communication technologies, the use of multimedia data is rapidly increasing. Recently, there has been a growing demand in various fields for high-quality and high-resolution images, such as high-definition (HD) and ultra-high-definition (UHD) images, as well as for reality media services such as virtual reality and augmented reality. Specifically, the amount of data generated by processing multi-view images captured by multiple camera devices for 360-degree images in virtual reality and augmented reality has increased dramatically, but the performance of image processing systems is insufficient to handle such large amounts of data.

[0004] As mentioned above, in existing image encoding and decoding methods and apparatuses, there is a need to improve the performance of image processing, especially the performance of image encoding / decoding. Summary of the Invention

[0005] Technical issues

[0006] The object of this invention is to provide a method for improving image setup processing in the initial steps of encoding and decoding. More specifically, this invention aims to provide an encoding and decoding method and apparatus for improving image setup processing while taking into account the characteristics of 360-degree images.

[0007] Technical solutions

[0008] According to one aspect of the present invention, a method for decoding 360-degree images is provided.

[0009] Here, the method for decoding a 360-degree image may include: receiving a bitstream including an encoded 360-degree image; generating a prediction image by referring to syntax information obtained from the received bitstream; obtaining a decoded image by combining the generated prediction image with a residual image obtained by inverse quantization and inverse transform of the bitstream; and reconstructing the decoded image into a 360-degree image according to a projection format.

[0010] Here, the syntax information may include projection format information for the 360-degree image.

[0011] Here, the projection format information can be information indicating at least one of the following: Equi-Rectangular Projection (ERP) format, in which a 360-degree image is projected onto a 2D plane; CubeMap Projection (CMP) format, in which a 360-degree image is projected onto a cube; Octahedron Projection (OHP) format, in which a 360-degree image is projected onto an octahedron; and IcoSahedral Projection (ISP) format, in which a 360-degree image is projected onto a polyhedron.

[0012] Here, reconstruction may include: obtaining arrangement information by referring to grammatical information based on regional packing; and rearranging the blocks of the decoded image based on the arrangement information.

[0013] Here, the generation of the predicted image may include: performing image expansion on a reference image obtained by recovering the bitstream; and generating the predicted image by referring to the reference image that has been image expanded.

[0014] Here, performing image expansion may include performing image expansion based on the partitioning units of the reference image.

[0015] Here, performing image expansion based on partitioning units can include generating an expansion region separately for each partitioning unit by using reference pixels of the partitioning unit.

[0016] Here, the extended region can be generated using the boundary pixels of the partition unit that is spatially adjacent to the partition unit to be extended, or using the boundary pixels of the partition unit that has image continuity with the partition unit to be extended.

[0017] Here, performing image expansion based on partitioning units can include generating an expanded image of the combined regions using the boundary pixels of regions that are spatially adjacent to each other in the partitioning units.

[0018] Here, performing image expansion based on partitioning units can include: using all neighboring pixel information of spatially adjacent partitioning units to generate an expanded region between adjacent partitioning units.

[0019] Here, performing image expansion based on partitioning units can include generating an expanded region using the average value of neighboring pixels of spatially adjacent partitioning units.

[0020] Here, the generation of the predicted image may include: obtaining a motion vector candidate group from motion information included in the syntactic information, the motion vector candidate group including motion vectors of blocks adjacent to the current block to be decoded; deriving a predicted motion vector from the motion vector candidate group based on selection information extracted from the motion information; and determining the predicted block of the current block to be decoded by using a final motion vector derived by adding the predicted motion vector to the differential motion vector extracted from the motion information.

[0021] Here, the motion vector candidate group may include only the motion vectors of the following blocks in the adjacent blocks, provided that the blocks adjacent to the current block are different from the face to which the current block belongs, and that the block belongs to the face with image continuity to which the current block belongs.

[0022] Here, the block adjacent to the current block can refer to the block that is adjacent to the current block in at least one of the following directions: top left, top, top right, bottom, and bottom left.

[0023] Here, the final motion vector can be based on the reference area included in at least one reference image and set in an area with image continuity between faces according to the projection format, as indicated by the current block.

[0024] Here, the reference image can be extended in the upward, downward, leftward, and rightward directions based on the image continuity according to the projection format, and then the reference area can be set.

[0025] Here, a reference image can be extended to a face, and a reference region can be set across the face boundary.

[0026] Here, motion information may include at least one of the following: a list of reference images to which the reference image belongs, an index of the reference image, and a motion vector indicating the reference area.

[0027] Here, the generation of the prediction block for the current block may include: dividing the current block into multiple sub-blocks, and generating a prediction block for each of the multiple sub-blocks generated by the division.

[0028] According to another aspect of the present invention, a method for decoding a 360-degree image is provided. The method may include: receiving a bitstream encoded with the 360-degree image, the bitstream including data of an extended two-dimensional image, the extended two-dimensional image including a two-dimensional image and a predetermined extended region, and the two-dimensional image being projected from an image having a three-dimensional projection structure and including at least one face; and reconstructing the extended two-dimensional image by decoding the data of the extended two-dimensional image, wherein the size of the extended region is determined based on width information indicating the width of the extended region, the width information being obtained from the bitstream, wherein sample values ​​of the extended region are determined differently according to a padding method selected from a plurality of padding methods, wherein reconstructing the extended two-dimensional image includes generating a prediction image, wherein the prediction image is generated by selecting a prediction mode among a plurality of prediction modes including intra-frame prediction and inter-frame prediction and performing prediction based on the selected prediction mode, and information about the selected prediction mode is obtained from the bitstream.

[0029] According to another aspect of the present invention, a method for encoding a 360-degree image is provided. The method may include: obtaining a two-dimensional image projected from an image having a three-dimensional projection structure and including at least one face; obtaining an extended two-dimensional image including the two-dimensional image and a predetermined extended region; and encoding data of the extended two-dimensional image into a bitstream encoding the 360-degree image, wherein the size of the extended region is encoded based on width information indicating the width of the extended region, the width information being encoded into the bitstream, wherein sample values ​​of the extended region are determined differently according to a padding method selected from a plurality of padding methods, wherein encoding the data of the extended two-dimensional image includes generating a prediction image, wherein the prediction image is generated by selecting a prediction mode from a plurality of prediction modes including intra-frame prediction and inter-frame prediction and performing prediction based on the selected prediction mode, and information about the selected prediction mode is obtained from the bitstream.

[0030] According to another aspect of the invention, a non-transitory computer-readable recording medium for storing a bitstream generated by a method for encoding a 360-degree image is provided. The method may include: obtaining a two-dimensional image projected from an image having a three-dimensional projection structure and including at least one face; obtaining an extended two-dimensional image including the two-dimensional image and a predetermined extended region; and encoding data of the extended two-dimensional image into a bitstream encoding the 360-degree image, wherein the size of the extended region is encoded based on width information indicating the width of the extended region, which is encoded into the bitstream, wherein sample values ​​of the extended region are determined differently according to a padding method selected from a plurality of padding methods, wherein encoding the data of the extended two-dimensional image includes generating a prediction image, wherein the prediction image is generated by selecting a prediction mode from a plurality of prediction modes including intra-frame prediction and inter-frame prediction and performing prediction based on the selected prediction mode, and information about the selected prediction mode is obtained from the bitstream.

[0031] Beneficial effects of the present invention

[0032] The image encoding / decoding method and apparatus according to embodiments of the present invention can enhance compression performance. In particular, compression performance can be enhanced for 360-degree images. Attached Figure Description

[0033] Figure 1 This is a block diagram of an image encoding apparatus according to an embodiment of the present invention.

[0034] Figure 2 This is a block diagram of an image decoding apparatus according to an embodiment of the present invention.

[0035] Figure 3 This is an example diagram of dividing image information into multiple layers to compress the image.

[0036] Figure 4 This is a conceptual diagram illustrating an example of image segmentation according to an embodiment of the present invention.

[0037] Figure 5 This is another example diagram of the image segmentation method according to an embodiment of the present invention.

[0038] Figure 6 This is an example diagram of a common image resizing method.

[0039] Figure 7 This is an example diagram of image size adjustment according to an embodiment of the present invention.

[0040] Figure 8 This is an example diagram illustrating a method for constructing an expanded region in an image resizing method according to an embodiment of the present invention.

[0041] Figure 9 This is an example diagram of a method for constructing the region to be deleted and the region to be generated in an image resizing method according to an embodiment of the present invention.

[0042] Figure 10 This is an example diagram of image reconstruction according to an embodiment of the present invention.

[0043] Figure 11 This is an example diagram showing images before and after image setting processing according to an embodiment of the present invention.

[0044] Figure 12 This is an example diagram illustrating the adjustment of the size of each segmentation unit of an image according to an embodiment of the present invention.

[0045] Figure 13 This is an example diagram of the settings or size adjustments of the division units in an image.

[0046] Figure 14 This is an example diagram illustrating both the process of adjusting the image size and the process of adjusting the size of the partitioned units in the image.

[0047] Figure 15 This is an example diagram showing a 3D space and a 2D planar space for displaying a three-dimensional (3D) image.

[0048] Figures 16a to 16d This is a conceptual diagram illustrating a projection format according to an embodiment of the present invention.

[0049] Figure 17 This is a conceptual diagram illustrating the projection format included in a rectangular image according to an embodiment of the present invention.

[0050] Figure 18 This is a conceptual diagram of a method for converting a projection format into a rectangular shape according to an embodiment of the present invention, namely, a method for performing rearrangement on a surface to exclude meaningless regions.

[0051] Figure 19 This is a conceptual diagram illustrating the execution of a region-wise packing process according to an embodiment of the present invention to convert a CMP projection format into a rectangular image.

[0052] Figure 20 This is a conceptual diagram of 360-degree image segmentation according to an embodiment of the present invention.

[0053] Figure 21 This is an example diagram of 360-degree image segmentation and image reconstruction according to an embodiment of the present invention.

[0054] Figure 22This is an example of how an image packaged or projected by CMP is divided into tiles.

[0055] Figure 23 This is a conceptual diagram illustrating an example of adjusting the size of a 360-degree image according to an embodiment of the present invention.

[0056] Figure 24 This is a conceptual diagram illustrating the continuity between surfaces under a projection format (e.g., CHP, OHP, or ISP) according to an embodiment of the present invention.

[0057] Figure 25 This is a conceptual diagram showing the continuity of the surfaces of part 21c, which is an image obtained through image reconstruction processing in CMP projection format or region packing processing.

[0058] Figure 26 This is an example diagram illustrating image size adjustment under the CMP projection format according to an embodiment of the present invention.

[0059] Figure 27 This is an example diagram illustrating the resizing of an image transformed and packaged in CMP projection format according to an embodiment of the present invention.

[0060] Figure 28 This is an example diagram illustrating a data processing method for adjusting the size of a 360-degree image according to an embodiment of the present invention.

[0061] Figure 29 This is an example diagram showing a tree-based block format.

[0062] Figure 30 This is an example diagram showing a type-based block format.

[0063] Figure 31 This is an example diagram showing various types of blocks that can be obtained by the block division section of the present invention.

[0064] Figure 32 This is an example diagram illustrating tree-based partitioning according to an embodiment of the present invention.

[0065] Figure 33 This is an example diagram illustrating tree-based partitioning according to an embodiment of the present invention.

[0066] Figure 34 This is an example diagram illustrating various cases of obtaining predicted blocks through inter-frame prediction.

[0067] Figure 35 This is an example diagram illustrating the composition of a list of reference pictures according to an embodiment of the present invention.

[0068] Figure 36This is a conceptual diagram illustrating a non-translation motion model according to an embodiment of the present invention.

[0069] Figure 37 This is an example diagram illustrating motion estimation in a sub-block unit according to an embodiment of the present invention.

[0070] Figure 38 This is an example diagram illustrating a block referenced in motion information prediction of the current block according to an embodiment of the present invention.

[0071] Figure 39 This is an example diagram illustrating a block used for predicting motion information of the current block in a non-translational motion model according to an embodiment of the present invention.

[0072] Figure 40 This is an example diagram illustrating the use of extended images to perform inter-frame prediction according to an embodiment of the present invention.

[0073] Figure 41 This is a conceptual diagram illustrating an extension of a surface element according to an embodiment of the present invention.

[0074] Figure 42 This is an example diagram illustrating the use of extended images to perform inter-frame prediction according to an embodiment of the present invention.

[0075] Figure 43 This is an example diagram illustrating the use of extended reference images to perform inter-frame prediction according to an embodiment of the present invention.

[0076] Figure 44 This is an example diagram illustrating the composition of motion information prediction candidate groups for inter-frame prediction in 360-degree images according to an embodiment of the present invention. Detailed Implementation

[0077] While the invention can be subject to various modifications and alternatives, specific embodiments thereof are shown by way of example in the accompanying drawings and will be described in detail herein. However, it should be understood that the invention is not intended to be limited to the specific forms disclosed, but rather, the invention covers all modifications, equivalents, and alternatives falling within the spirit and scope of the invention.

[0078] It should be understood that although the terms “first,” “second,” etc., may be used herein to describe various elements, these elements should not be limited by these terms. These terms are used only to distinguish one element from another. For example, without departing from the scope of the invention, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element. As used herein, the term “and / or” includes any and all combinations of one or more of the associated listed items.

[0079] It should be understood that when an element is referred to as "connected" or "coupled" to another element, it can be directly connected to or coupled to the other element, or there may be intermediate elements. In contrast, when an element is referred to as "directly connected" or "directly coupled" to another element, there are no intermediate elements. Other words used to describe the relationship between elements should be interpreted in a similar manner (i.e., "between" and "directly between," "adjacent" and "directly adjacent," etc.).

[0080] The technical terms used herein are for the purpose of describing particular embodiments only and are not intended to limit the invention. As used herein, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the terms “comprises,” “comprising,” “includes,” and / or “including” as used herein specify the presence of the stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0081] Unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in common dictionaries should be interpreted as having the meaning consistent with their meaning in the context of the relevant field and will not be interpreted in an idealized or overly formal sense unless expressly defined herein.

[0082] Both the image encoding device and the image decoding device can be user terminals such as personal computers (PCs), laptops, personal digital assistants (PDAs), portable multimedia players (PMPs), portable game consoles (PSPs), wireless communication terminals, smartphones and televisions, virtual reality (VR) devices, augmented reality (AR) devices, mixed reality (MR) devices, head-mounted display (HMD) devices, and smart glasses, or server terminals such as application servers and service servers. They can also include various devices such as communication modems for communicating with various devices or wired / wireless communication networks; memory for storing various programs and data for encoding or decoding images or executing inter-frame or intra-frame prediction for encoding or decoding; and processors for executing programs to perform computational and control operations. Furthermore, images encoded into a bitstream by the image encoding device can be transmitted to the image decoding device in real-time or non-real-time via wired / wireless communication networks such as the Internet, short-range wireless networks, wireless local area networks (LANs), WiBro networks, and mobile communication networks, or via various communication interfaces such as cables and universal serial buses (USB). Then, the bitstream can be decoded by an image decoding device to restore and replay the bitstream as an image.

[0083] Furthermore, an image encoded into a bitstream by an image encoding device can be transmitted from the image encoding device to the image decoding device via a computer-readable recording medium.

[0084] The aforementioned image encoding and decoding devices can be separate devices, but they can be configured as a single image encoding / decoding device depending on the implementation method. In this case, some elements of the image encoding device can be substantially the same as some elements of the image decoding device, and can be implemented to include at least the same structure or perform the same function.

[0085] Therefore, redundant descriptions of the corresponding technical elements will be omitted in the following detailed description of the technical elements and their working principles.

[0086] Furthermore, the image decoding device corresponds to the computing device that applies the image encoding method executed by the image encoding device to the decoding process, and therefore the following description will focus on the image encoding device.

[0087] The computing device may include: a memory configured to store programs or software patterns for implementing image encoding methods and / or image decoding methods; and a processor connected to the memory to execute the programs. Furthermore, the image encoding device may also be called an encoder, and the image decoding device may also be called a decoder.

[0088] Typically, an image can consist of a series of still images. Still images can be classified into groups of pictures (GOPs), and each still image can be referred to as a picture. In this case, a picture can indicate one of the frames and fields in progressive and interlaced signals. When encoding / decoding is performed on a frame-based basis, a picture can be represented as a "frame," and when encoding / decoding is performed on a field-based basis, a picture can be represented as a "field." This invention assumes progressive signals but can also be applied to interlaced signals. As a higher concept, units such as GOPs and sequences can exist, and each picture can also be divided into predetermined regions such as slices, tiles, blocks, etc. Furthermore, a GOP can include units such as I-pictures, P-pictures, and B-pictures. An I-picture can refer to a picture that is autonomously encoded / decoded without using a reference picture, and P-pictures and B-pictures can refer to pictures that are encoded / decoded by performing processes such as motion estimation and motion compensation using a reference picture. Typically, a P-picture can use I-pictures and B-pictures as reference pictures, and a B-picture can use I-pictures and P-pictures as reference pictures. However, the above definitions can also be changed by the encoding / decoding settings.

[0089] Here, the image referenced during encoding / decoding is called the reference image, and the block or pixel referenced during encoding / decoding is called the reference block or reference pixel. Furthermore, the reference data can include various types of encoding / decoding information and frequency domain coefficients, as well as spatial domain pixel values, generated and determined during the encoding / decoding process. For example, the reference data may correspond to intra-frame prediction information or motion information in the prediction unit, transform information in the transform / inverse transform unit, quantization information in the quantization / inverse quantization unit, encoding / decoding information (context information) in the encoding / decoding unit, filter information in the in-loop filter unit, etc.

[0090] The smallest unit of an image can be a pixel, and the number of bits used to represent a pixel is called the bit depth. Typically, the bit depth can be 8 bits, and bit depths of 8 bits or more can be supported depending on the encoding settings. At least one bit depth can be supported based on the color space. Furthermore, at least one color space can be included based on the image color format. One or more images with the same size or one or more images with different sizes can be included based on the color format. For example, YCbCr 4:2:0 can consist of one luminance component (Y in this example) and two chrominance components (Cb / Cr in this example). In this case, the composition ratio of the chrominance component to the luminance component can be 1:2 in width and height. As another example, YCbCr 4:4:4 can have the same composition ratio in width and height. Similar to the example above, when including one or more color spaces, the image can be divided into color spaces.

[0091] This invention will be described based on any color space (Y in this example) of any color format (YCbCr), and this description will be applied in the same or similar manner (depending on the settings of the specific color space) to other color spaces of that color format (Cb and Cr in this example). However, each color space may be given partial differences (independent of the settings of the specific color space). That is, settings dependent on each color space may refer to settings that are proportional to or dependent on the composition ratio of each component (e.g., 4:2:0, 4:2:2, or 4:4:4), and settings independent of each color space may refer to settings of the corresponding color space that are independent of or unrelated to the composition ratio of each component. In this invention, some elements may have independent settings or dependent settings, depending on the encoder / decoder.

[0092] The setup information or syntax elements required during image encoding processing can be determined hierarchically by units such as video, sequence, picture, slice, tile, and block. These units include video parameter sets (VPS), sequence parameter sets (SPS), picture parameter sets (PPS), slice headers, tile headers, and block headers. The encoder can add units to the bitstream and send the bitstream to the decoder. The decoder can parse the bitstream at the same level, recover the setup information sent by the encoder, and use the setup information in image decoding processing. Furthermore, relevant information can be sent via the bitstream in the form of supplementary enhancement information (SEI) or metadata, which can then be parsed and used. Each parameter set has a unique ID value, and a lower-level parameter set can have the ID value of a higher-level parameter set to be referenced. For example, a lower-level parameter set can reference information from one or more higher-level parameter sets with corresponding ID values. In the various examples of the units described above, when any unit comprises one or more different units, any unit can be referred to as a higher-level unit, and the included units can be referred to as lower-level units.

[0093] Setting information occurring within such units can include settings independent of each unit or settings dependent on previous, subsequent, or higher-level units. Here, it will be understood that dependent settings use tagging information corresponding to the settings of previous, subsequent, or higher-level units (e.g., a 1-bit flag; 1 indicates following, and 0 indicates not following) to indicate the setting information of the corresponding unit. In this invention, examples of independent settings will be focused on to describe the setting information. However, examples may also be included where relationships between the setting information of previous, subsequent, or higher-level units dependent on the current unit are added to or replace independent settings.

[0094] Figure 1This is a block diagram of an image encoding apparatus according to an embodiment of the present invention. Figure 2 This is a block diagram of an image decoding apparatus according to an embodiment of the present invention.

[0095] Reference Figure 1 The image encoding apparatus may be configured to include a prediction unit, a subtractor, a transform unit, a quantization unit, an inverse quantization unit, an inverse transform unit, an adder, an in-loop filter unit, a memory, and / or an encoding unit, some of which may not necessarily be included. Depending on the implementation, some or all of these elements may be selectively included, and additional elements not shown herein may be included.

[0096] Reference Figure 2 The image decoding apparatus may be configured to include a decoding unit, a prediction unit, an inverse quantization unit, an inverse transform unit, an adder, an in-loop filter unit, and / or a memory, some of which may not necessarily be included. Depending on the implementation, some or all of these elements may be selectively included, and some additional elements not shown herein may be included.

[0097] Image encoding and decoding devices can be separate devices, but they can also be configured as a single image encoding / decoding device depending on the implementation. In this case, some elements of the image encoding device can be substantially the same as some elements of the image decoding device, and can be implemented to include at least the same structure or perform the same function. Therefore, redundant descriptions of the corresponding technical elements will be omitted in the following detailed description of the technical elements and their working principles. The image decoding device corresponds to the computing device that applies the image encoding method performed by the image encoding device to the decoding process; therefore, the following description will focus on the image encoding device. The image encoding device can also be called an encoder, and the image decoding device can also be called a decoder.

[0098] The prediction unit can be implemented using a prediction module and can generate prediction blocks by performing intra-frame prediction or inter-frame prediction on the block to be encoded. The prediction unit generates prediction blocks by predicting the current block to be encoded in the image. In other words, the prediction unit can predict the pixel values ​​of the pixels of the current block to be encoded in the image through intra-frame prediction or inter-frame prediction to generate prediction blocks with the predicted pixel values ​​of the pixels. Furthermore, the prediction unit can deliver the information needed to generate prediction blocks to the encoding unit, enabling the encoding of prediction mode information. The encoding unit adds the corresponding information to the bitstream and transmits the bitstream to the decoder. The decoding unit of the decoder can parse the corresponding information, recover the prediction mode information, and then use the prediction mode information to perform intra-frame prediction or inter-frame prediction.

[0099] The subtractor subtracts the prediction block from the current block to generate a residual block. In other words, the subtractor can calculate the difference between the pixel value of each pixel in the current block to be encoded and the predicted pixel value of each pixel in the prediction block generated by the prediction unit to generate a residual block as a block-type residual signal.

[0100] The transform unit can transform a signal belonging to the spatial domain into a signal belonging to the frequency domain. In this case, the signal obtained through the transform process is called the transform coefficient. For example, a residual block with a residual signal delivered from a subtractor can be transformed into a transform block with transform coefficients. In this case, the input signal is determined according to the encoding settings, and the input signal is not limited to the residual signal.

[0101] The transform unit can perform transformations on the residual blocks using transformation techniques such as the Hadamard Transform, the Discrete Sine Transform (DST), and the Discrete Cosine Transform (DCT). However, the invention is not limited to this, and various enhanced and modified transformation techniques can be used.

[0102] For example, at least one transformation technique can be supported, and at least one detailed transformation technique can be supported within each transformation technique. In this case, the at least one detailed transformation technique can be a transformation technique that constructs some basis vectors differently in each transformation technique. For example, as transformation techniques, DST-based transformations and DCT-based transformations can be supported. For DST, detailed transformation techniques such as DST-I, DST-II, DST-III, DST-V, DST-VI, DST-VII, and DST-VIII can be supported, and for DCT, detailed transformation techniques such as DCT-I, DCT-II, DCT-III, DCT-V, DCT-VI, DCT-VII, and DCT-VIII can be supported.

[0103] One of the transformation techniques can be set as the default transformation technique (e.g., one transformation technique && one detailed transformation technique), and additional transformation techniques can be supported (e.g., multiple transformation techniques || multiple detailed transformation techniques). Whether additional transformation techniques are supported can be determined at the unit level (sequence, image, slice, or tile), and related information can be generated based on the unit. When additional transformation techniques are supported, transformation technique selection information can be determined at the block level, and related information can be generated.

[0104] Transformations can be performed horizontally and / or vertically. For example, a two-dimensional (2D) transformation can be performed by using a one-dimensional (1D) transformation horizontally and vertically with basis vectors, so that pixel values ​​in the spatial domain can be transformed into the frequency domain.

[0105] Furthermore, transformations can be performed adaptively, horizontally and / or vertically. Specifically, whether to perform a transformation adaptively can be determined based on at least one coding setting. For intra-frame prediction, for example, DCT-I can be applied horizontally and DST-I can be applied vertically when the prediction mode is horizontal; DST-VI can be applied horizontally and DCT-VI can be applied vertically when the prediction mode is vertical; DCT-II can be applied horizontally and DCT-V can be applied vertically when the prediction mode is diagonally downward to the left; and DST-I can be applied horizontally and DST-VI can be applied vertically when the prediction mode is diagonally downward to the right.

[0106] The size and shape of a transform block can be determined based on the encoding costs of candidates for the size and shape of the transform block. The image data of the transform block and information about the determined size and shape of the transform block can then be encoded.

[0107] Within the transformation options, a square transformation can be set as the default, and additional transformation options (such as a rectangle) can be supported. Whether additional transformation options are supported can be determined at the unit level (sequence, image, slice, or tile), and related information can be generated based on the unit. Transformation selection information can be determined at the block level, and related information can be generated.

[0108] Furthermore, it can be determined whether a transform block format is supported based on the encoding information. In this case, the encoding information may correspond to slice type, encoding mode, block size and shape, block partitioning scheme, etc. That is, a transform format can be supported based on at least one piece of encoding information, and multiple transform formats can be supported based on at least one piece of encoding information. The former may be an implicit situation, while the latter may be an explicit situation. For the explicit situation, adaptive selection information indicating the best candidate group to be selected from multiple candidate groups can be generated, and this adaptive selection information is added to the bitstream. According to the present invention, in addition to this example, it will be understood that when encoding information is explicitly generated, the information is added to the bitstream in various units, and the relevant information is parsed by the decoder in various units and the relevant information is recovered into decoded information. Furthermore, it should be understood that when encoding / decoding information is implicitly processed, the encoder and decoder perform the processing through the same processing, rules, etc.

[0109] As an example, support for rectangular transformations can be determined based on the slice type. For I slices, supported transformations can be square transformations, and for P / B slices, supported transformations can be either square or rectangular transformations.

[0110] As an example, support for rectangular transforms can be determined based on the coding mode. For intra-frame prediction, the supported transform can be a square transform, and for inter-frame prediction, the supported transform can be a square transform and / or a rectangular transform.

[0111] As an example, support for rectangular transformations can be determined based on the size and shape of the block. Blocks of a certain size or larger support square transformations, while blocks smaller than a certain size support both square and / or rectangular transformations.

[0112] As an example, support for rectangular transformations can be determined based on the block partitioning scheme. When the block to be transformed is obtained through a quadtree partitioning scheme, the supported transformation can be a square transformation. When the block to be transformed is obtained through a binary tree partitioning scheme, the supported transformation can be either a square transformation or a rectangular transformation.

[0113] The above examples are examples of support for transform forms based on a single encoded piece of information, and multiple pieces of information can be combined and associated with additional transform form support settings. The above examples are merely examples of additional transform form support based on various encoding settings. However, the invention is not limited thereto and various modifications can be made thereto.

[0114] Transform processing can be omitted based on encoding settings or image characteristics. For example, transform processing (including inverse processing) can be omitted based on encoding settings (e.g., assuming a lossless compression environment in this example). As another example, transform processing can be omitted when the compression performance through the transform is not shown based on image characteristics. In this case, the transform can be omitted for all cells or one of the horizontal and vertical cells. Whether omission is supported can be determined based on the size and shape of the block.

[0115] For example, suppose horizontal and vertical transformations are set to be jointly omitted. When the transformation omission flag is 1, the transformation can be performed neither horizontally nor vertically, and when the transformation omission flag is 0, the transformation can be performed both horizontally and vertically. On the other hand, suppose horizontal and vertical transformations are set to be omitted independently. When the first transformation omission flag is 1, the horizontal transformation is not performed, and when the first transformation omission flag is 0, the horizontal transformation is performed. Moreover, when the second transformation omission flag is 1, the vertical transformation is not performed, and when the second transformation omission flag is 0, the vertical transformation is performed.

[0116] Transform omitting is supported when the block size corresponds to range A, but not when the block size corresponds to range B. For example, transform omitting is not supported when the block width is greater than M or the block height is greater than N. Transform omitting is supported when the block width is less than m or the block height is less than n. M(m) and N(n) can be the same or different from each other. The settings associated with the transform can be determined at the unit level, such as sequences, images, slices, etc.

[0117] When additional transform techniques are supported, the transform technique settings can be determined based on at least one piece of encoding information. In this case, the encoding information may correspond to slice type, encoding mode, block size and shape, prediction mode, etc.

[0118] As an example, support for transform techniques can be determined based on the coding mode. For intra-frame prediction, supported transform techniques may include DCT-I, DCT-III, DCT-VI, DST-II, and DST-III, and for inter-frame prediction, supported transform techniques may include DCT-II, DCT-III, and DST-III.

[0119] As an example, the supported transformation techniques can be determined based on the slice type. For I slices, supported transformation techniques may include DCT-I, DCT-II, and DCT-III; for P slices, supported transformation techniques may include DCT-V, DST-V, and DST-VI; and for B slices, supported transformation techniques may include DCT-I, DCT-II, and DST-III.

[0120] As an example, support for transformation techniques can be determined based on the prediction mode. Prediction mode A supports transformation techniques such as DCT-I and DCT-II, prediction mode B supports transformation techniques such as DCT-I and DST-I, and prediction mode C supports transformation techniques such as DCT-I. In this case, prediction modes A and B can both be directional modes, and prediction mode C can be a non-directional mode.

[0121] As an example, the supported transformation techniques can be determined based on the size and shape of the block. Transformation techniques supported by blocks of a specific size or larger may include DCT-II; transformation techniques supported by blocks smaller than a specific size may include DCT-II and DST-V; and transformation techniques supported by blocks of a specific size or larger, as well as blocks smaller than a specific size, may include DCT-I, DCT-II, and DST-I. Furthermore, transformation techniques supported in a square shape may include DCT-I and DCT-II, and transformation techniques supported in a rectangular shape may include DCT-I and DST-I.

[0122] The above examples can be seen as examples of support for transform techniques based on a single encoded piece of information, and multiple pieces of information can be combined and associated with additional transform technique support settings. The present invention is not limited to the above examples and modifications can be made to them. Furthermore, the transform unit can deliver information required to generate a transform block to the encoding unit, thereby encoding the information. The encoding unit adds the corresponding information to the bitstream and transmits the bitstream to the decoder. The decoding unit of the decoder can parse the information and use the parsed information in the inverse transform process.

[0123] The quantization unit can quantize the input signal. In this case, the signal obtained through quantization processing is called the quantization coefficient. For example, the quantization unit can quantize a residual block with residual transform coefficients delivered from the transform unit to obtain a quantized block with quantization coefficients. In this case, the input signal is determined according to the encoding settings and is not limited to the residual transform coefficients.

[0124] The quantization unit can use quantization techniques such as dead zone uniform threshold quantization and quantization weighting matrices to quantize the transformed residual blocks. However, the invention is not limited to this, and various improved and modified quantization techniques can be used. Whether additional quantization techniques are supported can be determined on a unit basis, such as a sequence, image, slice, or tile, and relevant information can be generated based on the unit. When additional quantization techniques are supported, quantization technique selection information can be determined on a block basis, and relevant information can be generated.

[0125] When additional quantization techniques are supported, the quantization technique settings can be determined based on at least one piece of encoding information. In this case, the encoding information may correspond to slice type, encoding mode, block size and shape, prediction mode, etc.

[0126] For example, the quantization unit can set different quantization weighting matrices corresponding to the coding mode and different weighting matrices applied according to inter-frame prediction / intra-frame prediction. Furthermore, the quantization unit can set different weighting matrices applied according to the intra-frame prediction mode. In this case, assuming the quantization weighting matrix has a size of M×N, the same as the size of the quantization block, the quantization weighting matrix can be a quantization matrix constructed differently for some quantization components.

[0127] Quantization can be omitted depending on encoding settings or image characteristics. For example, quantization (including inverse quantization) can be omitted based on encoding settings (e.g., in this example, assuming a lossless compression environment). As another example, quantization can be omitted when the compression performance after quantization is not shown based on image characteristics. In this case, some or all of the region can be omitted, and whether omission is supported can be determined based on the size and shape of the block.

[0128] Information about quantization parameters (QPs) can be generated in units such as sequences, images, slices, tiles, or blocks. For example, information about QPs can be generated in the higher-level unit where QP information is first generated. <1> A default QP is set in the middle, and the QP can be set to a value that is the same as or different from the QP set in the higher-level unit. The QP can be ultimately determined during quantization processing performed in some units through this process. In this case, units such as sequences and images can be... <1> Corresponding examples, such as slices, tiles, and blocks, can be units that are... <2> Corresponding examples, and units such as blocks can be related to <3> Corresponding example.

[0129] Information about QPs can be generated based on the QPs in each unit. Alternatively, a predetermined QP can be set as a predicted value, and information about the difference between the predicted QP and the QP in the unit can be generated. Alternatively, a QP obtained based on at least one of the QPs set in the parent unit, the same and previous units, or the adjacent units can be set as a predicted value, and information about the difference between the predicted QP and the QP in the current unit can be generated. Alternatively, a QP set in the parent unit and a QP obtained based on at least one piece of coding information can be set as predicted values, and information about the difference between the predicted QP and the QP in the current unit can be generated. In this case, the same and previous units can be units that can be defined according to the order in which units are encoded, adjacent units can be spatially adjacent units, and the coding information can be the slice type, coding mode, prediction mode, position information, etc. of the corresponding unit.

[0130] As an example, the QP in the current cell can be used to set the QP in the higher-level cell as the predicted value and generate difference information. Information about the difference between the QP set in the slice and the QP set in the image can be generated, or information about the difference between the QP set in the tile and the QP set in the image can be generated. Furthermore, information about the difference between the QP set in the block and the QP set in the slice or tile can be generated. Additionally, information about the difference between the QP set in the sub-block and the QP set in the block can be generated.

[0131] As an example, a QP obtained based on at least one QP in a neighboring cell or at least one QP in a previous cell can be used as the prediction value, and difference information can be generated. Information about the difference between the QP obtained and the QP obtained based on the QPs of neighboring blocks such as the left, top-left, bottom-left, top, and top-right sides of the current block can be generated. Alternatively, information about the difference between the QP and the QP of encoded images preceding the current image can be generated.

[0132] As an example, the QP in the current cell can be used to set the QP in the upper cell and the QP obtained based on at least one piece of coding information as the prediction value, and difference information can be generated. Furthermore, information about the difference between the QP in the current block and the QP of a slice corrected according to slice type (I / P / B) can be generated. Alternatively, information about the difference between the QP in the current block and the QP of a tile corrected according to coding mode (intra-frame / inter-frame) can be generated. Alternatively, information about the difference between the QP in the current block and the QP of a picture corrected according to prediction mode (directional / non-directional) can be generated. Alternatively, information about the difference between the QP in the current block and the QP of a picture corrected according to position information (x / y) can be generated. In this case, the correction can be an operation that adds an offset to or subtracts an offset from the QP used for prediction in the upper cell. In this case, at least one offset information can be supported according to the coding settings, and information that is implicitly processed or explicitly associated can be generated according to predetermined processing. The present invention is not limited to the above examples and modifications can be made to the above examples.

[0133] The above examples are permissible when a signal indicating a change in QP is provided or activated. For instance, when neither a signal indicating a change in QP is provided nor activated, no difference information is generated, and the predicted QP can be determined for each cell. As another example, when a signal indicating a change in QP is provided or activated, difference information is generated, and when the value of the difference information is 0, the predicted QP can be determined for each cell.

[0134] The quantization unit delivers the information needed to generate the quantized block to the encoding unit, enabling the information to be encoded. The encoding unit adds the corresponding information to the bitstream and transmits the bitstream to the decoder. The decoder's decoding unit can parse the information and use the parsed information in the inverse quantization process.

[0135] The above example has been described assuming that the residual block is transformed and quantized by the transform and quantization units. However, the residual signal of the residual block can be transformed into a residual block with transform coefficients without performing quantization. Alternatively, only quantization can be performed without transforming the residual signal of the residual block into transform coefficients. Alternatively, neither transform nor quantization can be performed. This can be determined based on the encoding settings.

[0136] The encoding unit can scan the quantization coefficients, transform coefficients, or residual signals of the generated residual blocks in at least one scanning order (e.g., zigzag scanning, vertical scanning, horizontal scanning, etc.), generate a quantization coefficient string, transform coefficient string, or signal string, and encode the quantization coefficient string, transform coefficient string, or signal string using at least one entropy coding technique. In this case, information about the scanning order can be determined according to the encoding settings (e.g., encoding mode, prediction mode, etc.), and the information about the scanning order can be used to generate implicitly determined information or explicitly associated information. For example, a scanning order can be selected from multiple scanning orders according to the intra-frame prediction mode.

[0137] Furthermore, the encoding unit can generate encoded data including the encoded information delivered from each element, and can output the encoded data as a bitstream. This can be implemented using a multiplexer (MUX). In this case, methods such as Exponential Columbus, Context Adaptive Variable Length Coding (CAVLC), and Context Adaptive Binary Arithmetic Coding (CABAC) can be used as encoding techniques to perform encoding. However, the present invention is not limited to this, and various encoding techniques obtained by improving and modifying the above-described encoding techniques can be used.

[0138] When entropy encoding (e.g., CABAC in this example) is performed on syntax elements such as information generated through encoding / decoding processes and residual block data, the entropy encoding apparatus may include a binarizer, a context modeler, and a binary arithmetic encoder. In this case, the binary arithmetic encoder may include a regular encoding engine and a bypass encoding engine.

[0139] The syntax elements input to the entropy encoding device do not have to be binary values. Therefore, when the syntax element is not a binary value, the binarizer can binarize the syntax element and output a binary bit string (binstring) consisting of 0s and 1s. In this case, the binary bits (bin) represent bits consisting of 0s and 1s and can be encoded by a binary arithmetic encoder. In this case, one of the regular encoding engine and the bypass encoding engine can be selected based on the probability of 0s and 1s occurring, and this can be determined according to the encoding / decoding settings. When the syntax element is data where the frequency of 0s is equal to the frequency of 1s, the bypass encoding engine can be used; otherwise, the regular encoding engine can be used.

[0140] Various methods can be used when binarizing grammatical elements. For example, fixed-length binarization, unary binarization, truncated Rice binarization, and K-order exponential Golomb binarization can be used. Furthermore, signed or unsigned binarization can be performed depending on the range of values ​​of the grammatical element. The binarization processing of grammatical elements according to the present invention can include additional binarization methods as well as the binarization described in the examples above.

[0141] The inverse quantization unit and the inverse transform unit can be implemented by reversing the processes performed in the transform unit and the quantization unit. For example, the inverse quantization unit can inverse quantize the transform coefficients quantized by the quantization unit, and the inverse transform unit can inverse transform the inverse quantized transform coefficients to generate the recovered residual block.

[0142] The adder adds the predicted block and the recovered residual block to recover the current block. The recovered block can be stored in memory and used as reference data (for the prediction section, filter section, etc.).

[0143] The in-loop filter unit can additionally perform one or more post-processing filtering operations such as deblocking filter, Sample Adaptive Offset (SAO), and Adaptive Loop Filter (ALF). The deblocking filter can remove block distortion generated at the boundaries between blocks from the restored image. ALF can perform filtering based on values ​​obtained by comparing the input image with the restored image. Specifically, ALF can perform filtering based on values ​​obtained by comparing the input image with the restored image after filtering the blocks using the deblocking filter. Alternatively, ALF can perform filtering based on values ​​obtained by comparing the input image with the restored image after filtering the blocks using SAO. SAO can recover offset differences based on values ​​obtained by comparing the input image with the restored image, and can be applied in the form of band offset (BO), edge offset (EO), etc. Specifically, SAO can add an offset relative to the original image to the restored image to which the deblocking filter is applied, at least one pixel unit, and can be applied in the form of BO, EO, etc. In detail, SAO can add offsets relative to the original image to the image recovered after filtering the blocks by ALF, on a pixel-by-pixel basis, and can be applied in the form of BO, EO, etc.

[0144] As filtering information, setting information regarding whether to support each post-processing filter can be generated at the unit level of sequences, images, slices, tiles, etc. Furthermore, setting information regarding whether to execute each post-processing filter can be generated at the unit level of images, slices, tiles, blocks, etc. The scope of filter execution can be categorized into the interior of the image and the boundaries of the image. Setting information considering this category can be generated. Additionally, information regarding filtering operations can be generated at the unit level of images, slices, tiles, blocks, etc. Information can be processed implicitly or explicitly, and independent or dependent filtering can be applied to the filter based on color components, which can be determined according to the encoding settings. The in-loop filter unit can deliver the filtering information to the encoding unit for encoding. The encoding unit adds the corresponding information to the bitstream and transmits the bitstream to the decoder. The decoding unit of the decoder can parse the information and apply the parsed information to the in-loop filter unit.

[0145] The memory can store recovered blocks or images. The recovered blocks or images stored in the memory can be provided to the prediction unit, which performs intra-frame prediction or inter-frame prediction. Specifically, for processing, the space storing the bitstream compressed by the encoder in a queue can be set as the Encoded Picture Buffer (CPB), and the space storing the decoded images in picture units can be set as the Decoded Picture Buffer (DPB). The CPB can store the decoding unit in decoding order, simulate the decoding operation in the encoder, and store the compressed bitstream through simulation processing. The bitstream output from the CPB is recovered through decoding processing, and the recovered image is stored in the DPB. The images stored in the DPB can be referenced during image encoding / decoding processing.

[0146] The decoding unit can be implemented by reversing the processing of the encoding unit. For example, the decoding unit can receive a quantization coefficient string, a transform coefficient string, or a signal string from the bit stream, decode the string, parse the decoded data including the decoding information, and deliver the parsed decoded data to each element.

[0147] Next, image setting processing applied to an image encoding / decoding apparatus according to an embodiment of the present invention will be described. This is an example applied before encoding / decoding (initial image setting), but some processing may be examples to be applied to other steps (e.g., steps after encoding / decoding or sub-steps of encoding / decoding). Image setting processing can be performed considering, for example, network and user environments such as multimedia content characteristics, bandwidth, user terminal performance, and accessibility. For example, image partitioning, image resizing, image reconstruction, etc., can be performed according to the encoding / decoding settings. The following description of image setting processing focuses on rectangular images. However, the invention is not limited thereto, and image setting processing can be applied to polygonal images. The same image settings can be applied regardless of the image form, or different image settings can be applied, which can be determined according to the encoding / decoding settings. For example, after examining information about the image shape (e.g., rectangular or non-rectangular shape), information about the corresponding image settings can be constructed.

[0148] The following examples will be described under the assumption that a dependent setting is provided for the color space. However, independent settings can be provided for the color space. Furthermore, in the following examples, independent settings may include providing encoding / decoding settings independently for each color space. Although one color space is described, it is contemplated that examples of applying this description to other color spaces (e.g., an example where M is generated in the luminance component and N is generated in the chrominance component) be included, and such examples can be derived. Furthermore, dependent settings may include examples that are set proportionally to the color format composition ratio (e.g., 4:4:4, 4:2:2, 4:2:0, etc.) (e.g., for 4:2:0, the chrominance component is M / 2 when the luminance component is M). It is contemplated that examples of applying this description to each color space be included, and such examples can be derived. This description is not limited to the examples above and can be applied collectively to the present invention.

[0149] Some of the builds in the following examples can be applied to various coding techniques, such as spatial domain coding, frequency domain coding, block-based coding, object-based coding, etc.

[0150] Typically, the input image can be encoded or decoded either as is or after image segmentation. For example, segmentation can be performed to prevent corruption caused by packet loss during transmission, such as for error robustness. Alternatively, segmentation can be performed to classify regions with different properties within the same image based on image characteristics, type, etc.

[0151] According to the present invention, image segmentation processing may include segmentation processing and inverse segmentation processing. The following examples will focus on segmentation processing, but inverse segmentation processing can be derived from segmentation processing.

[0152] Figure 3 This is an example diagram of dividing image information into multiple layers to compress the image.

[0153] Part 3a is an example diagram in which the image sequence is composed of multiple GOPs. Furthermore, a GOP can consist of I-images, P-images, and B-images, as shown in Part 3b. An image can consist of slices, tiles, etc., as shown in Part 3c. As shown in Part 3d, slices, tiles, etc., can consist of multiple default coding portions, and as shown in Part 3e, a default coding portion can consist of at least one coding subunit. The image setup processing according to the invention will be described based on examples of units such as images, slices, and tiles as shown in Parts 3b and 3c.

[0154] Figure 4 This is a conceptual diagram illustrating an example of image segmentation according to an embodiment of the present invention.

[0155] Part 4a is a conceptual diagram in which an image (e.g., a picture) is divided at uniform intervals in both the horizontal and vertical directions. The divided regions can be called blocks. Each block can be a default encoded portion (or a maximum encoded portion) obtained through the image division and can be the basic unit to be applied to the division units described below.

[0156] Part 4b is a conceptual map in which the image is divided in at least one direction selected from the horizontal and vertical directions. The divided regions (T0 to T3) may be referred to as tiles, and each region may be encoded or decoded independently or in dependence on other regions.

[0157] Part 4c is a conceptual diagram in which an image is divided into groups of consecutive blocks. The divided regions (S0, S1) can be referred to as slices, and each region can be encoded or decoded independently or dependent on other regions. Groups of consecutive blocks can be defined according to the scanning order. Typically, groups of consecutive blocks conform to the raster scan order. However, the invention is not limited to this, and groups of consecutive blocks can be determined based on encoding / decoding settings.

[0158] Part 4D is a conceptual diagram of a group of blocks that divide an image into according to any user-defined settings. The division of regions (A0 to A2) can be called arbitrary division, and each region can be encoded or decoded independently or dependent on other regions.

[0159] Independent encoding / decoding means that when encoding or decoding some units (or regions), data in other units cannot be referenced. Specifically, multiple pieces of information used or generated during texture encoding and entropy encoding for some units can be encoded independently without referencing each other. Even in the decoder, for texture decoding and entropy decoding for some units, the parsed and recovered information in other units may not refer to each other. In this case, whether to reference data in other units (or regions) can be limited to spatial regions (e.g., between regions in an image), but can also be limited to temporal regions (e.g., between consecutive images or frames) depending on the encoding / decoding settings. For example, reference can be made when some units in the current image and some units in another image have continuity or the same encoding environment; otherwise, reference may be restricted.

[0160] Furthermore, dependency encoding / decoding can mean that when encoding or decoding some units, data in other units can be referenced. Specifically, multiple pieces of information used or generated during texture encoding and entropy encoding for some units can be dependently encoded and referenced to each other. Even in the decoder, for texture decoding and entropy decoding for some units, the parsed and recovered information in other units can be referenced to each other. That is, the above setup can be the same as or similar to the setup of a general encoding / decoding. In this case, in order to identify regions (here, surfaces generated according to the projection format)... <face>According to the present application, the image division processing can include an image division indication step, an image division type identification step, and an image division execution step. In this case, the image division indication step can include a step of indicating a division type of an image according to a characteristic, a type, or the like of the image. The image division type identification step can include a step of identifying a division type of an image according to a characteristic, a type, or the like of the image. The image division execution step can include a step of dividing an image according to a division type of the image.

[0161] In the above example, some units (slices, tiles, or the like) can be provided with independent encoding / decoding settings (for example, independent slice segments), and other units can be provided with dependent encoding / decoding settings (for example, dependent slice segments). According to the present application, the following description will focus on independent encoding / decoding settings.

[0162] As shown in section 4a, the default encoding section acquired by the image division section can be divided into default encoding blocks according to the color space, and can have a size and a shape determined according to the characteristic and the resolution of the image. The size or the shape of the supported blocks can be a square of NxN (2 n ) having a width and a height expressed as an exponential multiple of 2 (2 n × 2 n ; 256x256, 128x128, 64x64, 32x32, 16x16, 8x8, or the like; n is an integer in the range of 3 to 8) or a rectangle of MxN (2 m × 2 n ). For example, the input image can be divided according to the resolution, can be divided into 128x128 for an 8k UHD image, can be divided into 64x64 for a 1080p HD image, or can be divided into 16x16 for a WVGA image, and can be divided according to the image type, can be divided into 256x256 for a 360-degree image. The default encoding section can be divided into encoding sub-units, and then encoded or decoded. Information about the default encoding section can be added to the bitstream in units of sequence, picture, slice, tile, or the like, and can be parsed by the decoder to recover the relevant information.

[0163] The image encoding method and the image decoding method according to the embodiment of the present application can include the following image division step. In this case, the image division processing can include an image division indication step, an image division type identification step, and an image division execution step. In addition, the image encoding apparatus and the image decoding apparatus can be configured to include an image division indication section, an image division type identification section, and an image division execution section that respectively perform the image division indication step, the image division type identification step, and the image division execution step. For encoding, the relevant syntax elements can be generated. For decoding, the relevant syntax elements can be parsed.

[0164] In the block division processing, as shown in section 4a, the image division indication section can be omitted. The image division type identification section can check information about the size and the shape of the block, and the image division section can perform the division by the division type information identified in the default encoding section.

[0165] The block can be a unit to be always divided, but whether to divide other division units (tile, slice, etc.) can be determined according to the encoding / decoding setting. As a default setting, the picture division section can perform division in units of blocks, and then perform division in other units. In this case, block division can be performed based on the picture size.

[0166] In addition, division in units of blocks can be performed after division in other units (tile, slice, etc.). That is, block division can be performed based on the size of the division unit. This can be determined according to the encoding / decoding setting through explicit or implicit processing. The following examples describe the former case and will also focus on units other than blocks.

[0167] In the image division indication step, whether to perform image division can be determined. For example, when a signal indicating image division (e.g., tiles_enabled_flag) is confirmed, division can be performed. When a signal indicating image division is not confirmed, division can not be performed, or division can be performed by confirming other encoding / decoding information.

[0168] In detail, it is assumed that a signal indicating image division (e.g., tiles_enabled_flag) is confirmed. When the signal is activated (e.g., tiles_enabled_flag = 1), division can be performed in a plurality of units. When the signal is deactivated (e.g., tiles_enabled_flag = 0), division can not be performed. Alternatively, a signal indicating image division not being confirmed can mean that division is not performed or division is performed in at least one unit. Whether division is performed in a plurality of units can be confirmed by another signal (e.g., first_slice_segment_in_pic_flag).

[0169] In summary, when a signal indicating image division is provided, the corresponding signal is a signal for indicating whether division is performed in a plurality of units. Whether to divide the corresponding image can be determined according to the signal. For example, it is assumed that tiles_enabled_flag is a signal indicating whether to divide an image. Here, tiles_enabled_flag equal to 1 can mean that the image is divided into a plurality of tiles, and tiles_enabled_flag equal to 0 can mean that the image is not divided.

[0170] In summary, when a signal indicating image division is not provided, the division can not be performed, or it can be determined whether to divide the corresponding image through another signal. For example, first_slice_segment_in_pic_flag is not a signal indicating whether to perform image division, but a signal indicating a first slice segment in an image. Thus, it can be confirmed whether division is performed in two or more units (for example, marked as 0 indicates that the image is divided into a plurality of slices).

[0171] The present application is not limited to the above-described examples, and modifications can be made to the above-described examples. For example, a signal indicating image division can not be provided for each tile, and a signal indicating image division can be provided for each slice. Alternatively, a signal indicating image division can be provided based on the type, characteristics, etc. of an image.

[0172] In the image division type identification step, an image division type can be identified. The image division type can be defined by a division method, division information, etc.

[0173] In section 4b, a tile can be defined as a unit obtained by horizontal and vertical division. In detail, a tile can be defined as a group of adjacent blocks in a quadrangular space divided by at least one horizontal or vertical division line passing through an image.

[0174] Tile division information can include column and row boundary position information, tile number information of columns and rows, tile size information, etc. Tile number information can include the number of columns of tiles (for example, num_tile_columns) and the number of rows of tiles (for example, num_tile_rows). Thus, an image can be divided into a number (= number of columns x number of rows) of tiles. Tile size information can be obtained based on tile number information. The width or height of a tile can be uniform or non-uniform, and thus, under a predetermined rule, related information (for example, uniform_spacing_flag) can be implicitly determined or explicitly generated. In addition, tile size information can include size information of each column and each row of a tile (for example, column_width_tile[i] and row_height_tile[i]), or size information of the width and height of each tile. In addition, size information can be information that can be additionally generated depending on whether tile size is uniform (for example, in the case of non-uniform division because uniform_spacing_flag is 0).

[0175] In section 4c, a slice can be defined as a unit grouping consecutive blocks. In detail, a slice can be defined as a group of consecutive blocks in a predetermined scan order (here, in a raster scan).

[0176] The slice division information can include slice number information, slice position information (e.g., slice_segment_address), etc. In this case, the slice position information can be position information of a predetermined block (e.g., the first row in the scan order in the slice). In this case, the position information can be block scan order information.

[0177] In section 4d, various division settings are allowed for arbitrary division.

[0178] In section 4d, the division unit can be defined as a group of blocks that are spatially adjacent to each other, and the information about the division can include information about the size, form, and position of the division unit. This is only an example of arbitrary division, and various division forms can be allowed as shown in Figure 5

[0179] Figure 5 is another example diagram of an image division method according to an embodiment of the present invention.

[0180] In sections 5a and 5b, an image can be divided into a plurality of regions with at least one block interval in the horizontal or vertical direction, and the division can be performed based on block position information. Section 5a shows an example (A0, A1) in which division is performed horizontally based on row information of each block, and section 5b shows an example (B0 to B3) in which division is performed horizontally and vertically based on column information and row information of each block. The information about the division can include the number of division units, block interval information, division direction, etc., and some division information can not be generated when the division information is implicitly included according to a predetermined rule.

[0181] In sections 5c and 5d, an image can be divided into a group of consecutive blocks in the scan order. An additional scan order other than the conventional slice raster scan order can be applied to image division. Section 5c shows an example (C0, C1) in which scanning is performed clockwise or counterclockwise with respect to a starting block (Box-Out), and section 5d shows an example (D0, D1) in which scanning is performed vertically with respect to a starting block (vertical). The information about the division can include information about the number of division units, information about the position of the division unit (e.g., the first row in the scan order in the division unit), information about the scan order, etc., and some division information can not be generated when the division information is implicitly included according to a predetermined rule.

[0182] ​In part 5e, the image can be divided using the horizontal and vertical division lines. The existing tile can be divided by the horizontal or vertical division line. Thus, the division can be performed in the form of a quadrangular space, but it can not be possible to divide the image using the division line. For example, an example of dividing the image through some division lines of the image (e.g., a division line between the left boundary of E5 and the right boundaries of E1, E3, and E4) is possible, and an example of dividing the image through some division lines of the image (e.g., a division line between the lower boundaries of E2 and E3 and the upper boundary of E4) is not possible. In addition, the division can be performed based on the block unit (e.g., after first performing the block division), or the division can be performed through the horizontal or vertical division line (e.g., the division is performed through the division line regardless of the block division). Thus, each division unit can not be a multiple of the block. Thus, division information different from the division information of the existing tile can be generated, and the division information can include information on the number of division units, information on the position of the division unit, information on the size of the division unit, etc. For example, the information on the position of the division unit can be generated as position information (e.g., measured in pixels or in blocks) based on a predetermined position (e.g., the upper left corner of the image), and the information on the size of the division unit can be generated as information on the width and height of each division unit (e.g., measured in pixels or in blocks).

[0183] Similar to the above example, the division according to any user-defined setting can be performed by applying a new division method or by changing some elements of the existing division. That is, the division method can be supported by replacing or adding to the conventional division method, and the division method can be supported by changing some settings (slices, tiles, etc.) of the conventional division method (e.g., according to another scan order, by generating other division information using another division method in the form of a quadrangular shape, or according to a dependent encoding / decoding characteristic). In addition, a setting for configuring an additional division unit (e.g., a setting other than division according to a scan order or division according to a specific interval difference) can be supported, and an additional division unit form (e.g., a polygon form such as a triangle in addition to division into a quadrangular space) can be supported. In addition, the image division method can be supported based on the type, characteristic, etc. of the image. For example, a partial division method (e.g., a face of a 360-degree image) can be supported according to the type, characteristic, etc. of the image. The information on the division can be generated based on the support.

[0184] In the image division execution step, the image can be divided based on the identified division type information. That is, the image can be divided into a plurality of division units based on the identified division type, and the image can be encoded or decoded based on the obtained division unit.

[0185] In this case, it can be determined whether to have the encoding / decoding setting in each division unit according to the division type. That is, the setting information required during the encoding / decoding process for each division unit can be designated by a higher unit (e.g., a picture), or an independent encoding / decoding setting can be provided for each division unit.

[0186] In general, a slice can have an independent encoding / decoding setting for each division unit (e.g., a slice header), and a tile cannot have an independent encoding / decoding setting for each division unit and can have a setting depending on a picture's encoding / decoding setting (e.g., a PPS). In this case, information generated in association with a tile can be division information, and can be included in a picture's encoding / decoding setting. The present application is not limited to the above-described example, and the above-described example can be modified.

[0187] Encoding / decoding setting information for a tile can be generated in units of a video, a sequence, a picture, etc. At least one encoding / decoding setting information is generated in a higher unit, and the generated one encoding / decoding setting information can be referred to. Alternatively, independent encoding / decoding setting information (e.g., a tile header) can be generated in units of a tile. This is different from the case of following one encoding / decoding setting determined in a higher unit, in that encoding / decoding is performed in a case where at least one encoding / decoding setting is provided in units of a tile. That is, all tiles can be encoded or decoded according to the same encoding / decoding setting, or at least one tile can be encoded or decoded according to an encoding / decoding setting different from the encoding / decoding settings of other tiles.

[0188] The above examples focus on various encoding / decoding settings in a tile. However, the present application is not limited thereto, and even the same or similar settings can be applied to other division types.

[0189] As an example, in some division types, division information can be generated in a higher unit, and encoding or decoding can be performed according to a single encoding / decoding setting of the higher unit.

[0190] As an example, in some division types, division information can be generated in a higher unit, and independent encoding / decoding settings for each division unit in the higher unit can be generated, and encoding or decoding can be performed according to the generated encoding / decoding settings.

[0191] As an example, in some division types, division information can be generated in a higher unit, and a plurality of encoding / decoding setting information can be supported in the higher unit. Encoding or decoding can be performed according to an encoding / decoding setting referred to by each division unit.

[0192] As an example, in some division types, division information can be generated in a higher unit, and independent encoding / decoding settings including the division information can be generated in a corresponding division unit, and encoding or decoding can be performed according to the generated encoding / decoding settings.

[0193] As an example, in some division types, division information can be generated in a higher unit, and independent encoding / decoding settings including the division information can be generated in a corresponding division unit, and encoding or decoding can be performed according to the generated encoding / decoding settings.

[0194] The encoding / decoding setting information can include information required to encode or decode a tile, such as a tile type, information about a reference picture list, quantization parameter information, inter prediction setting information, in-loop filtering setting information, in-loop filtering control information, a scan order, whether to perform encoding or decoding, etc. The encoding / decoding setting information can be used to explicitly generate related information, or can have encoding / decoding settings that are implicitly determined according to the format, characteristics, etc. of an image determined in a higher unit. In addition, related information can be explicitly generated based on information obtained through settings.

[0195] Next, an example of performing image division in an encoding / decoding apparatus according to an embodiment of the present application will be described.

[0196] Division processing can be performed on an input image before starting encoding. An image can be divided using division information (e.g., image division information, division unit setting information, etc.), and then the image can be encoded in a division unit. Image encoding data can be stored in a memory after encoding is completed, and can be added to a bitstream and then transmitted.

[0197] Division processing can be performed before starting decoding. An image can be divided using division information (e.g., image division information, division unit setting information, etc.), and then image decoding data can be parsed and decoded in a division unit. After decoding is completed, image decoding data can be stored in a memory, and a plurality of division units are merged into a single unit, and thus an image can be output.

[0198] Through the above examples, image division processing has been described. In addition, according to the present application, a plurality of division processes can be performed.

[0199] For example, the image can be divided, and the divided unit of the image can be divided. The division can be the same division process (e.g., slice / slice, tile / tile, etc.) or a different division process (e.g., slice / tile, tile / slice, tile / surface, surface / tile, slice / surface, surface / slice, etc.). In this case, the subsequent division process can be performed based on the previous division result, and the division information generated during the subsequent division process can be generated based on the previous division result.

[0200] Further, a plurality of division processes (A) can be performed, and the division processes can be different division processes (e.g., slice / surface, tile / surface, etc.). In this case, the subsequent division process can be performed based on or independently of the previous division result, and the division information generated during the subsequent division process can be generated based on or independently of the previous division result.

[0201] The plurality of image division processes can be determined according to the encoding / decoding setting. However, the present application is not limited to the above-described example, and various modifications can be made to the above-described example.

[0202] The encoder can add the information generated during the above-described process to the bitstream in units of at least one of a sequence, a picture, a slice, a tile, etc., and the decoder can parse the related information from the bitstream. That is, the information can be added to one unit, and the information can be copied and added to a plurality of units. For example, a syntax element indicating whether some information is supported or a syntax element indicating whether activation is performed can be generated in some units (e.g., upper units), and the same or similar information can be generated in some units (e.g., lower units). That is, even in the case where the related information is supported and set in the upper units, the lower units can have separate settings. The description is not limited to the above-described example, and can be commonly applied to the present application. Further, the information can be included in the bitstream in the form of SEI or metadata.

[0203] In general, the input image can be encoded or decoded as it is, but the encoding or decoding can be performed after adjusting the image size (expansion or reduction; resolution adjustment). For example, in a layered encoding scheme (scalable video coding) for supporting spatial, temporal, and image quality scalability, image size adjustment such as overall expansion and reduction of the image can be performed. Alternatively, image size adjustment such as partial expansion and reduction of the image can be performed. The image size adjustment can be performed differently, that is, the image size adjustment can be performed for the purpose of adaptability to the encoding environment, for the purpose of encoding uniformity, for the purpose of encoding efficiency, for the purpose of image quality improvement, or according to the type, characteristics, etc. of the image.

[0204] As a first example, the size adjustment processing can be performed during processing performed in accordance with characteristics, types, and the like of an image (e.g., layered encoding, 360-degree image encoding, and the like).

[0205] As a second example, the size adjustment processing can be performed at an initial encoding / decoding step. The size adjustment processing can be performed before encoding or decoding is performed. The encoded or decoded image can be performed on the size-adjusted image.

[0206] As a third example, the size adjustment processing can be performed during a prediction step (intra prediction or inter prediction) or before prediction. During the size adjustment processing, image information (e.g., information on pixels of an intra prediction reference, information on an intra prediction mode, information on a reference picture for inter prediction, information on an inter prediction mode, and the like) can be used at the prediction step.

[0207] As a fourth example, the size adjustment processing can be performed during a filtering step or before filtering. During the size adjustment processing, image information (e.g., pixel information to be applied to a deblocking filter, pixel information to be applied to SAO, information on SAO filtering, pixel information applied to ALF, information on ALF filtering, and the like) in the filtering step can be used.

[0208] Further, after the size adjustment processing is performed, the image can be processed by inverse size adjustment processing and changed to the image before the size adjustment in terms of image size, or the image can not be changed. This can be determined in accordance with encoding / decoding settings (e.g., characteristics of performing size adjustment). In this case, the size adjustment processing can be expansion processing and the inverse size adjustment processing is reduction processing, and the size adjustment processing can be reduction processing and the inverse size adjustment processing is expansion processing.

[0209] When the size adjustment processing is performed in accordance with the first example to the fourth example, inverse size adjustment processing is performed in a subsequent step so that the image before the size adjustment can be acquired.

[0210] When the size adjustment processing is performed by layered encoding or in accordance with the third example (or when the size of a reference picture is adjusted in inter prediction), the inverse size adjustment processing can not be performed in a subsequent step.

[0211] In the embodiments of the present application, the image size adjustment processing can be performed alone or together with inverse processing. The following examples will be described with a focus on the size adjustment processing. In this case, since the inverse size adjustment processing is inverse processing for the size adjustment processing, the description of the inverse size adjustment processing will be omitted to prevent redundant description. However, it is obvious that the person skilled in the art can recognize the same content as that described literally.

[0212] Figure 6 is an example diagram of a general image resizing method.

[0213] Referring to Figure 6 a, an expanded image (P0+P1) can be obtained by adding a specific region (P1) to an initial image P0 (or an image before resizing; it is indicated by a thick solid line).

[0214] Referring to Figure 6 b, a reduced image (S0) can be obtained by removing a specific region (S1) from an initial image (S0+S1).

[0215] Referring to Figure 6 c, a resized image (T0+T1) can be obtained by adding a specific region (T1) to an initial image (T0+T2) and removing a specific region (T2) from the entire image.

[0216] According to the present application, the following description focuses on the resizing processing for expansion and the resizing processing for reduction. However, the present application is not limited to this, and should be understood to include the case where expansion and reduction are combined as shown in Figure 6 c.

[0217] Figure 7 is an example diagram of image resizing according to an embodiment of the present application.

[0218] During the resizing processing, the image expansion method will be described with reference to section 7a, and the image reduction method will be described with reference to section 7b.

[0219] In section 7a, the image before resizing is S0, and the image after resizing is S1. In section 7b, the image before resizing is T0, and the image after resizing is T1.

[0220] When expanding an image as shown in section 7a, the image can be expanded in the "upward” direction, the "downward” direction, the "leftward” direction, or the "rightward” direction (ET, EL, EB, ER). When reducing an image as shown in section Figure 7 b, the image can be reduced in the "upward” direction, the "downward” direction, the "leftward” direction, or the "rightward” direction (RT, RL, RB, RR).

[0221] Comparing image expansion and image reduction, the "upward” direction, the "downward” direction, the "leftward” direction, and the "rightward” direction of expansion can correspond to the "downward” direction, the "upward” direction, the "rightward” direction, and the "leftward” direction of reduction. Therefore, the following description focuses on image expansion, but should be understood to include the description of image reduction.

[0222] In the following description, image expansion or reduction is performed in an "up" direction, a "down" direction, a "left" direction, and a "right" direction. However, it should also be understood that resizing can be performed in an "up and left" direction, an "up and right" direction, a "down and left" direction, or a "down and right" direction.

[0223] In this case, when expansion is performed in a "down and right" direction, the regions RC and BC are acquired, and the region BR can or can not be acquired depending on the encoding / decoding setting. That is, the regions TL, TR, BL, and BR can or can not be acquired, but for ease of description, it will be described that the corner regions (i.e., the regions TL, TR, BL, and BR) can be acquired.

[0224] The image resizing processing according to the embodiments of the present application can be performed in at least one direction. For example, the image resizing processing can be performed in all directions such as up, down, left, and right, can be performed in two or more directions selected from among up, down, left, and right (left+right, up+down, up+left, up+right, down+left, down+right, up+left+right, down+left+right, up+down+left, up+down+right, etc.), or can be performed in only one direction selected from among up, down, left, and right.

[0225] For example, resizing can be performed in a "left+right" direction, an "up+down" direction, a "left and up+right and down" direction, and a "left and down+right and up" direction, which symmetrically expand to both ends with respect to the center of the image, resizing can be performed in a "left+right" direction, a "left and up+right and up" direction, and a "left and down+right and down" direction, which symmetrically expand vertically with respect to the image, and resizing can be performed in an "up+down" direction, a "left and up+left and down" direction, and a "right and up+right and down" direction, which symmetrically expand horizontally with respect to the image. Other resizing can be performed.

[0226] In sections 7a and 7b, the size of the image before the resizing (S0, T0) is defined as P_Width x P_Height, and the size of the image after the resizing (S1, T1) is defined as P'_Width x P'_Height. Here, when the resizing values in the "left" direction, "right" direction, "top" direction, and "bottom" direction are defined as Var_L, Var_R, Var_T, and Var_B (or collectively as Var_x), the image size after the resizing can be expressed as (P_Width + Var_L + Var_R) x (P_Height + Var_T + Var_B). In this case, Var_L, Var_R, Var_T, and Var_B as the resizing values in the "left" direction, "right" direction, "top" direction, and "bottom" direction can be Exp_L, Exp_R, Exp_T, and Exp_B (here, Exp_x is positive) for image expansion (in section 7a), and can be -Rec_L, -Rec_R, -Rec_T, and -Rec_B (which represent negative values for image reduction when Rec_L, Rec_R, Rec_T, and Rec_B are defined as positive values). Further, the upper-left corner coordinate, upper-right corner coordinate, lower-left corner coordinate, and lower-right corner coordinate of the image before the resizing can be (0, 0), (P_Width - 1, 0), (0, P_Height - 1), and (P_Width - 1, P_Height - 1), and the upper-left corner coordinate, upper-right corner coordinate, lower-left corner coordinate, and lower-right corner coordinate of the image after the resizing can be expressed as (0, 0), (P'_Width - 1, 0), (0, P'_Height - 1), and (P'_Width - 1, P'_Height - 1). The size of the region changed (or acquired or removed) by the resizing (here, TL to BR; i is an index for identifying TL to BR) can be M[i] x N[i] and can be expressed as Var_X x Var_Y (this example assumes that X is L or R, and Y is T or B). M and N can have various values, and can have the same settings regardless of i, or can have separate settings according to i. Various examples will be described below.

[0227] Referring to section 7a, S1 can be configured to include some or all of the regions TL to BR (top-left to bottom-right) that would be generated by expanding S0 in several directions. Referring to section 7b, T1 can be configured to exclude all or some of the regions TL to BR that would be removed by reducing T0 in several directions.

[0228] In part 7a, when the existing image (S0) is expanded in the "up" direction, the "down" direction, the "left" direction, and the "right" direction, the image can include the regions TC, BC, LC, and RC obtained through the resizing process and can further include the regions TL, TR, BL, and BR.

[0229] As an example, when expansion is performed in the "up" direction (ET), the image can be constructed by adding the region TC to the existing image (S0), and the image can include the region TL or TR and expansion in at least one different direction (EL or ER).

[0230] As an example, when expansion is performed in the "down" direction (EB), the image can be constructed by adding the region BC to the existing image (S0), and the image can include the region BL or BR and expansion in at least one different direction (EL or ER).

[0231] As an example, when expansion is performed in the "left" direction (EL), the image can be constructed by adding the region LC to the existing image (S0), and the image can include the region TL or BL and expansion in at least one different direction (ET or EB).

[0232] As an example, when expansion is performed in the "right" direction (ER), the image can be constructed by adding the region RC to the existing image (S0), and the image can include the region TR or BR and expansion in at least one different direction (ET or EB).

[0233] According to embodiments of the present application, it is possible to provide a setting (e.g., spa_ref_enabled_flag or tem_ref_enabled_flag) for spatially or temporally limiting the referability of a resized region (this example assumes expansion).

[0234] That is, it is possible to allow (e.g., spa_ref_enabled_flag = 1 or tem_ref_enabled_flag = 1) or limit (e.g., spa_ref_enabled_flag = 0 or tem_ref_enabled_flag = 0) the reference to data of a region that is resized spatially or temporally according to an encoding / decoding setting.

[0235] Encoding / decoding of the image (S0, T1) before resizing and the regions (TC, BC, LC, RC, TL, TR, BL, and BR) added or deleted during resizing can be performed as follows.

[0236] For example, when encoding or decoding an image before resizing and an added or deleted region, data about the image before resizing and data about the added or deleted region (data after encoding or decoding is completed; pixel values or prediction-related information) can be spatially or temporally referenced to each other.

[0237] Alternatively, the image before resizing and the data about the added or deleted region can be spatially referenced, the data about the image before resizing can be temporally referenced, and the data about the added or deleted region can not be temporally referenced.

[0238] That is, a setting for limiting the referenceability of the added or deleted region can be provided. Setting information about the referenceability of the added or deleted region can be explicitly generated or implicitly determined.

[0239] The image resizing process according to the embodiment of the present application can include an image resizing indication step, an image resizing type identification step, and / or an image resizing execution step. In addition, the image encoding apparatus and the image decoding apparatus can include an image resizing indication section, an image resizing type identification section, and an image resizing execution section configured to perform the image resizing indication step, the image resizing type identification step, and the image resizing execution step, respectively. For encoding, relevant syntax elements can be generated. For decoding, relevant syntax elements can be parsed.

[0240] In the image resizing indication step, whether to perform image resizing can be determined. For example, when a signal indicating image resizing (e.g., img_resizing_enabled_flag) is confirmed, resizing can be performed. When a signal indicating image resizing is not confirmed, resizing can not be performed, or resizing can be performed by confirming other encoding / decoding information. In addition, although a signal indicating image resizing is not provided, the signal indicating image resizing can be implicitly activated or deactivated according to encoding / decoding settings (e.g., characteristics, types, etc. of an image). When resizing is performed, corresponding resizing-related information can be generated, or corresponding resizing-related information can be implicitly determined.

[0241] When a signal indicating image resizing is provided, the corresponding signal is a signal for indicating whether to perform image resizing. Whether to resize a corresponding image can be determined according to the signal.

[0242] For example, assume that a signal indicating image size adjustment (e.g., img_resizing_enabled_flag) is confirmed. When the corresponding signal (e.g., img_resizing_enabled_flag = 1) is activated, image size adjustment can be performed. When the corresponding signal (e.g., img_resizing_enabled_flag = 0) is deactivated, image size adjustment can not be performed.

[0243] In addition, when a signal indicating image size adjustment is not provided, size adjustment can not be performed, or whether to adjust the size of a corresponding image can be determined through another signal.

[0244] For example, when an input image is divided in units of blocks, size adjustment can be performed according to whether the size (e.g., width or height) of the image is a multiple of the size (e.g., width or height) of the block (for extension in this example, assume that size adjustment processing is performed when the image size is not a multiple of the block size). That is, when the width of the image is not a multiple of the width of the block or when the height of the image is not a multiple of the height of the block, size adjustment can be performed. In this case, size adjustment information (e.g., size adjustment direction, size adjustment value, etc.) can be determined according to encoding / decoding information (e.g., size of the image, size of the block, etc.). Alternatively, size adjustment can be performed according to the characteristics, type (e.g., 360-degree image), etc. of the image, and size adjustment information can be explicitly generated or can be designated as a predetermined value. The present application is not limited to the above-described example, and the above-described example can be modified.

[0245] In the image size adjustment type identification step, an image size adjustment type can be identified. The image size adjustment type can be defined by a size adjustment method, size adjustment information, etc. For example, size adjustment based on a scaling factor, size adjustment based on an offset factor, etc. can be performed. The present application is not limited to the above-described example, and these methods can be applied in combination. For ease of description, the following description will focus on size adjustment based on a scaling factor and size adjustment based on an offset factor.

[0246] For a scaling factor, size adjustment can be performed by multiplication or division based on the size of the image. Information about the size adjustment operation (e.g., expansion or reduction) can be explicitly generated, and expansion or reduction processing can be performed according to the corresponding information. In addition, size adjustment processing can be performed as a predetermined operation (e.g., one of an expansion operation and a reduction operation) according to encoding / decoding settings. In this case, information about the size adjustment operation will be omitted. For example, when image size adjustment is activated in the image size adjustment indication step, image size adjustment can be performed as a predetermined operation.

[0247] The resizing direction can be at least one direction selected from among: up, down, left, and right. Depending on the resizing direction, at least one scale factor can be required. That is, one scale factor can be required for each direction (here, unidirectional), one scale factor can be required for horizontal or vertical direction (here, bidirectional), and one scale factor can be required for all directions of the image (here, omnidirectional). In addition, the resizing direction is not limited to the above-described examples, and the above-described examples can be modified.

[0248] The scale factor can have a positive value, and can have range information that is different according to the encoding / decoding setting. For example, when the information is generated by combining the resizing operation and the scale factor, the scale factor can be used as a multiplier. The scale factor greater than 0 or less than 1 can mean a reduction operation, the scale factor greater than 1 can mean an expansion operation, and the scale factor being 1 can mean that the resizing is not performed. As another example, when the scale factor information is generated independently of the resizing operation, the scale factor for the expansion operation can be used as a multiplier, and the scale factor for the reduction operation can be used as a divisor.

[0249] The process of changing the image (S0, T0) before the resizing into the image (here, S1 and T1) after the resizing will be described again with reference to parts 7a and 7b of FIG. 7. Figure 7

[0250] As an example, when one scale factor (referred to as sc) is used in all directions of the image and the resizing direction is the "down + right" direction, the resizing direction is ER and EB (or RR and RB), the resizing values Var_L (Exp_L or Rec_L) and Var_T (Exp_T or Rec_T) are 0, and Var_R (Exp_R or Rec_R) and Var_B (Exp_B or Rec_B) can be expressed as P_Width×(sc-1) and P_Height×(sc-1). Thus, the image after the resizing can be (P_Width×sc)×(P_Height×sc).

[0251] ​As an example, when respective scale factors (here, sc_w and sc_h) are used in the landscape direction or the portrait direction of the image and the resizing directions are the "left+right" direction and the "up+down" direction (up+down+left+right when both are operated), the resizing directions can be ET, EB, EL, and ER, the resizing values Var_T and Var_B can be P_Height x (sc_h-1) / 2, and Var_L and Var_R can be P_Width x (sc_w-1) / 2. Thus, the image after resizing can be (P_Width x sc_w) x (P_Height x sc_h).

[0252] For the offset factor, resizing can be performed by addition or subtraction based on the size of the image. Alternatively, resizing can be performed by addition or subtraction based on the encoding / decoding information of the image. Alternatively, resizing can be performed by independent addition or subtraction. That is, the resizing process can have a dependent setting or an independent setting.

[0253] Information about the resizing operation (e.g., expansion or reduction) can be explicitly generated, and the expansion or reduction process can be performed according to the respective information. In addition, the resizing operation can be performed as a predetermined operation (e.g., one of the expansion operation and the reduction operation) according to the encoding / decoding setting. In this case, information about the resizing operation can be omitted. For example, when the image resizing is activated in the image resizing indication step, the image resizing can be performed as a predetermined operation.

[0254] The resizing direction can be at least one direction selected from among up, down, left, and right. Depending on the resizing direction, at least one offset factor can be required. That is, one offset factor can be required for each direction (here, unidirectional), one offset factor can be required for the landscape or portrait direction (here, symmetric bidirectional), one offset factor can be required according to a partial combination of directions (here, asymmetric bidirectional), and one offset factor can be required for all directions of the image (here, omnidirectional). In addition, the resizing direction is not limited to the above-described examples, and the above-described examples can be modified.

[0255] The offset factor can have a positive value or both positive and negative values, and can have range information that is different according to the encoding / decoding setting. For example, when the offset factor is generated in conjunction with the size adjustment operation and the offset factor generation information (here, it is assumed that the offset factor has both positive and negative values), the offset factor can be used as a value to be added or subtracted according to the sign information of the offset factor. The offset factor greater than 0 can mean an expansion operation, the offset factor less than 0 can mean a reduction operation, and the offset factor being 0 can mean that the size adjustment is not performed. As another example, when the offset factor information is generated independently of the size adjustment operation (here, it is assumed that the offset factor has a positive value), the offset factor can be used as a value to be added or subtracted according to the size adjustment operation. The offset factor greater than 0 can mean that an expansion or reduction operation can be performed according to the size adjustment operation, and the offset factor being 0 can mean that the size adjustment is not performed.

[0256] The method of changing the image (S0, T0) before the size adjustment into the image (S1, T1) after the size adjustment using the offset factor will be described again with reference to parts 7a and 7b of FIG. 7. Figure 7

[0257] As an example, when one offset factor (referred to as os) is used in all directions of the image and the size adjustment direction is the "up + down + left + right" direction, the size adjustment direction can be ET, EB, EL, and ER (or RT, RB, RL, and RR) and the size adjustment values Var_T, Var_B, Var_L, and Var_R can be os. The size of the image after the size adjustment can be (P_Width + os) x (P_Height + os).

[0258] As an example, when the offset factors (os_w, os_h) are used in the horizontal or vertical direction of the image and the size adjustment direction is the "left + right" direction and the "up + down" direction ("up + down + left + right" direction when both operations are performed), the size adjustment direction can be ET, EB, EL, and ER (or RT, RB, RL, and RR), the size adjustment values Var_T and Var_B can be os_h, and the size adjustment values Var_L and Var_R can be os_w. The size of the image after the size adjustment can be {P_Width + (os_w x 2)} x {P_Height + (os_h x 2)}.

[0259] ​As an example, when the resizing direction is the "down" direction and the "right" direction (the "down+right" direction when operated together) and the offset factors (os_b, os_r) are used according to the resizing direction, the resizing direction can be EB and ER (or RB and RR), the resizing value Var_B can be os_b, and the resizing value Var_R can be os_r. The size of the image after resizing can be (P_Width+os_r) x (P_Height+os_b).

[0260] As an example, when the offset factors (os_t, os_b, os_l, os_r) are used according to the direction of the image and the resizing direction is the "up" direction, the "down" direction, the "left" direction, and the "right" direction (the "up+down+left+right" direction when all are operated), the resizing direction can be ET, EB, EL, and ER (or RT, RB, RL, and RR), the resizing value Var_T can be os_t, the resizing value Var_B can be os_b, the resizing value Var_L can be os_l, and the resizing value Var_R can be os_r. The size of the image after resizing can be (P_Width+os_l+os_r) x (P_Height+os_t+os_b).

[0261] The above examples indicate a case where the offset factor is used as the resizing value (Var_T, Var_B, Var_L, Var_R) during the resizing process. That is, this means that the offset factor is used as the resizing value without any change, which can be an example of independently performed resizing. Alternatively, the offset factor can be used as an input variable of the resizing value. In detail, the offset factor can be designated as the input variable, and the resizing value can be acquired through a series of processes according to the encoding / decoding setting, which can be an example of resizing performed based on predetermined information (e.g., image size, encoding / decoding information, etc.) or an example of dependency-based resizing.

[0262] For example, the offset factor can be a multiple (e.g., 1, 2, 4, 6, 8, and 16) or an exponential multiple (e.g., an exponential multiple of 2, such as 1, 2, 4, 8, 16, 32, 64, 128, and 256) of a predetermined value (here, an integer). Alternatively, the offset factor can be a multiple or an exponential multiple of a value (e.g., a value set based on a motion search range for inter prediction) acquired based on the encoding / decoding setting. Alternatively, the offset factor can be a multiple or an integer of a unit (here, assumed to be AxB) acquired from the picture division section. Alternatively, the offset factor can be a multiple (here, assumed to be ExF, such as a tile) of a unit acquired from the picture division section.

[0263] Alternatively, the offset factor can be a value less than or equal to the width and height of the unit obtained from the picture division section. In the above example, the multiple or the exponential multiple can have a value of 1. However, the present application is not limited to the above example, and the above example can be modified. For example, when the offset factor is n, Var_x can be 2 x n or 2 n .

[0264] In addition, a separate offset factor can be supported according to a color component. An offset factor for some color components can be supported, and thus an offset factor information for other color components can be derived. For example, when an offset factor (A) for a luminance component is explicitly generated (here, it is assumed that a composition ratio of the luminance component with respect to the chrominance component is 2:1), an offset factor (A / 2) for the chrominance component can be implicitly obtained. Alternatively, when an offset factor (A) for the chrominance component is explicitly generated, an offset factor (2A) for the luminance component can be implicitly obtained.

[0265] Information about a size adjustment direction and a size adjustment value can be explicitly generated, and a size adjustment process can be performed according to the corresponding information. In addition, the information can be implicitly determined according to an encoding / decoding setting, and a size adjustment process can be performed according to the determined information. At least one predetermined direction or size adjustment value can be assigned, and in this case, the relevant information can be omitted. In this case, the encoding / decoding setting can be determined based on characteristics, types, encoding information, etc. of an image. For example, at least one size adjustment direction can be predetermined according to at least one size adjustment operation, at least one size adjustment value can be predetermined according to at least one size adjustment operation, and at least one size adjustment value can be predetermined according to at least one size adjustment direction. In addition, a size adjustment direction, a size adjustment value, etc. during inverse size adjustment processing can be derived from a size adjustment direction, a size adjustment value, etc. applied during size adjustment processing. In this case, the size adjustment value implicitly determined can be one of the above examples (examples in which a size adjustment value is differently obtained).

[0266] In addition, multiplication or division has been described in the above examples, but a shift operation can be used according to an implementation of an encoder / decoder. Multiplication can be implemented by a left shift operation, and division can be implemented by a right shift operation. The description is not limited to the above examples, and can be commonly applied to the present application.

[0267] In the image size adjustment execution step, image size adjustment can be performed based on the identified size adjustment information. That is, image size adjustment can be performed based on information about a size adjustment type, a size adjustment operation, a size adjustment direction, a size adjustment value, etc., and encoding / decoding can be performed based on an image after size adjustment is obtained.

[0268] Further, in the image size adjustment execution step, the size adjustment can be performed using at least one data processing method. In detail, the size adjustment can be performed on the region to be size-adjusted according to the size adjustment type and the size adjustment operation by using at least one data processing method. For example, according to the size adjustment type, it can be determined how to fill data when the size adjustment is for expansion, and it can be determined how to remove data when the size adjustment is for reduction.

[0269] In summary, in the image size adjustment execution step, the image size adjustment can be performed based on the identified size adjustment information. Alternatively, in the image size adjustment execution step, the image size adjustment can be performed based on the size adjustment information and a data processing method. The two cases can differ from each other in that only the size of the image to be encoded or decoded is adjusted, or in that even the data processing of the region to be size-adjusted and the image size is considered. In the image size adjustment execution step, it can be determined whether to perform the data processing method according to the step, position, etc., to which the size adjustment process is applied. The following description focuses on an example of performing the size adjustment based on the data processing method, but the present application is not limited thereto.

[0270] When performing the size adjustment based on the offset factor, various methods can be used to perform the size adjustment for expansion and the size adjustment for reduction. For expansion, the size adjustment can be performed using at least one data filling method. For reduction, the size adjustment can be performed using at least one data removal method. In this case, when performing the size adjustment based on the offset factor, the size-adjusted region can be filled with new data or original image data directly or after modification (expansion), and the size-adjusted region can be simply removed or through a series of processes (reduction).

[0271] When performing the size adjustment based on the scale factor, in some cases (e.g., hierarchical encoding), the size adjustment for expansion can be performed by applying upsampling, and the size adjustment for reduction can be performed by applying downsampling. For example, at least one up-sampling filter can be used for expansion, and at least one down-sampling filter can be used for reduction. The filter applied horizontally can be the same as or different from the filter applied vertically. In this case, when performing the size adjustment based on the scale factor, neither new data is generated in the size-adjusted region nor new data is removed from the size-adjusted region, but the original image data can be rearranged using a method such as interpolation. The data processing method associated with the size adjustment can be classified according to the filter used for sampling. Furthermore, in some cases (e.g., cases similar to the case of the offset factor), the size adjustment for expansion can be performed using a method of padding at least one data, and the size adjustment for reduction can be performed using a method of removing at least one data. According to the present application, the following description focuses on the data processing method corresponding to the case of performing the size adjustment based on the offset factor.

[0272] In general, a predetermined data processing method can be used in the region to be size-adjusted, but at least one data processing method can also be used in the region to be size-adjusted, as in the following examples. Selection information for the data processing method can be generated. The former can mean that the size adjustment is performed by a fixed data processing method, and the latter can mean that the size adjustment is performed by an adaptive data processing method.

[0273] Furthermore, the data processing method can be applied to all regions (TL, TC, TR,..., BR in parts 7a and 7b) in the region to be added or deleted during the size adjustment or some regions (e.g., each or a combination of TL to BR in parts 7a and 7b).

[0274] Figure 8 is an example diagram of a method of constructing a region generated by expansion in an image size adjustment method according to an embodiment of the present application.

[0275] Referring to part 8a, for ease of description, the image can be divided into regions TL, TC, TR, LC, C, RC, BL, BC, and BR corresponding to the upper left, upper, upper right, left, center, right, lower left, lower, and lower right positions of the image. In the following description, the image is expanded in the "down+right" direction, but it should be understood that the description can be applied to other expansion directions.

[0276] The region added according to the expansion of the image can be constructed using various methods. For example, the region can be padded with arbitrary values, or can be padded with reference to some data of the image.

[0277] Referring to the portion 8b, the generated areas (A0, A2) can be filled with an arbitrary pixel value. Various methods can be used to determine the arbitrary pixel value.

[0278] As an example, the arbitrary pixel value can be one pixel in a pixel value range that can be represented using a bit depth (e.g., from 0 to 1<<(bit_depth)-1). For example, the arbitrary pixel value can be a minimum value, a maximum value, a median value (e.g., 1<<(bit_depth-1), etc.), etc. in the pixel value range (here, bit_depth indicates a bit depth).

[0279] As an example, the arbitrary pixel value can be one pixel in a pixel value range that belongs to pixels of the image (e.g., from min P to max P ; min P and max P indicate a minimum value and a maximum value among the pixels belonging to the image; min P is greater than or equal to 0; and max P is less than or equal to 1<<(bit_depth)-1). For example, the arbitrary pixel value can be a minimum value, a maximum value, a median value, an average value of (at least two pixels), a weighted sum, etc. in the pixel value range.

[0280] As an example, the arbitrary pixel value can be a value determined in a pixel value range that belongs to pixels of a specific area included in the image. For example, when A0 is constructed, the specific area can be TR+RC+BR. Further, the specific area can be set to an area corresponding to 3×9 of TR, RC, and BR or an area corresponding to 1×9 (which is assumed to be the rightmost line). This can depend on the encoding / decoding setting. In this case, the specific area can be a unit to be divided by the picture division section. In detail, the arbitrary pixel value can be a minimum value, a maximum value, a median value, an average value of (at least two pixels), a weighted sum, etc. in the pixel value range.

[0281] Referring again to the portion 8b, the area A1 to be added as the image is expanded can be filled with pattern information generated using a plurality of pixel values (e.g., a pattern is assumed to use a plurality of pixels; it is not necessary to follow certain rules). In this case, the pattern information can be defined according to the encoding / decoding setting, or the relevant information can be generated. The generated area can be filled with at least one piece of pattern information.

[0282] Referring to part 8c, a region added as the image is expanded can be constructed with reference to pixels of a specific region included in the image. In detail, the added region can be constructed by copying or padding pixels (hereinafter, referred to as reference pixels) in a region adjacent to the added region. In this case, the pixels in the region adjacent to the added region can be pixels before encoding or pixels after encoding (or decoding). For example, when resizing is performed in a pre-encoding step, the reference pixels can refer to pixels of an input image, and when resizing is performed in an intra prediction reference pixel generation step, a reference picture generation step, a filtering step, etc., the reference pixels can refer to pixels of a restored image. In this example, it is assumed that the nearest neighbor pixels are used in the added region, but the present application is not limited thereto.

[0283] A region (A0) generated when the image is expanded to the left or right in association with horizontal image resizing can be constructed by horizontally padding (Z0) external pixels adjacent to the generated region (A0), and a region (A1) generated when the image is expanded upward or downward in association with vertical image resizing can be constructed by vertically padding (Z1) external pixels adjacent to the generated region (A1). Also, a region (A2) generated when the image is expanded downward and to the right can be constructed by diagonally padding (Z2) external pixels adjacent to the generated region (A2).

[0284] Referring to part 8d, the generated regions (B'0 to B'2) can be constructed with reference to data of specific regions (B0 to B2) included in the image. In part 8d, unlike part 8c, a region that is not adjacent to the generated region can be referred to.

[0285] For example, when there is a region having high correlation with the generated region in the image, the generated region can be filled with pixels of the region having high correlation. In this case, position information, size information, etc. of the region having high correlation can be generated. Alternatively, when there is a region having high correlation by encoding / decoding information of characteristics, types, etc. of the image and position information, size information, etc. of the region having high correlation can be implicitly checked (for example, for a 360-degree image), the generated region can be filled with data of the corresponding region. In this case, position information, size information, etc. of the corresponding region can be omitted.

[0286] As an example, a region (B'2) generated when the image is expanded to the left or right in association with horizontal image resizing can be filled with reference to pixels in a region (B2) opposite to the generated region when the image is expanded to the left or right in association with horizontal resizing.

[0287] As an example, a region (B'1) generated when an image is expanded upward or downward in association with portrait image size adjustment can be filled with reference to pixels in a region (B1) opposite to the generated region when the image is expanded upward or downward in association with portrait size adjustment.

[0288] As an example, a region (B'0) generated when an image is expanded by some image size adjustment (here, diagonally with respect to the image center) can be filled with reference to pixels in a region (B0, TL) opposite to the generated region.

[0289] Examples in which continuity exists at a boundary between both ends of an image and data about a region symmetrical with respect to a size adjustment direction is acquired have been described. However, the present application is not limited to this, and data of other regions (TL to BR) can be acquired.

[0290] When a generated region is filled with data of a specific region of an image, data of the corresponding region can be copied as is and used to fill the generated region, or data of the corresponding region can be transformed based on characteristics, types, etc. of the image and used to fill the generated region. In this case, copying data as is can mean using pixel values of the corresponding region without any change, and performing a transformation process can mean not using pixel values of the corresponding region without any change. That is, at least one pixel value of the corresponding region can be changed by a transformation process. The generated region can be filled with the changed pixel value, or at least one of the positions of some pixels can be different from other positions. That is, in order to fill an AxB generated region, CxD data other than AxB data of the corresponding region can be used. In other words, at least one of the motion vectors applied to the pixels used to fill the generated region can be different from other pixels. In the above example, when a 360-degree image is composed of a plurality of faces according to a projection format, the generated region can be filled with data of other faces. The data processing method for filling a region generated when an image is expanded by image size adjustment is not limited to the above-described example. The data processing method can be improved or changed, or an additional data processing method can be used.

[0291] A plurality of candidate groups for a data processing method can be supported according to encoding / decoding settings, and information about selecting a data processing method from among the plurality of candidate groups can be generated and added to a bitstream. For example, one data processing method can be selected from among a filling method by using a predetermined pixel value, a filling method by copying an external pixel, a filling method by copying a specific region of an image, a filling method by transforming a specific region of an image, etc., and related selection information can be generated. Furthermore, a data processing method can be determined implicitly.

[0292] For example, the data processing method applied to all regions to be generated along with the expansion through the image size adjustment (here, the regions TL to BR in the portion 7a) can be one of a padding method by using a predetermined pixel value, a padding method by copying an external pixel, a padding method by copying a specific region of an image, a padding method by transforming a specific region of an image, and the like, and the relevant selection information can be generated. Further, one predetermined data processing method applied to the entire region can be determined.

[0293] Alternatively, the data processing method applied to each of the regions TL to BR in the portion 7a or two or more regions to be generated along with the expansion through the image size adjustment (here, the regions TL to BR in the portion 7a) can be one of a padding method by using a predetermined pixel value, a padding method by copying an external pixel, a padding method by copying a specific region of an image, a padding method by transforming a specific region of an image, and the like, and the relevant selection information can be generated. Further, one predetermined data processing method applied to at least one region can be determined. Figure 7

[0294] Figure 9 is an example diagram of a method of constructing a region to be deleted through reduction and a region to be generated in an image size adjustment method according to an embodiment of the present application.

[0295] The region to be deleted in the image reduction processing can not only be simply removed but also removed after a series of application processing.

[0296] Referring to the portion 9a, during the image reduction processing, the specific regions (A0, A1, A2) can be simply removed without additional application processing. In this case, the image (A) can be divided into the regions TL to BR as shown in the portion 8a.

[0297] Referring to the portion 9b, the regions (A0 to A2) can be removed, and the regions (A0 to A2) can be used as reference information when the image (A) is encoded or decoded. For example, the deleted regions (A0 to A2) can be utilized during a process of restoring or correcting the specific regions of the image (A) deleted through the reduction. During the restoration or correction processing, a weighted sum, an average, or the like of two regions (a deleted region and a generated region) can be used. Further, the restoration or correction processing can be a process that can be applied when the two regions have a high correlation.

[0298] ​As an example, the region (B'2) deleted when the image is scaled down to the left or right in association with a horizontal image resizing can be used to restore or correct pixels in the region (B2, LC) opposite to the region deleted when the image is scaled down to the left or right in association with a horizontal image resizing, and that region (B'2) can then be removed from memory.

[0299] As an example, the region (B'1) deleted when the image is scaled up or down in association with portrait image resizing can be used for encoding / decoding (restoration or correction) of the region (B1, TR) opposite to the region deleted when the image is scaled up or down in association with portrait image resizing, and the region (B'1) can then be removed from memory.

[0300] As an example, when an image is reduced in size by some image resizing (here, diagonally relative to the center of the image), the deleted region (B'0) can be used for encoding / decoding (restoration or correction) of the region (B0, TL) opposite to the deleted region, and the region (B'0) can then be removed from memory.

[0301] Examples have been described of data from regions that are continuous at the boundaries between the two ends of an image and symmetrical about the direction of size adjustment, which were used for restoration or correction. However, the invention is not limited thereto, and data from regions TL to BR other than the symmetrical regions can be used for restoration or correction and can then be removed from memory.

[0302] The data processing methods used to remove areas to be deleted are not limited to the examples above. The data processing methods can be improved or changed, or additional data processing methods can be used.

[0303] Multiple candidate groups for data processing methods can be supported based on encoding / decoding settings, and relevant selection information can be generated and added to the bitstream. For example, a data processing method can be selected from methods such as simply removing the region to be deleted, or removing the region after using it in a series of processes, and relevant selection information can be generated. Furthermore, the data processing method can be implicitly determined.

[0304] For example, applied to all areas that are to be deleted as the image size is adjusted (here, Figure 7 The data processing method for regions TL to BR in part 7b can be one of the following: simply removing the region to be deleted, or removing the region after using it in a series of processes, and related selection information can be generated. Furthermore, the data processing method can be implicitly determined.

[0305] Alternative locations are applied to each region to be deleted as it is scaled down by the image size adjustment (here, Figure 7 The data processing method for each of regions TL to BR in part 7b can be one of the following: simply removing the region to be deleted, or removing the region after using it in a series of processes, and related selection information can be generated. Furthermore, the data processing method can be implicitly determined.

[0306] An example of performing a size adjustment based on a size adjustment (expand or shrink) operation has been described. In some cases, this description can be applied to examples where a size adjustment operation (expanding in this case) is performed and then the reverse size adjustment operation (shrinking in this case) is performed.

[0307] For example, one could choose a method to fill the region generated with the expansion using some data from the image, and then choose a method to remove the region after it was deleted during the shrinking process in the process of restoring or correcting some data from the image. Alternatively, one could choose a method to fill the region generated with the expansion by copying outer pixels, and then choose a method to simply remove the region to be deleted during the shrinking process in the reverse processing. That is, the data processing method in the reverse processing can be determined based on the data processing method selected in the image resizing process.

[0308] Unlike the examples above, the data processing methods for image resizing and inverse resizing can be independent. That is, the data processing method in inverse resizing can be chosen regardless of the method selected in image resizing. For example, one could choose a method that uses some data from the image to fill in the area generated with the expansion, and then choose a method that simply removes the area to be deleted with the shrinking in inverse resizing.

[0309] According to the present invention, the data processing method during image resizing processing can be implicitly determined based on the encoding / decoding settings, and the data processing method during inverse processing can also be implicitly determined based on the encoding / decoding settings. Alternatively, the data processing method during image resizing processing can be explicitly generated, and the data processing method during inverse processing can also be explicitly generated. Alternatively, the data processing method during image resizing processing can be explicitly generated, and the data processing method during inverse processing can be implicitly determined based on this data processing method.

[0310] Next, an example of performing image size adjustment in the encoding / decoding apparatus according to the embodiment of the present application will be described. In the following description, as an example, the size adjustment process indicates expansion, and the inverse size adjustment process indicates reduction. Further, the difference between the image before the size adjustment and the image after the size adjustment can refer to the image size, and the size adjustment related information can have some pieces explicitly generated and other pieces implicitly determined according to the encoding / decoding settings. Further, the size adjustment related information can include information on the size adjustment process and the inverse size adjustment process.

[0311] As a first example, the process of adjusting the size of the input image can be performed before starting the encoding. The size of the input image can be adjusted using the size adjustment information (e.g., size adjustment operation, size adjustment direction, size adjustment value, data processing method, etc.; the data processing method is used during the size adjustment process), and then the input image can be encoded. The image encoded data (here, the image after the size adjustment) can be stored in the memory after the encoding is completed, and can be added to the bitstream and then transmitted.

[0312] The size adjustment process can be performed before starting the decoding. The size of the image decoded data can be adjusted using the size adjustment information (e.g., size adjustment operation, size adjustment direction, size adjustment value, etc.), and then the image decoded data can be parsed for decoding. The output image can be stored in the memory after the decoding is completed, and can be changed to the image before the size adjustment (here, using the data processing method, etc.; this is used for the inverse size adjustment process) by performing the inverse size adjustment process.

[0313] As a second example, the process of adjusting the size of the reference picture can be performed before starting the encoding. The size of the reference picture can be adjusted using the size adjustment information (e.g., size adjustment operation, size adjustment direction, size adjustment value, data processing method, etc.; the data processing method is used during the size adjustment process), and then the reference picture (here, the size-adjusted reference picture) can be stored in the memory. The image can be encoded using the size-adjusted reference picture. After the encoding is completed, the image encoded data (here, the data obtained by encoding using the reference picture) can be added to the bitstream and then transmitted. Further, when the encoded image is stored in the memory as the reference picture, the above-described size adjustment process can be performed.

[0314] Before starting decoding, a size adjustment process can be performed on the reference picture. The size of the reference picture can be adjusted using size adjustment information (e.g., size adjustment operation, size adjustment direction, size adjustment value, data processing method, etc.; data processing method is used during size adjustment process), and then the reference picture (here, the size-adjusted reference picture) can be stored in the memory. The image decoding data (here, encoded by using the reference picture by the encoder) can be parsed for decoding. After decoding is completed, the output image can be generated. When the decoded image is stored in the memory as the reference picture, the above-mentioned size adjustment process can be performed.

[0315] As a third example, a size adjustment process can be performed on the image before filtering the image (here, assuming a deblocking filter) and after encoding (in detail, after encoding is completed except for the filtering process). The size of the image can be adjusted using size adjustment information (e.g., size adjustment operation, size adjustment direction, size adjustment value, data processing method, etc.; data processing method is used during size adjustment), and then the image after size adjustment can be generated and then filtered. After filtering is completed, inverse size adjustment process is performed so that the image after size adjustment can be changed to the image before size adjustment.

[0316] After decoding is completed (in detail, after decoding is completed except for the filtering process), and before filtering, a size adjustment process can be performed on the image. The size of the image can be adjusted using size adjustment information (e.g., size adjustment operation, size adjustment direction, size adjustment value, data processing method, etc.; data processing method is used during size adjustment), and then the image after size adjustment can be generated and then filtered. After filtering is completed, inverse size adjustment process is performed so that the image after size adjustment can be changed to the image before size adjustment.

[0317] In some cases (first example and third example), size adjustment process and inverse size adjustment process can be performed. In other cases (second example), only size adjustment process can be performed.

[0318] Further, in some cases (second and third examples), the same resizing process can be applied to the encoder and the decoder. In other cases (first example), the same or different resizing process can be applied to the encoder and the decoder. Here, the resizing process of the encoder and the decoder can differ in terms of the resizing execution step. For example, in some cases (here, the encoder), the resizing execution step considering image resizing and data processing for the resized region can be included. In other cases (here, the decoder), the resizing execution step considering image resizing can be included. Here, the former data processing can correspond to the latter data processing during the inverse resizing process.

[0319] Further, in some cases (third example), the resizing process can be applied only to the corresponding step, and the resized region can not be stored in the memory. For example, in order to use the resized region in the filtering process, the resized region can be stored in a temporary memory, filtered, and then removed through the inverse resizing process. In this case, there is no change in the size of the image due to the resizing. The present application is not limited to the above-described examples, and modifications can be made to the above-described examples.

[0320] The size of the image can be changed by the resizing process, and thus the coordinates of some pixels of the image can be changed by the resizing process. This can affect the operation of the picture partitioning section. According to the present application, by this process, block-based partitioning can be performed based on the image before the resizing or the image after the resizing. Further, unit (e.g., tile, slice, etc.) based partitioning can be performed based on the image before the resizing or the image after the resizing, which can be determined according to the encoding / decoding setting. According to the present application, the following description focuses on the case where the picture partitioning section operates based on the image after the resizing (e.g., image partitioning process after the resizing process), but other modifications can be made. The above-described examples will be described under a plurality of image settings to be described below.

[0321] The encoder can add information generated during the above-described process to the bitstream in units of at least one of a sequence, a picture, a slice, a tile, etc., and the decoder can parse the related information from the bitstream. Further, the information can be included in the bitstream in the form of SEI or metadata.

[0322] In general, the input image can be encoded or decoded as is or after image reconstruction. For example, image reconstruction can be performed to improve image encoding efficiency, image reconstruction can be performed to consider network and user environments, and image reconstruction can be performed according to the type, characteristics, etc. of the image.

[0323] According to the present application, the image reconstruction process can include a reconstruction process alone or can include a reconstruction process and an inverse reconstruction process. The following examples will describe focusing on the reconstruction process, but the inverse reconstruction process can be derived in reverse according to the reconstruction process.

[0324] Figure 10 is an example diagram of image reconstruction according to an embodiment of the present application.

[0325] The portion 10a shows an initial input image. The portions 10a to 10d are example diagrams of image rotation including predetermined angles of 0 degrees (for example, a candidate group can be generated by sampling 360 degrees into k portions; k can have values of 2, 4, 8, etc.; in this example, it is assumed that k is 4). The portions 10e to 10h are example diagrams having an inverse (or symmetric) relationship with respect to the portion 10a or with respect to the portions 10b to 10d.

[0326] The start position or scan order of the image can be changed according to the image reconstruction, but the start position and scan order can be predetermined regardless of the reconstruction, which can be determined according to the encoding / decoding settings. The following embodiments assume that the start position (for example, the top-left position of the image) and the scan order (for example, raster scan) are predetermined regardless of the image reconstruction.

[0327] The image encoding method and the image decoding method according to the embodiment of the present application can include the following image reconstruction steps. In this case, the image reconstruction process can include an image reconstruction indication step, an image reconstruction type identification step, and an image reconstruction execution step. Furthermore, the image encoding apparatus and the image decoding apparatus can be configured to include an image reconstruction indication section, an image reconstruction type identification section, and an image reconstruction execution section that respectively perform the image reconstruction indication step, the image reconstruction type identification step, and the image reconstruction execution step. For encoding, the relevant syntax elements can be generated. For decoding, the relevant syntax elements can be parsed.

[0328] In the image reconstruction indication step, it can be determined whether to perform the image reconstruction. For example, when a signal indicating the image reconstruction (for example, convert_enabled_flag) is confirmed, the reconstruction can be performed. When a signal indicating the image reconstruction is not confirmed, the reconstruction can not be performed, or the reconstruction can be performed by confirming other encoding / decoding information. Furthermore, although a signal indicating the image reconstruction is not provided, the signal indicating the image reconstruction can be implicitly activated or deactivated according to the encoding / decoding settings (for example, the characteristics, type, etc. of the image). When the reconstruction is performed, the corresponding reconstruction-related information can be generated, or the corresponding reconstruction-related information can be implicitly determined.

[0329] When a signal indicating image reconstruction is provided, the corresponding signal is a signal for indicating whether to perform image reconstruction. Whether to reconstruct the corresponding image can be determined according to the signal. For example, it is assumed that a signal indicating image reconstruction (e.g., convert_enabled_flag) is confirmed. When the corresponding signal is activated (e.g., convert_enabled_flag = 1), reconstruction can be performed. When the corresponding signal is deactivated (e.g., convert_enabled_flag = 0), reconstruction can not be performed.

[0330] In addition, when a signal indicating image reconstruction is not provided, reconstruction can not be performed, or whether to reconstruct the corresponding image can be determined through another signal. For example, reconstruction can be performed according to the characteristics, type, etc. of the image (e.g., a 360-degree image), and reconstruction information can be explicitly generated or can be designated as a predetermined value. The present application is not limited to the above-described example, and the above-described example can be modified.

[0331] In the image reconstruction type identification step, the image reconstruction type can be identified. The image reconstruction type can be defined by a reconstruction method, reconstruction mode information, etc. The reconstruction method (e.g., convert_type_flag) can include flipping, rotation, etc., and the reconstruction mode information can include a mode of the reconstruction method (e.g., convert_mode). In this case, the reconstruction-related information can consist of the reconstruction method and the mode information. That is, the reconstruction-related information can consist of at least one syntax element. In this case, the number of candidate sets of the mode information can be the same or different depending on the reconstruction method.

[0332] As an example, rotation can include candidates having uniform intervals (here, 90 degrees), as shown in parts 10a to 10d. Part 10a shows 0-degree rotation, part 10b shows 90-degree rotation, part 10c shows 180-degree rotation, and part 10d shows 270-degree rotation (here, it is measured clockwise).

[0333] As an example, flipping can include candidates as shown in parts 10a, 10e, and 10f. When part 10a shows no flipping, parts 10e and 10f show horizontal flipping and vertical flipping, respectively.

[0334] In the above example, the setting for rotation having uniform intervals and the setting for flipping have been described. However, this is only an example of image reconstruction, and the present application is not limited thereto, and the present application can include another interval difference, another flipping operation, etc. that can be determined according to encoding / decoding settings.

[0335] Alternatively, comprehensive information (e.g., convert_com_flag) generated by mixing a reconstruction method and corresponding mode information can be included. In this case, reconstruction-related information can consist of a reconstruction method and mode information mixed.

[0336] For example, the comprehensive information can include candidates as shown in parts 10a to 10f, which can be examples of 0-degree rotation, 90-degree rotation, 180-degree rotation, 270-degree rotation, horizontal flipping, and vertical flipping with respect to part 10a.

[0337] Alternatively, the comprehensive information can include candidates as shown in parts 10a to 10h, which can be examples of 0-degree rotation, 90-degree rotation, 180-degree rotation, 270-degree rotation, horizontal flipping, vertical flipping, 90-degree rotation and then horizontal flipping (or horizontal flipping and then 90-degree rotation), and 90-degree rotation and then vertical flipping (or vertical flipping and then 90-degree rotation) or examples of 0-degree rotation, 90-degree rotation, 180-degree rotation, 270-degree rotation, horizontal flipping, 180-degree rotation and then horizontal flipping (or horizontal flipping and then 180-degree rotation), 90-degree rotation and then horizontal flipping (or horizontal flipping and then 90-degree rotation), and 270-degree rotation and then horizontal flipping (or horizontal flipping and then 270-degree rotation).

[0338] The candidate group can be configured to include a rotation mode, a flipping mode, and a combined mode of rotation and flipping. The combined mode can simply include mode information in a reconstruction method, and can include a mode generated by mixing mode information in each method. In this case, the combined mode can include a mode generated by mixing at least one mode (e.g., rotation) of some methods and at least one mode (e.g., flipping) of other methods. In the above example, the combined mode includes a case where a mode of some methods is combined with multiple modes of some methods (here, 90-degree rotation + multiple flipping / horizontal flipping + multiple rotation). The information of the mixed construction can include a case where reconstruction is not applied (here, part 10a) as a candidate group, and can include a case where reconstruction is not applied as a first candidate group (e.g., #0 is designated as an index).

[0339] Alternatively, the information of the mixed construction can include mode information corresponding to a predetermined reconstruction method. In this case, reconstruction-related information can consist of mode information corresponding to a predetermined reconstruction method. That is, information about a reconstruction method can be omitted, and reconstruction-related information can consist of one syntax element associated with mode information.

[0340] For example, the reconstruction-related information can be configured to include rotation-specific candidates as shown in parts 10a to 10d. Alternatively, the reconstruction-related information can be configured to include flip-specific candidates as shown in parts 10a, 10e, and 10f.

[0341] The image before the image reconstruction processing and the image after the image reconstruction processing can have the same size or at least one different length, which can be determined according to the encoding / decoding setting. The image reconstruction processing can be a processing of rearranging pixels in the image (here, inverse pixel rearrangement processing is performed during the inverse image reconstruction processing; this can be derived inversely according to the pixel rearrangement processing), and thus can change the position of at least one pixel. The pixel rearrangement can be performed according to a rule based on the image reconstruction type information.

[0342] In this case, the pixel rearrangement can be affected by the size and shape (e.g., square or rectangular) of the image. In detail, the width and height of the image before the reconstruction processing and the width and height of the image after the reconstruction processing can serve as variables during the pixel rearrangement processing.

[0343] For example, at least one of the ratio information (e.g., former / latter or latter / former) regarding the ratio of the width of the image before the reconstruction processing to the width of the image after the reconstruction processing, the ratio of the width of the image before the reconstruction processing to the height of the image after the reconstruction processing, the ratio of the height of the image before the reconstruction processing to the width of the image after the reconstruction processing, and the ratio of the height of the image before the reconstruction processing to the height of the image after the reconstruction processing can serve as a variable during the pixel rearrangement processing.

[0344] In this example, when the image before the reconstruction processing and the image after the reconstruction processing have the same size, the ratio of the width of the image to the height of the image can serve as a variable during the pixel rearrangement processing. Further, when the image is in a square shape, the ratio of the length of the image before the reconstruction processing to the length of the image after the reconstruction processing can serve as a variable during the pixel rearrangement processing.

[0345] In the image reconstruction performing step, the image reconstruction can be performed based on the identified reconstruction information. That is, the image reconstruction can be performed based on the information regarding the reconstruction type, the reconstruction mode, and the like, and the encoding / decoding can be performed based on the reconstructed image.

[0346] Next, an example of performing the image reconstruction in the encoding / decoding apparatus according to the embodiment of the present application will be described.

[0347] The process of reconstructing the input image can be performed before starting encoding. The reconstruction can be performed using reconstruction information (e.g., image reconstruction type, reconstruction mode, etc.), and the reconstructed image can be encoded. The image encoding data can be stored in a memory after the encoding is completed, and can be added to a bitstream and then transmitted.

[0348] The reconstruction process can be performed before starting decoding. The reconstruction can be performed using reconstruction information (e.g., image reconstruction type, reconstruction mode, etc.), and the image decoding data can be parsed for decoding. The image can be stored in a memory after the decoding is completed, and can be changed into the image before the reconstruction and then output by performing an inverse reconstruction process.

[0349] The encoder can add information generated during the above-described processes to a bitstream in units of at least one of a sequence, a picture, a slice, a tile, etc., and the decoder can parse the related information from the bitstream. Furthermore, the information can be included in the bitstream in the form of SEI or metadata.

[0350] [Table 1]

[0351]

[0352] Table 1 represents example syntax elements associated with partitioning among image settings. The following description will focus on additional syntax elements. Furthermore, in the following examples, the syntax elements are not limited to any particular unit, and can be supported in various units such as a sequence, a picture, a slice, and a tile. Alternatively, the syntax elements can be included in SEI, metadata, etc. Furthermore, the type, order, condition, etc. of the syntax elements supported in the following examples are limited to the example, and thus can be changed and determined according to encoding / decoding settings.

[0353] In Table 1, tile_header_enabled_flag represents a syntax element indicating whether encoding / decoding settings for a tile are supported. When the syntax element is activated (tile_header_enabled_flag = 1), encoding / decoding settings in a tile unit can be provided. When the syntax element is deactivated (tile_header_enabled_flag = 0), encoding / decoding settings in a tile unit cannot be provided, and encoding / decoding settings in an upper unit can be assigned.

[0354] Further, tile_coded_flag indicates a syntax element indicating whether a tile is encoded or decoded. When the syntax element is activated (tile_coded_flag = 1), the corresponding tile can be encoded or decoded. When the syntax element is deactivated (tile_coded_flag = 0), the corresponding tile cannot be encoded or decoded. Here, not performing encoding can mean that encoding data is not generated for the corresponding tile (here, it is assumed that the corresponding region is processed by a predetermined rule or the like; a meaningless region in some projection formats applicable to 360-degree images). Not performing decoding means that decoding data in the corresponding tile is no longer parsed (here, it is assumed that the corresponding region is processed by a predetermined rule). Further, not parsing the decoding data can mean that there is no encoding data in the corresponding unit and thus parsing is no longer performed, and can also mean that even if the encoding data exists, parsing is no longer performed by the flag. The header information of the tile unit can be supported according to whether the tile is encoded or decoded.

[0355] The above examples focus on tiles. However, the present application is not limited to tiles, and the above description can be modified and then applied to other division units of the present application. Further, the examples of the tile division setting are not limited to the above-described cases, and the above-described cases can be modified.

[0356] [Table 2]

[0357]

[0358] Table 2 indicates example syntax elements associated with reconstruction among image settings.

[0359] Referring to Table 2, convert_enabled_flag indicates a syntax element indicating whether reconstruction is performed. When the syntax element is activated (convert_enabled_flag = 1), a reconstructed image is encoded or decoded, and additional reconstruction-related information can be checked. When the syntax element is deactivated (convert_enabled_flag = 0), an original image is encoded or decoded.

[0360] Further, convert_type_flag indicates mixed information about reconstruction methods and mode information. One method can be determined from a plurality of candidate groups of a method applying rotation, a method applying flipping, and a method applying rotation and flipping.

[0361] [Table 3]

[0362]

[0363] Table 3 indicates example syntax elements associated with resizing among image settings.

[0364] Referring to Table 3, pic_width_in_samples and pic_height_in_samples denote syntax elements indicating the width and height of the picture. The size of the picture can be checked through the syntax elements.

[0365] In addition, img_resizing_enabled_flag denotes a syntax element indicating whether image size adjustment is performed. When the syntax element is activated (img_resizing_enabled_flag = 1), the picture is encoded or decoded after the size adjustment, and additional size adjustment-related information can be checked. When the syntax element is deactivated (img_resizing_enabled_flag = 0), the original picture is encoded or decoded. In addition, the syntax element can indicate size adjustment for intra prediction.

[0366] In addition, resizing_met_flag indicates a size adjustment method. One size adjustment method can be determined from a candidate group such as a scaling factor-based size adjustment method (resizing_met_flag = 0), an offset factor-based size adjustment method (resizing_met_flag = 1), etc.

[0367] In addition, resizing_mov_flag denotes a syntax element for a size adjustment operation. For example, one of expansion and reduction can be determined.

[0368] In addition, width_scale and height_scale denote scaling factors associated with horizontal and vertical size adjustments of the scaling factor-based size adjustment.

[0369] In addition, top_height_offset and bottom_height_offset denote an "up" direction offset factor and a "down" direction offset factor associated with horizontal size adjustment of the offset factor-based size adjustment, and left_width_offset and right_width_offset denote a "left" direction offset factor and a "right" direction offset factor associated with vertical size adjustment of the offset factor-based size adjustment.

[0370] The size of the picture after the size adjustment can be updated through the size adjustment-related information and the picture size information.

[0371] Further, the resizing_type_flag indicates a syntax element indicating a data processing method for the resized region. The number of candidate groups of the data processing method can be the same or different according to the resizing method and the resizing operation.

[0372] The image setting processing applied to the above-described image encoding / decoding apparatus can be performed individually or in combination. The following examples will focus on examples in which a plurality of image setting processes are performed in combination.

[0373] Figure 11 FIG. 11 is an example diagram showing images before and after the image setting processing according to the embodiment of the present application. In detail, part 11a shows an example before performing image reconstruction on the divided image (e.g., an image projected during 360-degree image encoding), and part 11b shows an image after performing image reconstruction on the divided image (e.g., an image packed during 360-degree image encoding). That is, it can be understood that part 11a is an example diagram before performing the image setting processing, and part 11b is an example diagram after performing the image setting processing.

[0374] In this example, image division (here, it is assumed that tiles) and image reconstruction are described as the image setting processing.

[0375] In the following examples, image reconstruction is performed after performing image division. However, according to the encoding / decoding setting, image division can be performed after performing image reconstruction, and can be modified. Further, the above-described image reconstruction processing (including inverse processing) can be applied identically or similarly to the reconstruction processing in the division unit in the image in the present embodiment.

[0376] Image reconstruction can or can not be performed in all division units in the image, and image reconstruction can be performed in some division units. Therefore, the division unit before reconstruction (e.g., some of P0 to P5) can be the same as or different from the division unit after reconstruction (e.g., some of S0 to S5). Through the following examples, various image reconstruction cases will be described. Further, for ease of description, it is assumed that the unit of the image is a picture, the unit of the divided image is a tile, and the division unit is a rectangular shape.

[0377] As an example, whether to perform image reconstruction can be determined in some cells (e.g., sps_convert_enabled_flag, SEI, or metadata, etc.). Alternatively, whether to perform image reconstruction can be determined in some cells (e.g., pps_convert_enabled_flag). This can be enabled when it first appears in the corresponding cell (here, the image) or when it is activated in a higher-level cell (e.g., sps_convert_enabled_flag = 1). Alternatively, whether to perform image reconstruction can be determined in some cells (e.g., tile_convert_flag[i]; i is the tile cell index). This can be enabled when it first appears in the corresponding cell (here, the tile) or when it is activated in a higher-level cell (e.g., pps_convert_enabled_flag = 1). Furthermore, in part, whether to perform image reconstruction can be implicitly determined based on the encoding / decoding settings, so this information can be omitted.

[0378] As an example, it can be determined whether to reconstruct partitions in an image based on a signal indicating image reconstruction (e.g., pps_convert_enabled_flag). More specifically, it can be determined whether to reconstruct all partitions in the image based on this signal. In this case, a single signal indicating image reconstruction can be generated within the image.

[0379] As an example, it can be determined whether to reconstruct partitions in an image based on a signal indicating image reconstruction (e.g., tile_convert_flag[i]). More specifically, it can be determined whether to reconstruct some partitions in the image based on this signal. In this case, at least one signal indicating image reconstruction can be generated (e.g., a number of signals equal to the number of partitions).

[0380] As an example, it can be determined whether to reconstruct an image based on a signal indicating image reconstruction (e.g., pps_convert_enabled_flag), and it can be determined whether to reconstruct partitioning units in the image based on a signal indicating image reconstruction (e.g., tile_convert_flag[i]). Specifically, when any signal is activated (e.g., pps_convert_enabled_flag = 1), any other signal (e.g., tile_convert_flag[i]) can be additionally checked, and it can be determined whether to reconstruct some partitioning units in the image based on that signal (here, tile_convert_flag[i]). In this case, multiple signals indicating image reconstruction can be generated.

[0381] When a signal indicative of image reconstruction is activated, image reconstruction-related information can be generated. The following examples will describe various types of image reconstruction-related information.

[0382] As an example, reconstruction information can be generated for an image. Specifically, a single piece of reconstruction information can be used as the reconstruction information for all partitioning units in the image.

[0383] As an example, reconstruction information can be generated for partitioning units in an image. Specifically, at least one piece of reconstruction information can be used as reconstruction information for some partitioning units in the image. That is, one piece of reconstruction information can be used as reconstruction information for one partitioning unit, or one piece of reconstruction information can be used as reconstruction information for multiple partitioning units.

[0384] The following example will be described in conjunction with an example of performing image reconstruction.

[0385] For example, when a signal indicating image reconstruction (e.g., pps_convert_enabled_flag) is activated, reconstruction information commonly applied to partitioning units in the image can be generated. Alternatively, when a signal indicating image reconstruction (e.g., pps_convert_enabled_flag) is activated, reconstruction information individually applied to partitioning units in the image can be generated. Alternatively, when a signal indicating image reconstruction (e.g., tile_convert_flag[i]) is activated, reconstruction information individually applied to partitioning units in the image can be generated. Alternatively, when a signal indicating image reconstruction (e.g., tile_convert_flag[i]) is activated, reconstruction information commonly applied to partitioning units in the image can be generated.

[0386] Reconstructed information can be processed implicitly or explicitly based on encoding / decoding settings. For implicit processing, the reconstructed information can be specified as predetermined values ​​based on the characteristics and type of the image.

[0387] P0 to P5 in part 11a can correspond to S0 to S5 in part 11b, and a reconstruction process can be performed on the partitioned units. For example, P0 can be left unreconstructed and then assigned to S0. P1 can be rotated 90 degrees and then assigned to S1. P2 can be rotated 180 degrees and then assigned to S2. P3 can be horizontally flipped and then assigned to S3. P4 can be rotated 90 degrees and horizontally flipped and then assigned to S4. P5 can be rotated 180 degrees and horizontally flipped and then assigned to S5.

[0388] However, the present invention is not limited to the above examples, and various modifications can be made to the above examples. Similar to the above examples, it is possible not to reconstruct the partitioning units in the image, or at least one of reconstruction using rotation, reconstruction using flipping, and reconstruction using a combination of rotation and flipping can be performed.

[0389] When image reconstruction is applied to partitioning units, additional reconstruction processing, such as rearranging the partitioning units, can be performed. That is, the image reconstruction processing according to the invention can be configured to include rearranging the partitioning units in the image as well as rearranging the pixels in the image, and can be represented using some of the syntax elements in Table 4 (e.g., part_top, part_left, part_width, part_height, etc.). This means that image partitioning processing and image reconstruction processing can be understood in combination. In the example above, the image has been described as being divided into multiple units.

[0390] P0 to P5 in part 11a can correspond to S0 to S5 in part 11b, and a reconstruction process can be performed on the partitioned units. For example, P0 can be left unreconstructed and then assigned to S0. P1 can be left unreconstructed and then assigned to S2. P2 can be rotated 90 degrees and then assigned to S1. P3 can be horizontally flipped and then assigned to S4. P4 can be rotated 90 degrees and horizontally flipped and then assigned to S5. P5 can be horizontally flipped and then rotated 180 degrees and then assigned to S3. The invention is not limited thereto and various modifications are possible.

[0391] also, Figure 7 P_Width and P_Height can correspond to Figure 11 P_Width and P_Height, and Figure 7 P'_Width and P'_Height can correspond to Figure 11 P'_Width and P'_Height. Figure 7 The dimensions of the resized image, P'_Width × P'_Height, can be expressed as (P_Width + Exp_L + Exp_R) × (P_Height + Exp_T + Exp_B), and Figure 11 The dimensions of the resized image, P'_Width×P'_Height, can be represented as (P_Width+Var0_L+Var1_L+Var2_L+Var0_R+Var1_R+Var2_R)×(P_Height+Var0_T+Var1_T+Var0_B+Var1_B) or (Sub_P0_Width+Sub_P1_Width+Sub_P2_Width+Var0_L+Var1_L+Var2_L+Var0_R+Var1_R+Var2_R)×(Sub_P0_Height+Sub_P1_Height+Var0_T+Var1_T+Var0_B+Var1_B).

[0392] Similar to the example above, for image reconstruction, one can perform a rearrangement of pixels within the image's partitions, or a rearrangement of the image's partitions, or both. In this case, the rearrangement of the image's partitions can be performed after the rearrangement of pixels within the partitions, or vice versa.

[0393] Whether to perform a rearrangement of the partitioned units in the image can be determined based on the signal indicating image reconstruction. Alternatively, a signal for rearranging the partitioned units in the image can be generated. Specifically, this signal can be generated when the signal indicating image reconstruction is activated. Alternatively, this signal can be processed implicitly or explicitly based on encoding / decoding settings. For implicit processing, the signal can be determined based on the characteristics, type, etc., of the image.

[0394] Furthermore, information regarding the rearrangement of partition units in an image can be implicitly or explicitly executed based on encoding / decoding settings, and this information can be determined based on the characteristics and type of the image. That is, each partition unit can be arranged according to pre-determined arrangement information for the partition units.

[0395] Next, an example of reconstructing segmentation units in an image in an encoding / decoding apparatus according to an embodiment of the present invention will be described.

[0396] The input image can be partitioned using partitioning information before encoding begins. Reconstruction information can be used to reconstruct the partitioned units, and the reconstructed image for each partitioned unit can be encoded. The encoded image data can be stored in memory after encoding is complete, and can be added to a bitstream and then transmitted.

[0397] Partitioning information can be used to perform partitioning processing before decoding begins. Reconstruction information can be used to perform reconstruction processing on the partitioned units, and image decoding data can be parsed to perform decoding within the reconstructed partitioned units. After decoding is complete, the image decoding data can be stored in memory, and multiple partitioned units can be merged into a single unit after performing inverse reconstruction processing within the partitioned units, thereby outputting the image.

[0398] Figure 12 This is an example diagram illustrating the adjustment of the size of each segmentation unit of an image according to an embodiment of the present invention. Figure 12 P0 to P5 correspond to Figure 11 P0 to P5, and Figure 12 S0 to S5 correspond to Figure 11 S0 to S5.

[0399] In the following examples, the description will focus on the case where image resizing is performed after image segmentation. However, depending on the encoding / decoding settings, image segmentation can be performed after image resizing, and this process can be modified. Furthermore, the image resizing process described above (including the inverse process) can be applied in the same or similar manner as the image segmentation unit resizing process in this embodiment.

[0400] For example, Figure 7 The TL to BR can correspond to Figure 12 The partitioning units SX (S0 to S5) are TL to BR. Figure 7 S0 and S1 can correspond to Figure 12 PX and SX. Figure 7 P_Width and P_Height can correspond to Figure 12 Sub_PX_Width and Sub_PX_Height. Figure 7 P'_Width and P'_Height can correspond to Figure 12 Sub_SX_Width and Sub_SX_Height. Figure 7 Exp_L, Exp_R, Exp_T, and Exp_B can correspond to Figure 12 VarX_L, VarX_R, VarX_T, and VarX_B, and other factors can also correspond.

[0401] The processing of adjusting the size of the partitioned units in the images in parts 12a to 12f and Figure 7 The difference between image expansion and reduction in parts 7a and 7b lies in the fact that the settings for image expansion or reduction can be proportional to the number of partition units. Furthermore, the process of adjusting the size of partition units in an image can differ from image expansion or reduction in terms of settings applied commonly or individually to the partition units in the image. In the following examples, various size adjustment scenarios will be described, and size adjustment processes can be performed while taking into account the above descriptions.

[0402] According to the present invention, image resizing may or may not be performed on all partitioned units in an image, and image resizing may be performed on some partitioned units. Various image resizing scenarios will be described through the following examples. Furthermore, for ease of description, it is assumed that the resizing operation is used for expansion, the resizing operation is based on an offset factor, the resizing direction is "up," "down," "left," and "right," the resizing direction is set to operate based on resizing information, the unit of the image is a picture, and the unit dividing the image is a tile.

[0403] As an example, whether to perform image resizing can be determined in some cells (e.g., sps_img_resizing_enabled_flag, SEI, or metadata, etc.). Alternatively, whether to perform image resizing can be determined in some cells (e.g., pps_img_resizing_enabled_flag). This can be allowed when it first appears in the corresponding cell (here, the image) or when it is activated in a higher-level cell (e.g., sps_img_resizing_enabled_flag = 1). Alternatively, whether to perform image resizing can be determined in some cells (e.g., tile_resizing_flag[i]; i is the cell index). This can be allowed when it first appears in the corresponding cell (here, the tile) or when it is activated in a higher-level cell. Furthermore, in part, whether to perform image resizing can be implicitly determined based on encoding / decoding settings, so this information can be omitted.

[0404] As an example, it can be determined whether to adjust the size of the partitioned units in an image based on a signal indicating image resizing (e.g., pps_img_resizing_enabled_flag). More specifically, it can be determined whether to adjust the size of all partitioned units in the image based on this signal. In this case, a single signal indicating image resizing can be generated.

[0405] As an example, it can be determined whether to adjust the size of the partitioning units in an image based on a signal indicating image resizing (e.g., tile_resizing_flag[i]). More specifically, it can be determined whether to adjust the size of some partitioning units in the image based on this signal. In this case, at least one signal indicating image resizing can be generated (e.g., a number of signals equal to the number of partitioning units).

[0406] As an example, it can be determined whether to resize an image based on a signal indicating image resizing (e.g., pps_img_resizing_enabled_flag), and it can be determined whether to resize the partitioned units in the image based on a signal indicating image resizing (e.g., tile_resizing_flag[i]). Specifically, when any signal is activated (e.g., pps_img_resizing_enabled_flag = 1), any other signal (e.g., tile_resizing_flag[i]) can be additionally checked, and the resizing of some partitioned units in the image can be performed based on that signal (here, tile_resizing_flag[i]). In this case, multiple signals indicating image resizing can be generated.

[0407] When a signal indicating image resizing is activated, image resizing-related information can be generated. The following examples will describe various types of image resizing-related information.

[0408] As an example, sizing information can be generated for an image. Specifically, a single sizing information or a set of sizing information can be used as the sizing information for all partitioning units in the image. For example, a single sizing information (or a sizing value applied to all supported or allowed sizing directions in the partitioning unit; in this example, a single piece of information) can be generated that is commonly applied to the "up," "down," "left," and "right" directions of the partitioning unit, or a set of sizing information (or a number of sizing information equal to the number of allowed or supported sizing directions in the partitioning unit; in this example, up to four pieces of information) can be generated individually for the "up," "down," "left," and "right" directions.

[0409] As an example, size adjustment information can be generated for partitioning units in an image. Specifically, at least one piece of size adjustment information or a set of size adjustment information can be used as size adjustment information for all partitioning units in the image. That is, one piece of size adjustment information or a set of size adjustment information can be used as size adjustment information for one partitioning unit or as size adjustment information for multiple partitioning units. For example, size adjustment information can be generated that is commonly applied to one partitioning unit in the image in the "up," "down," "left," and "right" directions, or a set of size adjustment information can be generated that is individually applied to the "up," "down," "left," and "right" directions. Alternatively, size adjustment information can be generated that is commonly applied to multiple partitioning units in the image in the "up," "down," "left," and "right" directions, or a set of size adjustment information can be generated that is individually applied to the "up," "down," "left," and "right" directions. The configuration of this size adjustment group implies size adjustment value information regarding at least one size adjustment direction.

[0410] In summary, size adjustment information can be generated that is commonly applied to the partitioning units in an image. Alternatively, size adjustment information can be generated that is applied individually to the partitioning units in an image. The following example will be described with reference to an example of performing image size adjustment.

[0411] For example, when a signal indicating image resizing (e.g., pps_img_resizing_enabled_flag) is activated, resizing information commonly applied to partitioning units in the image can be generated. Alternatively, when a signal indicating image resizing (e.g., pps_img_resizing_enabled_flag) is activated, resizing information individually applied to partitioning units in the image can be generated. Alternatively, when a signal indicating image resizing (e.g., tile_resizing_flag[i]) is activated, resizing information individually applied to partitioning units in the image can be generated. Alternatively, when a signal indicating image resizing (e.g., tile_resizing_flag[i]) is activated, resizing information commonly applied to partitioning units in the image can be generated.

[0412] Image resizing direction and resizing information can be processed implicitly or explicitly based on encoding / decoding settings. For implicit processing, resizing information can be specified to predetermined values ​​based on image characteristics and type.

[0413] The dimension adjustment direction in the dimension adjustment process of the present invention has been described as being at least one of an "up" direction, a "down" direction, a "left" direction, and a "right" direction, and the dimension adjustment direction and dimension adjustment information can be processed explicitly or implicitly. That is, the dimension adjustment value (including 0; which means no dimension adjustment) can be implicitly predetermined for some directions, and the dimension adjustment value (including 0; which means no dimension adjustment) can be explicitly specified for other directions.

[0414] Even within the subdivisions of an image, the size adjustment direction and size adjustment information can be set to implicit or explicit processing, and this can be applied to the subdivisions of the image. For example, there can be settings applied to one subdivision in the image (here, a setting equal to the number of subdivisions can appear), settings applied to multiple subdivisions in the image, or settings applied to all subdivisions in the image (here, a single setting can appear), and at least one setting can appear in the image (e.g., a setting up to the number of subdivisions can appear). Setting information applied to the subdivisions in the image can be collected, and a single set of settings can be defined.

[0415] Figure 13 This is an example diagram of the settings or size adjustments of the division units in an image.

[0416] In detail, Figure 13 Various examples of implicitly or explicitly processing the sizing direction and sizing information for partitioning units in an image are shown. In the following examples, for ease of description, implicit processing assumes that the sizing value for some sizing directions is 0.

[0417] As shown in section 13a, size adjustments can be explicitly handled when the boundaries of the partitioning units match the boundaries of the image (here, thick solid lines), and implicitly handled when the boundaries of the partitioning units do not match the boundaries of the image (thin solid lines). For example, P0 can be adjusted in the "up" and "left" directions (a2, a0), P1 can be adjusted in the "up" direction (a2), P2 can be adjusted in the "up" and "right" directions (a2, a1), P3 can be adjusted in the "down" and "left" directions (a3, a0), P4 can be adjusted in the "down" direction (a3), and P5 can be adjusted in the "down" and "right" directions (a3, a1). In this case, size adjustments in other directions may not be allowed.

[0418] As shown in section 13b, some directions of the partitioning unit (here, up and down) allow explicit size adjustment, and some directions of the partitioning unit (here, left and right) allow explicit size adjustment (here, thick solid lines) when the boundary of the partitioning unit matches the boundary of the image, and allow implicit size adjustment (here, thin solid lines) when the boundary of the partitioning unit does not match the boundary of the image. For example, P0 can be adjusted in the "up", "down", and "left" directions (b2, b3, b0); P1 can be adjusted in the "up" and "down" directions (b2, b3); P2 can be adjusted in the "up", "down", and "right" directions (b2, b3, b1); P3 can be adjusted in the "up", "down", and "left" directions (b3, b4, b0); P4 can be adjusted in the "up" and "down" directions (b3, b4); and P5 can be adjusted in the "up", "down", and "right" directions (b3, b4, b1). In this case, sizing in other directions may not be allowed.

[0419] As shown in section 13c, some directions of the partitioning unit (here, left and right) allow explicit size adjustment, and some directions of the partitioning unit (here, up and down) allow explicit size adjustment (here, thick solid lines) when the boundary of the partitioning unit matches the boundary of the image, and allow implicit size adjustment (here, thin solid lines) when the boundary of the partitioning unit does not match the boundary of the image. For example, P0 can be adjusted in the "up", "left", and "right" directions (c4, c0, c1); P1 can be adjusted in the "up", "left", and "right" directions (c4, c1, c2); P2 can be adjusted in the "up", "left", and "right" directions (c4, c2, c3); P3 can be adjusted in the "down", "left", and "right" directions (c5, c0, c1); P4 can be adjusted in the "down", "left", and "right" directions (c5, c1, c2); and P5 can be adjusted in the "down", "left", and "right" directions (c5, c2, c3). In this case, sizing in other directions may not be allowed.

[0420] Similar to the example above, settings related to image resizing can have various configurations. Multiple sets of settings are supported, allowing setting group selection information to be generated explicitly or implicitly determined based on encoding / decoding settings (e.g., image characteristics, type, etc.).

[0421] Figure 14 This is an example diagram illustrating both the process of adjusting the image size and the process of adjusting the size of the partitioned units in the image.

[0422] Reference Figure 14 The image resizing process and its inverse can be performed in directions e and f, and the resizing process and its inverse can be performed in directions d and g. That is, image resizing can be performed, and then the resizing process can be performed on the resizing units within the image. The order of resizing is not fixed. This means that multiple resizing processes are possible.

[0423] In summary, image resizing can be categorized into image resizing (or resizing the image before segmentation) and resizing of segmented units within the image (or resizing the image after segmentation). Either image resizing or resizing of segmented units within the image may not be performed, or either or both may be performed, depending on the encoding / decoding settings (e.g., image characteristics, type, etc.).

[0424] When performing multiple resizing processes in this example, the image can be resized in at least one of the "up," "down," "left," and "right" directions, and the size of at least one division unit in the image can be adjusted. In this case, the resizing of the division unit to be resized can be performed in at least one of the "up," "down," "left," and "right" directions.

[0425] Reference Figure 14 The dimensions of the image (A) before resizing can be defined as P_Width × P_Height, the dimensions of the image after primary resizing (or the image before secondary resizing; B) can be defined as P'_Width × P'_Height, and the dimensions of the image after secondary resizing (or the image after final resizing; C) can be defined as P”_Width × P”_Height. The image (A) before resizing represents an image without resizing, the image after primary resizing (B) represents an image with some resizing, and the image after secondary resizing (C) represents an image with all resizing. For example, the image after primary resizing (B) can represent an image resizing according to the image division units shown in sections 13a to 13c, and the image after secondary resizing (C) can represent an image resizing by means of... Figure 7 The image is obtained by completely adjusting the size of the image (B) after the primary size adjustment, as shown in part 7a. The opposite is also possible. However, the invention is not limited to the above example, and various modifications can be made to the above example.

[0426] In the dimensions of the image (B) after the initial resizing, P'_Width can be obtained by P_Width and at least one horizontal resizing value for the lateral resizing, and P'_Height can be obtained by P_Height and at least one vertical resizing value for the longitudinal resizing. In this case, the resizing values ​​can be the resizing values ​​generated in the partitioning cells.

[0427] In the dimensions of the image (C) after secondary resizing, P”_Width can be obtained by P'_Width and at least one horizontal resizing value for horizontal resizing, and P”_Height can be obtained by P'_Height and at least one vertical resizing value for vertical resizing. In this case, the resizing values ​​can be the resizing values ​​generated in the image.

[0428] In summary, the size of the resized image can be obtained using at least one resizing value and the size of the image before resizing.

[0429] In the resized areas of an image, information about the data processing methods can be generated. Various data processing methods will be described through the following examples. Data processing methods generated during desizing can be applied in the same or similar manner as those used in resizing. The data processing methods in resizing and desizing will be described through various combinations described below.

[0430] As an example, data processing methods can be generated for application to an image. Specifically, a data processing method or a set of data processing methods can be used as the data processing method for all partitioned units in the image (here, it is assumed that the size of all partitioned units will be adjusted). For example, a data processing method (or a data processing method applied to all supported or allowed size adjustment directions in the partitioned units, etc.; in this example, one message) can be generated for the "up," "down," "left," and "right" directions in the image (or a set of data processing methods applied to the "up," "down," "left," and "right" directions, or a number of data processing methods equal to the number of supported or allowed size adjustment directions in the partitioned units; in this example, up to four messages).

[0431] As an example, data processing methods can be generated for application to partitioned units in an image. Specifically, at least one data processing method or a set of data processing methods can be used as data processing methods for some partitioned units in the image (here, it is assumed that the size of the partitioned units is to be adjusted). That is, a data processing method or a set of data processing methods can be used as a data processing method for one partitioned unit or for multiple partitioned units. For example, a data processing method can be generated that is commonly applied to the "up," "down," "left," and "right" directions of one partitioned unit in the image, or a set of data processing methods can be generated that are individually applied to the "up," "down," "left," and "right" directions. Alternatively, a data processing method can be generated that is commonly applied to the "up," "down," "left," and "right" directions of multiple partitioned units in the image, or a set of data processing methods can be generated that are individually applied to the "up," "down," "left," and "right" directions. The configuration of this set of data processing methods implies a data processing method for adjusting the direction of at least one size.

[0432] In summary, a data processing method commonly applied to partitioning units in an image can be used. Alternatively, a data processing method applied separately to partitioning units in an image can be used. The data processing method can be a predetermined method. A predetermined data processing method can be provided as at least one method. This corresponds to implicit processing, and selection information for the data processing method can be explicitly generated, which can be determined based on encoding / decoding settings (e.g., image characteristics, type, etc.).

[0433] That is, a data processing method commonly applied to the partitioning units in the image can be used. A predetermined method can be used, or one of several data processing methods can be selected. Alternatively, a data processing method applied separately to the partitioning units in the image can be used. Depending on the partitioning units, a predetermined method can be used, or one of several data processing methods can be selected.

[0434] The following examples will describe some cases of adjusting the size of the partitioning units in an image (here, it is assumed that the size adjustment is used for expansion) (here, the size-adjusted region is filled with some data from the image).

[0435] The dimensions of specific regions TL to BR (e.g., S0 to S5 in parts 12a to 12f) of some elements (P0 to P5) can be adjusted using data from specific regions tl to br of some elements (in parts 12a to 12f). In this case, some elements can be the same as each other (e.g., S0 and P0) or different (e.g., S0 and P1). That is, the regions TL to BR to be resized can be filled with some data tl to br of the corresponding partitioned elements, and can also be filled with some data from partitioned elements other than the corresponding partitioned elements.

[0436] As an example, the size of the region TL to BR of the current partition unit can be adjusted using the data tl to br of the current partition unit. For example, TL of S0 can be filled with data tl of P0, RC of S1 can be filled with data tr+rc+br of P1, BL+BC of S2 can be filled with data bl+bc+br of P2, and TL+LC+BL of S3 can be filled with data tl+lc+bl of P3.

[0437] As an example, the size of the region TL to BR of the current partition unit can be adjusted using the data tl to br of the partition units spatially adjacent to the current partition unit. For example, in the "up" direction, TL+TC+TR of S4 can be filled with the data bl+bc+br of P1; in the "down" direction, BL+BC of S2 can be filled with the data tl+tc+tr of P5; in the "left" direction, LC+BL of S2 can be filled with the data tl+rc+bl of P1; in the "right" direction, RC of S3 can be filled with the data tl+lc+bl of P4; and in the "down + left" direction, BR of S0 can be filled with the data tl of P4.

[0438] As an example, the size of the region TL to BR of the current partition unit can be adjusted using data tl to br from partition units that are not spatially adjacent to the current partition unit. For example, data in the boundary regions between the two ends of an image (e.g., horizontal, vertical, etc.) can be acquired. The LC of S3 can be obtained using the data tr+rc+br of S5, the RC of S2 can be obtained using the data tl+lc of S0, the BC of S4 can be obtained using the data tc+tr of S1, and the TC of S1 can be obtained using the data bc of S4.

[0439] Alternatively, data can be obtained for specific regions of the image (regions that are not spatially adjacent to the resized region but are determined to be highly correlated with the resized region). The BC of S1 can be obtained using the data tl+lc+bl of S3, the RC of S3 can be obtained using the data tl+tc of S1, and the RC of S5 can be obtained using the data bc of S0.

[0440] Furthermore, some cases of adjusting the size of the partitioning units in the image (here, assuming the size adjustment is for reduction) are as follows (here, removal is performed by restoring or correcting some data of the image).

[0441] Specific regions TL to BR of some cells (e.g., S0 to S5 in sections 12a to 12f) can be used for the recovery or correction processing of specific regions tl to br of some cells P0 to P5. In this case, some cells may be the same as each other (e.g., S0 and P0) or different (e.g., S0 and P2). That is, regions to be resized can be used to recover some data of the corresponding partition cell and then removed, and regions to be resized can be used to recover some data of partition cells other than the corresponding partition cell and then removed. Detailed examples can be derived from the reverse of the extended process and will therefore be omitted.

[0442] This example can be applied to situations where highly correlated data exists in the area to be resized, and information about the referenced location for the resizing can be explicitly generated or implicitly obtained according to predetermined rules. Alternatively, relevant information can be examined in combination. This could be an example that can be applied when data is obtained from another area with continuity in the encoding of a 360-degree image.

[0443] Next, an example of adjusting the size of the segmentation units in an image in an encoding / decoding apparatus according to an embodiment of the present invention will be described.

[0444] The input image can be partitioned before encoding begins. Size adjustment information can be used to resize the partitioned units, and the image can be encoded after the partitioned units have been resized. The encoded image data can be stored in memory after encoding is complete, and can be added to a bitstream and then transmitted.

[0445] Partitioning information can be used to perform partitioning before decoding begins. Size adjustment information can be used to perform size adjustment on the partitioned units, and image decoding data can be parsed to decode within the size-adjusted partitioned units. After decoding is complete, the image decoding data can be stored in memory, and multiple partitioned units can be merged into a single unit after performing inverse size adjustment on the partitioned units, thus allowing the image to be output.

[0446] Other examples of the image resizing process described above can be applied. The invention is not limited thereto and can be modified.

[0447] In image settings processing, it is permissible to combine image resizing and image reconstruction. Image reconstruction can be performed after image resizing. Alternatively, image resizing can be performed after image reconstruction. Furthermore, it is permissible to combine image partitioning, image reconstruction, and image resizing. Image resizing and image reconstruction can be performed after image partitioning. The order of image settings is not fixed and can be changed, depending on the encoding / decoding settings. In this example, image settings processing is described as performing image reconstruction and image resizing after image partitioning. However, depending on the encoding / decoding settings, other orders are possible and can be modified.

[0448] For example, image setup processing can be performed in the following order: partitioning -> reconstruction; reconstruction -> partitioning; partitioning -> resizing; resizing -> partitioning; resizing -> reconstruction; reconstruction -> resizing; partitioning -> reconstruction -> resizing; partitioning -> resizing -> reconstruction; resizing -> partitioning -> reconstruction; resizing -> reconstruction -> partitioning; and reconstruction -> resizing -> partitioning, and combinations with additional image settings are possible. As mentioned above, image setup processing can be performed sequentially, but some or all of the setup processing can be performed simultaneously. Furthermore, as some image setup processing, multiple processes can be performed based on encoding / decoding settings (e.g., image characteristics, type, etc.). The following examples illustrate various combinations of image setup processing.

[0449] As an example, P0 to P5 in section 11a can correspond to S0 to S5 in section 11b, and reconstruction processing (here, pixel rearrangement) and size adjustment processing (here, adjusting the size of the division units to have the same size) can be performed in the partitioning units. For example, the size of P0 to P5 can be adjusted based on the offset, and P0 to P5 can be assigned to S0 to S5. Alternatively, P0 can be reconstructed without reconstructing, and then P0 can be assigned to S0. P1 can be rotated 90 degrees and then assigned to S1. P2 can be rotated 180 degrees and then assigned to S2. P3 can be rotated 270 degrees and then assigned to S3. P4 can be horizontally flipped and then assigned to S4. P5 can be vertically flipped and then assigned to S5.

[0450] As an example, P0 to P5 in section 11a may correspond to the same or different positions as S0 to S5 in section 11b, and reconstruction processing (here, rearrangement of pixels and division units) and size adjustment processing (here, adjusting the size of the division units to have the same size) can be performed in the division units. For example, the size of P0 to P5 can be adjusted based on a ratio, and P0 to P5 can be assigned to S0 to S5. Alternatively, P0 may not be reconstructed, and then P0 can be assigned to S0. P1 may not be reconstructed, and then P1 can be assigned to S2. P2 may be rotated 90 degrees, and then it can be assigned to S1. P3 may be horizontally flipped, and then it can be assigned to S4. P4 may be rotated 90 degrees and horizontally flipped, and then it can be assigned to S5. P5 may be horizontally flipped and then rotated 180 degrees, and then it can be assigned to S3.

[0451] As an example, P0 to P5 in section 11a can correspond to E0 to E5 in section 5e, and reconstruction processing (here, rearrangement of pixels and division units) and size adjustment processing (here, adjusting the size of the division units to have different sizes) can be performed in the division units. For example, P0 can be resized and reconstructed without being resized and then assigned to E0, P1 can be resized based on the scale but not reconstructed and then assigned to E1, P2 can be resized without being resized but reconstructed and then assigned to E2, P3 can be resized based on the offset but not reconstructed and then assigned to E4, P4 can be resized without being resized but reconstructed and then assigned to E5, and P5 can be resized based on the offset and reconstructed and then assigned to E3.

[0452] Similar to the examples above, the absolute or relative positions of the partitioning units in the image before and after image setup processing can be maintained or changed, which can be determined based on encoding / decoding settings (e.g., image characteristics, type, etc.). Furthermore, various combinations of image setup processing are possible. The invention is not limited thereto, and therefore various modifications are possible.

[0453] The encoder can add information generated during the above processing to the bitstream in units of at least one of sequences, images, slices, tiles, etc., and the decoder can parse the relevant information from the bitstream. Furthermore, the information can be included in the bitstream in the form of SEI or metadata.

[0454] [Table 4]

[0455]

[0456] Table 4 shows example syntax elements associated with multiple image settings. The following description will focus on additional syntax elements. Furthermore, in the following examples, syntax elements are not limited to any specific unit and can be supported in various units such as sequences, images, slices, and tiles. Alternatively, syntax elements can be included in SEI, metadata, etc.

[0457] Referring to Table 4, `parts_enabled_flag` represents a syntax element indicating whether to divide the image into units. When this syntax element is activated (`parts_enabled_flag = 1`), the image can be divided into multiple units, and these units can be encoded or decoded. Additionally, supplementary division information can be checked. When this syntax element is deactivated (`parts_enabled_flag = 0`), the original image is encoded or decoded. In this example, the description will focus on rectangular division units such as tiles, and different settings can be provided for existing tiles and division information.

[0458] Here, num_partitions refers to a syntax element that indicates the number of partitions. num_partitions plus 1 equals the number of partitions.

[0459] Furthermore, `part_top[i]` and `part_left[i]` are syntax elements indicating the position information of the partitioning unit, representing the horizontal and vertical start positions of the partitioning unit (e.g., the top-left position of the partitioning unit). Additionally, `part_width[i]` and `part_height[i]` are syntax elements indicating the size information of the partitioning unit, representing the width and height of the partitioning unit. In this case, the start position and size information can be set at the pixel level or at the block level. Furthermore, the syntax elements can be syntax elements that can be generated during image reconstruction processing or syntax elements that can be generated when combining image partitioning and image reconstruction processing.

[0460] In addition, `part_header_enabled_flag` represents a syntax element indicating whether encoding / decoding settings for partition units are supported. When this syntax element is activated (`part_header_enabled_flag = 1`), encoding / decoding settings for partition units can be provided. When this syntax element is deactivated (`part_header_enabled_flag = 0`), encoding / decoding settings cannot be provided, but encoding / decoding settings for higher-level units can be assigned.

[0461] The above examples are not limited to examples of syntax elements associated with resizing and reconstruction within a partitioning unit in image settings, and can be modified for other partitioning units and settings of the present invention. This example has been described assuming that resizing and reconstruction are performed after partitioning, but the present invention is not limited thereto and can be modified with other image setting orders, etc. Furthermore, the types, orders, conditions, etc., of syntax elements supported in the following examples are limited to this example and can therefore be changed and determined according to encoding / decoding settings.

[0462] [Table 5]

[0463]

[0464] Table 5 shows example syntax elements associated with reconstruction in the partitioned units within the image settings.

[0465] Referring to Table 5, `part_convert_flag[i]` represents a syntax element indicating whether to reconstruct the partition unit. This syntax element can be generated for each partition unit. When this syntax element is activated (`part_convert_flag[i] = 1`), the reconstructed partition unit can be encoded or decoded, and additional reconstruction-related information can be checked. When this syntax element is deactivated (`part_convert_flag[i] = 0`), the original partition unit is encoded or decoded. Here, `convert_type_flag[i]` refers to the pattern information about the reconstruction of the partition unit, and can be information about pixel rearrangement.

[0466] Furthermore, syntactic elements indicating additional reconstructions, such as partition unit rearrangement, can be generated. In this example, partition unit rearrangement can be performed via part_top and part_left, which are syntactic elements indicating the aforementioned image partitioning, or syntactic elements associated with partition unit rearrangement (e.g., index information) can be generated.

[0467] [Table 6]

[0468]

[0469] Table 6 shows example syntax elements related to size adjustments in the partitioned units within the image settings.

[0470] Referring to Table 6, `part_resizing_flag[i]` represents a syntax element indicating whether to adjust the size of the partitioned units in the image. This syntax element can be generated for each partitioned unit. When this syntax element is activated (`part_resizing_flag[i] = 1`), the resized partitioned units can be encoded or decoded after resizing, and additional resizing-related information can be checked. When this syntax element is deactivated (`part_resizing_flag[i] = 0`), the original partitioned units are encoded or decoded.

[0471] In addition, width_scale[i] and height_scale[i] represent the scale factors associated with the horizontal and vertical size adjustments based on the scaling factor in the partitioned unit.

[0472] In addition, top_height_offset[i] and bottom_height_offset[i] represent the offset factors for the "up" direction and the offset factors for the "down" direction associated with the size adjustment based on the offset factor in the partitioning unit, and left_width_offset[i] and right_width_offset[i] represent the offset factors for the "left" direction and the offset factors for the "right" direction associated with the size adjustment based on the offset factor in the partitioning unit.

[0473] Furthermore, `resizing_type_flag[i][j]` represents a syntax element indicating the data processing method for the resized region within the partitioned unit. This syntax element indicates a separate data processing method for the resized direction. For example, syntax elements indicating separate data processing methods for the resized region in the "up," "down," "left," and "right" directions can be generated. This syntax element can be generated based on resized information (e.g., it can be generated only if resized in some directions).

[0474] The image settings processing described above can be applied based on the characteristics, type, etc., of the image. In the following examples, unless specifically mentioned, the image settings processing described above can be applied with or without any changes. The descriptions in the following examples will focus on the additions or changes made in the examples described above.

[0475] For example, 360-degree images or omnidirectional images generated by a 360-degree camera device have characteristics that are different from those of images acquired by a regular camera device, and have a coding environment that is different from the coding environment for compressing a general image.

[0476] Unlike regular images, 360-degree images may not have boundary regions with discontinuities, and the data in all regions of a 360-degree image can be continuous. Furthermore, devices such as HMDs may require high-resolution images because the images should be reproduced in front of the eye through lenses. The amount of image data processed can increase when images are acquired using stereoscopic camera devices. Various image setups considering 360-degree images can be performed to provide an efficient encoding environment including the examples described above.

[0477] A 360-degree camera device can be multiple camera devices or a camera device with multiple lenses and sensors. The camera devices or lenses can cover all directions around any center point captured by the camera devices.

[0478] Various methods can be used to encode 360-degree images. For example, various image processing algorithms in 3D space can be used to encode 360-degree images, and various image processing algorithms can be used to convert 360-degree images to 2D space and encode them. According to the present invention, the following description will focus on methods for converting 360-degree images to 2D space and encoding or decoding the converted images.

[0479] A 360-degree image encoding apparatus according to an embodiment of the present invention may include Figure 1 The device may include some or all of the elements shown, and may also include a preprocessing unit configured to preprocess the input image (stitching, projection, region packing). Meanwhile, the 360-degree image decoding apparatus according to embodiments of the present invention may include... Figure 2 The image may include some or all of the elements shown, and may also include a post-processing unit configured to post-process the encoded image to reproduce the output image before decoding the encoded image.

[0480] In other words, the encoder can preprocess the input image, encode the preprocessed image, and transmit a bitstream including the image, and the decoder can parse, decode, and post-process the transmitted bitstream to generate an output image. In this case, the transmitted bitstream may include information generated during the preprocessing process and information generated during the encoding process, and the bitstream can be parsed and used during the decoding and post-processing processes.

[0481] The operation method for the 360-degree image encoder will then be described in more detail, and those skilled in the art can easily derive the operation method for the 360-degree image decoder. Since the operation method for the 360-degree image decoder is the opposite of the operation method for the 360-degree image encoder, a detailed description of the operation method for the 360-degree image decoder will be omitted.

[0482] The input image can undergo stitching and projection processing on a sphere-based 3D projection structure, and the image data on the 3D projection structure can be projected into a 2D image through this processing.

[0483] The projected image can be configured to include some or all of the 360-degree content based on encoding settings. In this case, the position information of the region (or pixel) to be placed at the center of the projected image can be generated implicitly as a predetermined value, or the position information of the region (or pixel) can be generated explicitly. Furthermore, when the projected image includes a specific region of the 360-degree content, the extent and position information of the included region can be generated. Additionally, the extent information (e.g., width and height) and position information (e.g., measured based on the upper left of the image) of the region of interest (ROI) can be generated based on the projected image. In this case, a specific region of high importance within the 360-degree content can be set as the ROI. A 360-degree image can allow viewing of all content in the "up," "down," "left," and "right" directions, but the user's gaze may be limited to a portion of the image; this limitation can be taken into account when setting that portion of the image as the ROI. For efficient encoding purposes, the ROI can be set to have good quality and high resolution, while other regions can be set to have lower quality and lower resolution than the ROI.

[0484] Among various 360-degree image transmission schemes, a single-stream transmission scheme allows the transmission of a full image or viewport image in a single bitstream for the user. A multi-stream transmission scheme allows the transmission of several full images with different image qualities in multiple bitstreams, thus enabling selection of image quality based on user environment and communication conditions. A tile-based streaming scheme allows the transmission of individually encoded, tile-based partial images in multiple bitstreams, thus enabling selection of tiles based on user environment and communication conditions. Therefore, a 360-degree image encoder can generate and transmit bitstreams with two or more qualities, and a 360-degree image decoder can set the Region of Interest (ROI) based on the user's view and selectively decode the bitstream based on the ROI. That is, the area where the user is looking can be set as the ROI using a head-tracking or eye-tracking system, and only the required portion can be presented.

[0485] A projected image can be converted into a packed image obtained by performing region-wise packing. Region-wise packing can include the step of dividing the projected image into multiple regions, and the regions can be arranged (or rearranged) within the packed image according to the region-wise packing settings. When a 360-degree image is converted into a 2D image (or a projected image), region-wise packing can be performed to increase spatial continuity. Therefore, image size can be reduced through region-wise packing. Furthermore, region-wise packing can be performed to reduce image quality degradation during rendering, enable viewport-based projection, and provide other types of projection formats. Region-wise packing can be performed or not, depending on encoding settings, which can be determined based on signals indicating whether region-wise packing should be performed (e.g., `regionwise_packing_flag`; information about region-wise packing can only be generated when `regionwise_packing_flag` is activated).

[0486] When performing regional packaging, settings (or mapping information) can be displayed (or generated) to assign (or arrange) specific regions of the projected image to specific regions of the packaged image. When not performing regional packaging, the projected image and the packaged image can be the same image.

[0487] In the above description, stitching, projection, and regional packaging are defined as separate processes, but some (e.g., stitching + projection, projection + regional packaging) or all (e.g., stitching + projection + regional packaging) of the processes can be defined as a single process.

[0488] At least one packed image can be generated from the same input image based on the settings for stitching, projection, and region-based packing. Furthermore, at least one encoded data can be generated for the same projected image based on the settings for region-based packing.

[0489] Packed images can be divided by performing a tiling process. In this case, tiling, as a process of dividing an image into multiple regions and then transmitting it, can be an example of a 360-degree image transmission scheme. As mentioned above, tiling can be performed considering the user environment for partial decoding, and it can also be performed for efficiently processing large amounts of data in a 360-degree image. For example, when an image consists of a single unit, the entire image can be decoded to decode the ROI. On the other hand, when an image consists of multiple unit regions, decoding only the ROI may be efficient. In this case, the division can be performed as a unit of tiles, which are division units according to a conventional encoding scheme, or it can be performed according to various division units (e.g., quadrilateral division blocks, etc.) already described according to the present invention. Furthermore, the division unit can be a unit for performing independent encoding / decoding. Tiling can be performed independently or based on the projected image or the packed image. That is, division can be performed based on the surface boundaries of the projected image, the surface boundaries of the packed image, the packing settings, etc., and division can be performed independently for each division unit. This may affect the generation of division information during the tiling process.

[0490] Next, the projected or packed image can be encoded. Encoded data and information generated during preprocessing can be added to the bitstream, and the bitstream can be transmitted to the 360-degree image decoder. The information generated during preprocessing can be added to the bitstream in the form of SEI or metadata. In this case, the bitstream can contain: at least one piece of encoded data with settings partially different for the encoding process; and at least one piece of preprocessing information with settings partially different for the preprocessing process. This is to construct the decoded image by combining multiple pieces of encoded data (encoded data + preprocessing information) according to the user environment. Specifically, the decoded image can be constructed by selectively combining multiple pieces of encoded data. Furthermore, this processing can be performed in two parts for application to a stereo system, and the processing can be performed on an additional depth image.

[0491] Figure 15 These are example diagrams illustrating 2D planar space and 3D space illustrating a 3D image.

[0492] Typically, for the purpose of a 360-degree 3D virtual space, three degrees of freedom (3DoF) may be required, and three rotations relative to the X-axis (pitch), Y-axis (yaw), and Z-axis (roll) may be supported. DoF refers to spatial degrees of freedom, 3DoF refers to degrees of freedom including rotation about the X-axis, Y-axis, and Z-axis as shown in section 15a, and 6DoF refers to additionally allowing translation along the X-axis, Y-axis, and Z-axis, in addition to 3DoF degrees of freedom. The following description will focus on the image encoding and image decoding apparatus of the present invention having 3DoF. When supporting 3DoF or greater (3DoF+), the image encoding and image decoding apparatus may be modified or combined with additional processing or apparatus not shown.

[0493] Referring to section 15a, yaw can have a range from -π (-180 degrees) to π (180 degrees), pitch can have a range from -π / 2 radians (or -90 degrees) to π / 2 radians (or 90 degrees), and roll can have a range from -π / 2 radians (or -90 degrees) to π / 2 radians (or 90 degrees). In this case, when Φ and θ are assumed to be longitude and latitude in a map representation of the Earth, 3D spatial coordinates (x, y, z) can be transformed from 2D spatial coordinates (Φ, θ). For example, 3D spatial coordinates can be derived from 2D spatial coordinates according to the transformation formulas x = cos(θ)cos(Φ), y = sin(θ), and z = -cos(θ)sin(Φ).

[0494] Furthermore, (Φ, θ) can be transformed into (x, y, z). For example, this can be achieved using the transformation formula Φ = tanθ. -1 (-Z / X) and θ = sin -1 (Y / (X 2 +Y 2 +Z 2 ) 1 / 2 Derive 2D spatial coordinates from 3D spatial coordinates.

[0495] When pixels in 3D space are accurately transformed into pixels in 2D space (e.g., integer unit pixels in 2D space), pixels in 3D space can be mapped to pixels in 2D space. When pixels in 3D space are not accurately transformed into pixels in 2D space (e.g., fractional unit pixels in 2D space), pixels obtained through interpolation can be mapped to 2D pixels. In this case, nearest neighbor interpolation, bilinear interpolation, B-spline interpolation, bicubic interpolation, etc., can be used as interpolation. In this case, relevant information can be explicitly generated by selecting one of several interpolation candidates, or the interpolation method can be implicitly determined according to predetermined rules. For example, predetermined interpolation filters can be used based on the 3D model, projection format, color format, and tile / pattern type. Furthermore, when interpolation information is explicitly generated, information about the filters (e.g., filter coefficients) can be included.

[0496] Part 15b shows an image transformed from 3D space to 2D space (2D planar coordinate system). The image can be sampled based on its dimensions (width and height) as pairs (i, j). Here, i can range from 0 to P_Width-1, and j can range from 0 to P_Height-1.

[0497] (Φ, θ) can be the center point (or reference point) of an image arranged 360 degrees relative to the projected image; depicted as... Figure 15 The center point is a point C with coordinates (Φ, θ) = (0, 0). The center point can be set in 3D space, and its position information can be generated explicitly or implicitly predetermined. For example, center position information can be generated for yaw, pitch, roll, etc. When no value is specified individually, each value can be assumed to be zero.

[0498] The above has described an example of transforming an entire 360-degree image from 3D space to 2D space. However, specific regions of a 360-degree image can be transformed, and positional information (e.g., some locations belonging to that region; in this example, positional information about the center point), extent information, etc., of a specific region can be explicitly generated, or the positional and extent information of a specific region can implicitly follow predetermined positional and extent information. For example, center position information in yaw, center position information in pitch, center position information in roll, extent information in yaw, extent information in pitch, extent information in roll, etc., can be generated, and the specific region can be at least one region. Therefore, positional and extent information of multiple regions can be processed. When the values ​​of information are not specified individually, the entire 360-degree image can be assumed.

[0499] H0 to H6 and W0 to W5 in part 15a indicate some latitude and longitude in part 15b, which can be represented as coordinates (C, j) and (i, C) in part 15b (where C is the longitude or latitude component). Unlike general images, when a 360-degree image is converted to 2D space, distortion or warping of the content in the image may occur. This may depend on the region of the image, and different encoding / decoding settings can be applied to the location of the image or the region divided according to that location. When encoding / decoding settings are adaptively applied based on encoding / decoding information in this invention, location information (e.g., x-component, y-component, or a range defined by x and y) may be included as an example of encoding / decoding information.

[0500] The definitions of 3D and 2D spaces are provided to aid in describing embodiments of the invention. However, the invention is not limited thereto, and the above description may be modified in detail or applied to other situations.

[0501] As described above, images acquired by a 360-degree camera device can be transformed into 2D space. In this case, a 3D model can be used to map the 360-degree image, and various 3D models such as spheres, cubes, cylinders, pyramids, and polyhedra can be used. When the model-mapped 360-degree image is transformed into 2D space, projection processing can be performed according to the model-based projection format.

[0502] Figures 16a to 16d This is a conceptual diagram illustrating a projection format according to an embodiment of the present invention.

[0503] Figure 16a This illustrates the equal rectangular projection (ERP) format in which a 360-degree image is projected onto a 2D plane. Figure 16b The cube map projection (CMP) format, in which a 360-degree image is projected onto a cube, is shown. Figure 16c This illustrates the octahedral projection (OHP) format, which projects a 360-degree image onto an octahedron. Figure 16d This illustrates a 360-degree image projected onto an icosahedral projection (ISP) format. However, the invention is not limited to this, and various projection formats can be used. Figures 16a to 16d The left side shows the 3D model, and the right side shows an example transformed into 2D space through projection processing. Various sizes and shapes can be provided depending on the projection format. Each shape can consist of surfaces or faces, and each face can be represented as a circle, triangle, quadrilateral, etc.

[0504] In this invention, projection formats can be defined by 3D models, face settings (e.g., the number of faces, face shapes, face shape configurations, etc.), projection processing settings, etc. Projection formats can be considered different projection formats when at least one element differs in definition. For example, ERP consists of a sphere model (3D model), one face (number of faces), and quadrilateral faces (face shapes). However, formats can be classified as different formats, such as ERP1 and ERP2, when some settings in the projection processing settings (e.g., the formula used during the transformation from 3D space to 2D space; i.e., elements with the same residual projection settings that produce differences in at least one pixel of the projected image during projection processing) differ. As another example, CMP consists of a cube model, six faces, and quadrilateral faces. Formats can be classified as different formats, such as CMP1 and CMP2, when some settings during the projection processing (e.g., the sampling method applied during the transformation from 3D space to 2D space) differ.

[0505] When multiple projection formats are used in place of a predetermined projection format, projection format identification information (or projection format information) can be explicitly generated. The projection format identification information can be configured using various methods.

[0506] As an example, a projection format can be identified by assigning index information (e.g., proj_format_flag) to multiple projection formats. For instance, #0 can be assigned to ERP, #1 to CMP, #2 to OHP, #3 to ISP, #4 to ERP1, #5 to CMP1, #6 to OHP1, #7 to ISP1, #8 to CMP compact, #9 to OHP compact, #10 to ISP compact, and #11 or higher can be assigned to other formats.

[0507] As an example, a projection format can be identified using at least one element of information constituting the projection format. In this case, the element of information constituting the projection format may include 3D model information (e.g., 3d_model_flag; #0 indicates a sphere, #1 indicates a cube, #2 indicates a cylinder, #3 indicates a pyramid, #4 indicates polyhedron 1, and #5 indicates polyhedron 2), face number information (e.g., num_face_flag; incrementing by 1 starting from 1; the number of faces generated in the projection format is specified as index information, i.e., #0 indicates one, #1 indicates three, #2 indicates six, #3 indicates eight, and #4 indicates twenty), face shape information (e.g., shape_face_flag; #0 indicates a quadrilateral, #1 indicates a circle, #2 indicates a triangle, #3 indicates a quadrilateral + circle, and #4 indicates a quadrilateral + triangle), projection processing settings information (e.g., 3d_2d_convert_idx), etc.

[0508] As an example, a projection format can be identified using element information constituting the projection format and projection format index information. For instance, as projection format index information, #0 can be assigned to ERP, #1 to CMP, #2 to OHP, #3 to ISP, and #4 or higher can be assigned to other formats. Projection formats (e.g., ERP, ERP1, CMP, CMP1, OHP, OHP1, ISP, and ISP1) can be identified together with element information constituting the projection format (here, projection processing settings information). Alternatively, projection formats (e.g., ERP, CMP, CMP compact, OHP, OHP compact, ISP, and ISP compact) can be identified together with element information constituting the projection format (here, regional packaging).

[0509] In summary, a projection format can be identified using projection format index information, at least one projection format element information, or both projection format index information and at least one projection format element information. This can be defined according to encoding / decoding settings. In this invention, the following description assumes that a projection format is identified using a projection format index. In this example, the description will focus on projection formats expressed using faces with the same size and shape, but configurations with faces of different sizes and shapes are possible. Furthermore, the configuration of each face can be... Figures 16a to 16d The configurations shown may be the same or different, the number of each face is used as a symbol to identify the corresponding face, and there is no restriction on a particular order. For ease of description, the following description assumes that for projected images, ERP is a projection format including one face + quadrilateral, CMP is a projection format including six faces + quadrilateral, OHP is a projection format including eight faces + triangle, ISP is a projection format including twenty faces + triangle, and that the faces have the same size and shape. However, this description can be applied in the same or similar way even for different settings.

[0510] like Figures 16a to 16d As shown, projection formats can be categorized into one face (e.g., ERP) or multiple faces (e.g., CMP, OHP, and ISP). Furthermore, the shape of each face can be categorized as quadrilateral, triangle, etc. The categorization can be an example of the type, characteristics, etc., of the image according to the invention, which can be applied when different encoding / decoding settings are provided according to the projection format. For example, the image type can be a 360-degree image, and the image characteristics can be one of the categories (e.g., each projection format, projection formats with one or more faces, projection formats with quadrilateral or non-quadrilateral faces).

[0511] A 2D planar coordinate system (e.g., (I, j)) can be defined in each face of a 2D projected image, and the characteristics of the coordinate system can vary depending on the projection format, the position of each face, etc. An ERP can have one 2D planar coordinate system, and other projection formats can have multiple 2D planar coordinate systems depending on the number of faces. In this case, the coordinate system can be represented as (k, i, j), where k can indicate the index information of each face.

[0512] Figure 17 This is a conceptual diagram illustrating a projection format included in a rectangular image according to an embodiment of the present invention.

[0513] That is, it is understandable that portions 17a to 17c show Figures 16b to 16d The projection format is implemented as a rectangular image.

[0514] Referring to sections 17a to 17c, each image format can be configured in a rectangular shape to encode or decode 360-degree images. For ERP, a single coordinate system can be used as is. However, for other projection formats, the coordinate systems of the surfaces can be integrated into a single coordinate system, and its detailed description will be omitted.

[0515] Referring to sections 17a to 17c, when constructing a rectangular image, it can be confirmed that regions filled with meaningless data such as blanks or background are generated. That is, a rectangular image can consist of regions including actual data (here, faces; valid regions) and meaningless regions added to construct the rectangular image (here, assuming these regions are filled with any pixel values; invalid regions). This can degrade performance due to the increase in encoded data, i.e., the increase in image size caused by the encoding / decoding of meaningless regions and actual image data.

[0516] Therefore, additional processing can be performed to construct the image by excluding meaningless regions and using regions that include actual data.

[0517] Figure 18 This is a conceptual diagram of a method for converting a projection format into a rectangular shape according to an embodiment of the present invention, namely, a method for performing rearrangement on a surface to exclude meaningless regions.

[0518] Referring to sections 18a to 18c, examples for rearranging sections 17a to 17c can be identified, and this process can be defined as a region-based packing process (CMP compact, OHP compact, ISP compact, etc.). In this case, faces can not only be rearranged, but also divided and then rearranged (OHP compact, ISP compact, etc.). This can be performed to remove meaningless regions and improve coding performance through efficient face arrangement. For example, when images are arranged consecutively between faces (e.g., B2-B3-B1, B5-B0-B4, etc. in section 18a), prediction accuracy during coding is enhanced, thus improving coding performance. Here, region-based packing according to the projection format is merely an example, and the invention is not limited thereto.

[0519] Figure 19 This is a conceptual diagram illustrating the execution of region packing processing according to an embodiment of the present invention to convert a CMP projection format into a rectangular image.

[0520] Referring to sections 19a to 19c, the CMP projection format can be arranged as 6×1, 3×2, 2×3, and 1×6. Furthermore, when adjusting the size of some faces, the arrangement can be as shown in sections 19d and 19e. In sections 19a to 19e, CMP is used as an example. However, the invention is not limited to this, and other projection formats can be applied. The arrangement of faces in an image acquired through region packing can follow predetermined rules corresponding to the projection format, or information about the arrangement can be explicitly generated.

[0521] The 360-degree image encoding and decoding apparatus according to an embodiment of the present invention can be configured to include Figure 1 and Figure 2 Some or all of the elements of the image encoding and decoding apparatus shown are included. In particular, a format conversion unit configured to transform the projection format and an inverse format conversion unit configured to perform an inverse transformation of the projection format may also be included in the image encoding apparatus and the image decoding apparatus, respectively. That is, the input image can be processed by the format conversion unit and then by... Figure 1 The image encoding device encodes the bitstream, and the bitstream can be encoded by the image encoding device. Figure 2 The image decoding device decodes the image and then processes it through an inverse format transformation unit to generate an output image. The following description will focus on the processing performed by the encoder (here, the input image, encoding, etc.), and the processing performed by the decoder can be derived from the encoder in reverse. Furthermore, redundant descriptions of the foregoing will be omitted.

[0522] The following description assumes that the input image is the same as a packaged image or a 2D projected image obtained by performing a preprocessing procedure using a 360-degree encoding device. That is, the input image can be an image obtained by performing projection processing or regional packaging processing according to some projection format. The projection format pre-applied to the input image can be one of various projection formats, which can be considered a common format and referred to as the first format.

[0523] The format conversion unit can perform conversions to projection formats other than the first format. In this case, the projection format to which the conversion is to be performed can be referred to as the second format. For example, ERP can be set as the first format, and ERP can be converted to the second format (e.g., ERP2, CMP, OHP, and ISP). In this case, ERP2 includes EPR formats that have the same conditions such as 3D models and face configurations, but with some different settings. Alternatively, the projection format can be the same format with the same projection format settings (e.g., ERP = ERP2), and can have different image sizes or resolutions. Alternatively, some of the following image setting processes can be applied. For ease of description, such examples have been mentioned, but each of the first and second formats can be one of various projection formats. However, the invention is not limited thereto and can be modified thereto.

[0524] During format transformation processing, due to the different coordinate system characteristics, the pixels of the transformed image (integer pixels) can be obtained from both fractional and integer pixel units in the image before transformation, thus enabling interpolation. The interpolation filter used in this case can be the same as or similar to the interpolation filter described above. In this case, relevant information can be explicitly generated by selecting one of several interpolation filter candidates, or the interpolation filter can be implicitly determined according to predetermined rules. For example, a predetermined interpolation filter can be used based on the projection format, color format, and tile / pattern type. Furthermore, when the interpolation filter is explicitly provided, information about the filter (e.g., filter coefficients) can be included.

[0525] In the format conversion section, the projection format can be defined to include regional packing, etc. That is, projection and regional packing can be performed during format conversion processing. Alternatively, processing such as regional packing can be performed after format conversion processing and before encoding.

[0526] The encoder can add information generated during the above processing to the bitstream in units of at least one of sequences, images, slices, tiles, etc., and the decoder can parse the relevant information from the bitstream. Furthermore, the information can be included in the bitstream in the form of SEI or metadata.

[0527] Next, image setting processing applied to a 360-degree image encoding / decoding apparatus according to an embodiment of the present invention will be described. The image setting processing according to the present invention can be applied to the preprocessing process, postprocessing process, format conversion processing, inverse format conversion processing, and general encoding / decoding processing of a 360-degree image encoding / decoding apparatus. The following description of image setting processing will focus on the 360-degree image encoding apparatus and may include the aforementioned image settings. Redundant descriptions of the aforementioned image setting processing will be omitted. Furthermore, the following examples will focus on image setting processing, and inverse image setting processing can be derived from the image setting processing. Several points can be confirmed through the various embodiments of the present invention described above.

[0528] The image setting process according to the invention can be performed in the 360-degree image projection step, the regional packaging step, the format conversion step, or other steps.

[0529] Figure 20 This is a conceptual diagram of 360-degree image segmentation according to an embodiment of the present invention. Figure 20 In this context, we assume that the image is projected through the ERP.

[0530] Section 20a illustrates an image projected via ERP, and various methods can be used to divide the image. In this example, the description focuses on slices or tiles, and it is assumed that W0 to W2 and H0 and H1 are the dividing boundaries of the slices or tiles and follow the raster scan order. The following examples focus on slices and tiles. However, the invention is not limited thereto, and other dividing methods can be applied.

[0531] For example, partitioning can be performed on a slice basis, and H0 and H1 can be set as partition boundaries. Alternatively, partitioning can be performed on a block basis, and W0 to W2, H0, and H1 can be set as partition boundaries.

[0532] Section 20b illustrates an example of an image projected via ERP being divided into tiles (assuming the same tile division boundaries as shown in Section 20a (W0 to W2, H0, and H1 are all activated)). When assuming region P is the entire image and region V is the area where the user's gaze rests, or the viewport, various methods exist to provide an image corresponding to the viewport. For example, the region corresponding to the viewport can be obtained by decoding the entire image (e.g., tiles a to i). In this case, the entire image can be decoded, and in the case of a divided image, tiles a to i (here, region A + region B) can be decoded. Alternatively, the region corresponding to the viewport can be obtained by decoding the region belonging to the viewport. In this case, in the case of a divided image, the region corresponding to the viewport can be obtained from the image recovered by decoding tiles f, g, j, and k (here, region B). The former case can be referred to as full decoding (or viewport-independent encoding), while the latter case can be referred to as partial decoding (or viewport-dependent encoding). The latter case can be an example that may occur in a 360-degree image with a large amount of data. Compared to slice-based partitioning methods, tile-based partitioning methods can be used more frequently because they offer greater flexibility in acquiring partitioned regions. For partial decoding, the referability of partitioning units can be constrained (implicitly handled here) spatially or temporally, since it's impossible to determine where the viewpoint will occur, and this constraint can be taken into account when performing encoding / decoding. The following examples will focus on full decoding, but will concentrate on describing 360-degree image partitioning using tiles (or the rectangular partitioning method of this invention) in preparation for partial decoding. However, the following description can be applied to other partitioning units in the same or modified manner.

[0533] Figure 21 This is an example diagram of 360-degree image segmentation and image reconstruction according to an embodiment of the present invention. Figure 21 In this context, we assume that the image is projected via CMP.

[0534] Section 21a shows an image projected by CMP, and various methods can be used to divide the image. It is assumed that W0 to W2, H0, and H1 are the dividing boundaries of faces, slices, and tiles and follow the raster scan order.

[0535] For example, partitioning can be performed on slices, with H0 and H1 set as partition boundaries. Alternatively, partitioning can be performed on tiles, with W0 to W2, H0, and H1 set as partition boundaries. Alternatively, partitioning can be performed on faces, with W0 to W2, H0, and H1 set as partition boundaries. In this example, it is assumed that the face is part of the partitioning unit.

[0536] In this context, a face can be a partitioning unit (here, a plane coordinate system for each face) that has different properties in the same image, performed to classify or distinguish them based on the image's characteristics, type (in this example, a 360-degree image and a projection format), etc. (depending on encoding / decoding). A slice or tile can be a partitioning unit that is performed to divide the image according to user definitions (here, independent encoding / decoding). Furthermore, a face can be a unit partitioned during projection processing according to a projection format by a predetermined definition (or induced by projection format information), while a slice or tile can be a unit partitioned by explicitly generating partitioning information according to user definitions. Additionally, faces can have polygonal partitioning shapes, including quadrilaterals, depending on the projection format; slices can have any partitioning shape that cannot be defined as quadrilaterals or polygons; and tiles can have quadrilateral partitioning shapes. The settings for the partitioning units can be defined only for the description of this example.

[0537] In this example, a face has been described as a partitioning unit for classifying regions. However, a face can also be a unit used to perform independent encoding / decoding based on encoding / decoding settings as at least one face unit, and can have settings for performing independent encoding / decoding in combination with tiles, slices, etc. In this case, when a face is combined with tiles, slices, etc., explicit information about the tiles and slices can be generated, or tiles and slices can be implicitly combined based on face information. Alternatively, explicit information about tiles and slices can be generated based on face information.

[0538] As a first example, an image partitioning process (here, a face) is performed, and the partitioning information (obtained from the projection format information) can be implicitly omitted. This example is for cases that depend on encoding / decoding settings and can correspond to situations where the referability between face elements is unrestricted.

[0539] As a second example, an image partitioning process (here, a face) is performed, and the image partitioning can explicitly generate partitioning information. This example is for cases that depend on encoding / decoding settings and can correspond to situations where the referability between face cells is unrestricted.

[0540] As a third example, multiple image partitioning processes (here, faces and tiles) are performed. Some image partitions (here, faces) may implicitly omit or explicitly generate partitioning information, while other image partitions (here, tiles) may explicitly generate partitioning information. In this example, one image partitioning process (here, faces) precedes the other image partitioning processes (here, tiles).

[0541] As a fourth example, multiple image partitioning processes are performed. Some image partitions (here, faces) can implicitly omit or explicitly generate partitioning information, while other image partitions (here, tiles) can explicitly generate partitioning information based on these image partitions (here, faces). In this example, one image partitioning process (here, faces) precedes the other image partitioning processes (here, tiles). In some cases of this example (assuming the second example), it can be the same as explicitly generating partitioning information, but the configuration of the partitioning information may differ.

[0542] As a fifth example, multiple image partitioning processes are performed. Some image partitions (here, faces) can implicitly omit partitioning information, and other image partitions (here, tiles) can implicitly omit partitioning information based on these image partitions (here, faces). For example, face units can be set as tile units individually, or multiple face units (here, face units are grouped when adjacent faces are continuous; otherwise, face units are not grouped; B2-B3-B1 and B4-B0-B5 in part 18a) can be set as tile units. Face units can be set as tile units according to predetermined rules. This example is for independent encoding / decoding settings and can be an example corresponding to cases where the referability between face units is limited. That is, in some cases (assuming the first example), the partitioning information can be processed in the same way as implicitly, but the encoding / decoding settings may differ.

[0543] This example can be a description of a scenario where partitioning can be performed in a projection step, a region packing step, an initial encoding / decoding step, etc., and can be any other image partitioning process performed in the encoder / decoder.

[0544] In part 21a, a rectangular image can be constructed by adding regions (B) that do not contain data to regions (A) that include data. In this case, the position, size, shape, number, etc., of regions A and B can be information that can be checked through the projection format or when explicitly generating information about the projected image, and the relevant information can be represented by the aforementioned image segmentation information, image reconstruction information, etc. For example, information about a specific region of the projected image (e.g., part_top, part_left, part_width, part_height, and part_convert_flag) can be represented as shown in Tables 4 and 5. However, the invention is not limited to this and can be applied to other situations (e.g., other projection formats, other projection settings, etc.).

[0545] Regions B and A can be constructed as a single image and then encoded or decoded. Alternatively, region-specific characteristics can be considered when performing partitioning, and different encoding / decoding settings can be applied. For example, region B can be encoded or decoded without using information about whether encoding or decoding is performed (e.g., the `tile_coded_flag` when the partitioning unit is assumed to be a tile). In this case, the corresponding region can be restored to some data (here, any pixel value) according to predetermined rules. Alternatively, in the image partitioning process described above, region B can have different encoding / decoding settings than region A. Alternatively, the corresponding region can be removed by performing a region-specific packing process.

[0546] Section 21b illustrates an example of an image packaged via CMP being divided into tiles, slices, or faces. In this case, the packaged image is an image that has undergone face rearrangement or region-based packaging processing, and can be an image obtained by performing image partitioning and image reconstruction according to the invention.

[0547] In part 21b, the rectangular shape can be constructed to include regions containing data. In this case, the location, size, shape, number, etc., of the regions can be information that can be checked through predetermined settings or information that can be checked when explicitly generating information about the packaged image, and the relevant information can be represented by the aforementioned image segmentation information, image reconstruction information, etc. For example, information about a specific region of the packaged image (e.g., part_top, part_left, part_width, part_height, and part_convert_flag) can be represented as shown in Tables 4 and 5.

[0548] Various partitioning methods can be used to partition a packaged image. For example, partitioning can be performed on a slice basis, with H0 set as the partition boundary. Alternatively, partitioning can be performed on a tile basis, with W0, W1, and H0 set as partition boundaries. Alternatively, partitioning can be performed on a polygon basis, with W0, W1, and H0 set as partition boundaries.

[0549] Image partitioning and image reconstruction processing according to the present invention can be performed on the projected image. In this case, the reconstruction processing can be used to rearrange the faces and pixels in the image. This can be a possible example when the image is divided into multiple faces or is composed of multiple faces. The following examples will be described focusing on the case of dividing the image into patches based on face units.

[0550] In section 21a, SX and Y (S0,0 to S3,2) can correspond to S'U and V (S'0,0 to S'2,1) in section 21b (here, X and Y can be the same as or different from U and V), and reconstruction processing can be performed on a face-by-face basis. For example, S2,1, S3,1, S0,1, S1,2, S1,1, and S1,0 can be assigned to S'0,0, S'1,0, S'2,0, S'0,1, S'1,1, and S'2,1 (face rearrangement). Furthermore, S2,1, S3,1, and S0,1 may not be reconstructed (pixel rearrangement), and S1,2, S1,1, and S1,0 can be rotated 90 degrees and then reconstructed. This can be represented as shown in section 21c. In section 21c, the horizontally placed symbols (S1,0, S1,1, S1,2) can be images that are horizontally placed to maintain the continuity of the image.

[0551] Surface reconstruction can be handled implicitly or explicitly based on encoding / decoding settings. Implicit processing can be performed according to predetermined rules, taking into account the image type (here, a 360-degree image) and characteristics (here, projection format, etc.).

[0552] For example, for S'0,0 and S'1,0, S'1,0 and S'2,0, S'0,1 and S'1,1, S'1,1 and S'2,1 in part 21c, there exists image continuity (or correlation) between two faces relative to the face boundaries, and part 21c can be an example of continuity between three upper faces and three lower faces. In cases where an image is divided into multiple faces by projection processing from 3D space to 2D space and then the image is packaged for each region, reconstruction can be performed to increase image continuity between faces in order to efficiently reconstruct the faces. Such face reconstructions can be predetermined and processed.

[0553] Alternative sites can perform refactoring through explicit processing and generate refactoring information.

[0554] For example, when examining information about an M×N build (e.g., 6×1, 3×2, 2×3, 1×6, etc. for CMP compaction; in this example, a 3×2 configuration is assumed) via region-based packing processing (e.g., one of implicitly acquired and explicitly generated information), face reconstruction can be performed based on the M×N build, and information about the face reconstruction can then be generated. For example, when rearranging faces in an image, index information (or information about their position in the image) can be assigned to each face. When rearranging pixels within a face, pattern information for reconstruction can be assigned.

[0555] Index information can be like Figure 18 As shown in sections 18a to 18c, they are predefined. In sections 21a to 21c, SX,Y or S'U,V represent each face using positional information indicating width and height (e.g., S[i][j]) or using a single positional information (e.g., S[i]; assuming the positional information is assigned in raster scan order starting from the top left of the image), and each face can be assigned an index.

[0556] For example, when indexing is assigned using positional information indicating width and height, face index #2 can be assigned to S'0,0, face index #3 can be assigned to S'1,0, face index #1 can be assigned to S'2,0, face index #5 can be assigned to S'0,1, face index #0 can be assigned to S'1,1, and face index #4 can be assigned to S'2,1, as shown in section 21c. Alternatively, when indexing is assigned using a single positional information, face index #2 can be assigned to S[0], face index #3 can be assigned to S[1], face index #1 can be assigned to S[2], face index #5 can be assigned to S[3], face index #0 can be assigned to S[4], and face index #4 can be assigned to S[5]. For ease of description, in the following examples, S'0,0 to S'2,1 can be referred to as a to f. Alternatively, each face can be represented based on positional information indicating the width and height of a pixel or block unit at the top left corner of the image.

[0557] For packaged images obtained through image reconstruction processing (or region-based packaging processing), the face scanning order may be the same as or different from the image scanning order, depending on the reconstruction settings. For example, when a scanning order (e.g., raster scanning) is applied to the image shown in part 21a, a, b, and c may have the same scanning order, while d, e, and f may have different scanning orders. For instance, when the scanning order of part 21a or the scanning order of a, b, and c follows (0,0)->(1,0)->(0,1)->(1,1), the scanning order of d, e, and f may follow (1,0)->(1,1)->(0,0)->(0,1). This can be determined based on the image reconstruction settings, and such settings can even be applied to other projection formats.

[0558] In the image partitioning process shown in section 21b, tiles can be individually set as surface units. For example, each of surfaces a through f can be set as a tile unit. Alternatively, multiple surface units can be set as tiles. For example, surfaces a through c can be set as one tile, and surfaces d through f can be set as one tile. The configuration can be determined based on surface characteristics (e.g., continuity between surfaces, etc.), and unlike the example above, different tile settings for surfaces can be possible.

[0559] The following is an example of segmentation information processed based on multiple image partitioning. In this example, it is assumed that the partitioning information of faces is omitted, the units other than faces are tiles, and the partitioning information is processed differently.

[0560] As a first example, image segmentation information can be obtained based on surface information, and this information can be implicitly omitted. For example, a surface can be set as a tile individually, or multiple surfaces can be set as tiles. In this case, when at least one surface is set as a tile, this can be determined based on surface information (e.g., continuity or correlation) according to predetermined rules.

[0561] As a second example, image segmentation information can be explicitly generated independent of face information. For instance, when generating segmentation information using the number of columns (here, num_tile_columns) and rows (here, num_tile_rows) of tiles, the segmentation information can be generated in the image segmentation processing method described above. For example, the number of columns of a tile can range from 0 to the width of the image or the width of the block (here, the unit obtained from the image segmentation section), and the number of rows of a tile can range from 0 to the height of the image or the height of the block. Furthermore, additional segmentation information (e.g., uniform_spacing_flag) can be generated. In this case, depending on the segmentation settings, the boundaries of the faces and the boundaries of the segmentation units can match or not match each other.

[0562] As a third example, image partitioning information can be explicitly generated based on face information. For instance, when generating partitioning information using the number of columns and rows of a tile, it can be based on face information (here, the number of columns ranges from 0 to 2, and the number of rows ranges from 0 to 1; because the face configuration in the image is 3x2). For example, the number of columns of a tile can be in the range of 0 to 2, and the number of rows of a tile can be in the range of 0 to 1. Furthermore, additional partitioning information (e.g., uniform_spacing_flag) may not be generated. In this case, the boundaries of the faces and the boundaries of the partitioning units can match each other.

[0563] In some cases (such as the second and third examples), the syntax elements for partitioning information can be defined differently, or even when using the same syntax elements, different syntax element settings can be applied (e.g., binarization settings; other binarizations can be used when the range of candidate groups for the syntax elements is limited and small). The examples described above have been described for some of the various elements of partitioning information. However, the invention is not limited thereto, and it is understood that other settings are possible depending on whether the partitioning information is generated based on surface information.

[0564] Figure 22 This is an example image of an image that has been packaged or projected using CMP and divided into tiles.

[0565] In this case, assuming it has the same Figure 21 The tile boundary shown in part 21a is the same as the tile boundary (W0 to W2, H0 and H1 are all activated) and has the same Figure 21 The tile partitioning boundaries shown in section 21b are the same (W0, W1, and H0 are all activated). When assuming region P represents the entire image and region V represents the viewport, full decoding or partial decoding can be performed. This example will focus on partial decoding. In section 22a, tiles e, f, and g can be decoded against the CMP (left side), and tiles a, c, and e can be decoded against the CMP compact (right side) to obtain the region corresponding to the viewport. In section 22b, tiles b, f, and i can be decoded against the CMP, and tiles d, e, and f can be decoded against the CMP compact to obtain the region corresponding to the viewport.

[0566] The above examples have already described the cases of partitioning based on surface elements (or surface boundaries) such as slicing and patching. However, as... Figure 20 As shown in part 20a, partitioning can be performed inside a surface (e.g., an image consists of one surface in ERP and multiple surfaces in other projection formats), or partitioning can be performed on the boundaries of a surface and inside the surface.

[0567] Figure 23 This is a conceptual diagram illustrating an example of adjusting the size of a 360-degree image according to an embodiment of the present invention. In this case, it is assumed that the image is projected via ERP. Furthermore, the following examples will focus on an extended case.

[0568] The size of the projected image can be adjusted using either a scaling factor or an offset factor, depending on the image size adjustment type. Here, the image before size adjustment can be P_Width × P_Height, and the image after size adjustment can be P'_Width × P'_Height.

[0569] For the scaling factor, after adjusting the width and height of the image using the scaling factor (here, a for width and b for height), the image width (P_Width × a) and height (P_Height × b) can be obtained. For the offset factor, after adjusting the width and height of the image using the offset factor (here, L and R for width and T and B for height), the image width (P_Width + L + R) and height (P_Height + T + B) can be obtained. Resizing can be performed using a predefined method, or by using a method selected from several options.

[0570] The data processing methods in the following examples will focus on the case of offset factors. For offset factors, data processing methods may include filling methods using predetermined pixel values, filling methods by copying external pixels, filling methods by copying a specific region of the image, and filling methods by transforming a specific region of the image.

[0571] The size of a 360-degree image can be adjusted by considering the continuity at the image boundaries. For ERPs, there are no outer boundaries in 3D space, but outer boundaries may exist when the 3D space is transformed to 2D space through projection processing. Data in the boundary regions includes data with outward continuity, but may have boundaries in terms of spatial properties. These properties can be considered when performing size adjustments. In this case, continuity can be checked according to the projection format, etc. For example, an ERP image can be an image with the property that both ends of the boundary are continuous. This example will be described under the assumption that the left and right boundaries of the image are continuous with each other, and the top and bottom boundaries of the image are continuous with each other. The data processing methods will focus on filling methods by copying specific regions of the image and filling methods by transforming specific regions of the image.

[0572] When resizing the image to the left, the resized region (LC or TL+LC+BL) can be filled with data from the right region of the image (tr+rc+br), which is continuous with the left side of the image. When resizing the image to the right, the resized region (RC or TR+RC+BR) can be filled with data from the left region of the image (tl+lc+bl), which is continuous with the right side of the image. When resizing the image upwards, the resized region (TC or TL+TC+TR) can be filled with data from the lower region of the image (bl+bc+br), which is continuous with the upper side of the image. When resizing the image downwards, the resized region (BC or BL+BC+BR) can be filled with data.

[0573] When the size or length of the resized region is m, the coordinates of the resized region relative to the image before resizing (here, x is in the range from 0 to P_Width-1) can have a range from (-m, y) to (-1, y) (resizing to the left) or from (P_Width, y) to (P_Width+m-1, y) (resizing to the right). The position (x') of the region used to obtain the data of the resized region can be derived from the formula x' = (x+P_Width)%P_Width. In this case, x represents the coordinates of the resized region relative to the image before resizing, and x' represents the coordinates of the region referencing the resized region relative to the image before resizing. For example, when the image size is adjusted to the left, m is 4, and the image width is 16, the corresponding data (-4, y) can be obtained from (12, y), the corresponding data (-3, y) can be obtained from (13, y), the corresponding data (-2, y) can be obtained from (14, y), and the corresponding data (-1, y) can be obtained from (15, y). Alternatively, when the image size is adjusted to the right, m ​​is 4, and the image width is 16, the corresponding data (16, y) can be obtained from (0, y), the corresponding data (17, y) can be obtained from (1, y), the corresponding data (18, y) can be obtained from (2, y), and the corresponding data (19, y) can be obtained from (3, y).

[0574] When the size or length of the resized region is n, the coordinates of the resized region relative to the image before resizing (here, y is in the range from 0 to P_Height-1) can have a range from (x, -n) to (x, -1) (resize up) or from (x, P_Height) to (x, P_Height+n-1) (resize down). The position (y') of the region used to obtain the data of the resized region can be derived from the formula y' = (y + P_Height) % P_Height. In this case, y represents the coordinates of the resized region relative to the image before resizing, and y' represents the coordinates of the region referencing the resized region relative to the image before resizing. For example, when the image size is adjusted upwards, n is 4, and the image height is 16, the corresponding data (x, -4) can be obtained from (x, 12), the corresponding data (x, -3) can be obtained from (x, 13), the corresponding data (x, -2) can be obtained from (x, 14), and the corresponding data (x, -1) can be obtained from (x, 15). Alternatively, when the image size is adjusted downwards, n is 4, and the image height is 16, the corresponding data (x, 16) can be obtained from (x, 0), the corresponding data (x, 17) can be obtained from (x, 1), the corresponding data (x, 18) can be obtained from (x, 2), and the corresponding data (x, 19) can be obtained from (x, 3).

[0575] After filling the resized region with data, you can perform resizing relative to the coordinates of the resized image (here, x is in the range from 0 to P'_Width-1, and y is in the range from 0 to P'_Height-1). This example can be applied to both latitude and longitude coordinate systems.

[0576] Various size adjustment combinations are available as follows.

[0577] As an example, the image size can be adjusted to the left by m. Alternatively, the image size can be adjusted to the right by n. Alternatively, the image size can be adjusted upwards by o. Alternatively, the image size can be adjusted downwards by p.

[0578] As an example, the image size can be adjusted to the left by m and to the right by n. Alternatively, the image size can be adjusted upwards by o and downwards by p.

[0579] As an example, the image size can be adjusted to the left by m, to the right by n, and upwards by o. Alternatively, the image size can be adjusted to the left by m, to the right by n, and downwards by p. Alternatively, the image size can be adjusted to the left by m, upwards by o, and downwards by p. Alternatively, the image size can be adjusted to the right by n, upwards by o, and downwards by p.

[0580] As an example, the image size can be adjusted to the left by m, to the right by n, up by o, and down by p.

[0581] Similar to the examples above, at least one resizing operation can be performed. Image resizing can be performed implicitly based on encoding / decoding settings, or resizing information can be generated implicitly, and then image resizing can be performed based on the generated resizing information. That is, m, n, o, and p in the examples above can be determined as predetermined values, or they can be explicitly generated using the resizing information. Alternatively, some of m, n, o, and p can be determined as predetermined values, and other values ​​can be explicitly generated.

[0582] The above example has focused on acquiring data from specific regions of an image, but other methods can also be applied. The data can be pixels before or after encoding, and can be determined based on the resizing step or the characteristics of the image to be resized. For example, when resizing is performed in preprocessing and pre-encoding steps, the data can refer to input pixels of the projected image, packed image, etc., and when resizing is performed in post-processing, intra-frame prediction reference pixel generation, reference image generation, filtering, etc., the data can refer to the recovered pixels. Furthermore, resizing can be performed in each resized region by using a separate data processing method.

[0583] Figure 24 This is a conceptual diagram illustrating the continuity between surfaces under a projection format (e.g., CHP, OHP, or ISP) according to an embodiment of the present invention.

[0584] In detail, Figure 24 Examples of images composed of multiple faces can be shown. Continuity can be a property generated in adjacent regions in 3D space. Sections 24a to 24c show, respectively, cases with both spatial adjacency and continuity when transformed into 2D space by projection processing (A), cases with spatial adjacency but no continuity (B), cases with no spatial adjacency but continuity (C), and cases with neither spatial adjacency nor continuity (D). In contrast, images in general are classified into cases with both spatial adjacency and continuity (A) and cases with neither spatial adjacency nor continuity (D). In this case, the cases with continuity correspond to some examples (A or C).

[0585] That is, referring to sections 24a to 24c, cases with both spatial adjacency and continuity (described here with reference to section 24a) can be shown as b0 to b4, and cases without spatial adjacency but with continuity can be shown as B0 to B6. In other words, these cases indicate adjacent regions in 3D space, and coding performance can be enhanced by using the continuity of b0 to b4 and B0 to B6 in the coding process.

[0586] Figure 25 This is a conceptual diagram showing the surface continuity in part 21c, which is an image obtained through image reconstruction processing or region packing processing under CMP projection format.

[0587] Here, Figure 21 Part 21c shows a rearrangement of the 360-degree image unfolded in the cube shape in part 21a, thus preserving the applied... Figure 21 The surface continuity of part 21a. That is, as shown in part 25a, surface S2,1 can be horizontally continuous with surfaces S1,1 and S3,1, and can be vertically continuous with surfaces S1,0 rotated by 90 degrees and S1,2 rotated by -90 degrees.

[0588] In the same manner, the continuity of surfaces S3,1, S0,1, S1,2, S1,1 and S1,0 can be checked in parts 25b to 25f.

[0589] The continuity between surfaces can be defined according to projection format settings, etc. However, the present invention is not limited to this and can be modified. Assumptions will be made as follows... Figure 24 and Figure 25 The following example describes the situation where continuity exists as shown.

[0590] Figure 26 This is an example diagram illustrating image size adjustment under the CMP projection format according to an embodiment of the present invention.

[0591] Section 26a shows an example of adjusting the image size, section 26b shows an example of adjusting the size of the surface unit (or partitioning unit), and section 26c shows an example of adjusting the size of the image and the surface unit (or an example of performing multiple size adjustments).

[0592] The size of a projected image can be adjusted using either a scaling factor or an offset factor, depending on the image size adjustment type. Here, the image before size adjustment can be P_Width × P_Height, and the image after size adjustment can be P'_Width × P'_Height, with the size of the face being F_Width × F_Height. The size can be the same or different depending on the face, and the width and height can also be the same or different depending on the face. However, for ease of description, this example will be described assuming all faces in the image have the same size and square shape. Furthermore, the description assumes the adjusted values ​​(WX and HY in this case) are the same. In the examples below, the focus will be on the case of the offset factor, and also on describing the data processing methods by copying specific regions of the image and by transforming specific regions of the image. The above settings can even be applied to... Figure 27 The situation shown in the figure.

[0593] For portions 26a to 26c, the boundary of a surface may be continuous with the boundary of another surface (here, it is assumed that it has a corresponding boundary). Figure 24 (Continuity of part 24a). Here, continuity can be classified into the case of having spatial adjacency and image continuity in the 2D plane (first example) and the case of having image continuity but no spatial adjacency in the 2D plane (second example).

[0594] For example, when assuming Figure 24 In the continuity of part 24a, the upper, left, right and lower regions of S1,1 can be spatially adjacent to the lower, right, left and upper regions of S1,0,S0,1,S2,1 and S1,2 and have image continuity with the lower, right, left and upper regions of S1,0,S0,1,S2,1 and S1,2 (first example).

[0595] Alternatively, the left and right regions of S1,0 are not spatially adjacent to the upper regions of S0,1 and S2,1, but can have image continuity with the upper regions of S0,1 and S2,1 (second example). Furthermore, the left regions of S0,1 can be spatially non-adjacent to each other, but have image continuity with each other (second example). Additionally, the left and right regions of S1,2 can be continuous with the lower regions of S0,1 and S2,1 (second example). This can only be a limited example, and other configurations can be applied depending on the definition and settings of the projection format. For ease of description, S0,0 to S3,2 in part 26a are referred to as a to l.

[0596] Part 26a could be an example of a filling method using data from regions having continuity toward the outer boundary of the image. The range from region A (excluding data) to the resized regions (here, a0 to a2, c0, d0 to d2, i0 to i2, k0, and l0 to l2) can be filled with any predetermined value or by filling with outer pixels, and the range from region B (including actual data) to the resized regions (here, b0, e0, h0, and j0) can be filled with data from regions (or faces) having image continuity. For example, b0 can be filled with data from the upper side of face h, e0 with data from the right side of face h, h0 with data from the left side of face e, and j0 with data from the lower side of face h.

[0597] Specifically, as an example, b0 can be filled with data of the lower side of the face obtained by rotating face h by 180 degrees, and j0 can be filled with data of the upper side of the face obtained by rotating face h by 180 degrees. However, this example (including the following examples) may only represent the position of the reference face, and may be considered in the context of... Figure 24 and Figure 25 Data is obtained from the sized region after the continuity between the surfaces is adjusted (e.g., rotated, etc.).

[0598] Part 26b can be an example of a filling method using data from regions with continuity toward the inner boundaries of the image. In this example, different resizing operations can be performed for each face. A shrinking process can be performed in region A, and an expanding process can be performed in region B. For example, the size of face a can be adjusted to the right (here, shrinking) w0, and the size of face b can be adjusted to the left (here, expanding) w0. Alternatively, the size of face a can be adjusted downwards (here, shrinking) h0, and the size of face e can be adjusted upwards (here, expanding) h0. In this example, when observing the change in the width of the image through faces a, b, c, and d, face a shrinks by w0, face b expands by w0 and w1, and face c can shrink by w1. Therefore, the width of the image before resizing is the same as the width of the image after resizing. When observing the change in the height of the image through faces a, e, and i, face a shrinks by h0, face e expands by h0 and h1, and face i can shrink by h1. Therefore, the height of the image before resizing is the same as the height of the image after resizing.

[0599] Considering that the region shrinks from region A which does not contain data, the resized regions (here b0, e0, be, b1, bg, g0, h0, e1, ej, j0, gi, g1, j1, and h1) can be simply removed, and considering that the region expands from region B which contains actual data, the resized regions can be filled with data from regions with continuity.

[0600] For example, b0 can be filled with the data on the top side of face e; e0 can be filled with the data on the left side of face b; be can be filled with the data of the left side of face b, the top side of face e, or the weighted sum of the left side of face b and the top side of face e; b1 can be filled with the data on the top side of face g; bg can be filled with the data of the left side of face b, the top side of face g, or the weighted sum of the right side of face b and the top side of face g; g0 can be filled with the data on the right side of face b; h0 can be filled with the data on the top side of face b; e1 can be filled with the data on the left side of face j; ej can be filled with the data of the bottom side of face e, the left side of face j, or the weighted sum of the bottom side of face e and the left side of face j; j0 can be filled with the data on the bottom side of face e; gj can be filled with the data of the bottom side of face g, the left side of face j, or the weighted sum of the bottom side of face g and the right side of face j; g1 can be filled with the data on the right side of face j; j1 can be filled with the data on the bottom side of face g; and h1 can be filled with the data on the bottom side of face j.

[0601] In the example above, when the resized region is filled with data from a specific region of the image, the data from the corresponding region can be copied and then used to fill the resized region, or it can be transformed based on the image's characteristics, type, etc., and then used to fill the resized region. For example, when a 360-degree image can be transformed into 2D space according to a projection format, a coordinate system (e.g., a 2D planar coordinate system) can be defined for each face. For ease of description, assume that (x, y, z) in 3D space is transformed into (x, y, C), (x, C, z), or (C, y, z) for each face. The example above indicates the following situation: data from faces other than the corresponding face is obtained from the resized region of a face. That is, when resizing is performed on the current face, data from other faces with different coordinate system characteristics can be copied as is and then used. In this case, there is a possibility that continuity may be distorted based on the resizing boundary. Therefore, data from other faces obtained according to the previous coordinate system characteristics can be transformed and used to fill the resized region. This transformation is only an example of a data processing method, and the invention is not limited thereto.

[0602] When data from a specific region of an image is copied and used to fill a resized region, distorted continuity (or fundamentally altered continuity) may be included in the boundary region between the resized region (e) and the resized region (e0). For example, continuity may change relative to the boundary, and straight edges may be curved relative to the boundary.

[0603] When data from a specific region of an image is transformed and used to fill a resized region, a gradually changing continuity may be included in the boundary regions between the resized regions.

[0604] The above examples can be seen as examples of the data processing method of the present invention for transforming data of a specific region of an image based on the characteristics, type, etc. of the image and filling the resized region with the transformed data.

[0605] Part 26c may be an example of combining the image resizing processes corresponding to parts 26a and 26b to fill the resized region with data from regions having continuity toward the boundaries (inner and outer boundaries) of the image. This example of resizing processing can be derived from the resizing processes of parts 26a and 26b, and its detailed description will be omitted.

[0606] Part 26a may be an example of a process for adjusting the image size, and part 26b may be an example of adjusting the size of the partitioning units in the image. Part 26c may be an example of multiple size adjustment processes that include the process for adjusting the image size and the process for adjusting the size of the partitioning units in the image.

[0607] For example, the size of an image acquired through projection processing (here, the first format) (region C) can be adjusted, and the size of an image acquired through format transformation processing (here, the second format) (region D) can be adjusted. In this example, the size of an image projected via ERP (the entire image) can be adjusted, and the image can be transformed into an image projected via CMP by the format transformation unit, and the size of the image projected via CMP (the surface unit) can be adjusted. The above example is an example of performing multiple size adjustment operations. However, the present invention is not limited thereto and can be modified thereto.

[0608] Figure 27 This is an example diagram illustrating the resizing of an image transformed and packaged in CMP projection format according to an embodiment of the present invention. Figure 27 Furthermore, assuming that Figure 25 The continuity between the faces shown means that the boundary of one face can be continuous with the boundary of another face.

[0609] In this example, the offset factors for W0 to W5 and H0 to H3 can have various values ​​(here, it is assumed that the offset factors are used as size adjustment values). For example, the offset factors can be derived from predetermined values, the motion search range of inter-frame prediction, units obtained from the image partitioning unit, etc., and other cases are also possible. In this case, the units obtained from the pixel partitioning unit can include faces. That is, the size adjustment value can be determined based on F_Width and F_...

Claims

1. A method for decoding an image using a decoding device, comprising: Receive the bitstream encoded with the image; Obtain index information from the bitstream to specify the block partitioning type of the current block in the image; Based on the index information, the block partitioning type of the current block is determined from the predefined candidate groups in the decoding device. The candidate group includes multiple candidate partitioning types. The plurality of candidate partition types include at least one of non-partition, first quadrilateral partition, second quadrilateral partition, binary partition, or trilateral partition, and Wherein, the first four-way partition represents dividing a decoding block into four decoding blocks based on a horizontal line and a vertical line, the second four-way partition represents dividing a decoding block into four decoding blocks based on three horizontal lines or three vertical lines, the binary partition represents dividing a decoding block into two decoding blocks, and the triangular partition represents dividing a decoding block into three decoding blocks. Based on the determined block partitioning type, the current block is divided into multiple sub-blocks; and Each sub-block is decoded with reference to the syntax information obtained from the bitstream. Decoding each of the sub-blocks includes generating a prediction block for each of the sub-blocks.

2. The method according to claim 1, in, The current block is set to be equal to the maximum decoded block or is set to be a block obtained by dividing the maximum decoded block. The size of the maximum decoded block is variably determined based on information about the maximum decoded block. The information regarding the largest decoded block is signaled from the bit stream.

3. The method according to claim 2, in, Based on the information regarding the maximum decoded block, the size of the maximum decoded block is set to be equal to 128×128 or 64×64.

4. The method according to claim 1, in, The number of candidate partition types that can be used for the current block varies depending on at least one of the size or shape of the current block.

5. The method according to claim 4, in, When the size of the current block is equal to or greater than a predetermined threshold, the candidate groups of the current block include the non-partition, the first quadrilateral partition, the second quadrilateral partition, the binary partition, and the trilateral partition. Wherein, when the size of the current block is less than the predetermined threshold, the candidate group of the current block does not include at least one of the second quadrilateral partition or the third quadrilateral partition.

6. The method according to claim 5, in, The predetermined threshold represents 16.

7. A method for encoding an image using an encoding device, comprising: The block partitioning type of the current block in the image is determined from a predefined candidate group in the encoding device. The candidate group includes multiple candidate partitioning types. The plurality of candidate partition types include at least one of non-partition, first quadrilateral partition, second quadrilateral partition, binary partition, or trilateral partition, and Wherein, the first four-way partition represents dividing a decoding block into four decoding blocks based on a horizontal line and a vertical line, the second four-way partition represents dividing a decoding block into four decoding blocks based on three horizontal lines or three vertical lines, the binary partition represents dividing a decoding block into two decoding blocks, and the triangular partition represents dividing a decoding block into three decoding blocks. Encode the index information used to specify the determined block partitioning type of the current block; and Each sub-block obtained from partitioning the current block based on the determined block partitioning type is encoded. Encoding each of the sub-blocks includes generating a prediction block for each of the sub-blocks.

8. A non-transitory computer-readable recording medium for storing a bitstream generated by encoding an image using an encoding device, the method comprising: The block partitioning type of the current block in the image is determined from a predefined candidate group in the encoding device. The candidate group includes multiple candidate partitioning types. The plurality of candidate partition types include at least one of non-partition, first quadrilateral partition, second quadrilateral partition, binary partition, or trilateral partition, and Wherein, the first four-way partition represents dividing a decoding block into four decoding blocks based on a horizontal line and a vertical line, the second four-way partition represents dividing a decoding block into four decoding blocks based on three horizontal lines or three vertical lines, the binary partition represents dividing a decoding block into two decoding blocks, and the triangular partition represents dividing a decoding block into three decoding blocks. Encode the index information used to specify the determined block partitioning type of the current block; and Each sub-block obtained from partitioning the current block based on the determined block partitioning type is encoded. Encoding each of the sub-blocks includes generating a prediction block for each of the sub-blocks.

Images