Image encoding / decoding method and method of transmitting bitstream

Abstract

Disclosed are an image encoding / decoding method and a method of transmitting a bitstream. A method of decoding a 360-degree image performed by an image decoding apparatus includes receiving a bitstream in which a 360-degree image is encoded, the bitstream including data of an extended two-dimensional image, the extended two-dimensional image including a two-dimensional image and a predetermined extension region, and the two-dimensional image being projected from an image having a three-dimensional projection structure and including one or more faces; generating a predicted image by performing prediction based on information about prediction included in the bitstream; and reconstructing the extended two-dimensional image based on the predicted image and a residual image.

Description

[0001] This application is a divisional application of application number 201780073662.3, filed on October 10, 2017, entered the national phase on May 28, 2019, and entitled "Image Data Encoding / Decoding Method and Apparatus". Technical Field

[0002] This invention relates to image data encoding and decoding technology, and more specifically, to methods and apparatus for encoding and decoding 360-degree images of real-world media services. Background Technology

[0003] With the widespread adoption of the internet and mobile devices, and the development of information and communication technologies, the use of multimedia data is rapidly increasing. Recently, there has been a growing demand in various fields for high-quality and high-resolution images, such as high-definition (HD) and ultra-high-definition (UHD) images, as well as for reality media services such as virtual reality and augmented reality. Specifically, the amount of data generated by processing multi-view images captured by multiple camera devices for 360-degree images in virtual reality and augmented reality has increased dramatically, but the performance of image processing systems is insufficient to handle such large amounts of data.

[0004] As mentioned above, in existing image encoding and decoding methods and apparatuses, there is a need to improve the performance of image processing, especially the performance of image encoding / decoding. Summary of the Invention

[0005] Technical issues

[0006] The object of this invention is to provide a method for improving image setup processing in the initial steps of encoding and decoding. More specifically, this invention aims to provide an encoding and decoding method and apparatus for improving image setup processing while taking into account the characteristics of 360-degree images.

[0007] Technical solutions

[0008] According to one aspect of the present invention, a method for decoding 360-degree images is provided.

[0009] Here, the method for decoding a 360-degree image may include: receiving a bitstream including an encoded 360-degree image; generating a prediction image by referring to syntax information obtained from the received bitstream; obtaining a decoded image by combining the generated prediction image with a residual image obtained by inverse quantization and inverse transform of the bitstream; and reconstructing the decoded image into a 360-degree image according to a projection format.

[0010] Here, the syntax information may include projection format information for the 360-degree image.

[0011] Here, the projection format information can be information indicating at least one of the following: Equi-Rectangular Projection (ERP) format, in which a 360-degree image is projected onto a 2D plane; CubeMap Projection (CMP) format, in which a 360-degree image is projected onto a cube; Octahedron Projection (OHP) format, in which a 360-degree image is projected onto an octahedron; and IcoSahedral Projection (ISP) format, in which a 360-degree image is projected onto a polyhedron.

[0012] Here, reconstruction may include: obtaining arrangement information by referring to grammatical information based on regional packing; and rearranging the blocks of the decoded image based on the arrangement information.

[0013] Here, the generation of the predicted image may include: performing image expansion on a reference image obtained by recovering the bitstream; and generating the predicted image by referring to the reference image that has been image expanded.

[0014] Here, performing image expansion may include performing image expansion based on the partitioning units of the reference image.

[0015] Here, performing image expansion based on partitioning units can include generating an expanded region separately for each partitioning unit by using reference pixels of the partitioning unit.

[0016] Here, the extended region can be generated using the boundary pixels of the partition unit that is spatially adjacent to the partition unit to be extended, or using the boundary pixels of the partition unit that has image continuity with the partition unit to be extended.

[0017] Here, performing image expansion based on partitioning units can include generating an expanded image of the combined regions using the boundary pixels of regions that are spatially adjacent to each other in the partitioning units.

[0018] Here, performing image expansion based on partitioning units can include: using all neighboring pixel information of spatially adjacent partitioning units to generate an expanded region between adjacent partitioning units.

[0019] Here, performing image expansion based on partitioning units can include generating an expanded region using the average value of neighboring pixels of spatially adjacent partitioning units.

[0020] Beneficial effects of the present invention

[0021] The image encoding / decoding method and apparatus according to embodiments of the present invention can enhance compression performance. In particular, compression performance can be enhanced for 360-degree images. Attached Figure Description

[0022] Figure 1 This is a block diagram of an image encoding apparatus according to an embodiment of the present invention.

[0023] Figure 2 This is a block diagram of an image decoding apparatus according to an embodiment of the present invention.

[0024] Figure 3 This is an example diagram of dividing image information into multiple layers to compress the image.

[0025] Figure 4 This is a conceptual diagram illustrating an example of image segmentation according to an embodiment of the present invention.

[0026] Figure 5 This is another example diagram of the image segmentation method according to an embodiment of the present invention.

[0027] Figure 6 This is an example diagram of a common image resizing method.

[0028] Figure 7 This is an example diagram of image size adjustment according to an embodiment of the present invention.

[0029] Figure 8 This is an example diagram illustrating a method for constructing an expanded region in an image resizing method according to an embodiment of the present invention.

[0030] Figure 9 This is an example diagram illustrating a method for constructing the region to be deleted and the region to be generated in an image resizing method according to an embodiment of the present invention.

[0031] Figure 10 This is an example diagram of image reconstruction according to an embodiment of the present invention.

[0032] Figure 11 This is an example diagram showing images before and after image setting processing according to an embodiment of the present invention.

[0033] Figure 12 This is an example diagram illustrating the adjustment of the size of each segmentation unit of an image according to an embodiment of the present invention.

[0034] Figure 13 This is an example diagram of the settings or size adjustments of the division units in an image.

[0035] Figure 14This is an example diagram illustrating both the process of adjusting the image size and the process of adjusting the size of the partitioned units in the image.

[0036] Figure 15 This is an example diagram showing a 3D space and a 2D planar space for displaying a three-dimensional (3D) image.

[0037] Figures 16a to 16d This is a conceptual diagram illustrating a projection format according to an embodiment of the present invention.

[0038] Figure 17 This is a conceptual diagram illustrating the projection format included in a rectangular image according to an embodiment of the present invention.

[0039] Figure 18 This is a conceptual diagram of a method for converting a projection format into a rectangular shape according to an embodiment of the present invention, namely, a method for performing rearrangement on a surface to exclude meaningless regions.

[0040] Figure 19 This is a conceptual diagram illustrating the execution of a region-wise packing process according to an embodiment of the present invention to convert a CMP projection format into a rectangular image.

[0041] Figure 20 This is a conceptual diagram of 360-degree image segmentation according to an embodiment of the present invention.

[0042] Figure 21 This is an example diagram of 360-degree image segmentation and image reconstruction according to an embodiment of the present invention.

[0043] Figure 22 This is an example of how an image packaged or projected by CMP is divided into tiles.

[0044] Figure 23 This is a conceptual diagram illustrating an example of adjusting the size of a 360-degree image according to an embodiment of the present invention.

[0045] Figure 24 This is a conceptual diagram illustrating the continuity between surfaces under a projection format (e.g., CHP, OHP, or ISP) according to an embodiment of the present invention.

[0046] Figure 25 This is a conceptual diagram showing the continuity of the surfaces of part 21c, which is an image obtained through image reconstruction processing in CMP projection format or region packing processing.

[0047] Figure 26 This is an example diagram illustrating image size adjustment under the CMP projection format according to an embodiment of the present invention.

[0048] Figure 27 This is an example diagram illustrating the resizing of an image transformed and packaged in CMP projection format according to an embodiment of the present invention.

[0049] Figure 28 This is an example diagram illustrating a data processing method for adjusting the size of a 360-degree image according to an embodiment of the present invention.

[0050] Figure 29 This is an example diagram illustrating a tree-based block format.

[0051] Figure 30 This is an example diagram showing a type-based block format.

[0052] Figure 31 This is an example diagram showing various types of blocks that can be obtained by the block division section of the present invention.

[0053] Figure 32 This is an example diagram illustrating tree-based partitioning according to an embodiment of the present invention.

[0054] Figure 33 This is an example diagram illustrating tree-based partitioning according to an embodiment of the present invention. Detailed Implementation

[0055] While the invention can be subject to various modifications and alternatives, specific embodiments thereof are shown by way of example in the accompanying drawings and will be described in detail herein. However, it should be understood that the invention is not intended to be limited to the specific forms disclosed, but rather, the invention covers all modifications, equivalents, and alternatives falling within the spirit and scope of the invention.

[0056] It should be understood that although the terms “first,” “second,” etc., may be used herein to describe various elements, these elements should not be limited by these terms. These terms are used only to distinguish one element from another. For example, without departing from the scope of the invention, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element. As used herein, the term “and / or” includes any and all combinations of one or more of the associated listed items.

[0057] It should be understood that when an element is referred to as "connected" or "coupled" to another element, it can be directly connected to or coupled to the other element, or there may be intermediate elements. In contrast, when an element is referred to as "directly connected" or "directly coupled" to another element, there are no intermediate elements. Other words used to describe the relationship between elements should be interpreted in a similar manner (i.e., "between" and "directly between," "adjacent" and "directly adjacent," etc.).

[0058] The technical terms used herein are for the purpose of describing particular embodiments only and are not intended to limit the invention. As used herein, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the terms “comprises,” “comprising,” “includes,” and / or “including” as used herein specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0059] Unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in common dictionaries should be interpreted as having the meaning consistent with their meaning in the context of the relevant field and will not be interpreted in an idealized or overly formal sense unless expressly defined herein.

[0060] Both the image encoding device and the image decoding device can be user terminals such as personal computers (PCs), laptops, personal digital assistants (PDAs), portable multimedia players (PMPs), portable game consoles (PSPs), wireless communication terminals, smartphones and televisions, virtual reality (VR) devices, augmented reality (AR) devices, mixed reality (MR) devices, head-mounted display (HMD) devices, and smart glasses, or server terminals such as application servers and service servers. They can also include various devices such as communication modems for communicating with various devices or wired / wireless communication networks; memory for storing various programs and data for encoding or decoding images or executing inter-frame or intra-frame prediction for encoding or decoding; and processors for executing programs to perform computational and control operations. Furthermore, images encoded into a bitstream by the image encoding device can be transmitted to the image decoding device in real-time or non-real-time via wired / wireless communication networks such as the Internet, short-range wireless networks, wireless local area networks (LANs), WiBro networks, and mobile communication networks, or via various communication interfaces such as cables and universal serial buses (USB). Then, the bitstream can be decoded by an image decoding device to restore and replay the bitstream as an image.

[0061] Furthermore, an image encoded into a bitstream by an image encoding device can be transmitted from the image encoding device to the image decoding device via a computer-readable recording medium.

[0062] The aforementioned image encoding and decoding devices can be separate devices, but they can be configured as a single image encoding / decoding device depending on the implementation method. In this case, some elements of the image encoding device can be substantially the same as some elements of the image decoding device, and can be implemented to include at least the same structure or perform the same function.

[0063] Therefore, redundant descriptions of the corresponding technical elements will be omitted in the following detailed description of the technical elements and their working principles.

[0064] Furthermore, the image decoding device corresponds to the computing device that applies the image encoding method executed by the image encoding device to the decoding process, and therefore the following description will focus on the image encoding device.

[0065] The computing device may include: a memory configured to store programs or software patterns for implementing image encoding methods and / or image decoding methods; and a processor connected to the memory to execute the programs. Furthermore, the image encoding device may also be called an encoder, and the image decoding device may also be called a decoder.

[0066] Typically, an image can consist of a series of still images. Still images can be classified into groups of pictures (GOPs), and each still image can be referred to as a picture. In this case, a picture can indicate one of the frames and fields in progressive and interlaced signals. When encoding / decoding is performed on a frame-based basis, a picture can be represented as a "frame," and when encoding / decoding is performed on a field-based basis, a picture can be represented as a "field." This invention assumes progressive signals but can also be applied to interlaced signals. As a higher concept, units such as GOPs and sequences can exist, and each picture can also be divided into predetermined regions such as slices, tiles, blocks, etc. Furthermore, a GOP can include units such as I-pictures, P-pictures, and B-pictures. An I-picture can refer to a picture that is autonomously encoded / decoded without using a reference picture, and P-pictures and B-pictures can refer to pictures that are encoded / decoded by performing processes such as motion estimation and motion compensation using a reference picture. Typically, a P-picture can use I-pictures and B-pictures as reference pictures, and a B-picture can use I-pictures and P-pictures as reference pictures. However, the above definitions can also be changed by the encoding / decoding settings.

[0067] Here, the image referenced during encoding / decoding is called the reference image, and the block or pixel referenced during encoding / decoding is called the reference block or reference pixel. Furthermore, the reference data can include various types of encoding / decoding information and frequency domain coefficients, as well as spatial domain pixel values, generated and determined during the encoding / decoding process. For example, the reference data may correspond to intra-frame prediction information or motion information in the prediction unit, transform information in the transform / inverse transform unit, quantization information in the quantization / inverse quantization unit, encoding / decoding information (context information) in the encoding / decoding unit, filter information in the in-loop filter unit, etc.

[0068] The smallest unit of an image can be a pixel, and the number of bits used to represent a pixel is called the bit depth. Typically, the bit depth can be 8 bits, and bit depths of 8 bits or more can be supported depending on the encoding settings. At least one bit depth can be supported based on the color space. Furthermore, at least one color space can be included based on the image color format. One or more images with the same size or one or more images with different sizes can be included based on the color format. For example, YCbCr 4:2:0 can consist of one luminance component (Y in this example) and two chrominance components (Cb / Cr in this example). In this case, the composition ratio of the chrominance component to the luminance component can be 1:2 in width and height. As another example, YCbCr 4:4:4 can have the same composition ratio in width and height. Similar to the example above, when including one or more color spaces, the image can be divided into color spaces.

[0069] The invention will be described based on any color space (Y in this example) of any color format (YCbCr), and this description will be applied in the same or similar manner (depending on the settings of the specific color space) to other color spaces of that color format (Cb and Cr in this example). However, each color space may be given partial differences (independent of the settings of the specific color space). That is, settings dependent on each color space may refer to settings that are proportional to or dependent on the composition ratio of each component (e.g., 4:2:0, 4:2:2, or 4:4:4), and settings independent of each color space may refer to settings of the corresponding color space that are independent of or unrelated to the composition ratio of each component. In this invention, some elements may have independent settings or dependent settings depending on the encoder / decoder.

[0070] The setup information or syntax elements required during image encoding processing can be determined hierarchically by units such as video, sequence, picture, slice, tile, and block. These units include video parameter sets (VPS), sequence parameter sets (SPS), picture parameter sets (PPS), slice headers, tile headers, and block headers. The encoder can add units to the bitstream and send the bitstream to the decoder. The decoder can parse the bitstream at the same level, recover the setup information sent by the encoder, and use the setup information in image decoding processing. Furthermore, relevant information can be sent via the bitstream in the form of supplementary enhancement information (SEI) or metadata, which can then be parsed and used. Each parameter set has a unique ID value, and a lower-level parameter set can have the ID value of a higher-level parameter set to be referenced. For example, a lower-level parameter set can reference information from one or more higher-level parameter sets with corresponding ID values. In the various examples of the units described above, when any unit comprises one or more different units, any unit can be referred to as a higher-level unit, and the included units can be referred to as lower-level units.

[0071] Setting information occurring within such units can include settings independent of each unit or settings dependent on previous, subsequent, or higher-level units. Here, it will be understood that dependent settings use tagging information corresponding to the settings of previous, subsequent, or higher-level units (e.g., a 1-bit flag; 1 indicates following, and 0 indicates not following) to indicate the setting information of the corresponding unit. In this invention, examples of independent settings will be focused on to describe the setting information. However, examples may also be included where relationships between the setting information of previous, subsequent, or higher-level units dependent on the current unit are added to or replace independent settings.

[0072] Figure 1 This is a block diagram of an image encoding apparatus according to an embodiment of the present invention. Figure 2 This is a block diagram of an image decoding apparatus according to an embodiment of the present invention.

[0073] Reference Figure 1 The image encoding apparatus may be configured to include a prediction unit, a subtractor, a transform unit, a quantization unit, an inverse quantization unit, an inverse transform unit, an adder, an in-loop filter unit, a memory, and / or an encoding unit, some of which may not necessarily be included. Depending on the implementation, some or all of these elements may be selectively included, and additional elements not shown herein may be included.

[0074] Reference Figure 2The image decoding apparatus may be configured to include a decoding unit, a prediction unit, an inverse quantization unit, an inverse transform unit, an adder, an in-loop filter unit, and / or a memory, some of which may not necessarily be included. Depending on the implementation, some or all of these elements may be selectively included, and some additional elements not shown herein may be included.

[0075] Image encoding and decoding devices can be separate devices, but they can also be configured as a single image encoding / decoding device depending on the implementation. In this case, some elements of the image encoding device can be substantially the same as some elements of the image decoding device, and can be implemented to include at least the same structure or perform the same function. Therefore, redundant descriptions of the corresponding technical elements will be omitted in the following detailed description of the technical elements and their working principles. The image decoding device corresponds to the computing device that applies the image encoding method performed by the image encoding device to the decoding process; therefore, the following description will focus on the image encoding device. The image encoding device can also be called an encoder, and the image decoding device can also be called a decoder.

[0076] The prediction unit can be implemented using a prediction module and can generate prediction blocks by performing intra-frame prediction or inter-frame prediction on the block to be encoded. The prediction unit generates prediction blocks by predicting the current block to be encoded in the image. In other words, the prediction unit can predict the pixel values ​​of the pixels of the current block to be encoded in the image through intra-frame prediction or inter-frame prediction to generate prediction blocks with the predicted pixel values ​​of the pixels. Furthermore, the prediction unit can deliver the information needed to generate prediction blocks to the encoding unit, enabling the encoding of prediction mode information. The encoding unit adds the corresponding information to the bitstream and transmits the bitstream to the decoder. The decoding unit of the decoder can parse the corresponding information, recover the prediction mode information, and then use the prediction mode information to perform intra-frame prediction or inter-frame prediction.

[0077] The subtractor subtracts the prediction block from the current block to generate a residual block. In other words, the subtractor can calculate the difference between the pixel value of each pixel in the current block to be encoded and the predicted pixel value of each pixel in the prediction block generated by the prediction unit to generate a residual block as a block-type residual signal.

[0078] The transform unit can transform a signal belonging to the spatial domain into a signal belonging to the frequency domain. In this case, the signal obtained through the transform process is called the transform coefficient. For example, a residual block with a residual signal delivered from a subtractor can be transformed into a transform block with transform coefficients. In this case, the input signal is determined according to the encoding settings, and the input signal is not limited to the residual signal.

[0079] The transform unit can perform transformations on the residual blocks using transformation techniques such as the Hadamard Transform, the Discrete Sine Transform (DST), and the Discrete Cosine Transform (DCT). However, the invention is not limited to this, and various enhanced and modified transformation techniques can be used.

[0080] For example, at least one transformation technique can be supported, and at least one detailed transformation technique can be supported within each transformation technique. In this case, the at least one detailed transformation technique can be a transformation technique that constructs some basis vectors differently in each transformation technique. For example, as transformation techniques, DST-based transformations and DCT-based transformations can be supported. For DST, detailed transformation techniques such as DST-I, DST-II, DST-III, DST-V, DST-VI, DST-VII, and DST-VIII can be supported, and for DCT, detailed transformation techniques such as DCT-I, DCT-II, DCT-III, DCT-V, DCT-VI, DCT-VII, and DCT-VIII can be supported.

[0081] One of the transformation techniques can be set as the default transformation technique (e.g., one transformation technique && one detailed transformation technique), and additional transformation techniques can be supported (e.g., multiple transformation techniques || multiple detailed transformation techniques). Whether additional transformation techniques are supported can be determined at the sequence, image, slice, or tile level, and related information can be generated based on the unit. When additional transformation techniques are supported, transformation technique selection information can be determined at the block level, and related information can be generated.

[0082] Transformations can be performed horizontally and / or vertically. For example, a two-dimensional (2D) transformation can be performed by using a one-dimensional (1D) transformation horizontally and vertically with basis vectors, so that pixel values ​​in the spatial domain can be transformed into the frequency domain.

[0083] Furthermore, transformations can be performed adaptively, horizontally and / or vertically. Specifically, whether to perform a transformation adaptively can be determined based on at least one coding setting. For intra-frame prediction, for example, DCT-I can be applied horizontally and DST-I can be applied vertically when the prediction mode is horizontal; DST-VI can be applied horizontally and DCT-VI can be applied vertically when the prediction mode is vertical; DCT-II can be applied horizontally and DCT-V can be applied vertically when the prediction mode is diagonally downward to the left; and DST-I can be applied horizontally and DST-VI can be applied vertically when the prediction mode is diagonally downward to the right.

[0084] The size and shape of a transform block can be determined based on the encoding costs of candidates for the size and shape of the transform block. The image data of the transform block and information about the determined size and shape of the transform block can then be encoded.

[0085] Among the transformation methods, a square transformation can be set as the default transformation method, and additional transformation methods (such as a rectangular transformation) can be supported. Whether to support additional transformation methods can be determined at the unit level (sequence, image, slice, or tile), and related information can be generated based on the unit. Transformation method selection information can be determined at the block level, and related information can be generated.

[0086] Furthermore, it can be determined whether a transform block format is supported based on the encoding information. In this case, the encoding information may correspond to slice type, encoding mode, block size and shape, block partitioning scheme, etc. That is, a transform format can be supported based on at least one piece of encoding information, and multiple transform formats can be supported based on at least one piece of encoding information. The former may be an implicit situation, while the latter may be an explicit situation. For the explicit situation, adaptive selection information indicating the best candidate group to be selected from multiple candidate groups can be generated, and this adaptive selection information is added to the bitstream. According to the present invention, in addition to this example, it will be understood that when encoding information is explicitly generated, the information is added to the bitstream in various units, and the relevant information is parsed by the decoder in various units and the relevant information is recovered into decoded information. Furthermore, it should be understood that when encoding / decoding information is implicitly processed, the encoder and decoder perform the processing through the same processing, rules, etc.

[0087] As an example, support for rectangular transformations can be determined based on the slice type. For I slices, supported transformations can be square transformations, and for P / B slices, supported transformations can be either square or rectangular transformations.

[0088] As an example, support for rectangular transforms can be determined based on the coding mode. For intra-frame prediction, the supported transform can be a square transform, and for inter-frame prediction, the supported transform can be a square transform and / or a rectangular transform.

[0089] As an example, support for rectangular transformations can be determined based on the size and shape of the block. Blocks of a certain size or larger support square transformations, while blocks smaller than a certain size support both square and / or rectangular transformations.

[0090] As an example, support for rectangular transformations can be determined based on the block partitioning scheme. When the block to be transformed is obtained through a quadtree partitioning scheme, the supported transformation can be a square transformation. When the block to be transformed is obtained through a binary tree partitioning scheme, the supported transformation can be either a square transformation or a rectangular transformation.

[0091] The above examples are examples of support for transform forms based on a single encoded piece of information, and multiple pieces of information can be combined and associated with additional transform form support settings. The above examples are merely examples of additional transform form support based on various encoding settings. However, the invention is not limited thereto and various modifications can be made thereto.

[0092] Transform processing can be omitted based on encoding settings or image characteristics. For example, transform processing (including inverse processing) can be omitted based on encoding settings (e.g., assuming a lossless compression environment in this example). As another example, transform processing can be omitted when the compression performance through the transform is not shown based on image characteristics. In this case, the transform can be omitted for all cells or one of the horizontal and vertical cells. Whether omission is supported can be determined based on the size and shape of the block.

[0093] For example, suppose horizontal and vertical transformations are set to be jointly omitted. When the transformation omission flag is 1, the transformation can be performed neither horizontally nor vertically, and when the transformation omission flag is 0, the transformation can be performed both horizontally and vertically. On the other hand, suppose horizontal and vertical transformations are set to be omitted independently. When the first transformation omission flag is 1, the horizontal transformation is not performed, and when the first transformation omission flag is 0, the horizontal transformation is performed. Moreover, when the second transformation omission flag is 1, the vertical transformation is not performed, and when the second transformation omission flag is 0, the vertical transformation is performed.

[0094] Transform omitting is supported when the block size corresponds to range A, but not when the block size corresponds to range B. For example, transform omitting is not supported when the block width is greater than M or the block height is greater than N. Transform omitting is supported when the block width is less than m or the block height is less than n. M(m) and N(n) can be the same or different from each other. The settings associated with the transform can be determined at the unit level, such as sequences, images, slices, etc.

[0095] When additional transform techniques are supported, the transform technique settings can be determined based on at least one piece of encoding information. In this case, the encoding information may correspond to slice type, encoding mode, block size and shape, prediction mode, etc.

[0096] As an example, support for transform techniques can be determined based on the coding mode. For intra-frame prediction, supported transform techniques may include DCT-I, DCT-III, DCT-VI, DST-II, and DST-III, and for inter-frame prediction, supported transform techniques may include DCT-II, DCT-III, and DST-III.

[0097] As an example, the supported transformation techniques can be determined based on the slice type. For I slices, supported transformation techniques may include DCT-I, DCT-II, and DCT-III; for P slices, supported transformation techniques may include DCT-V, DST-V, and DST-VI; and for B slices, supported transformation techniques may include DCT-I, DCT-II, and DST-III.

[0098] As an example, support for transformation techniques can be determined based on the prediction mode. Prediction mode A supports transformation techniques such as DCT-I and DCT-II, prediction mode B supports transformation techniques such as DCT-I and DST-I, and prediction mode C supports transformation techniques such as DCT-I. In this case, prediction modes A and B can both be directional modes, and prediction mode C can be a non-directional mode.

[0099] As an example, the supported transformation techniques can be determined based on the size and shape of the block. Transformation techniques supported by blocks of a specific size or larger may include DCT-II; transformation techniques supported by blocks smaller than a specific size may include DCT-II and DST-V; and transformation techniques supported by blocks of a specific size or larger, as well as blocks smaller than a specific size, may include DCT-I, DCT-II, and DST-I. Furthermore, transformation techniques supported in a square shape may include DCT-I and DCT-II, and transformation techniques supported in a rectangular shape may include DCT-I and DST-I.

[0100] The above examples can be seen as examples of support for transform techniques based on a single encoded piece of information, and multiple pieces of information can be combined and associated with additional transform technique support settings. The present invention is not limited to the above examples and modifications can be made to them. Furthermore, the transform unit can deliver information required to generate a transform block to the encoding unit, thereby encoding the information. The encoding unit adds the corresponding information to the bitstream and transmits the bitstream to the decoder. The decoding unit of the decoder can parse the information and use the parsed information in the inverse transform process.

[0101] The quantization unit can quantize the input signal. In this case, the signal obtained through quantization processing is called the quantization coefficient. For example, the quantization unit can quantize a residual block with residual transform coefficients delivered from the transform unit to obtain a quantized block with quantization coefficients. In this case, the input signal is determined according to the encoding settings and is not limited to the residual transform coefficients.

[0102] The quantization unit can use quantization techniques such as Dead Zone Uniform Threshold Quantization (DVT) and quantization weighting matrices to quantize the transformed residual blocks. However, the invention is not limited to this, and various improved and modified quantization techniques can be used. Whether additional quantization techniques are supported can be determined on a unit basis, such as a sequence, image, slice, or tile, and relevant information can be generated based on the unit. When additional quantization techniques are supported, quantization technique selection information can be determined on a block basis, and relevant information can be generated.

[0103] When additional quantization techniques are supported, the quantization technique settings can be determined based on at least one piece of encoding information. In this case, the encoding information may correspond to slice type, encoding mode, block size and shape, prediction mode, etc.

[0104] For example, the quantization unit can set different quantization weighting matrices corresponding to the coding mode and different weighting matrices applied according to inter-frame prediction / intra-frame prediction. Furthermore, the quantization unit can set different weighting matrices applied according to the intra-frame prediction mode. In this case, assuming the quantization weighting matrix has a size of M×N, the same as the size of the quantization block, the quantization weighting matrix can be a quantization matrix constructed differently for some quantization components.

[0105] Quantization can be omitted depending on encoding settings or image characteristics. For example, quantization (including inverse quantization) can be omitted based on encoding settings (e.g., in this example, assuming a lossless compression environment). As another example, quantization can be omitted when the compression performance after quantization is not shown based on image characteristics. In this case, some or all of the region can be omitted, and whether omission is supported can be determined based on the size and shape of the block.

[0106] Information about quantization parameters (QPs) can be generated in units such as sequences, images, slices, tiles, or blocks. For example, information about QPs can be generated in the higher-level unit where QP information is first generated. <1> A default QP is set in the middle, and the QP can be set to a value that is the same as or different from the QP set in the higher-level unit. The QP can be ultimately determined during quantization processing performed in some units through this process. In this case, units such as sequences and images can be... <1> Corresponding examples, such as slices, tiles, and blocks, can be units that are... <2> Corresponding examples, and units such as blocks can be related to <3> Corresponding example.

[0107] Information about QPs can be generated based on the QPs in each unit. Alternatively, a predetermined QP can be set as a predicted value, and information about the difference between the predicted QP and the QP in the unit can be generated. Alternatively, a QP obtained based on at least one of the QPs set in the parent unit, the same and previous units, or the adjacent units can be set as a predicted value, and information about the difference between the predicted QP and the QP in the current unit can be generated. Alternatively, a QP set in the parent unit and a QP obtained based on at least one piece of coding information can be set as predicted values, and information about the difference between the predicted QP and the QP in the current unit can be generated. In this case, the same and previous units can be units that can be defined according to the order in which units are encoded, adjacent units can be spatially adjacent units, and the coding information can be the slice type, coding mode, prediction mode, position information, etc. of the corresponding unit.

[0108] As an example, the QP in the current cell can be used to set the QP in the higher-level cell as the predicted value and generate difference information. Information about the difference between the QP set in the slice and the QP set in the image can be generated, or information about the difference between the QP set in the tile and the QP set in the image can be generated. Furthermore, information about the difference between the QP set in the block and the QP set in the slice or tile can be generated. Additionally, information about the difference between the QP set in the sub-block and the QP set in the block can be generated.

[0109] As an example, a QP obtained based on at least one QP in a neighboring cell or at least one QP in a previous cell can be used as the prediction value, and difference information can be generated. Information about the difference between the QP obtained and the QP obtained based on the QPs of neighboring blocks such as the left, top-left, bottom-left, top, and top-right sides of the current block can be generated. Alternatively, information about the difference between the QP and the QP of encoded images preceding the current image can be generated.

[0110] As an example, the QP in the current cell can be used to set the QP in the upper cell and the QP obtained based on at least one piece of coding information as the prediction value, and difference information can be generated. Furthermore, information about the difference between the QP in the current block and the QP of a slice corrected according to slice type (I / P / B) can be generated. Alternatively, information about the difference between the QP in the current block and the QP of a tile corrected according to coding mode (intra-frame / inter-frame) can be generated. Alternatively, information about the difference between the QP in the current block and the QP of a picture corrected according to prediction mode (directional / non-directional) can be generated. Alternatively, information about the difference between the QP in the current block and the QP of a picture corrected according to position information (x / y) can be generated. In this case, the correction can be an operation that adds an offset to or subtracts an offset from the QP used for prediction in the upper cell. In this case, at least one offset information can be supported according to the coding settings, and information that is implicitly processed or explicitly associated can be generated according to predetermined processing. The present invention is not limited to the above examples and modifications can be made to the above examples.

[0111] The above examples are permissible when a signal indicating a change in QP is provided or activated. For instance, when neither a signal indicating a change in QP is provided nor activated, no difference information is generated, and the predicted QP can be determined for each cell. As another example, when a signal indicating a change in QP is provided or activated, difference information is generated, and when the value of the difference information is 0, the predicted QP can be determined for each cell.

[0112] The quantization unit delivers the information needed to generate the quantized block to the encoding unit, enabling the information to be encoded. The encoding unit adds the corresponding information to the bitstream and transmits the bitstream to the decoder. The decoder's decoding unit can parse the information and use the parsed information in the inverse quantization process.

[0113] The above example has been described assuming that the residual block is transformed and quantized by the transform and quantization units. However, the residual signal of the residual block can be transformed into a residual block with transform coefficients without performing quantization. Alternatively, only quantization can be performed without transforming the residual signal of the residual block into transform coefficients. Alternatively, neither transform nor quantization can be performed. This can be determined based on the encoding settings.

[0114] The encoding unit can scan the quantization coefficients, transform coefficients, or residual signals of the generated residual blocks in at least one scanning order (e.g., zigzag scanning, vertical scanning, horizontal scanning, etc.), generate a quantization coefficient string, transform coefficient string, or signal string, and encode the quantization coefficient string, transform coefficient string, or signal string using at least one entropy coding technique. In this case, information about the scanning order can be determined according to the encoding settings (e.g., encoding mode, prediction mode, etc.), and the information about the scanning order can be used to generate implicitly determined information or explicitly associated information. For example, a scanning order can be selected from multiple scanning orders according to the intra-frame prediction mode.

[0115] Furthermore, the encoding unit can generate encoded data including the encoded information delivered from each element, and can output the encoded data as a bitstream. This can be implemented using a multiplexer (MUX). In this case, methods such as Exponential Columbus, Context Adaptive Variable Length Coding (CAVLC), and Context Adaptive Binary Arithmetic Coding (CABAC) can be used as encoding techniques to perform encoding. However, the present invention is not limited to this, and various encoding techniques obtained by improving and modifying the above-described encoding techniques can be used.

[0116] When entropy encoding (e.g., CABAC in this example) is performed on syntax elements such as information generated through encoding / decoding processes and residual block data, the entropy encoding apparatus may include a binarizer, a context modeler, and a binary arithmetic encoder. In this case, the binary arithmetic encoder may include a regular encoding engine and a bypass encoding engine.

[0117] The syntax elements input to the entropy encoding device do not have to be binary values. Therefore, when a syntax element is not a binary value, the binarizer can binarize the syntax element and output a binary bit string (binstring) consisting of 0s and 1s. In this case, the binary bits (bin) represent bits consisting of 0s and 1s and can be encoded by a binary arithmetic encoder. In this case, one of the regular encoding engine and the bypass encoding engine can be selected based on the probability of occurrence of 0s and 1s, and this can be determined according to the encoding / decoding settings. When the syntax element is data where the frequency of 0s is equal to the frequency of 1s, the bypass encoding engine can be used; otherwise, the regular encoding engine can be used.

[0118] Various methods can be used when binarizing grammatical elements. For example, fixed-length binarization, unary binarization, truncated Rice binarization, and K-order exponential Golomb binarization can be used. Furthermore, signed or unsigned binarization can be performed depending on the range of values ​​of the grammatical element. The binarization processing of grammatical elements according to the present invention can include additional binarization methods as well as the binarization described in the examples above.

[0119] The inverse quantization unit and the inverse transform unit can be implemented by reversing the processes performed in the transform unit and the quantization unit. For example, the inverse quantization unit can inverse quantize the transform coefficients quantized by the quantization unit, and the inverse transform unit can inverse transform the inverse quantized transform coefficients to generate the recovered residual block.

[0120] The adder adds the predicted block and the recovered residual block to recover the current block. The recovered block can be stored in memory and used as reference data (for the prediction section, filter section, etc.).

[0121] The in-loop filter unit can additionally perform one or more post-processing filtering operations such as deblocking filter, Sample Adaptive Offset (SAO), and Adaptive Loop Filter (ALF). The deblocking filter can remove block distortion generated at the boundaries between blocks from the restored image. ALF can perform filtering based on values ​​obtained by comparing the input image with the restored image. Specifically, ALF can perform filtering based on values ​​obtained by comparing the input image with the restored image after filtering the blocks using the deblocking filter. Alternatively, ALF can perform filtering based on values ​​obtained by comparing the input image with the restored image after filtering the blocks using SAO. SAO can recover offset differences based on values ​​obtained by comparing the input image with the restored image, and can be applied in the form of band offset (BO), edge offset (EO), etc. Specifically, SAO can add an offset relative to the original image to the restored image to which the deblocking filter is applied, at least one pixel unit, and can be applied in the form of BO, EO, etc. In detail, SAO can add offsets relative to the original image to the image recovered after filtering the blocks by ALF, on a pixel-by-pixel basis, and can be applied in the form of BO, EO, etc.

[0122] As filtering information, setting information regarding whether to support each post-processing filter can be generated at the unit level of sequences, images, slices, tiles, etc. Furthermore, setting information regarding whether to execute each post-processing filter can be generated at the unit level of images, slices, tiles, blocks, etc. The scope of filter execution can be categorized into the interior of the image and the boundaries of the image. Setting information considering this category can be generated. Additionally, information regarding filtering operations can be generated at the unit level of images, slices, tiles, blocks, etc. Information can be processed implicitly or explicitly, and independent or dependent filtering can be applied to the filter based on color components, which can be determined according to the encoding settings. The in-loop filter unit can deliver the filtering information to the encoding unit for encoding. The encoding unit adds the corresponding information to the bitstream and transmits the bitstream to the decoder. The decoding unit of the decoder can parse the information and apply the parsed information to the in-loop filter unit.

[0123] The memory can store recovered blocks or images. The recovered blocks or images stored in the memory can be provided to the prediction unit, which performs intra-frame prediction or inter-frame prediction. Specifically, for processing, the space storing the bitstream compressed by the encoder in a queue can be set as the Encoded Picture Buffer (CPB), and the space storing the decoded images in picture units can be set as the Decoded Picture Buffer (DPB). The CPB can store the decoding unit in decoding order, simulate the decoding operation in the encoder, and process the stored compressed bitstream through simulation. The bitstream output from the CPB is recovered through decoding, and the recovered image is stored in the DPB. The images stored in the DPB can be referenced during image encoding / decoding processing.

[0124] The decoding unit can be implemented by reversing the processing of the encoding unit. For example, the decoding unit can receive a quantization coefficient string, a transform coefficient string, or a signal string from the bit stream, decode the string, parse the decoded data including the decoding information, and deliver the parsed decoded data to each element.

[0125] Next, image setting processing applied to an image encoding / decoding apparatus according to an embodiment of the present invention will be described. This is an example applied before encoding / decoding (initial image setting), but some processing may be examples to be applied to other steps (e.g., steps after encoding / decoding or sub-steps of encoding / decoding). Image setting processing can be performed considering, for example, network and user environments such as multimedia content characteristics, bandwidth, user terminal performance, and accessibility. For example, image partitioning, image resizing, image reconstruction, etc., can be performed according to the encoding / decoding settings. The following description of image setting processing focuses on rectangular images. However, the invention is not limited thereto, and image setting processing can be applied to polygonal images. The same image settings can be applied regardless of the image form, or different image settings can be applied, which can be determined according to the encoding / decoding settings. For example, after examining information about the image shape (e.g., rectangular or non-rectangular shape), information about the corresponding image settings can be constructed.

[0126] The following examples will be described under the assumption that a dependent setting is provided for the color space. However, independent settings can be provided for the color space. Furthermore, in the following examples, independent settings may include providing encoding / decoding settings independently for each color space. Although one color space is described, it is contemplated that examples of applying this description to other color spaces (e.g., an example where M is generated in the luminance component and N is generated in the chrominance component), and such examples can be derived. Furthermore, dependent settings may include examples that are set proportionally to the color format composition ratio (e.g., 4:4:4, 4:2:2, 4:2:0, etc.) (e.g., for 4:2:0, the chrominance component is M / 2 when the luminance component is M). It is contemplated that examples of applying this description to each color space are included, and such examples can be derived. This description is not limited to the examples above and can be applied collectively to the present invention.

[0127] Some of the builds in the following examples can be applied to various coding techniques, such as spatial domain coding, frequency domain coding, block-based coding, object-based coding, etc.

[0128] Typically, the input image can be encoded or decoded either as is or after image segmentation. For example, segmentation can be performed to prevent corruption caused by packet loss during transmission, such as for error robustness. Alternatively, segmentation can be performed to classify regions with different properties within the same image based on image characteristics, type, etc.

[0129] According to the present invention, image segmentation processing may include segmentation processing and inverse segmentation processing. The following examples will focus on segmentation processing, but inverse segmentation processing can be derived from segmentation processing.

[0130] Figure 3 This is an example diagram of dividing image information into multiple layers to compress the image.

[0131] Part 3a is an example diagram in which the image sequence is composed of multiple GOPs. Furthermore, a GOP can consist of I-images, P-images, and B-images, as shown in Part 3b. An image can consist of slices, tiles, etc., as shown in Part 3c. As shown in Part 3d, slices, tiles, etc., can consist of multiple default coding portions, and as shown in Part 3e, a default coding portion can consist of at least one coding subunit. The image setup processing according to the invention will be described based on examples of units such as images, slices, and tiles as shown in Parts 3b and 3c.

[0132] Figure 4 This is a conceptual diagram illustrating an example of image segmentation according to an embodiment of the present invention.

[0133] Part 4a is a conceptual diagram in which an image (e.g., a picture) is divided at uniform intervals in both the horizontal and vertical directions. The divided regions can be called blocks. Each block can be a default encoded portion (or a maximum encoded portion) obtained through the image division and can be the basic unit to be applied to the division units described below.

[0134] Part 4b is a conceptual map in which the image is divided in at least one direction selected from the horizontal and vertical directions. The divided regions (T0 to T3) may be referred to as tiles, and each region may be encoded or decoded independently or in dependence on other regions.

[0135] Part 4c is a conceptual diagram in which an image is divided into groups of consecutive blocks. The divided regions (S0, S1) can be referred to as slices, and each region can be encoded or decoded independently or dependent on other regions. Groups of consecutive blocks can be defined according to the scanning order. Typically, groups of consecutive blocks conform to the raster scan order. However, the invention is not limited to this, and groups of consecutive blocks can be determined based on encoding / decoding settings.

[0136] Part 4D is a conceptual diagram of a group of blocks that divide an image into according to any user-defined settings. The division of regions (A0 to A2) can be called arbitrary division, and each region can be encoded or decoded independently or dependent on other regions.

[0137] Independent encoding / decoding means that when encoding or decoding some units (or regions), data in other units cannot be referenced. Specifically, multiple pieces of information used or generated during texture encoding and entropy encoding for some units can be encoded independently without referencing each other. Even within the decoder, for texture decoding and entropy decoding for some units, the parsed and reconstructed information in other units may not refer to each other. In this case, whether to reference data in other units (or regions) can be limited to spatial regions (e.g., between regions within an image), but can also be limited to temporal regions (e.g., between consecutive images or frames) depending on the encoding / decoding settings. For example, reference can be made when some units in the current image and some units in another image have continuity or the same encoding environment; otherwise, reference may be restricted.

[0138] Furthermore, dependency encoding / decoding can mean that when encoding or decoding some units, data in other units can be referenced. Specifically, multiple pieces of information used or generated during texture encoding and entropy encoding for some units can be dependently encoded and referenced to each other. Even in the decoder, for texture decoding and entropy decoding for some units, the parsed and recovered information in other units can also be referenced to each other. That is, the above setup can be the same as or similar to the setup of a general encoding / decoding. In this case, in order to identify regions (here, surfaces generated according to the projection format)... <face>(etc.) can divide regions based on the characteristics and type of the image (e.g., a 360-degree image).

[0139] In the above examples, independent encoding / decoding settings (e.g., independent slice segments) can be provided for some units (slices, tiles, etc.), while dependent encoding / decoding settings (e.g., dependent slice segments) can be provided for other units. According to the present invention, the following description will focus on independent encoding / decoding settings.

[0140] As shown in part 4a, the default encoding portion obtained through the image segmentation unit can be divided into default encoding blocks according to the color space, and can have a size and shape determined according to the characteristics and resolution of the image. The supported block size or shape can be a power of 2 (2^2 + 2^3 ... n A square of N×N width and height (2) n ×2 n ; 256×256, 128×128, 64×64, 32×32, 16×16, 8×8, etc.; n is an integer in the range of 3 to 8) or an M×N rectangle (2 m ×2 n For example, the input image can be divided according to resolution: an 8k UHD image can be divided into 128×128, a 1080p HD image into 64×64, or a WVGA image into 16×16. The input image can also be divided according to image type: a 360-degree image can be divided into 256×256. The default encoding section can be divided into encoding sub-units, which are then encoded or decoded. Information about the default encoding section can be added to the bitstream in units such as sequences, images, slices, and tiles, and can be parsed by the decoder to recover the relevant information.

[0141] The image encoding and image decoding methods according to embodiments of the present invention may include the following image segmentation steps. In this case, the image segmentation process may include an image segmentation indication step, an image segmentation type identification step, and an image segmentation execution step. Furthermore, the image encoding apparatus and the image decoding apparatus may be configured to include an image segmentation indication unit, an image segmentation type identification unit, and an image segmentation execution unit that respectively execute the image segmentation indication step, the image segmentation type identification step, and the image segmentation execution step. For encoding, relevant syntax elements may be generated. For decoding, relevant syntax elements may be parsed.

[0142] In the block partitioning process, as shown in section 4a, the image partitioning instruction unit can be omitted. The image partitioning type recognition unit can check information about the size and shape of the blocks, and the image partitioning unit can perform partitioning based on the partitioning type information recognized in the default encoding section.

[0143] A block can always be the unit to be divided, but whether to divide it into other units (tiles, slices, etc.) can be determined based on the encoding / decoding settings. By default, the image division department can perform division by blocks and then by other units. In this case, block division can be performed based on the image size.

[0144] Furthermore, partitioning can be performed on a block-by-block basis after partitioning by other units (tiles, slices, etc.). That is, block partitioning can be performed based on the size of the partitioning unit. This can be determined through explicit or implicit processing depending on the encoding / decoding settings. The following example describes the assumption of the former case, and will also focus on units other than blocks.

[0145] In the image segmentation indication step, it can be determined whether to perform image segmentation. For example, segmentation can be performed when a signal indicating image segmentation is confirmed (e.g., tiles_enabled_flag). If no signal indicating image segmentation is confirmed, segmentation can be skipped, or it can be performed by confirming other encoding / decoding information.

[0146] Specifically, assume a signal indicating image segmentation is confirmed (e.g., tiles_enabled_flag). When the signal is activated (e.g., tiles_enabled_flag=1), segmentation can be performed by multiple units. When the signal is deactivated (e.g., tiles_enabled_flag=0), segmentation can be omitted. Alternatively, the absence of a signal indicating image segmentation may indicate that segmentation is not performed or that segmentation is performed by at least one unit. Whether segmentation by multiple units is performed can be confirmed by another signal (e.g., first_slice_segment_in_pic_flag).

[0147] In summary, when a signal indicating image partitioning is provided, the corresponding signal is used to indicate whether partitioning is performed in multiple units. This signal can be used to determine whether to partition the corresponding image. For example, suppose `tiles_enabled_flag` is a signal indicating whether to partition the image. Here, `tiles_enabled_flag` equal to 1 indicates that the image is partitioned into multiple tiles, and `tiles_enabled_flag` equal to 0 indicates that the image is not partitioned.

[0148] In summary, when no signal indicating image segmentation is provided, segmentation may not be performed, or alternative signals may be used to determine whether to segment the image. For example, `first_slice_segment_in_pic_flag` is not a signal indicating whether image segmentation is performed, but rather a signal indicating the first slice segment in the image. Therefore, it can be determined whether to perform segmentation in two or more units (e.g., a flag of 0 indicates that the image is divided into multiple slices).

[0149] This invention is not limited to the above examples, and modifications can be made to them. For example, a signal indicating image segmentation may not be provided for each tile, but rather for each slice. Alternatively, the signal indicating image segmentation may be provided based on the image type, characteristics, etc.

[0150] In the image segmentation type recognition step, the image segmentation type can be identified. The image segmentation type can be defined by segmentation methods, segmentation information, etc.

[0151] In part 4b, a tile can be defined as a unit obtained by horizontal and vertical division. Specifically, a tile can be defined as a group of adjacent blocks in a quadrilateral space divided by at least one horizontal or vertical dividing line passing through the image.

[0152] Tile partitioning information can include column and row boundary position information, the number of tiles in each column and row, and tile size information. The tile number information can include the number of columns (e.g., num_tile_columns) and the number of rows (e.g., num_tile_rows) of a tile. Therefore, an image can be divided into a number of tiles (= number of columns × number of rows). Tile size information can be obtained based on the tile number information. The width or height of a tile can be uniform or non-uniform, so under predetermined rules, relevant information can be implicitly determined or explicitly generated (e.g., uniform_spacing_flag). Furthermore, tile size information can include the size information of each column and each row of the tile (e.g., column_width_tile[i] and row_height_tile[i]), or the width and height size information of each tile. Additionally, the size information can be additionally generated based on whether the tile size is uniform (e.g., in the case where the partitioning is non-uniform because uniform_spacing_flag is 0).

[0153] In part 4c, a slice can be defined as a unit that groups consecutive blocks. Specifically, a slice can be defined as a group of consecutive blocks in a predetermined scan order (here, raster scan).

[0154] Slice partitioning information may include the number of slices and slice location information (e.g., slice_segment_address). In this case, the slice location information may be the location information of a predetermined block (e.g., the first row in the slice according to the scan order). In this case, the location information may be the block scan order information.

[0155] In some 4D models, various partitioning settings are allowed for arbitrary partitioning.

[0156] In some 4D models, a partitioning unit can be defined as a group of spatially adjacent blocks, and information about the partitioning can include details about the size, shape, and location of the partitioning units. This is merely an example of an arbitrary partitioning, and as... Figure 5 As shown, it allows for various partitioning methods.

[0157] Figure 5 This is another example diagram of the image segmentation method according to an embodiment of the present invention.

[0158] In sections 5a and 5b, an image can be divided into multiple regions horizontally or vertically with at least one block interval, and the division can be performed based on block location information. Section 5a shows an example of performing horizontal division based on the row information of each block (A0, A1), and section 5b shows examples of performing horizontal and vertical division based on the column and row information of each block (B0 to B3). The information regarding the division may include the number of division units, block interval information, division direction, etc., and some division information may not be generated when it is implicitly included according to predetermined rules.

[0159] In sections 5c and 5d, the image can be divided into groups of consecutive blocks according to the scanning order. Additional scanning orders besides the traditional slicing raster scanning order can be applied to image partitioning. Section 5c shows examples of scanning clockwise or counterclockwise relative to the starting block (Box-Out) (C0, C1), and section 5d shows examples of scanning perpendicularly relative to the starting block (vertical) (D0, D1). Information about the partitioning can include information about the number of partition units, information about the position of the partition units (e.g., the first row in the partition units according to the scanning order), information about the scanning order, etc., and some partitioning information may not be generated when it is implicitly included according to predetermined rules.

[0160] In part 5e, the image can be divided using both horizontal and vertical dividing lines. Existing tiles can be divided using either horizontal or vertical dividing lines. Therefore, division can be performed in the form of a quadrilateral space, but it may not be possible to divide the image using dividing lines. For example, it is possible to divide the image using some dividing lines (e.g., the dividing line between the left boundary of E5 and the right boundaries of E1, E3, and E4), while it is not possible to divide the image using some dividing lines (e.g., the dividing line between the lower boundaries of E2 and E3 and the upper boundary of E4). Furthermore, division can be performed based on block units (e.g., after first performing block division), or it can be performed using either horizontal or vertical dividing lines (e.g., division using dividing lines, regardless of block division). Therefore, each dividing unit may not be a multiple of the number of blocks. Therefore, division information different from the division information of existing tiles can be generated, and the division information may include information about the number of dividing units, information about the location of the dividing units, information about the size of the dividing units, etc. For example, information about the position of the partitioning unit can be generated as positional information (e.g., measured in pixels or blocks) based on a predetermined position (e.g., at the top left corner of the image), and information about the size of the partitioning unit can be generated as information about the width and height of each partitioning unit (e.g., measured in pixels or blocks).

[0161] Similar to the examples above, partitioning can be performed based on any user-defined settings by applying new partitioning methods or by changing some elements of existing partitions. That is, partitioning methods can be supported by replacing or adding to traditional partitioning methods, and can be supported by changing some settings of traditional partitioning methods (slicing, tiles, etc.) (e.g., generating additional partitioning information by using a different partitioning method with quadrilateral shapes based on a different scan order, or based on dependent encoding / decoding characteristics). Furthermore, settings for configuring additional partitioning units can be supported (e.g., settings other than partitioning based on scan order or partitioning based on a specific interval difference), and additional partitioning unit forms can be supported (e.g., polygonal forms such as triangles in addition to partitioning into quadrilateral spaces). Moreover, image partitioning methods can be supported based on image type, characteristics, etc. For example, partial partitioning methods (e.g., faces of a 360-degree image) can be supported based on image type, characteristics, etc. Information about partitioning can be generated based on support.

[0162] In the image segmentation execution step, the image can be segmented based on the identified segmentation type information. That is, the image can be divided into multiple segmentation units based on the identified segmentation type, and the image can be encoded or decoded based on the acquired segmentation units.

[0163] In this case, the need for encoding / decoding settings in each partition unit can be determined based on the partition type. That is, the setting information required during encoding / decoding processing for each partition unit can be specified by the higher-level unit (e.g., an image), or independent encoding / decoding settings can be provided for each partition unit.

[0164] Typically, slices can have independent encoding / decoding settings for each segmentation unit (e.g., slice header), while tiles cannot have independent encoding / decoding settings for each segmentation unit and can have settings that depend on the image's encoding / decoding settings (e.g., PPS). In this case, the information generated in association with the tile can be segmentation information and can be included in the image's encoding / decoding settings. The invention is not limited to the above examples and modifications can be made to them.

[0165] Encoding / decoding settings for tiles can be generated at the unit level, such as video, sequences, and images. At least one encoding / decoding setting is generated in the higher-level unit, and this generated setting can be referenced. Alternatively, independent encoding / decoding settings (e.g., tile headers) can be generated at the tile level. This differs from following a single encoding / decoding setting determined in the higher-level unit in that encoding / decoding is performed while at least one encoding / decoding setting is provided at the tile level. That is, all tiles can be encoded or decoded according to the same encoding / decoding settings, or at least one tile can be encoded or decoded according to encoding / decoding settings different from those of other tiles.

[0166] The examples above focus on various encoding / decoding settings in tiles. However, the invention is not limited to this, and the same or similar settings can be applied to other partitioning types.

[0167] As an example, in some partitioning types, partitioning information can be generated in the higher-level unit, and encoding or decoding can be performed based on the individual encoding / decoding settings of the higher-level unit.

[0168] As an example, in some partitioning types, partitioning information can be generated in the superordinate unit, and independent encoding / decoding settings for each partitioning unit can be generated in the superordinate unit, and encoding or decoding can be performed according to the generated encoding / decoding settings.

[0169] As an example, in some partitioning types, partitioning information can be generated in the higher-level unit, and multiple encoding / decoding settings can be supported in the higher-level unit. Encoding or decoding can be performed based on the encoding / decoding settings referenced by each partitioning unit.

[0170] As an example, in some partitioning types, partitioning information can be generated in the higher-level unit, and independent encoding / decoding settings can be generated in the corresponding partitioning unit. Encoding or decoding can then be performed based on the generated encoding / decoding settings.

[0171] As an example, in some partitioning types, independent encoding / decoding settings including partitioning information can be generated in the corresponding partitioning unit, and encoding or decoding can be performed according to the generated encoding / decoding settings.

[0172] Encoding / decoding settings information can include information required for encoding or decoding tiles, such as tile type, information about the reference image list, quantization parameters, inter-frame prediction settings, in-loop filtering settings, in-loop filtering control information, scan order, and whether encoding or decoding is performed. Encoding / decoding settings information can be used to explicitly generate relevant information, or it can have encoding / decoding settings implicitly determined based on the image format, characteristics, etc., determined in the higher-level unit. Furthermore, relevant information can be explicitly generated based on information obtained through these settings.

[0173] Next, an example of performing image segmentation in an encoding / decoding apparatus according to an embodiment of the present invention will be described.

[0174] The input image can be partitioned before encoding begins. The image can be partitioned using partitioning information (e.g., image partitioning information, partitioning unit setting information, etc.), and then encoded within the partitioned units. The encoded image data can be stored in memory after encoding is complete, and can be added to a bitstream and then transmitted.

[0175] Partitioning can be performed before decoding begins. The image can be partitioned using partitioning information (e.g., image partitioning information, partitioning unit setting information, etc.), and then the image decoding data can be parsed and decoded within the partitioned units. After decoding is complete, the image decoding data can be stored in memory, and multiple partitioned units are merged into a single unit, thus allowing the image to be output.

[0176] The image segmentation process has been described through the examples above. Furthermore, according to the present invention, multiple segmentation processes can be performed.

[0177] For example, an image can be segmented, and the segmentation units of the image can be divided. The segmentation can be the same segmentation process (e.g., slice / slice, patch / patch, etc.) or different segmentation processes (e.g., slice / patch, patch / slice, patch / face, face / patch, slice / face, face / slice, etc.). In this case, subsequent segmentation processes can be performed based on the preceding segmentation results, and segmentation information generated during subsequent segmentation processes can be generated based on the preceding segmentation results.

[0178] Furthermore, multiple partitioning processes (A) can be performed, and these partitioning processes can be different (e.g., slice / face, tile / face, etc.). In this case, subsequent partitioning processes can be performed based on or independently of the preceding partitioning results, and partitioning information generated during subsequent partitioning processes can be generated based on or independently of the preceding partitioning results.

[0179] Multiple image segmentation processes can be determined based on encoding / decoding settings. However, the present invention is not limited to the above examples, and various modifications can be made to the above examples.

[0180] The encoder can add information generated during the above processing to the bitstream in units of at least one of sequences, images, slices, tiles, etc., and the decoder can parse the relevant information from the bitstream. That is, information can be added to one unit, and information can be copied and added to multiple units. For example, syntax elements indicating whether some information is supported or indicating whether active syntax elements are executed can be generated in some units (e.g., higher-level units), and the same or similar information can be generated in some units (e.g., lower-level units). That is, even if relevant information is supported and set in the higher-level unit, the lower-level unit can have separate settings. This description is not limited to the above examples and can be applied collectively to the present invention. Furthermore, information can be included in the bitstream in the form of SEI or metadata.

[0181] Typically, an input image can be encoded or decoded as is, but encoding or decoding can be performed after resizing the image (expanding or shrinking; resolution adjustment). For example, in a layered coding scheme (adaptive video coding) used to support spatial, temporal, and image quality adaptability, image resizing adjustments such as overall image expansion and reduction can be performed. Alternatively, image resizing adjustments such as partial image expansion and reduction can be performed. Image resizing can be performed in various ways, i.e., for the purpose of adapting to the coding environment, for coding uniformity, for coding efficiency, for image quality improvement, or according to the type and characteristics of the image.

[0182] As a first example, resizing can be performed during processing based on the characteristics, type, etc. of the image (e.g., layered encoding, 360-degree image encoding, etc.).

[0183] As a second example, resizing can be performed at the initial encoding / decoding step. Resizing can be performed before encoding or decoding. The resized image can then be encoded or decoded.

[0184] As a third example, resizing can be performed during or before the prediction step (intra-frame prediction or inter-frame prediction). During resizing, image information (e.g., information about the pixels referenced for intra-frame prediction, information about the intra-frame prediction mode, information about the reference image used for inter-frame prediction, information about the inter-frame prediction mode, etc.) can be used at the prediction step.

[0185] As a fourth example, resizing can be performed during or before the filtering step. In the resizing process, image information from the filtering step can be used (e.g., pixel information to be applied to the deblocking filter, pixel information to be applied to the SAO, information about the SAO filter, pixel information to be applied to the ALF filter, information about the ALF filter, etc.).

[0186] Furthermore, after performing resizing, the image can be processed by inverse resizing, changing it back to its original size (in terms of image dimensions), or the image can remain unchanged. This can be determined based on the encoding / decoding settings (e.g., the characteristics of the resizing process). In this case, the resizing process can be an expansion process and the inverse resizing process can be a reduction process, and vice versa.

[0187] When the resizing process is performed according to the first to fourth examples, the inverse resizing process is performed in subsequent steps so that the image before the resizing can be obtained.

[0188] When resizing is performed via layered encoding or based on a third example (or when resizing a reference image in inter-frame prediction), inverse resizing may not be performed in subsequent steps.

[0189] In embodiments of the present invention, the image resizing process can be performed alone or together with the inverse process. The following examples will focus on the resizing process. In this case, since the inverse resizing process is the inverse of the resizing process, a description of the inverse resizing process will be omitted to prevent redundancy. However, it will be apparent to those skilled in the art that the content is identical to what is literally described.

[0190] Figure 6 This is an example diagram of a common image resizing method.

[0191] Referring to section 6a, an extended image (P0+P1) can be obtained by adding a specific region (P1) to the initial image P0 (or the image before resizing; indicated by a thick solid line).

[0192] Referring to section 6b, a reduced image (S0) can be obtained by removing a specific region (S1) from the initial image (S0+S1).

[0193] Referring to section 6c, a resized image (T0+T1) can be obtained by adding a specific region (T1) to the initial image (T0+T2) and removing the specific region (T2) from the entire image.

[0194] According to the present invention, the following description focuses on the sizing adjustment process for expansion and the sizing adjustment process for reduction. However, the present invention is not limited thereto and should be understood to include cases in which expansion and reduction are applied in combination, as shown in section 6c.

[0195] Figure 7 This is an example diagram of image size adjustment according to an embodiment of the present invention.

[0196] During the resizing process, an image expansion method will be described with reference to section 7a, and an image reduction method will be described with reference to section 7b.

[0197] In section 7a, the image before resizing is S0, and the image after resizing is S1. In section 7b, the image before resizing is T0, and the image after resizing is T1.

[0198] When expanding an image as shown in section 7a, the image can be expanded in the "up", "down", "left", or "right" direction (ET, EL, EB, ER). When shrinking an image as shown in section 7b, the image can be shrunk in the "up", "down", "left", or "right" direction (RT, RL, RB, RR).

[0199] Comparing image expansion and image reduction, the "up," "down," "left," and "right" directions of expansion can correspond to the "down," "up," "right," and "left" directions of reduction. Therefore, the following description focuses on image expansion, but it should be understood that a description of image reduction is included.

[0200] In the following description, image scaling is performed in the "up", "down", "left", and "right" directions. However, it should also be understood that resizing can be performed in the "up and left", "up and right", "down and left", or "down and right" directions.

[0201] In this scenario, when expansion is performed in the "down and right" directions, regions RC and BC are acquired, and region BR may or may not be acquired depending on the encoding / decoding settings. That is, regions TL, TR, BL, and BR may or may not be acquired, but for ease of description, corner regions (i.e., regions TL, TR, BL, and BR) will be described as being able to be acquired.

[0202] Image resizing processing according to embodiments of the present invention can be performed in at least one direction. For example, image resizing processing can be performed in all directions such as up, down, left, and right; it can be performed in two or more directions selected from up, down, left, and right (left + right, up + down, up + left, up + right, down + left, down + right, up + left + right, down + left + right, up + down + left, up + down + right, etc.); or it can be performed only in one direction selected from up, down, left, and right.

[0203] For example, resizing can be performed in the "left + right" direction, the "up + down" direction, the "left and up + right and down" direction, and the "left and down + right and up" direction, which can be symmetrically extended to both ends relative to the center of the image. Resizing can also be performed in the "left + right" direction, the "left and up + right and up" direction, and the "left and down + right and down" direction, which can be vertically symmetrically extended relative to the image. Furthermore, resizing can be performed in the "up + down" direction, the "left and up + left and down" direction, and the "right and up + right and down" direction, which can be horizontally symmetrically extended relative to the image. Other resizing adjustments can also be performed.

[0204] In sections 7a and 7b, the dimensions of the image (S0, T0) before resizing are defined as P_Width × P_Height, and the dimensions of the image (S1, T1) after resizing are defined as P'_Width × P'_Height. Here, when the resizing values ​​in the "left", "right", "up", and "down" directions are defined as Var_L, Var_R, Var_T, and Var_B (or collectively referred to as Var_x), the dimensions of the resizing image can be expressed as (P_Width + Var_L + Var_R) × (P_Height + Var_T + Var_B). In this case, Var_L, Var_R, Var_T, and Var_B, which are the size adjustment values ​​in the "left", "right", "up", and "down" directions, can be Exp_L, Exp_R, Exp_T, and Exp_B for image expansion (in part 7a) (where Exp_x is positive), and can be -Rec_L, -Rec_R, -Rec_T, and -Rec_B for image reduction (which is represented as a negative value for image reduction when Rec_L, Rec_R, Rec_T, and Rec_B are defined as positive values). Furthermore, the coordinates of the top-left, top-right, bottom-left, and bottom-right corners of the image before resizing can be (0, 0), (P_Width-1, 0), (0, P_Height-1), and (P_Width-1, P_Height-1), respectively, and the coordinates of the top-left, top-right, bottom-left, and bottom-right corners of the image after resizing can be represented as (0, 0), (P'_Width-1, 0), (0, P'_Height-1), and (P'_Width-1, P'_Height-1). The size of the region changed (or acquired or removed) by resizing (here, TL to BR; i is the index used to identify TL to BR) can be M[i] × N[i] and can be represented as Var_X × Var_Y (this example assumes X is L or R, and Y is T or B). M and N can have various values ​​and can have the same settings regardless of i, or they can have separate settings depending on i. Various examples are described below.

[0205] Referring to section 7a, S1 can be configured to include some or all of the regions TL to BR (upper left to lower right) generated by extending S0 in several directions. Referring to section 7b, T1 can be configured to exclude all or some of the regions TL to BR that will be removed by shrinking in several directions from T0.

[0206] In part 7a, when the existing image (S0) expands in the "up", "down", "left" and "right" directions, the image may include regions TC, BC, LC and RC obtained through size adjustment processing and may further include regions TL, TR, BL and BR.

[0207] As an example, when performing an extension in the "up" direction (ET), an image can be constructed by adding a region TC to an existing image (S0), and the image can include a region TL or TR as well as extensions in at least one different direction (EL or ER).

[0208] As an example, when performing an extension in the "downward" direction (EB), an image can be constructed by adding a region BC to an existing image (S0), and the image can include a region BL or BR and an extension in at least one different direction (EL or ER).

[0209] As an example, when performing an extension in the "left" direction (EL), an image can be constructed by adding a region LC to an existing image (S0), and the image can include a region TL or BL and an extension in at least one different direction (ET or EB).

[0210] As an example, when performing an extension in the "right" direction (ER), an image can be constructed by adding a region RC to an existing image (S0), and the image can include a region TR or BR as well as extensions in at least one different direction (ET or EB).

[0211] According to embodiments of the invention, reference-worthy settings (e.g., spa_ref_enabled_flag or tem_ref_enabled_flag) can be provided for spatially or temporally limiting a sized region (this example assumes expansion).

[0212] That is, it is permissible (e.g., spa_ref_enabled_flag=1 or tem_ref_enabled_flag=1) or restricted (e.g., spa_ref_enabled_flag=0 or tem_ref_enabled_flag=0) to reference data in regions whose size is adjusted spatially or temporally according to encoding / decoding settings.

[0213] Encoding / decoding of the image before resizing (S0, T1) and regions added or deleted during resizing (TC, BC, LC, RC, TL, TR, BL, and BR) can be performed as follows.

[0214] For example, when encoding or decoding an image before resizing and regions that have been added or removed, data about the image before resizing and data about the regions that have been added or removed (data after encoding or decoding; pixel values ​​or prediction-related information) can be referenced to each other spatially or temporally.

[0215] The alternative site can spatially reference the image before the resizing and data about the added or deleted areas, and temporally reference data about the image before the resizing, but cannot temporally reference data about the added or deleted areas.

[0216] That is, it can provide settings for restricting the referrability of areas being added or deleted. The settings information regarding the referrability of areas being added or deleted can be generated explicitly or determined implicitly.

[0217] The image resizing process according to embodiments of the present invention may include an image resizing instruction step, an image resizing type identification step, and / or an image resizing execution step. Furthermore, the image encoding apparatus and the image decoding apparatus may include an image resizing instruction unit, an image resizing type identification unit, and an image resizing execution unit, respectively configured to perform the image resizing instruction step, the image resizing type identification step, and the image resizing execution step. For encoding, relevant syntax elements may be generated. For decoding, relevant syntax elements may be parsed.

[0218] In the image resizing instruction step, it can be determined whether to perform image resizing. For example, resizing can be performed when a signal indicating image resizing (e.g., `img_resizing_enabled_flag`) is confirmed. If no signal indicating image resizing is confirmed, resizing can be omitted, or it can be performed by confirming other encoding / decoding information. Furthermore, even if no signal indicating image resizing is provided, it can be implicitly activated or deactivated based on encoding / decoding settings (e.g., image characteristics, type, etc.). When resizing is performed, corresponding resizing-related information can be generated, or it can be implicitly determined.

[0219] When a signal indicating image resizing is provided, the corresponding signal indicates whether image resizing should be performed. This signal can be used to determine whether to resize the corresponding image.

[0220] For example, suppose a signal indicating image resizing is confirmed (e.g., img_resizing_enabled_flag). When the corresponding signal is activated (e.g., img_resizing_enabled_flag=1), image resizing can be performed. When the corresponding signal is deactivated (e.g., img_resizing_enabled_flag=0), image resizing can be prevented.

[0221] In addition, if no signal indicating image resizing is provided, resizing may not be performed, or alternative signals may be used to determine whether to resize the corresponding image.

[0222] For example, when dividing an input image into blocks, resizing can be performed based on whether the image size (e.g., width or height) is a multiple of the block size (e.g., width or height). (For an extension of this example, it is assumed that resizing is performed when the image size is not a multiple of the block size.) That is, resizing can be performed when the image width is not a multiple of the block width or when the image height is not a multiple of the block height. In this case, resizing information (e.g., resizing direction, resizing value, etc.) can be determined based on encoding / decoding information (e.g., image size, block size, etc.). Alternatively, resizing can be performed based on image characteristics, type (e.g., 360-degree image), etc., and resizing information can be explicitly generated or specified as a predetermined value. The present invention is not limited to the above examples and modifications can be made to them.

[0223] In the image resizing type identification step, the image resizing type can be identified. The image resizing type can be defined by resizing methods, resizing information, etc. For example, resizing based on a scaling factor, resizing based on an offset factor, etc., can be performed. This invention is not limited to the above examples, and these methods can be combined. For ease of description, the following description will focus on resizing based on a scaling factor and resizing based on an offset factor.

[0224] For scaling factors, resizing can be performed via multiplication or division based on the image size. Information about the resizing operation (e.g., expanding or shrinking) can be explicitly generated, and the expanding or shrinking process can be performed based on the corresponding information. Alternatively, the resizing process can be performed as a pre-defined operation (e.g., one of expanding and shrinking operations) based on encoding / decoding settings. In this case, information about the resizing operation is omitted. For example, when image resizing is activated in an image resizing instruction step, the image resizing can be performed as a pre-defined operation.

[0225] The resizing direction can be selected from at least one of the following: up, down, left, and right. Depending on the resizing direction, at least one scaling factor may be required. That is, a scaling factor may be required for each direction (unidirectional in this case), a scaling factor may be required for the horizontal or vertical direction (bidirectional in this case), and a scaling factor may be required for all directions of the image (omnidirectional in this case). Furthermore, the resizing direction is not limited to the examples above and can be modified.

[0226] Scale factors can have positive values ​​and can have range information that varies depending on the encoding / decoding settings. For example, when generating information by combining a sizing operation and a scale factor, the scale factor can be used as a multiplicand. A scale factor greater than 0 or less than 1 can indicate a shrinking operation, a scale factor greater than 1 can indicate an expanding operation, and a scale factor of 1 can indicate that no sizing is performed. As another example, when generating scale factor information independently of sizing operations, the scale factor used for expanding operations can be used as a multiplicand, and the scale factor used for shrinking operations can be used as a divisor.

[0227] Will refer again Figure 7 Sections 7a and 7b describe the process of changing the images (S0, T0) before resizing into images (S1 and T1) after resizing.

[0228] As an example, when a scaling factor (called sc) is used in all directions of the image and the resizing direction is "down + right", the resizing directions are ER and EB (or RR and RB), the resizing values ​​Var_L (Exp_L or Rec_L) and Var_T (Exp_T or Rec_T) are 0, and Var_R (Exp_R or Rec_R) and Var_B (Exp_B or Rec_B) can be represented as P_Width×(sc-1) and P_Height×(sc-1). Therefore, the resized image can be (P_Width×sc)×(P_Height×sc).

[0229] As an example, when using the corresponding scaling factors (sc_w and sc_h in this case) in the horizontal or vertical direction of an image, and the resizing direction is "left + right" and "up + down" (or up + down + left + right when both are operated on), the resizing direction can be ET, EB, EL, and ER. The resizing values ​​Var_T and Var_B can be P_Height × (sc_h - 1) / 2, and Var_L and Var_R can be P_Width × (sc_w - 1) / 2. Therefore, the resized image can be (P_Width × sc_w) × (P_Height × sc_h).

[0230] For the offset factor, resizing can be performed by addition or subtraction based on the image size. Alternatively, resizing can be performed by addition or subtraction based on the image's encoding / decoding information. Alternatively, resizing can be performed by independent addition or subtraction. That is, the resizing process can have dependent or independent settings.

[0231] Information about resizing operations (e.g., expanding or shrinking) can be explicitly generated, and the expanding or shrinking process can be performed based on the corresponding information. Furthermore, the resizing operation can be performed as a pre-defined operation (e.g., one of expanding and shrinking operations) based on encoding / decoding settings. In this case, information about the resizing operation can be omitted. For example, when image resizing is activated in an image resizing instruction step, the image resizing can be performed as a pre-defined operation.

[0232] The resizing direction can be selected from at least one of the following: up, down, left, and right. Depending on the resizing direction, at least one offset factor may be required. That is, one offset factor may be required for each direction (unidirectional in this case), one offset factor may be required for the horizontal or vertical direction (symmetric bidirectional in this case), one offset factor may be required based on a partial combination of directions (asymmetric bidirectional in this case), and one offset factor may be required for all directions of the image (omnidirectional in this case). Furthermore, the resizing direction is not limited to the examples above and can be modified from the examples above.

[0233] The offset factor can have a positive value or both positive and negative values, and can have range information that varies depending on the encoding / decoding settings. For example, when combining a sizing operation with offset factor generation information (here, assuming the offset factor has both positive and negative values), the offset factor can be used as a value to be added to or subtracted based on the sign information of the offset factor. An offset factor greater than 0 can mean an expansion operation, an offset factor less than 0 can mean a shrinking operation, and an offset factor of 0 can mean no sizing operation is performed. As another example, when offset factor information is generated independently of the sizing operation (here, assuming the offset factor has a positive value), the offset factor can be used as a value to be added to or subtracted based on the sizing operation. An offset factor greater than 0 can mean that an expansion or shrinking operation can be performed based on the sizing operation, and an offset factor of 0 can mean no sizing operation is performed.

[0234] Will refer again Figure 7 Sections 7a and 7b describe a method for using an offset factor to transform an image (S0, T0) before resizing into an image (S1, T1) after resizing.

[0235] As an example, when an offset factor (called os) is used in all directions of the image and the resizing direction is "up + down + left + right", the resizing direction can be ET, EB, EL, and ER (or RT, RB, RL, and RR), and the resizing values ​​Var_T, Var_B, Var_L, and Var_R can be os. The size of the resized image can be (P_Width + os) × (P_Height + os).

[0236] As an example, when using offset factors (os_w, os_h) in the horizontal or vertical direction of an image and the resizing direction is "left + right" and "up + down" (or "up + down + left + right" when both are operated on), the resizing direction can be ET, EB, EL, and ER (or RT, RB, RL, and RR), the resizing values ​​Var_T and Var_B can be os_h, and the resizing values ​​Var_L and Var_R can be os_w. The size of the resized image can be {P_Width + (os_w × 2)} × {P_Height + (os_h × 2)}.

[0237] As an example, when the resizing direction is "down" and "right" (or "down + right" when operated together) and offset factors (os_b, os_r) are used based on the resizing direction, the resizing direction can be EB and ER (or RB and RR), the resizing value Var_B can be os_b, and the resizing value Var_R can be os_r. The size of the resized image can be (P_Width + os_r) × (P_Height + os_b).

[0238] As an example, when using offset factors (os_t, os_b, os_l, os_r) based on the image orientation and the resizing direction is "up", "down", "left", or "right" (or "up + down + left + right" for all operations), the resizing direction can be ET, EB, EL, and ER (or RT, RB, RL, and RR). The resizing value Var_T can be os_t, Var_B can be os_b, Var_L can be os_l, and Var_R can be os_r. The size of the resized image can be (P_Width + os_l + os_r) × (P_Height + os_t + os_b).

[0239] The examples above illustrate cases where offset factors are used as size adjustment values ​​(Var_T, Var_B, Var_L, Var_R) during resizing processing. That is, this means the offset factors are used as size adjustment values ​​without any changes, which could be an example of a resizing performed independently. Alternatively, the offset factors can be used as input variables for size adjustment values. Specifically, the offset factors can be specified as input variables, and the size adjustment values ​​can be obtained through a series of processes based on encoding / decoding settings. This could be an example of a resizing performed based on predetermined information (e.g., image size, encoding / decoding information, etc.) or an example of a resizing performed dependently.

[0240] For example, the offset factor can be a multiple (e.g., 1, 2, 4, 6, 8, and 16) or an exponential multiple (e.g., exponential multiples of 2, such as 1, 2, 4, 8, 16, 32, 64, 128, and 256). Alternatively, the offset factor can be a multiple or an exponential multiple of a value obtained based on encoding / decoding settings (e.g., a value based on motion search range settings for inter-frame prediction). Alternatively, the offset factor can be a multiple or an integer of units obtained from the image partitioning (here, assuming A×B). Alternatively, the offset factor can be a multiple of units obtained from the image partitioning (here, assuming E×F, such as tiles).

[0241] Alternatively, the offset factor can be a value less than or equal to the width and height of the unit obtained from the image division. In the example above, the multiple or exponential multiple can have a value of 1. However, the invention is not limited to the above example and can be modified. For example, when the offset factor is n, Var_x can be 2×n or 2 n .

[0242] Furthermore, individual offset factors can be supported based on color components. Offset factors for some color components can be supported, and thus offset factor information for other color components can be derived. For example, when an offset factor (A) for the luminance component is explicitly generated (here, assuming the luminance component to chrominance component composition ratio is 2:1), an offset factor (A / 2) for the chrominance component can be implicitly obtained. Alternatively, when an offset factor (A) for the chrominance component is explicitly generated, an offset factor (2A) for the luminance component can be implicitly obtained.

[0243] Information regarding the resizing direction and resizing value can be explicitly generated, and resizing processing can be performed based on the corresponding information. Alternatively, information can be implicitly determined based on encoding / decoding settings, and resizing processing can be performed based on the determined information. At least one predetermined direction or resizing value can be assigned, and in this case, related information can be omitted. In this case, encoding / decoding settings can be determined based on image characteristics, type, encoding information, etc. For example, at least one resizing direction can be predetermined based on at least one resizing operation, at least one resizing value can be predetermined based on at least one resizing operation, and at least one resizing value can be predetermined based on at least one resizing direction. Furthermore, the resizing direction, resizing value, etc., applied during the resizing process can be derived from the resizing direction, resizing value, etc., applied during the resizing process. In this case, the implicitly determined resizing value can be one of the above examples (examples of obtaining resizing values ​​differently).

[0244] Furthermore, while multiplication or division has been described in the examples above, shift operations can be used depending on the encoder / decoder implementation. Multiplication can be implemented using a left shift, and division can be implemented using a right shift. This description is not limited to the examples above and can be applied collectively to this invention.

[0245] In the image resizing step, image resizing can be performed based on the identified resizing information. That is, image resizing can be performed based on information such as the resizing type, resizing operation, resizing direction, and resizing value, and encoding / decoding can be performed based on the acquired resized image.

[0246] Furthermore, in the image resizing step, at least one data processing method can be used to perform the resizing. Specifically, the resizing can be performed on the area to be resized based on the resizing type and the resizing operation, using at least one data processing method. For example, depending on the resizing type, it can be determined how to fill data when the resizing is used for expansion, and how to remove data when the resizing is used for shrinking.

[0247] In summary, in the image resizing execution step, image resizing can be performed based on the identified resizing information. Alternatively, in the image resizing execution step, image resizing can be performed based on resizing information and a data processing method. The difference between the two cases lies in whether only the size of the image to be encoded or decoded is adjusted, or whether data processing of both the image size and the area to be resized is considered. In the image resizing execution step, whether to perform a data processing method can be determined based on the step, location, etc., where the resizing processing is applied. The following description focuses on examples of resizing based on a data processing method, but the invention is not limited thereto.

[0248] When performing offset-based sizing, various methods can be used to perform sizing adjustments for expansion and reduction. For expansion, at least one data padding method can be used to perform the sizing adjustment. For reduction, at least one data removal method can be used to perform the sizing adjustment. In this case, when performing offset-based sizing, the sizing area can be directly or after modification filled with new data or original image data (expansion), and the sizing area can be simply or through a series of processes removed (reduction).

[0249] When performing scaling factor-based resizing, in some cases (e.g., hierarchical encoding), scaling for expansion can be performed by applying upsampling, and scaling for reduction can be performed by applying downsampling. For example, at least one upsampling filter can be used for expansion, and at least one downsampling filter can be used for reduction. The horizontally applied filter can be the same as or different from the vertically applied filter. In this case, when performing scaling factor-based resizing, neither new data is generated in the resized region nor is new data removed from the resized region, but methods such as interpolation can be used to rearrange the original image data. Data processing methods associated with resizing can be categorized according to the filters used for sampling. Furthermore, in some cases (e.g., similar to the case of offset factors), scaling for expansion can be performed by padding at least one data point, and scaling for reduction can be performed by removing at least one data point. According to the invention, the following description focuses on data processing methods corresponding to the case of performing scaling factor-based resizing.

[0250] Typically, a predetermined data processing method can be used in the area to be resized, but at least one data processing method can also be used in the area to be resized, as shown in the following example. Selection information for the data processing method can be generated. The former could mean performing the sizing using a fixed data processing method, while the latter could mean performing the sizing using an adaptive data processing method.

[0251] Furthermore, the data processing method can be applied to all regions (TL, TC, TR, ..., BR in parts 7a and 7b) or some regions (e.g., each or a combination of TL to BR in parts 7a and 7b) that are to be added or deleted during sizing.

[0252] Figure 8 This is an example diagram illustrating a method for constructing an expanded region in an image resizing method according to an embodiment of the present invention.

[0253] Referring to section 8a, for ease of description, the image can be divided into regions TL, TC, TR, LC, C, RC, BL, BC, and BR corresponding to the upper left, upper, upper right, left, center, right, lower left, lower, and lower right positions of the image. In the following description, the image extends in the "downward + rightward" direction, but it should be understood that this description can be applied to other directions of extension.

[0254] Various methods can be used to construct regions that are added based on the expansion of the image. For example, the region can be filled with arbitrary values, or it can be filled with some data from the image.

[0255] Referring to section 8b, the generated regions (A0, A2) can be filled with arbitrary pixel values. Various methods can be used to determine these arbitrary pixel values.

[0256] As an example, any pixel value can be a pixel within a range of pixel values ​​that can be represented using bit depth (e.g., from 0 to 1 << (bit_depth) - 1). For example, any pixel value can be the minimum, maximum, median (e.g., 1 << (bit_depth - 1) etc.) of the range of pixel values ​​(where bit_depth indicates the bit depth).

[0257] As an example, any pixel value can be a pixel within a range of pixel values ​​that belong to the image's pixels (e.g., from min...). P to max P min P and max P Indicates the minimum and maximum values ​​of pixels belonging to the image; min P Greater than or equal to 0; max P Less than or equal to 1 << (bit_depth) - 1). For example, any pixel value can be the minimum, maximum, median, average (of at least two pixels), weighted sum, etc., within the range of pixel values.

[0258] As an example, any pixel value can be a value determined within a range of pixel values ​​belonging to a specific region included in the image. For example, when constructing A0, the specific region could be TR+RC+BR. Furthermore, the specific region can be set to a region corresponding to 3×9 of TR, RC, and BR, or a region corresponding to 1×9 (which is assumed to be the rightmost line). This can depend on the encoding / decoding settings. In this case, the specific region can be a unit to be divided by the image partitioning unit. Specifically, any pixel value can be the minimum, maximum, median, average (of at least two pixels), weighted sum, etc., within the range of pixel values.

[0259] Referring again to section 8b, region A1, to be added as the image expands, can be filled with pattern information generated using multiple pixel values ​​(e.g., assuming the pattern uses multiple pixels; no specific rules need to be followed). In this case, the pattern information can be defined based on encoding / decoding settings, or relevant information can be generated. The generated region can be filled with at least one piece of pattern information.

[0260] Referring to section 8c, regions added as the image expands can be constructed by referencing pixels in a specific region included in the image. Specifically, the added region can be constructed by copying or filling pixels in a region adjacent to the added region (hereinafter referred to as reference pixels). In this case, pixels in the region adjacent to the added region can be pixels before encoding or pixels after encoding (or decoding). For example, when resizing is performed in the pre-encoding step, the reference pixel can refer to pixels in the input image, and when resizing is performed in the intra-frame prediction reference pixel generation step, reference image generation step, filtering step, etc., the reference pixel can refer to pixels in the recovered image. In this example, it is assumed that the nearest neighbor pixel is used in the added region, but the invention is not limited thereto.

[0261] When an image is expanded left or right in association with a horizontal image resizing, the generated region (A0) can be constructed by horizontally filling (Z0) the outer pixels adjacent to the generated region (A0). Similarly, when an image is expanded upward or downward in association with a vertical image resizing, the generated region (A1) can be constructed by vertically filling (Z1) the outer pixels adjacent to the generated region (A1). Furthermore, when an image is expanded downward and to the right, the generated region (A2) can be constructed by diagonally filling (Z2) the outer pixels adjacent to the generated region (A2).

[0262] Referring to part 8d, the generated region (B'0 to B'2) can be constructed by referencing data from specific regions (B0 to B2) included in the image. Unlike part 8c, in part 8d, regions that are not adjacent to the generated region can be referenced.

[0263] For example, when a region in the image is highly correlated with the generated region, the generated region can be filled by referencing the pixels of the highly correlated region. In this case, the location information, size information, etc., of the highly correlated region can be generated. Alternatively, when a highly correlated region exists through encoding / decoding information such as the image's characteristics and type, and the location information, size information, etc., of the highly correlated region can be implicitly checked (e.g., for a 360-degree image), the generated region can be filled with data from the corresponding region. In this case, the location information, size information, etc., of the corresponding region can be omitted.

[0264] As an example, the pixels in region (B2) can be used to fill region (B'2) generated when the image is expanded to the left or right in association with a horizontal image resizing, with region (B2) being relative to the region generated when the image is expanded to the left or right in association with a horizontal image resizing.

[0265] As an example, the pixels in region (B1) can be used to fill region (B'1) generated when the image is expanded upward or downward in association with portrait image resizing, with region (B1) being relative to the region generated when the image is expanded upward or downward in association with portrait image resizing.

[0266] As an example, the pixels in the region (B0, TL) can be used to fill the region (B'0) generated when the image is expanded by some image size adjustment (here, diagonally relative to the center of the image), with the region (B0, TL) being relative to the generated region.

[0267] An example has been described where continuity exists at the boundary between the two ends of an image and data is acquired from a region symmetrical about the scaling direction. However, the invention is not limited to this and data from other regions (TL to BR) can be acquired.

[0268] When filling a generated region using data from a specific region of an image, the data of the corresponding region can be copied as is and used to fill the generated region, or the data of the corresponding region can be transformed based on the characteristics, type, etc. of the image and used to fill the generated region. In this case, copying the data as is may mean using the pixel values ​​of the corresponding region without any changes, and performing transformation processing may mean not using the pixel values ​​of the corresponding region without any changes. That is, at least one pixel value of the corresponding region can be changed through transformation processing. The generated region can be filled with the changed pixel values, or at least one of the positions of some pixels can be different from the other positions. That is, to fill a generated region of A×B, C×D data other than A×B data of the corresponding region can be used. In other words, at least one of the motion vectors applied to the pixels used to fill the generated region can be different from the other pixels. In the example above, when a 360-degree image is composed of multiple faces according to a projection format, the generated region can be filled with data from the other faces. The data processing method used to fill the region generated when the image is expanded by image resizing is not limited to the above example. The data processing method can be improved or changed, or additional data processing methods can be used.

[0269] Multiple candidate groups for data processing methods can be supported based on encoding / decoding settings, and information about selecting a data processing method from multiple candidate groups can be generated and added to the bitstream. For example, a data processing method can be selected from methods such as padding using predetermined pixel values, padding by copying external pixels, padding by copying a specific region of the image, and padding by transforming a specific region of the image, and related selection information can be generated. Furthermore, the data processing method can be implicitly determined.

[0270] For example, the data processing method applied to all regions to be generated as an extension through image size adjustment (here, regions TL to BR in part 7a) could be one of the following: a filling method using predetermined pixel values, a filling method by copying external pixels, a filling method by copying a specific region of the image, or a filling method by transforming a specific region of the image, and related selection information could be generated. Furthermore, a predetermined data processing method applied to the entire region can be determined.

[0271] Alternative locations are applied to the regions to be expanded as the image size is adjusted (here, Figure 7 The data processing method for each or two or more regions (TL to BR) in part 7a can be one of the following: a filling method using predetermined pixel values, a filling method by copying external pixels, a filling method by copying a specific region of the image, or a filling method by transforming a specific region of the image, and related selection information can be generated. Furthermore, a predetermined data processing method can be determined for application to at least one region.

[0272] Figure 9 This is an example diagram illustrating a method for constructing the region to be deleted by shrinking and the region to be generated in an image resizing method according to an embodiment of the present invention.

[0273] The regions to be removed during image downsizing can not only be removed simply, but can also be removed after a series of applied processing steps.

[0274] Referring to section 9a, during image downsizing, specific regions (A0, A1, A2) can be easily removed without additional processing. In this case, the image (A) can be divided into regions TL to BR as shown in section 8a.

[0275] Referring to section 9b, regions (A0 to A2) can be removed and used as reference information when encoding or decoding the image (A). For example, the removed regions (A0 to A2) can be utilized during the process of restoring or correcting specific regions of the image (A) that have been deleted by reducing its size. During the restoration or correction process, a weighted sum, average, etc., of the two regions (the deleted region and the generated region) can be used. Furthermore, the restoration or correction process can be applied when the two regions have a high correlation.

[0276] As an example, the region (B'2) deleted when the image is scaled down to the left or right in association with a horizontal image resizing can be used to restore or correct pixels in the region (B2, LC) opposite to the region deleted when the image is scaled down to the left or right in association with a horizontal image resizing, and that region (B'2) can then be removed from memory.

[0277] As an example, the region (B'1) deleted when the image is scaled up or down in association with portrait image resizing can be used for encoding / decoding (restoration or correction) of the region (B1, TR) opposite to the region deleted when the image is scaled up or down in association with portrait image resizing, and the region (B'1) can then be removed from memory.

[0278] As an example, when an image is reduced in size by some image resizing (here, diagonally relative to the center of the image), the deleted region (B'0) can be used for encoding / decoding (restoration or correction) of the region (B0, TL) opposite to the deleted region, and then the region (B'0) can be removed from memory.

[0279] Examples have been described of data from regions that are continuous at the boundaries between the two ends of an image and symmetrical about the direction of size adjustment, which were used for restoration or correction. However, the invention is not limited thereto, and data from regions TL to BR other than the symmetrical regions can be used for restoration or correction and can then be removed from memory.

[0280] The data processing methods used to remove areas to be deleted are not limited to the examples above. The data processing methods can be improved or changed, or additional data processing methods can be used.

[0281] Multiple candidate groups for data processing methods can be supported based on encoding / decoding settings, and relevant selection information can be generated and added to the bitstream. For example, a data processing method can be selected from methods such as simply removing the region to be deleted, or removing the region after using it in a series of processes, and relevant selection information can be generated. Furthermore, the data processing method can be implicitly determined.

[0282] For example, apply to all areas that are to be deleted as the image size is adjusted (here, Figure 7 The data processing method for regions TL to BR in part 7b can be one of the following: simply removing the region to be deleted, or removing the region after using it in a series of processes, and related selection information can be generated. Furthermore, the data processing method can be implicitly determined.

[0283] Alternative locations are applied to each region to be deleted as it is scaled down by the image size adjustment (here, Figure 7 The data processing method for each of regions TL to BR in part 7b can be one of the following: simply removing the region to be deleted, or removing the region after using it in a series of processes, and related selection information can be generated. Furthermore, the data processing method can be implicitly determined.

[0284] An example of performing a size adjustment based on a size adjustment (expand or shrink) operation has been described. In some cases, this description can be applied to examples where a size adjustment operation (expanding in this case) is performed and then the reverse size adjustment operation (shrinking in this case) is performed.

[0285] For example, one could choose a method to fill the region generated with the expansion using some data from the image, and then choose a method to remove the region after it was deleted during the shrinking process in the process of restoring or correcting some data from the image. Alternatively, one could choose a method to fill the region generated with the expansion by copying outer pixels, and then choose a method to simply remove the region to be deleted during the shrinking process in the reverse processing. That is, the data processing method in the reverse processing can be determined based on the data processing method selected in the image resizing process.

[0286] Unlike the examples above, the data processing methods for image resizing and inverse resizing can be independent. That is, the data processing method in inverse resizing can be chosen regardless of the method selected in image resizing. For example, one could choose a method that uses some data from the image to fill in the area generated with the expansion, and then choose a method that simply removes the area to be deleted with the shrinking in inverse resizing.

[0287] According to the present invention, the data processing method during image resizing processing can be implicitly determined based on the encoding / decoding settings, and the data processing method during inverse processing can also be implicitly determined based on the encoding / decoding settings. Alternatively, the data processing method during image resizing processing can be explicitly generated, and the data processing method during inverse processing can also be explicitly generated. Alternatively, the data processing method during image resizing processing can be explicitly generated, and the data processing method during inverse processing can be implicitly determined based on this data processing method.

[0288] Next, an example of performing image resizing in an encoding / decoding apparatus according to an embodiment of the present invention will be described. In the following description, as an example, the resizing process indicates expansion, and the inverse resizing process indicates reduction. Furthermore, the difference between the image before and after resizing may refer to the image size, and the resizing-related information may include some explicitly generated segments and other segments implicitly determined according to the encoding / decoding settings. Additionally, the resizing-related information may include information about the resizing process and the inverse resizing process.

[0289] As a first example, the input image can be resized before encoding begins. The input image can be resized using resizing information (e.g., resizing operation, resizing direction, resizing value, data processing method, etc.; the data processing method is used during the resizing process), and then the input image can be encoded. The encoded image data (here, the resized image) can be stored in memory after encoding is complete, and can be added to a bitstream and then transmitted.

[0290] Resizing can be performed before decoding begins. Resizing information (e.g., resizing operation, resizing direction, resizing value, etc.) can be used to resize the image decoding data, which can then be parsed for decoding. The output image can be stored in memory after decoding is complete and can be reverted to its original size by performing inverse resizing (here, data processing methods, etc., are used for inverse resizing).

[0291] As a second example, the size adjustment of the reference image can be performed before encoding begins. The size of the reference image can be adjusted using size adjustment information (e.g., size adjustment operation, size adjustment direction, size adjustment value, data processing method, etc.; the data processing method is used during the size adjustment process), and then the reference image (here, the resized reference image) can be stored in memory. The image can then be encoded using the resized reference image. After encoding is complete, the image encoding data (here, the data obtained by encoding using the reference image) can be added to the bitstream and then transmitted. Furthermore, the aforementioned size adjustment process can be performed while the encoded image is stored in memory as a reference image.

[0292] Before decoding begins, the reference image can be resized. Resizing information (e.g., resizing operation, resizing direction, resizing value, data processing method, etc.; the data processing method is used during the resizing process) can be used to resize the reference image, and then the resized reference image (here, the resized reference image) can be stored in memory. Image decoding data (here, encoded using the reference image by the encoder) can be parsed for decoding. After decoding is complete, an output image can be generated. The aforementioned resizing process can be performed while the decoded image is stored in memory as a reference image.

[0293] As a third example, image resizing can be performed before filtering (here, assuming a deblocking filter) and after encoding (specifically, after encoding other than filtering is complete). The image size can be adjusted using resizing information (e.g., resizing operation, resizing direction, resizing value, data processing method, etc.; the data processing method is used during resizing), and then a resized image can be generated and then filtered. After filtering, inverse resizing is performed so that the resized image can be transformed back to the original image.

[0294] After decoding is complete (specifically, after decoding except for filtering), and before filtering, image resizing can be performed. The image size can be adjusted using resizing information (e.g., resizing operation, resizing direction, resizing value, data processing method, etc.; data processing method used during resizing); and then the resized image can be generated and then filtered. After filtering, inverse resizing is performed, so that the resized image can be transformed back to the original image.

[0295] In some cases (the first and third examples), both sizing and reverse sizing can be performed. In other cases (the second example), only sizing can be performed.

[0296] Furthermore, in some cases (the second and third examples), the same resizing process can be applied to both the encoder and decoder. In other cases (the first example), the same or different resizing processes can be applied to both the encoder and decoder. Here, the resizing processes for the encoder and decoder can differ in terms of the resizing execution steps. For example, in some cases (here, the encoder), resizing execution steps may include considering image resizing for the resized region and data processing. In other cases (here, the decoder), resizing execution steps may include considering image resizing. Here, the preceding data processing may correspond to the subsequent data processing during the inverse resizing process.

[0297] Furthermore, in some cases (the third example), the resizing process can be applied only to the corresponding steps, and the resizing region may not need to be stored in memory. For example, to use the resizing region in a filtering process, the resizing region can be stored in temporary memory, filtered, and then removed by inverse resizing. In this case, there is no change in image size due to resizing. The invention is not limited to the above examples, and modifications can be made to them.

[0298] The size of an image can be changed through resizing, thus altering the coordinates of some pixels. This can affect the operation of the image segmentation unit. According to the invention, this processing allows for block-based segmentation based on either the image before or after resizing. Furthermore, unit-based segmentation (e.g., tiles, slices, etc.) can be performed based on either the image before or after resizing, depending on the encoding / decoding settings. The following description focuses on the case where the image segmentation unit operates based on the image after resizing (e.g., image segmentation processing after resizing), but other modifications are possible. The examples described above will be presented under various image settings described below.

[0299] The encoder can add information generated during the above processing to the bitstream in units of at least one of the following: sequences, images, slices, tiles, etc., and the decoder can parse the relevant information from the bitstream. Furthermore, the information can be included in the bitstream in the form of SEI or metadata.

[0300] Typically, the input image can be encoded or decoded either as is or after image reconstruction. For example, image reconstruction can be performed to improve image encoding efficiency, image reconstruction can be performed to take into account network and user environments, and image reconstruction can be performed based on the image type, characteristics, etc.

[0301] According to the present invention, image reconstruction processing may include reconstruction processing alone or may include reconstruction processing and inverse reconstruction processing. The following examples will focus on reconstruction processing, but inverse reconstruction processing can be derived from reconstruction processing.

[0302] Figure 10 This is an example diagram of image reconstruction according to an embodiment of the present invention.

[0303] Assume that part 10a shows the initial input image. Parts 10a to 10d are example images of image rotation including predetermined angles of 0 degrees (e.g., candidate groups can be generated by sampling 360 degrees into k parts; k can have values ​​of 2, 4, 8, etc.; in this example, k is assumed to be 4). Parts 10e to 10h are example images that have an opposite (or symmetrical) relationship with respect to part 10a or with respect to parts 10b to 10d.

[0304] The starting position or scanning order of an image can be changed based on image reconstruction, but the starting position and scanning order can be predetermined independently of reconstruction, which can be determined based on encoding / decoding settings. The following implementation assumes that the starting position (e.g., the upper left position of the image) and scanning order (e.g., raster scanning) are predetermined independently of image reconstruction.

[0305] The image encoding and image decoding methods according to embodiments of the present invention may include the following image reconstruction steps. In this case, the image reconstruction process may include an image reconstruction instruction step, an image reconstruction type identification step, and an image reconstruction execution step. Furthermore, the image encoding apparatus and the image decoding apparatus may be configured to include an image reconstruction instruction unit, an image reconstruction type identification unit, and an image reconstruction execution unit that respectively execute the image reconstruction instruction step, the image reconstruction type identification step, and the image reconstruction execution step. For encoding, relevant syntax elements may be generated. For decoding, relevant syntax elements may be parsed.

[0306] In the image reconstruction instruction step, it can be determined whether to perform image reconstruction. For example, reconstruction can be performed when a signal indicating image reconstruction (e.g., convert_enabled_flag) is confirmed. When no signal indicating image reconstruction is confirmed, reconstruction may not be performed, or it can be performed by confirming other encoding / decoding information. Furthermore, even if no signal indicating image reconstruction is provided, it can be implicitly activated or deactivated based on encoding / decoding settings (e.g., image characteristics, type, etc.). When reconstruction is performed, corresponding reconstruction-related information can be generated, or the corresponding reconstruction-related information can be implicitly determined.

[0307] When a signal indicating image reconstruction is provided, the corresponding signal indicates whether image reconstruction should be performed. The decision to reconstruct the corresponding image can be determined based on the signal. For example, suppose a signal indicating image reconstruction is confirmed (e.g., `convert_enabled_flag`). When the corresponding signal is activated (e.g., `convert_enabled_flag=1`), reconstruction can be performed. When the corresponding signal is deactivated (e.g., `convert_enabled_flag=0`), reconstruction can be prevented.

[0308] Furthermore, when no signal indicating image reconstruction is provided, reconstruction may not be performed, or alternative signals may be used to determine whether to reconstruct the corresponding image. For example, reconstruction may be performed based on the characteristics, type, etc., of the image (e.g., a 360-degree image), and reconstruction information may be explicitly generated or specified as predetermined values. This invention is not limited to the above examples and modifications may be made to them.

[0309] In the image reconstruction type identification step, the image reconstruction type can be identified. The image reconstruction type can be defined by reconstruction methods, reconstruction mode information, etc. Reconstruction methods (e.g., `convert_type_flag`) can include flipping, rotation, etc., and reconstruction mode information can include the mode of the reconstruction method (e.g., `convert_mode`). In this case, reconstruction-related information can consist of reconstruction methods and mode information. That is, reconstruction-related information can consist of at least one syntax element. In this case, depending on the reconstruction method, the number of candidate groups for mode information can be the same or different.

[0310] As an example, rotations may include candidates with uniform intervals (90 degrees in this case), as shown in sections 10a to 10d. Section 10a shows a 0-degree rotation, section 10b shows a 90-degree rotation, section 10c shows a 180-degree rotation, and section 10d shows a 270-degree rotation (here, it is measured clockwise).

[0311] As an example, flipping can include candidates as shown in sections 10a, 10e, and 10f. While section 10a shows no flipping, sections 10e and 10f show horizontal and vertical flipping, respectively.

[0312] In the examples above, settings for rotations with uniform intervals and settings for flipping have been described. However, these are merely examples of image reconstruction, and the invention is not limited thereto. The invention may include additional interval differences, additional flipping operations, etc., which may be determined based on the encoding / decoding settings.

[0313] Alternatively, it may include comprehensive information generated by combining reconstruction methods and corresponding schema information (e.g., convert_com_flag). In this case, reconstruction-related information can be composed of a mixture of reconstruction methods and schema information.

[0314] For example, the comprehensive information may include candidates as shown in sections 10a to 10f, which may be examples of 0-degree rotation, 90-degree rotation, 180-degree rotation, 270-degree rotation, horizontal flip, and vertical flip relative to section 10a.

[0315] Alternatively, the aggregate information may include candidates as shown in sections 10a to 10h, which may be examples of 0-degree rotation, 90-degree rotation, 180-degree rotation, 270-degree rotation, horizontal flip, vertical flip, 90-degree rotation and then horizontal flip (or horizontal flip and then 90-degree rotation), and 90-degree rotation and then vertical flip (or vertical flip and then 90-degree rotation), or examples of 0-degree rotation, 90-degree rotation, 180-degree rotation, 270-degree rotation, horizontal flip, 180-degree rotation and then horizontal flip (or horizontal flip and then 180-degree rotation), 90-degree rotation and then horizontal flip (or horizontal flip and then 90-degree rotation), and 270-degree rotation and then horizontal flip (or horizontal flip and then 270-degree rotation).

[0316] Candidate groups can be configured to include rotation patterns, flip patterns, and combinations of rotation and flip. Combination patterns can simply include pattern information from the refactoring methods, and can also include patterns generated by mixing pattern information from each method. In this case, combination patterns can include patterns generated by mixing at least one pattern from some methods (e.g., rotation) with at least one pattern from other methods (e.g., flip). In the example above, combination patterns include cases generated by combining one pattern from some methods with multiple patterns from some methods (here, 90-degree rotation + multiple flips / horizontal flip + multiple rotations). The information from the mixed build can include cases where no refactoring is applied (here, part 10a) as candidate groups, and can include cases where no refactoring is applied as the first candidate group (e.g., specifying #0 as the index).

[0317] Alternatively, the information in the hybrid construction may include schema information corresponding to the predetermined refactoring method. In this case, the refactoring-related information may consist of schema information corresponding to the predetermined refactoring method. That is, information about the refactoring method can be omitted, and the refactoring-related information may consist of a syntactic element associated with the schema information.

[0318] For example, the reconstruction-related information can be configured to include rotation-specific candidates as shown in sections 10a to 10d. Alternatively, the reconstruction-related information can be configured to include flip-specific candidates as shown in sections 10a, 10e, and 10f.

[0319] The image before and after image reconstruction can have the same size or at least one different length, depending on the encoding / decoding settings. Image reconstruction can be a process of rearranging pixels in the image (here, inverse pixel rearrangement is performed during inverse image reconstruction; this can be derived in reverse from pixel rearrangement), and therefore the position of at least one pixel can be changed. Pixel rearrangement can be performed according to rules based on image reconstruction type information.

[0320] In this context, pixel rearrangement may be influenced by the size and shape of the image (e.g., square or rectangular). Specifically, the width and height of the image before and after reconstruction can act as variables during pixel rearrangement.

[0321] For example, ratio information about at least one of the following (e.g., former / later or latter / former) can serve as variables during pixel rearrangement processing: the ratio of the width of the image before reconstruction to the width of the image after reconstruction, the ratio of the width of the image before reconstruction to the height of the image after reconstruction, the ratio of the height of the image before reconstruction to the width of the image after reconstruction, and the ratio of the height of the image before reconstruction to the height of the image after reconstruction.

[0322] In this example, when the image before and after reconstruction has the same dimensions, the ratio of the image width to the image height can act as a variable during pixel rearrangement. Furthermore, when the image is square, the ratio of the image length before and after reconstruction can act as a variable during pixel rearrangement.

[0323] In the image reconstruction execution step, image reconstruction can be performed based on the identified reconstruction information. That is, image reconstruction can be performed based on information such as reconstruction type and reconstruction mode, and encoding / decoding can be performed based on the acquired reconstructed image.

[0324] Next, an example of performing image reconstruction in an encoding / decoding apparatus according to an embodiment of the present invention will be described.

[0325] The input image reconstruction process can be performed before encoding begins. Reconstruction can be performed using reconstruction information (e.g., image reconstruction type, reconstruction mode, etc.), and the reconstructed image can be encoded. Image encoding data can be stored in memory after encoding is complete, and can be added to a bitstream and then transmitted.

[0326] Reconstruction processing can be performed before decoding begins. Reconstruction can be performed using reconstruction information (e.g., image reconstruction type, reconstruction mode, etc.), and image decoding data can be parsed for decoding. After decoding is complete, the image can be stored in memory, and the image can be transformed back to its original state by performing inverse reconstruction processing before being output.

[0327] The encoder can add information generated during the above processing to the bitstream in units of at least one of the following: sequences, images, slices, tiles, etc., and the decoder can parse the relevant information from the bitstream. Furthermore, the information can be included in the bitstream in the form of SEI or metadata.

[0328] [Table 1]

[0329]

[0330] Table 1 shows example syntax elements associated with partitioning in the image settings. The following description will focus on additional syntax elements. Furthermore, in the following examples, syntax elements are not limited to any specific unit and can be supported in various units such as sequences, images, slices, and tiles. Alternatively, syntax elements can be included in SEI, metadata, etc. Additionally, the types, order, conditions, etc., of the syntax elements supported in the following examples are limited to this example and can therefore be changed and determined according to encoding / decoding settings.

[0331] In Table 1, `tile_header_enabled_flag` represents a syntax element indicating whether encoding / decoding settings for tiles are supported. When this syntax element is activated (`tile_header_enabled_flag=1`), encoding / decoding settings can be provided in tile units. When this syntax element is deactivated (`tile_header_enabled_flag=0`), encoding / decoding settings in tile units cannot be provided, but encoding / decoding settings in higher-level units can be assigned.

[0332] Furthermore, `tile_coded_flag` represents a syntax element indicating whether a tile should be encoded or decoded. When this syntax element is activated (`tile_coded_flag=1`), the corresponding tile can be encoded or decoded. When this syntax element is deactivated (`tile_coded_flag=0`), the corresponding tile cannot be encoded or decoded. Here, not performing encoding can mean that encoded data is not generated for the corresponding tile (here, it is assumed that the corresponding region is processed by predetermined rules, etc.; this applies to meaningless regions in some projection formats of 360-degree images). Not performing decoding means that decoded data in the corresponding tile is no longer parsed (here, it is assumed that the corresponding region is processed by predetermined rules). Furthermore, not parsing decoded data can mean that encoded data does not exist in the corresponding unit and therefore parsing is no longer performed, and it can also mean that even if encoded data exists, parsing is no longer performed via this flag. The header information of tile units can be supported based on whether the tile is encoded or decoded.

[0333] The above examples focus on blocks. However, the present invention is not limited to blocks, and the above description can be modified and then applied to other dividing units of the present invention. Furthermore, the examples of block division settings are not limited to the above cases, and the above cases can be modified.

[0334] [Table 2]

[0335]

[0336] Table 2 shows example syntax elements associated with reconstruction in image settings.

[0337] Referring to Table 2, `convert_enabled_flag` represents a syntax element that indicates whether reconstruction is performed. When this syntax element is activated (`convert_enabled_flag=1`), the reconstructed image is encoded or decoded, and additional reconstruction-related information can be checked. When this syntax element is deactivated (`convert_enabled_flag=0`), the original image is encoded or decoded.

[0338] Furthermore, `convert_type_flag` represents mixed information about the reconstruction method and pattern. A method can be determined from multiple candidate groups: methods applying rotation, methods applying flip, and methods applying both rotation and flip.

[0339] [Table 3]

[0340]

[0341] Table 3 shows example syntax elements associated with resizing in image settings.

[0342] Referring to Table 3, `pic_width_in_samples` and `pic_height_in_samples` represent syntax elements that indicate the width and height of an image. The dimensions of an image can be checked using these syntax elements.

[0343] Additionally, `img_resizing_enabled_flag` is a syntax element indicating whether image resizing is performed. When this syntax element is activated (`img_resizing_enabled_flag=1`), the image is encoded or decoded after resizing, and additional resizing-related information can be checked. When this syntax element is deactivated (`img_resizing_enabled_flag=0`), the original image is encoded or decoded. Furthermore, this syntax element can indicate resizing for intra-frame prediction.

[0344] In addition, `resizing_met_flag` indicates the sizing method. A sizing method can be determined from a candidate group such as a sizing method based on a scaling factor (resizing_met_flag=0), a sizing method based on an offset factor (resizing_met_flag=1), etc.

[0345] In addition, `resizing_mov_flag` represents a syntax element used for resizing operations. For example, it can determine one of expansion or shrinking.

[0346] In addition, width_scale and height_scale represent the scale factors associated with horizontal and vertical size adjustments based on the scaling factor.

[0347] In addition, top_height_offset and bottom_height_offset represent the offset factors in the "up" and "down" directions associated with the horizontal size adjustment based on the offset factor, and left_width_offset and right_width_offset represent the offset factors in the "left" and "right" directions associated with the vertical size adjustment based on the offset factor.

[0348] The size of the resized image can be updated using size adjustment-related information and image size information.

[0349] In addition, `resizing_type_flag` is a syntax element indicating the data processing method used for the resized region. The number of candidate groups for the data processing method can be the same or different, depending on the resizing method and the resizing operation.

[0350] Image setup processing applied to the aforementioned image encoding / decoding apparatus can be performed individually or in combination. The following examples will focus on examples of performing multiple image setup processes in combination.

[0351] Figure 11 These are example diagrams illustrating images before and after image setup processing according to embodiments of the present invention. Specifically, portion 11a shows an example before image reconstruction is performed on the segmented image (e.g., an image projected during 360-degree image encoding), and portion 11b shows an image after image reconstruction is performed on the segmented image (e.g., an image packaged during 360-degree image encoding). That is, it can be understood that portion 11a is an example diagram before image setup processing is performed, and portion 11b is an example diagram after image setup processing is performed.

[0352] In this example, image partitioning (here, we assume tiles) and image reconstruction are described as image setup processes.

[0353] In the following example, image reconstruction is performed after image segmentation. However, depending on the encoding / decoding settings, image segmentation can be performed after image reconstruction, and this can be modified. Furthermore, the above-described image reconstruction process (including inverse processing) can be applied in the same or similar manner as the reconstruction process in the segmentation unit of the image in this embodiment.

[0354] Image reconstruction may or may not be performed in all partitioned units of the image, and it may be performed in some partitioned units. Therefore, the partitioned units before reconstruction (e.g., some of P0 to P5) may be the same as or different from the partitioned units after reconstruction (e.g., some of S0 to S5). Various image reconstruction scenarios will be described through the following examples. Furthermore, for ease of description, it is assumed that the units of the image are pictures, the units that partition the image are tiles, and the partitioned units are rectangular in shape.

[0355] As an example, whether to perform image reconstruction can be determined in some cells (e.g., sps_convert_enabled_flag, SEI, or metadata, etc.). Alternatively, whether to perform image reconstruction can be determined in some cells (e.g., pps_convert_enabled_flag). This can be enabled when it first appears in the corresponding cell (here, the image) or when it is activated in a higher-level cell (e.g., sps_convert_enabled_flag=1). Alternatively, whether to perform image reconstruction can be determined in some cells (e.g., tile_convert_flag[i]; i is the tile cell index). This can be enabled when it first appears in the corresponding cell (here, the tile) or when it is activated in a higher-level cell (e.g., pps_convert_enabled_flag=1). Furthermore, in part, whether to perform image reconstruction can be implicitly determined based on the encoding / decoding settings, so this information can be omitted.

[0356] As an example, it can be determined whether to reconstruct partitions in an image based on a signal indicating image reconstruction (e.g., pps_convert_enabled_flag). More specifically, it can be determined whether to reconstruct all partitions in the image based on this signal. In this case, a single signal indicating image reconstruction can be generated within the image.

[0357] As an example, it can be determined whether to reconstruct the partitioning units in the image based on a signal indicating image reconstruction (e.g., tile_convert_flag[i]). More specifically, it can be determined whether to reconstruct some partitioning units in the image based on this signal. In this case, at least one signal indicating image reconstruction can be generated (e.g., a number of signals equal to the number of partitioning units).

[0358] As an example, it can be determined whether to reconstruct an image based on a signal indicating image reconstruction (e.g., pps_convert_enabled_flag), and it can be determined whether to reconstruct partitioning units in the image based on a signal indicating image reconstruction (e.g., tile_convert_flag[i]). Specifically, when any signal is activated (e.g., pps_convert_enabled_flag=1), any other signal (e.g., tile_convert_flag[i]) can be additionally checked, and it can be determined whether to reconstruct some partitioning units in the image based on that signal (here, tile_convert_flag[i]). In this case, multiple signals indicating image reconstruction can be generated.

[0359] When a signal indicative of image reconstruction is activated, image reconstruction-related information can be generated. The following examples will describe various types of image reconstruction-related information.

[0360] As an example, reconstruction information can be generated for an image. Specifically, a single piece of reconstruction information can be used as the reconstruction information for all partitioning units in the image.

[0361] As an example, reconstruction information can be generated for partitioning units in an image. Specifically, at least one piece of reconstruction information can be used as reconstruction information for some partitioning units in the image. That is, one piece of reconstruction information can be used as reconstruction information for one partitioning unit, or one piece of reconstruction information can be used as reconstruction information for multiple partitioning units.

[0362] The following example will be described in conjunction with an example of performing image reconstruction.

[0363] For example, when a signal indicating image reconstruction (e.g., pps_convert_enabled_flag) is activated, reconstruction information commonly applied to partitioning units in the image can be generated. Alternatively, when a signal indicating image reconstruction (e.g., pps_convert_enabled_flag) is activated, reconstruction information individually applied to partitioning units in the image can be generated. Alternatively, when a signal indicating image reconstruction (e.g., tile_convert_flag[i]) is activated, reconstruction information individually applied to partitioning units in the image can be generated. Alternatively, when a signal indicating image reconstruction (e.g., tile_convert_flag[i]) is activated, reconstruction information commonly applied to partitioning units in the image can be generated.

[0364] Reconstructed information can be processed implicitly or explicitly based on encoding / decoding settings. For implicit processing, the reconstructed information can be specified as predetermined values ​​based on the characteristics and type of the image.

[0365] P0 to P5 in part 11a can correspond to S0 to S5 in part 11b, and a reconstruction process can be performed on the partitioned units. For example, P0 can be left unreconstructed and then assigned to S0. P1 can be rotated 90 degrees and then assigned to S1. P2 can be rotated 180 degrees and then assigned to S2. P3 can be horizontally flipped and then assigned to S3. P4 can be rotated 90 degrees and horizontally flipped and then assigned to S4. P5 can be rotated 180 degrees and horizontally flipped and then assigned to S5.

[0366] However, the present invention is not limited to the above examples, and various modifications can be made to the above examples. Similar to the above examples, it is possible not to reconstruct the partitioning units in the image, or at least one of reconstruction using rotation, reconstruction using flipping, and reconstruction using a combination of rotation and flipping can be performed.

[0367] When image reconstruction is applied to partitioning units, additional reconstruction processing, such as rearranging the partitioning units, can be performed. That is, the image reconstruction processing according to the invention can be configured to include rearranging the partitioning units in the image as well as rearranging the pixels in the image, and can be represented using some of the syntax elements in Table 4 (e.g., part_top, part_left, part_width, part_height, etc.). This means that image partitioning processing and image reconstruction processing can be understood in combination. In the example above, the image has been described as being divided into multiple units.

[0368] P0 to P5 in part 11a can correspond to S0 to S5 in part 11b, and a reconstruction process can be performed on the partitioned units. For example, P0 can be left unreconstructed and then assigned to S0. P1 can be left unreconstructed and then assigned to S2. P2 can be rotated 90 degrees and then assigned to S1. P3 can be horizontally flipped and then assigned to S4. P4 can be rotated 90 degrees and horizontally flipped and then assigned to S5. P5 can be horizontally flipped and then rotated 180 degrees and then assigned to S3. The invention is not limited thereto and various modifications are possible.

[0369] also, Figure 7 P_Width and P_Height can correspond to Figure 11 P_Width and P_Height, and Figure 7 P'_Width and P'_Height can correspond to Figure 11 P'_Width and P'_Height. Figure 7 The dimensions of the resized image, P'_Width × P'_Height, can be expressed as (P_Width + Exp_L + Exp_R) × (P_Height + Exp_T + Exp_B), and Figure 11 The dimensions of the resized image, P'_Width×P'_Height, can be represented as (P_Width+Var0_L+Var1_L+Var2_L+Var0_R+Var1_R+Var2_R)×(P_Height+Var0_T+Var1_T+Var0_B+Var1_B) or (Sub_P0_Width+Sub_P1_Width+Sub_P2_Width+Var0_L+Var1_L+Var2_L+Var0_R+Var1_R+Var2_R)×(Sub_P0_Height+Sub_P1_Height+Var0_T+Var1_T+Var0_B+Var1_B).

[0370] Similar to the example above, for image reconstruction, one can perform a rearrangement of pixels within the image's partitions, or a rearrangement of the image's partitions, or both. In this case, the rearrangement of the image's partitions can be performed after the rearrangement of pixels within the partitions, or vice versa.

[0371] Whether to perform a rearrangement of the partitioned units in the image can be determined based on the signal indicating image reconstruction. Alternatively, a signal for rearranging the partitioned units in the image can be generated. Specifically, this signal can be generated when the signal indicating image reconstruction is activated. Alternatively, this signal can be processed implicitly or explicitly based on encoding / decoding settings. For implicit processing, the signal can be determined based on the characteristics, type, etc., of the image.

[0372] Furthermore, information regarding the rearrangement of partition units in an image can be implicitly or explicitly executed based on encoding / decoding settings, and this information can be determined based on the characteristics and type of the image. That is, each partition unit can be arranged according to pre-determined arrangement information for the partition units.

[0373] Next, an example of reconstructing segmentation units in an image in an encoding / decoding apparatus according to an embodiment of the present invention will be described.

[0374] The input image can be partitioned using partitioning information before encoding begins. Reconstruction information can be used to reconstruct the partitioned units, and the reconstructed image for each partitioned unit can be encoded. The encoded image data can be stored in memory after encoding is complete, and can be added to a bitstream and then transmitted.

[0375] Partitioning information can be used to perform partitioning processing before decoding begins. Reconstruction information can be used to perform reconstruction processing on the partitioned units, and image decoding data can be parsed to perform decoding within the reconstructed partitioned units. After decoding is complete, the image decoding data can be stored in memory, and multiple partitioned units can be merged into a single unit after performing inverse reconstruction processing within the partitioned units, thereby outputting the image.

[0376] Figure 12 This is an example diagram illustrating the adjustment of the size of each segmentation unit of an image according to an embodiment of the present invention. Figure 12 P0 to P5 correspond to Figure 11 P0 to P5, and Figure 12 S0 to S5 correspond to Figure 11 S0 to S5.

[0377] In the following examples, the description will focus on the case where image resizing is performed after image segmentation. However, depending on the encoding / decoding settings, image segmentation can be performed after image resizing, and this process can be modified. Furthermore, the image resizing process described above (including the inverse process) can be applied in the same or similar manner as the image segmentation unit resizing process in this embodiment.

[0378] For example, Figure 7 The TL to BR can correspond to Figure 12 The partitioning units SX (S0 to S5) are TL to BR. Figure 7 S0 and S1 can correspond to Figure 12 PX and SX. Figure 7 P_Width and P_Height can correspond to Figure 12 Sub_PX_Width and Sub_PX_Height. Figure 7 P'_Width and P'_Height can correspond to Figure 12 Sub_SX_Width and Sub_SX_Height. Figure 7 Exp_L, Exp_R, Exp_T, and Exp_B can correspond to Figure 12 VarX_L, VarX_R, VarX_T, and VarX_B, and other factors can also correspond.

[0379] The processing of adjusting the size of the partitioned units in the images in parts 12a to 12f and Figure 7 The difference between image expansion and reduction in parts 7a and 7b lies in the fact that the settings for image expansion or reduction can be proportional to the number of partition units. Furthermore, the process of adjusting the size of partition units in an image can differ from image expansion or reduction in terms of settings applied commonly or individually to the partition units in the image. In the following examples, various size adjustment scenarios will be described, and size adjustment processes can be performed while taking into account the above descriptions.

[0380] According to the present invention, image resizing may or may not be performed on all partitioned units in an image, and image resizing may be performed on some partitioned units. Various image resizing scenarios will be described through the following examples. Furthermore, for ease of description, it is assumed that the resizing operation is used for expansion, the resizing operation is based on an offset factor, the resizing direction is "up," "down," "left," and "right," the resizing direction is set to operate based on resizing information, the unit of the image is a picture, and the unit dividing the image is a tile.

[0381] As an example, whether to perform image resizing can be determined in some cells (e.g., sps_img_resizing_enabled_flag, SEI, or metadata, etc.). Alternatively, whether to perform image resizing can be determined in some cells (e.g., pps_img_resizing_enabled_flag). This can be allowed when it first appears in the corresponding cell (here, the image) or when it is activated in a higher-level cell (e.g., sps_img_resizing_enabled_flag=1). Alternatively, whether to perform image resizing can be determined in some cells (e.g., tile_resizing_flag[i]; i is the cell index). This can be allowed when it first appears in the corresponding cell (here, the tile) or when it is activated in a higher-level cell. Furthermore, in part, whether to perform image resizing can be implicitly determined based on encoding / decoding settings, so this information can be omitted.

[0382] As an example, it can be determined whether to adjust the size of the partitioned units in an image based on a signal indicating image resizing (e.g., pps_img_resizing_enabled_flag). More specifically, it can be determined whether to adjust the size of all partitioned units in the image based on this signal. In this case, a single signal indicating image resizing can be generated.

[0383] As an example, it can be determined whether to adjust the size of the partitioning units in an image based on a signal indicating image resizing (e.g., tile_resizing_flag[i]). More specifically, it can be determined whether to adjust the size of some partitioning units in the image based on this signal. In this case, at least one signal indicating image resizing can be generated (e.g., a number of signals equal to the number of partitioning units).

[0384] As an example, it can be determined whether to resize an image based on a signal indicating image resizing (e.g., pps_img_resizing_enabled_flag), and it can be determined whether to resize the partitions in the image based on a signal indicating image resizing (e.g., tile_resizing_flag[i]). Specifically, when any signal is activated (e.g., pps_img_resizing_enabled_flag=1), any other signal (e.g., tile_resizing_flag[i]) can be additionally checked, and the resizing of some partitions in the image can be performed based on that signal (here, tile_resizing_flag[i]). In this case, multiple signals indicating image resizing can be generated.

[0385] When a signal indicating image resizing is activated, image resizing-related information can be generated. The following examples will describe various types of image resizing-related information.

[0386] As an example, sizing information can be generated for an image. Specifically, a single sizing information or a set of sizing information can be used as the sizing information for all partitioning units in the image. For example, a single sizing information (or a sizing value applied to all supported or allowed sizing directions in the partitioning unit; in this example, a single piece of information) can be generated that is commonly applied to the "up," "down," "left," and "right" directions of the partitioning unit, or a set of sizing information (or a number of sizing information equal to the number of allowed or supported sizing directions in the partitioning unit; in this example, up to four pieces of information) can be generated individually for the "up," "down," "left," and "right" directions.

[0387] As an example, size adjustment information can be generated for partitioning units in an image. Specifically, at least one piece of size adjustment information or a set of size adjustment information can be used as size adjustment information for all partitioning units in the image. That is, one piece of size adjustment information or a set of size adjustment information can be used as size adjustment information for one partitioning unit or as size adjustment information for multiple partitioning units. For example, size adjustment information can be generated that is commonly applied to one partitioning unit in the image in the "up," "down," "left," and "right" directions, or a set of size adjustment information can be generated that is individually applied to the "up," "down," "left," and "right" directions. Alternatively, size adjustment information can be generated that is commonly applied to multiple partitioning units in the image in the "up," "down," "left," and "right" directions, or a set of size adjustment information can be generated that is individually applied to the "up," "down," "left," and "right" directions. The configuration of this size adjustment group implies size adjustment value information regarding at least one size adjustment direction.

[0388] In summary, size adjustment information can be generated that is generally applied to the partitioning units in an image. Alternatively, size adjustment information can be generated that is applied individually to the partitioning units in an image. The following example will be described with reference to an example of performing image size adjustment.

[0389] For example, when a signal indicating image resizing (e.g., pps_img_resizing_enabled_flag) is activated, resizing information commonly applied to partitioning units in the image can be generated. Alternatively, when a signal indicating image resizing (e.g., pps_img_resizing_enabled_flag) is activated, resizing information individually applied to partitioning units in the image can be generated. Alternatively, when a signal indicating image resizing (e.g., tile_resizing_flag[i]) is activated, resizing information individually applied to partitioning units in the image can be generated. Alternatively, when a signal indicating image resizing (e.g., tile_resizing_flag[i]) is activated, resizing information commonly applied to partitioning units in the image can be generated.

[0390] Image resizing direction and resizing information can be processed implicitly or explicitly based on encoding / decoding settings. For implicit processing, resizing information can be specified to predetermined values ​​based on image characteristics and type.

[0391] The dimension adjustment direction in the dimension adjustment process of the present invention has been described as being at least one of an "up" direction, a "down" direction, a "left" direction, and a "right" direction, and the dimension adjustment direction and dimension adjustment information can be processed explicitly or implicitly. That is, the dimension adjustment value (including 0; which means no dimension adjustment) can be implicitly predetermined for some directions, and the dimension adjustment value (including 0; which means no dimension adjustment) can be explicitly specified for other directions.

[0392] Even within the subdivisions of an image, the size adjustment direction and information can be set to implicit or explicit processing, and this can be applied to the subdivisions. For example, a setting can be applied to one subdivision (here, a setting equal to the number of subdivisions can appear), a setting can be applied to multiple subdivisions, or a setting can be applied to all subdivisions (here, a single setting can appear), and at least one setting can appear in the image (e.g., a setting up to the number of subdivisions can appear). Setting information applied to the subdivisions in the image can be collected, and a single set of settings can be defined.

[0393] Figure 13 This is an example diagram of the settings or size adjustments of the division units in an image.

[0394] In detail, Figure 13 Various examples of implicitly or explicitly processing the sizing direction and sizing information for partitioning units in an image are shown. In the following examples, for ease of description, implicit processing assumes that the sizing value for some sizing directions is 0.

[0395] As shown in section 13a, size adjustments can be explicitly handled when the boundaries of the partitioning units match the boundaries of the image (here, thick solid lines), and implicitly handled when the boundaries of the partitioning units do not match the boundaries of the image (thin solid lines). For example, P0 can be adjusted in the "up" and "left" directions (a2, a0), P1 can be adjusted in the "up" direction (a2), P2 can be adjusted in the "up" and "right" directions (a2, a1), P3 can be adjusted in the "down" and "left" directions (a3, a0), P4 can be adjusted in the "down" direction (a3), and P5 can be adjusted in the "down" and "right" directions (a3, a1). In this case, size adjustments in other directions may not be allowed.

[0396] As shown in section 13b, some directions of the partitioning unit (here, up and down) allow explicit size adjustment, and some directions of the partitioning unit (here, left and right) allow explicit size adjustment (here, thick solid lines) when the boundary of the partitioning unit matches the boundary of the image, and allow implicit size adjustment (here, thin solid lines) when the boundary of the partitioning unit does not match the boundary of the image. For example, P0 can be adjusted in the "up", "down", and "left" directions (b2, b3, b0); P1 can be adjusted in the "up" and "down" directions (b2, b3); P2 can be adjusted in the "up", "down", and "right" directions (b2, b3, b1); P3 can be adjusted in the "up", "down", and "left" directions (b3, b4, b0); P4 can be adjusted in the "up" and "down" directions (b3, b4); and P5 can be adjusted in the "up", "down", and "right" directions (b3, b4, b1). In this case, sizing in other directions may not be allowed.

[0397] As shown in section 13c, some directions of the partitioning unit (here, left and right) allow explicit size adjustment, and some directions of the partitioning unit (here, up and down) allow explicit size adjustment (here, thick solid lines) when the boundary of the partitioning unit matches the boundary of the image, and allow implicit size adjustment (here, thin solid lines) when the boundary of the partitioning unit does not match the boundary of the image. For example, P0 can be adjusted in the "up", "left", and "right" directions (c4, c0, c1); P1 can be adjusted in the "up", "left", and "right" directions (c4, c1, c2); P2 can be adjusted in the "up", "left", and "right" directions (c4, c2, c3); P3 can be adjusted in the "down", "left", and "right" directions (c5, c0, c1); P4 can be adjusted in the "down", "left", and "right" directions (c5, c1, c2); and P5 can be adjusted in the "down", "left", and "right" directions (c5, c2, c3). In this case, sizing in other directions may not be allowed.

[0398] Similar to the example above, settings related to image resizing can have various configurations. Multiple sets of settings are supported, allowing setting group selection information to be generated explicitly or implicitly determined based on encoding / decoding settings (e.g., image characteristics, type, etc.).

[0399] Figure 14 This is an example diagram illustrating both the process of adjusting the image size and the process of adjusting the size of the partitioned units in the image.

[0400] Reference Figure 14 The image resizing process and its inverse can be performed in directions e and f, and the resizing process and its inverse can be performed in directions d and g. That is, image resizing can be performed, and then the resizing process can be performed on the resizing units within the image. The order of resizing is not fixed. This means that multiple resizing processes are possible.

[0401] In summary, image resizing can be categorized into image resizing (or resizing the image before segmentation) and resizing of segmented units within the image (or resizing the image after segmentation). Either image resizing or resizing of segmented units within the image may not be performed, or either or both may be performed, depending on the encoding / decoding settings (e.g., image characteristics, type, etc.).

[0402] When performing multiple resizing processes in this example, the image can be resized in at least one of the "up," "down," "left," and "right" directions, and the size of at least one division unit in the image can be adjusted. In this case, the resizing of the division unit to be resized can be performed in at least one of the "up," "down," "left," and "right" directions.

[0403] Reference Figure 14 The dimensions of the image (A) before resizing can be defined as P_Width × P_Height, the dimensions of the image after primary resizing (or the image before secondary resizing; B) can be defined as P'_Width × P'_Height, and the dimensions of the image after secondary resizing (or the image after final resizing; C) can be defined as P''_Width × P''_Height. The image (A) before resizing represents an image without resizing, the image after primary resizing (B) represents an image with some resizing, and the image after secondary resizing (C) represents an image with all resizing. For example, the image after primary resizing (B) can represent an image resizing according to the image division units shown in sections 13a to 13c, and the image after secondary resizing (C) can represent an image resizing by means of... Figure 7 The image is obtained by completely adjusting the size of the image (B) after the primary size adjustment, as shown in part 7a. The opposite is also possible. However, the present invention is not limited to the above example, and various modifications can be made to the above example.

[0404] In the dimensions of the image (B) after the initial resizing, P'_Width can be obtained by P_Width and at least one horizontal resizing value for the lateral resizing, and P'_Height can be obtained by P_Height and at least one vertical resizing value for the longitudinal resizing. In this case, the resizing values ​​can be the resizing values ​​generated in the partitioning cells.

[0405] In the dimensions of the image (C) after secondary resizing, P''_Width can be obtained by P'_Width and at least one horizontal resizing value for horizontal resizing, and P''_Height can be obtained by P'_Height and at least one vertical resizing value for vertical resizing. In this case, the resizing values ​​can be the resizing values ​​generated in the image.

[0406] In summary, the size of the resized image can be obtained using at least one resizing value and the size of the image before resizing.

[0407] In the resized areas of an image, information about the data processing methods can be generated. Various data processing methods will be described through the following examples. Data processing methods generated during desizing can be applied in the same or similar manner as those used in resizing. The data processing methods in resizing and desizing will be described through various combinations described below.

[0408] As an example, data processing methods can be generated for application to an image. Specifically, a data processing method or a set of data processing methods can be used as the data processing method for all partitioned units in the image (here, it is assumed that the size of all partitioned units will be adjusted). For example, a data processing method (or a data processing method applied to all supported or allowed size adjustment directions in the partitioned units, etc.; in this example, one message) can be generated for the "up," "down," "left," and "right" directions in the image (or a set of data processing methods applied to the "up," "down," "left," and "right" directions, or a number equal to the number of supported or allowed size adjustment directions in the partitioned units; in this example, up to four messages).

[0409] As an example, data processing methods can be generated for application to partitioned units in an image. Specifically, at least one data processing method or a set of data processing methods can be used as data processing methods for some partitioned units in the image (here, it is assumed that the size of the partitioned units is to be adjusted). That is, a data processing method or a set of data processing methods can be used as a data processing method for one partitioned unit or for multiple partitioned units. For example, a data processing method can be generated that is commonly applied to the "up," "down," "left," and "right" directions of one partitioned unit in the image, or a set of data processing methods can be generated that are individually applied to the "up," "down," "left," and "right" directions. Alternatively, a data processing method can be generated that is commonly applied to the "up," "down," "left," and "right" directions of multiple partitioned units in the image, or a set of data processing methods can be generated that are individually applied to the "up," "down," "left," and "right" directions. The configuration of this set of data processing methods implies a data processing method for adjusting the direction of at least one size.

[0410] In summary, a data processing method that is commonly applied to partitioning units in an image can be used. Alternatively, a data processing method that is applied separately to partitioning units in an image can be used. The data processing method can be a predetermined method. A predetermined data processing method can be provided as at least one method. This corresponds to implicit processing, and selection information for the data processing method can be explicitly generated, which can be determined based on encoding / decoding settings (e.g., image characteristics, type, etc.).

[0411] That is, a data processing method commonly applied to the partitioning units in the image can be used. A predetermined method can be used, or one of several data processing methods can be selected. Alternatively, a data processing method applied separately to the partitioning units in the image can be used. Depending on the partitioning units, a predetermined method can be used, or one of several data processing methods can be selected.

[0412] The following examples will describe some cases of adjusting the size of the partitioning units in an image (here, it is assumed that the size adjustment is used for expansion) (here, the size-adjusted region is filled with some data from the image).

[0413] The dimensions of specific regions TL to BR (e.g., S0 to S5 in parts 12a to 12f) of some elements (P0 to P5) can be adjusted using data from specific regions tl to br of some elements (in parts 12a to 12f). In this case, some elements can be the same as each other (e.g., S0 and P0) or different (e.g., S0 and P1). That is, the regions TL to BR to be resized can be filled with some data tl to br of the corresponding partitioned elements, and can also be filled with some data from partitioned elements other than the corresponding partitioned elements.

[0414] As an example, the size of the region TL to BR of the current partition unit can be adjusted using the data tl to br of the current partition unit. For example, TL of S0 can be filled with data tl of P0, RC of S1 can be filled with data tr+rc+br of P1, BL+BC of S2 can be filled with data bl+bc+br of P2, and TL+LC+BL of S3 can be filled with data tl+lc+bl of P3.

[0415] As an example, the size of the region TL to BR of the current partition unit can be adjusted using the data tl to br of the partition units spatially adjacent to the current partition unit. For example, in the "up" direction, TL+TC+TR of S4 can be filled with the data bl+bc+br of P1; in the "down" direction, BL+BC of S2 can be filled with the data tl+tc+tr of P5; in the "left" direction, LC+BL of S2 can be filled with the data tl+rc+bl of P1; in the "right" direction, RC of S3 can be filled with the data tl+lc+bl of P4; and in the "down + left" direction, BR of S0 can be filled with the data tl of P4.

[0416] As an example, the size of the region TL to BR of the current partition unit can be adjusted using data tl to br from partition units that are not spatially adjacent to the current partition unit. For example, data in the boundary regions between the two ends of an image (e.g., horizontal, vertical, etc.) can be acquired. The LC of S3 can be obtained using the data tr+rc+br of S5, the RC of S2 can be obtained using the data tl+lc of S0, the BC of S4 can be obtained using the data tc+tr of S1, and the TC of S1 can be obtained using the data bc of S4.

[0417] Alternatively, data can be obtained for specific regions of the image (regions that are not spatially adjacent to the resized region but are determined to have a high correlation with the resized region). The BC of S1 can be obtained using the data tl+lc+bl of S3, the RC of S3 can be obtained using the data tl+tc of S1, and the RC of S5 can be obtained using the data bc of S0.

[0418] In addition, some cases of adjusting the size of the partitioning units in the image (here, assuming the size adjustment is for reduction) are as follows (here, removal is performed by restoring or correcting some data of the image).

[0419] Specific regions TL to BR of some cells (e.g., S0 to S5 in sections 12a to 12f) can be used for the recovery or correction processing of specific regions tl to br of some cells P0 to P5. In this case, some cells may be the same as each other (e.g., S0 and P0) or different (e.g., S0 and P2). That is, regions to be resized can be used to recover some data of the corresponding partition cell and then removed, and regions to be resized can be used to recover some data of partition cells other than the corresponding partition cell and then removed. Detailed examples can be derived from the reverse of the extended process and will therefore be omitted.

[0420] This example can be applied to situations where highly correlated data exists in the area to be resized, and information about the referenced location for the resizing can be explicitly generated or implicitly obtained according to predetermined rules. Alternatively, relevant information can be examined in combination. This could be an example that can be applied when data is obtained from another area with continuity in the encoding of a 360-degree image.

[0421] Next, an example of adjusting the size of the segmentation units in an image in an encoding / decoding apparatus according to an embodiment of the present invention will be described.

[0422] The input image can be partitioned before encoding begins. Size adjustment information can be used to resize the partitioned units, and the image can be encoded after the partitioned units have been resized. The encoded image data can be stored in memory after encoding is complete, and can be added to a bitstream and then transmitted.

[0423] Partitioning information can be used to perform partitioning before decoding begins. Size adjustment information can be used to perform size adjustment on the partitioned units, and image decoding data can be parsed to decode within the size-adjusted partitioned units. After decoding is complete, the image decoding data can be stored in memory, and multiple partitioned units can be merged into a single unit after performing inverse size adjustment on the partitioned units, thus allowing the image to be output.

[0424] Other examples of the image resizing process described above can be applied. The invention is not limited thereto and can be modified.

[0425] In image settings processing, it is permissible to combine image resizing and image reconstruction. Image reconstruction can be performed after image resizing. Alternatively, image resizing can be performed after image reconstruction. Furthermore, it is permissible to combine image partitioning, image reconstruction, and image resizing. Image resizing and image reconstruction can be performed after image partitioning. The order of image settings is not fixed and can be changed, depending on the encoding / decoding settings. In this example, image settings processing is described as performing image reconstruction and image resizing after image partitioning. However, depending on the encoding / decoding settings, other orders are possible and can be modified.

[0426] For example, image setup processing can be performed in the following order: partitioning -> reconstruction; reconstruction -> partitioning; partitioning -> resizing; resizing -> partitioning; resizing -> reconstruction; reconstruction -> resizing; partitioning -> reconstruction -> resizing; partitioning -> resizing -> reconstruction; resizing -> partitioning -> reconstruction; resizing -> reconstruction -> partitioning; and reconstruction -> resizing -> partitioning, and combinations with additional image settings are possible. As mentioned above, image setup processing can be performed sequentially, but some or all of the setup processing can be performed simultaneously. Furthermore, as some image setup processing, multiple processes can be performed based on encoding / decoding settings (e.g., image characteristics, type, etc.). The following examples illustrate various combinations of image setup processing.

[0427] As an example, P0 to P5 in section 11a can correspond to S0 to S5 in section 11b, and reconstruction processing (here, rearrangement of pixels) and size adjustment processing (here, adjusting the size of the division units to have the same size) can be performed in the partitioning units. For example, the size of P0 to P5 can be adjusted based on the offset, and P0 to P5 can be assigned to S0 to S5. Alternatively, P0 can be reconstructed without reconstructing, and then assigned to S0. P1 can be rotated 90 degrees and then assigned to S1. P2 can be rotated 180 degrees and then assigned to S2. P3 can be rotated 270 degrees and then assigned to S3. P4 can be horizontally flipped and then assigned to S4. P5 can be vertically flipped and then assigned to S5.

[0428] As an example, P0 to P5 in section 11a may correspond to the same or different positions as S0 to S5 in section 11b, and reconstruction processing (here, rearrangement of pixels and division units) and size adjustment processing (here, adjusting the size of the division units to have the same size) can be performed in the division units. For example, the size of P0 to P5 can be adjusted based on a ratio, and P0 to P5 can be assigned to S0 to S5. Alternatively, P0 may not be reconstructed, and then P0 can be assigned to S0. P1 may not be reconstructed, and then P1 can be assigned to S2. P2 may be rotated 90 degrees, and then it can be assigned to S1. P3 may be horizontally flipped, and then it can be assigned to S4. P4 may be rotated 90 degrees and horizontally flipped, and then it can be assigned to S5. P5 may be horizontally flipped and then rotated 180 degrees, and then it can be assigned to S3.

[0429] As an example, P0 to P5 in section 11a can correspond to E0 to E5 in section 5e, and reconstruction processing (here, rearrangement of pixels and division units) and size adjustment processing (here, adjusting the size of the division units to have different sizes) can be performed in the division units. For example, P0 can be resized and reconstructed without being resized and then assigned to E0, P1 can be resized based on the scale but not reconstructed and then assigned to E1, P2 can be resized without being resized but reconstructed and then assigned to E2, P3 can be resized based on the offset but not reconstructed and then assigned to E4, P4 can be resized without being resized but reconstructed and then assigned to E5, and P5 can be resized based on the offset and reconstructed and then assigned to E3.

[0430] Similar to the examples above, the absolute or relative positions of the partitioning units in the image before and after image setup processing can be maintained or changed, which can be determined based on encoding / decoding settings (e.g., image characteristics, type, etc.). Furthermore, various combinations of image setup processing are possible. The invention is not limited thereto, and therefore various modifications are possible.

[0431] The encoder can add information generated during the above processing to the bitstream in units of at least one of sequences, images, slices, tiles, etc., and the decoder can parse the relevant information from the bitstream. Furthermore, the information can be included in the bitstream in the form of SEI or metadata.

[0432] [Table 4]

[0433]

[0434] Table 4 shows example syntax elements associated with multiple image settings. The following description will focus on additional syntax elements. Furthermore, in the following examples, syntax elements are not limited to any specific unit and can be supported in various units such as sequences, images, slices, and tiles. Alternatively, syntax elements can be included in SEI, metadata, etc.

[0435] Referring to Table 4, `parts_enabled_flag` represents a syntax element indicating whether to divide the image into units. When this syntax element is activated (`parts_enabled_flag=1`), the image can be divided into multiple units, and these units can be encoded or decoded. Additionally, supplementary division information can be checked. When this syntax element is deactivated (`parts_enabled_flag=0`), the original image is encoded or decoded. In this example, the description will focus on rectangular division units such as tiles, and different settings can be provided for existing tiles and division information.

[0436] Here, num_partitions refers to a syntax element that indicates the number of partitions. num_partitions plus 1 equals the number of partitions.

[0437] Furthermore, `part_top[i]` and `part_left[i]` are syntax elements indicating the position information of the partitioning unit, representing the horizontal and vertical start positions of the partitioning unit (e.g., the top-left position of the partitioning unit). Additionally, `part_width[i]` and `part_height[i]` are syntax elements indicating the size information of the partitioning unit, representing the width and height of the partitioning unit. In this case, the start position and size information can be set at the pixel level or at the block level. Furthermore, the syntax elements can be syntax elements that can be generated during image reconstruction processing or syntax elements that can be generated when combining image partitioning and image reconstruction processing.

[0438] In addition, `part_header_enabled_flag` represents a syntax element indicating whether encoding / decoding settings for partition units are supported. When this syntax element is activated (`part_header_enabled_flag=1`), encoding / decoding settings for partition units can be provided. When this syntax element is deactivated (`part_header_enabled_flag=0`), encoding / decoding settings cannot be provided, but encoding / decoding settings for higher-level units can be assigned.

[0439] The above examples are not limited to examples of syntax elements associated with resizing and reconstruction within a partitioning unit in image settings, and can be modified for other partitioning units and settings of the present invention. This example has been described assuming that resizing and reconstruction are performed after partitioning, but the present invention is not limited thereto and can be modified with other image setting orders, etc. Furthermore, the types, orders, conditions, etc., of syntax elements supported in the following examples are limited to this example and can therefore be changed and determined according to encoding / decoding settings.

[0440] [Table 5]

[0441]

[0442] Table 5 shows example syntax elements associated with reconstruction in the partitioned units within the image settings.

[0443] Referring to Table 5, `part_convert_flag[i]` represents a syntax element indicating whether to reconstruct the partition unit. This syntax element can be generated for each partition unit. When this syntax element is activated (`part_convert_flag[i]=1`), the reconstructed partition unit can be encoded or decoded, and additional reconstruction-related information can be checked. When this syntax element is deactivated (`part_convert_flag[i]=0`), the original partition unit is encoded or decoded. Here, `convert_type_flag[i]` refers to the pattern information about the reconstruction of the partition unit, and can be information about pixel rearrangement.

[0444] Furthermore, additional syntax elements indicating refactoring, such as partition unit rearrangement, can be generated. In this example, partition unit rearrangement can be performed via part_top and part_left, which are syntax elements indicating the aforementioned image partitioning, or syntax elements associated with partition unit rearrangement (e.g., index information) can be generated.

[0445] [Table 6]

[0446]

[0447] Table 6 shows example syntax elements related to size adjustments in the partitioned units within the image settings.

[0448] Referring to Table 6, `part_resizing_flag[i]` represents a syntax element indicating whether to adjust the size of the partition units in the image. This syntax element can be generated for each partition unit. When this syntax element is activated (`part_resizing_flag[i]=1`), the resized partition units can be encoded or decoded after resizing, and additional resizing-related information can be checked. When this syntax element is deactivated (`part_resizing_flag[i]=0`), the original partition units are encoded or decoded.

[0449] In addition, width_scale[i] and height_scale[i] represent the scale factors associated with the horizontal and vertical size adjustments based on the scaling factor in the partitioned unit.

[0450] In addition, top_height_offset[i] and bottom_height_offset[i] represent the offset factors for the "up" direction and the offset factors for the "down" direction associated with the size adjustment based on the offset factor in the partitioning unit, and left_width_offset[i] and right_width_offset[i] represent the offset factors for the "left" direction and the offset factors for the "right" direction associated with the size adjustment based on the offset factor in the partitioning unit.

[0451] Furthermore, `resizing_type_flag[i][j]` represents a syntax element indicating the data processing method for the resized region within the partitioned unit. This syntax element indicates a separate data processing method for the resized direction. For example, syntax elements indicating separate data processing methods for the resized region in the "up," "down," "left," and "right" directions can be generated. This syntax element can be generated based on resized information (e.g., it can be generated only if resized in some directions).

[0452] The image settings processing described above can be applied based on the characteristics, type, etc., of the image. In the following examples, unless specifically mentioned, the image settings processing described above can be applied with or without any changes. The descriptions in the following examples will focus on the additions or changes made in the examples described above.

[0453] For example, 360-degree images or omnidirectional images generated by a 360-degree camera device have characteristics that are different from those of images acquired by a regular camera device, and have a coding environment that is different from the coding environment for compressing a general image.

[0454] Unlike regular images, 360-degree images may not have boundary regions with discontinuities, and the data in all regions of a 360-degree image can be continuous. Furthermore, devices such as HMDs may require high-resolution images because the images should be reproduced in front of the eye through lenses. The amount of image data processed can increase when images are acquired using stereoscopic camera devices. Various image setups considering 360-degree images can be performed to provide an efficient encoding environment including the examples described above.

[0455] A 360-degree camera device can be multiple camera devices or a camera device with multiple lenses and sensors. The camera devices or lenses can cover all directions around any center point captured by the camera devices.

[0456] Various methods can be used to encode 360-degree images. For example, various image processing algorithms in 3D space can be used to encode 360-degree images, and various image processing algorithms can be used to convert 360-degree images to 2D space and encode them. According to the present invention, the following description will focus on methods for converting 360-degree images to 2D space and encoding or decoding the converted images.

[0457] A 360-degree image encoding apparatus according to an embodiment of the present invention may include Figure 1 The device may include some or all of the elements shown, and may also include a preprocessing unit configured to preprocess the input image (stitching, projection, region packing). Meanwhile, the 360-degree image decoding apparatus according to embodiments of the present invention may include... Figure 2 The image may include some or all of the elements shown, and may also include a post-processing unit configured to post-process the encoded image to reproduce the output image before decoding the encoded image.

[0458] In other words, the encoder can preprocess the input image, encode the preprocessed image, and transmit a bitstream including the image, and the decoder can parse, decode, and post-process the transmitted bitstream to generate an output image. In this case, the transmitted bitstream may include information generated during the preprocessing process and information generated during the encoding process, and the bitstream can be parsed and used during the decoding and post-processing processes.

[0459] The operation method for the 360-degree image encoder will then be described in more detail, and those skilled in the art can easily derive the operation method for the 360-degree image decoder. Since the operation method for the 360-degree image decoder is the opposite of the operation method for the 360-degree image encoder, a detailed description of the operation method for the 360-degree image decoder will be omitted.

[0460] The input image can undergo stitching and projection processing on a sphere-based 3D projection structure, and the image data on the 3D projection structure can be projected into a 2D image through this processing.

[0461] The projected image can be configured to include some or all of the 360-degree content based on encoding settings. In this case, the positional information of the region (or pixel) to be placed at the center of the projected image can be generated implicitly as a predetermined value, or the positional information of the region (or pixel) can be generated explicitly. Furthermore, when the projected image includes a specific region of the 360-degree content, the extent and positional information of the included region can be generated. Additionally, the extent (e.g., width and height) and positional information (e.g., measured based on the upper left of the image) of the region of interest (ROI) can be generated based on the projected image. In this case, a specific region of high importance within the 360-degree content can be set as the ROI. A 360-degree image can allow viewing of all content in the "up," "down," "left," and "right" directions, but the user's gaze may be limited to a portion of the image; this limitation can be taken into account when setting that portion of the image as the ROI. For efficient encoding purposes, the ROI can be set to have good quality and high resolution, while other regions can be set to have lower quality and lower resolution than the ROI.

[0462] Among various 360-degree image transmission schemes, a single-stream transmission scheme allows the transmission of a full image or viewport image in a single bitstream for the user. A multi-stream transmission scheme allows the transmission of several full images with different image qualities in multiple bitstreams, thus enabling selection of image quality based on user environment and communication conditions. A tile-based streaming scheme allows the transmission of individually encoded, tile-based partial images in multiple bitstreams, thus enabling selection of tiles based on user environment and communication conditions. Therefore, a 360-degree image encoder can generate and transmit bitstreams with two or more qualities, and a 360-degree image decoder can set the Region of Interest (ROI) based on the user's view and selectively decode the bitstream based on the ROI. That is, the area where the user is looking can be set as the ROI using a head-tracking or eye-tracking system, and only the required portion can be presented.

[0463] A projected image can be converted into a packed image obtained by performing region-wise packing. Region-wise packing can include the step of dividing the projected image into multiple regions, and the regions can be arranged (or rearranged) within the packed image according to the region-wise packing settings. When a 360-degree image is converted into a 2D image (or a projected image), region-wise packing can be performed to increase spatial continuity. Therefore, image size can be reduced through region-wise packing. Furthermore, region-wise packing can be performed to reduce image quality degradation during rendering, enable viewport-based projection, and provide other types of projection formats. Region-wise packing can be performed or not, depending on encoding settings, which can be determined based on signals indicating whether region-wise packing should be performed (e.g., `regionwise_packing_flag`; information about region-wise packing can only be generated when `regionwise_packing_flag` is activated).

[0464] When performing regional packaging, settings (or mapping information) can be displayed (or generated) to assign (or arrange) specific regions of the projected image to specific regions of the packaged image. When not performing regional packaging, the projected image and the packaged image can be the same image.

[0465] In the above description, stitching, projection, and regional packaging are defined as separate processes, but some (e.g., stitching + projection, projection + regional packaging) or all (e.g., stitching + projection + regional packaging) of the processes can be defined as a single process.

[0466] At least one packed image can be generated from the same input image based on the settings for stitching, projection, and region-based packing. Furthermore, based on the settings for region-based packing, at least one encoded data can be generated for the same projected image.

[0467] Packed images can be divided by performing a tiling process. In this case, tiling, as a process of dividing an image into multiple regions and then transmitting it, can be an example of a 360-degree image transmission scheme. As mentioned above, tiling can be performed considering the user environment for partial decoding, and it can also be performed for efficiently processing large amounts of data in a 360-degree image. For example, when an image consists of a single unit, the entire image can be decoded to decode the ROI. On the other hand, when an image consists of multiple unit regions, decoding only the ROI may be efficient. In this case, the division can be performed as a unit of tiles, which are division units according to a conventional encoding scheme, or it can be performed according to various division units (e.g., quadrilateral division blocks, etc.) already described according to the present invention. Furthermore, the division unit can be a unit for performing independent encoding / decoding. Tiling can be performed independently or based on the projected image or the packed image. That is, division can be performed based on the surface boundaries of the projected image, the surface boundaries of the packed image, the packing settings, etc., and division can be performed independently for each division unit. This may affect the generation of division information during the tiling process.

[0468] Next, the projected or packed image can be encoded. Encoded data and information generated during preprocessing can be added to the bitstream, and the bitstream can be transmitted to the 360-degree image decoder. The information generated during preprocessing can be added to the bitstream in the form of SEI or metadata. In this case, the bitstream can contain: at least one piece of encoded data with settings partially different for the encoding process; and at least one piece of preprocessing information with settings partially different for the preprocessing process. This is to construct the decoded image by combining multiple pieces of encoded data (encoded data + preprocessing information) according to the user environment. Specifically, the decoded image can be constructed by selectively combining multiple pieces of encoded data. Furthermore, this processing can be performed in two parts for application to a stereo system, and the processing can be performed on an additional depth image.

[0469] Figure 15 These are example diagrams illustrating 2D planar space and 3D space illustrating a 3D image.

[0470] Typically, for the purpose of a 360-degree 3D virtual space, three degrees of freedom (3DoF) may be required, and three rotations relative to the X-axis (pitch), Y-axis (yaw), and Z-axis (roll) may be supported. DoF refers to spatial degrees of freedom, 3DoF refers to degrees of freedom including rotation about the X-axis, Y-axis, and Z-axis as shown in section 15a, and 6DoF refers to additionally allowing translation along the X-axis, Y-axis, and Z-axis, as well as 3DoF degrees of freedom. The following description will focus on the image encoding and image decoding apparatus of the present invention having 3DoF. When supporting 3DoF or greater (3DoF+), the image encoding and image decoding apparatus may be modified or combined with additional processing or apparatus not shown.

[0471] Referring to section 15a, yaw can have a range from -π (-180 degrees) to π (180 degrees), pitch can have a range from -π / 2 radians (or -90 degrees) to π / 2 radians (or 90 degrees), and roll can have a range from -π / 2 radians (or -90 degrees) to π / 2 radians (or 90 degrees). In this case, when Φ and θ are assumed to be longitude and latitude in a map representation of the Earth, 3D spatial coordinates (x, y, z) can be transformed from 2D spatial coordinates (Φ, θ). For example, 3D spatial coordinates can be derived from 2D spatial coordinates according to the transformation formulas x=cos(θ)cos(Φ), y=sin(θ), and z=-cos(θ)sin(Φ).

[0472] Furthermore, (Φ, θ) can be transformed into (x, y, z). For example, this can be achieved using the transformation formula Φ = tanθ. -1 (-Z / X) and θ=sin -1 (Y / (X 2 +Y 2 +Z 2 ) 1 / 2 Derive 2D spatial coordinates from 3D spatial coordinates.

[0473] When pixels in 3D space are accurately transformed into pixels in 2D space (e.g., integer unit pixels in 2D space), pixels in 3D space can be mapped to pixels in 2D space. When pixels in 3D space are not accurately transformed into pixels in 2D space (e.g., fractional unit pixels in 2D space), pixels obtained through interpolation can be mapped to 2D pixels. In this case, nearest neighbor interpolation, bilinear interpolation, B-spline interpolation, bicubic interpolation, etc., can be used as interpolation methods. In this case, relevant information can be explicitly generated by selecting one of several interpolation candidates, or the interpolation method can be implicitly determined according to predetermined rules. For example, predetermined interpolation filters can be used based on the 3D model, projection format, color format, and tile / pattern type. Furthermore, when interpolation information is explicitly generated, information about the filters (e.g., filter coefficients) can be included.

[0474] Part 15b shows an image transformed from 3D space to 2D space (2D planar coordinate system). Samples (i, j) can be taken from (Φ, θ) based on the image's dimensions (width and height). Here, i can range from 0 to P_Width-1, and j can range from 0 to P_Height-1.

[0475] (Φ, θ) can be the center point (or reference point) for arranging the 360-degree image relative to the projected image; depicted as Figure 15 The center point is a point C with coordinates (Φ, θ) = (0, 0). The center point can be set in 3D space, and its position information can be generated explicitly or implicitly determined as a predetermined value. For example, center position information can be generated for yaw, pitch, roll, etc. When no value is specified individually, each value can be assumed to be zero.

[0476] The above has described an example of transforming an entire 360-degree image from 3D space to 2D space. However, specific regions of a 360-degree image can be transformed, and positional information (e.g., some locations belonging to that region; in this example, positional information about the center point), extent information, etc., of a specific region can be explicitly generated, or the positional and extent information of a specific region can implicitly follow predetermined positional and extent information. For example, center position information in yaw, center position information in pitch, center position information in roll, extent information in yaw, extent information in pitch, extent information in roll, etc., can be generated, and the specific region can be at least one region. Therefore, positional and extent information of multiple regions can be processed. When the values ​​of information are not specified individually, the entire 360-degree image can be assumed.

[0477] H0 to H6 and W0 to W5 in part 15a indicate some latitude and longitude in part 15b, which can be represented as coordinates (C, j) and (i, C) in part 15b (where C is the longitude or latitude component). Unlike general images, when a 360-degree image is converted to 2D space, distortion or warping of the content in the image may occur. This may depend on the region of the image, and different encoding / decoding settings can be applied to the location of the image or the region divided according to that location. When encoding / decoding settings are adaptively applied based on encoding / decoding information in this invention, location information (e.g., x-component, y-component, or a range defined by x and y) may be included as an example of encoding / decoding information.

[0478] The definitions of 3D and 2D spaces are provided to aid in describing embodiments of the invention. However, the invention is not limited thereto, and the above description may be modified in detail or applied to other situations.

[0479] As described above, images acquired by a 360-degree camera device can be transformed into 2D space. In this case, a 3D model can be used to map the 360-degree image, and various 3D models such as spheres, cubes, cylinders, pyramids, and polyhedra can be used. When the model-mapped 360-degree image is transformed into 2D space, projection processing can be performed according to the model-based projection format.

[0480] Figures 16a to 16d This is a conceptual diagram illustrating a projection format according to an embodiment of the present invention.

[0481] Figure 16a This illustrates the equal rectangular projection (ERP) format in which a 360-degree image is projected onto a 2D plane. Figure 16b The cube map projection (CMP) format, in which a 360-degree image is projected onto a cube, is shown. Figure 16c This illustrates the octahedral projection (OHP) format, which projects a 360-degree image onto an octahedron. Figure 16d This illustrates a 360-degree image projected onto an icosahedral projection (ISP) format. However, the invention is not limited to this, and various projection formats can be used. Figures 16a to 16d The left side shows the 3D model, and the right side shows an example transformed into 2D space through projection processing. Various sizes and shapes can be provided depending on the projection format. Each shape can consist of surfaces or faces, and each face can be represented as a circle, triangle, quadrilateral, etc.

[0482] In this invention, projection formats can be defined by 3D models, face settings (e.g., the number of faces, face shapes, face shape configurations, etc.), projection processing settings, etc. Projection formats can be considered different projection formats when at least one element differs in definition. For example, ERP consists of a sphere model (3D model), one face (number of faces), and quadrilateral faces (face shapes). However, formats can be classified as different formats, such as ERP1 and ERP2, when some settings in the projection processing settings (e.g., the formula used during the transformation from 3D space to 2D space; i.e., elements with the same residual projection settings that produce differences in at least one pixel of the projected image during projection processing) differ. As another example, CMP consists of a cube model, six faces, and quadrilateral faces. Formats can be classified as different formats, such as CMP1 and CMP2, when some settings during the projection processing (e.g., the sampling method applied during the transformation from 3D space to 2D space) differ.

[0483] When multiple projection formats are used in place of a predetermined projection format, projection format identification information (or projection format information) can be explicitly generated. The projection format identification information can be configured using various methods.

[0484] As an example, a projection format can be identified by assigning index information (e.g., proj_format_flag) to multiple projection formats. For instance, #0 can be assigned to ERP, #1 to CMP, #2 to OHP, #3 to ISP, #4 to ERP1, #5 to CMP1, #6 to OHP1, #7 to ISP1, #8 to CMP compact, #9 to OHP compact, #10 to ISPcompact, and #11 or higher can be assigned to other formats.

[0485] As an example, a projection format can be identified using at least one element of information constituting the projection format. In this case, the element of information constituting the projection format may include 3D model information (e.g., 3d_model_flag; #0 indicates a sphere, #1 indicates a cube, #2 indicates a cylinder, #3 indicates a pyramid, #4 indicates polyhedron 1, and #5 indicates polyhedron 2), face number information (e.g., num_face_flag; a method of incrementing by 1 starting from 1; the number of faces generated in the projection format is specified as index information, i.e., #0 indicates one, #1 indicates three, #2 indicates six, #3 indicates eight, and #4 indicates twenty), face shape information (e.g., shape_face_flag; #0 indicates a quadrilateral, #1 indicates a circle, #2 indicates a triangle, #3 indicates a quadrilateral + circle, and #4 indicates a quadrilateral + triangle), projection processing settings information (e.g., 3d_2d_convert_idx), etc.

[0486] As an example, a projection format can be identified using element information constituting the projection format and projection format index information. For instance, as projection format index information, #0 can be assigned to ERP, #1 to CMP, #2 to OHP, #3 to ISP, and #4 or higher can be assigned to other formats. Projection formats (e.g., ERP, ERP1, CMP, CMP1, OHP, OHP1, ISP, and ISP1) can be identified together with element information constituting the projection format (here, projection processing settings information). Alternatively, projection formats (e.g., ERP, CMP, CMP compact, OHP, OHP compact, ISP, and ISP compact) can be identified together with element information constituting the projection format (here, regional packaging).

[0487] In summary, a projection format can be identified using projection format index information, at least one projection format element information, or both projection format index information and at least one projection format element information. This can be defined according to encoding / decoding settings. In this invention, the following description assumes that a projection format is identified using a projection format index. In this example, the description will focus on projection formats expressed using faces with the same size and shape, but configurations with faces of different sizes and shapes are possible. Furthermore, the configuration of each face can be... Figures 16a to 16d The configurations shown may be the same or different, the number of each face is used as a symbol to identify the corresponding face, and there is no restriction on a particular order. For ease of description, the following description assumes that for projected images, ERP is a projection format including one face + quadrilateral, CMP is a projection format including six faces + quadrilateral, OHP is a projection format including eight faces + triangle, ISP is a projection format including twenty faces + triangle, and that the faces have the same size and shape. However, this description can be applied in the same or similar way even for different settings.

[0488] like Figures 16a to 16d As shown, projection formats can be categorized into one face (e.g., ERP) or multiple faces (e.g., CMP, OHP, and ISP). Furthermore, the shape of each face can be categorized as quadrilateral, triangle, etc. The categorization can be an example of the type, characteristics, etc., of the image according to the invention, which can be applied when different encoding / decoding settings are provided according to the projection format. For example, the image type can be a 360-degree image, and the image characteristics can be one of the categories (e.g., each projection format, projection formats with one or more faces, projection formats with quadrilateral or non-quadrilateral faces).

[0489] A 2D planar coordinate system (e.g., (I, j)) can be defined in each face of a 2D projected image, and the characteristics of the coordinate system can vary depending on the projection format, the position of each face, etc. An ERP can have one 2D planar coordinate system, and other projection formats can have multiple 2D planar coordinate systems depending on the number of faces. In this case, the coordinate system can be represented as (k, i, j), where k can indicate the index information of each face.

[0490] Figure 17 This is a conceptual diagram illustrating a projection format included in a rectangular image according to an embodiment of the present invention.

[0491] That is, it is understandable that portions 17a to 17c show Figures 16b to 16d The projection format is implemented as a rectangular image.

[0492] Referring to sections 17a to 17c, each image format can be configured in a rectangular shape to encode or decode 360-degree images. For ERP, a single coordinate system can be used as is. However, for other projection formats, the coordinate systems of the surfaces can be integrated into a single coordinate system, and its detailed description will be omitted.

[0493] Referring to sections 17a to 17c, when constructing a rectangular image, it can be confirmed that regions filled with meaningless data such as blanks or background are generated. That is, a rectangular image can consist of regions including actual data (here, faces; valid regions) and meaningless regions added to construct the rectangular image (here, assuming these regions are filled with any pixel values; invalid regions). This can degrade performance due to the increase in encoded data, i.e., the increase in image size caused by the encoding / decoding of meaningless regions and actual image data.

[0494] Therefore, additional processing can be performed to construct the image by excluding meaningless regions and using regions that include actual data.

[0495] Figure 18 This is a conceptual diagram of a method for converting a projection format into a rectangular shape according to an embodiment of the present invention, namely, a method for performing rearrangement on a surface to exclude meaningless regions.

[0496] Referring to sections 18a to 18c, examples for rearranging sections 17a to 17c can be identified, and this process can be defined as a region-based packing process (CMP compact, OHP compact, ISP compact, etc.). In this case, faces can not only be rearranged, but also divided and then rearranged (OHP compact, ISP compact, etc.). This can be performed to remove meaningless regions and improve coding performance through efficient face arrangement. For example, when images are arranged consecutively between faces (e.g., B2-B3-B1, B5-B0-B4, etc. in section 18a), prediction accuracy during encoding is enhanced, thus improving coding performance. Here, region-based packing according to the projection format is merely an example, and the invention is not limited thereto.

[0497] Figure 19 This is a conceptual diagram illustrating the execution of region packing processing according to an embodiment of the present invention to convert a CMP projection format into a rectangular image.

[0498] Referring to sections 19a to 19c, the CMP projection format can be arranged as 6×1, 3×2, 2×3, and 1×6. Furthermore, when adjusting the size of some faces, the arrangement can be as shown in sections 19d and 19e. In sections 19a to 19e, CMP is used as an example. However, the invention is not limited to this, and other projection formats can be applied. The arrangement of faces in an image acquired through region packing can follow predetermined rules corresponding to the projection format, or information about the arrangement can be explicitly generated.

[0499] The 360-degree image encoding and decoding apparatus according to an embodiment of the present invention can be configured to include Figure 1 and Figure 2 Some or all of the elements of the image encoding and decoding apparatus shown are included. In particular, a format conversion unit configured to transform the projection format and an inverse format conversion unit configured to perform an inverse transformation of the projection format may also be included in the image encoding apparatus and the image decoding apparatus, respectively. That is, the input image can be processed by the format conversion unit and then by... Figure 1 The image encoding device encodes the bitstream, and the bitstream can be encoded by the image encoding device. Figure 2 The image decoding device decodes the image and then processes it through an inverse format transformation unit to generate an output image. The following description will focus on the processing performed by the encoder (here, the input image, encoding, etc.), and the processing performed by the decoder can be derived from the encoder in reverse. Furthermore, redundant descriptions of the foregoing will be omitted.

[0500] The following description assumes that the input image is the same as a packaged image or a 2D projected image obtained by performing a preprocessing procedure using a 360-degree encoding device. That is, the input image can be an image obtained by performing projection processing or regional packaging processing according to some projection format. The projection format pre-applied to the input image can be one of various projection formats, which can be considered a common format and referred to as the first format.

[0501] The format conversion unit can perform conversions to projection formats other than the first format. In this case, the projection format to which the conversion is to be performed can be referred to as the second format. For example, ERP can be set as the first format, and ERP can be converted to the second format (e.g., ERP2, CMP, OHP, and ISP). In this case, ERP2 includes EPR formats that have the same conditions such as 3D models and face configurations, but with some different settings. Alternatively, the projection format can be the same format with the same projection format settings (e.g., ERP=ERP2), and can have different image sizes or resolutions. Alternatively, some of the following image setting processes can be applied. For ease of description, such examples have been mentioned, but each of the first and second formats can be one of various projection formats. However, the invention is not limited thereto and can be modified thereto.

[0502] During format transformation processing, due to the different coordinate system characteristics, pixels in the transformed image (integer pixels) can be obtained from both fractional and integer pixel units in the image before transformation, thus enabling interpolation. The interpolation filter used in this case can be the same as or similar to the interpolation filter described above. In this case, relevant information can be explicitly generated by selecting one of several interpolation filter candidates, or the interpolation filter can be implicitly determined according to predetermined rules. For example, predetermined interpolation filters can be used based on projection format, color format, and tile / pattern type. Furthermore, when the interpolation filter is explicitly provided, information about the filter (e.g., filter coefficients) can be included.

[0503] In the format conversion section, the projection format can be defined to include regional packing, etc. That is, projection and regional packing can be performed during format conversion processing. Alternatively, processing such as regional packing can be performed after format conversion processing and before encoding.

[0504] The encoder can add information generated during the above processing to the bitstream in units of at least one of sequences, images, slices, tiles, etc., and the decoder can parse the relevant information from the bitstream. Furthermore, the information can be included in the bitstream in the form of SEI or metadata.

[0505] Next, image setting processing applied to a 360-degree image encoding / decoding apparatus according to an embodiment of the present invention will be described. The image setting processing according to the present invention can be applied to the preprocessing process, postprocessing process, format conversion processing, inverse format conversion processing, and general encoding / decoding processing of a 360-degree image encoding / decoding apparatus. The following description of image setting processing will focus on the 360-degree image encoding apparatus and may include the aforementioned image settings. Redundant descriptions of the aforementioned image setting processing will be omitted. Furthermore, the following examples will focus on image setting processing, and inverse image setting processing can be derived from the image setting processing. Several points can be confirmed through the various embodiments of the present invention described above.

[0506] The image setting process according to the invention can be performed in the 360-degree image projection step, the regional packaging step, the format conversion step, or other steps.

[0507] Figure 20 This is a conceptual diagram of 360-degree image segmentation according to an embodiment of the present invention. Figure 20 In this context, we assume that the image is projected through the ERP.

[0508] Section 20a illustrates an image projected via ERP, and various methods can be used to divide the image. In this example, the description focuses on slices or tiles, and it is assumed that W0 to W2 and H0 and H1 are the dividing boundaries of the slices or tiles and follow the raster scan order. The following examples focus on slices and tiles. However, the invention is not limited thereto, and other dividing methods can be applied.

[0509] For example, partitioning can be performed on a slice basis, and H0 and H1 can be set as partition boundaries. Alternatively, partitioning can be performed on a block basis, and W0 to W2, H0, and H1 can be set as partition boundaries.

[0510] Section 20b illustrates an example of an image projected via ERP being divided into tiles (assuming the same tile division boundaries as shown in Section 20a (W0 to W2, H0, and H1 are all activated)). When assuming region P is the entire image and region V is the area where the user's gaze rests, or the viewport, various methods exist to provide an image corresponding to the viewport. For example, the region corresponding to the viewport can be obtained by decoding the entire image (e.g., tiles a to i). In this case, the entire image can be decoded, and in the case of a divided image, tiles a to i (here, region A + region B) can be decoded. Alternatively, the region corresponding to the viewport can be obtained by decoding the region belonging to the viewport. In this case, in the case of a divided image, the region corresponding to the viewport can be obtained from the image recovered by decoding tiles f, g, j, and k (here, region B). The former case can be referred to as full decoding (or viewport-independent encoding), while the latter case can be referred to as partial decoding (or viewport-dependent encoding). The latter case can be an example that may occur in a 360-degree image with a large amount of data. Compared to slice-based partitioning methods, tile-based partitioning methods can be used more frequently because they offer greater flexibility in acquiring partitioned regions. For partial decoding, the referability of partitioning units can be constrained (implicitly handled here) spatially or temporally, since it's impossible to determine where the viewpoint will occur, and this constraint can be taken into account when performing encoding / decoding. The following examples will focus on full decoding, but will concentrate on describing 360-degree image partitioning using tiles (or the rectangular partitioning method of this invention) in preparation for partial decoding. However, the following description can be applied to other partitioning units in the same or modified manner.

[0511] Figure 21 This is an example diagram of 360-degree image segmentation and image reconstruction according to an embodiment of the present invention. Figure 21 In this context, we assume that the image is projected via CMP.

[0512] Section 21a shows an image projected by CMP, and various methods can be used to divide the image. It is assumed that W0 to W2, H0, and H1 are the dividing boundaries of faces, slices, and tiles and follow the raster scan order.

[0513] For example, partitioning can be performed on slices, with H0 and H1 set as partition boundaries. Alternatively, partitioning can be performed on tiles, with W0 to W2, H0, and H1 set as partition boundaries. Alternatively, partitioning can be performed on faces, with W0 to W2, H0, and H1 set as partition boundaries. In this example, it is assumed that the face is part of the partitioning unit.

[0514] In this context, a face can be a partitioning unit (here, a plane coordinate system for each face) that has different properties in the same image, performed to classify or distinguish them based on the image's characteristics, type (in this example, a 360-degree image and a projection format), etc. (depending on encoding / decoding). A slice or tile can be a partitioning unit that is performed to divide the image according to user definitions (here, independent encoding / decoding). Furthermore, a face can be a unit partitioned during projection processing according to a projection format by a predetermined definition (or induced by projection format information), while a slice or tile can be a unit partitioned by explicitly generating partitioning information according to user definitions. Additionally, a face can have a polygonal partitioning shape, including quadrilaterals, depending on the projection format; a slice can have any partitioning shape that cannot be defined as a quadrilateral or polygon; and a tile can have a quadrilateral partitioning shape. The settings for the partitioning units can be defined only for the description of this example.

[0515] In this example, a face has been described as a partitioning unit for classifying regions. However, a face can also be a unit used to perform independent encoding / decoding based on encoding / decoding settings as at least one face unit, and can have settings for performing independent encoding / decoding in combination with tiles, slices, etc. In this case, when a face is combined with tiles, slices, etc., explicit information about the tiles and slices can be generated, or tiles and slices can be implicitly combined based on face information. Alternatively, explicit information about tiles and slices can be generated based on face information.

[0516] As a first example, an image partitioning process (here, a face) is performed, and the partitioning information can be implicitly omitted (it is obtained based on the projection format information). This example is for cases that depend on encoding / decoding settings and can correspond to situations where the referability between face elements is unrestricted.

[0517] As a second example, an image partitioning process (here, a face) is performed, and the image partitioning can explicitly generate partitioning information. This example is for cases that depend on encoding / decoding settings and can correspond to situations where the referability between face cells is unrestricted.

[0518] As a third example, multiple image partitioning processes (here, faces and tiles) are performed. Some image partitions (here, faces) may implicitly omit or explicitly generate partitioning information, while other image partitions (here, tiles) may explicitly generate partitioning information. In this example, one image partitioning process (here, faces) precedes the other image partitioning processes (here, tiles).

[0519] As a fourth example, multiple image partitioning processes are performed. Some image partitions (here, faces) can implicitly omit or explicitly generate partitioning information, while other image partitions (here, tiles) can explicitly generate partitioning information based on these image partitions (here, faces). In this example, one image partitioning process (here, faces) precedes the other image partitioning processes (here, tiles). In some cases of this example (assuming the second example), it can be the same as explicitly generating partitioning information, but the partitioning information configuration may differ.

[0520] As a fifth example, multiple image partitioning processes are performed. Some image partitions (here, faces) can implicitly omit partitioning information, and other image partitions (here, tiles) can implicitly omit partitioning information based on these image partitions (here, faces). For example, face units can be set as tile units individually, or multiple face units (here, face units are grouped when adjacent faces are continuous; otherwise, face units are not grouped; B2-B3-B1 and B4-B0-B5 in part 18a) can be set as tile units. Face units can be set as tile units according to predetermined rules. This example is for independent encoding / decoding settings and can be an example corresponding to cases where the referability between face units is limited. That is, in some cases (assuming the first example), the partitioning information can be the same as implicitly processed, but the encoding / decoding settings may differ.

[0521] This example can be a description of a scenario where partitioning can be performed in a projection step, a region packing step, an initial encoding / decoding step, etc., and can be any other image partitioning process performed in the encoder / decoder.

[0522] In part 21a, a rectangular image can be constructed by adding regions (B) that do not contain data to regions (A) that include data. In this case, the position, size, shape, number, etc., of regions A and B can be information that can be checked through the projection format or when explicitly generating information about the projected image, and the relevant information can be represented by the aforementioned image segmentation information, image reconstruction information, etc. For example, information about a specific region of the projected image (e.g., part_top, part_left, part_width, part_height, and part_convert_flag) can be represented as shown in Tables 4 and 5. However, the invention is not limited to this and can be applied to other situations (e.g., other projection formats, other projection settings, etc.).

[0523] Regions B and A can be constructed as a single image and then encoded or decoded. Alternatively, region-specific characteristics can be considered when performing partitioning, and different encoding / decoding settings can be applied. For example, region B can be encoded or decoded without using information about whether encoding or decoding is performed (e.g., the `tile_coded_flag` when the partitioning unit is assumed to be a tile). In this case, the corresponding region can be restored to some data (here, any pixel value) according to predetermined rules. Alternatively, in the image partitioning process described above, region B can have different encoding / decoding settings than region A. Alternatively, the corresponding region can be removed by performing region-specific packing processing.

[0524] Section 21b illustrates an example of an image packaged via CMP being divided into tiles, slices, or faces. In this case, the packaged image is an image that has undergone face rearrangement or region-based packaging processing, and can be an image obtained by performing image partitioning and image reconstruction according to the invention.

[0525] In part 21b, the rectangular shape can be constructed to include regions containing data. In this case, the location, size, shape, number, etc., of the regions can be information that can be checked through predetermined settings or information that can be checked when explicitly generating information about the packaged image, and the relevant information can be represented by the aforementioned image segmentation information, image reconstruction information, etc. For example, information about a specific region of the packaged image (e.g., part_top, part_left, part_width, part_height, and part_convert_flag) can be represented as shown in Tables 4 and 5.

[0526] Various partitioning methods can be used to partition a packaged image. For example, partitioning can be performed on a slice basis, with H0 set as the partition boundary. Alternatively, partitioning can be performed on a tile basis, with W0, W1, and H0 set as partition boundaries. Alternatively, partitioning can be performed on a polygon basis, with W0, W1, and H0 set as partition boundaries.

[0527] Image partitioning and image reconstruction processing according to the present invention can be performed on the projected image. In this case, the reconstruction processing can be used to rearrange the faces and pixels in the image. This can be a possible example when the image is divided into multiple faces or is composed of multiple faces. The following examples will be described focusing on the case of dividing the image into patches based on face units.

[0528] In section 21a, SX and Y (S0,0 to S3,2) can correspond to S'U and V (S'0,0 to S'2,1) in section 21b (here, X and Y can be the same as or different from U and V), and reconstruction processing can be performed on a face-by-face basis. For example, S2,1, S3,1, S0,1, S1,2, S1,1, and S1,0 can be assigned to S'0,0, S'1,0, S'2,0, S'0,1, S'1,1, and S'2,1 (face rearrangement). Furthermore, S2,1, S3,1, and S0,1 may not be reconstructed (pixel rearrangement), and S1,2, S1,1, and S1,0 can be rotated 90 degrees and then reconstructed. This can be represented as shown in section 21c. In section 21c, the horizontally placed symbols (S1,0, S1,1, S1,2) can be images that are horizontally placed to maintain the continuity of the image.

[0529] The reconstruction of surfaces can be handled implicitly or explicitly based on the encoding / decoding settings. Implicit processing can be performed according to predetermined rules, taking into account the image type (here, a 360-degree image) and characteristics (here, projection format, etc.).

[0530] For example, for S'0,0 and S'1,0, S'1,0 and S'2,0, S'0,1 and S'1,1, S'1,1 and S'2,1 in part 21c, there exists image continuity (or correlation) between two faces relative to the face boundaries, and part 21c can be an example of continuity between three upper faces and three lower faces. In cases where an image is divided into multiple faces by projection processing from 3D space to 2D space and then the image is packaged for each region, reconstruction can be performed to increase image continuity between faces in order to efficiently reconstruct the faces. Such face reconstructions can be predetermined and processed.

[0531] Alternative sites can perform refactoring through explicit processing and generate refactoring information.

[0532] For example, when examining information about an M×N build (e.g., 6×1, 3×2, 2×3, 1×6, etc. for CMP compaction; in this example, a 3×2 configuration is assumed) via region-based packing processing (e.g., one of implicitly acquired and explicitly generated information), face reconstruction can be performed based on the M×N build, and information about the face reconstruction can then be generated. For example, when rearranging faces in an image, index information (or information about their position in the image) can be assigned to each face. When rearranging pixels within a face, pattern information for reconstruction can be assigned.

[0533] Index information can be as follows Figure 18 As shown in sections 18a to 18c, they are predefined. In sections 21a to 21c, SX,Y or S'U,V represent each face using positional information indicating width and height (e.g., S[i][j]) or using a single positional information (e.g., S[i]; assuming the positional information is assigned in raster scan order starting from the top left of the image), and each face can be assigned an index.

[0534] For example, when indexing is assigned using positional information indicating width and height, face index #2 can be assigned to S'0,0, face index #3 can be assigned to S'1,0, face index #1 can be assigned to S'2,0, face index #5 can be assigned to S'0,1, face index #0 can be assigned to S'1,1, and face index #4 can be assigned to S'2,1, as shown in section 21c. Alternatively, when indexing is assigned using a single positional information, face index #2 can be assigned to S[0], face index #3 can be assigned to S[1], face index #1 can be assigned to S[2], face index #5 can be assigned to S[3], face index #0 can be assigned to S[4], and face index #4 can be assigned to S[5]. For ease of description, in the following examples, S'0,0 to S'2,1 can be referred to as a to f. Alternatively, each face can be represented based on positional information indicating the width and height of a pixel or block unit at the top left corner of the image.

[0535] For packaged images obtained through image reconstruction processing (or region-based packaging processing), the face scanning order may be the same as or different from the image scanning order, depending on the reconstruction settings. For example, when a scanning order (e.g., raster scanning) is applied to the image shown in part 21a, a, b, and c may have the same scanning order, while d, e, and f may have different scanning orders. For instance, when the scanning order of part 21a or the scanning order of a, b, and c follows the order (0,0)->(1,0)->(0,1)->(1,1), the scanning order of d, e, and f may follow the order (1,0)->(1,1)->(0,0)->(0,1). This can be determined based on the image reconstruction settings, and such settings can even be applied to other projection formats.

[0536] In the image partitioning process shown in section 21b, tiles can be individually set as surface units. For example, each of surfaces a through f can be set as a tile unit. Alternatively, multiple surface units can be set as tiles. For example, surfaces a through c can be set as one tile, and surfaces d through f can be set as one tile. The configuration can be determined based on surface characteristics (e.g., continuity between surfaces, etc.), and unlike the example above, different tile settings for surfaces can be possible.

[0537] The following is an example of partitioning information processed based on multiple image partitioning. In this example, it is assumed that the partitioning information of faces is omitted, the units other than faces are tiles, and the partitioning information is processed differently.

[0538] As a first example, image segmentation information can be obtained based on surface information, and this information can be implicitly omitted. For example, a surface can be set as a tile individually, or multiple surfaces can be set as tiles. In this case, when at least one surface is set as a tile, this can be determined based on surface information (e.g., continuity or correlation) according to predetermined rules.

[0539] As a second example, image segmentation information can be explicitly generated independent of face information. For instance, when generating segmentation information using the number of columns (here, num_tile_columns) and rows (here, num_tile_rows) of tiles, the segmentation information can be generated in the image segmentation processing method described above. For example, the number of columns of a tile can range from 0 to the width of the image or the width of the block (here, the unit obtained from the image segmentation section), and the number of rows of a tile can range from 0 to the height of the image or the height of the block. Furthermore, additional segmentation information (e.g., uniform_spacing_flag) can be generated. In this case, depending on the segmentation settings, the boundaries of the faces and the boundaries of the segmentation units may or may not match each other.

[0540] As a third example, image partitioning information can be explicitly generated based on face information. For instance, when generating partitioning information using the number of columns and rows of a tile, it can be based on face information (here, the number of columns ranges from 0 to 2, and the number of rows ranges from 0 to 1; because the face configuration in the image is 3x2). For example, the number of columns of a tile can be in the range of 0 to 2, and the number of rows of a tile can be in the range of 0 to 1. Furthermore, additional partitioning information (e.g., uniform_spacing_flag) may not be generated. In this case, the boundaries of the faces and the boundaries of the partitioning units can match each other.

[0541] In some cases (such as the second and third examples), the syntax elements for partitioning information can be defined differently, or even when using the same syntax elements, different syntax element settings can be applied (e.g., binarization settings; other binarizations can be used when the range of candidate groups for the syntax elements is limited and small). The examples described above have been described for some of the various elements of partitioning information. However, the invention is not limited thereto, and it is understood that other settings are possible depending on whether the partitioning information is generated based on surface information.

[0542] Figure 22 This is an example image of an image that has been packaged or projected using CMP and divided into tiles.

[0543] In this case, assuming it has the same Figure 21 The tile partitioning boundaries shown in part 21a are the same as those in the tile partitioning boundaries (W0 to W2, H0 and H1 are all activated) and have the same... Figure 21 The tile partitioning boundaries shown in section 21b are the same (W0, W1, and H0 are all activated). When assuming region P represents the entire image and region V represents the viewport, full decoding or partial decoding can be performed. This example will focus on partial decoding. In section 22a, tiles e, f, and g can be decoded against the CMP (left side), and tiles a, c, and e can be decoded against the CMP compact (right side) to obtain the region corresponding to the viewport. In section 22b, tiles b, f, and i can be decoded against the CMP, and tiles d, e, and f can be decoded against the CMP compact to obtain the region corresponding to the viewport.

[0544] The above examples have already described the cases of partitioning based on surface elements (or surface boundaries) such as slicing and patching. However, as... Figure 20 As shown in part 20a, partitioning can be performed inside a surface (e.g., an image consists of one surface in ERP and multiple surfaces in other projection formats), or partitioning can be performed on the boundaries of a surface and inside the surface.

[0545] Figure 23 This is a conceptual diagram illustrating an example of adjusting the size of a 360-degree image according to an embodiment of the present invention. In this case, it is assumed that the image is projected via ERP. Furthermore, the following examples will focus on an extended case.

[0546] The size of the projected image can be adjusted using either a scaling factor or an offset factor, depending on the image size adjustment type. Here, the image before size adjustment can be P_Width × P_Height, and the image after size adjustment can be P'_Width × P'_Height.

[0547] For the scaling factor, after adjusting the width and height of the image using the scaling factor (here, a for width and b for height), the image width (P_Width × a) and height (P_Height × b) can be obtained. For the offset factor, after adjusting the width and height of the image using the offset factor (here, L and R for width and T and B for height), the image width (P_Width + L + R) and height (P_Height + T + B) can be obtained. Resizing can be performed using a predefined method, or by using a method selected from several options.

[0548] The data processing methods in the following examples will focus on the case of offset factors. For offset factors, data processing methods may include filling methods using predetermined pixel values, filling methods by copying external pixels, filling methods by copying a specific region of the image, and filling methods by transforming a specific region of the image.

[0549] The size of a 360-degree image can be adjusted by considering the continuity at the image boundaries. For ERPs, there are no outer boundaries in 3D space, but outer boundaries may exist when the 3D space is transformed to 2D space through projection processing. Data in the boundary regions includes data with outward continuity, but may have boundaries in terms of spatial properties. These properties can be considered when performing size adjustments. In this case, continuity can be checked according to the projection format, etc. For example, an ERP image can be an image with the property that both ends of the boundary are continuous. This example will be described under the assumption that the left and right boundaries of the image are continuous with each other, and the top and bottom boundaries of the image are continuous with each other. The data processing methods will focus on filling methods by copying specific regions of the image and filling methods by transforming specific regions of the image.

[0550] When resizing the image to the left, the resized region (LC or TL+LC+BL) can be filled with data from the right region of the image (tr+rc+br), which is continuous with the left side of the image. When resizing the image to the right, the resized region (RC or TR+RC+BR) can be filled with data from the left region of the image (tl+lc+bl), which is continuous with the right side of the image. When resizing the image upwards, the resized region (TC or TL+TC+TR) can be filled with data from the lower region of the image (bl+bc+br), which is continuous with the upper side of the image. When resizing the image downwards, the resized region (BC or BL+BC+BR) can be filled with data.

[0551] When the size or length of the resized region is m, the coordinates of the resized region relative to the image before resizing (here, x is in the range from 0 to P_Width-1) can have a range from (-m, y) to (-1, y) (resizing to the left) or from (P_Width, y) to (P_Width+m-1, y) (resizing to the right). The position (x') of the region used to obtain the data of the resized region can be derived from the formula x'=(x+P_Width)%P_Width. In this case, x represents the coordinates of the resized region relative to the image before resizing, and x' represents the coordinates of the region referencing the resized region relative to the image before resizing. For example, when the image size is adjusted to the left, m is 4, and the image width is 16, the corresponding data (-4, y) can be obtained from (12, y), the corresponding data (-3, y) can be obtained from (13, y), the corresponding data (-2, y) can be obtained from (14, y), and the corresponding data (-1, y) can be obtained from (15, y). Alternatively, when the image size is adjusted to the right, m ​​is 4, and the image width is 16, the corresponding data (16, y) can be obtained from (0, y), the corresponding data (17, y) can be obtained from (1, y), the corresponding data (18, y) can be obtained from (2, y), and the corresponding data (19, y) can be obtained from (3, y).

[0552] When the size or length of the resized region is n, the coordinates of the resized region relative to the image before resizing (here, y is in the range from 0 to P_Height-1) can have a range from (x, -n) to (x, -1) (resize upwards) or from (x, P_Height) to (x, P_Height+n-1) (resize downwards). The position (y') of the region used to obtain the data of the resized region can be derived from the formula y'=(y+P_Height)%P_Height. In this case, y represents the coordinates of the resized region relative to the image before resizing, and y' represents the coordinates of the region referencing the resized region relative to the image before resizing. For example, when resizing the image upwards, n is 4, and the image height is 16, the corresponding data (x, -4) can be obtained from (x, 12), the corresponding data (x, -3) can be obtained from (x, 13), the corresponding data (x, -2) can be obtained from (x, 14), and the corresponding data (x, -1) can be obtained from (x, 15). Alternatively, when the image size is adjusted downwards, n is 4 and the image height is 16, the corresponding data (x, 16) can be obtained from (x, 0), the corresponding data (x, 17) can be obtained from (x, 1), the corresponding data (x, 18) can be obtained from (x, 2), and the corresponding data (x, 19) can be obtained from (x, 3).

[0553] After filling the resized region with data, you can perform resizing relative to the coordinates of the resized image (here, x is in the range from 0 to P'_Width-1, and y is in the range from 0 to P'_Height-1). This example can be applied to both latitude and longitude coordinate systems.

[0554] Various size adjustment combinations are available as follows.

[0555] As an example, the image size can be adjusted to the left by m. Alternatively, the image size can be adjusted to the right by n. Alternatively, the image size can be adjusted upwards by o. Alternatively, the image size can be adjusted downwards by p.

[0556] As an example, the image size can be adjusted to the left by m and to the right by n. Alternatively, the image size can be adjusted upwards by o and downwards by p.

[0557] As an example, the image size can be adjusted to the left by m, to the right by n, and upwards by o. Alternatively, the image size can be adjusted to the left by m, to the right by n, and downwards by p. Alternatively, the image size can be adjusted to the left by m, upwards by o, and downwards by p. Alternatively, the image size can be adjusted to the right by n, upwards by o, and downwards by p.

[0558] As an example, the image size can be adjusted to the left by m, to the right by n, up by o, and down by p.

[0559] Similar to the examples above, at least one resizing operation can be performed. Image resizing can be performed implicitly based on encoding / decoding settings, or resizing information can be generated implicitly, and then image resizing can be performed based on the generated resizing information. That is, m, n, o, and p in the examples above can be determined as predetermined values, or they can be explicitly generated using the resizing information. Alternatively, some of m, n, o, and p can be determined as predetermined values, and other values ​​can be explicitly generated.

[0560] The above example has focused on acquiring data from specific regions of an image, but other methods can also be applied. The data can be pixels before or after encoding, and can be determined based on the resizing step or the characteristics of the image to be resized. For example, when resizing is performed in preprocessing and pre-encoding steps, the data can refer to input pixels of the projected image, packed image, etc., and when resizing is performed in post-processing, intra-frame prediction reference pixel generation, reference image generation, filtering, etc., the data can refer to the recovered pixels. Furthermore, resizing can be performed in each resized region by using a separate data processing method.

[0561] Figure 24 This is a conceptual diagram illustrating the continuity between surfaces under a projection format (e.g., CHP, OHP, or ISP) according to an embodiment of the present invention.

[0562] In detail, Figure 24 Examples of images composed of multiple faces can be shown. Continuity can be a property generated in adjacent regions in 3D space. Sections 24a to 24c show, respectively, cases with both spatial adjacency and continuity when transformed into 2D space by projection processing (A), cases with spatial adjacency but no continuity (B), cases with no spatial adjacency but continuity (C), and cases with neither spatial adjacency nor continuity (D). In contrast, images in general are classified into cases with both spatial adjacency and continuity (A) and cases with neither spatial adjacency nor continuity (D). In this case, the cases with continuity correspond to some examples (A or C).

[0563] That is, referring to sections 24a to 24c, cases with both spatial adjacency and continuity (described here with reference to section 24a) can be shown as b0 to b4, and cases without spatial adjacency but with continuity can be shown as B0 to B6. In other words, these cases indicate adjacent regions in 3D space, and coding performance can be enhanced by using the continuity characteristic of b0 to b4 and B0 to B6 in the coding process.

[0564] Figure 25 This is a conceptual diagram illustrating the surface continuity in part 21c, which is an image obtained through image reconstruction processing or region packing processing under the CMP projection format.

[0565] Here, Figure 21 Part 21c shows a rearrangement of the 360-degree image unfolded in the cube shape in part 21a, thus preserving the applied... Figure 21 The surface continuity of part 21a. That is, as shown in part 25a, surface S2,1 can be horizontally continuous with surfaces S1,1 and S3,1, and can be vertically continuous with surfaces S1,0 rotated by 90 degrees and S1,2 rotated by -90 degrees.

[0566] In the same manner, the continuity of surfaces S3,1, S0,1, S1,2, S1,1 and S1,0 can be checked in parts 25b to 25f.

[0567] The continuity between surfaces can be defined according to projection format settings, etc. However, the present invention is not limited to this and can be modified. Assumptions will be made as follows... Figure 24 and Figure 25 The following example describes the situation where continuity exists as shown.

[0568] Figure 26 This is an example diagram illustrating image size adjustment under the CMP projection format according to an embodiment of the present invention.

[0569] Section 26a shows an example of adjusting the image size, section 26b shows an example of adjusting the size of the surface unit (or partitioning unit), and section 26c shows an example of adjusting the size of the image and the surface unit (or an example of performing multiple size adjustments).

[0570] The size of a projected image can be adjusted using either a scaling factor or an offset factor, depending on the image size adjustment type. Here, the image before size adjustment can be P_Width × P_Height, and the image after size adjustment can be P'_Width × P'_Height, with the size of the face being F_Width × F_Height. The size can be the same or different depending on the face, and the width and height can also be the same or different depending on the face. However, for ease of description, this example will be described assuming all faces in the image have the same size and square shape. Furthermore, the description assumes the adjusted values ​​(WX and HY in this case) are the same. In the examples below, the focus will be on the case of the offset factor, and also on the methods of filling specific regions of the image by copying and by transforming the filling methods of specific regions of the image. The above settings can even be applied to... Figure 27 The situation shown in the figure.

[0571] For portions 26a to 26c, the boundary of a surface may be continuous with the boundary of another surface (here, it is assumed that it has a corresponding boundary). Figure 24 (Continuity of part 24a). Here, continuity can be classified into the case of having spatial adjacency and image continuity in the 2D plane (first example) and the case of having image continuity but no spatial adjacency in the 2D plane (second example).

[0572] For example, when assuming Figure 24 In the continuity of part 24a, the upper, left, right and lower regions of S1,1 can be spatially adjacent to the lower, right, left and upper regions of S1,0,S0,1,S2,1 and S1,2 and have image continuity with the lower, right, left and upper regions of S1,0,S0,1,S2,1 and S1,2 (first example).

[0573] Alternatively, the left and right regions of S1,0 are not spatially adjacent to the upper regions of S0,1 and S2,1, but can have image continuity with the upper regions of S0,1 and S2,1 (second example). Furthermore, the left regions of S0,1 can be spatially non-adjacent to each other, but have image continuity with each other (second example). Additionally, the left and right regions of S1,2 can be continuous with the lower regions of S0,1 and S2,1 (second example). This can only be a limited example, and other configurations can be applied depending on the definition and settings of the projection format. For ease of description, S0,0 to S3,2 in part 26a are referred to as a to l.

[0574] Part 26a can be an example of a filling method using data from regions having continuity toward the outer boundary of the image. The range from region A (excluding data) to the resized regions (here, a0 to a2, c0, d0 to d2, i0 to i2, k0, and l0 to l2) can be filled with any predetermined value or by filling with outer pixels, and the range from region B (including actual data) to the resized regions (here, b0, e0, h0, and j0) can be filled with data from regions (or faces) having image continuity. For example, b0 can be filled with data from the upper side of face h, e0 with data from the right side of face h, h0 with data from the left side of face e, and j0 with data from the lower side of face h.

[0575] Specifically, as an example, b0 can be filled with data of the lower side of the face obtained by rotating face h by 180 degrees, and j0 can be filled with data of the upper side of the face obtained by rotating face h by 180 degrees. However, this example (including the following examples) may only represent the position of the reference face, and may be considered in the context of... Figure 24 and Figure 25 Data is obtained from the sized region after the continuity between the surfaces is adjusted (e.g., rotated, etc.).

[0576] Part 26b can be an example of a filling method using data from regions with continuity toward the inner boundaries of the image. In this example, different resizing operations can be performed for each face. A shrinking process can be performed in region A, and an expanding process can be performed in region B. For example, the size of face a can be adjusted to the right (here, shrinking) w0, and the size of face b can be adjusted to the left (here, expanding) w0. Alternatively, the size of face a can be adjusted downwards (here, shrinking) h0, and the size of face e can be adjusted upwards (here, expanding) h0. In this example, when observing the change in the width of the image through faces a, b, c, and d, face a shrinks by w0, face b expands by w0 and w1, and face c can shrink by w1. Therefore, the width of the image before resizing is the same as the width of the image after resizing. When observing the change in the height of the image through faces a, e, and i, face a shrinks by h0, face e expands by h0 and h1, and face i can shrink by h1. Therefore, the height of the image before resizing is the same as the height of the image after resizing.

[0577] Considering that the region shrinks from region A, which does not contain data, the resized regions (here b0, e0, be, b1, bg, g0, h0, e1, ej, j0, gi, g1, j1, and h1) can be simply removed, and considering that the region expands from region B, which contains actual data, the resized regions can be filled with data from regions with continuity.

[0578] For example, b0 can be filled with the data on the top side of face e; e0 can be filled with the data on the left side of face b; be can be filled with the data of the left side of face b, the top side of face e, or the weighted sum of the left side of face b and the top side of face e; b1 can be filled with the data on the top side of face g; bg can be filled with the data of the left side of face b, the top side of face g, or the weighted sum of the right side of face b and the top side of face g; g0 can be filled with the data on the right side of face b; h0 can be filled with the data on the top side of face b; e1 can be filled with the data on the left side of face j; ej can be filled with the data of the bottom side of face e, the left side of face j, or the weighted sum of the bottom side of face e and the left side of face j; j0 can be filled with the data on the bottom side of face e; gj can be filled with the data of the bottom side of face g, the left side of face j, or the weighted sum of the bottom side of face g and the right side of face j; g1 can be filled with the data on the right side of face j; j1 can be filled with the data on the bottom side of face g; and h1 can be filled with the data on the bottom side of face j.

[0579] In the example above, when a resized region is filled with data from a specific region of the image, the data from that region can be copied and then used to fill the resized region, or it can be transformed based on the image's characteristics, type, etc., and then used to fill the resized region. For example, when a 360-degree image can be transformed into 2D space according to a projection format, a coordinate system (e.g., a 2D planar coordinate system) can be defined for each face. For ease of description, assume that (x, y, z) in 3D space is transformed into (x, y, C), (x, C, z), or (C, y, z) for each face. The example above indicates the following situation: data from faces other than the corresponding face is obtained from the resized region of a face. That is, when resizing is performed on the current face, data from other faces with different coordinate system characteristics can be copied as is and then used. In this case, there is a possibility that continuity may be distorted based on the resizing boundary. Therefore, data from other faces obtained according to the coordinate system characteristics of the preceding face can be transformed and used to fill the resized region. This transformation is only an example of a data processing method, and the invention is not limited thereto.

[0580] When data from a specific region of an image is copied and used to fill a resized region, distorted continuity (or fundamentally altered continuity) may be included in the boundary region between the resized region (e) and the resized region (e0). For example, continuity may change relative to the boundary, and straight edges may be curved relative to the boundary.

[0581] When data from a specific region of an image is transformed and used to fill a resized region, a gradually changing continuity may be included in the boundary regions between the resized regions.

[0582] The above examples can be seen as examples of the data processing method of the present invention for transforming data of a specific region of an image based on the characteristics, type, etc. of the image and filling the resized region with the transformed data.

[0583] Part 26c may be an example of combining the image resizing processes corresponding to parts 26a and 26b to fill the resized region with data from regions having continuity toward the boundaries (inner and outer boundaries) of the image. This example of resizing processing can be derived from the resizing processes of parts 26a and 26b, and its detailed description will be omitted.

[0584] Part 26a may be an example of a process for adjusting the image size, and part 26b may be an example of adjusting the size of the partitioning units in the image. Part 26c may be an example of multiple size adjustment processes that include the process for adjusting the image size and the process for adjusting the size of the partitioning units in the image.

[0585] For example, the size of an image acquired through projection processing (here, the first format) (region C) can be adjusted, and the size of an image acquired through format transformation processing (here, the second format) (region D) can be adjusted. In this example, the size of an image projected via ERP (here, the entire image) can be adjusted, and the image can be transformed into an image projected via CMP by the format transformation unit, and the size of the image projected via CMP (here, the surface unit) can be adjusted. The above example is an example of performing multiple size adjustment operations. However, the present invention is not limited thereto and can be modified thereto.

[0586] Figure 27 This is an example diagram illustrating the resizing of an image transformed and packaged in CMP projection format according to an embodiment of the present invention. Figure 27 Furthermore, assuming that Figure 25 The continuity between the faces shown means that the boundary of one face can be continuous with the boundary of another face.

[0587] In this example, the offset factors for W0 to W5 and H0 to H3 can have various values ​​(here, it is assumed that the offset factors are used as size adjustment values). For example, the offset factors can be derived from predetermined values, the motion search range of inter-frame prediction, units obtained from the image partitioning unit, etc., and other cases are also possible. In this case, the units obtained from the pixel partitioning unit can include faces. That is, the size adjustment value can be determined based on F_Width and F_Height.

[0588] Section 27a is an example of individually adjusting the size of a single face (here, relative to facing up, down, left, and right) and filling an extended region with data of continuous areas. For example, the outer regions a0 to a6 of face a can be filled with continuous data, and the outer regions b0 to b6 of face b can be filled with continuous data.

[0589] Section 27b is an example of adjusting the size of multiple faces (here, relative to the multiple faces upward, downward, left, and right) and filling an extended region with data of regions with continuity. For example, faces a, b, and c can be extended to the outer regions a0 to a4, b0 and b1, and c0 to c4.

[0590] Part 27c could be an example of adjusting the size of the entire image (here, relative to the entire image up, down, left, and right) and filling the extended region with data from regions with continuity. For example, the entire image consisting of faces a to f could be extended to the outer regions a0 to a2, b0, c0 to c2, d0 to d2, and f0 to f2.

[0591] That is, sizing can be performed in a single face cell, in multiple face cells that are continuous with each other, and in the entire image cell.

[0592] In the example above, the resized region (here, a0 to f7) can be filled with data of a continuous region (or face), as shown in section 24a. That is, the resized region can be filled with data of the top, bottom, left, and right sides of faces a to f.

[0593] Figure 28 This is an example diagram illustrating a data processing method for adjusting the size of a 360-degree image according to an embodiment of the present invention.

[0594] Reference Figure 28 Region B (a0 to a2, ad0, b0, c0 to c2, cf1, d0 to d2, e0, f0 to f2) can be filled as a resized region using data from regions that are continuous between pixel data belonging to a to f. Furthermore, region C (ad1, be, cf0) can be filled as another resized region using data from the region to be resized and data from spatially adjacent but not continuous regions. Alternatively, since resizing is performed between two regions selected from a to f (e.g., a and d, b and e, and c and f), region C can be filled using multiple data from the two regions in combination. For example, faces b and e can be spatially adjacent but not continuous. The size of the resized region located between faces b and e can be adjusted using data from faces b and e. For example, the region can be filled with a value obtained by averaging the data from faces b and e, or with a value obtained by a distance-based weighted sum. In this case, the pixels used to fill the sized regions in faces b and e can be either boundary pixels of each face or interior pixels of each face.

[0595] In summary, the resized regions between image partitioning units can be filled with data generated by combining multiple data from two units.

[0596] Under certain conditions (here, when resizing multiple regions), data processing methods can be supported.

[0597] In sections 27a and 27b, the regions to be resized between the partitioning units are constructed separately for each partitioning unit (in section 27a, a6 and d1 are constructed for a and d respectively). Figure 28 In this context, a single region to be resized between adjacent partitioning units can be constructed (ad1 is constructed for a and d). It should be understood that this method can be included in the candidate group of data processing methods in parts 27a and 27b, and even in... Figure 28 In this case, different data processing methods can be used to perform size adjustments than in the example above.

[0598] In the image resizing process according to the present invention, a predetermined data processing method may be implicitly used in the resized region, or one of a plurality of data processing methods may be used for explicitly related information. The predetermined data processing method may be one of the following: a filling method using any pixel value, a filling method by copying external pixels, a filling method by copying a specific region of the image, a filling method by transforming a specific region of the image, a filling method using data derived from multiple regions of the image, etc. For example, when the resized region is located inside an image (e.g., a packaged image) and the regions on either side (e.g., faces) are spatially adjacent but not continuous, a data processing method may be used to fill the resized region with data derived from multiple regions. Furthermore, resizing can be performed using one of a plurality of data processing methods, and related selection information can be explicitly generated. This can be an example applicable to general images as well as 360-degree images.

[0599] The encoder can add information generated during the above processing to the bitstream in units of at least one of sequences, images, slices, tiles, etc., and the decoder can parse the relevant information from the bitstream. Furthermore, the information can be included in the bitstream in the form of SEI or metadata. Segmentation, reconstruction, and resizing processes for 360-degree images have been described focusing on some projection formats such as ERP and CMP. However, the invention is not limited thereto, and the above description can be applied to other projection formats either as is or with modifications.

[0600] The image setup processing for the aforementioned 360-degree image encoding / decoding device has been described as being applicable to preprocessing, postprocessing, format conversion processing, inverse format conversion processing, and encoding / decoding processing.

[0601] In summary, projection processing can be structured to include image setup processing. Specifically, in addition to at least one of the image setup processes, projection processing can also be performed. Partitioning can be performed on the projected image at the region (or surface) level. Depending on the projection format, partitioning can be performed on a single region or multiple regions. For partitioning, partitioning information can be generated. Furthermore, the size of the projected image or the size of the projected region can be adjusted. In this case, size adjustment can be performed on at least one region. For size adjustment, size adjustment information can be generated. Additionally, the projected image can be reconstructed (or surface-to-surface arrangement), or the projected region can be reconstructed. In this case, reconstruction can be performed on at least one region. For reconstruction, reconstruction information can be generated.

[0602] In summary, a regional packing process can be constructed to include image setup processing. Specifically, in addition to at least one of the image setup processes, regional packing projection processing can also be performed. Division processing can be performed on the packing image at the region (or surface) level. Depending on the regional packing settings, division can be performed on a single region or multiple regions. For division, division information can be generated. Furthermore, the size of the packing image or the size of the packing region can be adjusted. In this case, size adjustment can be performed on at least one region. For size adjustment, size adjustment information can be generated. Additionally, the packing image or the packing region can be reconstructed. In this case, reconstruction can be performed on at least one region. For reconstruction, reconstruction information can be generated.

[0603] During projection processing, all or some image settings processing can be performed, and image settings information can be included. This information can be settings specific to the projected image. More specifically, this information can be settings specific to a region within the projected image.

[0604] During regional packing processing, all or some image setup processing can be performed, and image setup information can be included. This information can be setup information specific to the packed image. More specifically, this information can be setup information specific to a region within the packed image. Alternatively, this information can be mapping information between the projected image and the packed image (see, for example, reference...). Figure 11 The description can be understood as assuming P0 and P1 indicate the projected image, and S0 to S5 indicate the packaged image. More specifically, this information can be mapping information between a specific region in the projected image and a specific region in the packaged image. That is, this information can be setting information for assigning a specific region from the projected image to a specific region in the packaged image.

[0605] Image information can be represented as information obtained through the various embodiments described above during the image setting process of the present invention. For example, when using at least one syntax element from Tables 1 to 6 to represent relevant information, the setting information of the projected image may include pic_width_in_samples, pic_height_in_samples, part_top[i], part_left[i], part_width[i], part_height[i], etc., and the setting information of the packaged image may include pic_width_in_samples, pic_height_in_samples, part_top[i], part_left[i], part_width[i], part_height[i], convert_type_flag[i], part_resizing_flag[i], top_height_offset[i], bottom_height_offset[i], left_width_offset[i], right_width_offset[i], resizing_type_flag[i], etc. The above example could be an example of explicitly generating information about the face (e.g., part_top[i], part_left[i], part_width[i], and part_height[i] in the settings of a projected image).

[0606] Some of the image setup processing can be included in the projection processing or region packing processing corresponding to the projection format through predefined operations.

[0607] For example, ERP uses a method of filling regions that expand left by m and right by n using data from regions in the direction opposite to the image's resizing direction, thus implicitly including resizing processing. Alternatively, CMP uses a method of filling regions that expand upward by m, downward by n, left by o, and right by p using data from regions that are continuous with the resizing region, thus implicitly including resizing processing.

[0608] In the examples above, the projection format may be an example of an alternative format that can replace the conventional projection format or an example of an additional format (e.g., ERP1 and CMP1) for the conventional projection format. However, the invention is not limited thereto, and various examples of image setting processes of the invention may be alternatively combined, and similar applications are possible for other formats.

[0609] Although not in Figure 1 and Figure 2 The image encoding and decoding apparatus are shown, but may also include a block partitioning unit. Information about a default encoding unit can be obtained from the image partitioning unit, and the default encoding unit may refer to the default (or starting) unit used for prediction, transformation, quantization, etc., during image encoding / decoding processing. In this case, the encoding unit may consist of one luma encoding block and two chroma encoding blocks according to the color format (here, YCbCr), and the size of the block may be determined according to the color format. The following example describes the block (here, the luma component). In this case, it is assumed that the block is a unit that can be obtained after each unit is determined, and it is also assumed that a similar setup applies to other types of blocks.

[0610] A block division section can be configured in association with each element of the image encoding and image decoding apparatus. Through this process, the size and shape of the blocks can be determined. In this case, different blocks can be defined for each element. A block can be a prediction block for the prediction section, a transformation block for the transformation section, a quantization block for the quantization section, etc. However, the invention is not limited to this, and additional block units can be defined for other elements. The size and shape of the block can be defined by the width and height of the block.

[0611] A block can be represented as M×N by its block partitioning and can be obtained within a range from the minimum to the maximum value. For example, when the block supports a square shape and has a maximum value of 256×256 and a minimum value of 8×8, a size of 2 can be obtained. m ×2 m Blocks of size 2m×2m (where m is an integer from 3 to 8; for example, 8×8, 16×16, 32×32, 64×64, 128×128, and 256×256), blocks of size 2m×2m (where m is an integer from 4 to 128), or blocks of size m×m (where m is an integer from 8 to 128). Alternatively, when the blocks support square and rectangular forms and have the same range as above, blocks of size 2m×2m can be obtained. m ×2 n The block size can be 8×8, 8×16, 16×8, 16×16, 16×32, 32×16, 32×32, 32×64, 64×32, 64×64, 64×128, 128×64, 128×128, 128×256, 256×128, or 256×256. The aspect ratio can be unrestricted, or a maximum aspect ratio can exist based on the encoding / decoding settings. Alternatively, a block size of 2m×2n can be obtained (where m and n are integers from 4 to 128). Alternatively, a block size of m×n can be obtained (where m and n are integers from 8 to 256).

[0612] The available blocks can be determined based on encoding / decoding settings (e.g., block type, partitioning scheme, partitioning settings, etc.). For example, blocks of size 2 can be obtained. m ×2 n The block is used as the coding block, and blocks of size 2m×2n or m×n can be obtained as prediction blocks, and blocks of size 2m×2n or m×n can be obtained as prediction blocks. m ×2 n The block is used as a transform block. Information about the size and range of the block (e.g., information related to exponents and multiples) can be generated based on the setting...

Claims

1. A method for decoding a 360-degree image, the method comprising: Receive a bitstream encoded with the 360-degree image, the bitstream including data of an extended two-dimensional image, the extended two-dimensional image including a two-dimensional image and a predetermined extended region, and the two-dimensional image being projected from an image having a three-dimensional projection structure and including one or more surfaces; A predicted image is generated by performing a prediction based on information about the prediction included in the bitstream. as well as The extended two-dimensional image is reconstructed based on the predicted image and the residual image. The size of the extended region to be filled is determined based on a first width information of the extended region on the left side of the face and a second width information of the extended region on the right side of the face, both of which are obtained separately from the bitstream. The number of syntax elements used to signal the size of the extended region is determined differently based on the projection format used for the three-dimensional projection structure. This projection format is one of several, including ERP (Projection Processing) format which projects the 360-degree image onto a two-dimensional plane and CMP (Projection Processing) format which projects the 360-degree image onto a cube. The predicted image is added to the residual image to reconstruct the extended two-dimensional image. The residual image is obtained by decoding the residual information included in the bitstream, and The predicted image is generated by performing either intra-frame prediction or inter-frame prediction.

2. The method according to claim 1, wherein, The sample values ​​of the extended region are determined differently depending on the filling method selected from multiple filling methods.

3. The method according to claim 2, wherein, The padding method is selected from the plurality of padding methods based on selection information obtained from the bitstream.

4. The method according to claim 2, wherein, The plurality of filling methods include a filling method that horizontally copies the sample values ​​of the face to the sample values ​​of the extended region.

5. A method for encoding a 360-degree image, the method comprising: Obtain a two-dimensional image projected from an image having a three-dimensional projection structure and including at least one face; Obtain an extended two-dimensional image including the two-dimensional image and the predetermined extended region; A predicted image is generated by performing a prediction, and information about the prediction is encoded into a bitstream; as well as The data of the extended two-dimensional image is encoded into the bitstream based on the predicted image and the residual image. The size of the extended region to be filled is encoded based on a first width information of the extended region on the left side of the face and a second width information of the extended region on the right side of the face. Both the first width information and the second width information are encoded separately into the bitstream. The number of syntax elements used to signal the size of the extended region is determined differently based on the projection format used for the three-dimensional projection structure. This projection format is one of several, including ERP (Projection Processing) format which projects the 360-degree image onto a two-dimensional plane and CMP (Projection Processing) format which projects the 360-degree image onto a cube. The residual image is obtained based on the extended 2D image and the predicted image. Specifically, by encoding the residual image, residual information is included in the bitstream, and The predicted image is generated by performing either intra-frame prediction or inter-frame prediction.

6. A method for transmitting a bit stream, comprising: The method for encoding a 360-degree image according to claim 5 is used to generate a bitstream; as well as Transmit the bit stream.

Images