Image data encoding / decoding method and apparatus

Abstract

To provide an image data encoding / decoding method and an apparatus for improving compression performance.SOLUTION: A 360-degree image decoding method includes: receiving a bitstream obtained by encoding a 360-degree image; referring to syntax information acquired from the bitstream to create a prediction image; coupling the prediction image to a residual image obtained by performing inverse quantization and inverse transformation of the bitstream to acquire a decoded image; and reconstructing the image into a 360-degree image according to a projection format. The creation of the prediction image includes: acquiring, from motion information included in the syntax information, a motion vector candidate group including a motion vector of a block adjacent to a current block to be decoded; deriving a prediction motion vector from the motion vector candidate group on the basis of selection information extracted from the motion information; and determining a prediction block for the current block to be decoded by using a final motion vector derived by adding the prediction motion vector to a differential motion vector extracted from the motion information.SELECTED DRAWING: Figure 44

Description

Technical Field

[0001] The present invention relates to image data encoding and decoding technology, and more particularly, to a method and apparatus for processing encoding and decoding of 360-degree images for immersive media services.

Background Art

[0002] With the spread of the Internet and mobile terminals and the development of information and communication technology, the use of multimedia data has increased rapidly. Recently, demands for high-resolution images and high-quality images, such as HD (High Definition) images and UHD (Ultra High Definition) images, have arisen in various fields, and the demand for immersive media services such as virtual reality and augmented reality has increased rapidly. In particular, in the case of 360-degree images for virtual reality and augmented reality, since multi-view images captured by a plurality of cameras are processed, the amount of data generated thereby increases enormously, but the performance of the image processing system for processing this is insufficient.

[0003] As described above, in the conventional image encoding / decoding method and apparatus, improvement in performance for image processing, particularly image encoding / decoding, is required.

Summary of the Invention

Problems to be Solved by the Invention

[0004] The present invention is for solving the problems as described above, and an object thereof is to provide a method for improving an image setting process in an initial stage of encoding and decoding. More specifically, it is to provide an encoding and decoding method and apparatus for improving an image setting process considering the characteristics of 360-degree images.

Means for Solving the Problems

[0005] One aspect of the present invention for achieving the above object is to provide a method for decoding a 360-degree image.

[0006] Here, the method for decoding a 360-degree image may include the steps of: receiving a bitstream in which a 360-degree image has been encoded; generating a predicted image by referring to syntactic information obtained from the received bitstream; obtaining a decoded image by combining the generated predicted image with a residual image obtained by inverse quantization and inverse transformation of the bitstream; and reconstructing the decoded image into a 360-degree image in a projection format.

[0007] Here, the syntax information can include projection format information for the 360-degree image.

[0008] Here, the projection format information may refer to at least one of the following: the ERP (Equi-Rectangular Projection) format, which projects the 360-degree image onto a two-dimensional plane; the CMP (CubeMap Projection) format, which projects the 360-degree image onto a cube; the OHP (OctaHedron Projection) format, which projects the 360-degree image onto an octahedron; and the ISP (IcoSahedral Projection) format, which projects the 360-degree image onto a polyhedron.

[0009] Here, the reconstruction step may include the step of obtaining placement information by regional packing by referring to the syntactic information, and the step of rearranging each block of the decoded image based on the placement information.

[0010] Here, the step of generating the predicted image may include the step of performing image augmentation on a reference picture obtained by restoring the bitstream, and the step of generating a predicted image by referring to the image augmented reference picture.

[0011] Here, the step of performing image augmentation may include a step of performing image augmentation based on the division units of the reference image.

[0012] In this step, the image expansion is performed based on the division units, and the boundary pixels of the division units are used to generate regions that are individually expanded for each division unit.

[0013] Here, the extended region can be generated using boundary pixels of division units that are spatially adjacent to the division unit being extended, or boundary pixels of division units that have image continuity with the division unit being extended.

[0014] In this step, the image expansion based on the division units can be performed by using boundary pixels of regions formed by combining two or more spatially adjacent division units to generate an expanded image for the combined region.

[0015] In this step, the image expansion based on the division units can generate an expanded region between adjacent division units by using all the adjacent pixel information of spatially adjacent division units.

[0016] In this step, the image expansion based on the division units can be performed by generating the expanded region using the average value of adjacent pixels of each of the spatially adjacent division units.

[0017] Here, the step of generating the predicted image may include the steps of: obtaining a group of motion vector candidates from the motion information contained in the syntactic information, including motion vectors of blocks adjacent to the current block to be decoded; deriving a predicted motion vector from the group of motion vector candidates based on selection information extracted from the motion information; and determining the predicted block of the current block to be decoded using the final motion vector derived by adding the predicted motion vector with the difference motion vector extracted from the motion information.

[0018] Here, the motion vector candidate group can consist only of motion vectors for blocks that belong to a surface with image continuity with the surface to which the current block belongs, if the block adjacent to the current block is on a different surface from the surface to which the current block belongs.

[0019] Here, the adjacent block can mean a block adjacent to the current block in at least one of the following directions: upper left, top, upper right, left, and lower left.

[0020] Here, the final motion vector can point to a reference region that, relative to the current block, belongs to at least one reference picture and is set to a region where there is image continuity between surfaces according to the projection format.

[0021] Here, the reference picture is expanded in the upward, downward, left, and right directions based on the image continuity according to the projection format, after which the reference region can be set.

[0022] Here, the reference picture is extended on a surface-by-surface basis, and the reference region can be set to extend across the boundary of the surface.

[0023] Here, the motion information may include at least one of the following: the list of reference pictures to which the reference picture belongs, the index of the reference picture, and a motion vector pointing to the reference area.

[0024] Here, the step of generating a predicted block for the current block may include dividing the current block into a plurality of subblocks and generating a predicted block for each of the divided subblocks. [Effects of the Invention]

[0025] When using the image coding / decoding method and apparatus according to the embodiments of the present invention described above, compression performance can be improved. In particular, compression performance can be improved for 360-degree images. [Brief explanation of the drawing]

[0026] [Figure 1] This is a block diagram of an image encoding device according to one embodiment of the present invention. [Figure 2] This is a block diagram of an image decoding device according to one embodiment of the present invention. [Figure 3] This is an example diagram illustrating how image information can be divided hierarchically to compress an image. [Figure 4] This is a conceptual diagram illustrating various examples of image segmentation related to one embodiment of the present invention. [Figure 5] This is another illustrative diagram of an image segmentation method according to one embodiment of the present invention. [Figure 6] This is an example diagram illustrating a common method for resizing images. [Figure 7] This is an illustrative diagram of image size adjustment according to one embodiment of the present invention. [Figure 8] This is an illustrative diagram of a method for configuring an expanded region in an image size adjustment method according to one embodiment of the present invention. [Figure 9] This is an illustrative diagram of a method for configuring the region to be deleted and the region to be generated by reduction in an image size adjustment method according to one embodiment of the present invention. [Figure 10] This is an illustrative diagram of image reconstruction according to one embodiment of the present invention. [Figure 11] This is an illustrative diagram showing images before and after the image setting process related to one embodiment of the present invention. [Figure 12] This is an illustrative diagram showing size adjustment for each division unit within an image according to one embodiment of the present invention. [Figure 13] This is an example diagram illustrating the size adjustment or setting of the division units within an image. [Figure 14] This is an illustrative diagram that shows both the image size adjustment process and the size adjustment process for the division units within the image. [Figure 15] This is an illustrative diagram showing a 3D image in a 3D space and a 2D planar space. [Figure 16a]This is a conceptual diagram illustrating the projection format for one embodiment of the present invention. [Figure 16b] This is a conceptual diagram illustrating the projection format for one embodiment of the present invention. [Figure 16c] This is a conceptual diagram illustrating the projection format for one embodiment of the present invention. [Figure 16d] This is a conceptual diagram illustrating the projection format for one embodiment of the present invention. [Figure 17] This is a conceptual diagram illustrating how a projection format according to one embodiment of the present invention is contained within a rectangular image. [Figure 18] This is a conceptual diagram of a method for converting a projection format to a rectangular shape according to one embodiment of the present invention, wherein the surface is rearranged to eliminate meaningless areas. [Figure 19] This is a conceptual diagram showing a regional packing process performed using a CMP projection format according to one embodiment of the present invention as a rectangular image. [Figure 20] This is a conceptual diagram of the division of a 360-degree image according to one embodiment of the present invention. [Figure 21] This is an illustrative diagram of the division and reconstruction of a 360-degree image according to an embodiment of the present invention. [Figure 22] This is an example diagram showing an image projected or packed using CMP divided into tiles. [Figure 23] This is a conceptual diagram illustrating an example of resizing a 360-degree image according to one embodiment of the present invention. [Figure 24] This is a conceptual diagram illustrating the continuity between surfaces in a projection format (e.g., CMP, OHP, ISP) according to one embodiment of the present invention. [Figure 25] This is a conceptual diagram illustrating the surface continuity of Figure 21c, which is an image obtained by the image reconstruction process or regional packing process in the CMP projection format. [Figure 26] This is an illustrative diagram illustrating image resizing in a CMP projection format according to one embodiment of the present invention. [Figure 27] This is an illustrative diagram illustrating size adjustment for an image that has been converted to and packed into a CMP projection format according to one embodiment of the present invention. [Figure 28] This is an illustrative diagram illustrating a data processing method for resizing a 360-degree image according to one embodiment of the present invention. [Figure 29] This is an illustrative diagram showing a tree-based block structure. [Figure 30] This is an illustrative diagram showing a type-based block shape. [Figure 31] This is an illustrative diagram showing various block shapes that can be obtained with the block division section of the present invention. [Figure 32] This is an illustrative diagram illustrating a tree-based partitioning mechanism related to one embodiment of the present invention. [Figure 33] This is an illustrative diagram illustrating a tree-based partitioning mechanism related to one embodiment of the present invention. [Figure 34] This is an illustrative diagram showing various cases in which prediction blocks are obtained through inter-screen prediction. [Figure 35] This is an illustrative diagram showing the configuration of a reference picture list according to one embodiment of the present invention. [Figure 36] This is a conceptual diagram showing a motion model outside of movement related to one embodiment of the present invention. [Figure 37] This is an illustrative diagram showing motion estimation at the subblock level according to one embodiment of the present invention. [Figure 38] This is an illustrative diagram showing a block referenced in the current block's motion information prediction according to one embodiment of the present invention. [Figure 39] This is an illustrative diagram showing a block currently referenced in predicting the motion information of a block in a non-movement motion model according to one embodiment of the present invention. [Figure 40] This is an illustrative diagram showing how to perform inter-screen prediction using an extended picture according to one embodiment of the present invention. [Figure 41] This is a conceptual diagram showing an extension of the surface unit according to one embodiment of the present invention. [Figure 42]This is an illustrative diagram showing how to perform inter-screen prediction using an extended image according to one embodiment of the present invention. [Figure 43] This is an illustrative diagram showing how to perform inter-screen prediction using an extended reference picture according to one embodiment of the present invention. [Figure 44] This is an illustrative diagram showing the configuration of a group of candidate motion information predictions for inter-screen prediction in a 360-degree image according to one embodiment of the present invention. [Modes for carrying out the invention]

[0027] While the present invention can be modified in various ways and may have many different embodiments, specific embodiments are described in detail here with illustrations in the drawings. However, this should not be understood as limiting the present invention to specific embodiments, but rather as including all modifications, equivalents, or substitutions that fall within the spirit and technical scope of the present invention.

[0028] Terms such as First, Second, A, and B are used to describe various components, but the components should not be limited by these terms. These terms are used solely for the purpose of distinguishing one component from another. For example, without exceeding the scope of the invention, the First component may be named the Second component, and similarly, the Second component may be named the First component. The term "and / or" includes a combination of multiple related descriptions or any of multiple related descriptions.

[0029] When one component is said to be "linked" or "connected" to another component, it means that it is directly linked or connected to the other component, but it should be understood that there may be other components intervening between them. Conversely, when one component is said to be "directly linked" or "directly connected" to another component, it should be understood that there are no other components intervening between them.

[0030] The terms used herein are used only to describe specific embodiments and do not limit the invention. A singular expression includes plural expressions unless the context clearly indicates otherwise. In this specification, terms such as “includes” or “having” are intended to specify the presence of features, figures, stages, operations, components, parts, or combinations thereof as described in the specification, and should be understood not to preemptively exclude the presence or possibility of adding one or more other features, figures, stages, operations, components, parts, or combinations thereof.

[0031] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by a person of ordinary skill in the art to which this invention pertains. Terms defined in commonly used dictionaries should be interpreted in accordance with their meaning in the context of the relevant art and not in an ideal or overly formal sense unless expressly defined herein.

[0032] Image encoding and decoding devices may be user terminals such as personal computers (PCs), laptops, personal digital assistants (PDAs), portable multimedia players (PMPs), PlayStation Portables (PSPs), wireless communication terminals (Wireless Communication Terminals), smartphones, TVs, virtual reality devices (VR), augmented reality devices (AR), mixed reality devices (MR), head-mounted displays (HMDs), and smart glasses, or server terminals such as application servers and service servers. They may also include various devices such as communication modems for communicating with various devices or wired / wireless networks, memory for storing various programs and data for encoding or decoding images or for predicting images within or between screens for encoding or decoding, and a processor for executing programs, performing calculations, and controlling them. Furthermore, images encoded into a bitstream by an image encoding device can be transmitted in real time or non-real time to an image decoding device via wired or wireless networks such as the Internet, short-range wireless communication systems, wireless LAN networks, WiBro networks, and mobile communication networks, or via various communication interfaces such as cables and Universal Serial Bus (USB), where they can be decoded and restored as images.

[0033] Furthermore, images encoded into a bitstream by an image encoding device can also be transmitted from the encoding device to a decoding device via a computer-readable recording medium.

[0034] The aforementioned image encoding device and image decoding device may be separate devices, but depending on the implementation, they may be combined into a single image encoding / decoding device. In that case, some components of the image encoding device may be substantially the same technical elements as some components of the image decoding device, and can be implemented to include at least the same structure or to perform at least the same function.

[0035] Therefore, in the detailed explanations of the following technical elements and their operating principles, redundant explanations of corresponding technical elements will be omitted.

[0036] Since an image decoding device corresponds to a computer device that applies the image encoding method performed by an image encoding device to decoding, the following explanation will focus on the image encoding device.

[0037] A computer device may include a memory for storing programs or software modules that implement image encoding and / or image decoding methods, and a processor connected to the memory to execute the programs. An image encoding device is sometimes called an encoder, and an image decoding device is sometimes called a decoder.

[0038] Typically, an image can be composed of a series of still images, and these still images can be divided into GOP (Group of Pictures) units. Each still image is sometimes called a picture. In this case, a picture can represent either a frame or a field in a progressive signal or an interlaced signal. An image can be represented as a "frame" when encoding / decoding is done in frame units, and as a "field" when it is done in field units. This invention explains the concepts assuming a progressive signal, but it is also applicable to interlaced signals. Higher-level concepts such as GOP and Sequence can exist. Furthermore, each picture can be divided into predetermined areas such as slices, tiles, and blocks. Also, a single GOP may contain units such as I-pictures, P-pictures, and B-pictures. An I-picture can refer to a picture that is encoded / decoded independently without using a reference picture, while P-pictures and B-pictures can refer to pictures that are encoded / decoded using a reference picture through processes such as motion estimation and motion compensation. Generally, in the case of a P-picture, an I-picture and a P-picture can be used as reference pictures, and in the case of a B-picture, an I-picture and a P-picture can be used as reference pictures, but this definition can also be changed depending on the encoding / decoding settings.

[0039] Here, the picture referenced during encoding / decoding is called the Reference Picture, and the referenced block or pixel is called the Reference Block or Reference Pixel. Furthermore, the referenced data can be not only pixel values ​​in the spatial domain, but also coefficient values ​​in the frequency domain, and various encoding / decoding information generated and determined during the encoding / decoding process. For example, in the prediction unit, this could be in-screen prediction-related information or motion-related information; in the transformation / inverse transformation unit, transformation-related information; in the quantization / inverse quantization unit, quantization-related information; in the encoding / decoding unit, encoding / decoding-related information (context information); and in the in-loop filter unit, filter-related information.

[0040] The smallest unit that makes up an image can be a pixel. The number of bits used to represent a single pixel is called the bit depth. Generally, the bit depth is 8 bits, but depending on the encoding settings, higher bit depths can be supported. Depending on the color space, at least one bit depth can be supported. Also, depending on the color format of the image, it can be composed of at least one color space. Depending on the color format, it can be composed of one or more pictures of a fixed size or one or more pictures of different sizes. For example, in the case of YCbCr4:2:0, it can be composed of one luminance component (Y in this example) and two chrominance components (Cb / Cr in this example). In this case, the ratio of the chrominance component to the luminance component can be 1:2 horizontally to 2 vertically. As another example, in the case of 4:4:4, the horizontal and vertical ratios can be the same. When composed of one or more color spaces as in the above examples, the picture can be divided into each color space.

[0041] In this invention, we will explain based on a certain color format (YCbCr in this example) and a certain color space (Y in this example). The same or similar application (settings dependent on a specific color space) can be made to other color spaces (Cb, Cr in this example) depending on the color format. However, it is also possible to make partial differences for each color space (settings independent of a specific color space). That is, settings dependent on each color space can mean that the settings are proportional to or dependent on the composition ratio of each component (for example, determined according to 4:2:0, 4:2:2, 4:4:4, etc.), while settings independent of each color space can mean that the settings are independent of the composition ratio of each component or that the settings are independent only for the color space in question. In this invention, depending on the encoder / decoder, some configurations may have independent or dependent settings.

[0042] The configuration information or syntax elements required in the image encoding process can be determined at the unit level, such as video, sequence, picture, slice, tile, or block. This information can be recorded in the bitstream in units such as VPS (Video Parameter Set), SPS (Sequence Parameter Set), PPS (Picture Parameter Set), Slice Header, Tile Header, and Block Header, and transmitted to the decoder. The decoder can then parsing the information at the same level to recover the configuration information transmitted from the encoder and use it in the image decoding process. In addition, related information can be transmitted to the bitstream in the form of SEI (Supplement Enhancement Information) or metadata, and can be parsed and used. Each parameter set has a unique ID value, and a sub-parameter set can have the ID value of the higher-level parameter set it references. For example, a sub-parameter set can reference information from one or more higher-level parameter sets that have a matching ID value. Among the various unit examples mentioned above, a unit that contains one or more other units is sometimes called a higher-level unit, and the units it contains are sometimes called lower-level units.

[0043] In the case of setting information generated at the aforementioned unit, it may include content for settings independent of the unit in question, or content for settings that depend on previous, subsequent, or higher-level units. Here, a dependent setting can be understood as flag information indicating that the setting information of the unit in question follows the settings of previous, subsequent, or higher-level units (for example, if a 1-bit flag is 1, the setting is followed; if it is 0, the setting is not followed). The setting information in this invention will be described mainly as an example of independent settings, but it may also include examples in which content for dependent relationships with the setting information of previous, subsequent, or higher-level units of the current unit is added to or replaced.

[0044] Figure 1 is a block diagram of an image encoding device according to one embodiment of the present invention. Figure 2 is a block diagram of an image decoding device according to one embodiment of the present invention.

[0045] Referring to Figure 1, the image coding device can be configured to include a prediction unit, a subtraction unit, a transformation unit, a quantization unit, an inverse quantization unit, an inverse transformation unit, an addition unit, an in-loop filter unit, a memory and / or a coding unit. Of the above configurations, some are not necessarily included, and some or all may be selectively included depending on the implementation, and additional configurations not shown may also be included.

[0046] Referring to Figure 2, the image decoding device can be configured to include a decoding unit, a prediction unit, an inverse quantization unit, an inverse transform unit, an addition unit, an in-loop filter unit, and / or a memory. Some of the above components are not necessarily included, and depending on the implementation, some or all of them may be selectively included, and additional components not shown may also be included.

[0047] Image encoding and decoding devices may be separate devices, but depending on the implementation, they may be combined into a single image encoding / decoding device. In that case, some components of the image encoding device may be substantially the same technical elements as some components of the image decoding device, and can be implemented to include at least the same structure or perform at least the same function. Therefore, in the detailed explanation of the following technical elements and their operating principles, redundant explanations of corresponding technical elements will be omitted. The image decoding device corresponds to a computer device that applies the image encoding method performed by the image encoding device to decoding, so the following explanation will focus on the image encoding device. The image encoding device is sometimes called an encoder, and the image decoding device is sometimes called a decoder.

[0048] The prediction unit can be implemented using a software module called the Prediction Module, and can generate predicted blocks for the blocks to be encoded using either an intra-prediction method or an inter-prediction method. The prediction unit generates predicted blocks by predicting the current block to be encoded in the image. That is, the prediction unit predicts the pixel value of each pixel in the current block to be encoded in the image via intra-prediction or inter-prediction, and generates predicted blocks having the predicted pixel value of each generated pixel. The prediction unit can also transmit the information necessary to generate predicted blocks to the encoding unit to encode information for the prediction mode, record this information in a bitstream and transmit it to the decoder, and the decoder's decoding unit parses this information to restore the information for the prediction mode, which can then be used for intra-prediction or inter-prediction.

[0049] The subtraction unit generates a residual block by subtracting the predicted block from the current block. In other words, the subtraction unit calculates the difference between the pixel value of each pixel in the current block to be encoded and the predicted pixel value of each pixel in the predicted block generated via the prediction unit, thereby generating a residual block, which is a residual signal in block form.

[0050] The conversion unit can convert signals belonging to the spatial domain into signals belonging to the frequency domain. At this time, the signal obtained through the conversion process is called the transformed coefficient. For example, a residual block containing a residual signal transmitted from the subtraction unit can be converted to obtain a transformed block containing the transformed coefficient, but the input signal is determined according to the encoding settings and is not limited to residual signals.

[0051] The transformation unit can transform the residual block using transformation techniques such as the Hadamard Transform, Discrete Sine Transform (DST-Based Transform), and Discrete Cosine Transform (DCT-Based Transform). However, it is not limited to these, and various transformation techniques that are improvements and modifications of these can be used.

[0052] For example, at least one of the above transformation techniques can be supported, and each transformation technique can support at least one detailed transformation technique. In this case, at least one detailed transformation technique may be a transformation technique in which a part of the basis vectors differs for each transformation technique. For example, DST-based transformations and DCT-based transformations can be supported as transformation techniques. In the case of DST, detailed transformation techniques such as DST-I, DST-II, DST-III, DST-V, DST-VI, DST-VII, and DST-VIII can be supported, and in the case of DCT, detailed transformation techniques such as DCT-I, DCT-II, DCT-III, DCT-V, DCT-VI, DCT-VII, and DCT-VIII can be supported.

[0053] Any of the above transformations (for example, one transformation technique and one detailed transformation technique) can be set as the basic transformation technique, and additional transformation techniques (for example, multiple transformation techniques || multiple detailed transformation techniques) can be supported. Whether or not to support additional transformation techniques is determined at the unit level, such as sequences, pictures, slices, and tiles, and related information can be generated at the unit level. If additional transformation techniques are supported, the transformation technique selection information is determined at the unit level, such as blocks, and related information can be generated at the unit level.

[0054] The transformation can be performed in the k / vertical direction. For example, by performing a one-dimensional transformation in the horizontal direction using the basis vectors in the transformation, and then a one-dimensional transformation in the vertical direction, a total two-dimensional transformation can be performed, thereby converting the pixel values ​​in the spatial domain to the frequency domain.

[0055] Furthermore, the horizontal / vertical conversion can be performed adaptively. More specifically, whether or not it is performed adaptively can be determined depending on at least one encoding setting. For example, if the prediction mode in on-screen prediction is horizontal mode, DCT-I can be applied horizontally and DST-I vertically; if the prediction mode in on-screen prediction is vertical mode, DST-VI can be applied horizontally and DCT-VI vertically; if the prediction mode is diagonal down left, DCT-II can be applied horizontally and DCT-V vertically; and if the prediction mode is diagonal down right, DST-I can be applied horizontally and DST-VI vertically.

[0056] The size and shape of each transformation block are determined according to the encoding cost for each candidate size and shape of the transformation block, and the image data and information such as the size and shape of each determined transformation block can be encoded.

[0057] Of the aforementioned transformation shapes, a square transformation can be set as the basic transformation shape, and additional transformation shapes (e.g., rectangular shapes) can be supported. Whether or not to support additional transformation shapes is determined at the sequence, picture, slice, tile, etc., and related information can be generated at the said unit level. Transformation shape selection information is determined at the block, etc., and related information can be generated at the block level.

[0058] Furthermore, the support for the transformation block shape can be determined according to the encoding information. In this case, the encoding information can include slice type, encoding mode, block size and shape, block partitioning method, etc. That is, one transformation shape can be supported according to at least one piece of encoding information, and multiple transformation shapes can be supported according to at least one piece of encoding information. The former is an implicit situation, and the latter can be an explicit situation. In the explicit case, adaptive selection information that points to the optimal candidate group from among multiple candidate groups can be generated and recorded in the bitstream. Including this example, in the present invention, when encoding information is explicitly generated, it can be understood that the relevant information is recorded in the bitstream in various units, and the decoder parses the related information in various units to restore it to the decoded information. Also, when the encoding / decoding information is processed implicitly, it can be understood that the encoder and decoder process it using the same process, rules, etc.

[0059] For example, rectangular transformation support can be determined depending on the slice type. In the case of I-slice, the supported transformation shape is a square, and in the case of P / B slice, the supported transformation shape may be a square or a rectangle.

[0060] For example, rectangular transformation support can be determined depending on the encoding mode. In the case of Intra, the supported transformation shape is a square, while in the case of Inter, the supported transformation shape can be either a square or a rectangle.

[0061] For example, rectangular transformation support can be determined based on the size and shape of the block. For blocks above a certain size, the supported transformation shape is a square, while for blocks below a certain size, the supported transformation shape can be either a square or a rectangle.

[0062] For example, the support for rectangular transformations can be determined according to the block partitioning method. If the block to be transformed is obtained using a quad tree partitioning method, the supported transformation shape is a square; if the block is obtained using a binary tree partitioning method, the supported transformation shape can be either a square or a rectangle.

[0063] The above example illustrates support for transformation shapes based on a single encoding piece of information; multiple pieces of information can be combined to contribute to additional transformation shape support settings. The above example is merely one example of additional transformation shape support depending on various encoding settings, and is not limited to the above; various variations are possible.

[0064] Depending on the encoding settings or image characteristics, the conversion process can be omitted. For example, depending on the encoding settings (assuming a lossless compression environment in this example), the conversion process (including the reverse process) can be omitted. As another example, if the compression performance achieved by the conversion is not realized due to the image characteristics, the conversion process can be omitted. In this case, the conversion to be omitted may be at the overall unit level, or at either the horizontal or vertical unit level. Whether or not to support such omissions can be determined depending on the size and shape of the blocks, etc.

[0065] For example, in a setting where the omission of horizontal and vertical conversions is grouped, if the conversion omission flag is 1, horizontal and vertical conversions will not be performed, and if the conversion omission flag is 0, horizontal and vertical conversions may be performed. In a setting where the omission of horizontal and vertical conversions operates independently, if the first conversion omission flag is 1, horizontal conversion will not be performed, if the first conversion omission flag is 0, horizontal conversion will be performed, if the second conversion omission flag is 1, vertical conversion will not be performed, and if the second conversion omission flag is 0, vertical conversion will be performed.

[0066] If the block size falls within range A, conversion omission can be supported; if the block size falls within range B, conversion omission cannot be supported. For example, if the block's width is greater than M or the block's height is greater than N, the conversion omission flag cannot be supported; if the block's width is less than m or the block's height is less than n, the conversion omission flag can be supported. M(m) and N(n) may be the same or different. The conversion-related settings can be determined in units such as sequences, pictures, and slices.

[0067] If additional transformation techniques are supported, the settings for these techniques can be determined based on at least one piece of encoding information. This encoding information may include slice type, encoding mode, block size and shape, and prediction mode.

[0068] For example, the supported conversion techniques can be determined according to the encoding mode. In the case of Intra, the supported conversion techniques are DCT-I, DCT-III, DCT-VI, DST-II, and DST-III, while in the case of Inter, the supported conversion techniques may be DCT-II, DCT-III, and DST-III.

[0069] For example, the supported conversion technique can be determined according to the slice type. For I-slices, the supported conversion techniques are DCT-I, DCT-II, and DCT-III; for P-slices, the supported conversion techniques are DCT-V, DST-V, and DST-VI; and for B-slices, the supported conversion techniques may be DCT-I, DCT-II, and DST-III.

[0070] For example, the support for the transformation technique can be determined according to the prediction mode. The transformation techniques supported in prediction mode A may be DCT-I and DCT-II, the transformation techniques supported in prediction mode B may be DCT-I and DST-I, and the transformation technique supported in prediction mode C may be DCT-I. In this case, prediction modes A and B may be directional modes, and prediction mode C may be a non-directional mode.

[0071] For example, the support for the transformation technique can be determined according to the size and shape of the block. The transformation technique supported for blocks above a certain size is DCT-II, the transformation techniques supported for blocks below a certain size are DCT-II and DST-V, and the transformation techniques supported for blocks above and below a certain size may be DCT-I, DCT-II, and DST-I. Also, the transformation techniques supported for square shapes may be DCT-I and DCT-II, and the transformation techniques supported for rectangular shapes may be DCT-I and DST-I.

[0072] The above example illustrates the support for a transformation technique based on a single encoded piece of information; however, multiple pieces of information can be combined to contribute to the setup of additional transformation technique support. The process is not limited to the above example and can be adapted to other cases. Furthermore, the transformation unit can transmit the information necessary to generate a transformation block to the encoding unit for encoding, record the resulting information into a bitstream, transmit it to the decoder, and the decoder's decoding unit can parse this information and use it in the inverse transformation process.

[0073] The quantization unit can quantize the input signal. The signal obtained through this quantization process is called the quantized coefficient. For example, a residual block having residual conversion coefficients transmitted from the conversion unit can be quantized to obtain a quantized block having quantized coefficients. However, the input signal is determined according to the coding settings and is not limited to residual conversion coefficients.

[0074] The quantization unit can quantize the transformed residual blocks using quantization techniques such as Dead Zone Uniform Threshold Quantization and Quantization Weighted Matrix, but is not limited to these; various quantization techniques that are improvements and modifications thereof can be used. Whether or not to support additional quantization techniques is determined at the sequence, picture, slice, tile, etc., and related information can be generated at the said level. If additional quantization techniques are supported, the quantization technique selection information is determined at the block level, etc., and related information can be generated.

[0075] If additional quantization techniques are supported, the settings for these techniques can be determined based on at least one piece of encoding information. This encoding information could include the slice type, encoding mode, block size and shape, and prediction mode.

[0076] For example, the quantization unit can set the quantization weight matrix corresponding to the coding mode and the weight matrix applied according to inter-screen prediction / intra-screen prediction to be different from each other. Furthermore, the weight matrix applied according to the intra-screen prediction mode can also be set to be different. In this case, assuming the quantization weight matrix is ​​M×N in size and the block size is the same as the quantization block size, some of the quantization components may be different quantization matrices.

[0077] Depending on the encoding settings or image characteristics, the quantization process can be omitted. For example, depending on the encoding settings (assuming a lossless compression environment in this example), the quantization process (including the inverse process) can be omitted. As another example, the quantization process can be omitted if the compression performance due to quantization is not achieved depending on the image characteristics. In this case, the region to be omitted may be the entire region or only a portion of it. Whether or not to support such omission can be determined depending on the size and shape of the blocks, etc.

[0078] Information about Quantization Parameters (QP) can be generated at units such as sequences, pictures, slices, tiles, and blocks. For example, the base QP can be set at the higher-level unit where the QP information is first generated. <1> The lower the unit level, the more you can set the QP to the same or different value as the QP set in the higher unit. <2> Through this process, the QP can be finally determined in a quantization process that takes place in some units. <3> At this time, the units such as sequences and pictures are <1> The units are slices, tiles, blocks, etc. <2> The units such as blocks are <3> This could be an example of that.

[0079] Information about QP can be generated based on the QP at each unit. Alternatively, a pre-set QP can be set as the predicted value, and difference information between that and the QP at each unit can be generated. Alternatively, a QP obtained based on at least one of the following—the QP set in a higher unit, the QP previously set in the same unit, or the QP set in an adjacent unit—can be set as the predicted value, and difference information between that and the QP at the current unit can be generated. Alternatively, a QP obtained based on the QP set in a higher unit and at least one piece of encoded information can be set as the predicted value, and difference information between that and the QP at the current unit can be generated. In this case, the previously same unit is a unit that can be defined according to the encoding order of each unit, an adjacent unit is a spatially adjacent unit, and the encoded information may be the slice type, encoding mode, prediction mode, location information, etc. of the unit in question.

[0080] For example, the current unit QP can generate difference information by setting the higher-level unit QP as the predicted value. Difference information can be generated between the QP set in a slice and the QP set in a picture, or between the QP set in a tile and the QP set in a picture. Furthermore, difference information can be generated between the QP set in a block and the QP set in a slice or tile. Additionally, difference information can be generated between the QP set in a subblock and the QP set in a block.

[0081] For example, the current unit's QP can generate difference information by setting the predicted value to the QP obtained based on at least one adjacent unit's QP, or the QP obtained based on at least one previous unit's QP. Difference information can be generated between the current block and the QP obtained based on the QP of adjacent blocks such as the left, upper left, lower left, top, and upper right of the current block. Alternatively, difference information can be generated between the current picture and the encoded picture's QP prior to the current picture.

[0082] As an example, the current unit QP can generate difference information by setting the predicted value to the QP obtained based on the QP of the higher unit and at least one piece of coding information. Difference information can be generated between the current block's QP and the slice's QP corrected according to the slice type (I / P / B). Alternatively, difference information can be generated between the current block's QP and the tile's QP corrected according to the coding mode (Intra / Inter). Alternatively, difference information can be generated between the current block's QP and the picture's QP corrected according to the prediction mode (directional / non-directional). Alternatively, difference information can be generated between the current block's QP and the picture's QP corrected according to the position information (x / y). In this case, the meaning of the correction can be that it is added to or subtracted in the form of an offset from the higher unit's QP used for prediction. In this case, at least one piece of offset information can be supported depending on the coding setting, and it may be implicitly processed or explicitly related information may be generated according to a predetermined process. This is not limited to the above example, and variations to other examples are possible.

[0083] The above example may be possible when a signal indicating QP variation is provided or activated. For example, if a signal indicating QP variation is not provided or is not activated, no difference value information is generated, and the predicted QP can be determined as QP for each unit. As another example, if a signal indicating QP variation is provided or activated, difference value information is generated, and when its value is 0, the predicted QP can be determined as QP for each unit.

[0084] The quantization unit can transmit the information necessary to generate quantization blocks to the encoding unit for encoding, record the resulting information into a bitstream, and transmit it to the decoder. The decoder's decoding unit can then parse this information and use it in the dequantization process.

[0085] In the above example, the explanation was based on the assumption that the residual block is transformed and quantized via a transformation unit and a quantization unit. However, it is not necessary to transform the residual signal to generate a residual block with transformation coefficients and perform the quantization process; it is possible to perform only the quantization process without converting the residual signal of the residual block to transformation coefficients; and it is even possible to perform neither the transformation nor the quantization process. This can be determined depending on the encoding configuration.

[0086] The coding unit can scan the quantization coefficients, transformation coefficients, or residual signals of the generated residual block according to at least one scan sequence (e.g., zigzag scan, vertical scan, horizontal scan, etc.) to generate a sequence of quantization coefficients, a sequence of transformation coefficients, or a sequence of signals, and encode them using at least one entropy coding technique. In this case, information about the scan sequence can be determined according to the coding settings (e.g., coding mode, prediction mode, etc.), and can be implicitly determined or the relevant information can be explicitly generated. For example, depending on the on-screen prediction mode, one of several scan sequences can be selected.

[0087] Furthermore, encoded data containing encoded information transmitted from each component can be generated and output to a bitstream, which can be achieved with a multiplexer (MUX). In this case, encoding can be performed using methods such as Exponential Golomb, Context Adaptive Variable Length Coding (CAVLC), and Context Adaptive Binary Arithmetic Coding (CABAC), but is not limited to these, and various encoding techniques that are improvements and modifications of these methods can be used.

[0088] When performing entropy coding (in this example, CABAC is assumed) on syntactic elements such as the residual block data and information generated during the coding / decoding process, the entropy coding device may include a binarizer, a context modeler, and a binary arithmetic coder. In this case, the binary arithmetic coder may include a regular coding engine and a bypass coding engine.

[0089] Since the syntactic elements input to the entropy encoding device may not be binary, if the syntactic elements are not binary, the binarization unit can binarize the syntactic elements and output a Bin String consisting of 0s or 1s. In this case, Bin represents bits consisting of 0s or 1s and can be encoded via the binary arithmetic encoding unit. In this case, either the regular coding unit or the bypass coding unit can be selected based on the probability of occurrence of 0s and 1s. This can be determined according to the encoding / decoding settings. If the syntactic elements are data where the frequency of 0s and the frequency of 1s are the same, the bypass coding unit can be used; otherwise, the regular coding unit can be used.

[0090] Various methods can be used when performing binarization on the aforementioned syntactic elements. For example, fixed-length binarization, unary binarization, truncated rice binarization, and K-th Exp-Golomb binarization can be used. Depending on the range of values ​​the syntactic elements possess, signed or unsigned binarization may be performed. The binarization process for syntactic elements generated in this invention may include not only the binarization methods mentioned in the examples above, but also other additional binarization methods.

[0091] The inverse quantization unit and the inverse transformation unit can be realized by reversing the processes in the transformation unit and the quantization unit. For example, the inverse quantization unit can inverse quantize the quantization transformation coefficients generated in the quantization unit, and the inverse transformation unit can inverse transform the inversely quantized transformation coefficients to generate the restored residual block.

[0092] The addition unit adds the predicted block and the restored residual block to restore the current block. The restored block is stored in memory and can be used as reference data (for the prediction unit and filter unit, etc.).

[0093] The in-loop filter section may include at least one post-processing filter step, such as a deblocking filter, a sample adaptive offset (SAO), or an adaptive loop filter (ALF). A deblocking filter can remove block distortion that occurs at the boundaries between blocks from the reconstructed image. An ALF can perform filtering based on a comparison between the reconstructed image and the input image. More specifically, filtering can be performed based on a comparison between the reconstructed image and the input image after the blocks have been filtered via the deblocking filter. Alternatively, filtering can be performed based on a comparison between the reconstructed image and the input image after the blocks have been filtered via the SAO. An SAO reconstructs the offset difference based on a comparison between the reconstructed image and the input image and can be applied in the form of a band offset (BO), edge offset (EO), etc. More specifically, an SAO adds an offset from the original image at least one pixel at a time to the reconstructed image to which a deblocking filter has been applied and can be applied in the form of BO, EO, etc. More specifically, after the blocks have been filtered via ALF and the image has been restored, an offset from the original image is added on a pixel-by-pixel basis, and this can be applied in forms such as BO and EO.

[0094] As filtering-related information, setting information regarding whether to support each post-processing filter can be generated in units such as sequences, pictures, slices, and tiles. Furthermore, setting information regarding whether to execute each post-processing filter can be generated in units such as pictures, slices, tiles, and blocks. The scope to which the filter execution is applied can be divided into the interior of the image and the image boundaries, and setting information considering this can be generated. Additionally, information related to the filtering operation can be generated in units such as pictures, slices, tiles, and blocks. This information can be processed implicitly or explicitly, and the filtering can be an independent or dependent filtering process depending on the color component. This can be determined according to the encoding settings. The in-loop filter unit can transmit the filtering-related information to the encoding unit for encoding, record the resulting information into a bitstream, and transmit it to the decoder. The decoder's decoding unit can then parse this information and apply it to the in-loop filter unit.

[0095] Memory can store the recovered blocks or pictures. The recovered blocks or pictures stored in memory can be provided to the prediction unit that performs in-screen or inter-screen prediction. Specifically, the storage space for the queue-type bitstream compressed by the encoder can be processed as a Coded Picture Buffer (CPB), and the space for storing the decoded images on a picture-by-picture basis can be processed as a Decoded Picture Buffer (DPB). In the case of CPB, the decoded units are stored according to the decoding order, the decoding operation is emulated within the encoder, the compressed bitstream is stored during the emulation process, the bitstream output from CPB is recovered through the decoding process, the recovered images are stored in DPB, and the pictures stored in DPB can be referenced in subsequent image encoding and decoding processes.

[0096] The decoding unit can be implemented by reversing the process in the encoding unit. For example, it can receive a sequence of quantization coefficients, a sequence of transformation coefficients, or a sequence of signals from a bitstream, decode them, and parse the decoded data containing the decoded information and transmit it to each component.

[0097] The following describes an image setting process applied to an image encoding / decoding device according to one embodiment of the present invention. This may be an example applied to a stage before encoding / decoding (initial image setup), but some processes may also be applicable to other stages (e.g., stages after encoding / decoding or internal stages of encoding / decoding). The image setting process may be performed considering the network and user environment, such as the characteristics of the multimedia content, bandwidth, and the performance and accessibility of the user terminal. For example, depending on the encoding / decoding settings, image division, image resizing, and image reconstruction may be performed. The image setting process described below will be explained mainly in terms of rectangular images, but is not limited to this and can also be applied to polygonal images. Regardless of the shape of the image, the same image settings may be applied or different image settings may be applied. This can be determined according to the encoding / decoding settings. For example, after confirming information about the shape of the image (e.g., rectangular or non-rectangular shape), information for the image settings can be constructed based on that information.

[0098] In the examples described later, we assume that settings are dependent on the color space, but it is also possible to set settings that are independent of the color space. Furthermore, in the case of independent settings in the examples described later, it is possible to include examples where encoding / decoding settings are set independently for each color space, and even when describing one color space, it is possible to assume and derive that examples applicable to other color spaces (for example, if M is generated from the luminance component, N is generated from the chrominance component) are included. Furthermore, in the case of dependent settings, it is possible to include examples where settings are set in proportion to the composition ratio of the color format (for example, 4:4:4, 4:2:2, 4:2:0, etc.) (for example, in the case of 4:2:0, if M is generated from the luminance component, M / 2 is generated from the chrominance component), and it is possible to assume and derive that examples applicable to each color space are included without special explanation. This explanation is not limited to the above examples and may be applicable to the present invention in general.

[0099] Some of the configurations in the examples described later may be applicable to various coding techniques, such as spatial domain coding, frequency domain coding, block-based coding, and object-based coding.

[0100] While it is common practice to encode / decode the input image as is, there are also cases where the image is divided before encoding / decoding. For example, division can be performed to prevent damage due to packet corruption during transmission, or for error tolerance. Alternatively, division can be performed to classify regions with different properties within the same image, depending on the characteristics and type of the image.

[0101] In the present invention, the image segmentation process may include a segmentation process and its reverse process. In the examples described later, the segmentation process will be explained in detail, but the content of the reverse segmentation process can be derived in reverse from the segmentation process.

[0102] Figure 3 is an example diagram illustrating how image information can be divided hierarchically to compress an image.

[0103] Figure 3a is an illustrative diagram showing an image sequence composed of multiple GOPs. One GOP can consist of I-pictures, P-pictures, and B-pictures, as shown in 3b. One picture can consist of slices, tiles, etc., as shown in 3c. Slices, tiles, etc., are composed of multiple basic coding units, as shown in 3d, and a basic coding unit can consist of at least one subcoding unit, as shown in Figure 3e. The image setting process in this invention will be explained based on examples applied to units such as pictures, slices, and tiles, as shown in 3b and 3c.

[0104] Figure 4 is a conceptual diagram showing various examples of image segmentation according to one embodiment of the present invention.

[0105] Figure 4a is a conceptual diagram showing an image (e.g., a picture) divided into horizontal and vertical sections of a fixed length. The divided regions can be called blocks, and each block is a basic coding unit (or maximum coding unit) obtained through the picture division section, and may also be the basic unit applied to the division unit described later.

[0106] Figure 4b is a conceptual diagram showing an image divided in at least one of two directions: horizontal and vertical. The divided regions T0 to T3 can be called tiles, and each region can be encoded / decoded independently or dependently on the other regions.

[0107] Figure 4c is a conceptual diagram showing an image divided into groups of consecutive blocks. The divided regions S0 and S1 can be called slices, and each region may perform encoding / decoding independently of or dependent on other regions. The groups of consecutive blocks can be defined according to the scan order, generally following the raster scan order, but not limited to this, and can be determined according to the encoding / decoding settings.

[0108] 4D is a conceptual diagram in which an image is divided into groups of blocks according to user-defined arbitrary settings. The divided regions A0 to A2 can be called arbitrary partitions, and each region may perform encoding / decoding independently of or dependent on the other regions.

[0109] Independent encoding / decoding can mean that when encoding / decoding some units (or regions), data from other units cannot be referenced. More specifically, the information used or generated in the texture encoding and entropy encoding of some units is encoded independently and does not reference each other, and the decoder does not need to reference the parsing and reconstruction information of other units for the texture decoding and entropy decoding of some units. In this case, the ability to reference data from other units (or regions) may be restricted in the spatial domain (e.g., between regions within a single image), but depending on the encoding / decoding settings, restrictions may also be placed in the temporal domain (e.g., between consecutive images or frames). For example, if some units in the current image and some units in other images have continuity or share the same encoding environment, referencing is possible; otherwise, referencing can be restricted.

[0110] Furthermore, dependent encoding / decoding can mean that when encoding / decoding some units, data from other units can be referenced. In detail, information used or generated in the texture encoding and entropy encoding of some units is referenced and encoded dependently, and similarly in the decoder, parsing information and reconstruction information from other units can be referenced for the texture decoding and entropy decoding of some units. In other words, the settings can be the same as or similar to general encoding / decoding. In this case, the region (in this example, the surface generated according to the projection format) can be referenced depending on the characteristics and type of the image (e.g., a 360-degree image). <face>It may have been split for the purpose of identifying (etc.)

[0111] In the above example, some units (slice, tile, etc.) can have independent encoding / decoding settings (e.g., independent slice segments), while some units can have dependent encoding / decoding settings (e.g., dependent slice segments). This invention will primarily focus on describing independent encoding / decoding settings.

[0112] As in 4a, the basic coding unit obtained via the picture division section is divided into basic coding blocks according to the color space, and the size and shape can be determined according to the image characteristics and resolution. The size or shape of the supported block is such that the width and height are raised to the power of 2 (2 n An N×N square (2 n ×2 n 256×256, 128×128, 64×64, 32×32, 16×16, 8×8, etc. (n is an integer between 3 and 8) or an M×N rectangle (2 m ×2 n ) is possible. For example, depending on the resolution, the input image can be divided into sizes such as 128x128 for 8k UHD images, 64x64 for 1080p HD images, and 16x16 for WVGA images. Depending on the image type, the input image can be divided into sizes such as 256x256 for 360-degree images. The basic coding unit can be divided into lower coding units and coded / decoded, and the information for the basic coding unit can be recorded in a bitstream in units such as sequences, pictures, slices, and tiles and transmitted. This can be parsed by a decoder to restore the relevant information.

[0113] An image encoding method and a decoding method according to one embodiment of the present invention may include the following image segmentation steps. In this case, the image segmentation process may include an image segmentation instruction step, an image segmentation type identification step, and an image segmentation execution step. Furthermore, the image encoding device and the decoding device may be configured to include an image segmentation instruction unit, an image segmentation type identification unit, and an image segmentation execution unit that realize the image segmentation instruction step, the image segmentation type identification step, and the image segmentation execution step. In the case of encoding, associated syntactic elements can be generated, and in the case of decoding, associated syntactic elements can be parsed.

[0114] In each block division process of 4a, the image division instruction unit is optional, and the image division type identification unit is a process of confirming information regarding the size and shape of the block, and the image division unit can perform division in basic coding units via the identified division type information.

[0115] In the case of blocks, they can always be the unit of division, but for other division units (tiles, slices, etc.), whether or not to divide can be determined depending on the encoding / decoding settings. The picture division unit can be set by default to perform division by block units first, followed by division by other units. In this case, block division may be performed based on the size of the picture.

[0116] Furthermore, data can be divided into blocks after being divided into other units (tiles, slices, etc.). That is, block division can be performed based on the size of the division unit. This can be determined through explicit or implicit processing depending on the encoding / decoding settings. The examples described later assume the former case and focus on units other than blocks.

[0117] At the image splitting instruction stage, it is possible to decide whether or not to perform image splitting. For example, if a signal instructing image splitting (e.g., tiles_enabled_flag) is detected, splitting can be performed. If no such signal is detected, splitting can be withheld, or other encoding / decoding information can be checked before splitting can be performed.

[0118] In detail, if a signal instructing image segmentation (e.g., tiles_enabled_flag) is detected and activated (e.g., tiles_enabled_flag=1), segmentation can be performed in multiple units. If the signal is deactivated (e.g., tiles_enabled_flag=0), segmentation can not be performed. Alternatively, if no signal instructing image segmentation is detected, it may mean that segmentation will not be performed or that segmentation will be performed in at least one unit. Whether segmentation will be performed in multiple units can be confirmed via another signal (e.g., first_slice_segment_in_pic_flag).

[0119] In summary, when a signal is provided to instruct image division, this signal indicates whether or not to divide the image into multiple units, and it is possible to check whether or not to divide the image according to this signal. For example, when `tiles_enabled_flag` is a signal indicating whether or not to divide the image, a value of `tiles_enabled_flag` of 1 can mean that the image will be divided into multiple tiles, and a value of 0 can mean that the image will not be divided.

[0120] In summary, if no signal is provided to instruct image segmentation, segmentation will not be performed, or whether or not the image in question is being segmented can be confirmed by other signals. For example, the `first_slice_segment_in_pic_flag` is not a signal indicating whether or not the image is being segmented, but rather a signal indicating whether or not it is the first slice segment in the image. This allows confirmation of whether or not the image is being segmented into two or more units (for example, if the flag is 0, it means that the image has been segmented into multiple slices).

[0121] The above example is not the only one that can be applied; variations to other examples are also possible. For example, a signal to instruct image segmentation may not be provided in the tile stage, but a signal to instruct image segmentation may be provided in the slice stage. Alternatively, a signal to instruct image segmentation may be provided depending on the type and characteristics of the image.

[0122] The image segmentation type identification stage allows for the identification of the image segmentation type. The image segmentation type can be defined by the method of segmentation, segmentation information, and other factors.

[0123] In 4b, a tile can be defined as a unit obtained by dividing the image horizontally and vertically. More specifically, it can be defined as a group of adjacent blocks within a rectangular space demarcated by at least one horizontal or vertical dividing line that crosses the image.

[0124] The division information for a tile can include information about the boundary positions between rows and columns, the number of tiles in each row and column, and the size of the tiles. The number of tiles can include the number of rows (e.g., num_tile_columns) and the number of columns (e.g., num_tile_rows), allowing the tile to be divided into (number of rows × number of columns) tiles. The size of the tiles can be obtained based on the number of tiles, but the width or height of the tiles may be uniform or uneven. This can be implicitly determined under a pre-set rule, or the relevant information (e.g., uniform_spacing_flag) can be explicitly generated. The size of the tiles can also include the size information for each row and column (e.g., column_width_tile[i], row_height_tile[i]), or the height and width information for each tile. Furthermore, the size information may also be information that can be generated depending on whether the tile sizes are uniform or not (for example, when uniform_spacing_flag is 0, which means unequal division).

[0125] In 4c, a slice can be defined as a group of consecutive blocks. More specifically, it can be defined as a group of consecutive blocks based on a predetermined scan order (raster scan in this example).

[0126] The segmentation information for a slice may include information about the number of slices and the location information of the slices (e.g., slice_segment_address). In this case, the slice location information may be the location information of a predetermined block (e.g., the first block in the scan order within the slice). In this case, the location information may be the scan order information of the block.

[0127] In 4D, any divided region can be divided into various subdivision settings.

[0128] In 4D, a division unit can be defined as a group of spatially adjacent blocks, and the division information for this unit can include information such as the size, shape, and position of the division unit. This is just one example for any division region, and a variety of division shapes are possible, as shown in Figure 5.

[0129] Figure 5 is another illustrative diagram of an image segmentation method according to one embodiment of the present invention.

[0130] In cases 5a and 5b, the image can be divided into multiple regions with at least one block interval in the horizontal or vertical direction, and the division may be performed based on the positional information of the blocks. 5a shows examples A0 and A1 in which the division is performed horizontally based on the column information of each block, and 5b shows examples B0 to B3 in which the division is performed vertically and horizontally based on the horizontal and column information of each block. The division information for this may include the number of division units, block interval information, division direction, etc., and if this is implicitly included according to a predetermined rule, some division information may not be generated.

[0131] In cases 5c and 5d, the image can be divided into groups of consecutive blocks based on the scan order. Additional scan orders other than the raster scan order of existing slices can be applied to image division. 5c shows examples C0 and C1 where scanning is performed clockwise or counterclockwise (Box-Out) around the starting block, and 5d shows examples D0 and D1 where scanning is performed vertically (Vertical) around the starting block. The division information for this may include information on the number of division units, position information of the division units (e.g., the first position in the scan order within the division unit), and information on the scan order. If this is implicitly included according to a predetermined rule, some division information may not be generated.

[0132] In the case of 5e, images can be divided by horizontal and vertical dividing lines. Existing tiles are divided by horizontal or vertical dividing lines, which can result in a quadrilateral spatial division shape, but it may not be possible to divide the image across the dividing lines. For example, it is possible to divide the image across a dividing line along part of the image (e.g., a dividing line forming the right boundary of E1, E3, and E4 and the left boundary of E5), but it is not possible to divide the image across a dividing line along part of the image (e.g., a dividing line forming the lower boundary of E2 and E3 and the upper boundary of E4). Furthermore, division can be performed based on block units (e.g., division is performed after block division), or by the aforementioned horizontal or vertical dividing lines (e.g., division is performed by the dividing lines regardless of block division), and as a result, each division unit may not consist of an integer multiple of a block. Therefore, division information different from that of existing tiles can be generated, and this division information may include information on the number of division units, the position information of the division units, and the size information of the division units. For example, position information for a division unit can be generated based on a predetermined position (e.g., the upper left corner of the image) (e.g., measured in pixel units or block units), and size information for a division unit can be generated as horizontal and vertical size information for each division unit (e.g., measured in pixel units or block units).

[0133] As in the example above, partitioning with arbitrary settings defined by the user can be performed by applying a new partitioning method or by modifying and applying some components of an existing partitioning method. In other words, it can be supported by substituting existing partitioning methods or by adding partitioning shapes, or by modifying and applying some settings to existing partitioning methods (slice, tile, etc.) (e.g., following a different scan order, other partitioning methods for quadrilateral shapes and the generation of other partitioning information as a result, dependent encoding / decoding characteristics, etc.). It is also possible to support settings that constitute additional partitioning units (e.g., settings other than partitioning according to the scan order or according to a difference in a fixed interval), and additional partitioning unit shapes (e.g., polygonal shapes such as triangles other than partitioning into quadrilateral spaces). Furthermore, it is possible to support image partitioning methods based on the type and characteristics of the image. For example, some partitioning methods (e.g., the surface of a 360-degree image) can be supported depending on the type and characteristics of the image, and partitioning information can be generated based on this.

[0134] During the image segmentation execution phase, the image can be segmented based on the identified segmentation type information. That is, the image can be segmented into multiple segments based on the identified segmentation type, and encoding / decoding can be performed based on the acquired segmentation units.

[0135] In this case, it is possible to determine whether each division unit has encoding / decoding settings depending on the division type. That is, the setting information required for the encoding / decoding process of each division unit can be assigned by a higher-level unit (e.g., picture), or each division unit can have its own independent encoding / decoding settings.

[0136] Generally, slices can have independent encoding / decoding settings for each division unit (e.g., slice header), while tiles cannot have independent encoding / decoding settings for each division unit and can have settings that depend on the picture encoding / decoding settings (e.g., PPS). In this case, the information generated in relation to the tile may be division information, which may be included in the picture encoding / decoding settings. The present invention is not limited to the cases described above, and other variations are possible.

[0137] Encoding / decoding setting information for tiles can be generated at the video, sequence, picture, etc. level, and at least one encoding / decoding setting information can be generated at a higher level, with any one of them being referenced. Alternatively, independent encoding / decoding setting information (e.g., tile header) can be generated at the tile level. This differs from relying on a single encoding / decoding setting determined at a higher level, in that encoding / decoding is performed based on at least one encoding / decoding setting at the tile level. In other words, all tiles can either rely on a single encoding / decoding setting, or at least one tile can perform encoding / decoding according to a different encoding / decoding setting than other tiles.

[0138] While the above example primarily explains various encoding / decoding settings for tiles, the explanation is not limited to this, and similar or identical settings can be applied to other partitioning types.

[0139] For example, some partitioning types allow for the generation of partitioning information at a higher level and the subsequent encoding / decoding according to the encoding / decoding settings of one of the higher levels.

[0140] For example, some partitioning types allow for the generation of partitioning information at a higher level, and the generation of independent encoding / decoding settings for each partition at that higher level, thereby enabling encoding / decoding.

[0141] For example, some partitioning types can generate partitioning information at a higher level, support multiple encoding / decoding settings at that level, and perform encoding / decoding according to the encoding / decoding settings referenced by each partitioning unit.

[0142] For example, some segmentation types allow for the generation of segmentation information at a higher level, the generation of independent encoding / decoding settings for that segment, and the subsequent encoding / decoding.

[0143] For example, some segmentation types allow for the generation of independent encoding / decoding settings containing segmentation information for each segment, thereby enabling encoding / decoding.

[0144] The encoding / decoding settings information can include information necessary for encoding / decoding tiles, such as the tile type, information about the referenced picture list, quantization parameter information, inter-screen prediction settings information, in-loop filtering settings information, in-loop filtering control information, scan order, and whether or not to perform encoding / decoding. The encoding / decoding settings information can be explicitly generated, or the settings for encoding / decoding can be implicitly determined based on the image format, characteristics, etc., determined at a higher level. Alternatively, the relevant information can be explicitly generated based on the information obtained from the settings.

[0145] The following shows an example of image segmentation performed by an encoding / decoding device according to one embodiment of the present invention.

[0146] The input image can be segmented before encoding begins. After segmentation using segmentation information (e.g., image segmentation information, segmentation unit setting information), the image can be encoded in segments. After encoding is complete, the results can be stored in memory, and the encoded image data can be recorded as a bitstream and transmitted.

[0147] The segmentation process can be performed before the start of decoding. After segmentation using segmentation information (e.g., image segmentation information, segmentation unit setting information, etc.), the image decoding data can be parsed and decoded for each segment. After decoding is complete, the data can be saved to memory, and multiple segmentation units can be merged into one image for output.

[0148] The image segmentation process was explained using the above example. Furthermore, in this invention, multiple segmentation processes may be performed.

[0149] For example, an image can be divided, and the image can be divided into division units. The division may be the same division process (e.g., slice / slice, tile / tile, etc.) or different division processes (e.g., slice / tile, tile / slice, tile / surface, surface / tile, slice / surface, surface / slice, etc.). In this case, subsequent division processes may be performed based on the results of preceding division processes. The division information generated in subsequent division processes can be generated based on the results of preceding division processes.

[0150] Furthermore, multiple division processes A can be performed, and these division processes may be different (e.g., slice / surface, tile / surface, etc.). In this case, subsequent division processes may be performed based on the results of preceding division processes, or they may be performed independently of the results of preceding division processes. The division information generated in subsequent division processes may be generated based on the results of preceding division processes or it may be generated independently.

[0151] The multiple image segmentation processes can be determined according to the encoding / decoding settings, and are not limited to the above example; various transformations are also possible.

[0152] In the encoder, the information generated in the above process is recorded in the bitstream in at least one unit from among sequences, pictures, slices, tiles, etc., and in the decoder, the related information is parsed from the bitstream. That is, it can be recorded in one unit, or it can be recorded redundantly in multiple units. For example, syntactic elements indicating whether or not some information is supported, or syntactic elements indicating whether or not it is activated, can be generated in some units (e.g., higher units), and the same or similar information can be generated in some units (e.g., lower units). That is, even if related information is supported and set in the higher units, it can still have individual settings in the lower units. This is not limited to the above example, and may be a description that applies commonly to the present invention. It can also be included in the bitstream in the form of SEI or metadata.

[0153] On the one hand, while it is common to encode / decode the input image as is, it is also possible to encode / decode the image after adjusting its size (expanding or shrinking it; adjusting the resolution). For example, hierarchical encoding schemes (Scalability Video Coding) that support spatial, temporal, and image quality scalability can be used to adjust the overall image size, such as expanding or shrinking it. Alternatively, it is also possible to adjust the image size, such as expanding or shrinking it partially. Image size adjustment can be done for various purposes, such as adaptability to the encoding environment, encoding uniformity, encoding efficiency, image quality improvement, or depending on the type and characteristics of the image.

[0154] As a first example, the resizing process may be carried out in a process that is performed according to the characteristics and type of the image (for example, hierarchical coding, 360-degree image coding, etc.).

[0155] As a second example, the resizing process may be performed in the initial stages of encoding / decoding. Alternatively, the resizing process may be performed before encoding / decoding. The resized image can then be encoded / decoded.

[0156] As a third example, a resizing process may be performed during the prediction stage (in-screen prediction or inter-screen prediction) or before prediction execution. During the resizing process, image information from the prediction stage (e.g., pixel information referenced for in-screen prediction, in-screen prediction mode-related information, reference image information used for inter-screen prediction, inter-screen prediction mode-related information, etc.) can be used.

[0157] As a fourth example, a resizing process may be performed during or before the filtering stage. During the resizing process, image information from the filtering stage (e.g., pixel information applied to the deblocking filter, pixel information applied to the SAO, SAO filtering-related information, pixel information applied to the ALF, ALF filtering-related information, etc.) can be used.

[0158] Furthermore, after the resizing process has been performed, the image may or may not be changed back to the pre-sizing image (in terms of image size) through the reverse resizing process. This can be determined according to the encoding / decoding settings (for example, the nature of the resizing). In this case, if the resizing process is expansion, the reverse resizing process may be reduction, and if the resizing process is reduction, the reverse resizing process may be expansion.

[0159] If the resizing process described in the first to fourth examples is performed, the reverse resizing process can be performed in a subsequent stage to obtain the image before resizing.

[0160] If a hierarchical encoding or a size adjustment process according to the third example is performed (or if the size of the reference image is adjusted by inter-screen prediction), it is not necessary to perform a reverse size adjustment process in subsequent stages.

[0161] In one embodiment of the present invention, the image resizing process can be performed independently or inversely. In the example described later, the focus will be on the resizing process. In this case, since the reverse resizing process is the opposite of the resizing process, the explanation of the reverse resizing process can be omitted to avoid redundant explanation, but it is clear that an ordinary engineer can understand it in the same way as if it were written explicitly.

[0162] Figure 6 is an illustrative diagram of a common method for resizing images.

[0163] Referring to 6a, an expanded image P0+P1 can be obtained by further including a portion of region P1 from the initial image (or the image before resizing; P0; thick solid line).

[0164] Referring to 6b, a reduced image S0 can be obtained by excluding a portion of region S1 from the initial image S0+S1.

[0165] Referring to 6c, a resized image T0+T1 can be obtained by further including a portion of region T1 in the initial image T0+T2 and excluding a portion of region T2.

[0166] In the following, the present invention will primarily describe the size adjustment process by expansion and the size adjustment process by reduction, but it should be understood that it is not limited to these, and also includes cases where size expansion and reduction are applied in combination, as in 6c.

[0167] Figure 7 is an illustrative diagram of image size adjustment according to one embodiment of the present invention.

[0168] Refer to 7a to explain how to enlarge an image during the resizing process, and refer to 7b to explain how to reduce an image.

[0169] In 7a, the image before resizing is S0, and the image after resizing is S1. In 7b, the image before resizing is T0, and the image after resizing is T1.

[0170] When expanding an image as in 7a, it can be expanded in the upward, downward, left, and right directions (ET, EL, EB, ER), and when shrinking an image as in 7b, it can be shrinked in the upward, downward, left, and right directions (RT, RL, RB, RR).

[0171] When comparing image expansion and image reduction, the up, down, left, and right directions in expansion correspond to the down, up, right, and left directions in reduction. Therefore, although the following explanation will be based on image expansion, it should be understood that it will also include explanations about image reduction.

[0172] Furthermore, while the following describes image expansion or reduction in the upward, downward, left, and right directions, it should be understood that size adjustments can also be performed in the upper left, upper right, lower left, and lower right directions.

[0173] In this case, when expanding in the lower right direction, the RC and BC regions are acquired, while the BR region may or may not be acquired depending on the encoding / decoding settings. That is, the TL, TR, BL, and BR regions may or may not be acquired, but for the sake of explanation below, we will assume that the corner regions (TL, TR, BL, and BR regions) are acquireable.

[0174] The image resizing process according to one embodiment of the present invention may be performed in at least one direction. For example, it may be performed in all up, down, left, and right directions, or in two or more directions selected from the up, down, left, and right directions (e.g., left + right, up + down, up + left, up + right, down + left, down + right, up + left + right, down + left + right, up + down + left, up + down + right), or in only one of the up, down, left, or right directions.

[0175] For example, the image may be resized symmetrically from the center to both ends in the left + right, top + bottom, top left + bottom right, and bottom left + top right directions; it may be resized vertically symmetrically from the image to the left + right, top left + top right, and bottom left + bottom right directions; it may be resized horizontally symmetrically from the image to the top + bottom, top left + bottom left, and top right + bottom right directions; and other resizing options are also available.

[0176] In 7a and 7b, the size of the image before resizing (S0, T0) was defined as P_Width (width) × P_Height (height), and the size of the image after resizing (S1, T1) was defined as P'_Width (width) × P'_Height (height). Here, if the resizing values ​​in the left, right, up, and down directions are defined as Var_L, Var_R, Var_T, and Var_B (or collectively called Var_x), then the size of the image after resizing can be expressed as (P_Width + Var_L + Var_R) × (P_Height + Var_T + Var_B). In this case, the size adjustment values ​​Var_L, Var_R, Var_T, and Var_B in the left, right, up, and down directions are Exp_L, Exp_R, Exp_T, and Exp_B (in this example, Exp_x is a positive number) in image expansion (Figure 7a), and can be -Rec_L, -Rec_R, -Rec_T, and -Rec_B (if Rec_L, Rec_R, Rec_T, and Rec_B are defined as positive numbers, they can be expressed as negative numbers depending on the image reduction). Furthermore, the coordinates of the top-left, top-right, bottom-left, and bottom-right of the image before resizing are (0,0), (P_Width-1,0), (0,P_Height-1), and (P_Width-1,P_Height-1), respectively, and the coordinates of the image after resizing can be expressed as (0,0), (P'_Width-1,0), (0,P'_Height-1), and (P'_Width-1,P'_Height-1). The size of the area that is changed (or acquired, deleted) by resizing (in this example, TL~BR, where i is the index separating TL~BR) can be M[i] × N[i]. This can be expressed as Var_X × Var_Y (in this example, X is assumed to be L or R, and Y is T or B). M and N can have various values ​​and can be the same regardless of i, or they can have individual settings depending on i. Various cases regarding this will be described later.

[0177] Referring to 7a, S1 can consist of all or part of the TL~BR (top left to bottom right) generated in S0 through expansion in various directions. Referring to 7b, T1 can consist of all or part of the TL~BR removed from T0 through contraction in various directions.

[0178] In 7a, when expanding the existing image S0 in the upward, downward, left, and right directions, the image can be constructed including the TC, BC, LC, and RC regions acquired through each size adjustment process, and can also include the TL, TR, BL, and BR regions.

[0179] For example, when performing expansion in the upward (ET) direction, the existing image S0 can be used to construct an image that includes the TC region, and the TL or TR region can be included depending on expansion in at least one different direction (EL or ER).

[0180] For example, when performing expansion in the downward (EB) direction, the existing image S0 can be used to construct an image that includes the BC region, and the BL or BR region can be included depending on expansion in at least one different direction (EL or ER).

[0181] For example, when performing expansion in the left (EL) direction, the existing image S0 can be used to construct an image that includes the LC region, and the TL or BL region can be included depending on expansion in at least one different direction (ET or EB).

[0182] For example, when performing expansion in the rightward (ER) direction, the existing image S0 can be used to construct an image that includes the RC region, and the TR or BR region can be included depending on expansion in at least one different direction (ET or EB).

[0183] According to one embodiment of the present invention, a setting (e.g., spa_ref_enabled_flag or tem_ref_enabled_flag) can be set to spatially or temporally restrict the referability of a resizable region (assumed to be an expansion in this example).

[0184] In other words, you can either reference data in a region that is spatially or temporally resized depending on the encoding / decoding settings (e.g., spa_ref_enabled_flag=1 or tem_ref_enabled_flag=1) or restrict the reference (e.g., spa_ref_enabled_flag=0 or tem_ref_enabled_flag=0).

[0185] The encoding / decoding of the image before resizing (S0, T1) and the regions added or deleted during resizing (TC, BC, LC, RC, TL, TR, BL, BR regions) can be performed as follows:

[0186] For example, in the encoding / decoding of an image before resizing and regions to be added or deleted, the data of the image before resizing and the data of the regions to be added or deleted (encoded / decoded data, such as pixel values ​​or predictive information) can be spatially or temporally referenced from each other.

[0187] Alternatively, the image before resizing and the data of the areas to be added or deleted can be spatially referenced, while the data of the image before resizing can be temporally referenced, but the data of the areas to be added or deleted cannot be temporally referenced.

[0188] In other words, settings can be placed to restrict the accessibility of areas being added or deleted. The setting information regarding the accessibility of areas being added or deleted can be explicitly generated or implicitly determined.

[0189] An image resizing process according to one embodiment of the present invention may include an image resizing instruction step, an image resizing type identification step, and / or an image resizing execution step. Furthermore, an image encoding device and a decoding device may include an image resizing instruction unit, an image resizing type identification unit, and an image resizing execution unit that implement the image resizing instruction step, the image resizing type identification step, and the image resizing execution step. In the case of encoding, associated syntactic elements can be generated, and in the case of decoding, associated syntactic elements can be parsed.

[0190] At the image resizing instruction stage, it is possible to decide whether or not to perform image resizing. For example, if a signal instructing image resizing (e.g., img_resizing_enabled_flag) is detected, resizing can be performed. If no such signal is detected, resizing can be withheld, or resizing can be performed after checking other encoding / decoding information. Furthermore, even if a signal instructing image resizing is not provided, the signal instructing resizing may be implicitly activated or deactivated depending on the encoding / decoding settings (e.g., image characteristics, type, etc.). If resizing is performed, resizing-related information can be generated, or resizing-related information can be implicitly determined.

[0191] When a signal instructing image resizing is provided, the signal indicates whether or not to resize the image, and it is possible to confirm whether or not to resize the image in question based on the signal.

[0192] For example, if a signal instructing image resizing (e.g., img_resizing_enabled_flag) is detected and activated (e.g., img_resizing_enabled_flag=1), image resizing can be performed. Conversely, if the signal is deactivated (e.g., img_resizing_enabled_flag=0), it can mean that image resizing will not be performed.

[0193] Furthermore, if no signal is provided to instruct image size adjustment, the image size adjustment will not be performed, or whether or not the image size adjustment has been performed can be confirmed by other signals.

[0194] For example, when dividing an input image into blocks, size adjustment can be performed depending on whether the image size (e.g., width or height) is an integer multiple of the block size (e.g., width or height). (In this example, in the case of expansion; it is assumed that the size adjustment process is performed when it is not an integer multiple.) That is, size adjustment can be performed if the width of the image is not an integer multiple of the width of the block, or if the height of the image is not an integer multiple of the height of the block. In this case, size adjustment information (e.g., size adjustment direction, size adjustment value, etc.) can be determined according to the encoding / decoding information (e.g., image size, block size, etc.). Alternatively, size adjustment can be performed according to the characteristics and type of the image (e.g., 360-degree image), and the size adjustment information can be explicitly generated or assigned to a predetermined value. This is not limited to the above example, and variations to other examples are also possible.

[0195] The image resizing type identification stage allows for the identification of the image resizing type. The image resizing type can be defined by the resizing method, resizing information, etc. For example, resizing can be performed using a scale factor, or using an offset factor. However, it is not limited to these methods; a combination of the above methods is also possible. For the sake of explanation, we will focus on resizing using scale and offset factors.

[0196] In the case of a scale factor, resizing can be performed by multiplying or dividing based on the image size. Information about the resizing operation (e.g., expansion or reduction) can be explicitly generated, and the expansion or reduction process can be performed according to this information. Alternatively, depending on the encoding / decoding settings, the resizing process can be performed with a predetermined operation (e.g., either expansion or reduction), in which case information about the resizing operation can be omitted. For example, if image resizing is activated at the image resizing instruction stage, the image resizing may be performed with a predetermined operation.

[0197] The resizing direction can be at least one direction selected from the up, down, left, and right directions. Depending on the resizing direction, at least one scale factor may be required. That is, one scale factor may be required for each direction (unidirectional in this example), one scale factor may be required depending on the horizontal or vertical direction (bidirectional in this example), and one scale factor may be required depending on the overall direction of the image (omnidirectional in this example). Furthermore, the resizing direction is not limited to the above example and can be transformed into other examples.

[0198] The scale factor can have a positive value and its range information can be set differently depending on the encoding / decoding settings. For example, when generating information by mixing sizing operations and scale factors, the scale factor can be used as a multiplicand. A value greater than 0 or less than 1 can indicate a shrinking operation, a value greater than 1 can indicate an expansion operation, and a value of 1 can indicate no sizing operation. As another example, when generating scale factor information separately from sizing operations, the scale factor can be used as a multiplicand in the case of expansion operations, and as a divisor in the case of shrinking operations.

[0199] Referring again to Figures 7a and 7b, the process of changing from the original image (S0, T0) to the resized image (S1, T1 in this example) using a scale factor can be explained.

[0200] For example, if a single scale factor (called sc) is used depending on the overall direction of the image, and the resizing direction is down + right, then the resizing direction is ER, EB (or RR, RB), the resizing values ​​Var_L (Exp_L or Rec_L) and Var_T (Exp_T or Rec_T) are 0, and Var_R (Exp_R or Rec_R) and Var_B (Exp_B or Rec_B) can be expressed as P_Width × (sc-1) and P_Height × (sc-1). Therefore, the resized image can be (P_Width × sc) × (P_Height × sc).

[0201] For example, if the scaling factors (sc_w and sc_h in this example) are used according to the horizontal or vertical direction of the image, and the resizing directions are left + right and up + down (when both are working, it becomes up + down + left + right), then the resizing directions are ET, EB, EL, and ER, and the resizing values ​​Var_T and Var_B can be P_Height × (sc_h-1) / 2, and Var_L and Var_R can be P_Width × (sc_w-1) / 2. Therefore, the resized image can be (P_Width × sc_w) × (P_Height × sc_h).

[0202] In the case of an offset factor, size adjustment can be performed by adding or subtracting based on the image size. Alternatively, size adjustment can be performed by adding or subtracting based on the image encoding / decoding information. Or, size adjustment can be performed by adding or subtracting independently. In other words, the size adjustment process can be set to be dependent or independent.

[0203] Information regarding resizing operations (e.g., expansion or reduction) can be explicitly generated, and the expansion or reduction process can be performed according to this information. Furthermore, depending on the encoding / decoding settings, resizing operations can be performed using a predetermined operation (e.g., either expansion or reduction), in which case information regarding the resizing operation can be omitted. For example, if image resizing is activated at the image resizing instruction stage, the image resizing may be performed using a predetermined operation.

[0204] The resizing direction can be at least one of the up, down, left, or right directions. Depending on the resizing direction, at least one offset factor may be required. That is, one offset factor may be required in each direction (unidirectional in this example), one offset factor may be required depending on the horizontal or vertical direction (symmetrical bidirectional in this example), one offset factor may be required depending on a partial combination of each direction (asymmetrical bidirectional in this example), and one offset factor may be required depending on the overall direction of the image (omnidirectional in this example). Furthermore, the resizing direction is not limited to the above example and can be transformed into other examples.

[0205] The offset factor can have a positive value or both positive and negative values, and its range information can be set differently depending on the encoding / decoding settings. For example, when generating information by mixing size adjustment operations and the offset factor (assuming in this example that it has both positive and negative values), the offset factor can be used as a value to be added or subtracted depending on the sign information of the offset factor. If the offset factor is greater than 0, it can mean an expansion operation; if it is less than 0, it can mean a contraction operation; and if it is 0, it can mean no size adjustment is performed. As another example, when generating offset factor information separately from size adjustment operations (assuming in this example that it has a positive value), the offset factor can be used as a value to be added or subtracted depending on the size adjustment operation. If it is greater than 0, it can mean that an expansion or contraction operation is performed depending on the size adjustment operation; and if it is 0, it can mean no size adjustment is performed.

[0206] Referring again to Figures 7a and 7b of Figure 7, we can explain how to change from the original image (S0, T0) to the resized image (S1, T1) using an offset factor.

[0207] For example, if a single offset factor (called os) is used depending on the overall orientation of the image, and the resizing direction is up + down + left + right, then the resizing direction is ET, EB, EL, ER (or RT, RB, RL, RR), and the resizing values ​​Var_T, Var_B, Var_L, Var_R can be os. The resized image size can be (P_Width + os) × (P_Height + os).

[0208] For example, if the offset factors (os_w, os_h) are used according to the horizontal or vertical direction of the image, and the resizing direction is left + right or up + down (when both are working, it becomes up + down + left + right), then the resizing direction is ET, EB, EL, ER (or RT, RB, RL, RR), and the resizing values ​​Var_T and Var_B can be os_h, while Var_L and Var_R can be os_w. The resized image size can be {P_Width + (os_w × 2)} × {P_Height + (os_h × 2)}.

[0209] For example, if the resizing direction is downward and to the right (when they work together, downward + to the right), and the respective offset factors (os_b, os_r) are used according to the resizing direction, then the resizing direction can be EB, ER (or RB, RR), and the resizing value Var_B can be os_b, and Var_R can be os_r. The resized image size can be (P_Width + os_r) × (P_Height + os_b).

[0210] For example, if we use different offset factors (os_t, os_b, os_l, os_r) depending on the orientation of the image, and the resizing directions are up, down, left, and right (when all are working, it's up + down + left + right), then the resizing directions are ET, EB, EL, ER (or RT, RB, RL, RR), and the resizing values ​​Var_T can be os_t, Var_B can be os_b, Var_L can be os_l, and Var_R can be os_r. The resized image size can be (P_Width + os_l + os_r) × (P_Height + os_t + os_b).

[0211] The above example shows a case where the offset factor is used as a size adjustment value (Var_T, Var_B, Var_L, Var_R) in the size adjustment process. That is, it means that the offset factor is used directly as a size adjustment value. This can be an example of an independently performed size adjustment. Alternatively, the offset factor can also be used as an input variable for the size adjustment value. In detail, the offset factor can be assigned as an input variable, and the size adjustment value can be obtained through a series of processes depending on the encoding / decoding settings. This can be an example of a size adjustment performed based on predetermined information (e.g., image size, encoding / decoding information, etc.), or an example of a dependent size adjustment.

[0212] For example, the offset factor may be a multiple of a predetermined value (in this example, an integer) (e.g., 1, 2, 4, 6, 8, 16, etc.) or an exponent (e.g., 2 raised to the power of an exponent, such as 1, 2, 4, 8, 16, 32, 64, 128, 256, etc.). Alternatively, it may be a multiple or exponent of a value obtained based on the encoding / decoding settings (e.g., a value set based on the motion search range for inter-screen prediction). Alternatively, it may be a multiple or exponent of a unit obtained from the picture division section (assuming A × B in this example). Alternatively, it may be a multiple of a unit obtained from the picture division section (assuming E × F in the case of tiles, etc.).

[0213] Alternatively, the values ​​may be smaller than or within the same range as the horizontal and vertical dimensions obtained from the picture division. The multiples or exponents in the above example may also include cases where the value is 1, and are not limited to the above example; other variations are possible. For example, if the offset factor is n, then Var_x is 2 × n or 2 n It is possible.

[0214] Furthermore, it is possible to support individual offset factors according to the color components, and by supporting the offset factor for some color components, offset factor information for other color components can be derived. For example, if offset factor A for the luminance component (assuming in this example the ratio of luminance component to chrominance component is 2:1) is explicitly generated, offset factor A / 2 for the chrominance component can be implicitly obtained. Alternatively, if offset factor A for the chrominance component is explicitly generated, offset factor 2A for the luminance component can be implicitly obtained.

[0215] Information regarding the size adjustment direction and size adjustment value can be explicitly generated, and the size adjustment process can be performed according to this information. Alternatively, it can be implicitly determined according to the encoding / decoding settings, thereby enabling the size adjustment process. At least one predetermined direction or adjustment value can be assigned, in which case related information can be omitted. In this case, the encoding / decoding settings can be determined based on image characteristics, type, encoding information, etc. For example, at least one size adjustment direction can be predetermined by at least one size adjustment operation, at least one size adjustment value can be predetermined by at least one size adjustment operation, and at least one size adjustment value can be predetermined by at least one size adjustment direction. Furthermore, the size adjustment direction and size adjustment value in the reverse size adjustment process can be derived from the size adjustment direction and size adjustment value applied in the size adjustment process. In this case, the implicitly determined size adjustment value may be one of the examples mentioned above (examples in which various size adjustment values ​​are obtained).

[0216] Furthermore, while the above examples described the cases of multiplication and division, these can also be implemented using shift operations depending on the implementation of the encoder / decoder. Multiplication can be implemented using a left shift operation, and division can be implemented using a right shift operation. This explanation is not limited to the above examples and may be applicable in general to the present invention.

[0217] During the image resizing execution phase, image resizing can be performed based on identified resizing information. Specifically, image resizing can be performed based on information such as resizing type, resizing operation, resizing direction, and resizing value, and encoding / decoding can be performed based on the acquired resized image.

[0218] Furthermore, during the image resizing execution phase, resizing can be performed using at least one data processing method. More specifically, resizing can be performed using at least one data processing method on the area to be resized, depending on the resizing type and resizing operation. For example, depending on the resizing type, it can be determined how to fill the data if the resizing is expansion, or how to remove data if the resizing process is reduction.

[0219] In summary, image resizing can be performed based on resizing information identified during the image resizing execution phase. Alternatively, image resizing can be performed during the image resizing execution phase based on the resizing information and a data processing method. The difference between these two cases lies in whether only the size of the image to be encoded / decoded is adjusted, or whether both the image size and data processing of the area to be resized are considered. Whether or not the data processing method is included in the image resizing execution phase can be determined depending on the application stage and position of the resizing process. The examples described later will mainly focus on examples where resizing is performed based on a data processing method, but are not limited to these.

[0220] When performing resizing using an offset factor, various methods can be used for both expansion and reduction. In the case of expansion, resizing can be performed by filling at least one data point, and in the case of reduction, resizing can be performed by removing at least one data point. In this case, when resizing using an offset factor, new data or existing image data can be directly or transformed to fill the resizing area (expansion), and in the resizing area (reduction), data can be removed by simple removal or removal through a series of processes.

[0221] When performing size adjustment using a scale factor, in some cases (e.g., hierarchical coding), expansion can be performed by applying upsampling, and reduction can be performed by applying downsampling. For example, in the case of expansion, at least one upsampling filter can be used, and in the case of reduction, at least one downsampling filter can be used, and the filters applied horizontally and vertically may be the same or different. In this case, when performing size adjustment using a scale factor, new data may be generated or removed in the size adjustment area, or the data of the existing image may be rearranged using methods such as interpolation. The data processing method related to performing size adjustment can be distinguished by the filter used for sampling. Also, in some cases (e.g., similar to an offset factor), expansion can be performed by filling at least one data point, and reduction can be performed by removing at least one data point. This invention will mainly describe the data processing method when performing size adjustment using an offset factor.

[0222] Generally, one predetermined data processing method can be used for the area to be resized, but as shown in the example below, at least one data processing method can also be used for the area to be resized, and selection information for the data processing method can be generated. In the former case, it can be said that resizing is performed via a fixed data processing method, and in the latter case, via an adaptive data processing method.

[0223] Furthermore, a data processing method common to all areas added or deleted during resizing (TL, TC, TR, ..., BR in Figures 7a and 7b) can be applied, or a data processing method can be applied to only a portion of the areas added or deleted during resizing (for example, each of TL to BR in Figures 7a and 7b, or a combination thereof).

[0224] Figure 8 is an illustrative diagram of a method for configuring the expanded region in an image size adjustment method according to one embodiment of the present invention.

[0225] Referring to 8a, the image can be divided into TL, TC, TR, LC, C, RC, BL, BC, and BR regions for explanatory purposes, which correspond to the upper left, top, upper right, left, center, right, lower left, bottom, and lower right positions of the image, respectively. The following describes the case where the image is extended in the downward and rightward directions, but it should be understood that this can be similarly applied to other directions as well.

[0226] The area added as the image is expanded can be constructed in various ways; for example, it can be filled with an arbitrary value or by referencing some of the image's data.

[0227] Referring to 8b, it is possible to fill the region (A0, A2) which can be extended to any pixel value. Any pixel value can be determined using various methods.

[0228] As an example, any pixel value can be one pixel belonging to the range of pixel values that can be represented by the bit depth {for example, from 0 to 1<<(bit_depth)-1}. For example, it can be the minimum value, the maximum value, the median value {such as 1<<(bit_depth-1), etc.} of the range of the pixel values (where bit_depth is the bit depth).

[0229] As an example, any pixel value can be one pixel belonging to the range of pixel values of the pixels belonging to the image {for example, from min P to max P up to. min P and max P are the minimum value and the maximum value of the pixels belonging to the image. min P is equal to or greater than 0, and max P is equal to or smaller than 1<<(bit_depth)-1}. For example, any pixel value can be the minimum value, the maximum value, the median value, the average (of at least two pixels), the weighted sum, etc. of the range of the pixel values.

[0230] As an example, any pixel value can be a value determined by the range of pixel values of a partial region belonging to the image. For example, when configuring A0, the partial region can be TR+RC+BR. Also, the partial region can be set as the corresponding region of 3×9 of TR, RC, BR, or can be set as the corresponding region of 1×9 <assuming the rightmost line>. This can be adjusted according to the encoding / decoding settings. At this time, the partial region may also be a unit divided from the picture segmentation unit. Specifically, any pixel value can be the minimum value, the maximum value, the median value, the average (of at least two pixels), the weighted sum, etc. of the range of the pixel values.

[0231] Referring again to 8b, the region A1 added in accordance with the image expansion can be filled with pattern information generated using multiple pixel values ​​(for example, a pattern is assumed to be something using multiple pixels; it does not necessarily have to follow a certain rule). In this case, the pattern information can be defined or related information can be generated according to the encoding / decoding settings, and the expanded region can be filled using at least one piece of pattern information.

[0232] Referring to 8c, the region added in response to image expansion can be constructed by referencing pixels of a portion of the image. More specifically, the added region can be constructed by copying or padding pixels of an adjacent region (hereinafter referred to as "reference pixels"). In this case, the pixels of the adjacent region to the added region may be pixels before encoding or pixels after encoding (or decoding). For example, when resizing is performed in the pre-encoding stage, the reference pixels may represent pixels of the input image, while when resizing is performed in the screen prediction reference pixel generation stage, reference image generation stage, or filtering stage, the reference pixels may represent pixels of the restored image. In this example, we assume, but are not limited to, the use of pixels closest to the added region.

[0233] Region A0, which is extended to the left or right in relation to adjusting the horizontal size of the image, can be constructed by horizontally padding (Z0) the surrounding pixels adjacent to the extended region A0. Region A1, which is extended upwards or downwards in relation to adjusting the vertical size of the image, can be constructed by vertically padding (Z1) the surrounding pixels adjacent to the extended region A1. Furthermore, region A2, which is extended downwards to the right, can be constructed by diagonally padding (Z2) the surrounding pixels adjacent to the extended region A2.

[0234] By referring to 8d, it is possible to construct the extended region B'0 to B'2 by referencing data B0 to B2 of a portion of the image. 8d can be distinguished from 8c in that it allows referencing regions that are not adjacent to the extended region.

[0235] For example, when there is an area with a high correlation with the area to be expanded within the image, the area to be expanded can be filled by referring to the pixels of the area with a high correlation. At this time, the position information, area size information, etc. of the area with a high correlation can be generated. Or, when there is an area with a high correlation through encoding / decoding information such as the characteristics and types of the image, and the position information, size information, etc. of the area with a high correlation can be implicitly confirmed (for example, in the case of a 360-degree image), the data of the corresponding area can be filled into the area to be expanded. At this time, the position information, area size information, etc. of the area can be omitted.

[0236] As an example, in the case of the area B'2 expanded in the left or right direction related to the horizontal size adjustment of the image, the area to be expanded can be filled by referring to the pixels of the area B2 on the opposite side of the area to be expanded in the left or right direction related to the horizontal size adjustment.

[0237] As an example, in the case of the area B'1 expanded in the up or down direction related to the vertical size adjustment of the image, the area to be expanded can be filled by referring to the pixels of the area B1 on the opposite side of the area to be expanded in the up or right direction related to the vertical size adjustment.

[0238] As an example, in the case of the area B'0 expanded in the diagonal direction (in this example, with the center of the image as the reference) for partial size adjustment of the image, the area to be expanded can be filled by referring to the pixels of the area B0, TL on the opposite side of the area to be expanded.

[0239] In the above example, the case of obtaining data of an area that has continuity at the boundaries at both ends of the image and is in a position symmetric to the size adjustment direction has been described, but it is not limited to this, and it is also possible to obtain data from other areas TL~BR.

[0240] When filling an expanded area with data from a portion of an image, the data from that portion can either be copied directly and used for filling, or the data from that portion can be transformed based on the image's characteristics and type before filling. Copying directly means using the pixel values ​​of that portion as they are, while transformation means not using the pixel values ​​of that portion as they are. That is, the transformation process may result in a change in at least one pixel value in that portion before filling the expanded area, or at least one pixel acquisition position may differ. In other words, to fill the expanded area A×B, the data from A×B of the portion in question may not be used, but rather the data from C×D may be used. To put it another way, at least one motion vector applied to the pixels being filled may differ. The above example could occur when filling an expanded area using data from other surfaces in a 360-degree image composed of multiple surfaces depending on the projection format. The data processing method for filling an expanded area through image resizing is not limited to the above example; it can be improved and modified, or additional data processing methods can be used.

[0241] Depending on the encoding / decoding settings, a group of candidate data processing methods can be provided, and data processing method selection information can be generated from among these candidates and included in the bitstream. For example, one data processing method can be selected from methods such as filling using predetermined pixel values, filling by copying outer pixels, filling by copying a portion of the image, or filling by transforming a portion of the image, and selection information for this method can be generated. Furthermore, the data processing method can be implicitly determined.

[0242] For example, the data processing method applied to the entire area expanded by resizing the image (in this example, TL to BR in FIG. 7a) can be any one of the methods of filling with a predetermined pixel value, copying and filling the outer pixels, copying and filling a partial area of the image, converting and filling a partial area of the image, and other methods, and selection information for this can be generated. Also, one predetermined data processing method applied to the entire area can be determined.

[0243] Alternatively, the data processing method applied to the area expanded by resizing the image (in this example, each area of TL to BR in FIG. 7a of FIG. 7 or two or more of them) can be any one of the methods of filling with a predetermined pixel value, copying and filling the outer pixels, copying and filling a partial area of the image, converting and filling a partial area of the image, and other methods, and selection information for this can be generated. Also, one predetermined data processing method applied to at least one area can be determined.

[0244] FIG. 9 is an exemplary diagram of a method of forming an area to be deleted and an area to be generated by reducing the image size in an image size adjustment method according to an embodiment of the present invention.

[0245] The area deleted in the image reduction process can be not only simply removed but also removed after a series of utilization processes.

[0246] Referring to 9a, in the image reduction process, partial areas A0, A1, and A2 can be simply removed without an additional utilization process. At this time, the image A may be subdivided and may be called TL to BR as in FIG. 8a.

[0247] Referring to 9b, a portion of region A0-A2 is removed, but it can be used as reference information during the encoding / decoding of image A. For example, the removed portion of region A0-A2 can be used in the restoration or correction process of a portion of the reduced image A. In the restoration or correction process, a weighted sum or average of the two regions (the removed region and the generated region) can be used. Furthermore, the restoration or correction process may be applicable when the two regions have a high correlation.

[0248] For example, region B'2, which is deleted when the image is scaled down to the left or right in relation to the horizontal size adjustment, can be used to restore or correct the pixels of region B2, LC on the opposite side of the scaled-down region in relation to the horizontal size adjustment, and then removed from memory.

[0249] For example, the region B'1 that is deleted upwards or downwards in relation to the vertical size adjustment of an image can be used in the encoding / decoding process (restoration or correction process) of the region B1, TR opposite to the region being reduced, in the upward or downward direction related to the vertical size adjustment, and then removed from memory.

[0250] For example, a region B'0 that is reduced in size (in this example, diagonally from the center of the image) can be used in the encoding / decoding process (such as restoration or correction process) of the opposite region B0, TL, and then removed from memory.

[0251] In the above example, we described the case where data recovery or correction is used for a region where continuity exists at both ends of the image boundary and is located symmetrically with respect to the size adjustment direction. However, this is not limited to this case; the data can also be used for data recovery or correction of other regions TL~BR that are not symmetrical, and then removed from memory.

[0252] The data processing method for removing the reduced area of ​​the present invention is not limited to the above example, and improvements and modifications thereto, or additional data processing methods, can be used.

[0253] Depending on the encoding / decoding settings, a group of candidate data processing methods can be provided, and selection information for these methods can be generated and included in the bitstream. For example, one data processing method can be selected from methods such as simply removing the area to be resized, or removing the area after using it in a series of processes, and selection information for this method can be generated. Furthermore, the data processing method can be determined implicitly.

[0254] For example, the data processing method applied to the entire area that is deleted when the image is resized (in this example, TL~BR in 7b of Figure 7) can be one of the following: simply removing it, removing it after using it in a series of processes, or other methods, and selection information for this can be generated. Furthermore, the data processing method can be implicitly determined.

[0255] Alternatively, the data processing method applied to individual regions that are reduced in size by image resizing (in this example, TL to BR in Figure 7b) can be one of the following: simply removing them, removing them after using them in a series of processes, or other methods, and selection information for these methods can be generated. Furthermore, the data processing method can be implicitly determined.

[0256] In the above example, we explained the case of performing size adjustment through size adjustment operations (expansion and reduction). However, in some cases, it may be possible to apply this to situations where a size adjustment operation (expansion in this example) is performed first, followed by the reverse process of a size adjustment operation (reduction in this example).

[0257] For example, one method might be selected to fill the expanded area with partial image data, and then, in the reverse process, use the area to be reduced in a partial image data recovery or correction process before removing it. Alternatively, one method might be selected to fill the expanded area with copies of the outer pixels, and then, in the reverse process, simply remove the area to be reduced. In other words, the data processing method in the reverse process can be determined based on the data processing method selected in the image size adjustment process.

[0258] Unlike the example above, the data processing methods for the image resizing process and the reverse process can also be independent of each other. That is, the data processing method in the reverse process can be selected independently of the data processing method selected in the image resizing process. For example, a method can be chosen to fill the expanded area with some of the image data, and a method can be chosen to simply remove the area to be reduced in the reverse process.

[0259] In this invention, the data processing method in the image size adjustment process can be implicitly determined according to the encoding / decoding settings, and the data processing method in the reverse process can be implicitly determined according to the encoding / decoding settings. Alternatively, the data processing method in the image size adjustment process can be explicitly generated, and the data processing method in the reverse process can be explicitly generated. Alternatively, the data processing method in the image size adjustment process can be explicitly generated, and the data processing method in the reverse process can be implicitly determined based on the above data processing method.

[0260] Next, an example of image resizing using an encoding / decoding device according to one embodiment of the present invention is shown. In the example described later, the resizing process is given as expansion, and the reverse resizing process is given as reduction. The difference between the "image before resizing" and the "image after resizing" can refer to the size of the image, and some of the resizing-related information can be explicitly generated or some can be implicitly determined depending on the encoding / decoding settings. Furthermore, the resizing-related information can include information for both the resizing process and the reverse resizing process.

[0261] As a first example, a size adjustment process for the input image can be performed before the start of encoding. After performing the size adjustment using size adjustment information (for example, size adjustment operation, size adjustment direction, size adjustment value, data processing method, etc. The data processing method is what is used in the size adjustment process), the image after size adjustment can be encoded. After completion of the encoding, it can be stored in memory, and the image encoding data (in this example, meaning the image after size adjustment) can be recorded in a bitstream and transmitted.

[0262] A size adjustment process can be performed before the start of decoding. After performing the size adjustment using size adjustment information (for example, size adjustment operation, size adjustment direction, size adjustment value, etc.), the image decoding data after size adjustment can be parsed and decoded. After completion of the decoding, it can be stored in memory, and the inverse size adjustment process (in this example, using data processing methods, etc. This is what is used in the inverse size adjustment process) can be performed to change the output image to the image before size adjustment.

[0263] As a second example, a size adjustment process for the reference image can be performed before the start of encoding. After performing the size adjustment using size adjustment information (for example, size adjustment operation, size adjustment direction, size adjustment value, data processing method, etc. The data processing method is what is used in the size adjustment process), the image after size adjustment (in this example, the reference image after size adjustment) can be stored in memory, and this can be used to encode the image. After completion of the encoding, the image encoding data (in this example, meaning what is encoded using the reference image) can be recorded in a bitstream and transmitted. Also, when the encoded image is stored in memory as a reference image, the size adjustment process can be performed as in the above process.

[0264] Before starting the decoding process, a resizing process can be performed on the reference image. Using the resizing information (e.g., resizing operation, resizing direction, resizing value, data processing method, etc.; the data processing method is the one used in the resizing process), the resized image (in this example, the resized reference image) can be stored in memory, and the image decoding data (in this example, the same as that encoded by the encoder using the reference image) can be parsed and decoded. After the decoding is complete, it can be generated as an output image, and if the decoded image is included in the reference image and stored in memory, the resizing process can be performed as described above.

[0265] As a third example, after the completion of encoding (more specifically, meaning the completion of encoding excluding the filtering process) and before the start of image filtering (in this example, assumed to be a deblocking filter), a resizing process can be performed on the image. After resizing using resizing information (e.g., resizing operation, resizing direction, resizing value, data processing method, etc. The data processing method is the one used in the resizing process), a resized image can be generated, and filtering can be applied to the resized image. After the completion of filtering, the reverse resizing process can be performed to revert to the image before resizing.

[0266] (More specifically, this means the completion of decoding excluding the filtering process.) After decoding is complete, an image resizing process can be performed on the image before the filtering process begins. After resizing using resizing information (e.g., resizing operation, resizing direction, resizing value, data processing method, etc. The data processing method is the one used in the resizing process), a resized image can be generated, and filtering can be applied to the resized image. After filtering is complete, the reverse resizing process can be performed to revert to the image before resizing.

[0267] In the above examples, in some cases (the first and third examples), the size adjustment process and the reverse size adjustment process may be performed, while in other cases (the second example), only the size adjustment process may be performed.

[0268] Furthermore, in some cases (the second and third examples), the size adjustment process in the encoder and decoder may be the same, while in other cases (the first example), the size adjustment process in the encoder and decoder may or may not be the same. In this case, the difference in the size adjustment process in the encoder / decoder may be the size adjustment execution stage. For example, in some cases (the encoder in this example), a size adjustment execution stage may be included that considers image size adjustment and data processing of the area to be resized, and in some cases (the decoder in this example), a size adjustment execution stage may be included that considers image size adjustment. In this case, the data processing in the former can correspond to the data processing in the reverse size adjustment process of the latter.

[0269] Furthermore, in some cases (e.g., the third example), the resizing process is applied only at that specific stage, and it is not necessary to save the resizing area to memory. For example, the area can be temporarily saved to memory for use in the filtering process, filtered, and then removed via the reverse resizing process. In this case, there is no change in the size of the image due to the resizing. The above examples are not the only ones that can be applied; variations to other examples are also possible.

[0270] The image size can be changed through the aforementioned resizing process, thereby changing the coordinates of some pixels in the image. This can affect the operation of the picture division unit. In this invention, division can be performed in block units based on the image before resizing through the above process, or based on the image after resizing. Furthermore, division can be performed in some units (e.g., tiles, slices, etc.) based on the image before resizing, or in some units based on the image after resizing. This can be determined according to the encoding / decoding settings. In this invention, the case in which the picture division unit operates based on the image after resizing (e.g., the image division process after the resizing process) will be described in detail, but other variations are also possible. These will be explained in the following section in relation to the above example using multiple image settings.

[0271] The encoder records the information generated during the above process into a bitstream in units of at least one of the following: sequence, picture, slice, or tile. The decoder then parses the related information from the bitstream. This information may also be included in the bitstream in the form of SEI or metadata.

[0272] While it is common practice to encode / decode the input image as is, this issue can also occur when the image is reconstructed before encoding / decoding. For example, image reconstruction can be performed to improve encoding efficiency, to take into account the network and user environment, and depending on the type and characteristics of the image.

[0273] In this invention, the image reconstruction process can be performed independently or as a reverse process. In the examples described later, the reconstruction process will be the main focus, but the content of the reverse reconstruction process can be derived in reverse from the reconstruction process.

[0274] Figure 10 is an illustrative diagram of image reconstruction according to one embodiment of the present invention.

[0275] When 10a is the initial input image, 10a to 10d are example images obtained by applying a rotation that includes 0 degrees to the image (for example, a group of candidates can be formed by sampling 360 degrees into k intervals. k can have values ​​such as 2, 4, or 8, and in this example, we assume it is 4), and 10e to 10h are example images obtained by applying inversion (or symmetry) based on 10a or 10b to 10d.

[0276] The starting position or scan order of the image can be changed depending on the image reconstruction, or it can follow a predetermined starting position and scan order regardless of whether reconstruction is performed or not. This can be determined depending on the encoding / decoding settings. In the embodiments described later, we will assume that the image follows a predetermined starting position (e.g., the upper left position of the image) and scan order (e.g., raster scan) regardless of whether image reconstruction is performed or not.

[0277] An image encoding method and decoding method according to one embodiment of the present invention may include the following image reconstruction steps. In this case, the image reconstruction process may include an image reconstruction instruction step, an image reconstruction type identification step, and an image reconstruction execution step. Furthermore, the image encoding device and decoding device may be configured to include an image reconstruction instruction unit, an image reconstruction type identification unit, and an image reconstruction execution unit that implement the image reconstruction instruction step, the image reconstruction type identification step, and the image reconstruction execution step. In the case of encoding, associated syntactic elements can be generated, and in the case of decoding, associated syntactic elements can be parsed.

[0278] During the image reconstruction instruction phase, it is possible to decide whether or not to perform image reconstruction. For example, if a signal instructing image reconstruction (e.g., convert_enabled_flag) is detected, reconstruction can be performed. If no such signal is detected, reconstruction may not be performed, or reconstruction may be performed after checking other encoding / decoding information. Furthermore, even if a signal instructing image reconstruction is not provided, the signal instructing reconstruction may be implicitly activated or deactivated depending on the encoding / decoding settings (e.g., image characteristics, type, etc.). If reconstruction is performed, reconstruction-related information can be generated or implicitly determined.

[0279] If a signal instructing image reconstruction is provided, this signal indicates whether or not to perform image reconstruction, and it is possible to confirm whether or not to perform image reconstruction according to this signal. For example, if a signal instructing image reconstruction (e.g., convert_enabled_flag) is confirmed and the signal is activated (e.g., convert_enabled_flag=1), reconstruction may be performed, but if the signal is deactivated (e.g., convert_enabled_flag=0), reconstruction may not be performed.

[0280] Furthermore, if no signal is provided to instruct image reconstruction, reconstruction will not be performed, or whether or not the image in question has been reconstructed can be confirmed by other signals. For example, reconstruction can be performed according to the characteristics and type of the image (e.g., a 360-degree image), and the reconstruction information can be explicitly generated or assigned to a predetermined value. This is not limited to the above example, and variations to other examples are also possible.

[0281] In the image reconstruction type identification stage, the image reconstruction type can be identified. The image reconstruction type can be defined by the reconstruction method, reconstruction mode information, etc. The reconstruction method (e.g., convert_type_flag) can include rotation, inversion, etc., and the reconstruction mode information can include the mode of the reconstruction method (e.g., convert_mode). In this case, the reconstruction-related information can consist of the reconstruction method and mode information. That is, it can consist of at least one syntactic element. At this time, the number of candidate mode information groups for each reconstruction method may be the same or different.

[0282] For example, in the case of rotation, candidates such as 10a to 10d may have a difference of a certain interval (90 degrees in this example), and when 10a is set to a 0-degree rotation, 10b to 10d may be examples of applying rotations of 90 degrees, 180 degrees, and 270 degrees, respectively (in this example, the angle is measured clockwise).

[0283] For example, in the case of inversion, the candidates could include 10a, 10e, and 10f. If 10a is not inverted, then 10e and 10f could be examples where horizontal inversion and vertical inversion are applied, respectively.

[0284] The above examples illustrate settings for rotations with a fixed interval and settings for partial inversions, but they are merely examples of image reconstruction and are not limited to the above cases. Other interval differences can include other inversion operations, etc. This can be determined depending on the encoding / decoding settings.

[0285] Alternatively, it may include integrated information (e.g., convert_com_flag) generated by combining the method of reconstruction and the resulting mode information. In this case, the reconstruction-related information can consist of information that combines the method of reconstruction and the mode information.

[0286] For example, the integrated information may include candidates such as 10a to 10f. This could be an example of applying 0-degree rotation, 90-degree rotation, 180-degree rotation, 270-degree rotation, horizontal flip, and vertical flip based on 10a.

[0287] Alternatively, the integrated information may include candidates such as 10a to 10h. This could be an example of applying 0-degree rotation, 90-degree rotation, 180-degree rotation, 270-degree rotation, horizontal flip, vertical flip, horizontal flip after 90-degree rotation (or 90-degree rotation after horizontal flip), vertical flip after 90-degree rotation (or 90-degree rotation after vertical flip), or an example of applying 0-degree rotation, 90-degree rotation, 180-degree rotation, 270-degree rotation, horizontal flip, horizontal flip after 180-degree rotation (180-degree rotation after horizontal flip), horizontal flip after 90-degree rotation (90-degree rotation after horizontal flip), horizontal flip after 270-degree rotation (270-degree rotation after horizontal flip).

[0288] The candidate group can consist of modes in which rotation is applied, modes in which inversion is applied, and modes in which rotation and inversion are mixed. The mixed-configuration mode simply contains mode information from the method of reconstruction, and can include modes generated by mixing the mode information from each method. In this case, it can include modes generated by mixing at least one mode from some methods (e.g., rotation) and at least one mode from some methods (e.g., inversion), and the above example includes the case where one mode from some methods and multiple modes from some methods are mixed to generate (in this example, 90-degree rotation + multiple inversions / left-right inversions + multiple rotations). The mixed-configuration information can consist of a candidate group that includes cases where reconstruction is not applied {10a in this example}, and cases where reconstruction is not applied can be included as the first candidate group (e.g., assigning index 0).

[0289] Alternatively, it may include mode information based on a predetermined reconstruction method. In this case, the reconstruction-related information can consist of mode information based on a predetermined reconstruction method. That is, information about the reconstruction method can be omitted and can consist of a single syntactic element related to the mode information.

[0290] For example, it can be constructed by including candidates such as 10a to 10d related to rotation. Alternatively, it can be constructed by including candidates such as 10a, 10e, and 10f related to inversion.

[0291] The sizes of the images before and after the image reconstruction process may be the same, or they may differ in at least one length. This can be determined depending on the encoding / decoding settings. The image reconstruction process is a process of rearranging pixels within the image (in this example, the reverse image reconstruction process is the reverse pixel rearrangement process, which can be derived in reverse from the pixel rearrangement process), and the position of at least one pixel can be changed. The rearrangement of the pixel may be performed according to rules based on the image reconstruction type information.

[0292] In this process, the pixel rearrangement can be influenced by the size and shape of the image (e.g., square or rectangular). More specifically, the width and height of the image before and after the reconstruction process can act as variables in the pixel rearrangement process.

[0293] For example, at least one ratio of the following (e.g., the ratio of the width of the image before reconstruction to the width of the image after reconstruction, the ratio of the width of the image before reconstruction to the height of the image after reconstruction, the ratio of the height of the image before reconstruction to the width of the image after reconstruction, and the ratio of the height of the image before reconstruction to the height of the image after reconstruction) can act as a variable in the pixel rearrangement process.

[0294] In the above example, if the image size is the same before and after the reconstruction process, the ratio of the image's width to its height can act as a variable in the pixel rearrangement process. Furthermore, if the image is square, the ratio of the image's length before the reconstruction process to its length after the reconstruction process can act as a variable in the pixel rearrangement process.

[0295] During the image reconstruction execution phase, image reconstruction can be performed based on identified reconstruction information. Specifically, image reconstruction can be performed based on information such as reconstruction type and reconstruction mode, and encoding / decoding can be performed based on the acquired reconstructed image.

[0296] Next, an example of image reconstruction using an encoding / decoding device according to one embodiment of the present invention is shown.

[0297] A reconstruction process can be performed on the input image before encoding begins. After reconstruction using reconstruction information (e.g., image reconstruction type, reconstruction mode, etc.), the reconstructed image can be encoded. After encoding is complete, the image can be stored in memory, and the encoded image data can be recorded in a bitstream and transmitted.

[0298] The reconstruction process can be performed before the start of decoding. After reconstruction using reconstruction information (e.g., image reconstruction type, reconstruction mode, etc.), the image decoding data can be parsed and decoded. After decoding is complete, the image can be saved to memory, and after performing the reverse reconstruction process to convert it back to the pre-reconstruction image, the image can be output.

[0299] The encoder records the information generated during the above process into a bitstream in units of at least one of the following: sequence, picture, slice, or tile. The decoder then parses the related information from the bitstream. This information may also be included in the bitstream in the form of SEI or metadata.

[0300] [Table 1]

[0301] Table 1 shows examples of syntactic elements related to splitting during image setup. The examples described later will focus on the added syntactic elements. Furthermore, the syntactic elements in the examples described later are not limited to a specific unit, but may be syntactic elements supported by various units such as sequences, pictures, slices, and tiles. Alternatively, they may be syntactic elements included in SEI, metadata, etc. Also, the types of syntactic elements supported in the examples described later, the order of syntactic elements, and the conditions are limited only to these examples and may be changed and determined depending on the encoding / decoding settings.

[0302] In Table 1, `tile_header_enabled_flag` is a syntactic element indicating whether or not to support encoding / decoding settings for tiles. When enabled (tile_header_enabled_flag=1), tiles can have per-tile encoding / decoding settings, while when disabled (tile_header_enabled_flag=0), tiles cannot have per-tile encoding / decoding settings and can instead be assigned to higher-level encoding / decoding settings.

[0303] `tile_coded_flag` is a syntactic element that indicates whether to encode / decode a tile. When activated (tile_coded_flag=1), encoding / decoding of the corresponding tile is possible, and when deactivated (tile_coded_flag=0), encoding / decoding is not possible. Here, not encoding can mean not generating encoded data for that tile (in this example, it is assumed that the area is processed according to a predetermined rule, etc. Applicable to meaningless areas in a partial projection format of a 360-degree image). Not decoding means no longer parsing the decoded data for that tile (in this example, it is assumed that the area is processed according to a predetermined rule, etc.). Furthermore, no longer parsing the decoded data can mean that there is no encoded data for the unit, so parsing is no longer performed, but it can also mean that even if encoded data exists, parsing is no longer performed due to the aforementioned flag. Depending on whether tile encoding / decoding is performed, tile-specific header information can be provided.

[0304] The above examples primarily focus on tiles, but they are not limited to tiles and can be applied to other division units in this invention. Furthermore, the above examples of tile division settings are not limited to this case and can be modified to other examples.

[0305] [Table 2]

[0306] Table 2 shows examples of syntactic elements related to reconstruction during image setup.

[0307] Referring to Table 2, `convert_enabled_flag` is a syntactic element indicating whether or not to perform reconstruction. When enabled (convert_enabled_flag=1), it means that the reconstructed image will be encoded / decoded, and additional reconstruction-related information can be viewed. When deactivated (convert_enabled_flag=0), it means that the existing image will be encoded / decoded.

[0308] `convert_type_flag` represents mixed information regarding the method and mode of reconstruction. It can be determined from a group of several candidates, including methods that apply rotation, methods that apply inversion, and methods that apply a mixture of rotation and inversion.

[0309] [Table 3]

[0310] Table 3 shows examples of syntax elements related to size adjustment during image settings.

[0311] Referring to Table 3, pic_width_in_samples and pic_height_in_samples are syntactic elements related to the width and height of an image, and the size of the image can be checked using these syntactic elements.

[0312] `img_resizing_enabled_flag` is a syntactic element that indicates whether to resize an image. When enabled (`img_resizing_enabled_flag=1`), it means that the resized image will be encoded / decoded, and additional resizing-related information can be viewed. When disabled (`img_resizing_enabled_flag=0`), it means that the existing image will be encoded / decoded. It can also be a syntactic element that indicates resizing for in-screen prediction.

[0313] `resizing_met_flag` is a syntactic element that indicates the resizing method. It can be set to one of the following options: resizing using a scale factor (resizing_met_flag=0), resizing using an offset factor (resizing_met_flag=1), or other resizing methods.

[0314] `resizing_mov_flag` is a syntactic element that specifies the resizing behavior. For example, it can be set to either expand or shrink.

[0315] width_scale and height_scale refer to the scale factors related to horizontal and vertical size adjustments, which are part of the size adjustments using scale factors.

[0316] top_height_offset and bottom_height_offset refer to the upper and lower offset factors related to horizontal size adjustment using the offset factor, while left_width_offset and right_width_offset refer to the left and right offset factors related to vertical size adjustment using the offset factor.

[0317] The size of the image after resizing can be updated via the aforementioned size adjustment-related information and image size information.

[0318] `resizing_type_flag` is a syntactic element that specifies how the data to be resized will be handled. Depending on the resizing method and operation, the number of candidate data handling methods may be the same or different.

[0319] The image setting process applied to the image encoding / decoding device described above may be performed individually or as a combination of multiple image setting processes. The example described below will explain the case where multiple image setting processes are performed in combination.

[0320] Figure 11 is an illustrative diagram showing images before and after the image setting process according to one embodiment of the present invention. Specifically, 11a is an example before image reconstruction is performed on the divided images (e.g., an image projected by 360-degree image coding), and 11b is an example after image reconstruction is performed on the divided images (e.g., an image packed by 360-degree image coding). In other words, 11a can be understood as an illustrative diagram before the image setting process is performed, and 11b as an illustrative diagram after the image setting process is performed.

[0321] The image setup process in this example explains the cases of image segmentation (assuming tiling in this example) and image reconstruction.

[0322] The following example will describe the case where image reconstruction is performed after image segmentation, but depending on the encoding / decoding settings, it is also possible for image segmentation to be performed after image reconstruction, and variations to other cases are also possible. Furthermore, the image reconstruction process described above (including the reverse process) can be applied in the same or similar way as the reconstruction process of segmented units within the image in this embodiment.

[0323] Image reconstruction may be performed on all division units within the image, or not, or on only some of the division units. Therefore, the division units before reconstruction (e.g., some of P0 to P5) may or may not be the same as the division units after reconstruction (e.g., some of S0 to S5). Various cases concerning the execution of image reconstruction will be explained through the examples described later. Also, for the sake of explanation, it will be assumed that the unit of the image is a picture, the unit of the divided image is a tile, and the division unit is square in shape.

[0324] For example, whether or not to perform image reconstruction can be determined by certain units (e.g., sps_convert_enabled_flag, SEI, or metadata). Alternatively, whether or not to perform image reconstruction can be determined by certain units (e.g., pps_convert_enabled_flag). This is possible if it first occurs in the relevant unit (in this example, the picture) or if it is activated in a higher-level unit (e.g., sps_convert_enabled_flag=1). Alternatively, whether or not to perform image reconstruction can be determined by certain units (e.g., tile_convert_flag[i], where i is the division unit index). This is possible if it first occurs in the relevant unit (in this example, the tile) or if it is activated in a higher-level unit (e.g., pps_convert_enabled_flag=1). Furthermore, whether or not to perform the aforementioned partial image reconstruction can be implicitly determined depending on the encoding / decoding settings, thereby omitting related information.

[0325] For example, depending on a signal that instructs image reconstruction (e.g., pps_convert_enabled_flag), it can be determined whether to reconstruct the division units within the image. More specifically, depending on the signal, it can be determined whether to reconstruct all division units within the image. In this case, a signal instructing the reconstruction of a single image may be generated.

[0326] For example, depending on a signal instructing image reconstruction (e.g., tile_convert_flag[i]), it can be determined whether to reconstruct a segment of the image. More specifically, depending on the signal, it can be determined whether to reconstruct a sub-segment of the image. In this case, a signal instructing reconstruction of at least one image (e.g., generated as many times as there are segmentation units) may be generated.

[0327] For example, depending on a signal that instructs image reconstruction (e.g., pps_convert_enabled_flag), it can be determined whether to perform image reconstruction, and depending on a signal that instructs image reconstruction (e.g., tile_convert_flag[i]), it can be determined whether to perform reconstruction of a segmented unit within the image. More specifically, if some signals are activated (e.g., pps_convert_enabled_flag=1), some other signals (e.g., tile_convert_flag[i]) can be checked, and depending on the aforementioned signal (tile_convert_flag[i] in this example), it can be determined whether to perform reconstruction of a segmented unit within the image. At this time, signals instructing the reconstruction of multiple images can be generated.

[0328] When a signal that instructs image reconstruction is activated, image reconstruction-related information can be generated. Various cases of image reconstruction-related information will be explained in the examples described later.

[0329] For example, reconstruction information can be generated that is applied to an image. More specifically, one piece of reconstruction information can be used as the reconstruction information for all division units within the image.

[0330] As an example, reconstruction information can be generated that is applied to the division units within an image. More specifically, at least one piece of reconstruction information can be used as reconstruction information for a subset of the division units within the image. That is, one piece of reconstruction information can be used as reconstruction information for one division unit, or one piece of reconstruction information can be used as reconstruction information for multiple division units.

[0331] The examples described later can be explained by combining them with examples of image reconstruction.

[0332] For example, when a signal instructing image reconstruction (e.g., pps_convert_enabled_flag) is activated, reconstruction information can be generated that is applied commonly to all division units within the image. Alternatively, when a signal instructing image reconstruction (e.g., pps_convert_enabled_flag) is activated, reconstruction information can be generated that is applied individually to all division units within the image. Alternatively, when a signal instructing image reconstruction (e.g., tile_convert_flag[i]) is activated, reconstruction information can be generated that is applied individually to all division units within the image. Alternatively, when a signal instructing image reconstruction (e.g., tile_convert_flag[i]) is activated, reconstruction information can be generated that is applied commonly to all division units within the image.

[0333] In the case of the aforementioned reconstruction information, implicit or explicit processing can be performed depending on the encoding / decoding settings. In the implicit case, the reset information can be assigned to predetermined values ​​according to the characteristics and type of the image.

[0334] P0 to P5 of 11a can correspond to S0 to S5 of 11b, and a reconstruction process can be performed on each division unit. For example, P0 can be assigned to SO without reconstruction, P1 can be assigned to S1 after a 90-degree rotation, P2 can be assigned to S2 after a 180-degree rotation, P3 can be assigned to S3 after a left-right inversion, P4 can be assigned to S4 after a left-right inversion following a 90-degree rotation, and P5 can be assigned to S5 after a left-right inversion following a 180-degree rotation.

[0335] However, examples of transformations are not limited to those mentioned above, and various other examples are possible. As in the example above, reconstruction may not be performed on the image division units, or at least one of the following reconstruction methods may be used: reconstruction with rotation, reconstruction with inversion, or reconstruction with a combination of rotation and inversion.

[0336] When image reconstruction is applied to division units, additional reconstruction processes such as division unit rearrangement may be performed. That is, the image reconstruction process of the present invention can consist of rearranging pixels within the image, as well as rearranging pixels within the division units, and can be represented by some syntactic elements (e.g., part_top, part_left, part_width, part_height, etc.) as shown in Table 4. This means that the image division and image reconstruction processes can be understood as a mixture. The above description may be an example of what is possible when an image is divided into multiple units.

[0337] P0 to P5 in 11a can correspond to S0 to S5 in 11b, and a reconstruction process can be performed on each division unit. For example, P0 can be assigned to S0 without reconstruction, P1 can be assigned to S2 without reconstruction, P2 can be assigned to S1 after a 90-degree rotation, P3 can be assigned to S4 after a left-right flip, P4 can be assigned to S5 after a left-right flip following a 90-degree rotation, and P5 can be assigned to S3 after a 180-degree rotation following a left-right flip. However, this is not limited to these examples, and various other transformations are possible.

[0338] Furthermore, P_Width and P_Height in Figure 7 can correspond to P_Width and P_Height in Figure 11, and P'_Width and P'_Height in Figure 7 can correspond to P'_Width and P'_Height in Figure 11. The size of the resized image in Figure 7 is P'_Width × P'_Height, which can be expressed as (P_Width + Exp_L + Exp_R) × (P_Height + Exp_T + Exp_B), and the size of the resized image in Figure 11 is P'_Width × P'_Height, which can be expressed as (P_Width + Var0_L + Var1_L + Var2_L + Var0_R + Var1_R + Var2_R) × (P It can be expressed as (_Height+Var0_T+Var1_T+Var0_B+Var1_B) or (Sub_P0_Width+Sub_P1_Width+Sub_P2_Width+Var0_L+Var1_L+Var2_L+Var0_R+Var1_R+Var2_R)×(Sub_P0_Height+Sub_P1_Height+Var0_T+Var1_T+Var0_B+Var1_B).

[0339] As shown in the example above, image reconstruction can involve rearranging pixels within the division units of an image, rearranging division units within an image, and rearranging division units within an image, not just within the division units of an image. In this case, it is possible to perform image-internal rearrangement of division units after performing pixel rearrangement of division units, or to perform pixel rearrangement of division units after performing image-internal rearrangement of division units.

[0340] The rearrangement of image segments can be determined based on a signal instructing image reconstruction. Alternatively, a signal for rearranging image segments can be generated. More specifically, this signal can be generated when the signal instructing image reconstruction is activated. Alternatively, implicit or explicit processing can be performed depending on the encoding / decoding settings. In the implicit case, it can be determined based on the characteristics and type of the image.

[0341] Furthermore, information regarding the rearrangement of division units within an image can be processed implicitly or explicitly depending on the encoding / decoding settings, and can be determined according to the characteristics and type of the image. In other words, each division unit can be arranged according to predetermined division unit arrangement information.

[0342] Next, an example of reconstructing the division units within an image using an encoding / decoding device according to one embodiment of the present invention is shown.

[0343] Before encoding begins, the input image can be segmented using segmentation information. Reconstruction can be performed on each segment using reconstruction information, and the reconstructed image can be encoded. After encoding is complete, the image can be stored in memory, and the encoded image data can be recorded as a bitstream and transmitted.

[0344] Before starting decoding, the segmentation process can be performed using segmentation information. Reconstruction can be performed on each segment using reconstruction information, and the image decoding data can be parsed and decoded on each reconstructed segment. After decoding is complete, the data can be saved to memory, and after performing the reverse reconstruction process on each segment, the segments can be merged into a single image for output.

[0345] Figure 12 is an example diagram illustrating size adjustment for each division unit within an image according to one embodiment of the present invention. P0 to P5 in Figure 12 correspond to P0 to P5 in Figure 11, and S0 to S5 in Figure 12 correspond to S0 to S5 in Figure 11.

[0346] In the examples described later, the focus will be on cases where image size adjustment is performed after image division, but it is also possible to perform image division after image size adjustment depending on the encoding / decoding settings, and variations to other cases are also possible. Furthermore, the image size adjustment process described above (including the reverse process) can be applied in the same or similar way as the size adjustment process for division units within the image in this embodiment.

[0347] For example, TL~BR in Figure 7 can correspond to TL~BR of the division unit SX (S0~S5) in Figure 12, S0 and S1 in Figure 7 can correspond to PX and SX in Figure 12, P_Width and P_Height in Figure 7 can correspond to Sub_PX_Width and Sub_PX_Height in Figure 12, P'_Width and P'_Height in Figure 7 can correspond to Sub_SX_Width and Sub_SX_Height in Figure 12, Exp_L, Exp_R, Exp_T, and Exp_B in Figure 7 can correspond to VarX_L, VarX_R, VarX_T, and VarX_B in Figure 12, and other factors can also be represented.

[0348] The image resizing process for the division units in images 12a to 12f can be distinguished from the image resizing in Figures 7a and 7b in that settings related to image resizing enlargement or reduction may exist in proportion to the number of division units. Furthermore, there may be differences in whether there are settings that apply commonly to all division units within the image, or settings that apply individually to each division unit. The following examples will explain various cases of resizing, and the resizing process can be performed considering the above points.

[0349] Image resizing in this invention may or may not be performed on all division units within an image, or it may be performed on only some division units. Various cases of image resizing will be explained through the examples described later. For the sake of explanation, it will be assumed that the resizing operation is expansion, the method of resizing is the offset factor, the resizing direction is up, down, left, and right, the resizing direction is set to operate according to the resizing information, the image unit is a picture, and the division image unit is a tile.

[0350] For example, whether or not to resize an image can be determined by certain units (e.g., sps_img_resizing_enabled_flag, or SEI or metadata). Alternatively, whether or not to resize an image can be determined by certain units (e.g., pps_img_resizing_enabled_flag). This is possible if it first occurs in the relevant unit (in this example, the picture) or if it is activated in a higher-level unit (e.g., sps_img_resizing_enabled_flag=1). Alternatively, whether or not to resize an image can be determined by certain units (e.g., tile_resizing_flag[i], where i is the division unit index). This is possible if it first occurs in the relevant unit (in this example, the tile) or if it is activated in a higher-level unit. Furthermore, whether or not to resize certain images can be implicitly determined depending on the encoding / decoding settings, thereby omitting related information.

[0351] For example, depending on a signal instructing image resizing (e.g., pps_img_resizing_enabled_flag), it can be determined whether to resize individual segments within an image. More specifically, depending on the aforementioned signal, it can be determined whether to resize all segments within the image. In this case, a signal instructing resizing a single image may be generated.

[0352] For example, depending on a signal instructing image resizing (e.g., tile_resizing_flag[i]), it can be determined whether to resize the subdivisions within the image. More specifically, depending on the signal, it can be determined whether to resize a subdivision within the image. In this case, a signal instructing resizing at least one image (e.g., generated as many times as there are subdivisions) may be generated.

[0353] For example, depending on a signal that instructs image resizing (e.g., pps_img_resizing_enabled_flag), it can be determined whether to resize the image, and depending on a signal that instructs image resizing (e.g., tile_resizing_flag[i]), it can be determined whether to resize the subdivisions within the image. More specifically, if some signals are activated (e.g., pps_img_resizing_enabled_flag=1), some other signals (e.g., tile_resizing_flag[i]) can be checked, and depending on the aforementioned signal (in this example, tile_resizing_flag[i]), it can be determined whether to resize some subdivisions within the image. At this time, signals instructing the resizing of multiple images can be generated.

[0354] When a signal instructing image resizing is activated, image resizing-related information can be generated. Various cases involving image resizing-related information will be explained in the examples below.

[0355] As an example, resizing information can be generated that is applied to an image. More specifically, one piece of resizing information or a set of resizing information can be used as the resizing information for all divisions within an image. For example, one piece of resizing information can be generated that is applied commonly to the up, down, left, and right directions of divisions within an image (or a resizing value that is applied to all resizing directions supported or allowed by the division; in this example, one piece of information), or one set of resizing information can be generated that is applied to each of the up, down, left, and right directions (or as many as the number of resizing directions supported or allowed by the division; in this example, up to four pieces of information).

[0356] As an example, size adjustment information can be generated that is applied to division units within an image. More specifically, at least one piece of size adjustment information or a set of size adjustment information can be used as size adjustment information for a partial division unit within an image. That is, one piece of size adjustment information or a set of size adjustment information can be used as size adjustment information for one division unit, or it can be used as size adjustment information for multiple division units. For example, one piece of size adjustment information or a set of size adjustment information can be generated that is applied commonly to the up, down, left, and right directions of one division unit within an image, or one set of size adjustment information can be generated that is applied to each of the up, down, left, and right directions. Alternatively, one piece of size adjustment information or a set of size adjustment information can be generated that is applied commonly to the up, down, left, and right directions of multiple division units within an image, or one set of size adjustment information can be generated that is applied to each of the up, down, left, and right directions. The configuration of a size adjustment set means size adjustment value information for at least one size adjustment direction.

[0357] In summary, size adjustment information can be generated that is applied commonly to all division units within an image. Alternatively, size adjustment information can be generated that is applied individually to each division unit within an image. The examples described later can be explained by combining them with examples of image size adjustment.

[0358] For example, when a signal instructing image resizing (e.g., pps_img_resizing_enabled_flag) is activated, resizing information can be generated that is applied commonly to all division units within the image. Alternatively, when a signal instructing image resizing (e.g., pps_img_resizing_enabled_flag) is activated, resizing information can be generated that is applied individually to all division units within the image. Alternatively, when a signal instructing image resizing (e.g., tile_resizing_flag[i]) is activated, resizing information can be generated that is applied individually to all division units within the image. Alternatively, when a signal instructing image resizing (e.g., tile_resizing_flag[i]) is activated, resizing information can be generated that is applied commonly to all division units within the image.

[0359] Image resizing direction and information can be handled implicitly or explicitly depending on the encoding / decoding settings. In the implicit case, resizing information can be assigned to predetermined values ​​depending on the image characteristics and type.

[0360] As explained above, the size adjustment direction in the size adjustment process of the present invention is at least one of the up, down, left, and right directions, and the size adjustment direction and size adjustment information can be processed explicitly or implicitly. That is, for some directions, the size adjustment value (including 0, i.e., no adjustment) is implicitly predetermined, and for some directions, the size adjustment value (including 0, i.e., no adjustment) is explicitly assigned.

[0361] Even within the division units of an image, settings can be implicitly or explicitly configured for size adjustment direction and size adjustment information, and these can be applied to the division units within the image. For example, a setting can be applied to a single division unit within an image (in this example, only the division unit is affected), or a setting can be applied to multiple division units within an image, or a setting can be applied to all division units within an image (in this example, one setting is affected), and at least one setting can be applied to an image (for example, one setting can generate settings equal to the number of division units). A set of settings can be defined by collecting the setting information applied to the division units within the image.

[0362] Figure 13 is an example diagram illustrating the size adjustment or setting of division units within an image.

[0363] In detail, various examples of implicit and explicit processing of the size adjustment direction and size adjustment information for division units within an image are shown. In the examples described later, for the sake of explanation, implicit processing is explained assuming that the size adjustment value for some size adjustment directions is 0.

[0364] As in 13a, if the boundary of the division unit coincides with the boundary of the image (thick solid line in this example), explicit resizing can be performed; if they do not coincide (thin solid line), implicit resizing can be performed. For example, P0 can be resized upwards and to the left (a2, a0), P1 upwards (a2), P2 upwards and to the right (a2, a1), P3 downwards and to the left (a3, a0), P4 downwards (a3), and P5 downwards and to the right (a3, a1), but resizing is not possible in other directions.

[0365] As shown in 13b, some directions of the division unit (up and down in this example) can be explicitly processed for size adjustment, and some directions of the division unit (left and right in this example) can be explicitly processed (thick solid line in this example) if the boundary of the division unit coincides with the boundary of the image, and implicitly processed if they do not coincide (thin solid line in this example). For example, P0 can be resized in the up, down, and left directions (b2, b3, b0), P1 in the up and down directions (b2, b3), P2 in the up, down, and right directions (b2, b3, b1), P3 in the up, down, and left directions (b3, b4, b0), P4 in the up and down directions (b3, b4), and P5 in the up, down, and right directions (b3, b4, b1), while size adjustment is not possible in other directions.

[0366] As shown in 13c, some directions of the division unit (left and right in this example) can be explicitly resized, while some directions of the division unit (up and down in this example) can be explicitly resized if the boundary of the division unit coincides with the boundary of the image (thick solid line in this example), and implicitly resized if they do not coincide (thin solid line in this example). For example, P0 can be resized in the up, left, and right directions (c4, c0, c1), P1 in the up, left, and right directions (c4, c1, c2), P2 in the up, left, and right directions (c4, c2, c3), P3 in the down, left, and right directions (c5, c0, c1), P4 in the down, left, and right directions (c5, c1, c2), and P5 in the down, left, and right directions (c5, c2, c3), while other directions cannot be resized.

[0367] As shown in the example above, settings related to image resizing can take various forms. Multiple sets of settings may be supported, allowing for explicit selection of a setting set, or a predetermined set of settings may be implicitly determined depending on the encoding / decoding settings (e.g., image characteristics, type, etc.).

[0368] Figure 14 is an illustrative diagram that shows both the image size adjustment process and the size adjustment process for the division units within the image together.

[0369] Referring to Figure 14, the image resizing process and its reverse process can proceed in directions e and f, while the resizing process and its reverse process for individual divisions within the image can proceed in directions d and g. In other words, the image can be resized, and the individual divisions within the image can be resized, and the order of the resizing processes is not fixed. This means that multiple resizing processes are possible.

[0370] In summary, the image resizing process can be classified into image resizing (or resizing the image before division) and resizing of the division units within the image (or resizing the images after division). It is not necessary to perform both image resizing and division unit resizing, one or the other may be performed, or both may be performed. This can be determined depending on the encoding / decoding settings (e.g., image characteristics, type, etc.).

[0371] In the example above, when performing multiple size adjustments, the image size can be adjusted in at least one direction from the top, bottom, left, and right directions, and the size can be adjusted in at least one of the division units within the image. In this case, the size adjustment can be performed in at least one direction from the top, bottom, left, and right directions of the division unit being adjusted.

[0372] Referring to Figure 14, the size of image A before resizing can be defined as P_Width × P_Height, the size of image after primary resizing (or image before secondary resizing, B) is P'_Width × P'_Height, and the size of image after secondary resizing (or final resizing, C) is P''_Width × P''_Height. Image A before resizing means an image that has not undergone any resizing, image B after primary resizing means an image that has undergone some resizing, and image C after secondary resizing means an image that has undergone all resizing. For example, image B after primary resizing can mean an image in which the size of the division units within the image has been adjusted, as shown in Figures 13a to 13c, and image C after secondary resizing can mean an image in which the entire primary resized image B has been resized, as shown in Figure 7a, and the reverse is also possible. Examples of various transformations are possible, not limited to the above examples.

[0373] In the image B after primary resizing, P'_Width can be obtained via P_Width and at least one resizing value in the left or right direction that allows for horizontal resizing, and P'_Height can be obtained via P_Height and at least one resizing value in the up or down direction that allows for vertical resizing. In this case, the resizing values ​​may be resizing values ​​that occur in units of division.

[0374] In the image C after secondary resizing, P''_Width can be obtained via P'_Width and at least one resizing value in the left or right direction that allows for horizontal resizing, and P''_Height can be obtained via P'_Height and at least one resizing value in the up or down direction that allows for vertical resizing. In this case, the resizing values ​​may be resizing values ​​generated from the image.

[0375] In summary, the size of an image after resizing can be obtained through the size of the image before resizing and at least one resizing value.

[0376] Information regarding data processing methods can be generated in the area of ​​the image being resized. Various data processing methods will be explained through examples described later, and the same or similar applications are possible for data processing methods generated in the reverse resizing process as in the resizing process, and the data processing methods in the resizing process and the reverse resizing process can be explained through various combinations of the cases described later.

[0377] As an example, a data processing method can be generated that applies to an image. More specifically, one data processing method or set of data processing methods can be used as a data processing method for all division units in an image (assuming in this example all division units are resized). For example, one data processing method can be generated that applies commonly to the up, down, left, and right directions of division units in an image (or a data processing method that applies to all resizing directions supported or allowed by the division unit; one piece of information in this example) or one set of data processing methods that applies to each of the up, down, left, and right directions (or as many as the number of resizing directions supported or allowed by the division unit; up to four pieces of information in this example).

[0378] As an example, a data processing method can be generated that is applied to the division units within an image. More specifically, at least one data processing method or set of data processing methods can be used as a data processing method for some of the division units within an image (assuming in this example, the division units to be resized). That is, one data processing method or set of data processing methods can be used as a data processing method for one division unit, or it can be used as a data processing method for multiple division units. For example, one data processing method can be generated that is applied commonly to the up, down, left, and right directions of one division unit within an image, or a set of data processing methods can be generated that are applied to each of the up, down, left, and right directions. Alternatively, one data processing method can be generated that is applied commonly to the up, down, left, and right directions of multiple division units within an image, or a set of data processing method information can be generated that is applied to each of the up, down, left, and right directions. The configuration of the data processing method set means a data processing method for at least one resizing direction.

[0379] In summary, a data processing method can be used that is applied commonly to all division units within an image. Alternatively, a data processing method can be used that is applied individually to each division unit within an image. The data processing method can be a predetermined method. There can be at least one predetermined data processing method. This falls under the implicit case, and explicit selection information regarding the data processing method can arise. This can be determined according to the encoding / decoding settings (e.g., image characteristics, type, etc.).

[0380] In other words, a data processing method can be used that is commonly applied to all division units within an image, and either a predetermined method can be used, or one of several data processing methods can be selected. Alternatively, a data processing method can be used that is individually applied to each division unit within an image, and either a predetermined method can be used, or one of several data processing methods can be selected, depending on the division unit.

[0381] The following example illustrates some cases related to resizing the size of division units within an image (assuming expansion in this example) (in this example, the resizing area is filled using some of the image data).

[0382] A portion of a certain unit (for example, S0 to S5 in Figures 12a to 12f) can have its size adjusted using data from a portion of another unit (P0 to P5 in Figures 12a to 12f) specifically from a portion of that unit,

[0383] For example, the region TL~BR currently being resized in units of division can be resized using the tl~br data of the current division unit. For instance, S0's TL can be filled using P0's tl data, S1's RC can be filled using P1's tr+rc+br data, S2's BL+BC can be filled using P2's bl+bc+br data, and S3's TL+LC+BL can be filled using P3's tl+lc+bl data.

[0384] As an example, the region TL~BR currently being resized by a division unit can be resized using the tl~br data of a division unit spatially adjacent to the current division unit. For example, TL+TC+TR of S4 can be filled using the b1+bc+br data of P1 in the upward direction, BL+BC of S2 can be filled using the tl+tc+tr data of P5 in the downward direction, LC+BL of S2 can be filled using the tl+rc+bl data of P1 in the left direction, RC of S3 can be filled using the tl+lc+bl data of P4 in the right direction, and BR of S0 can be filled using the tl data of P4 in the lower left direction.

[0385] For example, the region TL~BR currently being resized for a division unit can be resized using tl~br data from a division unit that is not spatially adjacent to the current division unit. For instance, data can be obtained from the edges of an image (e.g., left / right, top / bottom, etc.). LC in S3 can be obtained using tr+rc+br data from S5, RC in S2 using tl+lc data from S0, BC in S4 using tc+tr data from S1, and TC in S1 using bc data from S4.

[0386] Alternatively, data from a specific region of the image (a region that is not spatially adjacent but is judged to have a high correlation with the region to be resized) can be obtained. S1's BC can be obtained using S3's tl+lc+bl data, S3's RC can be obtained using S1's tl+tc data, and S5's RC can be obtained using S0's bc data.

[0387] Furthermore, some cases regarding size adjustment of division units within an image (assuming reduction in this example) (removed by restoring or correcting using some image data in this example) are as follows:

[0388] A portion of a unit (for example, S0 to S5 in Figures 12a to 12f) from region TL to BR can be used in the restoration or correction process of a portion of a unit P0 to P5 from region tl to br. In this case, the portion of the unit may be the same (e.g., S0 and P0) or different (e.g., S0 and P2). That is, the resized region can be used and removed to restore some data of the corresponding division unit, and the resized region can be used and removed to restore some data of a division unit different from the corresponding division unit. Detailed examples can be derived in reverse from the expansion process, so they are omitted.

[0389] The above example applies when there is data that is highly correlated with the area being resized. Information about the location referenced for resizing can be explicitly generated, implicitly obtained based on predetermined rules, or a combination of these to confirm the relevant information. This could be an example that applies when obtaining data from other areas where continuity exists in the encoding of a 360-degree image.

[0390] Next, an example of adjusting the size of the division units within an image using an encoding / decoding device according to one embodiment of the present invention is shown.

[0391] The input image can be segmented before encoding begins. Size adjustment information can be used for each segment, allowing for the encoding of the image after size adjustment for each segment. After encoding is complete, the image can be stored in memory, and the encoded image data can be recorded as a bitstream for transmission.

[0392] Before starting decoding, the segmentation process can be performed using segmentation information. A size adjustment process can be performed on each segment using size adjustment information, and the image decoding data can be parsed and decoded on the segmented units after size adjustment. After decoding is complete, the data can be saved to memory, and after performing the reverse size adjustment process on each segment, the segments can be merged into a single image for output.

[0393] In other cases during the image resizing process described above, the changes can be applied as shown in the example above, and are not limited to this; changes can also be made to other examples.

[0394] The image setup process described above allows for a combination of image resizing and image reconstruction. Image reconstruction can be performed after image resizing, or image resizing can be performed after image reconstruction. Furthermore, combinations of image splitting, image reconstruction, and image resizing are possible. Image resizing and image reconstruction can be performed after image splitting, and the order of image setup is not fixed but can be changed, which can be determined according to the encoding / decoding settings. In this example, the image setup process is described as one in which image reconstruction and image resizing are performed after image splitting, but other orders are possible and can be changed according to the encoding / decoding settings.

[0395] For example, the process may be performed in the following order: splitting → reconstruction, reconstruction → splitting, splitting → resizing, resizing → splitting, resizing → reconstruction, reconstruction → resizing, splitting → reconstruction → resizing, splitting → resizing → reconstruction, resizing → splitting → reconstruction, resizing → reconstruction → splitting, reconstruction → splitting → resizing, reconstruction → resizing → splitting, etc., and can also be combined with additional image settings. As mentioned above, the image setting process may be performed sequentially, but all or some of the setting processes may be performed simultaneously. In addition, some image setting processes may consist of multiple steps depending on the encoding / decoding settings (e.g., image characteristics, type, etc.). Next, examples of various combinations of image setting processes are shown.

[0396] For example, P0 to P5 in Figure 11a can correspond to S0 to S5 in Figure 11b, and a reconstruction process (in this example, pixel rearrangement) and a size adjustment process (in this example, the same size adjustment for each division) can be performed on each division unit. For example, P0 to P5 can be assigned to S0 to S5 by applying a size adjustment using an offset. Alternatively, P0 can be assigned to S0 without reconstruction, P1 can be assigned to S1 by applying a 90-degree rotation, P2 can be assigned to S2 by applying a 180-degree rotation, P3 can be assigned to S3 by applying a 270-degree rotation, P4 can be assigned to S4 by applying a horizontal flip, and P5 can be assigned to S5 by applying an vertical flip.

[0397] For example, P0 to P5 in Figure 11a can correspond to the same or different positions as S0 to S5 in Figure 11b, and a reconstruction process (rearrangement of pixels and division units in this example) and a size adjustment process (same size adjustment in this example) can be performed on each division unit. For example, P0 to P5 can be assigned to S0 to S5 by applying a size adjustment using a scale. Also, P0 can be assigned to S0 without reconstruction, P1 can be assigned to S2 without reconstruction, P2 can be assigned to S1 by applying a 90-degree rotation, P3 can be assigned to S4 by applying a horizontal flip, P4 can be assigned to S5 by applying a horizontal flip after a 90-degree rotation, and P5 can be assigned to S3 by applying a 180-degree rotation after a horizontal flip.

[0398] For example, P0 to P5 in Figure 11a can correspond to E0 to E5 in Figure 5e, and a reconstruction process (rearrangement of pixels and division units in this example) and a size adjustment process (size adjustment that is not the same for each division unit in this example) can be performed on each division unit. For example, P0 can be assigned to E0 without size adjustment or reconstruction, P1 can be assigned to E1 with size adjustment using scale but without reconstruction, P2 can be assigned to E2 with no size adjustment but with reconstruction, P3 can be assigned to E4 with size adjustment using offset but without reconstruction, P4 can be assigned to E5 with no size adjustment but with reconstruction, and P5 can be assigned to E3 with size adjustment using offset and with reconstruction.

[0399] As in the example above, the absolute or relative positions within the image of the segmented units before and after the image setting process may be maintained or changed. This can be determined according to the encoding / decoding settings (e.g., image characteristics, type, etc.). Furthermore, various combinations of image setting processes are possible, and the example is not limited to the one above; variations into various other examples are also possible.

[0400] The encoder records the information generated during the above process into a bitstream in units of at least one of the following: sequence, picture, slice, or tile. The decoder then parses the related information from the bitstream. This information may also be included in the bitstream in the form of SEI or metadata.

[0401] [Table 4]

[0402] Next, we will show examples of syntactic elements associated with multiple image settings. The examples described below will focus on the additional syntactic elements. Furthermore, the syntactic elements in the examples described below are not limited to a specific unit, but may be syntactic elements supported by various units such as sequences, pictures, slices, and tiles. Alternatively, they may be syntactic elements included in SEI or metadata.

[0403] Referring to Table 4, `parts_enabled_flag` is a syntactic element indicating whether or not to divide into partial units. When enabled (parts_enabled_flag=1), it means that the image will be divided into multiple units for encoding / decoding, and additional division information can be viewed. When deactivated (parts_enabled_flag=0), it means that the existing image will be encoded / decoded. This example focuses on rectangular division units such as tiles, and it is possible to have different settings for existing tiles and division information.

[0404] `num_partitions` is a syntactic element for the number of partition units, and a value obtained by adding 1 represents the number of partition units.

[0405] `part_top[i]` and `part_left[i]` represent syntactic elements for positional information of the division unit, meaning the horizontal and vertical starting positions of the division unit (for example, the position of the upper left corner of the division unit). `part_width[i]` and `part_height[i]` represent syntactic elements for size information of the division unit, meaning the width and height of the division unit. In this case, the starting position and size information can be set in pixel units or block units. Furthermore, these syntactic elements may be syntactic elements that can occur during the image reconstruction process, or syntactic elements that can occur when the image division process and the image reconstruction process are combined.

[0406] `part_header_enabled_flag` is a syntactic element that indicates whether or not to support encoding / decoding settings at the subdivision level. When enabled (part_header_enabled_flag=1), subdivisions can have encoding / decoding settings, while when disabled (part_header_enabled_flag=0), they cannot have encoding / decoding settings and can instead receive encoding / decoding settings from higher-level units.

[0407] The above example is one example of syntactic elements associated with resizing and reconstruction at the division unit level in the image settings described later, but is not limited to this, and other division units and settings of the present invention can be modified and applied. This example is explained under the assumption that resizing and reconstruction are performed after division, but is not limited to this, and can be modified and applied by other image setting orders, etc. Furthermore, the types of syntactic elements supported in the example described later, the order of syntactic elements, conditions, etc. are limited only to this example, and can be changed and determined according to the encoding / decoding settings.

[0408] [Table 5]

[0409] Table 5 shows examples of syntactic elements related to the reconstruction of division units in image settings.

[0410] Referring to Table 5, part_convert_flag[i] represents a syntactic element indicating whether or not a division unit is being reconstructed. This syntactic element can occur for each division unit, and if activated (part_convert_flag[i]=1), it means that the reconstructed division unit will be encoded / decoded, and additional reconstruction-related information can be viewed. If deactivated (part_convert_flag[i]=0), it means that an existing division unit will be encoded / decoded. convert_type_flag[i] represents mode information regarding the reconstruction of the division unit and may be information regarding pixel rearrangement.

[0411] Furthermore, syntactic elements may be generated for additional reconstructions, such as rearranging the division units. In this example, rearranging the division units can be performed via the syntactic elements part_top and part_left, which are related to image division as described above, or syntactic elements related to rearranging the division units (e.g., index information) may be generated.

[0412] [Table 6]

[0413] Table 6 shows examples of syntactic elements related to adjusting the size of the division unit in image settings.

[0414] Referring to Table 6, part_resizing_flag[i] is a syntactic element indicating whether to perform image resizing on a division unit. This syntactic element can occur for each division unit. When activated (part_resizing_flag[i]=1), it means that the resized division unit will be encoded / decoded, and additional size-related information can be viewed. When deactivated (part_resizing_flag[i]=0), it means that the existing division unit will be encoded / decoded.

[0415] width_scale[i] and height_scale[i] represent the scale factors for horizontal and vertical size adjustments when adjusting the size using scale factors in units of division.

[0416] top_height_offset[i] and bottom_height_offset[i] represent the upward and downward offset factors related to size adjustment using the offset factor per division unit, while left_width_offset[i] and right_width_offset[i] represent the leftward and rightward offset factors related to size adjustment using the offset factor per division unit.

[0417] `resizing_type_flag[i][j]` represents a syntactic element for the data processing method of a region that is resized in units of division. The syntactic element represents a specific data processing method in the direction of resizing. For example, syntactic elements can be generated for specific data processing methods of regions that are resized in the up, down, left, and right directions. This can also be generated based on resizing information (e.g., only possible when resizing in certain directions).

[0418] The image setup process described above may be applied depending on the characteristics and type of the image. In the examples described later, the image setup process described above can be applied similarly, or modified, even without special mention. The examples described later will focus on cases where the above examples are applied additionally or with modifications.

[0419] For example, images generated via a 360-degree camera (360-degree video or omnidirectional video) have different characteristics from images acquired via a general camera and require a different encoding environment than general image compression.

[0420] Unlike conventional images, 360-degree images do not have discontinuous boundaries, and data in all regions can be continuous. Furthermore, devices such as HMDs reproduce images in front of the eyes through lenses, requiring high-resolution images, and when images are acquired via a stereoscopic camera, the amount of image data processed can increase. Various image setup processes considering 360-degree images can be employed to provide an efficient encoding environment, including the examples above.

[0421] The 360-degree camera is a camera having multiple cameras or multiple lenses and sensors, and the cameras or lenses can handle all directions around any central point that the camera captures.

[0422] 360-degree images can be encoded using various methods. For example, they can be encoded using various image processing algorithms in three-dimensional space, or they can be converted to two-dimensional space and encoded using various image processing algorithms. This invention primarily describes a method for encoding / decoding 360-degree images by converting them to two-dimensional space.

[0423] A 360-degree image encoding device according to one embodiment of the present invention may be configured to include all or part of the configuration shown in Figure 1, and may further include a pre-processing unit that performs pre-processing (stitching, projection, region-wise packing) on ​​the input image. On the other hand, a 360-degree image decoding device according to one embodiment of the present invention may include all or part of the configuration shown in Figure 2, and may further include a post-processing unit that performs post-processing (rendering) before the image is decoded and reproduced as an output image.

[0424] To reiterate, the encoder performs pre-processing on the input image, then encodes it and transmits the resulting bitstream. The decoder then parses the transmitted bitstream to decode it, and after post-processing, generates the output image. In this process, the bitstream contains and transmits information generated during both the pre-processing and encoding processes, which the decoder parses and uses in both the decoding and post-processing processes.

[0425] Next, we will explain in more detail how the 360-degree image encoder works. Since the operation of the 360-degree image decoder is the inverse operation of the 360-degree image encoder, it can be easily derived by an average engineer, so a detailed explanation will be omitted.

[0426] The input image can undergo stitching and projection processes to create a 3D projection structure on a sphere basis. Through these processes, the image data on the 3D projection structure can be projected onto a 2D image.

[0427] The projected image can consist of all or part of the 360-degree content, depending on the encoding settings. In this case, the positional information of the region (or pixel) located in the center of the projected image can be implicitly generated as a predetermined value, or the positional information can be explicitly generated. Furthermore, when the projected image consists of a portion of the 360-degree content, the extent and positional information of the included region can be generated. Additionally, extent information (e.g., height, width) and positional information (e.g., measured relative to the upper left of the image) for the Region of Interest (ROI) in the projected image can be generated. In this case, a portion of the 360-degree content with high importance can be set as the Region of Interest. While a 360-degree image allows viewing of all content in the up, down, left, and right directions, the user's gaze may be limited to a portion of the image, and this can be taken into consideration when setting the Region of Interest. For efficient encoding, the Region of Interest can be set to have high quality and resolution, while other areas can be set to have lower quality and resolution than the Region of Interest.

[0428] Among 360-degree image transmission methods, the Single Stream method can transmit the entire image or viewport image to the user using a single, individual bitstream. The Multi Stream method transmits multiple entire images with different image qualities using multiple bitstreams, allowing the user to select the image quality according to their environment and communication conditions. The Tiled Stream method transmits individually encoded tile-level partial images using multiple bitstreams, allowing the user to select the tile according to their environment and communication conditions. Therefore, a 360-degree image encoder can generate and transmit bitstreams with two or more qualities, and a 360-degree image decoder can set a region of interest according to the user's gaze and selectively decode according to that region of interest. In other words, the region of interest can be set to where the user's gaze lingers via a head tracking or eye tracking system, and only the necessary parts can be rendered.

[0429] The projected image can undergo a region-wise packing process to be converted into a packed image. The region-wise packing process may include a step of dividing the projected image into multiple regions. Each divided region can then be placed (or rearranged) within the packed image according to the region-wise packing settings. Region-wise packing may be performed to enhance spatial continuity when converting a 360-degree image to a 2D image (or projected image). Image size can be reduced through region-wise packing. It can also reduce image quality degradation during rendering, enable viewport-based projection, and provide other types of projection formats. Region-wise packing may or may not be performed depending on the encoding settings, and can be determined based on a signal indicating whether to perform it (e.g., regionwise_packing_flag; in the example below, region-wise_packing-related information is only available when regionwise_packing_flag is activated).

[0430] When regional packing is performed, configuration information (or mapping information) can be displayed (or generated) indicating that a portion of the projected image is assigned (or placed) as a portion of the packed image. When regional packing is not performed, the projected image and the packed image may be the same image.

[0431] In the above, stitching, projection, and regional packing processes were defined as separate processes, but a part of these processes (e.g., stitching + projection, projection + regional packing) or all of them (e.g., stitching + projection + regional packing) can be defined as a single process.

[0432] Depending on the settings for the stitching, projection, and regional packing processes, at least one packed image can be generated for the same input image. Furthermore, depending on the settings for the regional packing process, at least one encoded data can be generated for the same projected image.

[0433] A tiling process can be performed to divide a packed image. In this case, tiling is a process of dividing and transmitting an image into multiple regions, and can be an example of the 360-degree image transmission method. As mentioned above, tiling can be performed for the purpose of partial decoding, taking into account the user's environment, and for the purpose of efficiently processing the vast amount of data in a 360-degree image. For example, if an image consists of a single unit, the entire image can be decoded for decoding of the region of interest, but if the image consists of multiple unit regions, it is efficient to decode only the region of interest. In this case, the division can be performed by dividing it into tiles, which are division units according to existing encoding schemes, or by dividing it into various division units (square divisions, blocks, etc.) as described in the present invention. Furthermore, the division unit may be a unit that performs independent encoding / decoding. Tiling can be performed based on the projected image or the packed image, or independently. That is, it can be divided based on the surface boundary of the projected image, the surface boundary of the packed image, the packing settings, etc., and each division unit can be divided independently. This can affect the generation of partition information during the tiling process.

[0434] Next, the projected or packed image can be encoded. The encoded data, along with information generated during the preprocessing process, can be recorded in a bitstream and transmitted to a 360-degree image decoder. The information generated during the preprocessing process may be recorded in the bitstream in the form of SEI or metadata. In this case, the bitstream may include at least one encoded data and at least one piece of preprocessing information that differs in some settings of the encoding process or some settings of the preprocessing process. This may be for the purpose of the decoder mixing multiple encoded data (encoded data + preprocessing information) according to the user's environment to construct a decoded image. In detail, multiple encoded data can be selectively combined to construct a decoded image. Furthermore, the process may be performed in two separate steps for application in a stereoscope system, or the process may be performed on an additional depth image.

[0435] Figure 15 is an illustrative diagram showing a 3D image in a 3D space and a 2D planar space.

[0436] Generally, 3DoF (Degree of Freedom) is required for a 360-degree three-dimensional virtual space, which can support three rotations around the X (Pitch), Y (Yaw), and Z (Roll) axes. DoF refers to degrees of freedom in space, 3DoF refers to degrees of freedom including rotations around the X, Y, and Z axes as in 15a, and 6DoF refers to degrees of freedom that further allow for movement along the X, Y, and Z axes in addition to 3DoF. The image encoding and decoding devices of the present invention are described primarily in the case of 3DoF, and when supporting 3DoF or more (3DoF+), they may be combined with or modified to include additional processes or devices not shown in the present invention.

[0437] Referring to 15a, Yaw can range from -π (-180 degrees) to π (180 degrees), Pitch can range from -π / 2rad (or -90 degrees) to π / 2rad (or 90 degrees), and Roll can range from -π / 2rad (or -90 degrees) to π / 2rad (or 90 degrees). In this case, assuming that ψ and θ are Longitude and Latitude in the map representation of the Earth, (x, y, z) in 3D space can be transformed from (ψ, θ) in 2D space. For example, coordinates in 3D space can be derived from coordinates in 2D space based on the transformation formulas x=cos(θ)cos(ψ), y=sin(θ), and z=-cos(θ)sin(ψ).

[0438] Furthermore, (ψ, θ) can be converted to (x, y, z). For example, ψ = tan -1 (-Z / X), θ=sin -1 (Y / (X 2 +Y 2 + Z 2 ) 1 / 2 Based on the transformation formula, it is possible to derive 2D spatial coordinates from 3D spatial coordinates.

[0439] If pixels in 3D space are accurately converted to 2D space (e.g., integer unit pixels in 2D space), then pixels in 3D space can be mapped to pixels in 2D space. If pixels in 3D space are not accurately converted to 2D space (e.g., decimal unit pixels in 2D space), then pixels obtained by interpolation can be mapped to 2D pixels. In this case, interpolation methods such as Nearest neighbor interpolation, Bi-linear interpolation, B-spline interpolation, and Bi-cubic interpolation can be used. In this case, one of several interpolation candidates can be selected and related information can be explicitly generated, or the interpolation method can be implicitly determined based on predetermined rules. For example, a predetermined interpolation filter can be used depending on the 3D model, projection format, color format, slice / tile type, etc. Also, when explicitly generating interpolation information, information about the filter (e.g., filter coefficients) may be included.

[0440] 15b shows an example of transformation from 3D space to 2D space (2D plane coordinate system). (ψ, θ) can be sampled (i, j) based on the image size (width, height), where i can range from 0 to P_Width-1 and j can range from 0 to P_Height-1.

[0441] (ψ, θ) is the center point of the projected image for positioning the 360-degree image {or reference point, the point denoted as C in Figure 15, with coordinates (ψ, θ) = (0, 0)}. The setting for the center point can be specified in three-dimensional space, and the position information for the center point can be explicitly generated or implicitly set to a pre-defined value. For example, center position information for Yaw, center position information for Pitch, center position information for Roll, etc., can be generated. If the values ​​for the aforementioned information are not specifically stated, each value can be assumed to be 0.

[0442] The above example illustrates the conversion of an entire 360-degree image from 3D space to 2D space. However, it is also possible to target only a portion of a 360-degree image. Positional information (e.g., the position of a part of the region; in this example, positional information relative to the center point), range information, etc., can be explicitly generated for this portion of the image, or implicitly followed by already set positional and range information. For example, it is possible to generate central positional information for Yaw, central positional information for Pitch, central positional information for Roll, range information for Yaw, range information for Pitch, and range information for Roll. In the case of a portion of the image, there is at least one region, and this allows for the processing of positional and range information for multiple regions. If the values ​​for the aforementioned information are not explicitly specified, they can be assumed to represent the entire 360-degree image.

[0443] H0 to H6 and W0 to W5 in 15a represent some of the latitude and longitude in 15b, respectively, and the coordinates of 15b can be expressed as (C, j) and (i, C) (where C is the longitude or latitude component). Unlike general images, 360-degree images can be distorted or the content within the image may warp when converted to a two-dimensional space. This varies depending on the area of ​​the image, and the encoding / decoding settings can be set differently for the position of the image or for areas demarcated according to the position. When the encoding / decoding settings are adaptively set based on the encoding / decoding information in the present invention, the position information (e.g., x, y components, or ranges defined by x and y) may be included as an example of the encoding / decoding information.

[0444] The above descriptions in three-dimensional and two-dimensional space are defined to aid in explaining the embodiments of the present invention and are not limited thereto; modifications to the details or applications in other cases are possible.

[0445] As mentioned above, images acquired by a 360-degree camera can be converted into a 2D space. In this process, a 3D model can be used to map the 360-degree image, and various 3D models such as spheres, cubes, cylinders, pyramids, and polyhedra can be used. When converting the 360-degree image mapped based on the aforementioned model into a 2D space, a projection process can be performed using a projection format based on the aforementioned model.

[0446] Figures 16a to 16d are conceptual diagrams illustrating a projection format according to one embodiment of the present invention.

[0447] Figure 16a shows the ERP (Equi-Rectangular Projection) format, where a 360-degree image is projected onto a two-dimensional plane. Figure 16b shows the CMP CubeMap Projection format, where a 360-degree image is projected onto a cube. Figure 16c shows the OHP (OctaHedron Projection) format, where a 360-degree image is projected onto an octahedron. Figure 16d shows the ISP (IcoSahedral Projection) format, where a 360-degree image is projected onto a polyhedron. However, the projection format is not limited to these, and various other projection formats can be used. The left side of Figures 16a to 16d shows a three-dimensional model, and the right side shows an example of conversion to two-dimensional space through the projection process. Depending on the projection format, there are various sizes and shapes, and each shape can be composed of faces, and surfaces can be represented by circles, triangles, quadrilaterals, etc.

[0448] In this invention, a projection format can be defined by a 3D model, surface settings (e.g., number of surfaces, surface morphology, surface morphological configuration, etc.), and projection process settings. If at least one element of the above definition differs, it can be considered a different projection format. For example, in the case of ERP, it consists of a sphere model (3D model), one surface (number of surfaces), and a square surface (surface pattern), but if some of the settings in the projection process (e.g., mathematical formulas used when converting from 3D space to 2D space; that is, the remaining projection settings are the same, and the element that creates the difference of at least one pixel in the projected image during the projection process) differs, it can be classified into different formats such as ERP1 and EPR2. As another example, in the case of CMP, it consists of a cube model, six surfaces, and a square surface, but if some of the settings in the projection process (e.g., sampling method when converting from 3D space to 2D space) differ, it can be classified into different formats such as CMP1 and CMP2.

[0449] When using multiple projection formats instead of a single already configured projection format, you can explicitly generate projection format identifiers (or projection format information). Projection format identifiers can be structured in various ways.

[0450] For example, multiple projection formats can be identified by assigning index information (e.g., proj_format_flag). For instance, ERP could be assigned number 0, CMP number 1, OHP number 2, ISP number 3, ERP1 number 4, CMP1 number 5, OHP1 number 6, ISP1 number 7, CMP compact number 8, OHP compact number 9, ISP compact number 10, and other formats number 11 or higher.

[0451] As an example, a projection format can be identified from at least one element of information that constitutes the projection format. In this case, the element of information that constitutes the projection format may include 3D model information (e.g., 3d_model_flag; 0 is a sphere, 1 is a cube, 2 is a cylinder, 3 is a pyramid, 4 is polyhedron 1, 5 is polyhedron 2, etc.), number of surfaces information (e.g., num_face_flag; starting from 1 and increasing by 1 each time, or assigning the number of surfaces generated in the projection format as index information, where 0 is 1, 1 is 3, 2 is 6, 3 is 8, 4 is 20, etc.), surface shape information (e.g., shape_face_flag; 0 is a quadrilateral, 1 is a circle, 2 is a triangle, 3 is a quadrilateral + circle, 4 is a quadrilateral + triangle, etc.), projection process setting information (e.g., 3d_2d_convert_idx, etc.).

[0452] For example, a projection format can be identified by projection format index information and element information that constitutes the projection format. For instance, projection format index information can be assigned as follows: ERP is 0, CMP is 1, OHP is 2, ISP is 3, and other formats are 4 or higher. Together with the element information that constitutes the projection format (in this example, projection process setting information), projection formats (e.g., ERP, ERP1, CMP, CMP1, OHP, OHP1, ISP, ISP1, etc.) can be identified. Alternatively, projection formats (e.g., ERP, CMP, CMP compact, OHP, OHP compact, ISP, ISP compact, etc.) can be identified together with the element information that constitutes the projection format (in this example, whether or not it is regional packing).

[0453] In summary, a projection format can be identified by projection format index information, by at least one projection format element information, or by projection format index information and at least one projection format element information. This can be defined according to the encoding / decoding settings, and in this invention, we will explain assuming that the projection format is identified by the projection format index. In this example, we will mainly explain the case where the projection format is represented by surfaces having the same size and shape, but configurations in which the size and shape of each surface are not the same are also possible. Furthermore, the configuration of each surface may be the same as or different from that in Figures 16a to 16d, and the numbers on each surface are used as symbols to identify each surface and are not limited to a specific order. For the sake of explanation, in the examples described later, based on the projection image, ERP is a projection format with one surface + rectangle, CMP is a projection format with six surfaces + rectangle, OHP is a projection format with eight surfaces + triangle, and ISP is a projection format with twenty surfaces + triangle, and we will explain assuming that the surfaces have the same size and shape, but the same or similar can be applied to other settings.

[0454] As shown in Figures 16a to 16d, the projection format can be divided into one surface (e.g., ERP) or multiple surfaces (e.g., CMP, OHP, ISP, etc.). Furthermore, each surface can be divided into shapes such as a rectangle and a triangle. The above divisions can be applied to cases where the encoding / decoding settings based on the projection format are different, and may be examples of image types and characteristics in the present invention. For example, the image type may be a 360-degree image, and the image characteristics may be any of the above divisions (e.g., each projection format, a projection format with one surface or multiple surfaces, a projection format with a rectangular or non-rectangular surface, etc.).

[0455] A two-dimensional planar coordinate system {e.g., (i, j)} can be defined for each surface of a two-dimensional projection image, and the properties of the coordinate system may vary depending on the projection format, the position of each surface, etc. In the case of ERP, one two-dimensional planar coordinate system is used, and other projection formats may have multiple two-dimensional planar coordinate systems depending on the number of surfaces. In this case, the coordinate system can be represented as (k, i, j), where k can be the index information of each surface.

[0456] Figure 17 is a conceptual diagram illustrating how a projection format according to one embodiment of the present invention is contained within a rectangular image.

[0457] In other words, 17a to 17c can be understood as the projection format of Figures 16b to 16d realized as rectangular images.

[0458] Referring to 17a to 17c, each image format can be configured into a rectangular shape for encoding / decoding 360-degree images. In the case of ERP, it can be used directly in a single coordinate system, but in the case of other projection formats, the coordinate systems of each surface can be integrated into a single coordinate system, and a detailed explanation of this is omitted.

[0459] Referring to 17a to 17c, it can be seen that during the process of constructing a rectangular image, areas filled with meaningless data such as blank spaces or backgrounds occur. That is, the image can consist of an area containing actual data (in this example, the surface; Active Area) and meaningless areas filled to construct the rectangular image (in this example, assumed to be filled with arbitrary pixel values; Inactive Area). This may degrade performance not only due to the encoding / decoding of the actual image data, but also due to the increase in the amount of encoded data caused by the increase in image size due to the aforementioned meaningless areas.

[0460] Therefore, further processes may be carried out to eliminate meaningless areas and construct the image from areas containing actual data.

[0461] Figure 18 is a conceptual diagram of a method for converting a projection format to a rectangular shape according to one embodiment of the present invention, which involves rearranging the surface to eliminate meaningless areas.

[0462] Referring to 18a to 18c, one example of rearranging 17a to 17c can be seen, and such a process can be defined as a regional packing process (CMP compact, OHP compact, ISP compact, etc.). In this case, not only the surface itself can be rearranged, but the surface can also be divided and rearranged (OHP compact, ISP compact, etc.). This can be done not only to remove meaningless areas, but also to improve coding performance through efficient surface arrangement. For example, when the image is arranged to have continuity between surfaces (e.g., B2-B3-B1, B5-B0-B4 in 18a, etc.), coding performance can be improved by improving prediction accuracy during coding. Here, regional packing by projection format is merely one example in the present invention and is not limited thereto.

[0463] Figure 19 is a conceptual diagram showing that the CMP projection format according to one embodiment of the present invention is a rectangular image used to perform the regional packing process.

[0464] Referring to 19a to 19c, the CMP projection format can be arranged as 6×1, 3×2, 2×3, 1×6, etc. Furthermore, if some surfaces are resized, they can be arranged as in 19d to 19e. Although CMP is given as an example in 19a to 19e, it is not limited to CMP and can be applied to other projection formats. The surface arrangement of images acquired via the regional packing follows predetermined rules of the projection format, or information regarding the arrangement can be explicitly generated.

[0465] A 360-degree image encoding / decoding device according to one embodiment of the present invention can be configured to include all or part of the image encoding / decoding device shown in Figures 1 and 2. In particular, a format conversion unit and a format inverse conversion unit for converting and inversely converting projection formats may be further included in the image encoding device and image decoding device, respectively. That is, the image encoding device in Figure 1 can encode the input image via the format conversion unit, and the image decoding device in Figure 2 can generate an output image after the bitstream has been decoded via the format inverse conversion unit. In the following, the encoder (in this example, "input image" to "encoding") for the above process will be explained in detail, and the process in the decoder can be derived in reverse from the encoder. Furthermore, explanations that overlap with the above content will be omitted.

[0466] Next, we will explain assuming that the input image is the same image as the 2D projection image or packing image obtained by performing the preprocessing process with the 360-degree encoding device described above. That is, the input image may be an image obtained by performing a projection process using a certain projection format or a regional packing process. The projection format already applied to the input image is one of various projection formats, and may be considered a common format, sometimes called the first format.

[0467] The format conversion unit can convert to projection formats other than the first format. In this case, the format to be converted to can be called the second format. For example, ERP can be set as the first format and converted to the second format (e.g., ERP2, CMP, OHP, ISP, etc.). In this case, ERP2 may be an ERP format that has the same 3D model, surface configuration, and other conditions, but with some settings different. Alternatively, it may be the same format with the same projection format settings (e.g., ERP=ERP2), but the image size or resolution may differ. Or, some of the image setting processes described later may be applied. For the sake of explanation, the above examples have been given, but the first and second formats are just some of the various projection formats, and are not limited to the above examples; changes to other formats are also possible.

[0468] During the conversion process between formats, due to the characteristics of the different coordinate systems between projection formats, pixels (integer pixels) in the converted image may be obtained not only from integer unit pixels in the original image but also from decimal unit pixels, allowing for interpolation. The interpolation filter used in this case can be the same or similar to those described above. The interpolation filter can be selected from several candidate filters, and related information can be explicitly generated, or it can be implicitly determined by pre-established rules. For example, a predetermined interpolation filter can be used depending on the projection format, color format, slice / tile type, etc. Furthermore, when an interpolation filter is explicitly sent, information about the filter (e.g., filter coefficients) may also be included.

[0469] The projection format in the format conversion unit may be defined to include regional packing, etc. In other words, projection and regional packing processes may be performed during the format conversion process. Alternatively, processes such as regional packing may be performed after format conversion but before encoding.

[0470] The encoder records the information generated during the above process into a bitstream in units of at least one of the following: sequence, picture, slice, or tile. The decoder then parses the related information from the bitstream. This information may also be included in the bitstream in the form of SEI or metadata.

[0471] Next, the image setting process applied to a 360-degree image encoding / decoding device according to one embodiment of the present invention will be described. The image setting process in the present invention can be applied not only to general encoding / decoding processes, but also to pre-processing processes, post-processing processes, format conversion processes, and inverse format conversion processes in a 360-degree image encoding / decoding device. The image setting process described later will be explained mainly in relation to the 360-degree image encoding device, and will include the content of the image setting described above. Repeated explanations of the image setting process described above will be omitted. In addition, the examples described later will be explained mainly in relation to the image setting process, and the inverse image setting process can be derived in reverse from the image setting process, and in some cases can be confirmed through the various embodiments of the present invention described above.

[0472] The image setting process in this invention may be performed during the 360-degree image projection stage, the regional packing stage, the format conversion stage, or any other stage.

[0473] Figure 20 is a conceptual diagram of 360-degree image segmentation according to one embodiment of the present invention. In Figure 20, the case of an image projected by ERP is assumed and explained.

[0474] Figure 20a shows an image projected by ERP, which can be divided using various methods. This example focuses on slicing and tiling, assuming that W0-W2 and H0, H1 are the division boundaries of the slice or tile, and that the order follows the raster scan sequence. The examples described later focus on slicing and tiling, but are not limited to these, and other division methods can be applied.

[0475] For example, the division can be performed in slice units, and the division boundaries can be H0 and H1. Alternatively, the division can be performed in tile units, and the division boundaries can be W0 to W2 and H0 and H1.

[0476] Figure 20b shows an example of dividing an image projected by ERP into tiles {assuming the same tile division boundaries as in Figure 20a (W0~W2, H0, H1 are all activated)}. Assuming that the P region is the overall image and the V region is the area where the user's gaze lingers or a viewport, there are various ways to provide the image corresponding to the viewport. For example, the overall image (e.g., tiles a~l) can be decoded to obtain the region corresponding to the viewport. In this case, the overall image can be decoded, and if it is divided, tiles a~l (in this example, the A+B region) can be decoded. Alternatively, the region corresponding to the viewport can be obtained by decoding the region belonging to the viewport. In this case, if it is divided, tiles f, g, j, k (in this example, the B region) can be decoded to obtain the region corresponding to the viewport from the reconstructed image. The former case can be called overall decoding (or Viewport Independent Coding), and the latter case can be called partial decoding (or Viewport Dependent Coding). The latter case is an example that can occur with 360-degree images that have a large amount of data, and because it is possible to flexibly obtain the divided regions, a tile-based division method is often used more than a slice-based division method. In the case of partial decoding, it is not possible to know where the viewport originates, so the referentiality of the division unit can be restricted spatially or temporally (implicitly processed in this example), and encoding / decoding can be performed taking this into account. The examples described later will mainly explain the case of whole decoding, but for the purpose of preparing for the case of partial decoding, the division of a 360-degree image will be explained mainly using tiles (or the quadrilateral division method of the present invention). The contents of the examples described later can be applied similarly or modified to other division units.

[0477] Figure 21 is an illustrative diagram of 360-degree image segmentation and image reconstruction according to an embodiment of the present invention. In Figure 21, the case of an image projected by CMP is assumed and explained.

[0478] 21a shows an image projected by CMP, which can be divided using various methods. W0~W2 and H0, H1 are assumed to be the division boundaries of the surface, slice, and tile, respectively, and are assumed to follow the raster scan order.

[0479] For example, division can be performed at the slice level, with division boundaries H0 and H1. Alternatively, division can be performed at the tile level, with division boundaries W0-W2 and H0 and H1. Or, division can be performed at the surface level, with division boundaries W0-W2 and H0 and H1. In this example, the surface is assumed to be part of the division unit.

[0480] In this case, a surface is a division unit (dependent encoding / decoding in this example) created for the purpose of classifying or dividing regions within the same image that have different properties (e.g., the planar coordinate system of each surface) according to the characteristics and type of the image (in this example, a 360-degree image, projection format), while a slice or tile may be a division unit (independent encoding / decoding in this example) created for the purpose of dividing the image according to the user's definition. Furthermore, a surface is a unit divided according to a predetermined definition (or derived from projection format information) during the projection process using the projection format, while a slice or tile may be a unit that is explicitly divided by generating division information according to the user's definition. Also, a surface may have a division shape that includes polygons, including quadrilaterals, depending on the projection format, a slice may have an arbitrary division shape that cannot be defined as a quadrilateral or polygon, and a tile may have a square division shape. The setting of the division units may be limited to the content defined for the purpose of explaining this example.

[0481] In the example above, a surface was described as a division unit classified for the purpose of region division. Depending on the encoding / decoding settings, it can be a unit that performs independent encoding / decoding at least one surface unit, or it can be configured to perform independent encoding / decoding in combination with tiles, slices, etc. In this case, when combining with tiles, slices, etc., explicit information of the tiles and slices may be generated, or there may be cases where tiles and slices are implicitly combined based on surface information. Alternatively, there may be cases where explicit information of tiles and slices is generated based on surface information.

[0482] As a first example, a single image segmentation process (a surface in this example) is performed, and the image segmentation implicitly omits segmentation information (obtaining segmentation information from projection format information). This example is an example of a dependent encoding / decoding setting and may apply when the referability between surface units is not restricted.

[0483] As a second example, a single image segmentation process (a surface in this example) is performed, and the image segmentation can explicitly generate segmentation information. This example is an example of a dependent encoding / decoding setting and may apply when the referability between surface units is not restricted.

[0484] As a third example, multiple image segmentation processes (in this example, surface and tile) are performed, and some image segmentation (in this example, surface) can implicitly omit or explicitly generate segmentation information, while some image segmentation (in this example, tile) can explicitly generate segmentation information. In this example, some image segmentation processes (in this example, surface) precede some image segmentation processes (in this example, tile).

[0485] As a fourth example, multiple image segmentation processes are performed, and some image segmentation (surfaces in this example) can implicitly omit or explicitly generate segmentation information, while some image segmentation (tiles in this example) can explicitly generate segmentation information based on some image segmentation (surfaces in this example). In this example, some image segmentation processes (surfaces in this example) precede some image segmentation processes (tiles in this example). In this example, although it is the same that segmentation information is explicitly generated in some cases {assuming it is the case in the second example}, there may be differences in the structure of the segmentation information.

[0486] As a fifth example, multiple image segmentation processes are performed, and some image segmentations (surfaces in this example) can implicitly omit segmentation information, and some image segmentations (tiles in this example) can implicitly omit segmentation information based on some image segmentations (surfaces in this example). For example, individual surface units can be set at the tile level, or multiple surface units (in this example, adjacent surfaces are grouped if they are continuous surfaces, and not grouped otherwise; B2-B3-B1 and B4-B0-B5 in 18a) can be set at the tile level. It is possible to set surface units to tile levels according to already established rules. This example is an example for independent encoding / decoding settings, and may be an example where the referability between surface units is limited. That is, in some cases {assuming the first example}, segmentation information is implicitly processed, but there may be differences in the encoding / decoding settings.

[0487] The above examples illustrate cases where segmentation processes may occur during the projection stage, regional packing stage, or initial encoding / decoding stage; however, other image segmentation processes that may occur within the encoder / decoder are also possible.

[0488] In 21a, a rectangular image can be constructed by including a data-free region B within a data-containing region A. In this case, the position, size, shape, and number of regions A and B are information that can be confirmed by the projection format, or by explicitly generating information for the projected image, and related information can be indicated using the aforementioned image segmentation information and image reconstruction information. For example, as shown in Tables 4 and 5, information for a part of the projected image (e.g., part_top, part_left, part_width, part_height, part_convert_flag, etc.) can be indicated, and this example is not limited to this example and may be applicable to other cases (e.g., other projection formats, other projection settings, etc.).

[0489] Region B can be combined with region A to form a single image for encoding / decoding. Alternatively, different encoding / decoding settings can be applied by dividing the image while considering the characteristics of each region. For example, encoding / decoding of region B may not be performed, based on information about whether or not to perform encoding / decoding (e.g., tile_coded_flag if the division unit is assumed to be a tile). In this case, the region can be restored to a certain data (arbitrary pixel value in this example) according to the already established rules. Alternatively, the encoding / decoding settings for region B can be set differently from those for region A during the image division process described above. Alternatively, the region can be removed by performing a regional packing process.

[0490] Figure 21b shows an example of dividing an image packed with CMP into tiles, slices, and surfaces. In this case, the packed image is an image that has undergone a surface rearrangement process or a regional packing process, and may be an image obtained by performing the image division and image reconstruction of the present invention.

[0491] In 21b, a rectangular shape can be formed including the region containing the data. At this time, the position, size, shape, and number of each region are information that can be confirmed by the already set settings, or by explicitly generating information for the packed image, and related information can be shown in the aforementioned image segmentation information, image reconstruction information, etc. For example, as shown in Tables 4 and 5, information for a part of the packed image (e.g., part_top, part_left, part_width, part_height, part_convert_flag, etc.) can be shown.

[0492] The packed image can be divided using various division methods. For example, it can be divided at the slice level, with division boundaries of H0. Alternatively, it can be divided at the tile level, with division boundaries of W0, W1 and H0. Or, it can be divided at the surface level, with division boundaries of W0, W1 and H0.

[0493] The image segmentation and reconstruction processes of the present invention can be performed on a projected image. In this case, the reconstruction process can rearrange not only pixels within a surface but also surfaces within the image. This is possible when the image is divided or composed of multiple surfaces. The examples described below will focus on cases where the image is divided into tiles based on surface units.

[0494] SX,Y (S0,0~S3,2) in 21a can correspond to S'U,V (S'0,0~S'2,1; in this example, X,Y may be the same as or different from U,V) in 21b, and a reconstruction process can be performed on a surface-by-surface basis. For example, S2,1, S3,1, S0,1, S1,2, S1,1, S1,0 can be assigned (or rearranged on the surface) to S'0,0, S'1,0, S'2,0, S'0,1, S'1,1, S'2,1. Also, S2,1, S3,1, S0,1 can be reconstructed (or have their pixels rearranged) without reconstruction (or pixel rearrangement), while S1,2, S1,1, S1,0 can be reconstructed by applying a 90-degree rotation, which can be shown as in Figure 21c. In 21c, the symbols displayed horizontally (S1,0, S1,1, S1,2) may be images that have been rotated horizontally to match the symbols in order to maintain image continuity.

[0495] Surface reconstruction can be implicit or explicit, depending on the encoding / decoding settings. In the implicit case, it can be performed according to pre-established rules, taking into account the image type (in this example, a 360-degree image) and characteristics (in this example, the projection format, etc.).

[0496] For example, in 21c, image continuity (or correlation) exists between S'0,0 and S'1,0, S'1,0 and S'2,0, S'0,1 and S'1,1, and S'1,1 and S'2,1 based on the surface boundaries, and 21c may be an example where continuity exists between the three upper surfaces and the three lower surfaces. A surface can be divided into multiple surfaces through a projection process from three-dimensional space to two-dimensional space, and then reconstructed through a regional packing process to enhance image continuity between surfaces for efficient surface reconstruction. Such surface reconstruction can already be set up and processed.

[0497] Alternatively, the reconstruction process can be carried out through explicit processing, and reconstruction information can be generated for that process.

[0498] For example, when verifying information (either implicitly acquired or explicitly generated) regarding an M×N configuration (e.g., 6×1, 3×2, 2×3, 1×6, etc. for CMP compact; in this example, a 3×2 configuration is assumed) via a regional packing process, the surface can be reconstructed according to the M×N configuration, and then the information can be generated. For example, in the case of in-image rearrangement of the surface, index information (or positional information in the image) can be assigned to each surface, and in the case of in-image pixel rearrangement, mode information for the reconstruction can be assigned.

[0499] The index information can already be defined as shown in Figures 18a to 18c, where SX, Y or S'U, V in 21a to 21c represent each surface with positional information indicating the horizontal and vertical directions (e.g., S[i][j]) or a single positional information (e.g., assuming that the positional information is assigned in raster scan order from the upper left surface of the image; S[i]), and an index for each surface can be assigned to this.

[0500] For example, when assigning indices to position information indicating the horizontal and vertical directions, in the case of Figure 21c, S'0,0 can be assigned to the index of surface 2, S'1,0 to surface 3, S'2,0 to surface 1, S'0,1 to surface 5, S'1,1 to surface 0, and S'2,1 to surface 4. Alternatively, when assigning an index to a single position information, S[0] can be assigned to the index of surface 2, S[1] to surface 3, S[2] to surface 1, S[3] to surface 5, S[4] to surface 0, and S[5] to surface 4. For the sake of explanation, in the examples described later, S'0,0 to S'2,1 will be referred to as a to f. Alternatively, it can also be expressed using position information indicating the horizontal and vertical directions of pixels or blocks based on the upper left side of the image.

[0501] For packed images acquired through an image reconstruction process (or regional packing process), the scan order of the surfaces may or may not be the same in the image, depending on the reconstruction settings. For example, if one scan order (e.g., raster scan) is applied to 21a, the scan orders of a, b, and c may be the same, while the scan orders of d, e, and f may not be the same. For example, if the scan order for 21a, a, b, and c follows the order (0, 0) → (1, 0) → (0, 1) → (1, 1), then the scan order for d, e, and f may follow the order (1, 0) → (1, 1) → (0, 0) → (0, 1). This can be determined according to the image reconstruction settings, and other projection formats may also have such settings.

[0502] In the image segmentation process in 21b, individual surface units can be set as tiles. For example, surfaces a to f can each be set as a tile unit. Alternatively, multiple surfaces can be set as a tile unit. For example, surfaces a to c can be set as one tile, and surfaces d to f can be set as one tile. The above configuration can be determined based on surface characteristics (e.g., continuity between surfaces), and different surface tile settings than the above example are possible.

[0503] Next, we will show an example of segmentation information obtained through multiple image segmentation processes. In this example, we will omit segmentation information for surfaces, assume that units other than surfaces are tiles, and explain the process assuming that segmentation information is processed in various ways.

[0504] As a first example, image segmentation information can be implicitly omitted by obtaining it based on surface information. For example, individual surfaces may be set as tiles, or multiple surfaces may be set as tiles. In this case, if at least one surface is set as a tile, it can be determined by a predetermined rule based on surface information (e.g., continuity or correlation).

[0505] As a second example, image segmentation information can be explicitly generated independently of surface information. For example, when generating segmentation information based on the number of horizontal tiles (num_tile_columns in this example) and the number of vertical tiles (num_tile_rows in this example), the segmentation information can be generated using the method described in the image segmentation process. For example, the possible ranges for the number of horizontal tiles and the number of vertical tiles can be from 0 to the image width / block width (unit obtained from the picture segmentation section in this example) and from 0 to the image height / block height. In addition, additional segmentation information (e.g., uniform_spacing_flag) can be generated. In this case, depending on the segmentation settings, the surface boundary and the segmentation unit boundary may or may not coincide.

[0506] As a third example, image segmentation information can be explicitly generated based on surface information. For example, when generating segmentation information based on the number of rows and columns of tiles, the segmentation information can be generated based on surface information (in this example, the range for the number of rows is 0 to 2, and the range for the number of columns is 0 to 1, since the surface configuration within the image is 3x2). For example, the possible ranges for the number of rows and columns of tiles can be from 0 to 2 and from 0 to 1. Furthermore, additional segmentation information (e.g., uniform_spacing_flag) does not need to be generated. In this case, the surface boundary and the boundary of the segmentation unit can coincide.

[0507] In some cases {assuming the cases of the second and third examples}, the syntactic elements of the segmentation information may be defined differently, or even if the same syntactic elements are used, the settings of the syntactic elements (e.g., binarization settings; if the range of candidate sets for a syntactic element is limited and small, other binarization methods may be used) may be different. The above examples illustrate only a part of the various configurations of segmentation information, but are not limited to these; they can be understood as examples of how different settings are possible depending on whether the segmentation information is generated based on surface information.

[0508] Figure 22 is an example diagram showing an image projected or packed by CMP divided into tiles.

[0509] At this time, we assume that the same tile division boundary (W0~W2, H0, H1 all activated) as in 21a of Figure 21 is present, and that the same tile division boundary (W0, W1, H0 all activated) as in 21b of Figure 21 is present. Assuming that the P region is the overall image and the V region is the viewport, overall decoding or partial decoding can be performed. This example will mainly explain partial decoding. In 22a, in the case of CMP (left), tiles e, f, and g are decoded, and in the case of CMP compact (right), tiles a, c, and e are decoded to obtain the region corresponding to the viewport. In 22b, in the case of CMP, tiles b, f, and i are decoded, and in the case of CMP compact, tiles d, e, and f are decoded to obtain the region corresponding to the viewport.

[0510] The above example described how to perform divisions such as slicing and tiling based on surface units (or surface boundaries). However, as shown in 20a of Figure 20, it is also possible to perform divisions within a surface (for example, ERP images are composed of a single surface, while other projection formats are composed of multiple surfaces) or to include surface boundaries in the division.

[0511] Figure 23 is a conceptual diagram illustrating an example of resizing a 360-degree image according to one embodiment of the present invention. In this example, the case of an image projected by an ERP system is assumed. Furthermore, the examples described later will focus primarily on the case of image expansion.

[0512] The projected image can be resized using either a scale factor or an offset factor, depending on the image size adjustment type. The image before resizing may be P_Width × P_Height, and the image after resizing may be P'_Width × P'_Height.

[0513] In the case of a scale factor, after adjusting the size using scale factors (in this example, width a, height b) on the image's width and height, the image's width (P_Width × a) and height (P_Height × b) can be obtained. In the case of an offset factor, after adjusting the size using offset factors (in this example, width L, R, height T, B) on the image's width and height, the image's width (P_Width + L + R) and height (P_Height + T + B) can be obtained. Size adjustment can be performed using a pre-configured method, or by selecting one of several methods.

[0514] The data processing methods described in the examples below will focus primarily on the case of the offset factor. In the case of the offset factor, possible data processing methods include filling using predetermined pixel values, filling by copying the outer pixels, filling by copying a portion of the image, and filling by transforming a portion of the image.

[0515] In the case of 360-degree images, resizing can be performed by considering the characteristic that the image boundaries have continuity. In the case of ERP, there is no outer boundary in 3D space, but when converted to 2D space through the projection process, an outer boundary region can exist. Data in the boundary region has continuous data outside the boundary, but due to spatial characteristics, it can have a boundary. Resizing can be performed by considering these characteristics. At this time, continuity can be confirmed depending on the projection format, etc. For example, in the case of ERP, the boundary at both ends may be continuous. In this example, we will explain assuming that the left and right boundaries of the image are continuous, and that the top and bottom boundaries of the image are continuous, and we will mainly explain the data processing methods, focusing on methods of copying and filling a part of the image region and methods of transforming and filling a part of the image region.

[0516] When resizing an image to the left, the area to be resized (LC or TL+LC+BL in this example) can be filled using data from the right-hand area of ​​the image (tr+rc+br in this example) that is continuous with the left-hand area. When resizing an image to the right, the area to be resized (RC or TR+RC+BR in this example) can be filled using data from the left-hand area of ​​the image (tl+lc+bl in this example) that is continuous with the right-hand area. When resizing an image to the top, the area to be resized (TC or TL+TC+TR in this example) can be filled using data from the bottom-hand area of ​​the image (bl+bc+br in this example) that is continuous with the top-hand area. When resizing an image to the bottom, the area to be resized (BC or BL+BC+BR in this example) can be filled using data from the area to be resized.

[0517] If the size or length of the area to be resized is m, the area to be resized can have a range of (-m, y) to (-1, y) (resizing to the left) or (P_Width, y) to (P_Width+m-1, y) (resizing to the right) relative to the coordinates of the image before resizing (in this example, x is 0 to P_Width-1). The position x' of the area for obtaining data of the area to be resized can be derived by the formula x'=(x+P_Width)%P_Width. In this case, x represents the coordinates of the area to be resized relative to the coordinates of the image before resizing, and x' represents the coordinates of the area referenced by the area to be resized relative to the coordinates of the image before resizing. For example, if you adjust the size on the left side, and m is 4 with an image width of 16, then (-4, y) can be obtained from (12, y), (-3, y) from (13, y), (-2, y) from (14, y), and (-1, y) from (15, y). Alternatively, if you adjust the size on the right side, and m is 4 with an image width of 16, then (16, y) can be obtained from (0, y), (17, y) from (1, y), (18, y) from (2, y), and (19, y) from (3, y).

[0518] If the size or length of the area to be resized is n, the area to be resized can have a range of (x, -n) to (x, -1) (upward resizing) or a range of (x, P_Height) to (x, P_Height+n-1) (downward resizing) relative to the coordinates of the image before resizing (in this example, y is 0 to P_Height-1). The position y' of the area for obtaining data of the area to be resized can be derived using a formula such as y'=(y+P_Height)%P_Height. In this case, y represents the coordinates of the area to be resized relative to the coordinates of the image before resizing, and y' represents the coordinates of the area referenced by the area to be resized relative to the coordinates of the image before resizing. For example, if the size is adjusted upwards, and n is 4 with a vertical image height of 16, then data can be obtained from (x, -4) at (x, 12), (x, -3) at (x, 13), (x, -2) at (x, 14), and (x, -1) at (x, 15). Alternatively, if the size is adjusted downwards, and n is 4 with a vertical image height of 16, then data can be obtained from (x, 16) at (x, 0), (x, 17) at (x, 1), (x, 18) at (x, 2), and (x, 19) at (x, 3).

[0519] After filling the data in the resizing area, the image coordinates can be adjusted based on the resized image coordinates (in this example, x is 0 to P'_Width-1, y is 0 to P'_Height-1). The above example may be applicable to latitude and longitude coordinate systems.

[0520] The following various size adjustment combinations are possible.

[0521] For example, the image may be resized by m to the left, or by n to the right, or by o to the top, or by p to the bottom.

[0522] For example, the image may be resized by m to the left and n to the right. Alternatively, the image may be resized by o to the top and p to the bottom.

[0523] For example, the image may be resized by m to the left, n to the right, and o to the top. Alternatively, the image may be resized by m to the left, n to the right, and p to the bottom. Alternatively, the image may be resized by m to the left, o to the top, and p to the bottom. Alternatively, the image may be resized by n to the right, o to the top, and p to the bottom.

[0524] For example, the image may be resized by m to the left, n to the right, o to the top, and p to the bottom.

[0525] As in the example above, at least one size adjustment is performed, and the image size may be implicitly adjusted depending on the encoding / decoding settings, or size adjustment information may be explicitly generated and the image size may be adjusted based on that information. That is, m, n, o, and p in the example above can be determined to predetermined values, or they can be explicitly generated as size adjustment information, or some can be determined to predetermined values ​​and some can be explicitly generated.

[0526] The above example mainly describes the case where data is obtained from a portion of an image, but other methods are also applicable. The data can be either pre-encoded pixels or post-encoded pixels, and this can be determined according to the characteristics of the image or stage on which the size adjustment is performed. For example, when size adjustment is performed in the pre-processing stage or pre-encoding stage, the data can represent input pixels such as projection images or packing images. When size adjustment is performed in the post-processing stage, such as the in-screen prediction reference pixel generation stage, reference image generation stage, or filtering stage, the data can represent restored pixels. Furthermore, size adjustment can be performed individually on each area to be adjusted using a data processing method.

[0527] Figure 24 is a conceptual diagram illustrating the continuity between surfaces using projection formats (e.g., CMP, OHP, ISP) according to one embodiment of the present invention.

[0528] More specifically, this could be an example for an image consisting of multiple surfaces. Continuity is a characteristic that occurs in adjacent regions in three-dimensional space, and Figures 24a to 24c can be classified into four cases when transformed into two-dimensional space via the projection process: (A) where the regions are spatially adjacent and have continuity, (B) where the regions are spatially adjacent but do not have continuity, (C) where the regions are not spatially adjacent but have continuity, and (D) where the regions are neither spatially adjacent nor have continuity. General images differ from those classified into (A) where the regions are spatially adjacent and have continuity, and (D) where the regions are not spatially adjacent but have no continuity. In this case, the cases where continuity exists are those described in some examples (A or C).

[0529] In other words, referring to 24a to 24c, cases where regions are spatially adjacent and have continuity (explained using 24a as the basis in this example) are represented as b0 to b4, and cases where regions are not spatially adjacent but have continuity are represented as B0 to B6. That is, this refers to the case of adjacent regions in three-dimensional space, and by using the characteristic that b0 to b4 and B0 to B6 have continuity in the encoding process, encoding performance can be improved.

[0530] Figure 25 is a conceptual diagram illustrating the surface continuity of Figure 21c, which is an image obtained through the image reconstruction process or regional packing process in the CMP projection format.

[0531] Here, 21c in Figure 21 is a rearrangement of 21a, which is a 360-degree image unfolded into a cube shape, so the continuity of the surface in 21a in Figure 21 is maintained in this case as well. That is, as in 25a, surface S2,1 can be continuous with S1,1 and S3,1 to the left and right, and can be continuous with the 90-degree rotated S1,0 and -90-degree rotated S1,2 surfaces to the right and left and right, respectively.

[0532] Using a similar method, the continuity with respect to surfaces S3,1, S0,1, S1,2, S1,1, and S1,0 can be confirmed from 25b to 25f.

[0533] The continuity between surfaces can be defined according to the projection format settings, and is not limited to the examples above; other variations are possible. The examples described later will be explained under the assumption that continuity exists as shown in Figures 24 and 25.

[0534] Figure 26 is an illustrative diagram illustrating image resizing in the CMP projection format according to one embodiment of the present invention.

[0535] 26a shows an example of adjusting the size of an image, 26b shows an example of adjusting the size on a surface-by-surface basis (or division-by-subdivision basis), and 26c shows an example of adjusting the size of an image and / or multiple surfaces.

[0536] The projected image can be resized using either a scale factor or an offset factor, depending on the image resizing type. The image before resizing is P_Width × P_Height, the image after resizing is P'_Width × P'_Height, and the surface size may be F_Width × F_Height. Surfaces may have the same or different sizes, and their width and height may be the same or different. However, for the sake of explanation, this example assumes that all surfaces in the image are the same size and have a square shape. It also assumes that the resizing values ​​(WX and HY in this example) are the same. The data processing methods in the examples described later will mainly focus on the case of the offset factor, and the data processing methods will mainly focus on copying and filling a portion of the image, and transforming and filling a portion of the image. The above settings can also be applied to Figure 27.

[0537] In cases 26a to 26c, the surface boundaries (assuming in this example that they have continuity as shown in 24a of Figure 24) can have continuity with the boundaries of other surfaces. In this case, it can be divided into two cases: one where the surfaces are spatially adjacent in a two-dimensional plane and have image continuity (first example), and another where the surfaces are not spatially adjacent in a two-dimensional plane but have image continuity (second example).

[0538] For example, assuming the continuity of 24a in Figure 24, the upper, left, right, and lower regions of S1,1 are spatially adjacent to the lower, right, left, and upper regions of S1,0, S0,1, S2,1, and S1,2, and the images may also be continuous (in the first example).

[0539] Alternatively, the left and right regions of S1,0 may not be spatially adjacent to the upper regions of S0,1 and S2,1, but their images may be continuous (in the second example). Similarly, the left region of S0,1 and the right region of S3,1 may not be spatially adjacent, but their images may be continuous (in the second example). Furthermore, the left and right regions of S1,2 may be continuous with the lower regions of S0,1 and S2,1 (in the second example). This is a limited example in this case, and different configurations are possible depending on the definition and settings of the projection format. For the sake of explanation, S0,0 to S3,2 in Figure 26a are referred to as a to l.

[0540] 26a may be an example of filling with data from regions where continuity exists in the direction of the outer boundary of the image. Regions resized from region A where no data exists (in this example, a0~a2, c0, d0~d2, i0~i2, k0, l0~l2) can be filled with a predetermined arbitrary value or through outer pixel padding, and regions resized from region B containing actual data (in this example, b0, e0, h0, j0) can be filled with data from regions (or surfaces) where image continuity exists. For example, b0 can be filled with data from the upper side of surface h, e0 with data from the right side of surface h, h0 with data from the left side of surface e, and j0 with data from the lower side of surface h.

[0541] In detail, b0 is an example of filling using the underside data of the surface obtained by applying a 180-degree rotation to surface h, and j0 may be an example of filling using the upper side data of the surface obtained by applying a 180-degree rotation to surface h. However, in this example (or including the example described later), only the position of the referenced surface is shown, and the data acquired in the resizing region can be obtained after undergoing an adjustment process (e.g., rotation) that takes into account the continuity between surfaces, as shown in Figures 24 and 25.

[0542] 26b could be an example of filling with data from regions where continuity exists in the direction of the internal boundary of the image. In this example, the resizing operation performed along the surface may differ. Region A can undergo a reduction process, and region B can undergo an expansion process. For example, in the case of surface a, the size may be adjusted by w0 to the right (reduction in this example), and in the case of surface b, the size may be adjusted by w0 to the left (expansion in this example). Alternatively, in the case of surface a, the size may be adjusted by h0 downwards (reduction in this example), and in the case of surface e, the size may be adjusted by h0 upwards (expansion in this example). In this example, looking at the change in the width of the image from surfaces a, b, c, and d, surface a is reduced by w0, surface b is expanded by w0 and w1, and surface c is reduced by w1, so the width of the image before resizing and the width of the image after resizing are the same. Observing the changes in the vertical height of the images from surfaces a, e, and i, we see that surface a is reduced by h0, surface e is expanded by h0 and h1, and surface i is reduced by h1. Therefore, the vertical height of the image before resizing is the same as the vertical height of the image after resizing.

[0543] The regions to be resized (in this example, b0, e0, be, b1, bg, g0, h0, e1, ej, j0, gi, g1, j1, h1) can be simply removed, considering that they will be reduced from region A where no data exists, or they can be newly filled with data from regions where continuity exists, considering that they will be expanded from region B which contains the actual data.

[0544] For example, the data can be filled using the following: b0 is the upper side of surface e, e0 is the left side of surface b, be is the left side of surface b, or the upper side of surface e, or the weighted sum of the left side of surface b and the upper side of surface e, b1 is the upper side of surface g, bg is the left side of surface b, or the upper side of surface g, or the weighted sum of the right side of surface b and the upper side of surface g, g0 is the right side of surface b, h0 is the upper side of surface b, e1 is the left side of surface j, ej is the lower side of surface e, or the left side of surface j, or the weighted sum of the lower side of surface e and the left side of surface j, j0 is the lower side of surface e, gj is the lower side of surface g, or the left side of surface j, or the weighted sum of the lower side of surface g and the right side of surface j, g1 is the right side of surface j, j1 is the lower side of surface g, and h1 is the lower side of surface j.

[0545] In the above example, when filling a resized area with data from a portion of an image, it is possible to either copy the data from that area and fill it, or to fill it with data acquired after undergoing a transformation process based on the image's characteristics and type. For example, when a 360-degree image is transformed into a 2D space according to its projection format, a coordinate system (e.g., a 2D planar coordinate system) can be defined for each surface. For the sake of explanation, let's assume that the coordinates (x, y, z) in 3D space are transformed into (x, y, C), (x, C, z), or (C, y, z) for each surface. The above example shows the case where data from a surface different from the current surface is acquired for a resized area of ​​a portion of that surface. That is, while resizing is performed around the current surface, if data from another surface with different coordinate system characteristics is simply copied and filled, there is a possibility that continuity may be distorted based on the resizing boundary. For this reason, it is also possible to transform data from other surfaces acquired to match the coordinate system characteristics of the current surface and fill it into the resized area. Transformation is merely one example of a data processing method and is not limited to it.

[0546] When copying and filling data from a portion of an image into a region to be resized, the boundary region between the region to be resized (e) and the region being resized (e0) may contain distorted continuity (or rapidly changing continuity). For example, continuous features may change with respect to the boundary, similar to how an edge that was straight becomes folded with respect to the boundary.

[0547] When converting data from a portion of an image to fill a region being resized, the boundary region between the region being resized and the region being resized can include a gradually changing continuity.

[0548] The above example may be an example of the data processing method of the present invention, in which data from a portion of an image is transformed during the resizing process (in this example, expansion) based on the characteristics and type of the image, and the acquired data is then filled into the resized area.

[0549] 26c may be an example of combining the image resizing processes of 26a and 26b to fill the image using data from regions where continuity exists in the direction of the image boundaries (inner boundary and outer boundary). The resizing process in this example can be derived from 26a and 26b, so a detailed explanation is omitted.

[0550] 26a may be an example of an image resizing process, and 26b may be an example of an image resizing process for a division unit within an image. 26c may be an example of multiple resizing processes in which both an image resizing process and image division unit resizing are performed.

[0551] For example, the image acquired through the projection process (the first format in this example) can be resized (region C in this example), and the image acquired through the format conversion process (the second format in this example) can be resized (region D in this example). In this example, the image projected by ERP is resized (the entire image in this example), which could be an example where the image projected by CMP via the format conversion unit is resized (on a surface-by-surface basis in this example). The above example is just one example of performing multiple resizing operations, and is not limited to this; variations to other cases are also possible.

[0552] Figure 27 is an illustrative diagram illustrating resizing for an image that has been converted to a CMP projection format and packed according to one embodiment of the present invention. As with Figure 27, the explanation is based on the premise of continuity between surfaces as shown in Figure 25, so the boundaries of one surface can have continuity with the boundaries of other surfaces.

[0553] In this example, the offset factors of W0 to W5 and H0 to H3 (assuming in this example the offset factors are used as size adjustment values) can have various values. For example, they can be derived from predetermined values, the motion search range for inter-screen prediction, units obtained from the picture division section, etc., and other cases are also possible. In this case, the units obtained from the picture division section may include surfaces. That is, the size adjustment values ​​can be determined based on F_Width and F_Height.

[0554] 27a is an example of filling each expanded region with data from a region where continuity exists, by performing size adjustments (in this example, in the top, bottom, left, and right directions of each surface). For example, continuous data can be filled into the outer regions a0 to a6 of surface a, and continuous data can be filled into the outer regions b0 to b6 of surface b.

[0555] 27b is an example of filling an expanded region using data from a region where continuity exists, by performing size adjustments on multiple surfaces (in this example, in the upward, downward, left, and rightward directions of multiple surfaces). For example, using surfaces a, b, and c as a base, the outer boundaries can be expanded to a0-a4, b0-b1, and c0-c4.

[0556] 27c is an example of filling an expanded region with data from a region where continuity exists, by adjusting the size of the entire image (in this example, in the top, bottom, left, and right directions of the entire image). For example, the outer edge of the entire image consisting of surfaces a to f can be expanded to a0 to a2, b0, c0 to c2, d0 to d2, e0, and f0 to f2.

[0557] In other words, it is possible to adjust the size on a single surface unit, on multiple continuous surface units, and on the entire surface unit.

[0558] The resizing region in the above example (a0 to f7 in this example) can be filled using data from a continuous region (or surface) as shown in Figure 24a. That is, the resizing region can be filled using data from the upper, lower, left, and right sides of surfaces a to f.

[0559] Figure 28 is an illustrative diagram illustrating a data processing method for resizing a 360-degree image according to one embodiment of the present invention.

[0560] Referring to Figure 28, the resizing region B (a0~a2, ad0, b0, c0~c2, cf1, d0~d2, e0, f0~f2) can be filled with pixel data belonging to a through f that have continuity. Alternatively, another resizing region C (ad1, be, cf0) can be filled by mixing data from the region to be resized with data from a region that is spatially adjacent but not continuity. Or, since resizing is performed between two regions selected from a through f (for example, a and d, b and e, c and f), the data from these two regions can be mixed and used to fill the region. For example, surfaces b and e can have a relationship where they are spatially adjacent but not continuity. The resizing region be between surfaces b and e can be resized using data from surfaces b and e. For example, the be region can be filled with a value obtained by averaging the data from surfaces b and e, or with a value obtained through a distance-weighted sum. In this case, the pixels used for the data to fill the region resized by surface b and surface e may be boundary pixels of each surface, but they may also be internal pixels of the surfaces.

[0561] In summary, the areas between the image division units that are resized can be filled with data generated using a mixture of data from both units.

[0562] The aforementioned data processing method may be a method that is supported in certain situations (in this example, when resizing is performed in multiple areas).

[0563] In Figures 27a and 27b, the regions where size adjustments are performed between division units are configured individually for each division unit (for example, in Figure 27a, a6 and d1 are configured for a and d respectively). However, in Figure 28, the regions where size adjustments are performed between division units can be configured one for each adjacent division unit (one ad1 for a and d). Of course, the aforementioned method can also be included in the candidate group of data processing methods in Figures 27a to 27c, and size adjustments can be performed in Figure 28 using a data processing method different from the above example.

[0564] In the image resizing process of the present invention, a predetermined data processing method can be implicitly applied to the region to be resized, or relevant information can be explicitly generated using one of a plurality of data processing methods. The predetermined data processing method may be any of the following: a method of filling with arbitrary pixel values, a method of filling by copying outer pixels, a method of filling by copying a portion of an image, a method of filling by transforming a portion of an image, or a method of filling with data derived from multiple regions of an image. For example, if the region to be resized is located inside an image (e.g., a packed image) and the regions on both sides (e.g., the surface) are spatially adjacent but not continuous, a data processing method that fills with data derived from multiple regions can be applied to fill the region to be resized. Furthermore, resizing can be performed by selecting one of the plurality of data processing methods, and the selection information for this can be explicitly generated. This may be applicable not only to 360-degree images but also to general images.

[0565] In the encoder, the information generated in the above process is recorded in the bitstream in units of at least one of the following: sequence, picture, slice, or tile. In the decoder, the related information is parsed from the bitstream. This information may also be included in the bitstream in the form of SEI or metadata. The division, reconstruction, and resizing processes for 360-degree images have been explained focusing on certain projection formats such as ERP and CMP, but are not limited to these, and the same or modified content can be applied to other projection formats.

[0566] As explained earlier, the image setup process applied to the aforementioned 360-degree image encoding / decoding device can be applied not only to the encoding / decoding process, but also to pre-processing, post-processing, format conversion, and inverse format conversion processes.

[0567] In summary, the projection process can be comprised of an image setup process. More specifically, the projection process may include at least one image setup process. The projected image can be divided into regions (or surfaces). Depending on the projection format, it can be divided into one or more regions. Division information can be generated based on the division. The size of the projected image can be adjusted, or the size of the projected region can be adjusted. In this case, the size can be adjusted to at least one region. Size adjustment information can be generated based on the size adjustment. The projected image can be reconstructed (or surface-arranged), or the projected region can be reconstructed. In this case, reconstruction can be performed in at least one region. Reconstruction information can be generated based on the reconstruction.

[0568] In summary, the regional packing process can be comprised of an image setup process. More specifically, the regional packing process can be comprised of at least one image setup process. A segmentation process can be performed on a region (or surface) basis based on the packed image. The image can be segmented into one or more regions according to the regional packing settings. Segmentation information can be generated based on the segmentation. Furthermore, the size of the packed image can be adjusted, or the size of the packed region can be adjusted. In this case, the size adjustment can be performed on at least one region. Size adjustment information can be generated based on the size adjustment. Furthermore, the packed image can be reconstructed, or the packed region can be reconstructed. In this case, the reconstruction can be performed on at least one region. Reconstruction information can be generated based on the reconstruction.

[0569] The projection process may include all or part of the image setting process and may contain image setting information. This may be setting information for the projected image. More specifically, it may be setting information for a region within the projected image.

[0570] The aforementioned regional packing process involves all or part of the image setting process and may include image setting information. This may be setting information for the packed image. More specifically, it may be setting information for regions within the packed image. Alternatively, it may be mapping information between the projected image and the packed image (see, for example, the explanation related to Figure 11; it can be understood by assuming that P0 to P1 are the projected image and S0 to S5 are the packed image). More specifically, it may be mapping information between a portion of the projected image and a portion of the packed image. In other words, it may be setting information assigned from a portion of the projected image to a portion of the packed image.

[0571] The aforementioned information can be represented by information obtained through the various embodiments described above during the image setting process of the present invention. For example, when the relevant information is shown in Tables 1 to 6 using at least one syntactic element, the setting information for projected images may include pic_width_in_samples, pic_height_in_samples, part_top[i], part_left[i], part_width[i], part_height[i], etc., and the setting information for packed images may include pic_width_in_samples, pic_height_in_samples, part_top[i], part_left[i], part_width[i], part_height[i], convert_type_flag[i], part_resizing_flag[i], top_height_offset[i], bottom_height_offset[i], left_width_offset[i], right_width_offset[i], resizing_type_flag[i], etc. The above example could be one instance of explicitly generating information for a surface (for example, part_top[i], part_left[i], part_width[i], and part_height[i] from the settings information of the projected image).

[0572] Part of the image setup process may be included in the projection process based on the projection format or the regional packing process through predetermined operations.

[0573] For example, in the case of ERP, the process of resizing may implicitly include a method of copying data from the region in the opposite direction of the image resizing direction (right and left in this example) and filling it into a region that has been expanded by m and n in the left and right directions, respectively. Alternatively, in the case of CMP, the process of resizing may implicitly include a method of transforming data from a region that has continuity with the region to be resized and filling it into a region that has been expanded by m, n, o, and p in the up, down, left, and right directions, respectively.

[0574] The projection formats in the above examples may be examples that replace existing projection formats, or examples of projection formats that are additional to existing projection formats (e.g., ERP1, CMP1). Not limited to the above examples, various examples of the image setting process of the present invention can be combined alternatively. Similar applications are possible for other formats.

[0575] On the other hand, although not shown in Figures 1 and 2 of the image encoding and decoding apparatus, a block division section may also be included. Information for the basic encoding unit can be obtained from the picture division section, and the basic encoding unit can mean the basic (or starting) unit for prediction, transformation, quantization, etc., in the image encoding / decoding process. In this case, the encoding unit can consist of one luminance encoding block and two chrominance encoding blocks depending on the color format (YCbCr in this example), and the size of each block can be determined according to the color format. In the example described later, the explanation will be based on the block (luminance component in this example). In this explanation, it will be a...

Claims

1. A method for decoding an image using a decoding device, The steps include receiving the bitstream from which the aforementioned image has been encoded, The step of determining the current block included in the aforementioned image, The steps include obtaining information about the residual block of the current block from the bitstream, A step of generating a residual block of the current block based on the information of the residual block, The steps include: reconstructing the current block based on the residual block to generate a reconstructed image; The step includes performing image processing on the reconstructed image based on the image processing information contained in the bitstream, The image processing includes the step of padding at least one region into the reconstructed image, The aforementioned padding is, This is done by selecting one of several padding methods. The aforementioned plurality of padding methods include a first padding method that uses a predetermined sample value for the area to be padded, The aforementioned image processing is performed after in-loop filtering is performed on the reconstructed image. The padding is performed individually for each of the multiple division units that make up the reconstructed image. Decryption method.

2. A method for encoding an image using an encoding device, The step of determining the current block included in the aforementioned image, The step of generating the residual block of the current block, The steps include encoding the information of the residual block into a bitstream, The step includes encoding image processing information into the bitstream, The aforementioned image processing information is used to perform image processing on the reconstructed image, and the reconstructed image is generated by reconstructing the current block based on the residual block. The image processing includes the step of padding at least one region into the reconstructed image, The aforementioned padding is, This is done by selecting one of several padding methods. The aforementioned plurality of padding methods include a first padding method that uses a predetermined sample value for the area to be padded, The aforementioned image processing is performed after in-loop filtering is performed on the reconstructed image. The padding is performed individually for each of the multiple division units that make up the reconstructed image. Encoding method.

3. A program that causes a computer to perform an image encoding method, The aforementioned encoding method is The step of determining the current block included in the aforementioned image, The step of generating the residual block of the current block, The steps include encoding the information of the residual block into a bitstream, The step includes encoding image processing information into the bitstream, The aforementioned image processing information is used to perform image processing on the reconstructed image, and the reconstructed image is generated by reconstructing the current block based on the residual block. The image processing includes the step of padding at least one region into the reconstructed image, The aforementioned padding is, This is done by selecting one of several padding methods. The aforementioned plurality of padding methods include a first padding method that uses a predetermined sample value for the area to be padded, The aforementioned image processing is performed after in-loop filtering is performed on the reconstructed image. The padding is performed individually for each of the multiple division units that make up the reconstructed image. program.

4. A method for transmitting a bitstream generated by an image encoding method, The aforementioned encoding method is The step of determining the current block included in the aforementioned image, The step of generating the residual block of the current block, The steps include encoding the information of the residual block into a bitstream, The step includes encoding image processing information into the bitstream, The aforementioned image processing information is used to perform image processing on the reconstructed image, and the reconstructed image is generated by reconstructing the current block based on the residual block. The image processing includes the step of padding at least one region into the reconstructed image, The aforementioned padding is, This is done by selecting one of several padding methods. The aforementioned plurality of padding methods include a first padding method that uses a predetermined sample value for the area to be padded, The aforementioned image processing is performed after in-loop filtering is performed on the reconstructed image. The padding is performed individually for each of the multiple division units that make up the reconstructed image. Sending method.

Images