Video encoding and decoding method and device for performing format transformation

Abstract

The present invention relates to a video encoding and decoding method and device for performing format transformation, and the encoding method comprises the steps of: receiving a video in an RGB format; transforming the received RGB format video into a video in a YCbCr format; encoding the transformed YCbCr format video; and transmitting transform information, which has been used to transform the RGB format video into the YCbCr format video, together with the encoded YCbCr format video.

Description

Image encoding and decoding method and device for performing format conversion processing

[0001] The present invention relates to an encoding and decoding method that enables efficient provision of cloud gaming services.

[0002] As the cloud computing industry develops, the adoption of cloud technology in the gaming market is changing the market structure that was previously centered on console games.

[0003] In the case of general online games or mobile games, the game is provided in a way that the user installs the desired game on the terminal and plays the game while minimizing the connection to the server.

[0004] In contrast, in the case of cloud gaming, all game-related information is stored on a cloud server and provided as a service that immediately streams data to the user's terminal.

[0005] In other words, cloud gaming is a game service that allows users to play games and save items by connecting to a game server without installing the game on a terminal such as a smartphone, TV, computer, or tablet.

[0006] In this case, the game's main operations are processed on the game server, and the user terminal only displays the game's progress screen, so it is possible to enjoy high-spec games even on terminals with low specifications.

[0007] To provide such cloud gaming services, the game server provides all game services requested by users in the form of real-time video streaming. Game videos are computer graphic processed data, and there are many differences from real-life videos captured by a camera, such as changes in pixel values.

[0008] Meanwhile, the video compression methods used in cloud gaming services include video compression standard technologies such as H.264 / AVC, H.265 / HEVC, VP8, VP9, ​​and AV1. However, these video compression standard technologies were developed to be suitable for real-life videos shot by cameras, and are not efficient for game videos generated through computer graphics processing.

[0009] In addition, unlike OTT services such as Netflix that pre-encode content and stream it in real time upon user request, cloud gaming services receive feedback on user action events and immediately reflect them in game videos to perform real-time video compression and streaming. Therefore, the complexity of video compression is a very important issue in cloud gaming services.

[0010] The purpose of the present invention is to provide a method and device for efficiently encoding and decoding an image through format conversion processing.

[0011] An image encoding method according to one embodiment of the present invention includes the steps of: receiving an image in RGB format; converting the received RGB format image into an image in YCbCr format; encoding the converted YCbCr format image; and transmitting conversion information used to convert the RGB format image into the YCbCr format image together with the encoded YCbCr format image.

[0012] The image encoding device according to one embodiment of the present invention performs the image encoding method as described above.

[0013] An image decoding method according to one embodiment of the present invention includes the steps of: receiving a bitstream including information on an encoded YCbCr format image and conversion information used to convert an RGB format image into the YCbCr format image; decoding the encoded YCbCr format image; and converting the decoded YCbCr format image into an RGB format image using the conversion information.

[0014] The image decoding device according to one embodiment of the present invention performs the image decoding method as described above.

[0015] Meanwhile, the conversion information may include at least one of color specification information indicating a color range that can be expressed by the RGB format, color conversion information indicating a conversion method used for color conversion from the RGB format to the YCbCr format, and conversion function information indicating a conversion function used for optical-electrical conversion for the RGB format image.

[0016] Meanwhile, at least some steps of the above-described video encoding and decoding method may be implemented as a computer-readable recording medium recording a program for execution on a computer, or may be provided as the program itself.

[0017] According to one embodiment of the present invention, conversion information used to convert an RGB format image into a YCbCr format image is transmitted to a decoder together with an encoded YCbCr format image, so that the color, etc. of the original image can be expressed as is in the decoder.

[0018] According to another embodiment of the present invention, when encoding an image generated by graphic processing, such as a game image, object position information or background transition information provided from a game server can be applied to block division or encoding mode determination, thereby reducing the computational complexity of the encoder.

[0019] Figure 1 is a block diagram showing the overall configuration of a cloud game service system to which the present invention can be applied.

[0020] Figure 2 is a flowchart showing an image encoding method according to one embodiment of the present invention.

[0021] FIG. 3 is a block diagram showing an embodiment of a configuration of a media server that performs a video encoding method.

[0022] Figure 4 is a flowchart showing an image decoding method according to one embodiment of the present invention.

[0023] Fig. 5 is a block diagram showing an example of a configuration of a user terminal that performs an image decryption method.

[0024] Figure 6 is a block diagram illustrating the configuration and operation of a video encoder for encoding an image.

[0025] Figures 7 to 9 are drawings for explaining embodiments of a method for dividing blocks of an image.

[0026] Figure 10 is a block diagram illustrating the configuration and operation of a video decoder for decoding an image.

[0027] Figure 11 is a drawing for comparing and explaining the characteristics of real-life images captured by a camera and game images.

[0028] Figure 12 is a drawing for explaining the characteristics according to the movement of an object in a game video.

[0029] Figure 13 is a flowchart illustrating an image encoding method according to another embodiment of the present invention.

[0030] FIGS. 14 to 17 are drawings for explaining embodiments of a method for determining block division and encoding mode according to object location information.

[0031] Hereinafter, with reference to the attached drawings, a method and device for encoding and decoding an image according to an embodiment of the present invention will be described in detail.

[0032] In the following description of the present invention, detailed descriptions of related known functions or configurations will be omitted if they are deemed to unnecessarily obscure the gist of the invention. Furthermore, the terms described below are defined in light of their functions within the present invention and may vary depending on the intent or custom of the user or operator. Therefore, their definitions should be based on the overall content of the present invention.

[0033] In addition, in order to efficiently explain the technical components that make up the present invention, the preferred embodiments of the present invention implemented below omit, as much as possible, the system functional components that are already provided in each system functional configuration or are commonly provided in the technical field to which the present invention belongs, and focus on explaining the functional components that must be additionally provided for the present invention.

[0034] Anyone having ordinary skill in the art to which the present invention pertains will be able to easily understand the functions of components that have been conventionally used among the functional configurations that are not illustrated below, and will also be able to clearly understand the relationship between the components omitted as described above and the components added for the present invention.

[0035] In this specification, a device that encodes an image to generate a video signal bitstream is referred to as an encoding device, an encoding device, or an encoder, and a device that decodes the video signal bitstream to restore an image is referred to as a decoding device, a decoding device, or a decoder.

[0036] A pixel or pel is the smallest unit that constitutes an image, and the terms pixel and sample can be used interchangeably. A sample can generally represent a pixel or a pixel value, or it can represent only a pixel or pixel value of the luminance component, or only a pixel or pixel value of the chroma component.

[0037] In addition, the unit is used to refer to a basic unit of image processing or a specific location of a picture, and represents an image area including at least one of a luminance component and a chrominance component. Specifically, the unit can be used as a concept including a coding tree unit (CTU), a coding unit (CU), a prediction unit (PU), and a transform unit (TU). In addition, a block represents an image area including a specific component among luminance components and chrominance components, and an MxN block can represent a set of samples or transform coefficients consisting of M columns and N rows. Here, the terms unit, block, partition, signal, and area may be used interchangeably.

[0038] Meanwhile, a picture refers to a field or a frame, and these can be used interchangeably. For example, if the image is an interlaced image, one frame is divided into an odd (or odd, top) field and an even (or even, bottom) field, and each field is composed of a single picture unit, which can be encoded or decoded. If the image is a progressive image, one frame can be composed as a picture, which can be encoded or decoded.

[0039]

[0040] FIG. 1 is a block diagram illustrating the overall configuration of a cloud game service system to which the present invention can be applied, and the cloud game service can be provided through a game server (10), a media server (20), and a user terminal (30).

[0041] Referring to FIG. 1, the game server (10) provides a game service requested by a user in the form of real-time video streaming, and for this purpose, can transmit a game video generated through computer graphics processing to a media server (20).

[0042] The media server (20) compresses game images received from the game server (10) and transmits them to the user terminal (30). For this purpose, a video encoder for encoding the images may be included in the media server (20).

[0043] The user terminal (30) decrypts and plays game video streamed in real time from the game server (10) through the media server (20), and for this purpose, a video decoder for decrypting the video may be included in the user terminal (30).

[0044] In addition, the user terminal (30) transmits an input signal corresponding to the user's motion event to the game server (10) for feedback, and the game server (10) can generate a game video reflecting the feedback and stream it through the media server (20).

[0045] According to one embodiment of the present invention, a game image generated by a game server (10) may be an image in RGB format, and in this case, the media server (20) may convert an RGB (Red, Green, Blue) format image into a YCbCr (Luminace, Chroma blue, Chroma red) format image, encode the converted image, and transmit the encoded image to a user terminal (30).

[0046] Figure 2 is a flowchart illustrating an image encoding method according to an embodiment of the present invention.

[0047] Referring to FIG. 2, when an image in RGB format is received (step S200), the received RGB format image is converted into an image in YCbCr format (step S210).

[0048] After that, the YCbCr format image converted in step S210 is encoded (step S220), and the conversion information used to convert the RGB format image into a YCbCr format image in step S210 is transmitted together with the YCbCr format image encoded in step S220 (step S230).

[0049] Here, the conversion information may include color specification information indicating a color space that can be expressed by the RGB format of the received image, and color conversion information indicating a conversion method used for color transformation from the RGB format to the YCbCr format.

[0050] Additionally, in step S210, optical-electro conversion may be performed on an RGB format image, in which case transfer function information representing a transfer function used in the optical-electro conversion may be included in the conversion information.

[0051] Therefore, the conversion information transmitted together with the YCbCr format image encoded in step S230 may include color specification information, color conversion information, and conversion function information, but the present invention is not limited thereto, and may include various pieces of information related to format conversion.

[0052] FIG. 3 is a block diagram illustrating an embodiment of a configuration of a media server that performs a video encoding method, and shows the configuration of a media server (20) for performing the encoding method described with reference to FIG. 2.

[0053] Referring to FIG. 3, the media server (20) may include an optical-electrical conversion unit (21), a color conversion unit (22), a quantization unit (23), a downsampling unit (24), and an encoding unit (27).

[0054] First, the game image received from the game server (10) to the media server (20) may be an image in RGB format, and the color range that can be expressed by the RGB format may be determined by the color standard.

[0055] For example, the color standard of the RGB format may be BT.709-6, P3, BT.2020 or BT.2020-2, but the present invention is not limited thereto.

[0056] The optical-electric conversion unit (21) can perform optical-electric conversion on an RGB format image received from a game server (10) to convert an optical signal (RGB linear-light) into an electrical signal (R'G'B'(float)).

[0057] The transfer function used for the above optical-electrical conversion may be an optical-electrical transfer function (OETF) or an inverse function of an electro-optical transfer function (EOTF).

[0058] For example, the conversion function used for optical-electrical conversion may be in the form of an inverse HLG or inverse PQ, which is an inverse function of the electro-optical conversion function (EOTF), HLG (Hybrid Log-Gamma), or PQ (Perceptual Quantizer), but the present invention is not limited thereto.

[0059] The color conversion unit (22) can convert the RGB format image (R'G'B'(float)) output from the optical-electric conversion unit (21) into a YCbCr format image (Y'Cb'Cr'(float)).

[0060] Color conversion from RGB format to YCbCr format may use a Constant Luminance (CL) or Non-Constant Luminance (NCL) conversion method, but the present invention is not limited thereto.

[0061] When changing from RGB format to YCbCr format, in the case of Constant Luminance (CL), the luminance (Y) value is decoupled from the chrominance (Cb, Cr) value and converted, and in the case of Non-Constant Luminance (NCL), the luminance (Y) value is not completely decoupled from the chrominance (Cb, Cr) value and converted.

[0062] Next, the quantization unit (23) can perform quantization on the YCbCr format image (Y'Cb'Cr'(float)) converted by the color conversion unit (22) to output a YCbCr format image (Y'Cb'Cr'(integer)) having an integer value.

[0063] The downsampling unit (24) can perform chroma downsampling on the image input from the quantization unit (23) to change the YUV444 format to the YUV420 format, thereby outputting a YCbCr format image (Y'Cb'Cr'420).

[0064] And the encoding unit (27) can compress and encode the YCbCr format image (Y'Cb'Cr'420) input from the downsampling unit (24).

[0065] In order to encode YCbCr format video, the encoding unit (27) can use video codec standard technologies such as H.264 / AVC, H.265 / HEVC, H.266 / VVC, VP8, VP9, ​​AV1, etc., but the present invention is not limited thereto.

[0066] The bitstream transmitted from the media server (20) to the user terminal (30) may include conversion information used to convert an RGB format image into a YCbCr format image, together with a YCbCr format image encoded in the encoding unit (27).

[0067] Here, the conversion information may be included in the bitstream as HLS (High-Level Syntax) so that it can be used for video playback, and in this case, it may be inserted into at least one of VUI (Video Usability Information), SEI (Supplemental Enhancement Information), VPS (Video Parameter Set), SPS (Sequence Parameter Set), PPS (Picture Parameter Set), and PH (Picture Header).

[0068] And the conversion information may include color specification information indicating a color range that can be expressed by the RGB format of the game image, color conversion information indicating a conversion method used in the color conversion unit (22) for conversion from the RGB format to the YCbCr format, conversion function information indicating a conversion function used in the optical-electrical conversion unit (21) for optical-electrical conversion of the RGB format image, etc.

[0069] Meanwhile, game footage can have different characteristics depending on the camera position (camera angle, point-of-view, or perspective), and the camera position can be divided into first-person view, third-person view, top (or top-down) view, side view, quarter (or isometric) view, shoulder (or back) view, etc.

[0070] The characteristics of the image depending on the camera position may be as follows:

[0071] - The first-person view is composed of a screen that the camera sees from a first-person perspective, and provides a better sense of presence and immersion than other viewpoints.

[0072] - Third person point of view can mean quarter or isometric point of view

[0073] - Top view is a viewpoint where the camera is mounted on the ceiling and viewed from the sky, showing only the top of the entire screen and the character.

[0074] - The side view is mainly used in 2D games, and is good for expressing the character's action because the camera is on the side.

[0075] - The quarter viewpoint is a view looking down from a diagonal line above the sky, giving a good sense of space.

[0076] - Shoulder viewpoint is when the camera is behind the character, providing a full view of the screen and the character.

[0077] Depending on the camera viewpoint, information such as the direction of camera movement, the degree and angle of movement of the video, and the location and size of the background and objects within the video can be inferred.

[0078] For example, in the top view, there is a single background that scrolls vertically, and small objects can exist on a large background screen. In the side view, the background mainly moves left and right, and small objects can exist on a large background screen. In the quarter view, there are more diagonal edges than horizontal and vertical edges due to the diagonal position of the camera, and small objects can exist on a large background screen.

[0079] Meanwhile, more than one camera viewpoint can be mixed. For example, the Back viewpoint can be mixed with the Quarter viewpoint or the Top viewpoint depending on the camera adjustment. However, there are viewpoints, such as the Top viewpoint and the Quarter viewpoint, that cannot be mixed.

[0080] As mentioned above, the characteristics of the video may differ depending on the camera viewpoint of the game video, and there may also be differences in the method of switching the background screen or the method of operating input devices such as the mouse.

[0081] Camera viewpoint information as described above may be included in the bitstream as HLS (High-Level Syntax) and inserted into at least one of VUI (Video Usability Information), SEI (Supplemental Enhancement Information), VPS (Video Parameter Set), SPS (Sequence Parameter Set), PPS (Picture Parameter Set), and PH (Picture Header).

[0082] Additionally, if there is a separate flag to indicate that the content is gaming content, camera viewpoint information may be added in the form of an index when the flag value is "on." The flag indicating that it is gaming content may also be included in the bitstream as HLS and inserted into at least one of VUI, SEI, VPS, SPS, PPS, and PH.

[0083] In the above, embodiments of the present invention have been described by taking as an example the encoding method according to the present invention being performed in a media server (20), but the present invention is not limited thereto, and at least some steps of the encoding method may be performed in a separate image processing device.

[0084] According to one embodiment of the present invention, an encoded YCbCr format image transmitted from a media server (20) can be decoded at a user terminal (30) and then converted into an RGB format image and played back, and the decoding method performed at the user terminal (30) can be a reverse process of the encoding method described above.

[0085] Figure 4 is a flowchart illustrating an image decoding method according to an embodiment of the present invention.

[0086] Referring to FIG. 4, a bitstream including information about an image in an encoded YCbCr format and conversion information used to convert an image in an RGB format into an image in a YCbCr format is received (step S400).

[0087] Next, the encoded YCbCr format image is decoded using information parsed from the bitstream received at step S400 (step 410), and the decoded YCbCr format image is converted into an RGB format image using conversion information (step S420).

[0088] Here, the conversion information may include color specification information for the RGB format (e.g., the RGB format of the game image generated by the game server (10)) and conversion information for the color conversion method used in the encoding process (e.g., the color conversion process of the media server (20)).

[0089] Additionally, in step S420, electro-optical conversion may be performed on the RGB format image, in which case, information on the optical-electrical conversion function used in the encoding process (e.g., the optical-electrical conversion process of the media server (20)) may be included in the conversion information.

[0090] Therefore, the conversion information included together with the YCbCr format image in the bitstream received at step S200 may include color specification information, color conversion information, and conversion function information, but the present invention is not limited thereto, and may include various pieces of information related to conversion between the RGB format and the YCbCr format.

[0091] FIG. 5 is a block diagram illustrating an example of a configuration of a user terminal that performs an image decoding method, and shows the configuration of a user terminal (30) for performing the decoding method described with reference to FIG. 4.

[0092] Referring to FIG. 5, the user terminal (30) may include a decoding unit (31), an up-sampling unit (35), an inverse quantization unit (36), a color conversion unit (37), and an electro-optical conversion unit (38).

[0093] The bitstream received from the media server (20) to the user terminal (30) may include information about an encoded YCbCr format image and conversion information used to convert an RGB format image into a YCbCr format image in the media server (20).

[0094] For example, the conversion information may be included in the bitstream as HLS (High-Level Syntax) in at least one of VUI (Video Usability Information), SEI (Supplemental Enhancement Information), VPS (Video Parameter Set), SPS (Sequence Parameter Set), PPS (Picture Parameter Set), and PH (Picture Header).

[0095] And the conversion information may include color specification information for the RGB format, color conversion information indicating a color conversion method, conversion function information indicating a conversion function used for optical-electrical conversion, etc.

[0096] The decoding unit (31) can decode an encoded YCbCr format image using information included in the bitstream.

[0097] In order to decode YCbCr format video, the decoding unit (31) can use video codec standard technologies such as H.264 / AVC, H.265 / HEVC, H.266 / VVC, VP8, VP9, ​​AV1, etc., but the present invention is not limited thereto.

[0098] Meanwhile, the bitstream received from the media server (20) to the user terminal (30) may additionally include camera viewpoint information as described above, and accordingly, the decoding unit (31) may use the differences according to the camera viewpoint to decode the video or use the decoding as metadata.

[0099] The up-sampling unit (35) performs chroma up-sampling on the decoded YCbCr format image (Y'Cb'Cr'420) output from the decoding unit (31) to change the YUV420 format to the YUV444 format, thereby outputting a YCbCr format image (Y'Cb'Cr'(integer)).

[0100] Next, the inverse quantization unit (36) can perform inverse quantization on the YCbCr format image (Y'Cb'Cr'(integer)) to output a YCbCr format image (Y'Cb'Cr'(float)) having a real value.

[0101] The color conversion unit (37) can convert a YCbCr format image (Y'Cb'Cr'(float)) into an RGB format image (R'G'B'(float)).

[0102] Here, the color conversion unit (37) can convert a YCbCr format image (Y'Cb'Cr'(float)) into an RGB format image (R'G'B'(float)) using the same color conversion method used in the media server (20) by using color conversion information among the conversion information included in the bitstream.

[0103] For example, when Constant Luminance (CL) is used for color conversion in a media server (20), the color conversion unit (37) converts a YCbCr format image (Y'Cb'Cr'(float)) into an RGB format image (R'G'B'(float)) using the Constant Luminance (CL) method, and when Non-Constant Luminance (NCL) is used, the color conversion unit (37) can convert a YCbCr format image (Y'Cb'Cr'(float)) into an RGB format image (R'G'B'(float)) using the same Non-Constant Luminance (NCL) method.

[0104] In addition, the color conversion unit (37) can convert the RGB format image (R'G'B'(float)) according to the standard having the same color range as the game image generated by the game server (10) by using the color standard information among the conversion information included in the bitstream.

[0105] Accordingly, in the video played on the user terminal (30), the intended color can be expressed as is when producing the game video.

[0106] For example, the color conversion unit (37) can convert a YCbCr image into an RGB image according to the BT.709-6 standard when the RGB color standard of the game image generated from the game server (10) is BT.709-6, convert the YCbCr image into an RGB image according to the P3 standard when it is P3, convert the YCbCr image into an RGB image according to the BT.2020 standard when it is BT.2020, and convert the YCbCr image into an RGB format image according to the BT.2020-2 standard when it is BT.2020-2.

[0107] And the electro-optical conversion unit (38) can perform electro-optical conversion on the RGB format image (R'G'B'(float)) output from the color conversion unit (37), thereby converting the electrical signal (R'G'B'(float)) into an optical signal (RGB linear-light).

[0108] Here, the electro-optical conversion unit (38) can output an RGB format image (RGB linear-light) converted into an optical signal by applying an inverse function (inverse OETF or EOTF) corresponding to the conversion function used in the media server (20) using conversion function information among the conversion information included in the bitstream.

[0109] For example, the electro-optical conversion unit (38) may convert an electrical signal (R'G'B'(float)) into an optical signal (RGB linear-light) using the HLG conversion function when the inverse HLG conversion function is used for the optical-electrical conversion in the media server (20), and may convert an electrical signal (R'G'B'(float)) into an optical signal (RGB linear-light) using the PQ conversion function when the inverse PQ conversion function is used.

[0110] Meanwhile, the conversion process described above can be performed by linking the conversion information and the bit-depth information per pixel.

[0111] For example, the electro-optical conversion unit (38) can perform electro-optical conversion using at least one of color specification information, color conversion information, and bit depth information in addition to conversion function information.

[0112] In addition, the color conversion unit (37) may perform color conversion using at least one of conversion function information, color specification information, and bit depth information in addition to color conversion information.

[0113] The RGB format image output from the electro-optical converter (38) can be played through the display of the user terminal (30).

[0114] As described above, since the conversion information used to convert an RGB format image into a YCbCr format image in the media server (20) is transmitted to the user terminal (30) via a bitstream, the image played in the user terminal (30) can be restored to have characteristics such as color saturation that are as similar as possible to the game image generated in the game server (10).

[0115] In the above, embodiments of the present invention have been described by taking as an example the decryption method according to the present invention being performed in a user terminal (30), but the present invention is not limited thereto, and at least some steps of the decryption method may be performed in a separate image processing device.

[0116]

[0117] Hereinafter, embodiments of encoding and decoding methods and devices applicable to the present invention will be described with reference to FIGS. 6 to 10.

[0118] FIG. 6 is a diagram illustrating the configuration and operation of a video encoder for encoding an image, and the illustrated video encoder (100) may be configured to include a transformation unit (110), a quantization unit (120), an inverse quantization unit (130), an inverse transformation unit (140), a filtering unit (150), a prediction unit (160), a DPB (Decoded Picture Buffer, 170), and an entropy coding unit (180).

[0119] Referring to FIG. 6, the conversion unit (110) converts the residual signal, which is the difference between the input video signal and the prediction signal generated by the prediction unit (160), to obtain a conversion coefficient value.

[0120] For example, the Discrete Cosine Transform (DCT), the Discrete Sine Transform (DST), or the Wavelet Transform can be used.

[0121] The transform kernel used for transforming the residual block may be a transform kernel having separable vertical and horizontal transform properties. In this case, the transform for the residual block may be performed separately as vertical and horizontal transforms. For example, the encoder may perform a vertical transform by applying the transform kernel in the vertical direction of the residual block. Alternatively, the encoder may perform a horizontal transform by applying the transform kernel in the horizontal direction of the residual block.

[0122] Meanwhile, the transform kernel may be used as a term referring to a set of parameters used for transforming the residual signal, such as a transform matrix, a transform array, a transform function, or a transform, and may be any one of a plurality of available kernels, and transform kernels based on different transform types may be used for each of the vertical transform and the horizontal transform.

[0123] The transformation coefficients may be distributed such that higher coefficients are distributed toward the upper left corner of the block, and coefficients closer to '0' are distributed toward the lower right corner of the block. Furthermore, as the current block size increases, there is a possibility that many '0' coefficients exist in the lower right area. To reduce the transformation complexity of large blocks, only the upper left area can be left, and the remaining areas can be reset to '0'.

[0124] Additionally, error signals may exist only in some regions of a coding block, in which case the conversion process may be performed only on some arbitrary regions. For example, in a block of size 2Nx2N, error signals may exist only in the first 2NxN block, in which case the conversion process may be performed only on the first 2NxN block, but the conversion process may not be performed on the second 2NxN block and may not be encoded or decoded.

[0125] The encoder may perform additional transformations before the transform coefficients are quantized. The transformation method described above may be referred to as a primary transform, and the additional transformation may be referred to as a secondary transform.

[0126] The secondary transform can be optional for each residual block, for example, the encoder can improve coding efficiency by performing the secondary transform for areas where it is difficult to concentrate energy in the low-frequency region using only the primary transform.

[0127] Specifically, a secondary transformation may be additionally performed on blocks in which residual values ​​appear significantly in directions other than the horizontal or vertical direction of the residual block, and the secondary transformation may not be performed separately into vertical transformation and horizontal transformation, unlike the primary transformation. The secondary transformation as described above may be referred to as a low frequency non-separable transform (LFNST).

[0128] The quantization unit (120) quantizes the transformation coefficient value output from the transformation unit (110).

[0129] In order to increase coding efficiency, a method is used in which, instead of coding the picture signal as is, a picture is predicted using an already coded area through a prediction unit (160), and a restored picture is obtained by adding a residual value between the original picture and the predicted picture to the predicted picture.

[0130] To avoid mismatches between the encoder and decoder, when performing prediction in the encoder, information available in the decoder must also be used, and for this purpose, the encoder can perform a process of reconstructing the current block that it has encoded.

[0131] The inverse quantization unit (130) inversely quantizes the transform coefficient values, and the inverse transformation unit (140) restores the residual values ​​using the inverse quantized transform coefficient values.

[0132] The filtering unit (150) performs filtering operations using a deblocking filter, a sample adaptive offset (SAO), an adaptive loop filter (ALF), etc. to improve the quality of the restored picture and enhance encoding efficiency.

[0133] A deblocking filter is a filter for removing distortion within a block generated at the boundary between blocks in a restored picture. The encoder can determine whether to apply a deblocking filter to a boundary based on the distribution of pixels included in several columns or rows based on an arbitrary boundary within a block.

[0134] When a deblocking filter is applied, the filtering unit (150) can apply a long filter, a strong filter, or a weak filter depending on the deblocking filtering strength, and can process horizontal filtering and vertical filtering in parallel.

[0135] Sample Adaptive Offset (SAO) can be used to correct the offset from the original image on a pixel-by-pixel basis for a residual block to which a deblocking filter is applied. In order to correct the offset for a specific picture, the filtering unit (150) can use a method (Band Offset) that divides the pixels included in the image into a certain number of regions, determines the regions to perform offset correction, and applies the offset to the regions. In addition, the filtering unit (150) can use a method (Edge Offset) that applies the offset by considering the edge information of each pixel.

[0136] Adaptive Loop Filtering (ALF) is a method that divides pixels in an image into predetermined groups, determines a filter to be applied to each group, and performs differential filtering for each group. Information regarding whether to apply an adaptive loop filter can be signaled on a coding unit basis, and the shape and filter coefficients of the ALF filter to be applied can vary depending on the block. Furthermore, the same adaptive loop filter can be applied regardless of the characteristics of the target block.

[0137] The filtered picture can be stored in DPB (170) to be used as a reference picture.

[0138] The prediction unit (160) includes an intra / IBC prediction unit (161) and an inter prediction unit (165). The intra / IBC prediction unit (161) performs intra prediction and IBC prediction within the current picture, and the inter prediction unit (165) performs inter prediction to predict the current picture using a reference picture stored in the DPB (170).

[0139] The intra / IBC prediction unit (161) performs intra prediction from restored areas within the current picture and transmits intra encoding information to the entropy coding unit (180). Here, the intra encoding information may include at least one of an intra prediction mode, an MPM (Most Probable Mode) flag, an MPM index, and information about a reference sample.

[0140] Additionally, the intra / IBC prediction unit (161) can perform IBC (Intra BLock Copy) prediction from restored samples within the current picture and transmit IBC encoding information to the entropy coding unit (180).

[0141] In this case, the intra / IBC prediction unit (161) obtains a block vector value indicating a reference area used for prediction of the current area by referring to a specific area within the current picture, and can perform IBC prediction using the obtained block vector value.

[0142] The intra / IBC prediction unit (161) can transmit IBC encoding information including at least one of the size information of the reference area and block vector information (index information for block vector prediction of the current block within the motion candidate list, block vector difference information) to the entropy coding unit (160).

[0143] The inter prediction unit (165) refers to a specific area of ​​the restored reference picture to find the part most similar to the current area, obtains a motion vector value which is the distance between the areas, and transmits motion information (reference direction indication information (L0 prediction, L1 prediction, bidirectional prediction), reference picture index, motion vector information, etc.) for the obtained reference area to the entropy coding unit (180).

[0144] Additionally, the inter prediction unit (165) performs motion compensation using motion information to generate a prediction block for the current block, and transmits inter encoding information including motion information for the reference area to the entropy coding unit (180).

[0145] Meanwhile, the quantized transform coefficients in the form of a two-dimensional array can be rearranged into a one-dimensional array for entropy coding.

[0146] The method of scanning the quantized transform coefficients can be determined according to the size of the transform block and the prediction mode within the screen, and diagonal, vertical, and horizontal scans can be applied, and the scan information can be signaled on a block-by-block basis or derived from the decoder according to a set rule.

[0147] The entropy coding unit (180) generates a bitstream by entropy coding information representing quantized transform coefficients, intra-coding information, and inter-coding information, and for this purpose, a variable length coding (VLC) method and an arithmetic coding method can be used.

[0148] Variable-length coding (VLC) converts input symbols into a series of codewords, each of which can be of variable length. For example, frequently occurring symbols can be represented by shorter codewords, while less frequently occurring symbols can be represented by longer codewords.

[0149] As a variable length coding method, a context-based adaptive variable length coding (CAVLC) method can be used.

[0150] Arithmetic coding converts consecutive data symbols into a single prime number using the probability distribution of each data symbol, thereby obtaining the optimal prime number bits required to express each symbol.

[0151] As an arithmetic coding method, the context-based adaptive binary arithmetic code (CABAC) method can be used.

[0152] CABAC is a binary arithmetic encoding method that utilizes multiple context models generated based on experimentally derived probabilities. The context models can also be referred to as context models. First, if the symbols are not in binary form, the encoder binarizes each symbol using methods such as exp-Golomb. The binarized 0s and 1s can be represented as bins.

[0153] The CABAC initialization process is divided into context initialization and arithmetic coding initialization. Context initialization initializes the occurrence probability of each symbol, determined by the symbol type, quantization parameter (QP), and slice type (I, P, B). A context model with this initialization information can use probability-based values ​​obtained through experiments.

[0154] The context model provides the probability of occurrence of the Least Probable Symbol (LPS) or Most Probable Symbol (MPS) for the symbol currently being coded, and information (valMPS) on which empty value between 0 and 1 corresponds to the MPS.

[0155] One of several context models is selected through a context index (ctxIdx), and the context index can be derived through information of the block currently to be encoded or information of surrounding blocks.

[0156] Initialization for binary arithmetic coding is performed based on the probability model selected from the context model. Binary arithmetic coding is performed by dividing the data into probability intervals based on the occurrence probabilities of 0 and 1, and then encoding the probability interval corresponding to the bin to be processed becomes the entire probability interval for the bin to be processed next.

[0157] The location information within the probability interval processed for the last bin is output. However, since the probability interval cannot be infinitely divided, if it shrinks to a certain size, a renormalization process is performed to expand the probability interval and output the corresponding location information. Furthermore, after each bin is processed, a probability update process can be performed, which sets a new probability for the next bin to be processed based on the information from the processed bin.

[0158] As described above, the generated bitstream is encapsulated into NAL (Network Abstraction Layer) units as basic units.

[0159] NAL units are divided into VCL (Video Coding Layer) NAL units that contain video data and non-VCL NAL units that contain parameter information for decoding video data, and there may be various types of VCL or non-VCL NAL units.

[0160] A NAL unit consists of NAL header information and data, a Raw Byte Sequence Payload (RBSP). The NAL header information includes summary information about the RBSP. The RBSP of a VCL NAL unit contains an integer number of encoded coding tree units.

[0161] In order to decode a bitstream in a decoder, the bitstream must first be divided into NAL units, and then each divided NAL unit must be decoded. Meanwhile, the information required for decoding the bitstream can be transmitted as part of a picture parameter set (PPS), a sequence parameter set (SPS), a video parameter set (VPS), etc.

[0162] Meanwhile, the configuration and operation of the encoder described with reference to FIG. 6 are according to one embodiment of the present invention, and some configurations may be omitted or added as needed.

[0163] Additionally, a single picture may be encoded by dividing it into sub-pictures, slices, tiles, etc. A sub-picture may include one or more slices or tiles. When a single picture is encoded by dividing it into multiple slices or tiles, all slices or tiles within the picture must be decoded before it can be displayed on the screen.

[0164] When a single picture is encoded into multiple subpictures, only any subpicture can be decoded and displayed on the screen. A slice can contain multiple tiles or subpictures, and a tile can contain multiple subpictures or slices.

[0165] Subpictures, slices, and tiles can be encoded or decoded independently of each other, which is effective for parallel processing and processing speed improvement, but the amount of bits may increase because the encoded information of adjacent subpictures, slices, and tiles cannot be used.

[0166] And subpictures, slices, and tiles can be encoded by being divided into multiple coding tree units (CTUs).

[0167] A coding tree unit can be composed of a luminance coding tree block (CTB) of size 128x128 and two chrominance coding tree blocks of size 64x64.

[0168] A single coding tree unit may be undivided and constitute a single coding unit (CU) itself, or may be divided into multiple coding units as illustrated in Fig. 7. A coding unit may be composed of a luminance coding block (CB) and two chrominance coding blocks.

[0169] A coding unit may consist of one transform unit (TU), or may be split into multiple transform units. A transform unit may consist of a luminance transform block (TB) and two chrominance transform blocks.

[0170] Here, a coding unit represents a basic unit for processing a picture in the process of prediction, transformation, quantization, entropy coding, and decoding, and the size and shape of a coding unit within a picture may not be constant.

[0171] A coding unit may have a square or non-square shape, and a rectangular coding unit may include a vertical coding unit whose height is greater than its width and a horizontal coding unit whose width is greater than its height.

[0172] The coding tree unit is first partitioned into a Quad Tree (QT) structure, such that a single node of size 2NX2N can be partitioned into four nodes of size NXN. Furthermore, the Quad Tree partitioning can be performed recursively, and not all nodes need to be partitioned to the same depth.

[0173] Leaf nodes of a quadtree can be further partitioned into a multi-type tree (MTT) structure. For example, in a multi-type tree structure, a single node can be partitioned into a binary or ternary tree structure with horizontal or vertical partitioning. Accordingly, a multi-type tree structure can have four partitioning structures: vertical binary partitioning, horizontal binary partitioning, vertical ternary partitioning, and horizontal ternary partitioning.

[0174] In each tree structure, both the width and height of a node can have a power of 2. For example, in a binary tree (BT) structure, a node of size 2NX2N can be split into two NX2N nodes by vertical binary splitting, and into two 2NXN nodes by horizontal binary splitting.

[0175] Also, in the Ternary Tree (TT) structure, a node of size 2NX2N can be split into nodes of size (N / 2)X2N, NX2N, and (N / 2)X2N by vertical ternary splitting, and into nodes of size 2NX(N / 2), 2NXN, and 2NX(N / 2) by horizontal ternary splitting. This multi-type tree splitting can be performed recursively.

[0176] A leaf node of a multi-type tree can be a coding unit. If the coding unit is no larger than the maximum transformation length, the coding unit can be used as a unit for prediction and transformation without further splitting. On the other hand, if the width or height of the coding unit is larger than the maximum transformation length, the coding unit can be split into multiple transformation units without explicit signaling regarding the splitting.

[0177] The tree partitioning structure as described above may have the same shape (Single Tree) for the luminance block and the chrominance block, or different shapes (Dual Tree) for the luminance block and the chrominance block.

[0178] Meanwhile, the block division from the coding tree unit (CTU) to the coding unit (CU) as described above can be performed through a rate-distortion optimization (RDO) process, whereby the division structure with the smallest rate-distortion cost (RD cost) value within the allowable size and depth conditions is selected to determine the final coding unit.

[0179] Rate-distortion cost (RD cost) is a cost value calculated by considering compression efficiency (rate) and image quality degradation (distortion) when a specific encoding mode is applied to a block. In order to calculate this, compression rate and image quality degradation values ​​must be obtained through an abbreviated encoding procedure using a specific encoding mode, which can significantly increase the complexity of the encoder.

[0180] For example, if the CTU size is 128x128, the encoder can perform an abbreviated encoding process for each of a plurality of (n) encoding modes for a 128x128 block, calculate n RD costs, and set the encoding mode corresponding to the smallest value among the n RD costs as the temporary encoding mode of the 128x128 block.

[0181] In the following, for convenience of explanation, it is assumed that the CTU size is 64x64, but the CTU size may be a value larger than 64x64, for example, 128x128.

[0182] Referring to Fig. 8, the encoding mode for the 64x64 block is set to “6” and the RD cost is set to “61”. After that, the 64x64 block is divided into four 32x32 blocks, and n RD costs for each of n encoding modes are calculated for each 32x32 block, and then a temporary encoding mode is set for each of the 32x32 blocks.

[0183] The encoding mode of the first 32x32 block is “2” and the RD cost is “20”, the encoding mode of the second 32x32 block is “13” and the RD cost is “10”, the encoding mode of the third 32x32 block is “5” and the RD cost is “15”, and the encoding mode of the fourth 32x32 block is “17” and the RD cost is “15”.

[0184] At this time, the sum of the RD costs for four 32x32 blocks is compared with the RD cost value of the 64x64 block. If the RD cost value of the 64x64 block is greater than the sum of the RD costs of the 32x32 blocks, division into 32x32 blocks is determined. If the RD cost value of the 64x64 block is less than the sum of the RD costs of the 32x32 blocks, the 64x64 block can be used as the final CU without block division.

[0185] In the case illustrated in Fig. 8, since the sum of the RD costs of the four 32x32 blocks, “50,” is less than the RD cost of the 64x64 block, “61,” division into 32x32 blocks can be determined.

[0186] Next, the 32x32 block is again divided into four 16x16 blocks, and n RD costs for each of n encoding modes are calculated for each 16x16 block, and then a temporary encoding mode is set for each of the 16x16 blocks, and whether to divide into 16x16 blocks can be determined based on the result of comparing the RD costs of the 32x32 block and the 16x16 blocks.

[0187] To determine the final coding unit (CU), the process described above is repeated recursively.

[0188] As described above, the decision on whether to split a block and the decision on the encoding mode are interrelated to determine the final coding unit, and this process can significantly increase the complexity of the encoder.

[0189] Referring to an example of the final block division result illustrated in FIG. 9, for a coding tree unit (CTU) of size 64x64, the sum of the RD costs of four 32x32 blocks is less than the RD cost of the coding tree unit, and among them, the “e” block has an RD cost less than the sum of the RD costs of the four 16x16 blocks into which it is divided, so it is determined as the final coding unit of size 32x32, and the remaining blocks are determined as the final coding units of size 32x32.

[0190] FIG. 10 is a diagram illustrating the configuration and operation of a video decoder for decoding an image, and the illustrated video decoder (200) may be configured to include an entropy decoding unit (210), an inverse quantization unit (220), an inverse transformation unit (230), a filtering unit (240), a prediction unit (260), and a DPB (Decoded Picture Buffer, 270).

[0191] Referring to FIG. 10, the entropy decoding unit (210) entropy decodes the bitstream to extract transform coefficient information, intra-coding information, inter-coding information, etc. for each region.

[0192] For example, the entropy decoding unit (210) can obtain a binary code for transform coefficient information of a specific area from a bitstream, and can obtain a quantized transform coefficient by de-binarizing the binary code.

[0193] The inverse quantization unit (220) inversely quantizes the quantized transform coefficients, and the inverse transformation unit (230) restores the residual value using the inverse quantized transform coefficients.

[0194] Meanwhile, the residual value obtained from the inverse transformation unit (230) is combined with the predicted value obtained from the prediction unit (260) to restore the original pixel value.

[0195] The filtering unit (240) performs a filtering operation using a deblocking filter, a sample adaptive offset, an adaptive loop filter, etc. to improve the image quality of the restored picture, and the filtered picture can be output or stored in the DPB (270) to be used as a reference picture for the next picture.

[0196] The prediction unit (260) includes an intra / IBC prediction unit (261) and an inter prediction unit (265), and generates a prediction picture by utilizing the encoding type decoded through the entropy decoding unit (210), the transform coefficient for each region, and the intra / inter encoding information.

[0197] To restore the current block for which decoding is performed, the decoded region of the current picture or other pictures containing the current block can be used. A picture (or tile / slice) that performs intra prediction or intra BC prediction using only the current picture for restoration is called an intra picture or I picture (or tile / slice), and a picture (or tile / slice) that can perform all of intra prediction, inter prediction, and intra BC prediction is called an inter picture (or tile / slice).

[0198] Meanwhile, a picture (or tile / slice) that uses at most one motion vector and reference picture index to predict sample values ​​of each block among inter-pictures (or tiles / slices) is called a predictive picture or P-picture (or tile / slice), and a picture (or tile / slice) that uses at most two motion vectors and reference picture indices is called a bi-predictive picture or B-picture (or tile / slice).

[0199] That is, a P picture (or tile / slice) uses at most one motion information set to predict each block, and a B picture (or tile / slice) uses at most two motion information sets to predict each block. Here, a motion information set may include one or more motion vectors and one reference picture index.

[0200] The intra / IBC prediction unit (261) generates a prediction block using intra encoding information and restored samples within the current picture, and the intra encoding information may include at least one of an intra prediction mode, an MPM (Most Probable Mode) flag, and an MPM index.

[0201] The intra / IBC prediction unit (261) can predict sample values ​​of the current block using restored samples located on the left and / or upper side of the current block as reference samples.

[0202] For example, the reference samples may be samples adjacent to the left boundary of the current block and / or samples adjacent to the upper boundary, and may be samples located on a line within a preset distance from the left boundary of the current block among samples of neighboring blocks of the current block and / or samples located on a line within a preset distance from the upper boundary of the current block. In this case, the neighboring blocks of the current block may include at least one of a left (L) block, an upper (A) block, a below left (BL) block, an above right (AR) block, or an above left (AL) block adjacent to the current block.

[0203] Additionally, the intra / IBC prediction unit (261) can restore the current region by referring to a specific region including restored samples within the current picture. To this end, the intra / IBC prediction unit (261) can perform IBC prediction using IBC encoding information obtained from the entropy decoding unit (210), and the IBC encoding information can include block vector information.

[0204] The inter prediction unit (265) generates a prediction block using the reference picture and inter encoding information stored in the DPB (270), and the inter encoding information may include a set of motion information (reference picture index, motion vector information, etc.) of the current block for the reference block.

[0205] Meanwhile, inter prediction can include L0 prediction, L1 prediction, and bi-prediction.

[0206] L0 prediction refers to prediction using a single reference picture included in the L0 picture list, and L1 prediction refers to prediction using a single reference picture included in the L1 picture list. For this, a set of motion information (e.g., motion vector and reference picture index) may be required.

[0207] In the biprediction method, up to two reference regions can be used. These two reference regions may exist in the same reference picture or in different pictures. Accordingly, in the two sets of motion information used in the biprediction method, two motion vectors may correspond to the same reference picture index or to different reference picture indices.

[0208] At this time, the reference pictures are pictures that are located temporally before or after the current picture, and may be completed pictures that have already been restored, and the two reference areas used in the pair prediction method may be areas selected from the L0 picture list and the L1 picture list, respectively.

[0209] The inter prediction unit (265) can obtain a reference block of the current block using a motion vector and a reference picture index, and the reference block exists in a reference picture corresponding to the reference picture index.

[0210] Additionally, the sample values ​​of a block specified by a motion vector or their interpolated values ​​can be used as a predictor of the current block. For motion prediction with subpel-level pixel accuracy, an 8-tap interpolation filter can be used for the luminance signal, and a 4-tap interpolation filter can be used for the chrominance signal.

[0211] Meanwhile, the configuration and operation of the decoder described with reference to FIG. 10 are according to one embodiment of the present invention, and some configurations may be omitted or added as needed, and the decoder may decode an image by performing the reverse process of the encoding method of the encoder described above.

[0212]

[0213] As described above, the game image generated from the game server (10) is computer graphic processed data, and there is a difference in pixel value changes compared to the real image captured by the camera.

[0214] The graph shown in (a) of Fig. 11 shows the pixel value distribution for a real-life image, and like an analog signal, the pixel values ​​are continuous. In contrast, the graph shown in (b) of Fig. 11 shows the pixel value distribution for a game image, and it can be seen that the pixel value distribution of a graphically processed game image has the characteristic of being concentrated on a few sample pixel values, like a digital signal.

[0215] This means that while the pixel values ​​of real-life images with added noise naturally have various values, in the case of randomly created game images, the distribution shows a simple shape as the values ​​are biased towards specific pixel values.

[0216] More specifically, in the case of real-world images captured by a camera, noise can occur due to increased dark current caused by loss of charge or sensor heat generation during the process of converting the charge corresponding to the brightness of light into an analog electrical signal and then into a digital signal through a CCD or CMOS image sensor. Furthermore, even subtle changes in light intensity or slight variations in air flow can cause differences in the pixel values ​​of the image.

[0217] Accordingly, in the case of real-world images, the probability that pixel values ​​at the same location in two consecutive images on the time axis, for example, the pixel value at location (10,10) in the (t)-th image and the pixel value at location (10,10) in the (t+1)-th image, are the same may be very low.

[0218] In contrast, in the case of game images generated through computer graphics processing, if an object in the image has not moved, there is a very high probability that the pixel values ​​at the same location in two consecutive images on the time axis, for example, the pixel values ​​corresponding to the object location in the (t)th image and the pixel values ​​corresponding to the object location in the (t+1)th image, are all identical.

[0219] Figure 12 is a diagram illustrating the characteristics according to the movement of an object in a game video. Figures 12 (a) and 12 (b) show the (t)th image and the (t+1)th image, respectively, in the time axis of the game video.

[0220] Referring to FIG. 12, the pixel values ​​of the (t)th image shown in (a) of FIG. 12 and the (t+1)th image shown in (b) of FIG. 12 are very likely to be different only in the areas where the object indicated by the square box in (b) of FIG. 12 moves, and are identical in the remaining areas.

[0221] Meanwhile, the game server (10) receives a user action event, renders the game screen, visualizes the screen, and transmits it to the media server (20). The media server (20) then compresses and streams the video received from the game server (10).

[0222] In this case, the game server (10) can obtain information about objects moving within the screen during the process of rendering the game screen and visualizing the screen. For example, the game server (10) may have information about areas where objects, indicated by rectangular boxes, move in an image such as that illustrated in (b) of FIG. 12.

[0223] This is because the game image is rendered through the game engine, so that the game server (10) can obtain information about the screen composition in real time for each frame, and through this, the game server (10) can obtain information about the position of an object moving in a specific frame or background transition information, and further information about an area where a change in pixel value occurred in the frame.

[0224] According to another embodiment of the present invention, when encoding an image generated by graphic processing, such as a game image, object position information or background transition information provided from a game server can be applied to an encoding process, such as block division or encoding mode determination, thereby reducing the computational complexity of the encoder.

[0225] FIG. 13 is a flowchart illustrating a video encoding method according to another embodiment of the present invention. Hereinafter, the method is described as being performed by a video encoder included in a media server (20), but the present invention is not limited thereto.

[0226] Referring to FIG. 13, when object position information is received from a game server (10) together with a game image (step S1300), the encoder divides the picture included in the image to be encoded into first units of a first size (step S1310).

[0227] For example, the first unit may be a coding tree unit (CTU) as described above, and the first size may be 128x128, but the present invention is not limited thereto, and the first size may be included in the bitstream as header information and transmitted to the decoder.

[0228] The encoder uses the object location information received in step S1300 to check whether an object is included within the area of ​​the first unit segmented in step S1310 (step S1320).

[0229] At step S1320, the encoder can determine whether there is a moving object or a newly appeared new object within the area of ​​the first unit based on the object position information received from the game server (10).

[0230] Afterwards, based on the verification result, the encoder determines whether to split the block for the first unit and at least one of the encoding modes (step S1330).

[0231] As a first embodiment of a method for determining whether to split a block and an encoding mode in step S1330, the encoder may configure the first unit as one coding unit (CU) if no object is included in the area of ​​the first unit, and determine the inter prediction mode for the first unit to be skip mode.

[0232] As described above, a coding tree unit (CTU) can be split into multiple coding units (CUs), and information about the minimum size (min CU) of a coding unit (CU) can be included as header information in the bitstream and transmitted to the decoder.

[0233] Meanwhile, for inter prediction, one of skip mode, merge mode, and MVP (Motion Vector Prediction) mode can be applied to derive a prediction sample for the current block.

[0234] In skip mode and merge mode, motion information of surrounding blocks can be used as motion information of the current block, and in skip mode, unlike merge mode, the residual value, which is the difference between the predicted sample and the original sample, is not transmitted.

[0235] In MVP mode, the motion vector of the surrounding block is used as a motion vector predictor, and the MVD value, which is the difference value between the motion vector prediction value obtained from the surrounding block and the final motion vector value, is transmitted.

[0236] Meanwhile, if an object is included within the area of ​​the first unit, the encoder can determine whether to split the block for the first unit and at least one of the encoding modes based on the rate-distortion cost (RD cost) through the rate-distortion optimization (RDO) process described above.

[0237] Figure 14 illustrates a coding tree unit (CTU). In the following, an example is given of a CTU size of 64x64, but the CTU size may be a value larger than 64x64, for example, 128x128.

[0238] If it is determined that no moving object is included in the area corresponding to the coding tree unit (CTU) based on the object position information received from the game server (10), the block division process and encoding mode determination for the coding tree unit (CTU) are skipped, the coding tree unit (CTU) is set as the final coding unit (CU), and the encoding mode can be determined as the skip mode.

[0239] Meanwhile, as illustrated in FIG. 15, if a moving object (Ob) with movement is included within an area corresponding to a coding tree unit (CTU), the block division and encoding mode can be determined through an RDO process for the corresponding coding tree unit (CTU).

[0240] As another example, if an object is included within the area of ​​the first unit, the encoder may divide the first unit into second units having a second size smaller than the first size, and use object location information received from the game server (10) to determine whether an object is included within the area of ​​the divided second unit.

[0241] At this time, if an object is not included in the area of ​​the second unit, the encoder may determine the second unit as one coding unit and determine the inter prediction mode for the second unit as the merge skip mode.

[0242] Meanwhile, if an object is included in the area of ​​the second unit, the encoder can divide the second unit into third units having a third size smaller than the second size, and use the object location information received from the game server (10) to check again whether an object is included in the area of ​​the divided third unit.

[0243] At this time, if an object is not included in the area of ​​the third unit, the encoder may determine the third unit as one coding unit and determine the inter prediction mode for the third unit as merge skip mode.

[0244] Meanwhile, the above-described process may be repeated until the first unit is divided into units of a specific size, and if an object is included within the area of ​​a unit of a specific size, the encoder may determine whether to split the unit into blocks and at least one of the encoding modes based on the rate-distortion cost (RD cost).

[0245] Here, the specific size into which the first unit, the coding tree unit (CTU), can be split may be a size corresponding to a preset specific depth, or a minimum size (min CU) that the coding unit (CU) can have.

[0246] As a second embodiment of a method for determining whether to split a block and an encoding mode in step S1330, the encoder can check whether a moving object with movement is included in a corresponding area up to a block level split at a specific depth from a coding tree unit (CTU), and if no object is included, set the corresponding area as a final coding unit (CU) and set the skip mode as the encoding mode.

[0247] In this case, if the encoder contains an object in the area, if the area is larger than a block divided into a specific depth, the encoder can divide the block once more and check the object again, and if the area is the same size as a block divided into a specific depth, the encoder can determine the block division and encoding mode based on the rate-distortion cost (RD cost).

[0248] As illustrated in FIG. 15, when a moving object (Ob) with movement is included within an area corresponding to a coding tree unit (CTU), the encoder can divide the coding tree unit (CTU) into four 32x32 areas without determining an encoding mode for the coding tree unit (CTU).

[0249] Next, the encoder checks whether each of the four 32x32 regions contains a moving object with motion, and skips the block segmentation process and encoding mode determination for the 32x32 region that does not contain an object, sets the 32x32 region as the final coding unit (CU), and the encoding mode can be determined as skip mode.

[0250] Meanwhile, when a specific depth as described above is set to a 32x32 block size, block division and encoding mode can be determined based on the rate-distortion cost (RD cost) for the 32x32 area containing the object.

[0251] As illustrated in FIG. 16, among the four 32x32 regions into which the coding tree unit (CTU) is divided, the block division and encoding mode are determined based on the rate-distortion cost (RD cost) for the region (U2) containing the object (Ob), and the remaining three 32x32 regions that do not contain the object (Ob) are all set as the final coding unit (CU), and the encoding mode can be determined as the skip mode.

[0252] As a third embodiment of a method for determining whether to split a block and an encoding mode in step S1330, the encoder can check whether a moving object with movement is included in an area from a coding tree unit (CTU) to a block level of a minimum size (min CU) that a coding block (CU) can have, and if no object is included, set the area as a final coding unit (CU) and set the skip mode as an encoding mode.

[0253] In this case, the encoder can determine the block division and encoding mode based on the rate-distortion cost (RD cost) if the area contains an object and if the area is larger than the minimum size (min CU), then perform block division once more to check the object again, and if the area is equal to the minimum size (min CU), then perform block division and encoding mode based on the rate-distortion cost (RD cost).

[0254] As shown in Fig. 16, when a moving object (Ob) with movement is included within an area (U2) of size 32x32, the encoder can divide the area (U2) into four 16x16 areas.

[0255] Next, the encoder checks whether each of the four 16x16 regions contains a moving object with motion, and skips the block segmentation process and encoding mode determination for the 16x16 regions that do not contain objects, sets the corresponding 16x16 regions as the final coding unit (CU), and the encoding mode can be determined as skip mode.

[0256] Meanwhile, when the minimum size (min CU) as described above is a 16x16 block size, block division and encoding mode can be determined based on the rate-distortion cost (RD cost) for the 16x16 region containing the object.

[0257] As illustrated in FIG. 17, for an area (U3) containing an object (Ob) among four 16x16 areas, block division and encoding mode are determined based on the rate-distortion cost (RD cost), and the remaining three 16x16 areas that do not contain an object (Ob) are all set as final coding units (CUs), and the encoding mode can be determined as skip mode.

[0258] In the above, the minimum size (min CU) that a coding block (CU) can have is described as a 16x16 block size, but the present invention is not limited thereto, and may have a smaller size, for example, a 4x4 block size.

[0259] According to another embodiment of the present invention, the encoder may additionally receive background change information from the game server (10), and determine to use intra prediction for a picture recognized as a scene change based on the background change information received from the game server (10).

[0260] For example, if the encoder determines that a scene has changed in a particular frame using background change information, the encoder can encode the frame as an intra frame regardless of the intra period or GOP (Group Of Pictures) size.

[0261] According to another embodiment of the present invention, when it is determined that a new object is included within the area of ​​the first unit based on object location information received from the game server (10), the encoding mode for at least one coding unit (CU) belonging to the first unit may be determined as an intra prediction mode.

[0262] For example, the encoder can use inter prediction to encode coding blocks (CUs) that are determined to contain a newly appeared new object using object position information in the inter frame.

[0263] Meanwhile, the object position information and background change information described above may be in vector or matrix form. Furthermore, the matrix information may include information related to scaling, rotation, and other factors in addition to translation.

[0264] And the game engine expresses the changes in objects or backgrounds, such as game characters in the 3D domain, in the form of a matrix, but the 3D information can be changed to 2D information in the image domain for the image encoding process.

[0265]

[0266] The methods described herein may be performed by a processor of a video encoder or video decoder. Furthermore, the encoder may generate a bitstream that is decoded by a video signal processing method, and the bitstream generated by the encoder may be stored in a computer-readable, non-transitory storage medium (recording medium).

[0267] The embodiments of the present invention described above may be implemented through various means. For example, the embodiments of the present invention may be implemented using hardware, firmware, software, or a combination thereof.

[0268] Some embodiments may also be implemented in the form of a computer-executable storage medium containing computer-executable instructions, such as program modules executed by a computer. Computer-readable media may be any available media that can be accessed by a computer, and includes both volatile and nonvolatile media, removable and non-removable media.

[0269] Additionally, computer-readable media may include both computer storage media and communication media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, or other data. Communication media typically includes other data, such as computer-readable instructions, data structures, or program modules, in a modulated data signal, or other transport mechanism, and includes any information delivery media.

[0270] In addition, although the preferred embodiments of the present invention have been illustrated and described above, the present invention is not limited to the specific embodiments described above, and various modifications can be made by a person having ordinary skill in the art to which the invention pertains without departing from the gist of the present invention as claimed in the claims, and such modifications should not be understood individually from the technical idea or prospect of the present invention.

Claims

1. In a method of encoding an image, Step of receiving an image in RGB format; A step of converting the received RGB format image into an image in YCbCr format; A step of encoding the above-mentioned converted YCbCr format image; and An image encoding method comprising the step of transmitting conversion information used to convert the RGB format image into the YCbCr format image together with the encoded YCbCr format image.

2. In the first paragraph, the conversion information is An image encoding method including color specification information indicating a color space that can be expressed by the above RGB format.

3. In the first paragraph, the conversion information is An image encoding method, wherein color conversion information indicating a conversion method used for color conversion from the RGB format to the YCbCr format is included in the conversion information.

4. In the first paragraph, the conversion step A step of performing optical-electro conversion on the RGB format image; An image encoding method in which transfer function information representing a transfer function used in the optical-electrical conversion is included in the conversion information.

5. In the first paragraph, the conversion step A step of performing quantization on the above-mentioned converted YCbCr format image; and An image encoding method comprising the step of performing chroma down-sampling on the quantized YCbCr format image.

6. In the first paragraph, the transmission step Transmitting a bitstream including at least one of the conversion information and camera viewpoint information in HLS (High-Level Syntax) along with information about the encoded YCbCr format image, At least one of the above conversion information and the above camera viewpoint information A video encoding method in which at least one of VUI (Video Usability Information), SEI (Supplemental Enhancement Information), VPS (Video Parameter Set), SPS (Sequence Parameter Set), PPS (Picture Parameter Set) and PH (Picture Header) is inserted.

7. In the first paragraph, the encoding step Step of receiving object location information; A step of dividing a picture included in an image to be encoded into first units of a first size; A step of using the received object location information to determine whether an object is included within the area of ​​the divided first unit; and A video encoding method comprising: a step of determining whether to divide a block for the first unit and at least one of an encoding mode, based on the above verification result.

8. In paragraph 7, the decision step is A video encoding method in which, if an object is not included within the area of ​​the first unit, the first unit is determined as one coding unit, and the inter prediction mode for the first unit is determined as skip mode.

9. In paragraph 7, the decision step is An image encoding method comprising: a step of determining whether to split a block and at least one of an encoding mode for the first unit based on a rate-distortion cost when an object is included within the area of ​​the first unit; 10. In paragraph 7, the decision step is If an object is contained within the area of ​​the above first unit, A step of dividing the first unit into second units having a second size smaller than the first size; and A step of checking whether an object is included within the area of ​​the divided second unit using the received object location information; and A video encoding method in which, if an object is not included in the area of ​​the second unit, the second unit is determined as one coding unit, and the inter prediction mode for the second unit is determined as skip mode.

11. In paragraph 10, the decision step is If an object is contained within the area of ​​the second unit, A step of dividing the second unit into third units having a third size smaller than the second size; and A step of using the received object location information to determine whether an object is included within the area of ​​the divided third unit; and A video encoding method in which, if an object is not included in the area of ​​the third unit, the third unit is determined as one coding unit, and the inter prediction mode for the third unit is determined as skip mode.

12. In paragraph 7, The above first unit is divided into units of a specific size, An image encoding method in which, when an object is included within an area of ​​the specific size unit, at least one of block division and encoding mode for the specific size unit is determined based on a rate-distortion cost (RD cost).

13. In paragraph 12, A video encoding method wherein the above specific size is the minimum size that the coding unit can have.

14. In paragraph 7, further comprising a step of receiving background change information; A video encoding method in which intra prediction is used for a picture recognized as a scene change based on the received background change information.

15. In paragraph 7, A video encoding method in which, when it is determined that a new object is included within the area of ​​the first unit according to the received object location information, the encoding mode for at least one coding unit belonging to the first unit is determined to be an intra prediction mode.

16. An image encoding device performing the method of any one of claims 1 to 15.

17. In a method for decrypting an image, A step of receiving a bitstream including information about an image in an encoded YCbCr format and conversion information used to convert an image in an RGB format into the YCbCr format image; A step of decoding an encoded YCbCr format image; and An image decoding method comprising: a step of converting the decoded YCbCr format image into an RGB format image using the above conversion information.

18. In paragraph 17, the conversion information is An image decoding method comprising at least one of color specification information indicating a color range that can be expressed by the RGB format, color conversion information indicating a conversion method used for color conversion from the RGB format to the YCbCr format, and conversion function information indicating a conversion function used for optical-electrical conversion for the RGB format image.

19. In the 17th paragraph, the conversion step It is performed by linking the above conversion information and bit-depth information per pixel, A step of performing chroma up-sampling on the decoded YCbCr format image; A step of performing inverse quantization on a YCbCr format image on which the above chroma up-sampling has been performed; A step of converting the image on which the above dequantization has been performed into RGB format; and An image decoding method comprising a step of performing electro-optical conversion on an image converted into the RGB format.

20. An image decoding device performing the method of any one of claims 17 to 19.

Images