Image encoding device, image decoding device, image processing system and program

Abstract

To reduce design and verification costs.SOLUTION: An image coding device 10 according to the present invention an image encoding unit 100 that is configured to perform encoding processing such that an even field is a Top field, an odd field is the Bottom field when an input image is an interlaced image, and the Top field and the Bottom field immediately after the Top field are paired.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an image encoding device, an image decoding device, an image processing system, and a program. [Background technology]

[0002] Non-Patent Document 2 discloses additional information for controlling the encoding and decoding processes of interlaced images. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] ITU-T Recommendation H.266 Versatile video coding [Non-Patent Document 2] ITU-T Recommendation H.274 Versatile supplemental enhancement information messagaes for coded video bitstreams Summary of the Invention [Problem to be solved by the invention]

[0004] However, the additional information disclosed in Non-Patent Document 2 allows for multiple patterns for the encoding and decoding methods of interlaced images, which poses the problem of increased design and verification costs for image encoding devices and image decoding devices.

[0005] Therefore, the present invention has been made in consideration of the above-mentioned problems, and aims to provide an image encoding device, an image decoding device, an image processing system, and a program that can reduce design costs and verification costs by limiting the encoding method and decoding method of interlaced images to specific patterns. [Means for solving the problem]

[0006] A first aspect of the present invention is an image coding apparatus, comprising: th field (The following is the even field) becomes the Top field, and odd th field (Below, odd field) The present invention relates to an image encoding unit that is configured to perform encoding processing so that a Top field becomes a Bottom field and the Top field and the Bottom field immediately following the Top field are paired.

[0007] A second feature of the present invention is summarized as an image decoding device including an image decoding unit configured to perform decoding processing such that, when the value of sps_field_flag in control data decoded from a bitstream is 1, an even field becomes a Top field, an odd field becomes a Bottom field, and the Top field and the Bottom field immediately following the Top field are paired.

[0008] A third feature of the present invention is summarized as an image processing system including an image encoding device and an image decoding device, wherein the image encoding device includes an image encoding unit configured to perform encoding processing such that, when an input image is an interlaced image, an even field becomes a Top field, an odd field becomes a Bottom field, and the Top field and the Bottom field immediately following the Top field are paired; and the image decoding device includes an image decoding unit configured to perform decoding processing such that, when a value of sps_field_flag is 1 among control data decoded from a bitstream, an even field becomes a Top field, an odd field becomes a Bottom field, and the Top field and the Bottom field immediately following the Top field are paired.

[0009] A fourth feature of the present invention is summarized as a program for causing a computer to function as an image coding device, the image coding device including an image coding unit configured to perform coding processing such that, when an input image is an interlaced image, an even field becomes a Top field, an odd field becomes a Bottom field, and the Top field and the Bottom field immediately following the Top field are paired.

[0010] A fifth feature of the present invention is summarized as a program for causing a computer to function as an image decoding device, the image decoding device including an image decoding unit configured to perform decoding processing such that, when a value of sps_field_flag in control data decoded from a bitstream is 1, an even field becomes a Top field, an odd field becomes a Bottom field, and the Top field and the Bottom field immediately following the Top field are paired. Effect of the Invention

[0011] According to the present invention, it is possible to provide an image encoding device, an image decoding device, an image processing system, and a program that can reduce design costs and verification costs by limiting the encoding method and decoding method of an interlaced image to a specific pattern. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of an image processing system 1 according to an embodiment. [Diagram 2] FIG. 1 is a diagram illustrating an example of functional blocks of an image encoding device 10 according to an embodiment. [Diagram 3] FIG. 3 is a diagram showing an example of functional blocks of the image encoding unit 100 of the image encoding device 10 according to an embodiment. [Figure 4] FIG. 4 is a flowchart showing an example of processing of the image encoding device 10 according to an embodiment. [Diagram 5] FIG. 5 is a diagram showing an example of a pre-processed image input to the image encoding unit 100 when adaptive downsampling is performed for each frame by the processing in steps S401 to S405 shown in FIG. [Figure 6] FIG. 6 is a diagram showing an example of functional blocks of the inter prediction unit 111 of the image encoding unit 100 of the image encoding device 10 according to an embodiment. [Figure 7] FIG. 7 is a diagram showing an example of functional blocks of the image decoding device 20 according to an embodiment. [Figure 8] FIG. 8 is a diagram showing an example of functional blocks of the image decoding unit 200 according to an embodiment. [Figure 9] FIG. 9 is a diagram illustrating an example of functional blocks of the inter prediction unit 241 according to an embodiment. [Figure 10] FIG. 10 is a diagram for explaining the second modification. [Figure 11] FIG. 11 is a diagram for explaining the second modification. [Figure 12] FIG. 12 is a diagram for explaining the third modification. [Figure 13] FIG. 13 is a diagram for explaining the third modification. [Figure 14] FIG. 14 is a diagram for explaining the third modification. [Figure 15] FIG. 15 is a diagram for explaining the third modification. [Figure 16] FIG. 16 is a diagram for explaining the fourth modification. [Figure 17] FIG. 17 is a diagram for explaining the fourth modification. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, the embodiments of the present invention will be described with reference to the drawings. Note that the components in the following embodiments can be appropriately replaced with existing components, and various variations including combinations with other existing components are possible. Therefore, the description of the following embodiments does not limit the content of the invention described in the claims.

[0014] (First embodiment) An image processing system 1 according to a first embodiment of the present invention will be described below with reference to Fig. 1 to Fig. 12. Fig. 1 is a diagram showing an image processing system 1 according to the first embodiment of the present invention.

[0015] As shown in FIG. 1, an image processing system 1 includes an image encoding device 10 and an image decoding device 20.

[0016] The image encoding device 10 is configured to generate encoded data by encoding an input image signal. The image decoding device 20 is configured to generate an output image signal by decoding the encoded data.

[0017] Here, the encoded data may be transmitted from the image encoding device 10 to the image decoding device 20 via a transmission path. Alternatively, the encoded data may be stored in a storage medium and then provided from the image encoding device 10 to the image decoding device 20.

[0018] (Image encoding device 10) The image encoding device 10 according to this embodiment will be described below with reference to Fig. 2. Fig. 2 is a diagram showing an example of functional blocks of the image encoding device 10 according to this embodiment.

[0019] As shown in FIG. 2, the image encoding device 10 includes a preprocessing unit 300, an analysis unit 400, a control unit 500, a UI unit 600, and the image encoding unit 100.

[0020] The pre-processing unit 300 is configured to receive an input image signal and control information output from the control unit 500, perform pre-processing, and output a pre-processed image signal and control data.

[0021] The analysis unit 400 is configured to receive an input image signal and output analysis information to the control unit 500 .

[0022] The control unit 500 is configured to receive the analysis information output from the analysis unit 400 and the setting information output from the UI unit 600 as input, and to output control information to the pre-processing unit 300 and the image encoding unit 100 .

[0023] The UI unit 600 is configured to output, to the control unit 500, setting information designated by the user via the GUI or CUI.

[0024] The image encoding unit 100 is configured to receive as input the preprocessed image signal and control data output from the preprocessing unit 300, and control information output from the control unit 500, and to output a bit stream.

[0025] (Image Encoding Unit 100) The image encoding unit 100 of the image encoding device 10 according to this embodiment will be described below with reference to Fig. 3. Fig. 3 is a diagram showing an example of functional blocks of the image encoding unit 100 according to this embodiment.

[0026] As shown in FIG. 3, the image encoding unit 100 includes an inter prediction unit 111, an intra prediction unit 112, a subtractor 121, an adder 122, a transform / quantization unit 131, an inverse transform / inverse quantization unit 132, an encoding unit 140, an in-loop filter processing unit 150, and a frame buffer 160.

[0027] The inter prediction unit 111 is configured to generate a prediction signal by inter prediction (inter-frame prediction) in the same manner as the inter prediction unit 241 described later.

[0028] Specifically, the inter prediction unit 111 is configured to identify a reference block contained in the reference frame by comparing a frame to be encoded (hereinafter, the target frame) with a reference frame stored in the frame buffer 160, and to determine a motion vector for the identified reference block.

[0029] Moreover, the inter prediction unit 111 is configured to generate a prediction signal included in the prediction block for each prediction block based on the reference block and the motion vector. The inter prediction unit 111 is configured to output the prediction signal to the subtractor 121 and the adder 122. Here, the reference frame is a frame different from the target frame.

[0030] The intra prediction unit 112 is configured to generate a prediction signal by intra prediction (intra-frame prediction) in the same manner as an intra prediction unit 242 described later.

[0031] Specifically, the intra prediction unit 112 is configured to identify a reference block included in a target frame, and generate a prediction signal for each prediction block based on the identified reference block. The intra prediction unit 112 is also configured to output the prediction signal to the subtractor 121 and the adder 122.

[0032] Here, the reference block is a block that is referred to for a block to be predicted (hereinafter, a target block). For example, the reference block is a block adjacent to the target block.

[0033] The subtractor 121 is configured to subtract the prediction signal from the input image signal, and output the prediction residual signal to the transformation and quantization unit 131. Here, the subtractor 121 is configured to generate a prediction residual signal that is the difference between the prediction signal generated by intra prediction or inter prediction and the input image signal.

[0034] The adder 122 is configured to add the prediction signal to the prediction residual signal output from the inverse transform / inverse quantization unit 132 to generate a pre-filtered decoded signal, and output the pre-filtered decoded signal to the intra prediction unit 112 and the in-loop filter processing unit 150.

[0035] Here, the unfiltered decoded signal forms a reference block used by the intra prediction unit 112 .

[0036] The transform / quantization unit 131 is configured to perform a transform process on the prediction residual signal and to obtain a coefficient level value. Furthermore, the transform / quantization unit 131 may be configured to quantize the coefficient level value.

[0037] Here, the transform process is a process of transforming a prediction residual signal into a frequency component signal. In the transform process, a basis pattern (transformation matrix) corresponding to a discrete cosine transform (DCT) may be used, or a basis pattern (transformation matrix) corresponding to a discrete sine transform (DST) may be used.

[0038] The inverse transform / inverse quantization unit 132 is configured to perform inverse transform processing on the coefficient level values ​​output from the transform / inverse quantization unit 131, similarly to the inverse transform / inverse quantization unit 220. Here, the inverse transform / inverse quantization unit 132 may be configured to perform inverse quantization of the coefficient level values ​​prior to the inverse transform processing.

[0039] Here, the inverse transformation process and inverse quantization are performed in the reverse order to the transformation process and quantization performed by the transformation and quantization unit 131 .

[0040] The encoding unit 140 is configured to encode the coefficient level values ​​output from the transform / quantization unit 131, and output encoded data.

[0041] Here, for example, the coding is entropy coding, which assigns codes of different lengths based on the probability of occurrence of coefficient level values.

[0042] The encoding unit 140 is also configured to encode control data for use in the decoding process in addition to the coefficient level values.

[0043] In addition to the control data input from the pre-processing unit 300, the control data may include size data such as an encoding block size, a prediction block size, and a transform block size.

[0044] Similar to in-loop filter processing unit 250, in-loop filter processing unit 150 is configured to perform filtering on the unfiltered decoded signal output from adder 122 and to output the filtered decoded signal to frame buffer 160.

[0045] The frame buffer 160 is configured to store reference frames used by the inter prediction unit 111 .

[0046] An example of the processing of the image encoding device 10 according to this embodiment will be described below with reference to Fig. 4. Fig. 4 is a flowchart showing an example of the processing of the image encoding device 10.

[0047] As shown in FIG. 4, in step S401, the analysis unit 400 analyzes an input image signal, estimates the coding difficulty of the input image signal, and outputs it as analysis information.

[0048] For example, the analysis unit 400 may calculate the encoding difficulty from a variance value of pixel values ​​(luminance signal values ​​or color difference signal values) of an input image signal. For example, the analysis unit 400 may estimate that the encoding difficulty is high when the variance value is greater than a predetermined threshold, and may estimate that the encoding difficulty is low when the variance value is equal to or less than the threshold.

[0049] For example, the analysis unit 400 may calculate the encoding difficulty from a difference value between pixel values ​​in a reference frame. Specifically, the analysis unit 400 may estimate the encoding difficulty using the sum of absolute values ​​of the difference values, the sum of squares of the difference values, or a difference value per pixel obtained by dividing each value by the number of pixels. For example, the analysis unit 400 may estimate that the encoding difficulty is high when the difference value is greater than a predetermined threshold, and may estimate that the encoding difficulty is low when the difference value is equal to or less than the threshold.

[0050] For example, the analysis unit 400 may estimate the encoding difficulty using a machine learning based technique such as a convolutional neural network (CNN) that is configured and trained to receive an input image signal and output an estimated value of the encoding difficulty.

[0051] Here, the analysis unit 400 may use a method other than the above, so long as it is configured to receive an input image signal and output an encoding difficulty level.

[0052] Furthermore, in the above description, the encoding difficulty level is divided into two levels, "high" and "low," but the encoding difficulty level may be divided into three or more levels.

[0053] In the above, an example has been described in which the analysis unit 400 outputs the encoding difficulty, but the analysis unit 400 may output information for determining the encoding difficulty (e.g., the above-mentioned variance value, etc.), and the encoding difficulty itself may be determined within the control unit 500.

[0054] Furthermore, as described below, when downsampling processing is executed for all input image signals according to user settings from the UI unit 600, this step S401 may be omitted.

[0055] After the above processing is performed, the procedure proceeds to step S402.

[0056] In step S402, the control unit 500 determines whether or not it is necessary to perform downsampling processing. For example, the control unit 500 may determine whether or not to perform downsampling processing based on the analysis information output from the analysis unit 400. For example, when it is determined that the encoding difficulty is high, the control unit 500 may determine that it is necessary to perform downsampling processing.

[0057] As described above, the control unit 500 may determine the encoding difficulty level internally and then determine whether or not downsampling processing needs to be performed.

[0058] As described above, when the encoding difficulty is defined in three or more stages, the control unit 500 may determine that the execution of the downsampling process is unnecessary when the encoding difficulty is the lowest, and may determine that the execution of the downsampling process is necessary in other cases, and may determine the execution method of the downsampling process so that the resolution after the downsampling process becomes smaller as the encoding difficulty becomes higher. Note that in this specification, the resolution refers to the size of one frame or one entire field of an image (the value obtained by multiplying the number of pixels in the horizontal direction by the number of pixels in the vertical direction).

[0059] Furthermore, for example, the control unit 500 may determine whether or not to perform downsampling processing based on the resolution of an input image (an image included in an input image signal).

[0060] For example, the control unit 500 may determine that it is necessary to perform downsampling processing when the resolution of the input image is greater than a predetermined resolution, and may determine that it is not necessary to perform downsampling processing when the resolution of the input image is equal to or less than the predetermined resolution.

[0061] For example, the control unit 500 may determine that it is necessary to perform downsampling processing when the resolution of the input image signal is 8K resolution, and may determine that it is not necessary to perform downsampling processing when the resolution of the input image signal is less than 8K resolution.

[0062] In addition, the control unit 500 may predefine the resolution of the image after the downsampling process, and if the resolution of the input image is greater than the resolution of the image, determine that it is necessary to perform the downsampling process, and decide to perform the downsampling process so that the resolution of the image after the downsampling process will be the same.

[0063] Also, for example, the control unit 500 may determine whether or not it is necessary to perform the downsampling process based on the control information input from the UI unit 600. For example, when the control information sets the downsampling process to be always performed, the control unit 500 may determine that it is always necessary to perform the downsampling process.

[0064] Furthermore, the control unit 500 may determine whether or not it is necessary to perform downsampling processing by combining the above-described determination methods.

[0065] For example, the control unit 500 may obtain information for controlling the On / Off of the downsampling process from the UI unit 600, and when the downsampling process is On and the resolution of the input image is greater than a predetermined resolution, the control unit 500 may determine whether or not to perform the downsampling process based on the encoding difficulty, and otherwise determine that it is not necessary to perform downsampling.

[0066] The number of encoding difficulty levels, the predetermined resolution, and the like used in the above processing may be designed so that the user can specify them via the UI unit 600 .

[0067] After the above processing is performed, the procedure proceeds to step S403.

[0068] In step S403, the procedure branches based on the result of step S402. If it is determined in step S402 that downsampling processing is required, the procedure proceeds to step S404, and if it is determined in step S402 that downsampling processing is not required, the procedure proceeds to step S406.

[0069] In step S404, the pre-processing unit 300 performs a downsampling process on the input image signal. Note that, if step S404 is not performed (if it is determined that the downsampling process is not necessary), the pre-processing unit 300 may output the input image signal to the encoding unit 100 as is.

[0070] The pre-processing unit 300 may perform the downsampling process using a one-dimensional or two-dimensional downsampling filter.

[0071] Furthermore, the pre-processing unit 300 may perform the downsampling process using a machine learning based technique such as CNN that learns downsampling and upsampling processes as a pair.

[0072] After the downsampling process is completed, the procedure proceeds to step S405.

[0073] In step S405, the pre-processing unit 300 determines the type of upsampling process to be performed by the image decoding device 20 and parameters to be used for the upsampling process based on the input image and the downsampled image, and outputs the determined type and parameters as control data.

[0074] The pre-processing unit 300 may prepare various types of upsampling processes in advance, such as a one-dimensional convolution filter, a two-dimensional convolution filter, a machine learning based method such as CNN, and an upsampling process identical to RPR described later, and output an index value indicating which of these to use as control data.

[0075] Furthermore, the pre-processing unit 300 may be provided with a plurality of candidates for the same type of filter (for example, a one-dimensional convolution filter) based on variations in filter coefficients.

[0076] Furthermore, the preprocessing unit 300 may determine filter coefficients and network parameters (weights, etc.) of a machine learning-based method such as CNN, and output them as control data.

[0077] As a method for determining the type and parameters of the upsampling process, the preprocessing unit 300 may select, for example, a method that minimizes an error (sum of absolute error, sum of squared error, etc.) between an input image and an image after the upsampling process. These control data may be coded as a user-defined SEI message in the image coding unit 100.

[0078] Furthermore, the pre-processing unit 300 may be configured not to specify the up-sampling process in particular and not to output it as control data.

[0079] The pre-processing unit 300 may switch whether or not to output control data related to the upsampling process based on control information input from the UI unit 600, for example.

[0080] After the above-mentioned processing is performed, the procedure proceeds to step S406.

[0081] In step S406, the image encoding unit 100 performs encoding processing and outputs a bit stream. Here, an example of the encoding processing in the image encoding unit 100 will be described with reference to Figs.

[0082] FIG. 5 shows an example of a pre-processed image input to the image encoding unit 100 when adaptive downsampling is performed for each frame by the processes in steps S401 to S405.

[0083] In FIG. 5, Fr0, Fr2, and Fr5 indicate frames on which downsampling processing was not performed, and Fr1, Fr3, and Fr4 indicate frames on which downsampling processing was performed.

[0084] In this way, when the image resolution differs for each frame, the inter prediction unit 111 uses a technique called RPR (Reference Picture Resampling) described in Non-Patent Document 1. The process of the inter prediction unit 111 when using RPR will be described with reference to FIG.

[0085] FIG. 6 is a diagram showing an example of functional blocks of the inter prediction unit 111 of the image encoding unit 100 of the image encoding device 10 according to this embodiment.

[0086] As shown in FIG. 6, the inter prediction unit 111 includes a resolution conversion unit 1111, a parameter determination unit 1112, and a predicted image generation unit 1113.

[0087] The resolution conversion unit 1111 is configured to convert the resolution of the reference frame used in inter prediction into the resolution of the frame to be coded, when the resolution of the frame to be coded differs from the resolution of the reference frame used in inter prediction.

[0088] If the resolution of the reference frame is greater than the resolution of the frame to be coded, downsampling is performed, and if the resolution of the reference frame is less than the resolution of the frame to be coded, upsampling is performed. The specific processing content is defined in Non-Patent Document 1, so a detailed description will be omitted.

[0089] The parameter determination unit 1112 is configured to determine parameters related to inter prediction. Such parameters are, for example, a motion vector, an index value indicating which reference frame to use, and the like.

[0090] The predicted image generating unit 1113 is configured to generate a predicted image using the parameters determined by the parameter determining unit 1112 .

[0091] (Image Decoding Device 20) Hereinafter, the image decoding device 20 according to this embodiment will be described with reference to Fig. 7. Fig. 7 is a diagram showing an example of functional blocks of the image decoding device 20 according to this embodiment.

[0092] As shown in FIG. 7, the image decoding device 20 includes an extraction unit 700, an image decoding unit 200, a post-processing unit 800, a control unit 900, and a UI unit 1000.

[0093] The extraction unit 700 is configured to receive as input a bitstream and control information output from the control unit 900, extract data to be input to the image decoding unit 200 from the bitstream based on the control information, and output the extracted data as a bitstream after extraction.

[0094] The image decoding unit 200 is configured to perform a decoding process using the extracted bit stream as input, and to output a decoded image signal and control data.

[0095] The post-processing unit 800 is configured to receive the decoded image signal, the control data, and the control information output from the control unit 900 as input, and to output a post-processed decoded signal.

[0096] The control unit 900 is configured to receive as input the decoded image signal and control data output from the image decoding unit 200 and the setting information output from the UI unit 1000, and to output control information.

[0097] (Image Decoding Unit 200) Hereinafter, the image decoding unit 200 of the image decoding device 20 according to this embodiment will be described with reference to Fig. 8. Fig. 8 is a diagram showing an example of functional blocks of the image decoding unit 200 according to this embodiment.

[0098] As shown in FIG. 8, the image decoding unit 200 includes a decoding unit 210, an inverse transform / inverse quantization unit 220, an adder 230, an inter prediction unit 241, an intra prediction unit 242, an in-loop filtering unit 250, and a frame buffer 260.

[0099] The decoding unit 210 is configured to decode the coded data generated by the image coding device 100, and to decode the coefficient level values.

[0100] Here, for example, the decoding is of entropy coded data that assigns codes of different lengths based on the probability of occurrence of coefficient level values.

[0101] The decoding unit 210 may be configured to obtain the control data by a decoding process of the encoded data.

[0102] Here, the control data may include size data such as a coding block (CU: Coding Unit) size, a prediction block (PU: Prediction Unit) size, and a transform block (TU: Transform Unit) size.

[0103] The inverse transform / inverse quantization unit 220 is configured to perform an inverse transform process on the coefficient level values ​​output from the decoding unit 210. Here, the inverse transform / inverse quantization unit 220 may be configured to perform an inverse quantization process on the coefficient level values ​​prior to the inverse transform process.

[0104] The adder 230 is configured to add the prediction signal to the prediction residual signal output from the inverse transform / inverse quantization unit 220 to generate a pre-filtered decoded signal, and output the pre-filtered decoded signal to the intra prediction unit 242 and the in-loop filter processing unit 250.

[0105] Here, the unfiltered decoded signal forms a reference block used by the intra prediction unit 242 .

[0106] The inter prediction unit 241 is configured to generate a prediction signal by inter prediction (inter-frame prediction).

[0107] Specifically, the inter prediction unit 241 is configured to generate a prediction signal for each prediction block based on a motion vector decoded from encoded data and a reference signal included in a reference frame. The inter prediction unit 241 is configured to output the prediction signal to the adder 230.

[0108] The intra prediction unit 242 is configured to generate a prediction signal by intra prediction (intra-frame prediction).

[0109] Specifically, the intra prediction unit 242 is configured to identify a reference block included in the target frame, and generate a prediction signal for each prediction block based on the identified reference block. The intra prediction unit 242 is configured to output the prediction signal to the adder 230.

[0110] The in-loop filtering unit 250 is configured to perform filtering on the unfiltered decoded signal output from the adder 230 , and to output the filtered decoded signal to the frame buffer 260 .

[0111] Here, the in-loop filter process may be composed of a plurality of filter processes. For example, the filter processes include a deblocking filter process that reduces distortion occurring at the boundary of a block (an encoding block, a prediction block, a transformation block, or a subblock obtained by dividing them), and an adaptive loop filter process that switches filters based on filter coefficients, filter selection information, local characteristics of an image pattern, and the like transmitted from the image encoding device 100.

[0112] The frame buffer 260 is configured to store reference frames used by the inter prediction unit 241 .

[0113] Here, the filtered decoded signal forms a reference frame used by the inter prediction unit 241.

[0114] (Inter prediction unit 241) An example of the process of the inter prediction unit 241 of the image decoding unit 200 of the image decoding device 20 according to this embodiment will be described with reference to Fig. 9. Fig. 9 is a diagram showing an example of functional blocks of the inter prediction unit 241 according to this embodiment.

[0115] As shown in FIG. 9, the inter prediction unit 241 includes a parameter decoding unit 2411, a resolution conversion unit 2412, and a predicted image generation unit 2413.

[0116] The parameter decoding unit 2411 is configured to decode parameters related to inter prediction from the bitstream. Here, the parameters include, for example, a motion vector, an index value indicating which reference frame is to be used, and the like.

[0117] The resolution conversion unit 2412 is configured to identify a reference frame based on the above-mentioned parameters, and convert the resolution of the reference frame when the resolution of the reference frame differs from the resolution of the frame to be decoded. As with the resolution conversion unit 1111, the resolution conversion unit 2412 can use the method defined in Non-Patent Document 1 as the method of converting the resolution.

[0118] The predicted image generating unit 2413 is configured to generate a predicted image, similar to the predicted image generating unit 1113.

[0119] (Post-processing unit 800) In this embodiment, the post-processing unit 800 is configured to perform upsampling processing as necessary, and generate and output a post-processed decoded image.

[0120] Here, the post-processing unit 800 may determine that an upsampling process needs to be performed, for example, when the resolution of the decoded image output from the image decoding unit 200 is smaller than the resolution of the image of the sequence to be decoded, and may perform the upsampling process.

[0121] The resolution of the images of the sequence to be decoded may be included in the control data decoded by the image decoding unit 200. The resolution of the images of the sequence to be decoded may be input as control information from the UI unit 1000 via the control unit 900.

[0122] When it is determined that the upsampling process needs to be performed, the post-processing unit 800 performs the upsampling process. Here, the post-processing unit 800 may select a method for performing the upsampling process from among a plurality of methods for performing the upsampling process.

[0123] The post-processing unit 800 may have various types of upsampling processing prepared in advance, such as a one-dimensional convolution filter, a two-dimensional convolution filter, a machine learning based method such as CNN, and an upsampling processing identical to the above-mentioned RPR.

[0124] Furthermore, the post-processing unit 800 may be provided with a plurality of candidates for the same type of filter (for example, a one-dimensional convolution filter) based on variations in filter coefficients.

[0125] In addition, the pre-processing unit 300 may determine filter coefficients and network parameters (weights, etc.) of a machine learning-based method such as CNN, and the post-processing unit 800 may perform upsampling processing based on values ​​(parameters determined by the pre-processing unit 300) input as control data via a bit stream.

[0126] The pre-processing unit 300 may determine which of a plurality of upsampling processes to use, and the post-processing unit 800 may determine which upsampling process to use based on a value input as control data via the bitstream (a value indicating the upsampling process determined by the pre-processing unit 300).

[0127] The UI unit 1000 may output setting information based on a user's setting, and the post-processing unit 800 may determine which upsampling process to use from among a plurality of upsampling processes based on the setting information.

[0128] The post-processing unit 800 may determine filter coefficients in the upsampling process and parameters (weights, etc.) of a network for a machine learning-based method such as CNN, based on the setting information output from the UI unit 1000 via the control unit 900.

[0129] (Variation 1) Hereinafter, Modification 1 of the present invention will be described with a focus on differences from the above-mentioned first embodiment. In Modification 1, a post-processing unit 800 is configured to perform post-filtering using the decoded image and control data output from the image decoding unit 200 as input.

[0130] In this modification example 1, the analysis unit 400 analyzes the area of ​​the screen content (such as a game screen or a remote desktop screen) in the input image signal, and outputs area information of the screen content (such as the number of areas, the position and size of each area, etc.) to the control unit 500.

[0131] Here, the analysis unit 400 may output information indicating only whether or not the entire image is a screen content.

[0132] The control unit 500 is configured to output the area information of the screen content output from the analysis unit 400 to the image encoding unit 100 .

[0133] Alternatively, the control unit 500 is configured to output the area specified by the UI unit 600 to the image encoding unit 100 as screen content area information.

[0134] The image encoding unit 100 is configured to encode the above-mentioned screen content area information as a part of the control data to generate a bitstream. At this time, the screen content area information may be encoded as a user-defined SEI message.

[0135] On the other hand, the image decoding unit 200 is configured to decode the screen content area information from the extracted bitstream described above, and output the decoded information to the post-processing unit 800 .

[0136] The post-processing unit 800 is configured to determine whether or not to perform post-filtering on each region of the decoded image signal based on the above-mentioned screen content region information, and to actually perform the post-filtering.

[0137] Here, the post-processing unit 800 may perform control so as not to perform post-filtering on an area that is defined as being a screen content based on the screen content area information.

[0138] In the above example, the image encoding device 10 transmits screen content area information to the image decoding device 20, and controls the post-filtering process in the image decoding device 20 based on the screen content area information.

[0139] As another example, the image encoding device 10 may transmit information about a post-filter application region (or a non-application region) and control whether or not to apply a post-filter in accordance with the application region (or non-application region) information.

[0140] (Variation 2) Hereinafter, the second modified example of the present invention will be described with reference to Figs. 10 and 11, focusing on the differences from the first embodiment and the first modified example.

[0141] In the present modified example 2, the image encoding device 10 and the image decoding device 20 are configured to switch between multi-channel organization and single-channel organization.

[0142] FIG. 10 shows an example of switching from a single-channel configuration to a multi-channel configuration, and FIG. 11 shows an example of switching from a multi-channel configuration to a single-channel configuration.

[0143] FIG. 10 shows an example in which, when switching from single-channel to multi-channel format, the input image signal is downsampled to reduce transmission capacity in order to broadcast content on a sub-channel other than the content broadcast during the single-channel format.

[0144] In the example of Fig. 10, switching from a single channel formation to a multi-channel formation is performed between Fr2 and Fr3. In the above-mentioned downsampling process, the UI unit 600 outputs control information indicating that the formation has been switched to a multi-channel formation via the control unit 500, and the pre-processing unit 300 performs the downsampling process based on the control information.

[0145] In the example of Fig. 11, switching from multi-channel to single-channel is performed between Fr2 and Fr3. In this case, as in the above example, the pre-processing unit 300 performs downsampling processing up to Fr2, but does not perform downsampling processing from Fr3 onwards.

[0146] The inter prediction unit 111 of the image encoding device 10 is configured to perform resolution conversion using RPR as described above when the resolution of the frame to be encoded and the resolution of the reference frame differ due to switching between multi-channel and single-channel organization.

[0147] Similarly, the inter prediction unit 241 of the image decoding device 20 is configured to perform resolution conversion using RPR as described above when the resolution of the decoding target frame differs from the resolution of the reference frame.

[0148] Finally, the post-processing unit 800 is configured to perform an upsampling process, where the content of the upsampling process is the same as in the above example.

[0149] That is, in this modified example 2, the image encoding device 10 includes a pre-processing unit 300 configured to output a pre-processed input image obtained by performing a downsampling process on the input image signal in multi-channel formation, and an image encoding unit 100 configured to encode the input image signals before and after switching between single-channel formation and multi-channel formation as the same bit stream by using an RPR that converts the resolution of the reference frame when the resolution of the reference frame and the resolution of the pre-processed input image differ.

[0150] In addition, in this modified example 2, the image decoding device 20 includes an image decoding unit 200 configured to decode the bit stream and output a decoded image, and a post-processing unit 800 configured to perform an upsampling process on the decoded image downsampled in the image encoding device 10 in the case of multi-channel organization.

[0151] Here, the image decoding unit 200 may include an inter prediction unit 241 configured to perform an RPR process for converting the resolution of the reference frame when the resolution of the frame to be decoded differs from the resolution of the reference frame.

[0152] In addition, the above-mentioned upsampling process may be a one-dimensional or two-dimensional convolution filter, a method using CNN, the same method as the RPR process, or may be performed based on control data decoded from the bitstream.

[0153] (Variation 3) 12 to 15, Modification 3 of the present invention will be described below, focusing on differences from the above-mentioned first embodiment and Modifications 1 and 2. Specifically, as Modification 3, an example of encoding and decoding processes when an input image is an interlaced image will be described.

[0154] In this third variant, when the input image is an interlaced image, the encoding process is performed so that the even field becomes the Top field and the odd field becomes the Bottom field, as shown in Figure 12, and the Top field is paired with the Bottom field immediately following it.

[0155] In the example of FIG. 12, the field number starts from 0, that is, from Fi0.

[0156] Here, the analysis unit 400 may be configured to determine whether the target input image is an interlaced image, and to output the determination result to the control unit 500.

[0157] For example, the analysis unit 400 may define in advance possible resolutions and information as to whether the resolutions correspond to an interlaced image or a progressive image, and determine whether the target input image is an interlaced image or a progressive image based on the resolution of the input image and such information.

[0158] Alternatively, the user may specify whether the target input image is an interlaced image via the UI unit 600. Here, the information (interlaced / progressive) specified by the user via the UI unit 600 is output to the control unit 500.

[0159] The control unit 500 is configured to receive as input the information output from the analysis unit 400 or the UI unit 600, and when the target input image is an interlaced image, to set control data so that the even field becomes the Top field, the odd field becomes the Bottom field, and the Top field and the Bottom field immediately following it are paired, as shown in FIG. 12, and to output the control data to the image encoding unit 100.

[0160] The image encoding unit 100 is configured to encode such control data and output it as a bit stream.

[0161] Here, the image encoding unit 100 may encode such control data as a Frame-field information SEI message defined in Non-Patent Document 2.

[0162] FIG. 13 shows an example of the setting values ​​of each syntax element when such control data is coded as a Frame-field information SEI message.

[0163] As another example, when the input image is an interlaced image, the image encoding unit 100 may combine two interlaced (field) images to generate a single frame image and encode it so that the even lines become the Top field and the odd lines become the Bottom field, as shown in Figure 14.

[0164] 14, the first line is line 0. The process of generating a frame image from a field image may be executed by the pre-processing unit 300, for example.

[0165] The control unit 500 may input to the pre-processing unit 300 control information as to whether or not to generate a frame image from a field image based on setting information input from the UI unit 600. Fig. 15 shows an example of the setting of the Frame-field information SEI message at this time.

[0166] Furthermore, when the input image is an interlaced image, the image encoding unit 100 may encode the image by setting the value of sps_field_flag described in Non-Patent Document 1 to 1 as control data.

[0167] This concludes the description of the processing in the image encoding device 10 according to Modification 3. Below, the processing in the image decoding device 20 will be described.

[0168] The image decoding unit 200 is configured to decode the control data from the bitstream and obtain the above-mentioned value of sps_field_flag and the value of the Frame-field information SEI message.

[0169] Here, the image decoding unit 200 may be configured to output an error when the value of sps_field_flag is 1 and the value of sps_field_flag and the setting value of the Frame-field information SEI message are different from the setting values ​​of Figure 13 or Figure 15.

[0170] In addition, even if the value of sps_field_flag is 1 and the value of sps_field_flag and the setting value of the Frame-field information SEI message are different from the setting values ​​of Figure 13 or Figure 15, the image decoding unit 200 may be configured to perform a decoding operation for the setting values ​​of Figure 13 when the decoded image signal is a field image, and to perform a decoding operation for the setting values ​​of Figure 15 when the decoded image signal is a frame image.

[0171] Here, the image decoding unit 200 determines whether the decoded image signal is a field image based on the resolution of the decoded image signal included in the control data or the setting information output from the UI unit 1000.

[0172] That is, in this modified example 3, the image encoding device 10 includes an image encoding unit 100 that is configured to perform encoding processing when the input image is an interlaced image such that the even field becomes the Top field, the odd field becomes the Bottom field, and the Top field and the Bottom field immediately following the Top field are paired.

[0173] Here, when the input image is an interlaced image, the image encoding unit 100 may be configured to set values ​​of the Frame-field information SEI message and encode them as control data so that the even field becomes the Top field, the odd field becomes the Bottom field, and the Top field and the Bottom field immediately following the Top field are paired, thereby generating a bitstream.

[0174] The image encoding device 10 may also include a UI unit 600 configured to set whether the input image is an interlaced image, and a control unit 500 configured to output control information that controls the encoding process based on the setting information output from the UI unit 600.

[0175] Here, the image encoding unit 100 may be configured to determine whether or not the input image is an interlaced image based on such control information.

[0176] Furthermore, in this modified example 3, the image decoding device 20 includes an image decoding unit 200 configured to perform decoding processing such that, when the value of sps_field_flag in the control data decoded from the bitstream is 1, the even field becomes the Top field, the odd field becomes the Bottom field, and the Top field and the Bottom field immediately following the Top field are paired.

[0177] Here, even if the image decoding unit 200 determines, based on the control data decoded from the bitstream, that the setting values ​​of the Frame-field information SEI message are not set so that the even field becomes the Top field, the odd field becomes the Bottom field, and the Top field and the Bottom field immediately following the Top field are paired, the image decoding unit 200 may be configured to perform decoding processing so that the even field becomes the Top field, the odd field becomes the Bottom field, and the Top field and the Bottom field immediately following the Top field are paired.

[0178] (Variation 4) 16 and 17, a fourth modification of the present invention will be described below, focusing on differences from the first embodiment and modifications 1 to 3. Specifically, as the fourth modification, an encoding process and a decoding process will be described for the case where spatial scalability is pseudo-realized using only the encoding technique defined in the H.266 Main profile specified in Non-Patent Document 1, using the above-mentioned RPR.

[0179] First, a description will be given of the process in the image encoding device 10. Specifically, a description will be given of an example in which an input image is input as shown in FIG.

[0180] In this fourth modification, images (Fr0' to Fr3') are generated by performing downsampling processing on each of the input frames (Fr0 to Fr3) as shown in Fig. 17. Such downsampling processing may be executed by the pre-processing unit 300, for example.

[0181] The UI unit 600 may be configured to output setting information indicating whether to implement the pseudo spatial scalability in this variant example 4 to the control unit 500, and the control unit 500 may be configured to output such setting information to the pre-processing unit 300 as control information.

[0182] The pre-processing unit 300 may perform a down-sampling process based on such control information.

[0183] That is, the preprocessing unit 300 is configured to generate low-resolution images (Fr0' to Fr3') from input images (Fr0 to Fr3) as shown in FIG.

[0184] The image encoding unit 100 is configured to perform encoding processing using as input low-resolution images (Fr0' to Fr3') generated from the input images (Fr0 to Fr3), respectively.

[0185] Here, the arrows pointing to each Fr / Fr′ in FIG.

[0186] For example, in Fig. 17, Fr3' indicates that Fr0' is used as the reference frame. Similarly, in Fig. 17, Fr1 indicates that Fr0, Fr3, and Fr1' are used as the reference frames.

[0187] Here, as shown in FIG. 17, the low resolution images (Fr0' to Fr3') use only the low resolution images (Fr0' to Fr3') as reference frames.

[0188] On the other hand, the high resolution images (Fr0 to Fr3) can use both the high resolution images and the low resolution images as reference frames.

[0189] The high-resolution image uses as a reference frame a low-resolution image (eg, Fr0' in the case of Fr0) obtained by performing downsampling processing on the target frame.

[0190] Note that the reference frame relationships shown in Fig. 17 are merely examples. As long as the above constraints that "low resolution images (Fr0' to Fr3') use only low resolution images (Fr0' to Fr3') as reference frames" and "high resolution images use low resolution images (Fr0' to Fr0', for example) obtained by performing downsampling processing on the target frame as a reference frame" are satisfied, the inter prediction unit 111 can use reference relationships other than those shown in Fig. 17.

[0191] When the resolution of the reference frame and the resolution of the frame to be encoded differ, the image encoding unit 100 uses the RPR in the same manner as in the above example.

[0192] At this time, the image encoding unit 100 may set the value of TemporalId defined in Non-Patent Document 1 as control data to be as shown in FIG. 17, and generate a bitstream.

[0193] That is, the image encoding unit 100 may encode the low-resolution image frame so that TemporalId=0 and the high-resolution image frame so that TemporalId=1.

[0194] The set value of TemporalId is merely an example, and it is sufficient that the TemporalId of the high-resolution image is set to be larger than the TemporalId of the low-resolution image.

[0195] In addition, in the above, an example has been described in which the high-resolution image uses the input image itself, but the image encoding unit 100 may also encode the error image between the input image and an image obtained by upsampling a low-resolution image as the high-resolution image.

[0196] Secondly, the process in the image decoding device 20 will be described.

[0197] The extractor 700 is configured to extract from the bitstream only the data that is actually to be decoded.

[0198] For example, the above-mentioned TemporalID value may be used for such extraction. That is, the extraction unit 700 may be configured to obtain the maximum value of the TemporalID to be decoded from the control unit 900, and output only the bitstream of the frame having the TemporalID equal to or less than the maximum value to the image decoding unit 200.

[0199] For example, if a low resolution image frame is coded so that TemporalId=0 and a high resolution image frame is coded so that TemporalId=1, then if the above-mentioned maximum value is set to 0, the extraction unit 700 will extract a bitstream related only to the low resolution image, and if the maximum value is set to 1, the extraction unit 700 will extract a bitstream including both the low resolution image and the high resolution image.

[0200] Here, the extraction unit 700 may determine this maximum value based on setting information from the UI unit 1000. Moreover, the maximum value may be set in advance for each image decoding device 20.

[0201] When a high-resolution image is also decoded, the inter prediction unit 241 uses the RPR if the resolution of the reference frame and the resolution of the frame to be decoded differ.

[0202] If the high resolution image is also decoded, the post-processing unit 800 is configured to discard the decoded image of the low resolution image and output only the decoded image of the high resolution image.

[0203] In addition, when an error image between an upsampled image of a low-resolution image and the input image is used as the high-resolution image, the post-processing unit 800 upsamples the low-resolution image and adds it to a decoded image of the high-resolution error image to generate a decoded image of the final high-resolution image.

[0204] Thereafter, in the same manner as described above, the post-processing unit 800 discards the low-resolution image and outputs only the decoded image of the high-resolution image.

[0205] The upsampling method may be performed based on a method or parameters specified by the image encoding device 10, as in the above example, or may be performed using a predetermined upsampling method.

[0206] When the image encoding device 10 specifies the upsampling method, the image encoding device 10 may specify that the image encoding device 10 and the image decoding device 20 perform the same upsampling process.

[0207] When the high-resolution image is not decoded, the post-processing unit 800 may be configured to output the decoded image of the low-resolution image output from the image decoding unit 200 as is.

[0208] That is, in the image encoding device 10 according to the present modified example 4, the image encoding unit 100 is configured to generate a low-resolution image by performing a downsampling process on a high-resolution image, and is configured to perform inter-prediction by referring to only the low-resolution image, and is configured to perform inter-prediction for the high-resolution image by referring to the low-resolution image at the same time and using RPR processing.

[0209] Here, the image encoding unit 100 may be configured to perform encoding by setting the Temporal ID of the high resolution image to a value greater than the Temporal ID of the low resolution image.

[0210] Furthermore, in the image decoding device 20 according to the present modified example 4, the image decoding unit 200 is configured to decode a high-resolution image and a low-resolution image of the same time as different frames, and to perform inter prediction using an RPR process when the resolution of the frame to be decoded differs from the resolution of the reference frame.

[0211] Here, the image decoding device 20 may include a control unit 900 configured to control whether to decode only the low resolution image or to decode both the low resolution image and the high resolution image based on the Temporal ID.

[0212] Furthermore, the image decoding device 20 may include a post-processing unit 800 configured to, when both the low-resolution image and the high-resolution image are decoded, discard the low-resolution image and output only the high-resolution image.

[0213] Furthermore, the above-described image encoding device 10 and image decoding device 20 may be realized as a program that causes a computer to execute each function (each process).

[0214] In the above embodiments and modified examples, the present invention has been described using the image encoding device 10 and the image decoding device 20 as examples. However, the present invention is not limited to such examples, and can be similarly applied to an image encoding / decoding system having the functions of the image encoding device 10 and the image decoding device 20. [Industrial Applicability]

[0215] According to this embodiment, for example, it is possible to improve the overall service quality in video communication, which makes it possible to contribute to Goal 9 of the Sustainable Development Goals (SDGs) led by the United Nations, which is to "build resilient infrastructure, promote sustainable industrialization and foster innovation." [Explanation of symbols]

[0216] 1. Image processing system 10...Image encoding device 100...Image encoding unit 111…Inter prediction section 1111...Resolution conversion section 1112... Parameter determination unit 1113...Prediction image generation unit 112…Intra prediction unit 121…Subtractor 122...Adder 131...Transformation and quantization section 132…Inverse transform and inverse quantization section 140...encoding section 300…Pretreatment section 400…Analysis department 500...Control unit 600...UI section 20...Image decoding device 200...Image decoding unit 210…Decoding section 220…Inverse transform and inverse quantization unit 230…Adder 241…Inter prediction section 2411...parameter decoding unit 2412…Resolution conversion section 2413…Prediction image generation unit 242…Intra prediction unit 250...In-loop filter processing section 260...Frame buffer 700...Extraction part 800…Post-processing section 900...Control unit 1000...UI section

Claims

1. An image decoding device, An image decoding device comprising an image decoding unit configured to perform a decoding process such that, when the value of sps_field_flag is 1 among control data decoded from a bit stream, an even-numbered field becomes a Top field, an odd-numbered field becomes a Bottom field, and the Top field and the Bottom field immediately following the Top field are paired.

2. 2. The image decoding device according to claim 1, wherein even when it is determined, based on the control data decoded from the bitstream, that a set value of a Frame-field information SEI message is not set so that an even-numbered field becomes the Top field, an odd-numbered field becomes the Bottom field, and the Top field and the Bottom field immediately following the Top field are paired, the image decoding unit is configured to perform the decoding process so that an even-numbered field becomes the Top field, an odd-numbered field becomes the Bottom field, and the Top field and the Bottom field immediately following the Top field are paired.

3. An image processing system including an image encoding device and an image decoding device, The image encoding device comprises: an image encoding unit configured to perform encoding processing such that, when an input image is an interlaced image, an even-numbered field becomes a Top field, an odd-numbered field becomes a Bottom field, and the Top field and the Bottom field immediately following the Top field are paired; The image decoding device comprises: An image processing system comprising an image decoding unit configured to perform decoding processing such that, when the value of sps_field_flag is 1 among control data decoded from a bit stream, an even-numbered field becomes a Top field, an odd-numbered field becomes a Bottom field, and the Top field and the Bottom field immediately following the Top field are paired.

4. A program for causing a computer to function as an image decoding device, The image decoding device comprises: A program comprising an image decoding unit configured to perform a decoding process such that, when the value of sps_field_flag is 1 among control data decoded from a bit stream, an even-numbered field becomes a Top field, an odd-numbered field becomes a Bottom field, and the Top field and the Bottom field immediately following the Top field are paired.

Images