Video encoding device, decoding device

Abstract

Conventional video decoding devices require a large amount of encoded data to encode and decode complex motion video signals with high image quality. Conversely, reducing the amount of encoded data results in a degradation of image quality. [Solution] The video decoding device 31, which takes encoded data Te as input and decodes it to output a decoded image Tb, includes a decoded picture buffer unit 307 that stores multiple decoded pictures, and an image generation AI unit 310 that generates a predicted image using the decoded picture and image generation information for generating a predicted image contained in the encoded data.

Description

[[Technical Field]]

[0001] Embodiments of the present invention relate to a moving picture encoding apparatus and a moving picture decoding apparatus. [[Background Art]]

[0002] To efficiently transmit or record moving pictures, a moving picture encoding apparatus that generates encoded data by encoding a moving picture, and a moving picture decoding apparatus that generates a decoded picture by decoding the encoded data are used.

[0003] Specific examples of moving picture encoding methods include H.264 / AVC and H.265 / HEVC (High-Efficiency Video Coding).

[0004] In such moving picture encoding methods, pictures constituting a moving picture are managed by a hierarchical structure consisting of slices obtained by dividing the picture, coding tree units (CTU: Coding Tree Unit) obtained by dividing the slices, coding units (sometimes referred to as CU: Coding Unit) obtained by dividing the coding tree units, and transform units (TU: Transform Unit) obtained by dividing the coding units, and are encoded and decoded on a CU-by-CU basis.

[0005] In addition, in such moving picture encoding methods, a prediction picture is usually generated based on a locally decoded picture obtained by encoding and decoding the input picture, and a prediction error (sometimes referred to as "difference image" or "residual image") obtained by subtracting the prediction picture from the input picture (original picture) is encoded. Examples of methods for generating a prediction picture include inter-picture prediction (inter prediction) and intra-picture prediction (intra prediction).

[0006] Non-Patent Document 1 is an example of recent moving picture encoding and decoding technology.

[0007] Non-patent document 1 describes a mechanism for managing reference picture lists in which multiple reference picture lists are defined and used by referencing them. [Prior art documents] [Non-patent literature]

[0008] [Non-Patent Document 1] Recommendation ITU-T H.266 (04 / 22) "Versatile Video Coding" [Overview of the project] [Problems that the invention aims to solve]

[0009] However, Non-Patent Document 1 stated that encoding and decoding complex motion video signals with high image quality requires a large amount of encoded data, and that reducing the amount of data results in a deterioration of image quality. [Means for solving the problem]

[0010] A video decoding device according to one aspect of the present invention, which receives encoded data as input and decodes it to output a decoded image, It has a decryption picture buffer section that stores multiple decrypted pictures, The aforementioned decrypted picture and, The system includes an image generation AI unit that generates a predicted image using image generation information for generating a predicted image included in the encoded data.

[0011] Furthermore, the encoded data that can decode a video signal according to one aspect of the present invention is: Input to the image generation AI unit that generates predicted images. Seed information for pseudorandom number generation and Information indicating which decoded picture to use in the decoded picture buffer section that stores multiple decoded pictures, Information indicating whether or not to use the predicted image generated by the aforementioned image generation AI unit for decoding, and comprising.

[0012] With such a configuration, the above problem can be solved.

Effects of the Invention

[0013] According to one aspect of the present invention, for a moving image signal with complex motion, generating a predicted image using image generation AI enables generation with image generation information of a small code amount, whereby the above problem can be solved.

Brief Description of Drawings

[0014] [Figure 1] FIG. 1 is a schematic diagram showing a configuration of an image transmission system according to the present embodiment. [Figure 2] FIG. 2 is a diagram showing a hierarchical structure of encoded data. [Figure 3] FIG. 3 is a diagram showing a hierarchical structure of encoded data in PU units. [Figure 4] FIG. 4 is a block diagram showing a configuration of a moving image decoding apparatus according to the present embodiment. [Figure 5] FIG. 5 is a block diagram showing a configuration of an image generation AI unit according to the present embodiment. [Figure 6] FIG. 6 is a diagram showing SPS syntax of image generation information according to the present embodiment. [Figure 7] FIG. 7 is a diagram showing PH syntax of image generation information according to the present embodiment. [Figure 8] FIG. 8 is a diagram showing syntax of image generation information indicating a decoded picture input to the image generation AI unit according to the present embodiment. [Figure 9] FIG. 9 is a diagram showing CU syntax of image generation information according to the present embodiment. [Figure 10] FIG. 10 is a block diagram showing a configuration of a moving image encoding apparatus according to the present embodiment.

Mode for Carrying Out the Invention

[0015] (First Embodiment) Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0016] FIG. 1 is a schematic diagram showing the configuration of an image transmission system 1 according to the present embodiment.

[0017] The image transmission system 1 includes a moving image encoding device 11, a network 21, a moving image decoding device 31, and a moving image display device 41.

[0018] An input moving image signal T is input to the moving image encoding device 11.

[0019] The network 21 transmits encoded data Te generated by the moving image encoding device 11 to the moving image decoding device 31. The network 21 is the Internet, a wide area network (WAN), a local area network (LAN), or a combination thereof. The network 21 is not necessarily limited to a two-way communication network, and may be a one-way communication network that transmits broadcast waves such as terrestrial digital broadcasting and satellite broadcasting. Further, the network 21 may be replaced with a storage medium storing the encoded data Te such as a DVD (Digital Versatile Disc: registered trademark), a BD (Blue-ray Disc: registered trademark) or the like.

[0020] The moving image decoding device 31 decodes each piece of the encoded data Te transmitted by the network 21, and supplies a decoded image Td to the moving image display device 41.

[0021] The moving image display device 41 displays all or part of the decoded image Td. The moving image display device 41 includes a display device such as a liquid crystal display or an organic EL (Electro-luminescence) display, for example. Examples of the form of the display include a stationary type, a mobile type, and an HMD. Further, when the moving image decoding device 31 has high processing capability, it displays an image with high image quality, and when it has lower processing capability, it displays an image that does not require high processing capability and display capability.

[0022] <Operator> The operators used in this specification are described below.

[0023] >> represents right bit shift, << represents left bit shift, & represents bitwise AND, | represents bitwise OR, |= represents the OR assignment operator, and || represents logical OR.

[0024] x ? y : z is a ternary operator that takes y when x is true (non-zero), and takes z when x is false (zero).

[0025] Clip3(a,b,c) is a function that clips c to a value between a and b inclusive, returns a when c<a, returns b when c>b, and returns c in other cases (provided that a<=b).

[0026] abs(a) is a function that returns the absolute value of a.

[0027] Int(a) is a function that returns the integer value of a.

[0028] Floor(a) is a function that returns the maximum integer not greater than a.

[0029] ceil(a) is a function that returns the minimum integer not less than a.

[0030] Max(a, b) is a function that returns the larger value between a and b.

[0031] Min(a, b) is a function that returns the smaller value between a and b.

[0032] a / d represents division of a by d (rounded down to the nearest integer).

[0033] a÷d and a over d represent division of a by d (no rounding down).

[0034] <Description> The descriptors used in this specification are described below.

[0035] ae(v): Descriptor for context-adaptive arithmetic entropy coding.

[0036] b(8): A byte descriptor containing a bit string (8 bits) of any pattern.

[0037] f(n): A descriptor of a fixed pattern bit sequence using n bits written from left to right, with the left bit leading. i(n): A descriptor for a signed integer using n bits.

[0038] se(v): Descriptor for a zero-order signed integer with the left bit as the leading bit in an Exp-Golomb encoding. u(n): Descriptor of an unsigned integer using n bits ue(v): Descriptor for Exp-Golomb coding of an unsigned integer with the left bit as the leading bit (encoded data) (The structure of Te) Prior to a detailed description of the image encoding device 11 and image decoding device 31 according to this embodiment, the data structure of the encoded data Te generated by the image encoding device 11 and decoded by the image decoding device 31 will be explained with reference to Figures 2 and 3.

[0039] The encoded data Te is a bitstream composed of multiple CVS (Coded Video Sequences) and EoB (End of Bitstream) NAL units, as shown in Figure 2. A CVS consists of multiple AU (Access Units) and EoS (End of Sequence) NAL units. The first AU in a CVS is called the CVSS (Coded Video Sequence Start) AU. A unit obtained by dividing a CVS by layer is called a CLVS (Coded Layer Video Sequence). An AU consists of one or more PUs (Picture Units) from the same output time. If a multilayer encoding scheme is not adopted, an AU consists of one PU. A PU is a unit of encoded data for a single decoded picture, composed of multiple NAL units. A CLVS consists of PUs from the same layer, and the first PU in a CLVS is called the CLVSS (Coded Layer Video Sequence Start) PU. CLVSS PUs are limited to PUs consisting of randomly accessible IRAP (Intra Random Access Pictures) or GDR (Gradual Decoder Refresh Picture). A NAL unit consists of a NAL unit header and RBSP (Raw Byte Sequence Payload) data. The NAL unit header consists of 2 bits of 0 data, followed by 6 bits of nuh_layer_id indicating the layer value, 5 bits of nuh_unit_type indicating the NAL unit type, and 3 bits of nuh_temporal_id_plus1 which is the Temporal ID value plus 1.

[0040] Figure 3 shows the hierarchical structure of encoded data Te at the PU level. Encoded data Te includes, exemplarily, a sequence and multiple pictures that make up the sequence. Figure 3 shows an encoded video sequence defining sequence SEQ, an encoded picture defining picture PICT, an encoded slice defining slice S, encoded slice data defining slice data, an encoded tree unit contained in the encoded slice data, and an encoded unit contained in the encoded tree unit.

[0041] In an encoded video sequence, a set of data that the image decoding device 31 references to decode the sequence SEQ to be processed is defined. As shown in Figure 3, the sequence SEQ includes a video parameter set (VPS), a sequence parameter set (SPS), a picture parameter set (PPS), an adaptation parameter set (APS), a picture (PICT), and supplemental enhancement information (SEI).

[0042] The Video Parameter Set (VPS) defines a set of encoding parameters common to multiple layers in a video composed of multiple layers, as well as a set of encoding parameters associated with the multiple layers included in the video and each individual layer.

[0043] The sequence parameter set (SPS) defines a set of encoding parameters that the image decoding device 31 references to decode the target sequence. For example, the width and height of the picture are defined. Multiple SPSs may exist. In that case, one of the SPSs is selected from the PPS.

[0044] (Picture layer) The picture layer defines the set of data that the image decoding device 31 references to decode the picture PICT to be processed. As shown in Figure 3, the picture PICT includes the picture header PH and slices 0 to NS-1 (NS being the total number of slices included in the picture PICT).

[0045] (Slice layer) The slice layer defines the set of data that the image decoding device 31 references to decode the slice S to be processed. As shown in Figure 3, the slice includes a slice header and slice data.

[0046] The slice header contains a set of encoding parameters that the image decoding device 31 references to determine the decoding method for the target slice. The slice type specification information (slice_type), which specifies the slice type, is an example of the encoding parameters included in the slice header.

[0047] The slice types that can be specified by the slice type specification information include: (1) I slices that use only intra prediction during encoding; (2) P slices that use single prediction (L0 prediction) or intra prediction during encoding; and (3) B slices that use single prediction (L0 prediction or L1 prediction), bi-prediction, or intra prediction during encoding. Note that inter-prediction is not limited to single or bi-prediction, and a prediction image may be generated using more reference pictures. Hereinafter, when referring to P slices and B slices, we mean slices that contain blocks on which inter-prediction can be used.

[0048] The slice header may also include a reference to the picture parameter set PPS (pic_parameter_set_id).

[0049] (Encoded slice data) The encoded slice data defines the set of data that the image decoding device 31 references to decode the slice data to be processed. The slice data includes CTUs, as shown in the encoded slice header of Figure 3. CTUs are fixed-size (e.g., 64x64) blocks that make up the slice.

[0050] (Code tree unit) Figure 3 defines the set of data that the image decoding device 31 references to decode the CTU to be processed. The CTU is divided into coding units CU, which are the basic units of encoding processing, by recursive quad tree partitioning (QT), binary tree partitioning (BT), or ternary tree partitioning (TT). The nodes of the tree structure obtained by recursive quad tree partitioning are called coding nodes. The intermediate nodes of the quad tree, binary tree, and ternary tree are coding nodes, and the CTU itself is defined as the top-level coding node.

[0051] (Encoding Unit) Figure 3 defines the set of data that the image decoding device 31 references to decode the encoding unit to be processed. Specifically, the CU consists of a CU header CUH, prediction parameters, transformation parameters, quantization transformation coefficients, etc. The CU header defines the prediction mode, etc.

[0052] Prediction processing may be performed at the CU (Unit) level, or at the subCU level, which is a further subdivision of the CU.

[0053] There are two types of predictions (prediction modes): intra-prediction and inter-prediction. Intra-prediction is prediction within the same picture, while inter-prediction refers to prediction processing performed between different pictures (for example, between display times or between layer images).

[0054] The transformation and quantization processes are performed in units of CUs, but the quantization transformation coefficients may be entropy-encoded in subblock units such as 4x4.

[0055] In this specification, when we refer to a "flag indicating whether or not XX is true," we mean that the flag is not 0 ( For example, 1) is considered true and 0 is considered false, and in logical negation, logical AND, etc., 1 is treated as true and 0 as false (and so on). However, in actual devices and methods, other values ​​may be used as true and false values.

[0056] (Configuration of the image decoding device) The configuration of the image decoding device 31 according to this embodiment will now be described. Figure 4 shows a block diagram of the video decoding device of this embodiment.

[0057] The video decoding device 31 receives encoded data Te, which is obtained by compressing and encoding the input video signal T, and outputs a decoded image Td. The video decoding device 31 of this embodiment consists of an entropy decoding unit 301, an inverse quantization / inverse transformation unit 302, an adder 303, a loop filter unit 304, a prediction parameter decoding unit 305, a prediction image generation unit 306, a decoded picture buffer unit 307, a generated image buffer unit 308, an image generation information decoding unit 309, and an image generation AI unit 310. The entropy decoding unit 301 decodes the encoded data that has been entropically coded with arithmetic codes or variable-length codes, and sends the decoding result to the inverse quantization / inverse transformation unit 302, the prediction parameter decoding unit 305, and the image generation information decoding unit 309 according to the syntax. The inverse quantization / inverse transformation unit 302 performs inverse quantization and inverse transformation on the transformation coefficient information decoded by the entropy decoding unit 301 to generate residual image information, which is then sent to the adder 303.

[0058] The prediction parameter decoding unit 305 also refers to the decoded picture in the decoded picture buffer unit 307 and performs template matching and other processing to generate prediction parameters decoded by the entropy decoding unit 301, and decodes prediction parameter information such as the reference picture index, motion vector, and merge index according to the prediction mode.

[0059] The image generation information decoding unit 309 generates the image generation information decoded by the entropy decoding unit 301 based on the syntax described later.

[0060] The prediction image generation unit 306 generates a prediction image from the prediction parameter information, the decoded picture from the decoded picture buffer unit 307, and the generated prediction image from the generated image buffer unit 308.

[0061] The adder 303 adds the residual information generated by the inverse quantization / inverse transform unit 302 and the predicted image generated by the predicted image generation unit 306 to create a temporary decoded picture.

[0062] The loop filter unit 304 performs loop filtering on the temporary decoded picture to create a decoded image.

[0063] The decoded picture buffer unit 307 is a buffer that stores multiple decoded pictures. The decoded pictures are referenced as needed by the prediction parameter decoding unit 305, the prediction image generation unit 306, and the image generation AI unit 310, and at the timing of display, a copy of the decoded picture stored in the buffer is output from the video decoding device 31 as the decoded image Td.

[0064] The image generation AI unit 310 generates a predicted image from the image generation information decoded by the image generation information decoding unit 309 and the decoded picture in the decoded picture buffer unit 307.

[0065] The generated image buffer unit 308 is a buffer that stores the generated predicted images generated by the image generation AI unit 310. The generated predicted images are referenced as predicted images by the predicted image generation unit 306 as needed.

[0066] (Configuration of the image generation AI unit) Figure 4 shows the image generation AI unit 310 and the video encoding device 11 within the video decoding device 31 of this embodiment. The block diagram of the internal image generation AI unit 112 is shown.

[0067] The image generation AI unit 310 takes one or more decoded pictures from the decoding buffer unit and image generation information (random number seed information, prompt information, reference picture information, etc.) as input and outputs a predicted image to be generated.

[0068] The image generation AI unit 310 consists of an image spatial analysis unit 3101, an image temporal analysis unit 3102, a conditional encoder unit 3103, a diffusion model unit 3104, and a noise generation unit 3105.

[0069] The image spatial analysis unit 3101 is a neural network that converts images into features, and it takes a decoded picture as input to generate image spatial analysis information. Specifically, it uses a CLIP (Contrastive Language-Image Pretraining) image encoder, a VAE (Variational Auto-Encoder), a neural network that obtains a depth map from an image (e.g., Depth Anything), or a combination thereof.

[0070] The image time analysis unit 3102 is a neural network that outputs optical flow and depth maps between multiple pictures. It takes multiple decoded pictures from the decoded picture buffer unit as input and outputs image time analysis information.

[0071] The condition encoder unit 3102 takes the image spatial analysis information generated by the image spatial analysis unit 3101, the prompt information from the image generation information, and the image temporal analysis information created by the image temporal analysis unit 3102 as inputs, and generates spatial and temporal condition information (keyframe attention information).

[0072] The diffusion model unit 3104 is a neural network model that generates images. It uses random number information from the noise generation unit 3105 and conditional information (keyframe attention information) obtained from the conditional encoder unit 3103 to generate a predicted image from the decoded picture.

[0073] The noise generation unit 3105 generates random number information to be input to the diffusion model unit based on the random number seed information of the image generation information.

[0074] Non-patent document 1 described the problem that encoding and decoding complex motion video signals with high image quality requires a large amount of encoded data, and reducing the amount of data results in a deterioration of image quality. With the motion video decoding device of this embodiment, the predicted image generated by the image generation AI unit 310 can be added to the prediction, making it possible to generate predicted images even for motion video signals with complex motion, thereby improving encoding efficiency.

[0075] (Syntax for image generation information) Figure 6 shows the syntax of the image generation information within seq_parameter_set_rbsp(), which is the encoded data of the SPS (Sequence Parameter Set) in this embodiment.

[0076] A value of 1 for the syntax element sps_generative_ai_pic_flag indicates that the generated predicted image generated by the image generation AI unit can be used. A value of 0 for sps_generative_ai_pic_flag indicates that the predicted image generated by the image generation AI unit cannot be used. If sps_generative_ai_pic_flag does not exist, its value is assumed to be 0.

[0077] If the value of the syntax element sps_generative_ai_pic_flag is 1, then the syntax element sps_generative_ai_seed_value exists. sps_generative_ai_seed_value indicates the random number seed information, which is represented here as a 16-bit positive integer. If sps_generative_ai_seed_value does not exist, its value is assumed to be 0.

[0078] Figure 7 shows the syntax of the image generation information of picture_header_structure(), which is the encoded data of the PH (Picture Header) in this embodiment.

[0079] If the value of the SPS syntax element sps_generative_ai_pic_flag is 1, the Picture header contains syntax for image generation information.

[0080] A value of 1 in the syntax element ph_generative_prompt_flag indicates that prompt information is present in the picture header. A value of 0 in ph_generative_prompt_flag indicates that prompt information is not present. When ph_generative_prompt_flag is 1, the encoded data is aligned to bytes using the syntax element ph_alignment_zero_bit, which is always a 1-bit value of 0. The function byte_aligned() indicates whether the position of the encoded data is in bytes or not.

[0081] The syntax element ph_generative_ai_prompt indicates the prompt information for the text input to the image generation AI unit. If ph_generative_ai_prompt exists, it must not be NULL.

[0082] A value of 1 for the syntax element ph_generative_ai_seed_update_flag indicates that the random number seed information input to the image generation AI unit in the Picture header will be updated. A value of 0 for ph_generative_ai_seed_update_flag indicates that the random number seed information input to the image generation AI unit in the Picture header will not be updated.

[0083] If the value of ph_generative_ai_seed_update_flag is 1, the syntax element ph_generative_seed_value exists. ph_generative_ai_seed_value indicates the random number seed information to be updated in the Picture header, and is shown here as a 16-bit positive integer. If ph_generative_ai_seed_value does not exist, the value of sps_generative_ai_seed_value is used as the random number seed information.

[0084] `generative_ai_ref_pic_list_struct()` is one of the image generation pieces of information (reference picture information) that specifies which decoded picture in the decoded picture buffer section should be input to the image generation AI section.

[0085] In this embodiment, we have shown a method of defining image generation information in the Picture header, but in another embodiment, image generation information may be similarly defined in the Slice header.

[0086] Figure 8 shows the syntax of the reference picture information generator_ai_ref_pic_list_struct() in this embodiment.

[0087] The syntax element num_generative_ref_entries indicates the number of decoded pictures to be input from the decoded picture buffer to the generated image AI unit.

[0088] The Picture Order Count (POC) is a variable that is one-to-one associated with each picture related to the sequence picture header, indicating the order in which the pictures are output from the decoded picture buffer. The syntax element abs_delta_poc[i] represents the absolute value of the variable DeltaPocVal. The variable DeltaPocVal represents the difference in POC values ​​in the output order from the picture currently being decoded.

[0089] The syntax element entry_sign_flag[i] indicates the sign information of abs_delta_poc[i]. If entry_sign_flag[i] is 0, it indicates that DeltaPocVal[i] is greater than or equal to 0. If entry_sign_flag[i] is 1, it indicates that DeltaPocVal[i] is less than 0. If it does not exist, the value of entry_sign_flag is assumed to be 0.

[0090] The variable DeltaPocVal is derived as follows:

[0091] for( i = 0; i < num_ref_entries; i++ ) DeltaPocVal[ i ] = ( 1 - 2 * entry_sign_flag[ i ] ) * AbsDeltaPoc[ i ] By defining the syntax for `generative_ai_ref_pic_list_struct()` in this way, it becomes possible to specify one or more decoded pictures in the decoded picture buffer section.

[0092] In this example, the variable DeltaPocVal is represented using two syntax elements: its absolute value and its sign. However, the Description of a signed integer can also be expressed using se(v).

[0093] In this embodiment, `generative_ai_ref_pic_list_struct()` was called per PH unit to define the decoded picture to be used by the image generation AI unit. However, multiple selection patterns may be defined in higher-level header syntaxes such as SPS or PPS, and then selected by index in the PH unit. Figure 9 shows the syntax of the image generation information of coding_uit(), which is the encoded data for each Coding Unit in this embodiment.

[0094] The variable CuPredMode[chType][x0][y0] indicates the prediction mode of the CU. Here, the variable chType indicates the index of the luminance and chrominance components, where 0 represents luminance Y, 1 represents chrominance Cb, and 2 represents chrominance Cr. The variables x0 and y0 indicate the horizontal and vertical addresses of the CU, respectively.

[0095] This embodiment describes the case where CuPredMode[chType][x0][y0] is not MODE_INTRA, which indicates intra mode, or MODE_PLT, which indicates palette mode, and is not DUAL_TREE_CHROMA, that is, the case of inter-mode (MODE_INTER) or intra-block copy mode (MODE_IBC).

[0096] The syntax element generative_ai_flag[ x0 ][ y0 ] is the syntax element cu_skip_flag[ x0 If the value of [y0] is 0 and the value of sps_generative_ai_flag is 1, then the syntax element generative_ai_flag[ x0 ][ y0 ] exists. If the value of generative_ai_flag[ x0 ][ y0 ] is 1, it indicates that the generated prediction image in the generated image buffer will be used as the prediction image. If the value of generative_ai_flag[ x0 ][ y0 ] is 0, it indicates that the generated prediction image in the generated image buffer will not be used as the prediction image. If generative_ai_flag[ x0 ][ y0 ] does not exist, its value is assumed to be 0.

[0097] In this embodiment, the generated predicted images in the generated image buffer units 309 and 110 contain less additional information such as motion vectors and merge indexes within the coding_unit compared to the conventional prediction mode. Therefore, specifying them with binary values, such as the syntax element generative_ai_flag[ x0 ][ y0 ], allows for very efficient encoding overall.

[0098] In another embodiment, if the value of generative_ai_flag[ x0 ][ y0 ] is 1, it indicates that the generated picture in the generated image buffer will be used as the prediction image, and if the value of generative_ai_flag[ x0 ][ y0 ] is 0, it indicates that the generated picture in the generated image buffer will not be used as the prediction image.

[0099] Furthermore, in this embodiment, we defined the use of a generated prediction image in generative_ai_flag[ x0 ][ y0 ], but in another embodiment, a generated prediction image may be used as one of the candidates for merge prediction. In this case, the generated prediction image (and its label) is stored in candList[]. The syntax element of merge_idx is decoded, and the merge candidate indicated by merge_idx (candList[merge_idx]) is determined. The predicted image is derived using ).

[0100] Non-patent document 1 described the problem that encoding and decoding complex motion video signals with high image quality requires a large amount of encoded data, and reducing the amount of data degrades image quality. According to the syntax of this embodiment, the decoded picture input to the image generation AI unit 310 or 112 and the image generation information can be efficiently defined and the generated predicted image can be added to the prediction, making it possible to generate predicted images of complex motion video signals with less image generation information, thereby improving encoding efficiency.

[0101] (Configuration of the video encoding device) Next, the configuration of the video encoding device 11 according to this embodiment will be described. Figure 4 is a block diagram showing the configuration of the video encoding device 11 according to this embodiment.

[0102] Figure 10 shows a block diagram of the video encoding device 11 of this embodiment. The video encoding device 11 receives an input video signal T, compresses and encodes it, and outputs encoded data Te. The video encoding device 11 of this embodiment consists of a conversion / quantization unit 101, an entropy encoding unit 102, an inverse quantization / inverse conversion unit 103, an adder 104, a prediction parameter creation unit 105, a prediction image generation unit 106, a decoded picture buffer unit 107, a loop filter unit 108, a prediction parameter encoding unit 109, a generated image buffer unit 110, an image generation information creation unit 111, an image generation AI unit 112, an image information encoding unit 113, and a subtractor 114.

[0103] The conversion and quantization unit 101 converts the difference signal between the input video signal T and the predicted image, and then quantizes the conversion coefficients.

[0104] The entropy coding unit 102 entropy codes the conversion coefficient information quantized by the conversion / quantization unit 101, the prediction parameter information created by the prediction parameter coding unit 108, and the image generation information created by the image generation information coding unit 112, etc., using arithmetic codes or variable-length codes based on the syntax, and creates coded data Te.

[0105] The inverse quantization / inverse transformation unit 103 performs inverse quantization and inverse transformation on the transformation coefficient information quantized by the transformation / quantization unit 101 to generate residual image information, which is then sent to the adder 104.

[0106] The prediction parameter creation unit 105 also refers to the decoded picture in the decoded picture buffer unit 107 and performs template matching processing to create prediction parameter information such as a reference picture index, motion vector, and merge index according to the prediction mode.

[0107] The prediction image generation unit 106 generates a prediction image from the prediction parameter information from the prediction parameter creation unit 105, the decoded picture from the decoded picture buffer unit 107, and the generated prediction image from the generated image buffer unit 110.

[0108] The decoded picture buffer unit 107 is a buffer that stores multiple decoded pictures, and allows the prediction parameter creation unit 105, the prediction image generation unit 106, and the image generation AI unit 112 to refer to the decoded pictures as needed.

[0109] The loop filter unit 108 performs loop filter processing on the decoded picture created by adding the residual signal and the predicted image in the adder 104, thereby creating a decoded picture.

[0110] The prediction parameter coding unit 109 creates prediction parameter information according to the syntax of the prediction parameters created by the prediction parameter creation unit 105 and sends it to the entropy coding unit 102.

[0111] The generated image buffer unit 110 is a buffer that stores the generated predicted images generated by the image generation AI unit 310. The generated predicted images are referenced as predicted images by the predicted image generation unit 306 as needed.

[0112] The image generation information creation unit 111 creates image generation information while referring to the input video signal T, and sends the image generation information to the image generation AI unit 112 and the image quality information encoding unit 113.

[0113] The image generation AI unit 112 generates a predicted image from the image generation information created by the image generation information creation unit 111 and the decoded picture in the decoded picture buffer unit 107.

[0114] The image generation information encoding unit 113 generates image generation information according to the syntax shown in Figures 6, 7, 8, and 9, and sends it to the entropy encoding unit.

[0115] The subtractor 114 generates a difference signal between the input video signal T and the predicted image generated by the predicted image generation unit 106.

[0116] Non-patent document 1 described the problem that encoding and decoding complex motion video signals with high image quality requires a large amount of encoded data, and reducing the amount of data results in a deterioration of image quality. With the motion video encoding device of this embodiment, since the predicted image generated by the image generation AI unit 110 can be added to the prediction, it becomes possible to generate predicted images even for motion video encoded signals with complex motion, thereby improving encoding efficiency.

[0117] Furthermore, some parts of the video encoding device 11 and video decoding device 31 in the above-described embodiment, such as the entropy decoding unit 301, the inverse quantization / inverse transformation unit 302, the adder 303, the loop filter unit 304, the prediction parameter decoding unit 305, the prediction image generation unit 306, the decoded picture buffer unit 307, the generated image buffer unit 308, the image generation information decoding unit 309, the image generation AI device 310, the transformation / quantization unit 101, the entropy encoding unit 102, the inverse quantization / inverse transformation unit 103, the adder 104, the prediction parameter creation unit 105, the prediction image generation unit 106, the decoded picture buffer unit 107, the loop filter unit 108, the prediction parameter encoding unit 109, the generated image buffer unit 110, the image generation information creation unit 111, the image generation AI unit 112, and the image information encoding unit 113, may be implemented using a computer. In that case, the control function may be implemented by recording a program for implementing this control function on a computer-readable recording medium, loading the program recorded on this recording medium into a computer system, and executing it. Here, "computer system" refers to a computer system built into either the video encoding device 11 or the video decoding device 31, and includes hardware such as the OS and peripheral devices. Furthermore, "computer-readable recording medium" refers to portable media such as flexible disks, magneto-optical disks, ROMs, CD-ROMs, and storage devices such as hard disks built into a computer system. Moreover, "computer-readable recording medium" may also include those that dynamically hold programs for a short period of time, such as communication lines used when transmitting programs via networks such as the Internet or communication lines such as telephone lines, and those that hold programs for a certain period of time, such as volatile memory inside a computer system that acts as a server or client in such a case. Furthermore, the above program may be for implementing only a part of the functions described above, and may also be able to implement the above functions in combination with programs already recorded in the computer system.

[0118] Furthermore, some or all of the video encoding device 11 and video decoding device 31 in the above-described embodiment may be implemented as an integrated circuit such as an LSI (Large Scale Integration). Each functional block of the video encoding device 11 and video decoding device 31 may be individually implemented as a processor, or some or all of them may be integrated into a single processor. In addition, the method of implementing the integrated circuit is not limited to LSIs; it may also be implemented using dedicated circuits or general-purpose processors. Furthermore, advances in semiconductor technology If a technology emerges that can better replace LSIs for integrated circuits, integrated circuits using that technology may be used.

[0119] Although one embodiment of this invention has been described in detail above with reference to the drawings, the specific configuration is not limited to that described above, and various design changes can be made without departing from the spirit of this invention.

[0120] [Application Examples] The video encoding device 11 and video decoding device 31 described above can be installed and used in various devices that transmit, receive, record, and play back video. The video may be natural video captured by a camera or the like, or it may be artificial video (including CG and GUI) generated by a computer or the like.

[0121] (Hardware implementation and software implementation) Furthermore, each block of the video decoding device 31 and video encoding device 11 described above may be implemented in hardware by logic circuits formed on an integrated circuit (IC chip), or it may be implemented in software using a CPU (Central Processing Unit).

[0122] The embodiments of the present invention are not limited to those described above, and various modifications are possible within the scope of the claims. That is, embodiments obtained by combining technical means that have been appropriately modified within the scope of the claims are also included in the technical scope of the present invention. [Industrial applicability]

[0123] Embodiments of the present invention can be suitably applied to a video decoding device that decodes encoded data from image data, and a video encoding device that generates encoded data from image data. Furthermore, they can be suitably applied to the data structure of encoded data generated by the video encoding device and referenced by the video decoding device. [Explanation of symbols]

[0124] 1. Image transmission system 11. Video Encoding Device 21 Network 31. Video Decoder 41. Video display device 301 Entropy Decoder 302, 103 Inverse Quantization / Inverse Transform Section 303, 104 Adder 304, 108 Loop filter section 305 Prediction parameter decoding unit 306.106 Predictive Image Generation Unit 307, 107 Decoded picture buffer section 308, 110 Generated image buffer section 309 Image generation information decoding unit 310, 112 Image generation AI device 101 Conversion / Quantization Section 102 Entropy coding unit 105 Prediction parameter creation unit 109 Prediction parameter coding unit 111 Image Generation Information Creation Unit 113 Image Information Encoding Unit 114 Subtractor 3101 Image Spatial Analysis Department 3102 Image Time Analysis Department 3103 Condition Encoder Unit 3104 Diffusion Model Section 3105 Noise generation unit

Claims

1. In a video decoding device that takes encoded data as input, decodes it, and outputs a decoded image, It has a decryption picture buffer section that stores multiple decrypted pictures, The aforementioned decrypted picture and, A video decoding device characterized by having an image generation AI unit that generates a predicted image using image generation information for generating a predicted image contained in the encoded data.

2. A video decoding device characterized in that the image generation information for generating a predicted image included in the encoded data of claim 1 includes seed information for generating pseudorandom numbers.

3. A video decoding device characterized in that the image generation information for generating a predicted image included in the encoded data of claim 1 includes information indicating which decoded picture in the decoded picture buffer unit should be used.

4. A video decoding device characterized in that the image generation information for generating a predicted image included in the encoded data of claim 1 includes information indicating whether or not to use the predicted image generated by the image generation AI unit for decoding.

5. In a video encoding device that encodes video signals and outputs encoded data, It has a decryption picture buffer section that stores multiple decrypted pictures, The aforementioned decrypted picture and, An image generation AI unit generates a predicted image using the image generation information included in the encoded data. A video encoding device characterized by having the following features.

6. In encoded data that can decode video signals, Input to the image generation AI unit that generates predicted images. Seed information for pseudorandom number generation and Information indicating which decoded picture to use in the decoded picture buffer section that stores multiple decoded pictures, Information indicating whether or not to use the predicted image generated by the aforementioned image generation AI unit for decoding, and Encoded data characterized by having the following features.

Images