Image decoding device, image encoding device, image decoding method, image encoding method

Abstract

In image generation neural networks, although it is possible to generate various images with little textual information, it is difficult to achieve fine control such as generating an image that is close to a specific image. Although it is possible to control the shape to a certain extent by inputting additional conditions, there is the problem that the relationship between the amount of input information and the quality of a generated image is not controlled and that application to image encoding and decoding is thus difficult. An image decoding device (30) according to one aspect of the present invention is characterized by comprising: a generated information decoding device (302) that decodes encoded data and creates generated information; and an image generation neural network device (301) that generates a decoded image from the generated information.

Description

Image decoding device, image encoding device, image decoding method, image encoding method

[0001] The present invention relates to an image encoding device and an image decoding device.

[0002] In order to efficiently transmit or record images, an image encoding device is used that encodes an image to generate encoded data, and an image decoding device is used that decodes the encoded data to generate a decoded image.

[0003] A specific example of an image encoding method is the H.266 / VVC (Versatile Video Coding) method.

[0004] In such traditional image coding methods, an image is divided into parts for encoding / decoding. First, a predicted image is generated based on a locally decoded image obtained by encoding an input image / decoding the encoded data. Next, the predicted image is subtracted from the input image (original image) to obtain a prediction error (sometimes called a "difference image" or "residual image"), which is then coded / decoded.

[0005] Meanwhile, a recent non-patent document 1 discloses a method called Stable Diffusion, which uses a diffusion model as an image generation method using a neural network. This method can generate an image based on text entered by a user called a prompt.

[0006] Furthermore, Non-Patent Document 2 discloses a method for generating images by controlling a neural network using additional input conditions in addition to prompts called ControlNet, which allows for the input of conditions such as edge maps, segmentation maps, and key points.

[0007] R. Rombach, A. Blattmann, D. Lorenz, P. Esser and B. Ommer, “High-Resolution Image Synthesis with Latent Diffusion Models,” Computer Vision and Pattern Recognition (CVPR) 2022. L. Zhang and M. Agrawala, “Adding Conditional Control to Text-to-Image Diffusion Models,” arXiv:2302.05543, Feb. 2023.

[0008] The method disclosed in Non-Patent Document 1 is capable of generating a variety of images using a small amount of text information, but has the problem that it is difficult to perform fine control such as making the image resemble a specific image. Therefore, it has been difficult to apply this method to image encoding and decoding.

[0009] Furthermore, the method disclosed in Non-Patent Document 2 allows for some control of the generated image by using additional condition inputs, but does not control the relationship between the amount of input information and the quality of the generated image, making it difficult to apply to image encoding and decoding.

[0010] An image decoding device according to one aspect of the present invention is characterized by having a generated information decoding device that decodes encoded data to create generated information, and an image generation neural network device that generates a decoded image from the generated information.

[0011] An image coding device according to one aspect of the present invention is characterized by comprising a generation information creation device that creates generation information for image generation, an image generation neural network device that generates an image from the generation information, a generation information coding device that codes the generation information, and a coding control device that calculates a cost value from the result of comparing the image generation result of the image generation neural network device with an input image signal and the code amount from the generation information coding device, and feeds the result back to the generation information creation device.

[0012] This configuration solves the problem of achieving fine control of image encoding and decoding with an extremely small amount of code using an image generation neural network.

[0013] FIG. 1 is a schematic diagram showing the configuration of an image transmission system according to the present embodiment; FIG. 2 is a diagram showing an example of the configuration of a generated information encoding device and a generated information decoding device; FIG. 3 is a diagram showing an example of the syntax of image information; FIG. 4 is a diagram showing an example of the syntax of text information; FIG. 5 is a diagram showing an example of the syntax of seed information; FIG. 6 is a diagram showing an example of the syntax of hint information; FIG. 7 is a diagram showing another configuration of a generated information encoding device and a generated information decoding device; FIG. 8 is a diagram showing an example of the syntax of image generation network information; FIG. 9 is a schematic diagram showing the configuration of another image transmission system according to the present embodiment; FIG. 10 is a diagram showing an example of the syntax when combining a plurality of generated images.

[0014] First Embodiment FIG. 1 is a conceptual diagram showing the configuration of an image transmission system according to this embodiment.

[0015] The image transmission system 1 comprises an image encoding device 10, a transmission network 20, an image decoding device 30, and an image display device 40.

[0016] The image encoding device 10 receives an image signal T as input and outputs encoded data Te.

[0017] The transmission network 20 transmits the encoded data Te from the image encoding device 10 to the image decoding device 30. The transmission network 20 is the Internet, a wide area network (WAN), a local area network (LAN), or a combination of these. The network 20 is not necessarily limited to a bidirectional communication network, but may also be a unidirectional communication network that transmits broadcast waves such as terrestrial digital broadcasting and satellite broadcasting. Furthermore, the transmission network 20 may be replaced by a storage medium on which the encoded data Te is recorded, such as a DVD (Digital Versatile Disc: trademark) or a BD (Blue-ray Disc: registered trademark).

[0018] The image decoding device 30 receives the coded data Te as input, outputs a decoded image Td, and sends it to the image display device 40.

[0019] The image display device 40 displays all or part of the decoded image Td output from the image decoding device 30. The image display device 40 includes a display device such as a liquid crystal display or an organic EL (Electro-luminescence) display. The display may be in the form of a stationary display, a mobile display, an HMD, or the like. Furthermore, if the image decoding device 30 has high processing power, it displays high-quality images, and if it has only low processing power, it displays images that do not require high processing power or display power.

[0020] The image coding device 10 comprises a generated information generating device 101, a coding control device 102, an image generating neural network device 103, and a generated information coding device 104.

[0021] The generated information generating device 101 receives the image signal T and a control signal from the encoding control device 102, generates generated information for image generation, and outputs it to the image generation neural network device 103 and the generated information encoding device 104.

[0022] The image generation neural network device 103 generates an image based on the generation information created by the generation information creating device 101, and sends the image generation results to the encoding control device .

[0023] The generated information encoding device 104 encodes the generated information to generate encoded data Te, and sends the code amount to the encoding control device 102 .

[0024] The encoding control device 102 calculates a cost value based on the comparison result between the image generation result of the image generation neural network device 103 and the input image signal T, and the code amount from the generated information encoding device 104, and feeds back the result (control signal) to the generated information creation device 101.

[0025] The comparison between the image generation result of the image generation neural network device 103 and the input image signal T uses an evaluation criterion D for image similarity such as mean square error, sum of absolute error, SSIM (Structural Similarity), MS-SSIM (Multi-Scale Structural Similarity), or LPIPS (Learned Perceptual Image Patch Similarity). The cost value is calculated as C=D+λR, for example, using an undetermined constant λ of Lagrange's undetermined multiplier method, the evaluation criterion D, and the code amount R of the generated information.

[0026] The image decoding device 30 comprises an image generation neural network device 301 and a generated information decoding device 302 .

[0027] The generated information decoding device 302 decodes the coded data Te based on the syntax, and sends the decoded generated information to the image generation neural network device 301 .

[0028] The image generation neural network 301 generates a decoded image Td based on the generation information decoded by the generation information decoding device 302, and sends it to the image display device 40. The image generation neural network 103 and the image generation neural network 301 are assumed to perform the same operation.

[0029] The generative information creation device 101 generates information for the image generation neural network device 103 to generate an image. Specifically, the generative information includes at least text information, seed information, and hint information. The text information is information that explains the content of the image to be generated. The seed information is a control-related parameter that is input to the generative image neural network. The hint information is image contour information that serves as a hint for the generated image, etc. The hint information may include hint information fidelity, which indicates the degree of the hint information.

[0030] Specifically, the image information indicates the following information about the image signals generated by the image generation neural network devices 103 and 301: ・Color difference format of the generated image (4:2:0, 4:2:2, 4:4:4) ・Number of pixels in the horizontal and vertical directions of the generated image ・Bit length of pixels in the generated image ・Alpha map (transparency information) of the generated image ・Top, bottom, left, and right offset values ​​of the generated image for deriving the cropped image to be output to the image display device 40 ・Image signal information described as VUI (Video Usability Information) parameters (contents defined in Recommendation ITU-T H.274 | ISO / IEC 23002-7)

[0031] The text information is information expressed as a combination of words or sentences called prompts and negative prompts that are input to the image generation neural network devices 103 and 301. A prompt is an expression related to the content of the generated image, while a negative prompt indicates an expression that does not exist in the content of the generated image.

[0032] The seed information (parameter information) is control-related information input to the image generation neural network devices 103 and 301. Specifically, it includes the following: Initial value seed (random number seed) Prompt fidelity information (e.g., CFG scale, Classifier Free Guidance scale) Scaler type information Scaler iteration number (steps) information, etc.

[0033] In addition to the above, parameters that can control the images generated by the image generation neural network devices 103 and 301 can also be used as seed information. The seed information (parameter information) may be a scalar value, or a set of scalar values ​​if there are multiple scalar values.

[0034] The seed information of the initial value is a seed value for generating an initial image of random numbers to be input to the image generation neural network devices 103 and 301. It is assumed that the image encoding device 10 and the image decoding device 30 share the same method for generating random numbers.

[0035] The fidelity information is a value that indicates the degree to which the prompt is applied, and the higher the value, the stronger the constraints imposed by the prompt. Although real values ​​between 4 and 14 can be used, CFG scale information that is an integer within a specified range (e.g., 0 to 127) can also be transmitted, and the value obtained by dividing the CFG scale information by a specified number (e.g., 8 or 10) can be input to the neural network.

[0036] The sampling method type information indicates the type of noise reduction (scaling) algorithm applied to the diffusion model. Note that when using a GAN model instead of a diffusion model, step information is not required, so no encoding or decoding is required.

[0037] The sampling repetition number (steps) information indicates the number of times the diffusion model is applied to remove noise, and is a parameter that determines how finely the image is rendered.

[0038] Hint information is extended information that serves as a hint for image generation and is input to the image generation neural network devices 103 and 301. Specifically, it includes any of the following: - Image outline information - Reduced image information (thumbnail information, rough image information) - Image skeleton (pose) information (posture information) - Color information for each region of the image (segmentation image information) - Image color histogram information - Image alpha map (transparency information) - Image depth map (depth information) Data other than the above that can control the images generated by the image generation neural network devices 103 and 301 can also be used as hint information.

[0039] Specifically, the image contour information may be a monochrome image obtained by performing contour extraction processing on the image signal T. It may be binary or multi-valued. Furthermore, the size of the original image signal T may be reduced by reducing the number of pixels in the horizontal and vertical directions.

[0040] Specifically, the reduced image information is an image obtained by performing a reduction process on the image signal T to reduce the number of pixels in the horizontal and vertical directions. For example, it may be an image of 240x135, 120x68, or 60x34, which is obtained by reducing 4K to 1 / 8, 1 / 16, 1 / 32, or 1 / 64. It may also be a square image such as 64x64 or 128x128.

[0041] Specifically, the skeletal information of an image is an image made up of skeletal information that represents the pose of a person or the like.

[0042] The region color information of an image is an image in which each region is expressed in a color that is similar to the image signal T after region (segmentation) separation processing is performed on the image signal T.

[0043] The color histogram information of an image is data that indicates the frequency of pixels in the image.

[0044] The alpha map of an image is a value that represents the transparency of the pixels of the image, with smaller values ​​representing higher transparency and larger values ​​representing higher opacity.

[0045] The depth map of an image is information that indicates the depth of the image from the screen, with smaller values ​​indicating farther away and larger values ​​indicating closer.

[0046] The image contour information, reduced image information, image skeleton information, and region color information may be encoded using a still image compression format such as PNG, JPEG, WebP, HEVC (HEIF), AVIF, JPEG-XL, etc. The information is encoded and decoded by a hint information encoder 1044 and a hint information decoder 3024, which will be described later.

[0047] The generation information creation device 101 modifies the generation information based on the cost value fed back from the encoding control device 102. The image generation neural network device 103 generates an image based on the modified generation information and sends it to the encoding control device 102. The generation information encoding device 104 encodes the modified generation information and sends the code amount to the encoding control device 102. The encoding control device 102 may calculate a cost value and update the generation information until the cost value becomes sufficiently small. The above process is repeated until an image generation result commensurate with the code amount of the generation information is obtained.

[0048] 2 shows an example of the generated information encoding device 104 and generated information decoding device 302 according to this embodiment. The generated information encoding device 104 is made up of an image information encoder 1041, a text information encoder 1042, a seed information encoder 1043, and a hint information encoder 1044. The generated information decoding device 302 is made up of an image information decoder 3021, a text information decoder 3022, a seed information decoder 3023, and a hint information decoder 3024.

[0049] As an example of an embodiment of the image information encoder 1041 and the image information decoder 3021, the syntax is shown in FIG.

[0050] image_parameter_set_rbsp() is a syntax table with a Raw Byte Sequence Payload (RBSP) structure, which is coded data containing an integer number of bytes encapsulated in a NAL unit.

[0051] The meaning of the Descriptor notation in the syntax tables below is interpreted as follows: ・b(8): Represents a byte value with an arbitrary bit string (8 bits) pattern. ・f(n): Represents a bit string with a fixed pattern using n bits written from the left bit in order (left to right). ・se(v): Represents a syntax element resulting from the 0th order Exp-Golomb encoding of a signed integer. ・st(v): Represents a string encoded in UTF-8 and terminated with a null. ・u(n): Represents an unsigned integer using n bits. When n is "v" in the syntax table, the number of bits changes depending on the values ​​of other syntax elements. ・ue(v): Represents a syntax element resulting from the 0th order Exp-Golomb encoding of an unsigned integer (left bit first).

[0052] The syntax element image_parameter_set_id indicates an identification value for reference by other syntax elements. Creation information for the same created image has the same value.

[0053] The syntax element chroma_format_idc represents the chrominance format. When the value of chroma_format_idc is 0, the format is monochrome, and the values ​​of the variables SubWidthC and SubHeightC are both 1. When the value of chroma_format_idc is 1, the format is 4:2:0, and the values ​​of the variables SubWidthC and SubHeightC are both 2. When the value of chroma_format_idc is 2, the format is 4:2:2, and the values ​​of the variables SubWidthC are 2 and 1. When the value of chroma_format_idc is 3, the format is 4:4:4, and the values ​​of the variables SubWidthC and SubHeightC are both 1. For RGB data, when the value of chroma_format_idc is 3, the luminance Y, chrominance Cb, and chrominance Cr of the 4:4:4 format are assigned in the order GBR.

[0054] The syntax element image_width represents the number of pixels in the horizontal direction for the luminance of the generated image.

[0055] The syntax element image_height represents the number of pixels in the vertical direction of the luminance of the generated image.

[0056] The value obtained by adding 8 to the syntax element bitdepth_minus8 represents the pixel bit length of the generated image.

[0057] The syntax element alpha_map_flag is a flag that indicates whether the generated image has an alpha map (transparency information) of pixel values. If the value is 1, an alpha map with the same number of pixels in the horizontal and vertical directions as the luminance of the generated image and the same pixel bit length as the generated image exists. If the value is 0, no alpha map exists. When using an alpha map with a real value between 0 and 1, the pixel value of the alpha map divided by (1<<(bitdepth_minus8+8))-1 is used.

[0058] The syntax element conformance_window_flag is a flag that is set to 1 when the generated image output by the image generation neural network device 301 is to be cropped at the top, bottom, left, and right and output to the image display device 40. When the value of conformance_window_flag is 0, the generated image output by the image generation neural network device 301 is output to the image display device 40 as is.

[0059] If the value of conformance_window_flag is 1, the syntax elements conf_win_left_offset, conf_win_right_offset, conf_win_top_offset, and conf_win_bottom_offset are encoded and decoded. If the value of conformance_window_flag is 0, the syntax elements conf_win_left_offset, conf_win_right_offset, conf_win_top_offset, and conf_win_bottom_offset are not encoded or decoded, and their values ​​are all set to 0.

[0060] The generated image sent to image display device 40 includes pixels with brightness ranging from horizontal image coordinate SubWidthC * conf_win_left_offset to image_width - (SubWidthC * conf_win_right_offset + 1) and vertical image coordinate SubHeightC * conf_win_top_offset to image_height - (SubHeightC * conf_win_bottom_offset + 1).

[0061] Note that the value of SubWidthC * (conf_win_left_offset + conf_win_right_offset) must be less than image_width, and the value of SubHeightC * (win_top_offset + conf_win_bottom_offset) must be less than image_height.

[0062] If the value of the syntax element vui_parameters_present_flag is equal to 1, it indicates that the VUI parameter vui_payload() is to be encoded and decoded. If the value of vui_parameters_present_flag is equal to 0, it is assumed that the VUI parameters are not present in the image information and are either unspecified or determined by the application by external means.

[0063] The value of the syntax element vui_payload_size_minus1 plus 1 specifies the number of bytes in the vui_payload() syntax structure. Note that the value of vui_payload_size_minus1 must be between 0 and 1023 inclusive.

[0064] The byte_aligned() function returns whether the current encoded data is in byte units. If it is not in byte units, it inserts the syntax element vui_alignment_zero_bit to adjust the bit position. The value of vui_alignment_zero_bit shall always be equal to 0.

[0065] In vui_payload(), the VUI parameters are encoded and decoded as information of the number of bytes of vui_payload_size_minus1 plus 1.

[0066] rbsp_trailing_bits() is a stuffing bit of 1 to 8 bits inserted so that the coded data of the RBSP structure is in byte units.

[0067] As an example of an embodiment of the text information encoder 1042 and the text information decoder 3022, the syntax of FIG. 4 is shown.

[0068] The text_parameter_set_rbsp() is a syntax table with a Raw Byte Sequence Payload (RBSP) structure, which is coded data containing an integer number of bytes encapsulated in a NAL unit.

[0069] The syntax element "text_parameter_set_id" indicates an identification value for reference by other syntax elements. Creation information for the same created image has the same value.

[0070] The syntax element num_prompts represents the number of prompts.

[0071] The byte_aliged() function returns whether the current encoded data is in byte units. If it is not in byte units, it adjusts the bit position by inserting the syntax element prompts_bit_equal_to_zero. The value of prompts_bit_equal_to_zero is always equal to 0.

[0072] The syntax element prompt[i] indicates the character string of the i-th prompt. The prompt is a word or sentence that represents the content of the generated image, and is part of the generation information sent to the generated image neural network devices 103 and 301.

[0073] The syntax element num_negative_prompts represents the number of negative prompts.

[0074] The byte_aliged() function returns whether the current encoded data is in byte units. If it is not in byte units, it adjusts the bit position by inserting the syntax element negative_prompts_bit_equal_to_zero. The value of negative_prompts_bit_equal_to_zero is always equal to 0.

[0075] The syntax element negative_prompt[i] indicates the character string of the i-th negative prompt. A negative prompt is a word or sentence that does not exist in the content of the generated image, and is part of the generation information sent to the generated image neural network devices 103 and 301.

[0076] rbsp_trailing_bits() is a stuffing bit of 1 to 8 bits inserted so that the coded data of the RBSP structure is in byte units.

[0077] 5 shows the syntax as an example of an embodiment of the seed information encoder 1043 and the seed information decoder 3023. seed_parameter_set_rbsp() is a syntax table with an RBSP (Raw Byte Sequence Payload) structure, and is coded data containing an integer number of bytes encapsulated in an NAL unit.

[0078] Instead of dividing the prompt into a prompt and a negative prompt, the negative prompt may be included in the prompt. In this case, the negative prompt is not encoded or decoded, and only the prompt is encoded and decoded. Also, instead of decoding the number of prompts, the prompts may be separated by a prompt delimiter (e.g., a space) and encoded and decoded as a single sentence.

[0079] The syntax element seed_parameter_set_id indicates an identification value for reference by other syntax elements. The same value is used for generation information for the same generated image.

[0080] The syntax element seed_value is an integer value of 0 or greater.

[0081] The syntax element cfg_scale indicates the prompt fidelity and is an integer value equal to or greater than 0. When expressed as a real number, it can be expressed as an integer multiplied by 10, for example, and the cfg_scale value can be divided by 10. Note that it may not be called cfg_scale, but may be called prompt_fidelity_value (prompt fidelity).

[0082] The syntax element sampler_idc is an integer value equal to or greater than 0. Specifically, the value 0 is an index value indicating the scaler type, such as DDIM (Denoising Diffusion Implicit Models), 1 is PLMS, 2 is LMS, 3 is Heun, 4 is DPM2, 5 is DPM2a, 6 is Euler, and 7 is Eulera. Note that sampling methods other than these methods may be used, and additional methods may also be used.

[0083] The syntax element sampler_steps is an integer value equal to or greater than 0.

[0084] rbsp_trailing_bits() is a stuffing bit of 1 to 8 bits inserted so that the coded data of the RBSP structure is in byte units.

[0085] As an example of an embodiment of the hint encoder 1044 and the hint decoder 3024, the syntax of FIG. 6 is shown.

[0086] hint_parameter_set_rbsp() is a syntax table with a Raw Byte Sequence Payload (RBSP) structure, which is coded data containing an integer number of bytes encapsulated in a NAL unit.

[0087] The syntax element hint_parameter_set_id indicates an identification value for reference by other syntax elements. The same value is used for generation information for the same generated image.

[0088] The syntax element outline_hint_flag is a flag indicating the presence of image outline information, and if the value is 1, coded image outline information is present. If the value of outline_hint_flag is 0, image outline information is not present.

[0089] The syntax element image_hint_flag is a flag indicating the presence of reduced image information, and if the value is 1, encoded reduced image information exists. If the value of image_hint_flag is 0, reduced image information does not exist.

[0090] The syntax element pose_hint_flag is a flag indicating the presence of image skeleton information, and if the value is 1, encoded image skeleton information exists. If the value of pose_hint_flag is 0, image skeleton information does not exist.

[0091] The syntax element segmentation_hint_flag is a flag indicating the presence of image region color information, and if the value is 1, coded image region color information exists. If the value of segmentation_hint_flag is 0, image region color information does not exist.

[0092] If the value of outline_hint_flag is 1, image outline information is present and the syntax element outline_payload_size is coded and decoded. outline_payload_size specifies the number of bytes in outline_payload_byte.

[0093] The byte_aligned() function returns whether the current encoded data is in byte units. If it is not in byte units, it inserts the syntax element outline_alignment_zero_bit to adjust the bit position. The value of outline_alignment_zero_bit shall always be equal to 0.

[0094] outline_payload_byte[i] is coded and decoded as coded data of image outline information of the number of bytes specified by outline_payload_size.

[0095] The hint information encoder 1044 encodes the image contour information using an existing image coding method, such as JPEG (ITU-T T.81 | ISO / IEC 109181-1), JPEG2000 (ITU-T T.800 | ISO / IEC 15444-1), AVC (ITU-T H.264 | ISO / IEC 14496-10), HEVC (ITU-T H.265 | ISO / IEC 23008-2), VVC (ITU-T H.266 | ISO / IEC 23090-3), or PNG (Portable Network Graphics, ISO / IEC 15948), and the resulting code is decoded by the hint information decoder 3024.

[0096] The syntax element outline_fidelity_value is an integer value that represents the fidelity of image outline information.

[0097] If the value of image_hint_flag is 1, reduced image information exists and the syntax element image_payload_size is encoded and decoded. image_payload_size specifies the number of bytes in image_payload_byte.

[0098] The byte_aligned() function returns whether the current encoded data is in byte units. If it is not in byte units, it inserts the syntax element image_alignment_zero_bit to adjust the bit position. The value of image_alignment_zero_bit shall always be equal to 0.

[0099] image_payload_byte[i] is coded and decoded as coded data of reduced image information of the number of bytes of image_payload_size.

[0100] The hint information encoder 1044 encodes the reduced image information using an existing image encoding method such as JPEG, JPEG2000, AVC, HEVC, VVC, PNG, etc., and the hint information decoder 3024 decodes the encoded image information.

[0101] The syntax element image_fidelity_value is an integer value that represents the fidelity of the reduced image information.

[0102] If the value of pose_hint_flag is 1, image skeleton information exists and the syntax element pose_payload_size is encoded and decoded. pose_payload_size specifies the number of bytes in pose_payload_byte.

[0103] The byte_aligned() function returns whether the current encoded data is in byte units. If it is not in byte units, it inserts the syntax element pose_alignment_zero_bit to adjust the bit position. The value of pose_alignment_zero_bit is always equal to 0.

[0104] Pose_payload_byte[i] is coded and decoded as coded data of image skeleton information of the number of bytes specified by pose_payload_size.

[0105] The hint information encoder 1044 encodes the image skeleton information using an existing image encoding method such as JPEG, JPEG2000, AVC, HEVC, VVC, PNG, etc., and the hint information decoder 3024 decodes it.

[0106] The syntax element pose_fidelity_value is an integer value that represents the fidelity of the image skeleton information.

[0107] If the value of segmentation_hint_flag is 1, image region color information is present and the syntax element segmentation_payload_size is coded and decoded. The segmentation_payload_size specifies the number of bytes in segmentation_payload_byte.

[0108] The byte_aligned() function returns whether the current encoded data is aligned to the byte. If it is not aligned to the byte, it inserts the syntax element segmentation_alignment_zero_bit to adjust the bit position. The value of segmentation_alignment_zero_bit is always equal to 0.

[0109] Segmentation_payload_byte[i] is coded and decoded as coded data of image region color information of the number of bytes specified by segmentation_payload_size.

[0110] The hint information encoder 1044 encodes the image region color information using an existing image encoding method such as JPEG, JPEG2000, AVC, HEVC, VVC, PNG, etc., and the hint information decoder 3024 decodes it.

[0111] The syntax element segmentation_fidelity_value is an integer value that represents the fidelity of the image region color information.

[0112] Although not shown, the syntax element color_hint_flag is a flag indicating the presence of color histogram information; a value of 1 indicates that color histogram information is present. A value of 0 indicates that image region color information is not present. When color_hint_flag = 1, color_histgram_val[i] (e.g., i = 0..255) is encoded and decoded using ue(v). color_histgram_val indicates, for example, the frequency of color (e.g., luminance) when the generated image is expressed as 8 bits with a pixel bit length of 0-255. It can also be expressed as RGB or YUV frequency, as color_histgram_val[cIdx][i] (e.g., cIdx = 0..2, i = 0..255). Furthermore, a coarser color range, such as 0..15, is acceptable, not limited to 0-255. Also, since adjacent histogram values ​​are often close to each other, color_histgram_val may be the difference between adjacent histogram values ​​(differential histogram value). In this case, the histogram value is derived from the previous histogram value and the transmitted differential histogram value.

[0113] rbsp_trailing_bits() is a stuffing bit of 1 to 8 bits inserted so that the coded data of the RBSP structure is in byte units.

[0114] 7 shows another example of the generated information encoding device 104 and generated information decoding device 302 according to this embodiment. The generated information encoding device 104 is made up of an image information encoder 1041, a text information encoder 1042, a seed information encoder 1043, a hint information encoder 1044, and an image generation neural network encoder 1045. The generated information decoding device 302 is made up of an image information decoder 3021, a text information decoder 3022, a seed information decoder 3023, a hint information decoder 3024, and an image generation neural network decoder 3025.

[0115] The difference from the configuration in Figure 2 is that the image generation network itself is encoded by image generation neural network encoder 1045, sent from image encoding device 10 to image decoding device 30, and decoded by image generation neural network decoder 3025. This allows the image generation neural network devices 103 and 301 to be dynamically changed.

[0116] The image generation neural network encoder 1045 and the image generation neural network decoder 3025 use a technique such as NNC (Neural Network Coding / ISO / IEC 15938-17) to encode and decode the image generation neural network in whole or in part.

[0117] Furthermore, the image generation neural network encoder 1045 and the image generation neural network decoder 3025 are provided with a plurality of pairs of identical image generation neural network devices 103 and 301 in the image encoding device 10 and the image decoding device 30. Index values ​​indicating these plurality of pairs of image generation network devices may then be encoded and decoded.

[0118] Furthermore, the image generation neural network encoder 1045 encodes a uniform resource identifier (URI) indicating the image generation neural network device, and the image generation neural network decoder 3025 decodes the URI. The image decoding device 30 identifies the image generation neural network device 301 from the URI.

[0119] Specifically, a plurality of pairs of identical image generation neural network devices 103 and 301 are prepared in the image encoding device 10 and the image decoding device 30, and a URI is used as an index value indicating the plurality of pairs of image generation neural network devices.

[0120] As another example, the image generating neural network device may be downloaded as data encoded by NNC or the like from an external website indicated by a URI and used by the image encoding device 10 and the image decoding device 30, respectively.

[0121] As an example of an embodiment of the image generating neural network encoder 1045 and the image generating neural network decoder 3025, the syntax of FIG. 8 is shown.

[0122] nn_parameter_set_rbsp() is a syntax table with a Raw Byte Sequence Payload (RBSP) structure, which is coded data containing an integer number of bytes encapsulated in a NAL unit.

[0123] The syntax element nn_parameter_set_id indicates an identification value for reference by other syntax elements. Creation information for the same created image has the same value.

[0124] The syntax element nn_mode_idc is a value that indicates the format in which the image generation network information is encoded and decoded. A value of 1 indicates that the image generation network information is specified by a URI, and a value of 0 indicates that the image generation network information is encoded and decoded by an NNC.

[0125] If the value of nn_mode_idc is 1, a URI specifying image generation neural network information exists.

[0126] The byte_aligned() function returns whether the current encoded data is in byte units. If it is not in byte units, it inserts the syntax element nn_alignment_zero_bit_a to adjust the bit position. The value of nn_alignment_zero_bit_a is always equal to 0.

[0127] The syntax element nn_tag_uri contains a tag URI with syntax and semantics specified in IETF RFC 4151, identifying the format and related information about the image-generating neural network used. If nn_tag_uri is equal to "tag:iso.org,2023:15938-17", it indicates that the encoded data for the image-generating neural network identified by nn_uri conforms to NNC (ISO / IEC 15938-17).

[0128] The syntax element nn_uri contains a URI with syntax and semantics specified in IETF Internet Standard 66 that identifies the image-generating neural network to be used.

[0129] If the value of nn_mode_idc is 0, it indicates that the image generation network information is encoded and decoded by the NNC, and the syntax element nn_payload_size is encoded and decoded. nn_payload_size specifies the number of bytes in nn_payload_byte.

[0130] The byte_aligned() function returns whether the current encoded data is in byte units. If it is not in byte units, it inserts the syntax element nn_alignment_zero_bit_b to adjust the bit position. The value of nn_alignment_zero_bit_b is always equal to 0.

[0131] nn_payload_byte[i] is coded and decoded as coded data of the generated image neural network of the number of bytes of nn_payload_size.

[0132] rbsp_trailing_bits() is a stuffing bit of 1 to 8 bits inserted so that the coded data of the RBSP structure is in byte units.

[0133] In this embodiment, it has been shown that an image transmission system can be realized using an image generation neural network.

[0134] Second Embodiment FIG. 9 is a conceptual diagram showing the configuration of another image transmission system according to this embodiment.

[0135] In this embodiment, the image transmission system enhances the confidentiality of the encoded data by separately encrypting seed information that is part of the generated information.

[0136] The image transmission system is composed of an image encoding device 10, a transmission network 20, an image decoding device 30, an image display device 40, a seed information encryption device 50, and a seed information decoding device 60.

[0137] The image encoding device 10 receives an image signal T as input and outputs encoded data Te.

[0138] The transmission network 20 transmits the encoded data Te from the image encoding device 10 to the image decoding device 30. The transmission network 20 is the Internet, a wide area network (WAN), a local area network (LAN), or a combination of these. The network 20 is not necessarily limited to a bidirectional communication network, but may also be a unidirectional communication network that transmits broadcast waves such as terrestrial digital broadcasting and satellite broadcasting. Furthermore, the transmission network 20 may be replaced by a storage medium on which the encoded data Te is recorded, such as a DVD (Digital Versatile Disc: trademark) or a BD (Blue-ray Disc: registered trademark).

[0139] The image decoding device 30 receives the coded data Te as input, outputs a decoded image Td, and sends it to the image display device 40.

[0140] The image display device 40 displays all or part of the decoded image Td output from the image decoding device 30. The image display device 40 includes a display device such as a liquid crystal display or an organic EL (Electro-luminescence) display. The display may be in the form of a stationary display, a mobile display, an HMD, or the like. Furthermore, if the image decoding device 30 has high processing power, it displays high-quality images, and if it has only low processing power, it displays images that do not require high processing power or display power.

[0141] The image coding device 10 comprises a generated information creating device 101, a coding control device 102, an image generating neural network device 103, and a generated image coding device 104.

[0142] The generated information creating device 101 receives the image signal T and a control signal from the encoding control device 102, creates generated information for image generation, and outputs it to the image generation neural network device 103 and the generated information encoding device 104. It also sends seed information, which is part of the generated information, to the seed information encryption device 50.

[0143] The image generation neural network device 103 generates an image based on the generation information created by the generation information creating device 101, and sends the image generation results to the encoding control device .

[0144] The generated information encoding device 104 encodes the generated information to generate encoded data Te, and sends the code amount to the encoding control device 102 .

[0145] The image decoding device 30 comprises an image generation neural network device 301 and a generated information decoding device 302 .

[0146] The encoding control device 102 receives the image generation result from the image generation neural network device 103 , the image signal T, and the code amount from the generated information encoding device 104 , and outputs a control signal to the generated information creating device 101 .

[0147] The seed information encryption device 50 encrypts the seed information and transmits the encrypted seed information. At the same time, it sends an encryption key (encrypted data) to the seed information decryption device 60 via a route separate from the transmission network used to transmit prompts, etc. The encryption may be performed using a common key system or a public key system. In the case of a common key system, the encryption key itself is transmitted, while in the case of a public key system, the encryption key is not transmitted, and seed information encrypted with a public key corresponding to the private key used by the destination seed information decryption device 60 is sent. Seed information encrypted with a public key can only be decrypted by a device that has the corresponding private key. The encrypted seed information itself may also be transmitted via a route separate from the transmission network.

[0148] It is also possible to encrypt and decrypt only a portion of the seed information, for example, the seed_value in FIG. 5, without encrypting the entire seed information.

[0149] The generated information decoding device 302 decodes the coded data Te based on the syntax, and sends the decoded generated information to the image generation neural network device 301 .

[0150] The seed information decryption device 60 decrypts the encrypted seed information using the encrypted data, and sends the obtained seed information to the image generation neural network device 301 .

[0151] The image decoding device 30 comprises an image generation neural network device 301 and a generated information decoding device 302 .

[0152] The image generation neural network device 301 generates a decoded image Td based on the generation information decoded by the generation information decoding device 302 and the seed information decoding device 60, and sends it to the image display device 40.

[0153] The generated information decoding device 302 decodes the coded data Te based on the syntax and sends the decoded generated information to the image generation network 301 .

[0154] It is assumed that the image generating neural network device 103 and the image generating neural network device 301 operate in the same manner.

[0155] In this embodiment, the confidentiality of the encoded data is improved by encrypting seed information, which is part of the generated information. Without knowing the seed information, the image generation neural network device 301 of the image decoding device 30 cannot generate an accurate image.

[0156] In this embodiment, the seed information is encrypted, but the text information or hint information may be encrypted and decrypted, and the same effect can be obtained.

[0157] Third Embodiment FIG. 10 is a conceptual diagram showing the configuration of another image transmission system according to this embodiment.

[0158] In this embodiment, the image transmission system aims to further improve performance by using a plurality of image generation neural networks.

[0159] Here we show an example using two image generation neural networks.

[0160] The image transmission system comprises an image encoding device 10, a transmission network 20, an image decoding device 30, and an image display device 40.

[0161] The image encoding device 10 receives an image signal T as input and outputs encoded data Te.

[0162] The transmission network 20 transmits the encoded data Te from the image encoding device 10 to the image decoding device 30. The transmission network 20 is the Internet, a wide area network (WAN), a local area network (LAN), or a combination of these. The network 20 is not necessarily limited to a bidirectional communication network, but may also be a unidirectional communication network that transmits broadcast waves such as terrestrial digital broadcasting and satellite broadcasting. Furthermore, the transmission network 20 may be replaced by a storage medium on which the encoded data Te is recorded, such as a DVD (Digital Versatile Disc: trademark) or a BD (Blue-ray Disc: registered trademark).

[0163] The image decoding device 30 receives the coded data Te as input, outputs a decoded image Td, and sends it to the image display device 40.

[0164] The image display device 40 displays all or part of the decoded image Td output from the image decoding device 30. The image display device 40 includes a display device such as a liquid crystal display or an organic EL (Electro-luminescence) display. The display may be in the form of a stationary display, a mobile display, an HMD, or the like. Furthermore, if the image decoding device 30 has high processing power, it displays high-quality images, and if it has only low processing power, it displays images that do not require high processing power or display power.

[0165] The image coding device 10 comprises a generated information creating device 101, a coding control device 102, a first image generating neural network device 1030, a second image generating neural network device 1031, a generated image coding device 104, and an image synthesis device 105.

[0166] The generated information creation device 101 inputs the image signal T and a control signal from the encoding control device 102, creates generated information for image generation, and outputs it to the first image generation neural network device 1030, the second image generation neural network device 1031, and the generated information encoding device 104.

[0167] First image generation neural network device 1030 and second image generation neural network device 1031 perform image generation based on the generation information created by generation information creation device 101 .

[0168] The image synthesizer 105 synthesizes the two outputs and sends the image generation result to the encoding control device 102 .

[0169] The generated information encoding device 104 encodes the generated information to generate encoded data Te, and sends the code amount to the encoding control device 102 .

[0170] The encoding control device 102 receives the image generation result from the image synthesis device 105, the image signal T, and the code amount from the generated information encoding device 104, and outputs a control signal to the generated information creation device 101. The image decoding device 30 is made up of a first image generation neural network device 3030, a second image generation neural network device 3031, and a generated information decoding device 302.

[0171] The generated information decoding device 302 decodes the coded data Te based on the syntax and sends the decoded generated information to the first image generation neural network device 3030 and the second image generation neural network device 3031. The generated information decoding device 302 decodes the seed information, hint information, etc., as well as the background information and synthesis information.

[0172] The first image generation neural network device 3030 and the second image generation neural network device 3031 perform image generation based on the generated information decoded by the generated information decoding device 302 .

[0173] The image synthesizer 105 synthesizes the two outputs to generate a decoded image Td, which is sent to the image display device 40.

[0174] It is assumed that the first image generating neural network device 1030 and the first image generating neural network device 3010 perform the same operation, and the second image generating neural network device 1031 and the second image generating neural network device 3011 perform the same operation.

[0175] Also, it is assumed that the image synthesis device 105 and the image synthesis device 303 perform the same operation.

[0176] Although an example of two image generating neural networks is shown, more than two may be used.

[0177] FIG. 11 shows an example of syntax for combining multiple generated images.

[0178] generative_image_composite_rbsp() is a syntax table with a Raw Byte Sequence Payload (RBSP) structure, which is coded data containing an integer number of bytes encapsulated in a NAL unit.

[0179] The syntax element background_image_width is the number of pixels in the horizontal direction for the brightness of the background image.

[0180] The syntax element background_image_height is the number of pixels in the vertical direction of the brightness of the background image.

[0181] The syntax element background_image_flag is a flag indicating that the background image is to be coded and decoded as coded data separate from the generated image.

[0182] If the value of background_image_flag is 1, a background image exists and the syntax element background_payload_size is coded and decoded. background_payload_size specifies the number of bytes in background_payload_byte.

[0183] The byte_aligned() function returns whether the current encoded data is in byte units. If it is not in byte units, it inserts the syntax element background_alignment_zero_bit to adjust the bit position. The value of background_alignment_zero_bit shall always be equal to 0.

[0184] background_payload_byte[i] is coded and decoded as coded data of the background image having the number of bytes specified by background_payload_size.

[0185] The background image is encoded and decoded using an existing image encoding method, such as JPEG, JPEG2000, AVC, HEVC, VVC, PNG, or the like.

[0186] The syntax element composite_idc is mode information that indicates how the generated images are composited. A value of 0 indicates that the average value of the generated images, excluding the background image, is overwritten on the background image; a value of 1 indicates that the generated image with the larger parameter_set_id value is used for overwriting; and a value of 2 indicates that the generated image with the smaller parameter_set_id value is used for overwriting. If the generated image includes an alpha map or if the alpha map is encoded and decoded as hint information, the alpha map value (pixel transparency information) may be used as a weighting coefficient to composite the background and generated image.

[0187] Furthermore, when a depth map is encoded and decoded using hint information, the generated image may be synthesized based on the depth information of the generated image.

[0188] The syntax element target_image_x_coordinate represents the horizontal pixel position from the top left corner of the background image, and the syntax element target_image_y_coordinate represents the vertical pixel position from the top left corner of the background image. The generated image with parameter_set_id i is positioned so that its top left corner is aligned with the top left corner of the background image.

[0189] Note that when these processes are performed, the chroma_format_idc and bitdepth_minus8 of all generated images must be the same.

[0190] rbsp_trailing_bits() is a stuffing bit of 1 to 8 bits inserted so that the coded data of the RBSP structure is in byte units.

[0191] In this embodiment, performance can be further improved by combining the image generation of a plurality of image generation neural networks.

[0192] Note that part or all of the image encoding device 10 and the image decoding device 30 in the first, second, and third embodiments described above may be implemented by a computer. In this case, a program for implementing the control functions may be recorded on a computer-readable recording medium, and the program may be loaded into a computer system and executed. Note that the term "computer system" as used herein refers to a computer system built into either the image encoding device 10 or the image decoding device 30, and includes hardware such as an OS and peripheral devices. Furthermore, the term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into a computer system. Furthermore, the term "computer-readable recording medium" may also include media that dynamically store programs for a short period of time, such as communication lines used when transmitting programs via networks such as the Internet or telephone lines, or media that store programs for a fixed period of time, such as volatile memory within a computer system that serves as a server or client in such cases. Furthermore, the program may be for implementing part of the functions described above, or may be capable of implementing the functions described above in combination with a program already recorded in the computer system.

[0193] Furthermore, part or all of the image encoding device 10 and the image decoding device 30 in the above-described embodiments may be realized as an integrated circuit such as an LSI (Large Scale Integration). Each functional block of the image encoding device 10 and the image decoding device 30 may be individually implemented as a processor, or part or all of them may be integrated into a processor. Furthermore, the integrated circuit implementation method is not limited to LSI, and may be implemented using a dedicated circuit or a general-purpose processor. Furthermore, if an integrated circuit implementation technology that can replace LSI emerges due to advances in semiconductor technology, an integrated circuit based on that technology may be used.

[0194] One embodiment of the present invention has been described in detail above with reference to the drawings, but the specific configuration is not limited to that described above, and various design changes and the like are possible within the scope that does not deviate from the gist of the present invention.

[0195] The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. In other words, embodiments obtained by combining technical means modified appropriately within the scope of the claims are also included in the technical scope of the present invention.

[0196]

[0009] Embodiments of the present invention can be suitably applied to an image decoding device that decodes coded data in which image data has been coded, and an image coding device that generates coded data in which image data has been coded. The present invention can also be suitably applied to the data structure of coded data generated by an image coding device and referenced by the image decoding device. (Cross-Reference to Related Applications) This application claims the benefit of priority to Japanese Patent Application No. 2023-144898, filed on September 7, 2023, the entire contents of which are incorporated herein by reference.

[0197] 1 Image transmission system 10 Image encoding device 20 Transmission network 30 Image decoding device 101 Generated information creation device 102 Encoding control device 103, 301 Image generation neural network device 1030, 3010 First image generation neural network device 1031, 3011 Second image generation neural network device 104 Generated information encoding device 1041 Image information encoder 1042 Text information encoder 1043 Seed information encoder 1044 Hint information encoder 1045 Image generation neural network encoder 105 Image synthesis device 302 Generated information decoding device 3021 Image information decoder 3022 Text information decoder 3023 Seed information decoder 3024 Hint information decoder 3025 Image generation neural network decoder 40 Image display device 50 Seed information encryption device 60 Seed information decoding device

Claims

1. An image decoding device comprising: a generated information decoding device that decodes encoded data to create generated information; and an image generation neural network device that generates a decoded image from the generated information.

2. The image decoding device of claim 1, characterized in that the generation information includes text information describing the content of the image to be generated, numerical seed information that serves as a seed for generating random numbers that serve as initial values ​​for the generated image, and image hint information that serves as a hint for the generated image.

3. An image decoding device according to claim 2, wherein the hint information includes at least one of contour information, reduced image information and skeletal information.

4. An image coding device comprising: a generation information creation device that creates generation information for image generation; an image generation neural network device that generates an image from the generation information; a generation information coding device that codes the generation information; and a coding control device that calculates a cost value from a comparison result between the image generation result of the image generation neural network device and an input image signal and the code amount from the generation information coding device, and feeds the result back to the generation information creation device.

5. The image encoding device of claim 4, characterized in that the generation information includes text information describing the contents of the image to be generated, numerical seed information that serves as a seed for generating random numbers that serve as initial values ​​for the generated image, and image hint information that serves as a hint for the generated image.

6. An image decoding method comprising: a generated information decoding step of decoding encoded data to generate generated information; and an image generating neural network step of generating a decoded image from the generated information.

7. An image encoding method comprising: a generation information creation step for creating generation information for image generation; an image generation neural network step for generating an image from the generation information; a generation information encoding step for encoding the generation information; and an encoding control step for calculating a cost value from a comparison result between the image generation result of the image generation neural network step and an input image signal and the code amount from the generation information encoding step, and feeding the result back to the generation information creation step.

Images