Video encoding device and decoding device

Abstract

To solve the problem of how to define a model and a control parameter required for defining a prompt being text information and other processing when neural network image processing using a generation AI is applied.SOLUTION: A video decoding device according to the present invention includes an image decoding device that decodes encoded data of an image signal, a generation information decoding device that decodes generation information from generated encoded data, a tag URI that identifies a format of the generation information, information acquired from the URI that identifies the generation information, and image information decoded by the image decoding device, and an image generating device that generates an image from the image information decoded by the image decoding device and the generation information decoded by the generation information decoding device.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a video encoding device and a video decoding device. [Background technology]

[0002] In order to efficiently transmit or record moving images, a moving image encoding device is used that generates encoded data by encoding an image, and a moving image decoding device is used that generates a decoded image by decoding the encoded data.

[0003] A specific example of a video encoding method is the H.266 / VVC (Versatile Video Coding) method.

[0004] In such traditional image coding methods, an image is divided into parts for encoding / decoding. First, a predicted image is generated based on a locally decoded image obtained by encoding an input image / decoding the encoded data. Next, the predicted image is subtracted from the input image (original image) to obtain a prediction error (sometimes called a "difference image" or "residual image"), which is then coded / decoded.

[0005] Recently, a generative AI technique called Stable Diffusion, which uses a diffusion model as an image generation method using a neural network, has been disclosed. This technique can generate images based on text entered by the user, called a prompt.

[0006] Furthermore, Non-Patent Document 1 defines a Supplemental Enhancement Information (SEI) message as a video encoding and decoding technique for transmitting image properties, display methods, timing, etc. simultaneously with encoded data. It also presents a Neural-Network Post-filter Activation SEI message that indicates the application of post-filter processing based on a neural network.

[0007] In Non-Patent Document 2, the method of Non-Patent Document 1 is extended to propose an SEI message that can target the purpose of any application. [Prior art documents] [Non-patent literature]

[0008] [Non-Patent Document 1] ITU-T Rec. H.274 V3 "Versatile supplemental enhancement information messages for coded video bitstreams" [Non-patent document 2] J. Boyce, J. Chen, S. Deshpande, MM Hannuksela, Hendry, S. McCarthy, GJ Sullivan and Y.-K. Wang, “Additional SEI messages for VSEI version 4 (Draft 2),” JVET Document, JVET-AH2006-v1, April 2024. Summary of the Invention [Problem to be solved by the invention]

[0009] The method disclosed in Non-Patent Document 1 does not support neural network image processing by AI generated from text information prompted by a prompt. The method disclosed in Non-Patent Document 2 makes it possible to define the purpose of any application, but there is a problem in that it is not clear how to define the information required by that specific application. For example, when applying neural network image processing using generative AI, prompts, which are text information, and other models and control parameters are required to define the processing, but it was unclear how to define them.

[0010] In Non-Patent Document 2, there is a problem in that the syntax elements that define the purpose of an application are not byte-aligned, even though they are string information.

[0011] Furthermore, in Non-Patent Document 1 and Non-Patent Document 2, there is a problem in that the Neural Network Post-Filter Activation SEI message that defines the application of a neural network cannot have its syntax extended. [Means for solving the problem]

[0012] A video decoding device according to one aspect of the present invention is characterized by having an image decoding device that decodes encoded data of an image signal, a generated information decoding device that decodes generated information from generated encoded data, a tag URI that identifies the format of the generated information, information obtained from the URI that identifies the generated information, and image information decoded by the image decoding device, and an image generating device that generates an image from the image information decoded by the image decoding device and the generated information decoded by the generated information decoding device. [Effects of the Invention]

[0013] By adopting such a configuration, it is possible to solve the problem of efficiently encoding and decoding moving images using an image generation technique. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a schematic diagram showing the configuration of an image transmission system according to an embodiment of the present invention. [Figure 2] FIG. 1 is a block diagram illustrating an example of an image generation processing apparatus according to an embodiment of the present invention. [Figure 3] FIG. 1 is a diagram illustrating an example of a block diagram of a generation information creating device according to an embodiment of the present invention. [Figure 4] FIG. 1 is a diagram illustrating an example of a block diagram of a generated information encoding device according to an embodiment of the present invention. [Figure 5] FIG. 2 is a diagram illustrating an example of a block diagram of a generated information decoding device according to the present embodiment. [Figure 6] FIG. 10 is a diagram showing the syntax of the NNPFC SEI message described in Non-Patent Document 2. [Figure 7] A figure showing an example of an extension of the NNPFC SEI message in this embodiment. [Figure 8] A figure showing an example of another extension of the NNPFC SEI message in this embodiment. [Figure 9] FIG. 10 is a diagram showing an example of an extension of the NNPFAE SEI message according to the present embodiment. [Figure 10] A figure showing an example of another extension of the NNPFC SEI message in this embodiment. [Figure 11] A figure showing an example of another extension of the NNPFC SEI message in this embodiment. [Figure 12] A figure showing an example of another extension of the NNPFC SEI message in this embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0015] (First embodiment) FIG. 1 is a conceptual diagram showing the configuration of an image transmission system according to this embodiment.

[0016] The image transmission system 1 comprises a video encoding device 10, a transmission network 20, a video decoding device 30, and an image display device 40.

[0017] The video encoding device 10 receives an input image signal T and outputs encoded data Te.

[0018] The transmission network 20 transmits the coded data Te from the video coding device 10 to the video decoding device 30. The transmission network 20 is the Internet, a wide area network (WAN), a local area network (LAN), or a combination of these. The network 20 is not necessarily limited to a two-way communication network, and may be a terrestrial digital The transmission network 20 may be a one-way communication network that transmits broadcast waves such as digital broadcasting, satellite broadcasting, etc. Furthermore, the transmission network 20 may be replaced by a storage medium on which encoded data Te is recorded, such as a DVD (Digital Versatile Disc: trademark) or a BD (Blue-ray Disc: registered trademark).

[0019] The video decoding device 30 receives the coded data Te as input, outputs a generated image Td, and sends it to the image display device 40.

[0020] The image display device 40 displays all or part of the generated image Td output from the video decoding device 30. The image display device 40 includes a display device such as a liquid crystal display or an organic EL (Electro-luminescence) display. The display may be in the form of a stationary display, a mobile display, an HMD, or the like. Furthermore, if the video decoding device 30 has high processing power, it displays high-quality images, and if it has only low processing power, it displays images that do not require high processing power or display power.

[0021] The video encoding device 10 comprises an image encoding device 101, a generated information creating device 102, and a generated information encoding device 103.

[0022] The image encoding device 101 encodes an input image signal T to generate encoded data Te, and sends the decoded image information to the generated information creating device 102 .

[0023] The generated information creating device 102 receives the input image signal T, external model data, and decoded image information from the video encoding device, creates generated information, and sends it to the generated information encoding device 103 .

[0024] The generated information encoding device 103 encodes the generated information and saves the generated information as URI data in a specified URI (Uniform Resource Identifier) ​​on a server or a specific storage location on a network, and generates supplemental extension information encoded data including the URI. A URI is a character string for identifying an abstract or physical resource, and a URI specified in RFC2396 or RFC3986 may be used. The URI may be the name of the information, such as a Uniform Resource Name (URN), or the location of the information, such as a Uniform Resource Locator (URL).

[0025] The video decoding device 30 comprises an image decoding device 301 , an image generation processing device 302 , and a generated information decoding device 303 .

[0026] The image decoding device 301 receives as input the live encoded data Te transmitted via the transmission network 20 , decodes the image information, and transmits the decoded image information to the generated information decoding device 302 and the image generating device 303 .

[0027] The generation information decoding device 302 decodes the auxiliary extension information of the encoded data Te based on the syntax, loads the image information created by the image decoding device 301 and the URI data from the storage location (a server on the network or a specific storage location) based on the decoded URI, creates generation information, and sends the generation information to the image generation device 301.

[0028] The image generation processing device 303 performs image generation processing using the image information decoded by the image decoding device 301, the generation information decoded by the generation information decoding device 303, and model data from outside, generates a generated image Td, and outputs it to the image display device 40.

[0029] In this embodiment, the image encoding device 101 and the image decoding device 301 are realized by applying a general-purpose video encoding and decoding method such as AVC, HEVC, or VVC.

[0030] FIG. 2 is a conceptual diagram showing the configuration of the image generation processing device of this embodiment. The image processing device in this system uses a generative image processing method based on so-called image generation AI, which is configured using neural networks such as diffusion models. It takes image information, generation information, and model data as inputs and outputs a generated image.

[0031] The image generation processing device 303 consists of an image generation unit 3031, a control unit 3032, and a control image generation unit 3033. The image generation unit 3031 uses a generated image processing method configured with a stable diffusion neural network. The control unit 3032 uses a control method configured with a neural network called a control net. The control image generation unit 3033 generates a control image signal from image information.

[0032] The control unit 3032 receives as input the image information, the control parameter information in the generation information, and the model data specified by the control parameters, and outputs control image information to be input to the image generation unit 3031. Here, the image information is the locally decoded image signal output by the image encoding device 101 or the decoded image signal output by the image decoding device 301.

[0033] Both are image information obtained by encoding and decoding an input image signal.

[0034] The control image signal is generated from image information by the control image generation unit 3033. Specifically, the control image signal uses the following information: The identification of these is included in the control parameters. Contour (Canny) image Soft edge image Sketch image Line art images Normal map image Depth map image Segmentation image Open Pose Image Wireframe (MLSD) images Inpainted image Reference Image Both are monochrome or color image information created using an image. Note that the control image signal is not limited to one image, and multiple different control image signals may exist for the same image information.

[0035] The generation information includes control parameter information, model information, model parameter information, prompt information, and the like.

[0036] The control parameter information is a parameter for controlling the control unit 3302 described above, and includes identification of the basic image of the image information, identification of the control image, model information of the control unit, and the like.

[0037] The model information is the name of a neural network model and model data of the neural network to be used for image generation by the image generation unit 3301. The model data is shared as URI information between the video encoding device 10 and the video decoding device 30, and is input as model data from outside to the image generation processing device 303. Alternatively, the video encoding device 10 and the video decoding device 30 may have the same model data in advance.

[0038] The model parameters are parameters for controlling the neural network, and are various kinds of numerical and character string information such as intensity values, number of steps, type of sampler, and seed information.

[0039] Prompt information is character string information that indicates the content of the image to be generated. Prompt information includes positive prompt information that indicates the content that is desired to be generated, and negative prompt information that indicates the content that is not desired to be generated.

[0040] The positive prompt information can be automatically generated by image analysis of the input image signal. Alternatively, the positive prompt information can be automatically generated by image analysis of the image information decoded by the image encoding device 101 or the image decoding device 301. The positive prompt may be encoded and decoded by using information created from the input image signal as part of the generated information. Alternatively, if information created from image information is used, the information may be created by the generated information decoding device 302, and mode information indicating this may be sent. Alternatively, the difference between the information created from the input image signal and the information created from the image information may be coded as part of the generated information, and the information created by the video decoding device 30 may be used to decode the information created from the input image signal.

[0041] The negative prompt information may be common to the video encoding device 10 and the video decoding device 30, and may be encoded and decoded as part of additional generated information when additional information needs to be sent.

[0042] 3 is a block diagram showing the configuration of generation information creation device 102 according to this embodiment. Generation information creation device 102 according to this embodiment receives an input image signal, image information created by video encoding device 10, and external model data, and outputs generation information that is sent to generation information encoding device 103.

[0043] The generated information creation device 102 is composed of a generated information creation unit 1021, an encoding control unit 1022, and an image generation processing device 1023. The image generation processing device 1023 is the same as the image generation processing device 303 described above, and outputs a generated image from the generated information, image information, and model data. The encoding control unit 1032 selects generated information based on two indices: an evaluation criterion D for image similarity, such as mean square error, absolute sum of error, SSIM (Structural Similarity), MS-SSIM (Multi-Scale Structural Similarity), or LPIPS (Learned Perceptual Image Patch Similarity), which is based on a comparison between the generated image result of the image generation device 303 and the input image signal; and the code amount R of the generated information created by the generated information creation unit 1021, and outputs the optimal one.

[0044] The generation information creating unit 1021 generates generation information by exchanging information with the encoding control unit 1022 and sends it to the image generation processing device 1023 .

[0045] The generation information encoding device 103 encodes the generation information created by the generation information creation device 102 and sends the auxiliary extension information encoded data together with the encoded data output by the image encoding device 101 as encoded data Te to the created transmission network 20.

[0046] The generated information decoding device 302 decodes the auxiliary extension encoded data out of the encoded data Te sent from the transmission network 20, and sends the decoding result to the image generation processing device 303 as generated information.

[0047] In this embodiment, encoding and decoding are performed as an SEI (Supplemental Enhancement Information) message based on a syntax described later. Note that the encoding and decoding method is not limited to the SEI message, and encoding and decoding may also be performed as a syntax in a video encoding and decoding method, such as an APS (Adaptation Parameter Set).

[0048] 4 is a block diagram showing the configuration of generation information encoding device 103 of this embodiment. Generation information encoding device 103 of this embodiment includes a supplemental extension information encoding unit 1031, a URI data encoding unit 1032, and a URI data saving unit 1033.

[0049] The supplementary extension information encoding unit 1031 defines an identifier for the generation information generated by the generation information creation device 102 as a URI (Uniform Resource Identifier), creates supplementary extension information encoded data as an SEI message described later, and sends it to the transmission network 20 as part of the encoded data Te.

[0050] The URI data encoding unit 1032 encodes the contents of the generated information whose identifier is defined by the supplementary extension information encoding unit 1031 as text information or compressed text data, converts it into URI data, and sends it to the URI data saving unit 1033.

[0051] The URI data saving unit 1033 saves the URI data encoded by the URI data encoding unit 1032 to the URI defined by the auxiliary extension information encoding unit 1031 in the location indicated by the URI (a server on the network or a specific storage location).

[0052] 5 is a block diagram showing the configuration of the generation information decoding device 302 according to this embodiment. The generation information decoding device 302 according to this embodiment includes a supplemental extension information decoding unit 3021, a URI data decoding unit 3022, and a URI data loading unit 3023.

[0053] The auxiliary extension information decoding unit 3021 decodes the auxiliary extension information encoded data out of the encoded data Te received from the transmission network 20. The auxiliary extension encoded data is decoded as an SEI message (described later), and the decoded URI is sent to the URI data loading unit 3023.

[0054] The URI data loading unit 3023 loads the encoded URI data from the location where it is stored (a server on the network or a specific storage location) based on the URI decoded by the supplemental extension information decoding unit 3021.

[0055] The URI data decoder 3022 decodes the generated information from the loaded URI data and sends it to the auxiliary extension decoder 3021 .

[0056] The auxiliary extension information decoding unit 3021 also combines the generation information created from the image information of the image decoding device 301, the decoding result of the auxiliary extension encoded data, and the decoding result of the generation information from the URI data, and outputs the generation information as the final result to the image generation processing device 303.

[0057] 6, 7, 8, and 9 show the syntax of generated information coded data that is coded and decoded by generated information coding device 103 and generated information decoding device 302 in this embodiment.

[0058] The meaning of the Descriptor notation in the following syntax tables is interpreted as follows: · b(8): Represents a byte value with any pattern of bit string (8 bits). f(n): represents a fixed-pattern bit string using n bits written left to right. se(v): represents a syntax element obtained by encoding a signed integer into 0th-order Exp-Golomb code. st(v): Represents a null-terminated string encoded in UTF-8. u(n): represents an unsigned integer using n bits. If n is "v" in the syntax table, the number of bits varies depending on the values ​​of other syntax elements. ·ue(v): represents the syntax element (left bit first) encoded as an unsigned integer with order 0 Exp-Golomb coding.

[0059] 6 shows part of the syntax of the NNPFC SEI message (Neural Network Post-Filter Characteristic SEI message) in Non-Patent Document 2. This SEI message can send tag information related to the purpose of the application as extended information.

[0060] The syntax elements in FIG. 6 will be explained below.

[0061] The syntax element nnpfc_purpose indicates the purpose of the NNPF. Here, if (nnpfc_purpose & bitMask) is not 0, it indicates that the NNPF has a purpose associated with the bitMask value. If the value of nnpfc_purpose is greater than 0 and the value of (nnpfc_purpose & bitMask) is 0, the purpose associated with the bitMask value does not apply to the NNPF. A bitMask value of 0x01 is intended to improve general visual quality.

[0062] A bitMask value of 0x02 is intended for upsampling of chrominance signals (from 4:2:0 format to 4:2:2 or 4:4:4 format, or from 4:2:2 format to 4:4:4 format).

[0063] A bitMask value of 0x04 is intended for resolution resampling (enlarging or reducing the width or height resolution).

[0064] A bitMask value of 0x08 is intended for picture rate upsampling.

[0065] A bitMask value of 0x10 is intended for pixel bit depth upsampling (increasing the luma pixel bit depth or chroma pixel bit depth).

[0066] A bitMask value of 0x20 is intended to colorize monochrome images.

[0067] A bitMask value of 0x40 is intended for temporal extrapolation (the generation of one or more future images).

[0068] A bitMask value of 0x80 is intended for spatial extrapolation (producing content outside the spatial domain of the input image).

[0069] If the value of nnpfc_purpose is 0, the NNPF can be used as determined by the application and specified by nnpfc_application_purpose_tag_uri.

[0070] All NNPFC SEI messages with a particular value of nnpfc_id in CLVS MUST have the same value of nnpfc_purpose. Values ​​of nnpfc_purpose MUST be in the range 0 to 255 inclusive, ... Decoders conforming to this version of this specification will ignore NNPFC SEI messages where nnpfc_purpose is in the range from 256 to 65535.

[0071] In the method of Non-Patent Document 2, the introduction of the syntax element nnpfc_application_purpose_tag_uri makes it possible to define the purpose of any application, but the syntax element nnpfc_purpose may be extended.

[0072] Specifically, for example, if the bitMask value is 0x0100, the target may be neural network post-filter processing by the generation AI.

[0073] The syntax element nnpfc_id indicates an identification number that can be used to identify an NNPF. The value of nnpfc_id MUST be in the range 0 to 2^32 - 2. Values ​​of nnpfc_id in the ranges 256 to 511 and 2^31 to 2^32 - 2 are reserved for future use. A decoder conforming to this version of this specification that encounters an NNPFC SEI message with an nnpfc_id in the range 256 to 511 or 2^31 to 2^32 - 2 shall Ignore the SEI message.

[0074] If the NNPFC SEI message is the first NNPFC SEI message, in decoding order, with a particular nnpfc_id value within the current CLVS, the following applies: This SEI message defines the base NNPF. This SEI message applies to the current decoded picture and all subsequent decoded pictures in the current layer, in output order, until the end of the current CLVS.

[0075] The syntax element nnpfc_base_flag is a flag that indicates whether this SEI message is for the base NNPF. If the value of nnpfc_base_flag is 1, this SEI message is for the base NNPF. If the value of nnpfc_base_flag is 0, this SEI message is an update to the base NNPF.

[0076] The following constraints apply to the value of nnpfc_base_flag: ·If the NNPFC SEI message is the first NNPFC SEI message with a particular nnpfc_id value in the current CLVS, in decoding order, the value of nnpfc_base_flag shall be 1. ·All NNPFC SEI messages with a particular nnpfc_id value and nnpfc_base_flag value of 1 within a CLVS must be identical.

[0077] Additionally, if the value of nnpfc_base_flag is 0, the following applies: This SEI message defines updates to the preceding base NNPF with the same nnpfc_id value in decoding order. Updates are not cumulative; each update applies to the base NNPF with a particular nnpfc_id value in the current CLVS for the first NNPF SEI message in decoding order. The NNPF defined in this SEI message is obtained by applying the updates defined in this SEI message to the base NNPF with the same nnpfc_id value. This SEI message applies to the current decoded picture and all subsequent decoded pictures in the current layer, in output order, up to the end of the current CLVS or to a picture associated with a subsequent NNPFC SEI message with nnpfc_base_flag equal to 0 and a specific nnpfc_id value in the current CLVS, in decoding order, excluding decoded pictures that follow the current decoded picture in output order within the current CLVS, whichever comes first.

[0078] The syntax element nnpfc_mode_idc is a value that identifies the neural network information. A value of 0 for nnpfc_mode_idc indicates that neural network information is included in the NNPFC SEI message and that the neural network information is in the form of an ISO / IEC 15938-17 bitstream. A value of 1 for nnpfc_mode_idc indicates that the neural network information is in the form identified by the tag URI nnpfc_tag_uri and is identified by the URI indicated by nnpfc_uri. The value of nnpfc_mode_idc MUST be in the range 0 to 255. Values ​​of nnpfc_mode_idc between 2 and 255 are reserved for future use and will not be present in bitstreams conforming to this version of this specification. Decoders conforming to this version of this specification ignore NNPFC SEI messages with nnpfc_mode_idc between 2 and 255.

[0079] The value of the syntax element nnpfc_alignment_zero_bit_a must be 0.

[0080] The syntax element indicates a tag URI. The nnpfc_tag_uri contains a tag URI with syntax and semantics defined in IETF RFC 4151 that indicates the format and related information of the neural network to be used as the base NNPF, or updates to be applied to the base NNPF with the same nnpfc_id value specified in the nnpfc_uri. Note that the use of the nnpfc_tag_uri allows the format of the neural network data specified in the nnpfc_uri to be registered without the need for a central registration authority. It can uniquely identify the formula. If nnpfc_tag_uri is equal to "tag:iso.org,2023:15938-17", it indicates that the neural network data identified by nnpfc_uri conforms to ISO / IEC 15938-17.

[0081] The nnpfc_uri contains a URI with syntax and semantics specified in IETF Internet Standard 66 that identifies the neural network used as the base NNPF, or an update to the base NNPF with the same nnpfc_id value.

[0082] The syntax element nnpfc_num_metadata_extension_bits indicates the number of bits extended for metadata. If nnpfc_num_metadata_extension_bits is 0, it indicates that nnpfc_reserved_metadata_extension does not exist. If nnpfc_num_metadata_extension_bits is greater than 0, the variable numSpecifiedMetadataExtensionBits is the number of bits representing all syntax elements between nnpfc_num_metadata_extension_bits and nnpfc_reserved_metadata_extension.

[0083] If nnpfc_num_metadata_extension_bits is greater than 0, it specifies the sum of numSpecifiedMetadataExtensionBits and the length (in bits) of nnpfc_reserved_metadata_extension. The value of nnpfc_num_metadata_extension_bits must be in the range of numSpecifiedMetadataExtensionBits to 2048. Values ​​of nnpfc_num_metadata_extension_bits in the range numSpecifiedMetadataExtensionBits + 1 through 2048, inclusive, are reserved for future use and will not be present in bitstreams conforming to this version of this specification. Decoders conforming to this version of this specification allow any value of nnpfc_num_metadata_extension_bits in the range 0 through numSpecifiedMetadataExtensionBits + 1 through 2048, inclusive.

[0084] The syntax element nnpfc_application_purpose_tag_uri_present_flag indicates whether the syntax element nnpfc_application_purpose_tag_uri is present in this NNPFC SEI message. If nnpfc_application_purpose_tag_uri_present_flag is 1, it indicates that the syntax element nnpfc_application_purpose_tag_uri is present in this NNPFC SEI message. If nnpfc_application_purpose_tag_uri_present_flag is 0, it indicates that the syntax element nnpfc_application_purpose_tag_uri is not present in this NNPFC SEI message. If it is not present, nnpfc_application_purpose_tag_uri_present_flag is inferred to be equal to 0.

[0085] The syntax element nnpfc_application_purpose_tag_uri, when nnpfc_purpose is 0, specifies a tag URI with syntax and semantics specified in IETF RFC 4151 that identifies an application-determined purpose in the NNPF. The nnpfc_application_purpose_tag_uri allows unique identification of an application-determined purpose in the NNPF without the need for a central registration authority.

[0086] The syntax element nnpfc_reserved_metadata_extension shall not be present in bitstreams conforming to this version of this specification. However, decoders conforming to this version of this specification shall ignore the presence and value of nnpfc_reserved_metadata_extension. If present, the length (in bits) of nnpfc_reserved_metadata_extension shall be equal to nnpfc_num_metadata_extension_bits - numSpecifiedMetadataExtensionBits.

[0087] In the method of Non-Patent Document 2, the introduction of the syntax element nnpfc_application_purpose_tag_uri makes it possible to define the purpose of any application, but there was a problem in that it was unclear how to define the information required by that specific application.

[0088] The syntax element nnpfc_application_purpose_tag_uri also includes string information. There was a problem where the data was not byte aligned.

[0089] Therefore, in this embodiment, a framework is provided that allows the necessary information to be defined for any application.

[0090] FIG. 7 shows part of the syntax of the NNPFC SEI message (Neural Network Post-Filter Characteristic SEI message) in this embodiment.

[0091] In Non-Patent Documents 1 and 2, only the values ​​0 and 1 are defined for the syntax element nnpfc_mode_idc, but here we define a value of 2 for nnpfc_mode_idc. Note that any identifiable value between 2 and 255 other than 0 and 1 may also be used.

[0092] The syntax element nnpfc_mode_idc is a value that identifies the neural network information. If the value of nnpfc_mode_idc is 0, it indicates that the neural network information is included in the NNPFC SEI message and that the neural network information is in the format of an ISO / IEC 15938-17 bitstream. If the value of nnpfc_mode_idc is 1, it indicates that the neural network information is in the format identified by the tag URI nnpfc_tag_uri and is identified by the URI indicated by nnpfc_uri.

[0093] When nnpfc_mode_idc is set to 2, it indicates that the application information for post-filter processing using a neural network is identified by the URI indicated by nnpfc_uri in a format identified by the tag URI nnpfc_tag_uri.

[0094] The value of nnpfc_mode_idc MUST be in the range 0 to 255. Values ​​of nnpfc_mode_idc from 3 to 255 are reserved for future use and will not be present in bitstreams conforming to this version of this specification. Decoders conforming to this version of this specification will ignore NNPFC SEI messages with nnpfc_mode_idc in the range 3 to 255.

[0095] By extending the syntax element nnpfc_mode_idc in this way, it becomes possible to define the information required by the specific application using tag URIs and URIs, thereby solving the problem.

[0096] Furthermore, since the byte alignment is not performed before the syntax element nnpfc_application_purpose_tag_uri in Non-Patent Document 2, there is a problem in that the character information cannot be used immediately after being decoded.

[0097] Therefore, after the syntax element nnpfc_application_purpose_tag_uri_present_flag, if the value of nnpfc_application_purpose_tag_uri_present_flag is 1, that is, if the syntax element nnpfc_application_purpose_tag_uri is present in this NNPFC SEI message, byte alignment is performed. Specifically, byte_aligned() is a function that returns whether the current encoded data is in byte units. If it is not in byte units, the syntax element nnpfc_metadata_alignment_zero_bit is inserted to adjust the bit position so that the next element is positioned on a byte boundary. nnpfc_metadata_alignment_zero_bit is set to 0.

[0098] In this way, the problem can be solved by inserting a byte alignment bit before the syntax element nnpfc_application_purpose_tag_uri.

[0099] FIG. 8 shows part of the syntax of an NNPFC SEI message (Neural Network Post-Filter Characteristic SEI message) in another embodiment.

[0100] In this example, first, after the syntax element nnpfc_application_purpose_tag_uri_present_flag, if the value of nnpfc_application_purpose_tag_uri_present_flag is 1, that is, if the syntax element nnpfc_application_purpose_tag_uri is present in this NNPFC SEI message, byte alignment is performed. Specifically, byte_aligned() is a function that returns whether the current encoded data is in byte units. If it is not in byte units, the syntax element nnpfc_metadata_alignment_zero_bit is inserted to adjust the bit position so that the next element is positioned on a byte boundary. nnpfc_metadata_alignment_zero_bit is set to 0.

[0101] In this way, the problem can be solved by inserting a byte alignment bit before the syntax element nnpfc_application_purpose_tag_uri.

[0102] Next, the syntax element nnpfc_application_purpose_tag_uri specifies a tag URI with syntax and semantics specified in IETF RFC 4151 that identifies an application-determined purpose of the NNPF when nnpfc_purpose is 0. The nnpfc_application_purpose_tag_uri allows unique identification of an application-determined purpose of the NNPF without the need for a central registration authority.

[0103] The syntax element nnpfc_application_data_uri identifies information about the application identified by the nnpfc_application_purpose_tag_uri. The nnpfc_application_data_uri contains a URI with syntax and semantics specified in IETF Internet Standard 66 that points to neural network and application information used as a base NNPF, or update information for a base NNPF with the same nnpfc_id value.

[0104] Note that instead of nnpfc_application_data_uri, string information of the syntax element nnpfc_application_data_string may be used.

[0105] In Non-Patent Documents 1 and 2, the Neural Network Post-Filter Characteristic SEI message has an extensible syntax structure, but the Neural Network Post-Filter Activation SEI message has a problem in that its syntax cannot be extended.

[0106] Therefore, in this embodiment, the Neural Network Post-Filter Activation Extension (NNPFAE) SEI message shown in Fig. 9 is shown. This SEI message can be used in addition to the existing Neural Network Post-Filter Activation (NNPFA) SEI message.

[0107] The syntax elements in FIG. 9 will be explained below.

[0108] A value of 1 for the syntax element nnpfa_extension_cancel_flag indicates that the SEI message cancels the persistence of the previous NNPFAE SEI message in the output order. A value of 0 for nnpfae_cancel_flag indicates that NNPFA extension information follows.

[0109] The syntax element nnpfa_extension_persistence_flag specifies the persistence of the NNPFAE SEI message for the current layer. A value of 0 for nnpfa_extension_persistence_flag specifies that the NNPFAE SEI message applies only to the current decoded picture. A value of 1 for nnpfa_extension_persistence_flag specifies that the NNPFAE SEI message applies to the current decoded picture and persists in output order for all subsequent pictures in the current layer until one or more of the following conditions become true: A new CLVS for the current hierarchy is started. The bitstream ends. ·The picture of the current layer in the AU associated with the NNPFAE SEI message is output following the current picture in output order.

[0110] The syntax element nnpfa_num_metadata_extension_bits indicates the number of bits extended for metadata. If nnpfa_num_metadata_extension_bits is 0, it indicates that nnpfa_reserved_metadata_extension does not exist. If nnpfa_num_metadata_extension_bits is greater than 0, the variable numSpecifiedActivationMetadataExtensionBits is the number of bits representing all syntax elements between nnpfa_num_metadata_extension_bits and nnpfa_reserved_metadata_extension.

[0111] If nnpfa_num_metadata_extension_bits is greater than 0, it specifies the sum of numSpecifiedActivationMetadataExtensionBits and the length (in bits) of nnpfa_reserved_metadata_extension. The value of nnpfa_num_metadata_extension_bits MUST be in the range of numSpecifiedActivationMetadataExtensionBits through 2048, inclusive. Values ​​of nnpfa_num_metadata_extension_bits in the range of numSpecifiedActivationMetadataExtensionBits + 1 through 2048, inclusive, are reserved for future use and will not be present in bitstreams conforming to this version of this specification. Decoders conforming to this version of this specification allow any value of nnpfa_num_metadata_extension_bits in the range of 0 through numSpecifiedActivationMetadataExtensionBits + 1 through 2048, inclusive.

[0112] The byte_aligned() function returns whether the current encoded data is aligned to the byte boundary. If it is not aligned to the byte boundary, it inserts the syntax element nnpfa_metadata_alignment_zero_bit to adjust the bit position so that the next element is aligned to the byte boundary. nnpfa_metadata_alignment_zero_bit is set to 0.

[0113] The syntax element nnpfa_ait_data_string is a text string containing the command prompt to be interpreted by the generative AI engine. The text prompt is encoded as specified in ISO / IEC 10646: Information technology - Universal Coded Character Set (UCS). UTF-8 of the UCS may be used here as specified by st(v).

[0114] The syntax element nnpfa_reserved_metadata_extension shall not be present in bitstreams conforming to this version of this specification. However, decoders conforming to this version of this specification shall ignore the presence and value of nnpfa_reserved_metadata_extension. If present, the length (in bits) of nnpfa_reserved_metadata_extension shall be equal to nnpfa_num_metadata_extension_bits - numSpecifiedActivationMetadataExtensionBits. The NNPFAE SEI message may be used simultaneously with the NNPFA SEI message, and the NNPFAE SEI message may be used in addition to the NNPFA SEI message if extensions are required.

[0115] As described above, in this embodiment, by defining a new NNPFAE SEI message that is an extension of the NNPFA SEI message, it has been shown that an image transmission system using a video encoding and decoding method that uses an image generation method can be realized by encoding and decoding character string information that includes a command prompt that is interpreted by the generation AI engine.

[0116] FIG. 10 shows another NNPFC SEI message (Neural Network Post-Filter This is part of the syntax of the Characteristic SEI message.

[0117] In Non-Patent Documents 1 and 2, only the values ​​0 and 1 are defined for the syntax element nnpfc_mode_idc, but the value 2 is defined for nnpfc_mode_idc as in the embodiment of FIG. 7. Note that values ​​other than 0 and 1 are also defined. Any other distinct value between 2 and 255 may be used.

[0118] The syntax element nnpfc_mode_idc is a value that identifies the neural network information. If the value of nnpfc_mode_idc is 0, it indicates that the neural network information is included in the NNPFC SEI message and that the neural network information is in the format of an ISO / IEC 15938-17 bitstream. If the value of nnpfc_mode_idc is 1, it indicates that the neural network information is in the format identified by the tag URI nnpfc_tag_uri and is identified by the URI indicated by nnpfc_uri.

[0119] If nnpfc_mode_idc is set to 2, it indicates that the application information for post-filter processing by the neural network is identified by the URI indicated by nnpfc_application_information_uri in a format identified by the tag URI, nnpfc_application_information_tag_uri.

[0120] As shown in Figure 10, when nnpfc_mode_idc is 2, character string information is included, so the start of the bitstream must be in byte units.

[0121] The byte_aliged() function returns whether the current encoded data is aligned to the byte boundary. If it is not aligned to the byte boundary, it inserts the syntax element nnpfc_application_information_alignment_zero_bit to adjust the bit position so that the next element is aligned to the byte boundary. The value of the syntax element nnpfc_application_information_alignment_zero_bit must be 0.

[0122] The syntax element nnpfc_application_information_tag_uri indicates a tag URI. The nnpfc_application_information_tag_uri contains a tag URI with syntax and semantics defined in IETF RFC 4151 that indicates the format of the application information and related information to be used as the base NNPF, or update information to be applied to the base NNPF with the same nnpfc_id value specified in the nnpfc_application_information_uri.

[0123] Note that the nnpfc_application_information_tag_uri can be used to uniquely identify the format of the application information specified in the nnpfc_application_information_uri without requiring a central registration authority.

[0124] For example, if nnpfc_application_information_tag_uri is equal to "tag:stable.diffusion.webui.170", it indicates that the application information identified by nnpfc_application_information_uri conforms to the application information generated by Stable Diffusion Webui 1.70.

[0125] The syntax element nnpfc_application_information_uri indicates a URI that identifies application information. The nnpfc_application_information_uri contains a URI with the syntax and semantics specified in IETF Internet Standard 66, and indicates application information used as the base NNPF, or update information for the base NNPF with the same nnpfc_id value. By extending the syntax element nnpfc_mode_idc in this way, it becomes possible to define the information required by a specific application independently using tag URIs and URIs, thereby solving the problem.

[0126] FIG. 11 shows another NNPFC SEI message (Neural Network Post-Filter This is part of the syntax of the Characteristic SEI message.

[0127] The syntax element num_processing_model determines the neural network model used by the application. The number of models for the post-filtering process is shown.

[0128] The syntax element num_processing_argment indicates the number of arguments for neural network post-filter processing by the application.

[0129] As shown in FIG. 11, before the character string information, the start of the bit stream is made to be in byte units.

[0130] The byte_aliged() function returns whether the current encoded data is in byte units. If it is not in byte units, it inserts the syntax element nnpfc_processing_alignment_zero_bit to adjust the bit position so that the next element is positioned on a byte boundary. The value of the syntax element nnpfc_processing_alignment_zero_bit must be 0. The syntax elements nnpfc_processing_tag_uri[ i ] and syntax elements nnpfc_processing_uri[ i ] for i=0 to i=num_processing_model-1 are encoded and decoded the same number of times as the number of num_processing_models.

[0131] The syntax element nnpfc_processing_tag_uri[ i ] indicates the tag URI. nnpfc_processing_tag_uri[ i ] contains a tag URI with syntax and semantics defined in IETF RFC 4151 that indicates the format of the application information and associated information to be used as the base NNPF, or update information to be applied to the base NNPF with the same nnpfc_id value specified in nnpfc_processing_tag_uri[ i ].

[0132] Note that nnpfc_processing_tag_uri[ i ] can be used to uniquely identify the format of the application information specified in nnpfc_processing_tag_uri[ i ] without the need for a central registration authority.

[0133] The syntax element nnpfc_processing_uri[ i ] indicates a URI that identifies application information. nnpfc_processing_uri[ i ] contains a URI with the syntax and semantics specified in IETF Internet Standard 66 that indicates application information used as a base NNPF or update information to a base NNPF with the same nnpfc_id value.

[0134] The syntax elements nnpfc_argment_content_type[ i ] and syntax elements nnpfc_argment_uri[ i ] for i=0 to i=num_processing_argment-1 are coded and decoded the number of times equal to the number of num_processing_argment.

[0135] The syntax element nnpfc_argment_content_type[ i ] represents a character string that indicates the type of argument information in the application information.

[0136] The syntax element nnpfc_argment_uri[ i ] indicates a URI that identifies application information. nnpfc_argment_uri contains a URI with the syntax and semantics specified in IETF Internet Standard 66 that indicates the argument information for application n that is used as a base NNPF or an update to a base NNPF with the same nnpfc_id value.

[0137] By extending the syntax element nnpfc_mode_idc in this way, it becomes possible to define the information required by that specific application independently using tag URIs and URIs, thereby solving the problem.

[0138] FIG. 12 shows another NNPFC SEI message (Neural Network Post-Filter This is part of the syntax of the Characteristic SEI message.

[0139] As shown in Figure 12, when nnpfc_mode_idc is 2, character string information is included, so the start of the bitstream must be in byte units.

[0140] The byte_aliged() function returns whether the current encoded data is in byte units. If it is not in byte units, it inserts the syntax element nnpfc_processing_alignment_zero_bit to adjust the bit position so that the next element is positioned on a byte boundary. The value of the syntax element nnpfc_processing_alignment_zero_bit must be 0. The syntax element nnpfc_processing_tag_uri indicates the tag URI. The nnpfc_processing_tag_uri contains a tag URI with syntax and semantics defined in IETF RFC 4151 that indicates the format of the application information and associated information to be used as the base NNPF, or update information to be applied to the base NNPF with the same nnpfc_id value specified in the nnpfc_processing_tag_uri.

[0141] Note that the nnpfc_processing_tag_uri allows the format of the application information specified by the nnpfc_processing_tag_uri to be uniquely identified without the need for a central registration authority.

[0142] The syntax element num_processing_model indicates the number of models for post-processing by the application.

[0143] The syntax element num_processing_argment indicates the number of arguments for post-processing by the application.

[0144] The syntax element nnpfc_processing_tag_uri indicates the tag URI. The nnpfc_processing_tag_uri contains a tag URI with syntax and semantics defined in IETF RFC 4151 that indicates the format of the application information and associated information to be used as the base NNPF, or update information to be applied to the base NNPF with the same nnpfc_id value specified in the nnpfc_processing_tag_uri.

[0145] Note that the nnpfc_processing_tag_uri allows the format of the application information specified by the nnpfc_processing_tag_uri to be uniquely identified without the need for a central registration authority.

[0146] The syntax element num_processing_model indicates the number of models for neural network post-filter processing by the application.

[0147] The syntax element num_processing_argment indicates the number of arguments for neural network post-filter processing by the application.

[0148] The syntax elements nnpfc_processing_uri[ i ] from i=0 to i=num_processing_model-1 are encoded and decoded the number of times equal to the number of num_processing_models.

[0149] The syntax element nnpfc_processing_uri[ i ] indicates a URI that identifies application information. nnpfc_processing_uri[ i ] contains a URI with the syntax and semantics specified in IETF Internet Standard 66 that indicates application information used as a base NNPF or update information to a base NNPF with the same nnpfc_id value.

[0150] The syntax elements nnpfc_argment_content_type[ i ] and syntax elements nnpfc_argment_uri[ i ] for i=0 to i=num_processing_argment-1 are coded and decoded the number of times equal to the number of num_processing_argment.

[0151] The syntax element nnpfc_argment_content_type[ i ] specifies the argument information of the application information. This indicates the type of information.

[0152] The syntax element nnpfc_argment_uri[ i ] indicates a URI that identifies application information. nnpfc_argment_uri contains a URI with the syntax and semantics specified in IETF Internet Standard 66 that indicates the argument information for application n that is used as a base NNPF or an update to a base NNPF with the same nnpfc_id value.

[0153] By extending the syntax element nnpfc_mode_idc in this way, it becomes possible to define the information required by that specific application independently using tag URIs and URIs, thereby solving the problem.

[0154] Note that part or all of the video encoding device 10 and the video decoding device 30 in the above-described embodiments may be implemented by a computer. In this case, a program for implementing the control functions may be recorded on a computer-readable recording medium, and the program may be loaded into a computer system and executed. Note that the term "computer system" as used herein refers to a computer system built into either the video encoding device 10 or the video decoding device 30, including hardware such as an OS and peripheral devices. Furthermore, the term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into a computer system. Furthermore, the term "computer-readable recording medium" may also include media that dynamically store programs for a short period of time, such as communication lines used when transmitting programs via networks such as the Internet or telephone lines, or media that store programs for a fixed period of time, such as volatile memory within a computer system that serves as a server or client in such cases. Furthermore, the program may be a program for implementing part of the above-described functions, or may be a program that can be implemented in combination with a program already stored in the computer system.

[0155] Furthermore, part or all of the video encoding device 10 and the video decoding device 30 in the above-described embodiments may be realized as an integrated circuit such as an LSI (Large Scale Integration). Each functional block of the video encoding device 10 and the video decoding device 30 may be individually implemented as a processor, or part or all of them may be integrated into a processor. Furthermore, the integrated circuit implementation method is not limited to LSI, and may be implemented using a dedicated circuit or a general-purpose processor. Furthermore, if an integrated circuit implementation technology that can replace LSI emerges due to advances in semiconductor technology, an integrated circuit based on that technology may be used.

[0156] One embodiment of the present invention has been described in detail above with reference to the drawings, but the specific configuration is not limited to that described above, and various design changes and the like are possible within the scope that does not deviate from the gist of the present invention.

[0157] The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. In other words, embodiments obtained by combining technical means modified appropriately within the scope of the claims are also included in the technical scope of the present invention. [Industrial Applicability]

[0158] The embodiments of the present invention can be suitably applied to a video decoding device that decodes coded data obtained by coding an image signal, and a video coding device that generates coded data obtained by coding image data, and can also be suitably applied to the data structure of coded data generated by a video coding device and referenced by the video decoding device. [Explanation of symbols]

[0159] 1. Image transmission system 10 Video Encoding Device 101 Image encoding device 102 Generative information creation device 1021 Generation Information Creation Department 1023, 303 Image generation processing device 1022 Encoding control unit 103 Generated information encoding device 1031 Supplementary Extension Code 1032 URI data encoding part 1033 URI data save section 20 Transmission Network 30 Video decoding device 301 Image decoding device 302 Generated information decoding device 3021 Supplementary extension information decoding unit 3022 URI data decoding unit 3023 URI Data Loading Unit 303 Image generation processing device 3031 Image Generation Unit 3032 Control Unit 3033 Control image generation unit 40 Image display device

Claims

1. an image decoding device that decodes encoded data of an image signal; a generated information decoding device that decodes the generated information from generated encoded data, a tag URI that identifies a format of the generated information, information acquired from the URI that identifies the generated information, and image information decoded by the image decoding device; A video decoding device comprising an image generating device that generates an image from the image information decoded by the image decoding device and the generated information decoded by the generated information decoding device.

2. 2. The video decoding device according to claim 1, wherein the generated information decoding device creates and uses generated information from the image information.

3. an image encoding device that encodes an image signal; a generation information creating device for creating generation information for generating an image from the image signal and image information decoded by the image encoding device; a tag URI that identifies the format of the generated information; A video encoding device comprising a URI for identifying the generated information and a generated information encoding device for encoding the generated information encoded data.

Images