Decoding device, encoding device, decoding method, and encoding method

Abstract

A decoding device (200) comprises a memory (252), and a circuit (251) connected to the memory (252). The circuit (251) uses the memory (252) to decode base data of a face image relating to a face moving image, and one or more pieces of extension data of the face image from a bit stream, decodes geometric information corresponding to each of a plurality of frames of the face moving image from the bitstream, generates a face moving image from the base data, the one or more pieces of extension data, and the geometric information. The base data is added to data corresponding to a first frame, which is a frame of the face moving image, in the bitstream. The one or more pieces of extension data are added to one or more pieces of data corresponding to one or more second frames of the face moving image in the bitstream.

Description

Decoding device, encoding device, decoding method, and encoding method

[0001] The present disclosure relates to a decoding device and the like.

[0002] Video coding technology has progressed from H.261 and MPEG-1 to H.264 / AVC (Advanced Video Coding), MPEG-LA, H.265 / HEVC (High Efficiency Video Coding), and H.266 / VVC (Versatile Video Codec). With this progress, there is a constant need to provide improvements and optimizations in video coding technology to handle the ever-increasing amount of digital video data in various applications. This disclosure relates to further advances, improvements, and optimizations in video coding.

[0003] Non-Patent Document 1 relates to an example of a conventional standard related to the above-mentioned video coding technology.

[0004] H. 265 (ISO / IEC 23008-2 HEVC) / HEVC (High Efficiency Video Coding)

[0005] With regard to the above-mentioned encoding methods, it is desirable to propose new methods to improve encoding efficiency, improve image quality, reduce the amount of processing, reduce the circuit scale, or appropriately select elements or operations such as filters, block sizes, motion vectors, reference pictures or reference blocks.

[0006] The present disclosure provides a configuration or method that can contribute to one or more of, for example, improved coding efficiency, improved image quality, reduced processing amount, reduced circuit size, improved processing speed, and appropriate selection of elements or operations, etc. Note that the present disclosure may include a configuration or method that can contribute to benefits other than those described above.

[0007] For example, a decoding device according to one aspect of the present disclosure includes a memory and a circuit connected to the memory, and the circuit uses the memory to decode base data of a facial image related to a facial video and one or more extended data of the facial image from a bit stream, decode geometric information from the bit stream, which is information corresponding to each of multiple frames of the facial video and indicates geometric attributes within an area including a person's face, and generate the facial video from the base data, the one or more extended data, and the geometric information using a generative model, wherein the base data is attached to data in the bit stream corresponding to a first frame, which is a frame of the facial video, and the one or more extended data are attached to one or more data in the bit stream corresponding to one or more second frames, which are one or more frames of the facial video and are one or more frames after the first frame.

[0008] Each embodiment of the present disclosure, or a partial configuration or method thereof, enables at least one of, for example, improved coding efficiency, improved image quality, reduced encoding / decoding processing volume, reduced circuit size, or improved encoding / decoding processing speed. Alternatively, each embodiment of the present disclosure, or a partial configuration or method thereof, enables appropriate selection of components / operations such as filters, block sizes, motion vectors, reference pictures, and reference blocks in encoding and decoding. Note that the present disclosure also includes disclosure of configurations or methods that may provide benefits other than those described above. For example, a configuration or method that improves coding efficiency while suppressing an increase in processing volume.

[0009] Further advantages and benefits of certain aspects of the present disclosure will become apparent from the specification and drawings. While such advantages and / or benefits may be obtained by several embodiments and features described in the specification and drawings, not all of them necessarily need to be provided to obtain one or more advantages and / or benefits.

[0010] These general or specific aspects may be realized by a system, an integrated circuit, a computer program, or a recording medium such as a computer-readable CD-ROM, or may be realized by any combination of a system, a method, an integrated circuit, a computer program, and a recording medium.

[0011] A configuration or method according to an aspect of the present disclosure may contribute to, for example, one or more of improved coding efficiency, improved image quality, reduced processing amount, reduced circuit size, improved processing speed, and appropriate selection of elements or operations, etc. Note that a configuration or method according to an aspect of the present disclosure may also contribute to benefits other than those described above.

[0012] FIG. 1 is a block diagram showing an example of the configuration of an encoding / decoding system according to an embodiment. FIG. 2 is a diagram showing an example of the hierarchical structure of data in a stream. FIG. 3 is a block diagram showing an example of the configuration of an encoding device according to an embodiment. FIG. 4 is a block diagram showing an example of the configuration of a decoding device according to an embodiment. FIG. 5 is a conceptual diagram showing an example of a facial image. FIG. 6 is a conceptual diagram showing an example of geometric attributes. FIG. 7 is a conceptual diagram showing an example of a facial moving image. FIG. 8 is a conceptual diagram showing an example of the configuration of data corresponding to each frame. FIG. 9 is a conceptual diagram showing the size of data in a CPB buffer. FIG. 10 is a conceptual diagram showing another example of the configuration of data corresponding to each frame. FIG. 11 is a conceptual diagram showing yet another example of the configuration of data corresponding to each frame. FIG. 12 is a conceptual diagram showing an example of the configuration of base data and extension data. FIG. 13 is a conceptual diagram showing another example of the configuration of base data and extension data. FIG. 14 is a conceptual diagram showing yet another example of the configuration of base data and extension data. FIG. 15 is a conceptual diagram showing yet another example of the configuration of base data and extension data. FIG. 16 is a conceptual diagram showing yet another example of the configuration of base data and extension data.

[0046] Fig. 17 is a conceptual diagram showing yet another example configuration of data corresponding to each frame. Fig. 18 is a conceptual diagram showing yet another example configuration of data corresponding to each frame. Fig. 19 is a conceptual diagram showing an example control of facial moving image generation. Fig. 20 is a block diagram showing yet another example configuration of an encoding device in an embodiment. Fig. 21 is a block diagram showing yet another example configuration of a decoding device in an embodiment. Fig. 22 is a flowchart showing an example operation of an encoding device in an embodiment. Fig. 23 is a flowchart showing an example operation of a decoding device in an embodiment. Fig. 24 is a block diagram showing yet another example configuration of an encoding device in an embodiment. Fig. 25 is a flowchart showing another example operation of an encoding device in an embodiment. Fig. 26 is a block diagram showing yet another example configuration of a decoding device in an embodiment. Fig. 27 is a flowchart showing another example operation of a decoding device in an embodiment. Fig. 28 is a block diagram showing yet another example configuration of a decoding device in an embodiment. Fig. 29 is a block diagram showing yet another example configuration of a decoding device in an embodiment.FIG. 30 is a diagram showing examples of various neural networks that can be used as generative models. FIG. 31 is a block diagram showing an example configuration for an encoding device according to an embodiment to encode moving images. FIG. 32 is a block diagram showing an example configuration for a decoding device according to an embodiment to decode moving images. FIG. 33 is a block diagram showing an example implementation of an encoding device according to an embodiment. FIG. 34 is a flowchart showing an example basic operation of an encoding device according to an embodiment. FIG. 35 is a block diagram showing an example implementation of a decoding device according to an embodiment. FIG. 36 is a flowchart showing an example basic operation of a decoding device according to an embodiment. FIG. 37 is a diagram showing the overall configuration of a content supply system that realizes a content distribution service. FIG. 38 is a diagram showing an example display screen of a web page. FIG. 39 is a diagram showing an example display screen of a web page. FIG. 40 is a diagram showing an example of a smartphone. FIG. 41 is a block diagram showing an example configuration of a smartphone.

[0013] Introduction For example, in video conferencing, an encoding device encodes video containing faces into a bitstream, and a decoding device decodes the video from the bitstream.

[0014] To reduce the amount of code, the encoding device may encode a facial image in a first frame and encode geometric information indicating geometric attributes within a region including a face for each frame.The decoding device may then decode the facial image in the first frame and decode the geometric information for each frame.The decoding device may then reconstruct the facial image for each frame based on the facial image decoded in the first frame and the geometric information decoded for each frame.

[0015] Here, the geometric attributes correspond to, for example, dynamic attributes and may be represented by a point cloud such as a facial landmark, or by a polygon model for representing the shape of an object by combining multiple polygons. Alternatively, the geometric attributes may be represented by other geometric models. Alternatively, the geometric attributes may be represented by the positions of facial features.

[0016] The amount of code required for the geometric information is assumed to be less than the amount of code required for the facial image. Therefore, by encoding and decoding the geometric information for each frame, it becomes possible to reproduce a facial moving image with a smaller amount of code than by encoding and decoding the facial image for each frame.

[0017] However, by encoding and decoding the face image in the first frame, the amount of code corresponding to the first frame increases, which may cause a delay.

[0018] Therefore, the decoding device of Example 1 includes a memory and a circuit connected to the memory, and the circuit uses the memory to decode base data of a facial image related to a facial video and one or more extension data of the facial image from a bit stream, decode geometric information from the bit stream, which is information corresponding to each of multiple frames of the facial video and indicates geometric attributes within an area including a person's face, and generate the facial video from the base data, the one or more extension data, and the geometric information using a generative model, wherein the base data is attached to data in the bit stream corresponding to a first frame, which is a frame of the facial video, and the one or more extension data are attached to one or more data in the bit stream corresponding to one or more second frames, which are one or more frames of the facial video and are one or more frames after the first frame.

[0019] This may make it possible to decode the base data and one or more extension data related to the face image by dividing them into multiple frames, thereby reducing the amount of code corresponding to one frame, and thus reducing delays.

[0020] Furthermore, the decoding device of Example 2 may be the decoding device of Example 1, in which the circuit decodes control information relating to the control of at least one of the multiple facial image data, which are the base data and the one or more extension data, from a header area.

[0021] This may make it possible to apply appropriate processing to facial image data in accordance with the control information when reproducing a facial moving image.

[0022] Furthermore, the decoding device of Example 3 may be the decoding device of Example 2, in which the control information includes presence information indicating whether or not facial image data that is one of the plurality of facial image data is included in an access unit controlled by the header area.

[0023] This may enable, in reproducing a facial moving image, identification of whether or not an access unit contains facial image data in accordance with the control information, thereby enabling appropriate processing to be applied to the facial image data.

[0024] Furthermore, the decoding device of Example 4 may be the decoding device of Example 2 or Example 3, wherein the control information includes, when an access unit controlled by the header area includes facial image data that is one of the plurality of facial image data, type information regarding whether the facial image data is the base data or extended data that is one of the one or more extended data, the type information indicating, when the access unit includes the base data, that the facial image data included in the access unit is the base data and will continue to be used up to the next base data, and the type information indicating, when the access unit includes the extended data, that the facial image data included in the access unit is the extended data and will be used together with the base data.

[0025] This may enable, in reproducing a facial moving image, identification of whether the facial image data is base data or extended data according to the control information, and may enable application of appropriate processing to the facial image data depending on whether the facial image data is base data or extended data.

[0026] Furthermore, the decoding device of Example 5 may be a decoding device that is any one of Examples 2 to 4, and the control information may include application information indicating whether or not facial image data that is one of the plurality of facial image data is applicable to generating and displaying a frame corresponding to the access unit among the plurality of frames of the facial moving image when the access unit controlled by the header area includes the facial image data.

[0027] This may make it possible to appropriately control whether or not to apply facial image data attached to the data corresponding to a frame of a facial moving image in accordance with the control information when reproducing the facial moving image.

[0028] Furthermore, the decoding device of Example 6 may be any of the decoding devices of Examples 1 to 5, in which the base data and the one or more extended data are each represented by a vector indicating facial features included in the facial image.

[0029] This may make it possible to reduce the amount of coding required for facial images, thereby suppressing delays.

[0030] Furthermore, the decoding device of Example 7 may be any of the decoding devices of Examples 1 to 5, in which the base data and the one or more extended data are each represented by an image related to the facial image.

[0031] This may make it possible to appropriately reflect the base data and extended data relating to the facial image as image data in the frames of the facial moving image when reproducing the facial moving image.

[0032] Furthermore, the decoding device of Example 8 may be any of the decoding devices of Examples 1 to 7, in which the circuit inputs the base data, at least one of the one or more extension data, and the geometric information into the generative model to generate frames of the facial video image.

[0033] This may make it possible to omit the process of generating an intermediate image from base data and extended data related to a facial image, thereby simplifying the process of generating a facial moving image.

[0034] Furthermore, the decoding device of Example 9 may be any of the decoding devices of Examples 1 to 7, in which the circuit generates an intermediate image from the base data and at least one of the one or more extension data, and inputs the intermediate image and the geometric information into the generative model to generate a frame of the facial video image.

[0035] This may enable appropriate generation of an intermediate image related to the facial image from the base data and extended data related to the facial image, and may enable appropriate reflection of the intermediate image related to the facial image in frames of the facial moving image.

[0036] Furthermore, the decoding device of Example 10 may be any of the decoding devices of Examples 1 to 7, in which the circuit refers to the base data, decodes extended data that is one of the one or more extended data, and inputs the extended data and the geometric information into the generative model to generate a frame of the facial video image.

[0037] This may enable efficient decoding of extended data with higher accuracy than the accuracy of the base data, and may enable highly accurate generation of facial motion image frames using the high-accuracy extended data.

[0038] Furthermore, the decoding device of Example 11 may be a decoding device that is any one of Examples 1 to 9, in which the base data is data of a part of a face included in the facial image, and the extended data that is one of the one or more extended data is data of another part of the face included in the facial image.

[0039] This may allow the facial image to be decoded in parts without complex processing.

[0040] Furthermore, the decoding device of Example 12 may be any of the decoding devices of Examples 1 to 9, in which the base data is data in a first frequency range in the facial image, and the extended data that is one of the one or more extended data is data in a second frequency range that is higher than the first frequency range in the facial image.

[0041] This may make it possible to decode low-frequency component data in the facial image as base data and high-frequency component data in the facial image as extended data. Furthermore, in reproducing a facial motion image, it may be possible to apply the low-frequency component data to generating a first frame of the facial motion image and apply both the low-frequency component data and the high-frequency component data to generating a second frame of the facial motion image. Therefore, it may be possible to suppress delays and to suppress unnaturalness in the facial motion image.

[0042] Furthermore, the decoding device of Example 13 may be a decoding device of any of Examples 1 to 10, wherein the base data is an image related to the facial image and corresponds to a first image which is an image having a first resolution, and the extended data which is one of the one or more extended data is an image related to the facial image, is an image decoded with reference to the first image, and corresponds to a second image which is an image having a second resolution higher than the first resolution.

[0043] This may enable decoding a low-resolution facial image as base data and decoding a high-resolution facial image as extended data. Furthermore, in reproducing a facial motion image, the low-resolution facial image may be applied to generating a first frame of the facial motion image, and the high-resolution facial image may be applied to generating a second frame of the facial motion image. Therefore, it may be possible to suppress delays and to suppress unnaturalness in the facial motion image.

[0044] Furthermore, the decoding device of Example 14 may be a decoding device of any of Examples 1 to 10, wherein the base data is an image related to the facial image and corresponds to a first image that is an image decoded with a first quantization width, and the extended data that is one of the one or more extended data is an image related to the facial image and corresponds to a second image that is an image decoded with a second quantization width that is finer than the first quantization width by referring to the first image.

[0045] This may make it possible to decode the coarse facial image as base data and decode the fine facial image as extended data. Furthermore, in reproducing a facial motion image, it may be possible to apply the coarse facial image to generating a first frame of the facial motion image and apply the fine facial image to generating a second frame of the facial motion image. Therefore, it may be possible to suppress delays and to suppress unnaturalness in the facial motion image.

[0046] Furthermore, the decoding device of Example 15 may be any of the decoding devices of Examples 2 to 5, in which the control information includes identification information for identifying each of the one or more extension data.

[0047] This may make it possible to identify each piece of extended data in accordance with the control information when reproducing a facial moving image, and thus to individually specify each piece of extended data and control the application of each piece of extended data.

[0048] Furthermore, the decoding device of Example 16 may be any of the decoding devices of Examples 2 to 5 and Example 15, in which the control information (i) is information included in the header area of ​​the access unit including the base data, and (ii) includes total number information that is information indicating the total number of the one or more extension data.

[0049] This may enable, in the reproduction of the facial motion image, to identify the total number of one or more pieces of augmented data according to the control information, and thus, to efficiently determine, according to the total number of one or more pieces of augmented data, one or more pieces of augmented data that may be used in the reproduction of the facial motion image.

[0050] Furthermore, the decoding device of Example 17 may be a decoding device of any one of Examples 2 to 5, Example 15, and Example 16, in which the control information (i) is information included in the header region of an access unit including the base data, and (ii) includes specific information that is extension data among the one or more extension data and is information for identifying extension data that is applicable to the generation and display of a second frame among the one or more second frames corresponding to the access unit including the extension data.

[0051] This may enable appropriate identification of extension data applicable to generating and displaying frames of facial motion images in accordance with the control information when reproducing facial motion images.

[0052] Furthermore, the decoding device of Example 18 may be a decoding device of any one of Examples 1 to 17, wherein the circuit decodes at least one control parameter for controlling a stream buffer in which the bitstream is stored in the memory, the control parameter being for controlling the buffer size of the stream buffer to be equal to or less than a reference size and for controlling the initial delay time for starting the decoding process to be equal to or less than the reference delay time.

[0053] This makes it possible to reduce resources required for decoding and shorten delay times.

[0054] Further, the encoding device of Example 19 includes a memory and a circuit connected to the memory, and the circuit uses the memory to encode base data of a facial image related to a facial video and one or more extension data of the facial image into a bit stream, and encodes geometric information into the bit stream, the geometric information being information corresponding to each of a plurality of frames of the facial video and indicating geometric attributes within an area including a person's face, the base data being attached to data corresponding to a first frame, which is a frame of the facial video, and the one or more extension data being attached to one or more data corresponding to one or more second frames, which are one or more frames of the facial video and one or more frames after the first frame.

[0055] This may make it possible to encode the base data and one or more extension data related to a face image separately into multiple frames, thereby reducing the amount of code corresponding to one frame and suppressing delays.

[0056] Furthermore, the encoding device of Example 20 may be the encoding device of Example 19, in which the circuit encodes control information regarding control of at least one of the multiple facial image data, which are the base data and the one or more extension data, into a header area.

[0057] This may make it possible to apply appropriate processing to facial image data in accordance with the control information when reproducing a facial moving image.

[0058] Furthermore, the encoding device of Example 21 may be the encoding device of Example 20, in which the control information includes presence information indicating whether or not facial image data that is one of the multiple facial image data is included in the access unit controlled by the header area.

[0059] This may enable, in reproducing a facial moving image, identification of whether or not an access unit contains facial image data in accordance with the control information, thereby enabling appropriate processing to be applied to the facial image data.

[0060] Furthermore, the encoding device of Example 22 may be the encoding device of Example 20 or Example 21, wherein the control information includes type information regarding whether, when an access unit controlled by the header area includes facial image data that is one of the plurality of facial image data, the facial image data is the base data or extended data that is one of the one or more extended data, and the type information indicates, when the access unit includes the base data, that the facial image data included in the access unit is the base data and will continue to be used up to the next base data, and the type information indicates, when the access unit includes the extended data, that the facial image data included in the access unit is the extended data and will be used together with the base data.

[0061] This may enable, in reproducing a facial moving image, identification of whether the facial image data is base data or extended data according to the control information, and may enable application of appropriate processing to the facial image data depending on whether the facial image data is base data or extended data.

[0062] Furthermore, the encoding device of Example 23 may be an encoding device that is any one of Examples 20 to 22, and the control information may include application information indicating whether or not facial image data that is one of the plurality of facial image data is applicable to generating and displaying a frame corresponding to the access unit among the plurality of frames of the facial moving image when the access unit controlled by the header area includes facial image data that is one of the plurality of facial image data.

[0063] This may make it possible to appropriately control whether or not to apply facial image data attached to the data corresponding to a frame of a facial moving image in accordance with the control information when reproducing the facial moving image.

[0064] Furthermore, the encoding device of Example 24 may be any of the encoding devices of Examples 19 to 23, in which the base data and the one or more extended data are each represented by a vector indicating facial features included in the facial image.

[0065] This may make it possible to reduce the amount of coding required for facial images, thereby suppressing delays.

[0066] Furthermore, the encoding device of Example 25 may be any of the encoding devices of Examples 19 to 23, in which the base data and the one or more extended data are each represented by an image related to the facial image.

[0067] This may make it possible to appropriately reflect the base data and extended data relating to the facial image as image data in the frames of the facial moving image when reproducing the facial moving image.

[0068] Furthermore, the encoding device of Example 26 may be any of the encoding devices of Examples 19 to 25, in which the circuit derives and encodes data of a portion of the face included in the facial image as the base data, and derives and encodes data of another portion of the face included in the facial image as extended data that is one of the one or more extended data.

[0069] This may allow a facial image to be coded in parts without complex processing.

[0070] Furthermore, the encoding device of Example 27 may be any of the encoding devices of Examples 19 to 25, in which the circuit derives and encodes data in a first frequency range in the facial image as the base data, and derives and encodes data in a second frequency range in the facial image that is higher than the first frequency range as extended data that is one of the one or more extended data.

[0071] This may make it possible to encode the low-frequency component data in the facial image as base data and encode the high-frequency component data in the facial image as extended data. Furthermore, in reproducing the facial motion image, it may be possible to apply the low-frequency component data to generating a first frame of the facial motion image and apply both the low-frequency component data and the high-frequency component data to generating a second frame of the facial motion image. Therefore, it may be possible to suppress delays and to suppress unnaturalness in the facial motion image.

[0072] Furthermore, the encoding device of Example 28 may be any of the encoding devices of Examples 19 to 25, in which the circuit encodes a first image, which is an image related to the facial image and has a first resolution, as the base data, and encodes a second image, which is an image related to the facial image, which is encoded with reference to the first image, and which has a second resolution higher than the first resolution, as the base data.

[0073] This may make it possible to encode a low-resolution facial image as base data and encode a high-resolution facial image as extended data. Furthermore, in reproducing a facial motion image, it may be possible to apply the low-resolution facial image to generating a first frame of the facial motion image and the high-resolution facial image to generating a second frame of the facial motion image. Therefore, it may be possible to suppress delays and to suppress unnaturalness in the facial motion image.

[0074] Further, the decoding method of Example 29 is a decoding method that decodes base data of a facial image related to a facial video and one or more extension data of the facial image from a bit stream, decodes geometric information from the bit stream, which is information corresponding to each of multiple frames of the facial video and is information indicating geometric attributes within an area including a person's face, generates the facial video from the base data, the one or more extension data, and the geometric information using a generative model, wherein the base data is attached to data in the bit stream that corresponds to a first frame, which is a frame of the facial video, and the one or more extension data are attached to one or more data in the bit stream that correspond to one or more second frames, which are one or more frames of the facial video and are one or more frames after the first frame.

[0075] This may make it possible to decode the base data and one or more extension data related to the face image by dividing them into multiple frames, thereby reducing the amount of code corresponding to one frame, and thus reducing delays.

[0076] In addition, the encoding method of Example 30 encodes base data of a facial image related to a facial video and one or more extension data of the facial image into a bit stream, encodes geometric information into the bit stream, the geometric information being information corresponding to each of multiple frames of the facial video and indicating geometric attributes within an area including a person's face, the base data being added to data corresponding to a first frame, which is a frame of the facial video, in the bit stream, and the one or more extension data being added to one or more data corresponding to one or more second frames, which are one or more frames of the facial video and are one or more frames after the first frame, in the bit stream.

[0077] This may make it possible to encode the base data and one or more extension data related to a face image separately into multiple frames, thereby reducing the amount of code corresponding to one frame and suppressing delays.

[0078] Furthermore, these comprehensive or specific aspects may be realized as a system, an apparatus, a method, an integrated circuit, a computer program, or a non-transitory recording medium such as a computer-readable CD-ROM, or may be realized as any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium.

[0079] [Definition of Terms] As an example, each term may be defined as follows.

[0080] (1) Image: A unit of data made up of a set of pixels, consisting of pictures or blocks smaller than pictures, and includes both moving images and still images.

[0081] (2) Picture: A processing unit of an image composed of a set of pixels, and is sometimes called a frame or field.

[0082] (3) Block: A processing unit for a set containing a specific number of pixels, and can be named anything, as shown in the following examples. It can also be shaped anything, including, for example, a rectangle made up of M×N pixels, a square made up of M×M pixels, a triangle, a circle, or any other shape.

[0083] (Examples of blocks) Slice / tile / brick CTU / superblock / basic division unit VPDU / hardware processing division unit CU / processing block unit / prediction block unit (PU) / orthogonal transform block unit (TU) / unit Sub-block

[0084] (4) Pixel / Sample A pixel / sample is a minimum unit point that constitutes an image, and includes not only pixels at integer positions but also pixels at decimal positions generated based on pixels at integer positions.

[0085] (5) Pixel Value / Sample Value: A value inherent to a pixel, including not only brightness value, color difference value, and RGB gradation, but also depth value or binary values ​​of 0 and 1.

[0086] (6) Flags: In addition to one bit, flags may be multi-bit, for example, parameters or indexes of two or more bits. In addition, flags may be multi-valued using other bases as well as two values ​​using binary numbers.

[0087] (7) Signal: A signal that is symbolized or coded to transmit information, including discrete digital signals as well as analog signals that take continuous values.

[0088] (8) Stream / Bitstream: A digital data string or flow. A stream / bitstream may consist of a single stream or multiple streams divided into multiple layers. It also includes transmission by serial communication over a single transmission path as well as transmission by packet communication over multiple transmission paths.

[0089] (9) Difference / Difference In the case of scalar quantities, in addition to simple difference (x-y), it is sufficient to include difference calculations, including absolute value of difference (|x-y|), squared difference (x^2-y^2), square root of difference (√(x-y)), weighted difference (ax-by: a, b are constants), and offset difference (x-y+a: a is an offset).

[0090] (10) Sum In the case of a scalar quantity, in addition to simple sum (x + y), it is sufficient if a sum operation is included, including the absolute value of the sum (|x + y|), sum of squares (x^2 + y^2), square root of the sum (√(x + y)), weighted sum (ax + by: a, b are constants), and offset sum (x + y + a: a is an offset).

[0091] (11) Based on: This includes cases where factors other than the one being based on are taken into consideration. It also includes cases where a result is obtained directly or via an intermediate result.

[0092] (12) Using (used, using) This includes cases where elements other than the target of use are taken into account. It also includes cases where a result is obtained directly, as well as cases where a result is obtained via an intermediate result.

[0093] (13) Prohibit (forbid) This can be rephrased as not being allowed. Also, not prohibiting or being allowed does not necessarily mean obligation.

[0094] (14) Limit (restriction / restrict / restricted) This can be rephrased as not being permitted. Also, not prohibiting something or being permitted does not necessarily mean that it is an obligation. Furthermore, it is sufficient if something is partially prohibited in terms of quantity or quality, and it also includes cases where something is completely prohibited.

[0095] (15) Chroma: An adjective, denoted by the symbols Cb and Cr, that specifies that a sample array or a single sample represents one of two color difference signals associated with a primary color. Instead of the term chroma, the term chrominance can also be used.

[0096] (16) Luma: An adjective, denoted by the symbol or subscript Y or L, that specifies that a sample array or a single sample represents a monochrome signal associated with a primary color. Instead of the term luma, the term luminance may also be used.

[0097] [Description] In the drawings, the same reference numerals refer to the same or similar elements, and the sizes and relative positions of the elements in the drawings are not necessarily drawn to scale.

[0098] Hereinafter, embodiments will be described in detail with reference to the drawings. Note that the embodiments described below are all comprehensive or specific examples. The numerical values, shapes, materials, components, the arrangement and connection of the components, steps, and the relationship and order of the steps shown in the following embodiments are merely examples and are not intended to limit the scope of the claims.

[0099] Below, embodiments of an encoding device and a decoding device will be described. The embodiments are examples of encoding devices and decoding devices to which the processes and / or configurations described in each aspect of the present disclosure can be applied. The processes and / or configurations can also be implemented in encoding devices and decoding devices different from the embodiments. For example, with regard to the processes and / or configurations applied to the embodiments, any of the following may be implemented.

[0100] (1) Any of the multiple components of the encoding device or decoding device of the embodiments described in each aspect of the present disclosure may be replaced or combined with other components described in any of the aspects of the present disclosure.

[0101] (2) In the encoding device or decoding device according to the embodiment, the functions or processes performed by some of the components of the encoding device or decoding device may be changed in any way, such as by adding, replacing, or deleting a function or process. For example, any function or process may be replaced with or combined with another function or process described in any of the aspects of the present disclosure.

[0102] (3) In the method implemented by the encoding device or decoding device according to the embodiment, some of the processes included in the method may be arbitrarily modified, such as by addition, replacement, deletion, etc. For example, any process in the method may be replaced with or combined with another process described in any of the aspects of the present disclosure.

[0103] (4) Some of the components constituting the encoding device or decoding device of the embodiment may be combined with components described in any of the aspects of the present disclosure, or may be combined with components having some of the functions described in any of the aspects of the present disclosure, or may be combined with components that perform some of the processing performed by the components described in each aspect of the present disclosure.

[0104] (5) A component having part of the functionality of the encoding device or decoding device of an embodiment, or a component that performs part of the processing of the encoding device or decoding device of an embodiment, may be combined or replaced with a component described in any of the aspects of the present disclosure, a component having part of the functionality described in any of the aspects of the present disclosure, or a component that performs part of the processing described in any of the aspects of the present disclosure.

[0105] (6) In the method implemented by the encoding device or decoding device of the embodiment, any of the multiple processes included in the method may be replaced or combined with the process described in any of the aspects of the present disclosure or any similar process.

[0106] (7) Some of the processes included in the method implemented by the encoding device or decoding device of the embodiment may be combined with the processes described in any of the aspects of the present disclosure.

[0107] (8) The implementation of the processes and / or configurations described in each aspect of the present disclosure is not limited to the encoding device or decoding device of the embodiments. For example, the processes and / or configurations may be implemented in a device used for a purpose other than video encoding or video decoding disclosed in the embodiments.

[0108] 1 is a block diagram showing an example of the configuration of a coding / decoding system according to this embodiment. The coding / decoding system includes a coding device 100 and a decoding device 200. In this embodiment, the coding / decoding system is used for face reproduction.

[0109] Facial reconstruction refers to the process of mapping the facial expressions and poses of one or more source persons onto images of one or more target persons while ensuring that the target persons' identities and attributes are preserved in the images. Facial reconstruction technology has been used in a wide range of applications, from video conferencing to entertainment. Currently, much research is being conducted to enhance photorealistic representations through the introduction of various techniques, such as motion representation extraction or image feature projection into a latent space.

[0110] For example, a video conferencing application is configured with an encoding / decoding architecture. First, a driving video consisting of one or more frames of a user corresponding to a source person is captured by an encoding device 100. Then, the driving video is transmitted to a decoding device 200 based on real-time communication. The decoding device 200 reconstructs and displays a facial video of a target person.

[0111] A facial image of a target person may be transmitted from the encoding device 100 to the decoding device 200. The facial image may be provided live from a camera, may be represented by one or more pre-defined animated avatars, or may be one or more pre-defined source images containing a face, or may be selected by a user.

[0112] Facial reconstruction technology has been widely adopted in the entertainment industry, such as in producing advertisements, editing movie scenes, and enhancing music videos, where emotions, expressions, and poses in one or more driving videos are transferred to the face of a target, while ensuring that the target's identity and appearance are preserved.

[0113] The person in the driving video and the target person may be the same person or different people. The target person is not limited to a real person, but may be a virtual person such as an avatar.

[0114] As various social media applications become more widespread and their usage increases, facial reconstruction technology offers unique customization, flexibility, convenience, and ease in generating expressions that represent emotions and personalities. Various facial reconstruction techniques have been proposed to address scenarios where driving videos are generated in real time or pre-recorded. Furthermore, output videos are generated to appear natural and without distortion. The attributes of the output video may also be adjustable by the user.

[0115] 1, an encoding device 100 receives as input a facial image of a target person and a driving video including multiple frames, encodes and compresses this information into one or more bitstreams, and then transmits the compressed bitstreams to a decoding device 200 using a transmission channel such as a communication network or a recording medium. Finally, the decoding device 200 reconstructs the facial video from the received bitstreams.

[0116] For example, a facial image is an image including a face, and may also be expressed as a reference image or an identity image. A facial image represents static visual features for reconstructing a facial video image. A driving video is a video including a face, and is a captured video image obtained by a camera. The driving video serves to impart movement to the facial image. A bit stream may also be simply expressed as a stream. Furthermore, the use of one bit stream is not limited, and multiple bit streams may also be used.

[0117] The person included in the face image and the person included in the driving video may or may not be the same person.

[0118] The encoding / decoding system according to the present embodiment can be applied to video conferencing, video generation and editing in the entertainment industry, social media, the e-commerce industry, etc. However, the scope of application is not limited to these.

[0119] [Data Structure] Figure 2 is a diagram showing an example of a hierarchical structure of data in a stream. The stream includes, for example, a video sequence. This video sequence includes, for example, a video parameter set (VPS), a sequence parameter set (SPS), a picture parameter set (PPS), supplemental enhancement information (SEI), and multiple pictures, as shown in Figure 2(a).

[0120] In a video composed of multiple layers, the VPS includes coding parameters common to multiple layers, and coding parameters related to multiple layers included in the video or to each individual layer.

[0121] The SPS includes parameters used for the sequence, i.e., encoding parameters that the decoding device 200 refers to in order to decode the sequence. For example, the encoding parameters may indicate the width or height of a picture. Note that there may be multiple SPSs.

[0122] The PPS includes parameters used for a picture, i.e., encoding parameters referenced by the decoding device 200 to decode each picture in a sequence. For example, the encoding parameters may include a reference value of the quantization width used in decoding the picture and a flag indicating the application of weighted prediction. Note that there may be multiple PPSs. Furthermore, the SPS and PPS may be simply referred to as parameter sets.

[0123] A picture may include a picture header and one or more slices, as shown in Fig. 2(b), where the picture header includes coding parameters that the decoding device 200 references to decode the one or more slices.

[0124] As shown in (c) of Fig. 2, a slice includes a slice header and one or more bricks. The slice header includes coding parameters that are referenced by the decoding device 200 to decode the one or more bricks.

[0125] A brick includes one or more coding tree units (CTUs), as shown in FIG. 2(d).

[0126] Note that a picture may not contain slices, but may instead contain tile groups, where a tile group contains one or more tiles, and a brick may contain slices.

[0127] A CTU is also called a superblock or a basic division unit. As shown in (e) of Fig. 2, such a CTU includes a CTU header and one or more coding units (CUs). The CTU header includes coding parameters that the decoding device 200 references to decode the one or more CUs.

[0128] A CU may be divided into multiple small CUs. Furthermore, as shown in (f) of FIG. 2, a CU includes a CU header, prediction information, and residual coefficient information. The prediction information is information for predicting the CU, and the residual coefficient information is information indicating a prediction residual, which will be described later. A CU is basically the same as a PU (Prediction Unit) and a TU (Transform Unit), but may include multiple TUs smaller than the CU, for example, in an SBT, which will be described later. A CU may also be processed for each VPDU (Virtual Pipeline Decoding Unit) that constitutes the CU. A VPDU is a fixed unit that can be processed in one stage, for example, when performing pipeline processing in hardware.

[0129] Note that the stream may not have some of the layers shown in FIG. 2 . The order of these layers may be changed, or some layers may be replaced with other layers. A picture currently being processed by a device such as the encoding device 100 or the decoding device 200 is referred to as a current picture. If the processing is encoding, the current picture is synonymous with a picture to be encoded, and if the processing is decoding, the current picture is synonymous with a picture to be decoded. A block, such as a CU or CU, currently being processed by a device such as the encoding device 100 or the decoding device 200 is referred to as a current block. If the processing is encoding, the current block is synonymous with a block to be encoded, and if the processing is decoding, the current block is synonymous with a block to be decoded.

[0130] Here, a region in which parameters used for encoding and decoding are described may be referred to as a header region. For example, the header region is a region including an SEI. The header region may further include a VPS, an SPS, a PPS, an SEI, a picture header, a slice header, a CTU header, and a CU header.

[0131] Furthermore, for example, pictures can be classified into one of several types, including I-pictures, P-pictures, and B-pictures. I-pictures are intra-predicted pictures, also called intra-pictures, that are coded and decoded without reference to other pictures. P-pictures are uni-predicted pictures that can be coded and decoded with reference to one other picture. B-pictures are bi-predicted pictures that can be coded and decoded with reference to two other pictures.

[0132] Furthermore, a moving image may be composed of a plurality of GOPs (Groups of Pictures). A GOP means a set of pictures. A GOP includes one or more I pictures. A GOP may include one or more P pictures, or one or more B pictures. A GOP may be a unit that allows video editing and random access. A GOP may have a fixed number of pictures, or may have a GOP structure in which I pictures, P pictures, and B pictures are arranged in a fixed order.

[0133] [Encoding and Decoding Configuration and Processing] Fig. 3 is a block diagram showing an example configuration of an encoding device 100 according to this embodiment. In this example, the encoding device 100 generates a bitstream including compressed facial images, compressed driving videos, and compressed background images from facial images, driving videos, and background images. The encoding device 100 includes a compressor 131, a deriver 132, a compressor 133, and a compressor 134. Each component is, for example, an electrical circuit that performs information processing. Two or more of the compressors 131, 133, and 134 may be integrated together.

[0134] Here, the facial image includes the face of the target person. Each of a plurality of facial images including the face of the target person may be used as the facial image. Furthermore, each driving frame of the driving video includes the face of the source person. Furthermore, each of a plurality of background images may be used as the background image.

[0135] First, compressor 131 compresses the face image by encoding it into a bitstream. Then, deriving unit 132 derives geometric information indicating the geometric attributes of the region including the face of the source person from each driving frame of the driving video captured directly from the camera. Then, compressor 133 compresses the geometric information by encoding it into a bitstream. Then, compressor 134 compresses the background image by encoding it into a bitstream.

[0136] Here, the geometric attributes correspond to, for example, dynamic attributes and may be represented by a point cloud such as a facial landmark, or by a polygon model for representing the shape of an object by combining multiple polygons. Alternatively, the geometric attributes may be represented by other geometric models. Alternatively, the geometric attributes may be represented by the positions of facial features.

[0137] For example, both the geometric information and the background image are transmitted from the encoding device 100 for each frame (each time instance). Also, for example, the face image is transmitted in the first frame (first time instance). The face image does not have to be transmitted for each frame. In particular, if the face image is the same as the face image in the previous frame, it does not have to be transmitted.

[0138] In a variant, the deriver 132 may derive a segmentation mask indicating foreground and background regions in the driving frame for each frame, and the compressor 133 may compress the segmentation mask by encoding it into a bitstream. The segmentation mask may then be transmitted to the decoding device 200. The segmentation mask of the driving frame corresponds to the segmentation mask of the frame in the facial video sequence to be reproduced.

[0139] Note that the background image is not an essential element, and therefore the encoding device 100 does not need to include the compressor 134, and the bitstream does not need to include a compressed background image.

[0140] 4 is a block diagram showing an example of the configuration of a decoding device 200 according to this embodiment. The decoding device 200 generates a facial moving image from a bitstream. In this example, the decoding device 200 includes a decompressor 231, a deriver 232, a decompressor 233, a deriver 237, a generator 234, and a decompressor 235. Each component is, for example, an electric circuit that performs information processing. Two or more of the decompressor 231, the decompressor 233, and the decompressor 235 may be integrated together.

[0141] The decompressor 231 decompresses the facial image by decoding it from the bitstream. Then, the deriving unit 232 derives facial information indicating facial attributes from the facial image. Here, the facial attributes are static visual attributes of a face. The facial attributes can also be expressed as identity.

[0142] The decompressor 233 decompresses the geometric information for each frame by decoding it from the bitstream, and the deriver 237 derives a segmentation mask from the geometric information for each frame, which indicates foreground and background regions in the facial video.

[0143] The decompressor 235 decodes the background image from the bitstream to be embedded in the facial motion image.

[0144] The generator 234 receives face information, geometric information, a segmentation mask, and a background image as input, and generates a face moving image including multiple frames. Specifically, the generator 234 generates a face moving image from the face information, geometric information, a segmentation mask, and the background image using a generative model such as a neural network. This makes it possible to reproduce a face moving image while suppressing the amount of coding.

[0145] For example, both the geometric information and the background image are transmitted to the decoding device 200 for each frame. Also, for example, a face image is transmitted in the first frame. A face image does not have to be transmitted for each frame. In particular, if a face image is not transmitted, a face image decoded in a previous frame may be used.

[0146] The background image is not an essential element, and therefore the decoding device 200 does not need to include the decompressor 235.

[0147] Alternatively, the decompressor 233 may decode the segmentation mask, and the decoded segmentation mask may be used in the generator 234. Alternatively, the segmentation mask is not an essential element. Therefore, the decoding device 200 may not include the deriving device 237. Alternatively, the derivation of face information may be omitted. Furthermore, a face image may be used instead of face information to generate a face moving image. Therefore, the decoding device 200 may not include the deriving device 232.

[0148] For example, the generator 234 may generate facial animation images based solely on facial images and geometric information.

[0149] 5 is a conceptual diagram showing an example of a face image. As in this example, the face image is an image that includes a face.

[0150] 6 is a conceptual diagram showing an example of a geometric attribute. In this example, the geometric attribute is a facial landmark. For example, geometric information indicating the geometric attribute is derived for each frame of a driving video.

[0151] 7 is a conceptual diagram showing an example of a facial motion image. As shown in this example, the facial motion image is a motion image including a face. In the facial motion image, for each of a plurality of frames, the geometric attributes of the frame are reflected in the facial image. This gives the facial image movement.

[0152] As mentioned above, the facial image may be transmitted in the first frame and not in each subsequent frame. This can reduce the amount of bits. However, in facial reproduction, large differences in the amount of bits between frames may cause large fluctuations in buffer size. Here, the buffer size refers to the buffer size of the stream buffer, i.e., the size of the data in the stream buffer.

[0153] 8 is a conceptual diagram showing an example of the structure of data corresponding to each frame. For example, POC#0 means that the POC (Picture Order Count) is 0. In this example, the coding order and display order match, and frames are coded and decoded in the order of POC#0, POC#1, POC#2, and POC#3. POC#0 may be the first frame in a sequence, i.e., a bitstream, or the first frame in a GOP.

[0154] In this example, facial image data is encoded in the first frame. Geometric information is encoded in each frame. The size of the geometric information data is smaller than the size of the facial image data. Therefore, the size of the data corresponding to the first frame in which a facial image is transmitted is large, and the size of the data corresponding to each subsequent frame in which a facial image is not transmitted is small. This can cause large fluctuations in buffer size, making low-latency transmission difficult.

[0155] For example, by reducing the buffer size of a coded picture buffer (CPB), decoding can be started early and low-delay transmission can be achieved. However, if the data size of the first frame is large, it is difficult to reduce the buffer size.

[0156] 9 is a conceptual diagram showing the size of data in the CPB buffer. Specifically, an example of buffer underflow occurring in the CPB is shown. Buffer underflow occurs when the buffer level falls below 0. In this case, there is no full frame data in the CPB buffer, so a decoded image cannot be generated, resulting in a delay in decoding.

[0157] Specifically, in this example, because the data size of the first frame is large, there is a shortage of data for the first frame when it is time to decode the first frame. This causes a delay. In other words, delays may occur due to differences in data size between frames. In other words, buffer underflow is more likely to occur when a frame with a larger data size than the data sizes of other frames is acquired. Therefore, it is better for the difference in data size between frames to be small.

[0158] For example, the encoding device 100 may encode at least one control parameter for controlling a stream buffer in which a bitstream is stored in a memory of the decoding device 200. The stream buffer may be, for example, a CPB. Then, the decoding device 200 may decode the at least one control parameter. Then, the decoding device 200 may control the stream buffer in accordance with the at least one control parameter.

[0159] The at least one control parameter may include at least one parameter specifying a buffer size of the stream buffer and an initial delay time for starting the decoding process, the initial delay time for starting the decoding process corresponding to the timing at which data of the bitstream is first obtained from the stream buffer.

[0160] The at least one control parameter may be set so that the buffer size of the stream buffer is equal to or smaller than a reference size and the initial delay time for starting the decoding process is equal to or smaller than a reference delay time, where the reference size may be any size smaller than a standard size and the reference delay time may be any delay time shorter than the standard delay time.

[0161] Furthermore, at least one control parameter may be set so that the buffer size of the stream buffer is minimized and the initial delay time for starting the decoding process is minimized within a range in which buffer underflow does not occur. In this case, the reference size may be regarded as the smallest size within a range in which buffer underflow does not occur, and the reference delay time may be regarded as the shortest delay time within a range in which buffer underflow does not occur.

[0162] By reducing the buffer size of the stream buffer, the size of data remaining in the stream buffer is reduced. With a small buffer size, data can be transmitted and processed without buffer underflow, resulting in low latency data processing. Therefore, it is better to keep the increase or decrease in the size of transmitted data between frames as small as possible.

[0163] 10 is a conceptual diagram showing another example of the configuration of data corresponding to each frame. In this example, facial image data is separated into multiple pieces of data and transmitted in different frames. Specifically, the facial image data is separated into base data and one or more pieces of extended data. The base data and the one or more pieces of extended data contain different contents of the facial image data.

[0164] The base data corresponds to, for example, a picture, also referred to as a base picture, and the picture decoded and output from the base data provides a reference texture from which a plurality of face pictures corresponding to a plurality of frames of a facial video sequence can be generated.

[0165] The augmented data may correspond to, for example, pictures used for fusion. The augmented data may be pictures input to a generative model for generating facial motion images, such as a neural network. That is, the augmented data may be used as driving pictures for augmenting facial images represented by the base data.

[0166] Fusion may be performed by inputting reconstructed data obtained by adding together the base data and the extended data into a generative model, or by inputting the base data and the extended data into a generative model.

[0167] In this example, the facial image data is separated into base data and three extension data. The facial image data is completed with the last extension data at POC#3. After POC#3, the facial image data is not transmitted.

[0168] Here, the data of a facial image is also referred to as facial image data. The facial image data may be data of the entire facial image, base data, or each of one or more pieces of extended data. Furthermore, the facial image data may be data reconstructed from the base data and at least one of the one or more pieces of extended data.

[0169] For example, after decoding the base data and geometric information corresponding to POC#0, a frame corresponding to POC#0 in a facial video is generated from the base data and geometric information corresponding to POC#0 using a generative model. The base data corresponding to POC#0 is also regarded as facial image data corresponding to POC#0.

[0170] Furthermore, after the extension data and geometric information corresponding to POC#1 are decoded, the extension data corresponding to POC#1 is added to the base data, and the facial image data corresponding to POC#1 is reconstructed. The facial image data and geometric information corresponding to POC#1 are used to generate a frame corresponding to POC#1 in the facial video.

[0171] When generating frames corresponding to POC#1 in a facial video, not only facial image data corresponding to POC#1 but also facial image data corresponding to POC#0 may be input to the generative model. Alternatively, base data corresponding to POC#0 and extended data corresponding to POC#1 may be input to the generative model.

[0172] Similarly, after decoding the extended data and geometric information corresponding to POC#2, the extended data corresponding to POC#2 is added to the facial image data corresponding to POC#1 to reconstruct the facial image data corresponding to POC#2. The facial image data and geometric information corresponding to POC#2 are used to generate a frame corresponding to POC#2 in the facial video.

[0173] When generating frames corresponding to POC#2 in a facial video, not only facial image data corresponding to POC#2 but also facial image data corresponding to POC#0 or POC#1 may be input to the generative model. Alternatively, base data corresponding to POC#0, extended data corresponding to POC#1, and extended data corresponding to POC#2 may be input to the generative model.

[0174] In this example, the data size for one frame is prevented from becoming too large, which makes it possible to reduce the size of the stream buffer and suppress delays.

[0175] Note that, as in the example of Fig. 10, the base data and one or more pieces of extended data may be transmitted in temporally consecutive POCs, which may enable all face image data to be quickly collected while reducing the maximum CPB buffer size.

[0176] If there is too little extended data, i.e., if the number of extended data is too small, the maximum CPB buffer size will not be reduced sufficiently. On the other hand, if there is too much extended data, i.e., if the number of extended data is too large, it will take a long time to collect all the facial image data. The number of extended data may be three. The number of extended data may be determined by the user based on delay, image quality, etc.

[0177] 11 is a conceptual diagram showing another example of the structure of data corresponding to each frame. In this example, the facial image data is separated into base data and two pieces of extended data. The facial image data is completed with the last piece of extended data in POC#4.

[0178] The one or more extension data may be transmitted in any order and in any frame, based on factors such as delay and picture quality.

[0179] Note that, as in the example of Fig. 11, the base data and one or more pieces of extended data may be transmitted in POCs that are not consecutive in time, which may reduce the maximum CPB buffer size while reducing the number of pieces of extended data and the amount of processing.

[0180] Furthermore, geometric information does not have to be transmitted for each of all frames. For example, geometric information does not have to be transmitted for frames in which base data is transmitted. Furthermore, geometric information does not have to be transmitted for frames in which extended data is transmitted. However, transmitting geometric information for each of all frames may result in the generation of a facial motion image with smooth movement.

[0181] Furthermore, control information related to the control of facial image data may be transmitted in a header area such as an SEI. For example, the control information may include presence information indicating whether or not facial image data is included in an access unit. Such presence information may be included in the header area of ​​the access unit. Note that an access unit is a data unit, and one access unit corresponds to one POC, i.e., one POC number.

[0182] The control information may also include type information indicating whether the access unit contains base data or extended data, and such type information may be included in the header area of ​​the access unit containing the facial image data.

[0183] The control information may also include identification information (e.g., count information) for identifying each extension data. Such identification information may be included in a header area of ​​an access unit including the extension data. The control information may also include total number information indicating the total number of one or more extension data. Such total number information may be included in a header area of ​​an access unit including the base data.

[0184] 12 is a conceptual diagram showing an example of the configuration of base data and augmented data. In this example, multiple elements included in a facial image are separated into base data and augmented data. That is, the base data includes one or more facial features (e.g., face shape, eyes, and mouth). The augmented data includes one or more other facial features (e.g., eyebrows, ears, and nose). The base data and augmented data are combined to reconstruct a facial image.

[0185] For example, the base data may be generated by using a segmentation mask to remove facial features (e.g., eyebrows, ears, and nose) from a facial image before encoding the base data using a video codec, and the augmented data may be generated by using the same segmentation mask to remove features from the facial image that were already encoded in the base data before encoding the augmented data.

[0186] Furthermore, intra prediction may be used for encoding the base data. Either intra prediction or inter prediction may be used for encoding the extended data. For example, the extended data may be data that encodes another facial feature, which is added to the facial features already encoded in the base data, as prediction error data in inter prediction.

[0187] In this example, by gradually transmitting multiple facial features, the occurrence of buffer underflow and delays can be suppressed, and facial expressions can be understood from partial facial features.

[0188] The facial features included in the base data may be parts that can significantly affect the impression of a facial expression, which may allow the facial expression to be understood earlier.

[0189] Also, in this example, frames of the facial motion image may be generated by the generative model only when all facial image data has been acquired. For example, frames that do not include faces, such as fixed frames, may be applied to the frames of the facial motion image until all facial image data has been acquired. This may prevent the display of faces that are missing features.

[0190] 13 is a conceptual diagram showing another example of the configuration of base data and extended data. In this example, the base data corresponds to a low-resolution image. The extended data corresponds to a high-resolution image, and more specifically, is prediction error data for reconstructing a high-resolution image from the low-resolution image. For example, by adding the extended data to an image obtained by using super-resolution technology to extend the resolution of an image corresponding to the base data, a high-resolution facial image can be reconstructed with high accuracy.

[0191] Specifically, for example, the base data may correspond to a facial image encoded at a low resolution (e.g., half or a quarter of the resolution of the full image). The extended data may correspond to a facial image encoded at full resolution with reference to the base data. Here, intra prediction may be used to encode the base data. Inter prediction may be used to encode the extended data.

[0192] More specifically, to encode an image of the extended data by inter prediction, an image of the base data may be used as a reference picture. In this case, a Reference Picture Resampling (RPR) technique may be used. In the RPR technique, motion compensation processing is performed using a scaling ratio. In this case, the scaling ratio may be determined by the difference between the resolution of the base data and the resolution of the extended data.

[0193] Furthermore, the plurality of pieces of extended data may correspond to a plurality of different resolutions, and may be coded in order from the lowest resolution to the highest resolution.

[0194] 14 is a conceptual diagram showing yet another example configuration of base data and extended data. The base data corresponds to an image composed of low-frequency components of a facial image. The extended data corresponds to an image composed of high-frequency components of a facial image. The base data corresponds to an image composed of frequency components in a first frequency range in the facial image, and the extended data corresponds to an image composed of frequency components in a second frequency range higher than the first frequency range in the facial image. The facial image is then reconstructed by adding the extended data to the base data.

[0195] For example, in encoding the base data, a low-frequency component image is obtained from a facial image through a low-pass filter (or a similar pre-processing technique), and then the low-frequency component image is compressed using a video codec and coded as the base data. In encoding the extended data, a high-frequency component image is obtained from the facial image through a high-pass filter, and then the high-frequency component image is compressed using a video codec and coded as the extended data.

[0196] The base data may correspond to an image of a face image that is made up of frequency components lower than a threshold, and the extended data may correspond to an image of a face image that is made up of frequency components not lower than a threshold.

[0197] Furthermore, the multiple extension data may correspond to multiple different frequency ranges, and may be coded in order from the lowest frequency range to the highest frequency range.

[0198] Furthermore, intra prediction may be used to encode the base data, and either intra prediction or inter prediction may be used to encode the extended data.

[0199] In this example, the face included in the facial moving image may gradually become clearer, which may reduce the sense of discomfort felt by the user viewing the facial moving image.

[0200] The extended data may be data corresponding to an image including both low-frequency components and high-frequency components. Specifically, the extended data may be data corresponding to an image including both low-frequency components and high-frequency components, and may be data corresponding to an image coded by inter prediction with reference to an image of the low-frequency components.

[0201] 15 is a conceptual diagram showing yet another example of the configuration of base data and extended data. The base data is a part of the face image and corresponds to an area included in the face image. The extended data is another part of the face image and corresponds to another area included in the face image. The face image is reconstructed by combining the base data and the extended data.

[0202] Also, if the face image is not facing forward, the base data may include only the captured portion of the face, and the augmented data may include other portions of the face captured in a different orientation than the base data.

[0203] In this example, a frame of the facial motion image may be generated by the neural network only when all the facial image data has been acquired. For example, a frame that does not include a face, such as a fixed frame, may be applied to the frame of the facial motion image until all the facial image data has been acquired. This may prevent the display of a face with a partially missing region.

[0204] The base data may also be coded using a Gradual Decoding Refresh (GDR) technique where only the left side of the image is intra-coded, and the extended data may also be coded using GDR where only the right side of the image is intra-coded, and after decoding the base data and extended data, a complete facial image may be reconstructed.

[0205] 16 is a conceptual diagram showing yet another example of the configuration of base data and extended data. In this example, the base data corresponds to data of a face image coded with a large QP (Quantization Parameter) value and intra prediction. The extended data corresponds to data of a face image coded with a small QP value and inter prediction.

[0206] Specifically, the QP value used for encoding the extended data is smaller than the QP value used for encoding the base data. This increases the accuracy of the face image encoded as the extended data. Meanwhile, the extended data is encoded using inter prediction with reference to the base data. This prevents the amount of code for the extended data from becoming too large.

[0207] Furthermore, the extended data is decoded by inter prediction with reference to the base data, thereby reconstructing a face image that reflects the base data and the extended data.

[0208] 12 to 16, after the augmented data is acquired, reconstructed data, which is facial image data reconstructed from the base data and the augmented data, may be input to the generative model. Alternatively, the base data and the augmented data may be input to the generative model. Alternatively, the base data and the reconstructed data may be input to the generative model. Alternatively, after multiple pieces of augmented data are acquired, the base data and multiple pieces of augmented data may be input to the generative model, or the base data and multiple pieces of reconstructed data may be input to the generative model.

[0209] For example, as shown in Figure 13, when the augmented data is prediction error data, reconstructed data corresponding to a reconstructed facial image may be obtained using the base data and the augmented data, and then the reconstructed data may be input to the generative model. At this time, the base data and the reconstructed data may be input to the generative model. Also, for example, as shown in Figure 12 or 15, when an image is represented by the base data and the augmented data, the base data and the augmented data may be input to the generative model.

[0210] 17 is a conceptual diagram showing another example of the structure of data corresponding to each frame. In this example, an SEI message is added to data corresponding to a frame of POC #0. This SEI message includes a parameter indicating the POC number that completes the facial image data.

[0211] For example, an SEI message containing this parameter may be transmitted in the first access unit in a bitstream, in the first access unit in a GOP, or in an access unit containing base data.

[0212] The SEI message may also include a parameter indicating whether incomplete facial image data can be used to reconstruct and display frames of facial video. Alternatively, the SEI message may include a parameter indicating frames that can be used even if the facial image data is incomplete. Then, only for frames indicated by the parameter, frames of facial video may be generated using a neural network from the facial image data added to the frames.

[0213] Furthermore, instead of the POC indicating the frame in which the facial image data is complete, an image identifier indicating the frame in which the facial image data is complete may be used.

[0214] Furthermore, since the SEI message has a syntax structure, the parameters in the SEI message can be expressed as syntax elements in the syntax structure of the SEI message. Furthermore, parameters indicating the same content as the content indicated by the above-described parameters may be included in a header field other than the SEI message.

[0215] 18 is a conceptual diagram showing yet another example of the structure of data corresponding to each frame. In this example, an SEI message is added to the data corresponding to each frame. That is, an SEI message is added to each access unit. The SEI message may include a parameter indicating whether or not facial image data is included in the access unit corresponding to the SEI message. Alternatively, the SEI message may be added only to access units that include facial image data.

[0216] When an access unit includes facial image data, the SEI message added to the access unit may include a parameter indicating whether the facial image data included in the access unit is base data or extended data. Alternatively, the target access unit may include a parameter indicating whether base data is included and a parameter indicating whether extended data is included.

[0217] If the facial image data included in the access unit is extended data, the SEI message attached to the access unit may include a parameter indicating the count (ordinal number) of the extended data. Also, if the facial image data is base data, the SEI message attached to the access unit may include a parameter indicating the total number of extended data associated with the base data.

[0218] Furthermore, if an access unit includes facial image data, the SEI message attached to the access unit may include a parameter indicating whether the facial image data of the access unit can be used to reconstruct and display corresponding frames of the facial video. Corresponding frames of the facial video may be generated from the facial image data of the access unit using a neural network only if the parameter indicates that the facial image data can be used to reconstruct and display corresponding frames of the facial video.

[0219] Additionally, an SEI message or other header area attached to an access unit may include one or more parameters indicating multiple identifiers of multiple access units each containing facial image data.

[0220] Fig. 19 is a conceptual diagram showing an example of control for generating a facial moving image. For example, the base data and the extended data correspond to different parts of a facial image, as in the examples of Fig. 12 and Fig. 15. In such a case, it may be better not to display an incomplete, partial image corresponding to the base data or the extended data than to display it.

[0221] Therefore, in POC#0, base data is acquired, but frames of facial motion images are generated without human faces, and in POC#1, the first extended data is acquired, but frames of facial motion images are generated without human faces.

[0222] In POC#2, the second augmented data is acquired. The face image can be almost completed by the base data and the two augmented data. Therefore, in POC#2, geometric information is further acquired, and frames of a facial motion image with a person's face are generated from the base data, the two augmented data, and the geometric information using a neural network.

[0223] In POC#3, the third augmented data is acquired. The face image can be completed by the base data and the three augmented data. Therefore, in POC#3, geometric information is further acquired, and frames of a facial motion image with a person's face are generated from the base data, the three augmented data, and the geometric information using a neural network.

[0224] Furthermore, in POC#3, frames of facial moving images are generated using more extended data than in POC#2, and therefore the completeness of the faces contained in frames generated in POC#3 is higher than the completeness of the faces contained in frames generated in POC#2.

[0225] For example, the control information transmitted in a header area such as an SEI may include application information indicating whether facial image data included in an access unit is applicable to the generation and display of a frame corresponding to the access unit. Also, the control information may include specification information, which is information included in the header area of ​​an access unit including base data, and is added to data corresponding to the frame and is information for specifying extension data applicable to the generation and display of the frame.

[0226] Another example of control of facial moving image generation is shown in Fig. 10. For example, the base data and the augmented data correspond to a plurality of different qualities of facial images, as in the examples of Fig. 13, Fig. 14, and Fig. 16. In such a case, it may be better to display an incomplete, low-quality image corresponding to the base data or the augmented data than not to display it at all.

[0227] Therefore, at POC#0, base data and geometric information are acquired, and frames of facial moving images are generated from the base data and geometric information using a neural network. At POC#1, first extended data and geometric information are acquired, and frames of facial moving images are generated from the base data, extended data, and geometric information using a neural network.

[0228] For subsequent frames, the neural network is used to generate frames of facial motion imagery in a similar manner. As the amount of augmented data used in a frame of facial motion imagery increases, the quality of the frame of facial motion imagery improves.

[0229] 20 is a block diagram showing another example configuration of the encoding device 100 according to this embodiment. The encoding device 100 generates a bitstream from a face image and a driving video. In this example, the encoding device 100 includes a preprocessor 141, a compressor 142, a compressor 143, a deriver 132, and a compressor 133. Each component is, for example, an electrical circuit that performs information processing. Two or more of the compressors 142, 143, and 133 may be integrated together.

[0230] The deriver 132 and the compressor 133 in Figure 20 correspond to the deriver 132 and the compressor 133 in Figure 3. The compressor 142 and the compressor 143 in Figure 20 correspond to the compressor 131 in Figure 3.

[0231] The preprocessor 141 acquires a facial image and separates the facial image into base data and one or more pieces of extended data. The compressor 142 acquires the base data and compresses it by bitstream encoding. The compressor 143 acquires one or more pieces of extended data and compresses it by bitstream encoding.

[0232] The deriver 132 obtains the driving video and derives geometric information from each frame of the driving video. The compressor 133 obtains the geometric information and compresses it by encoding it into a bitstream.

[0233] 21 is a block diagram showing another example configuration of the decoding device 200 according to this embodiment. The decoding device 200 generates a facial moving image from a bitstream. In this example, the decoding device 200 includes a decompressor 241, a decompressor 242, a decompressor 233, and a generator 234. Each component is, for example, an electric circuit that performs information processing. Two or more of the decompressor 241, the decompressor 242, and the decompressor 233 may be integrated together.

[0234] Decompressor 233 and generator 234 in Figure 21 correspond to decompressor 233 and generator 234 in Figure 4. Decompressor 241 and decompressor 242 in Figure 21 correspond to decompressor 231 in Figure 4.

[0235] The decompressor 241 decompresses the base data by decoding it from the bitstream. The decompressor 242 decompresses the one or more extension data by decoding it from the bitstream. The decompressor 233 decompresses the geometric information by decoding it from the bitstream. The generator 234 generates and outputs a facial motion image from the base data, the one or more extension data, and the geometric information using a generative model.

[0236] In generating a facial animation, the generator 234 may combine the base data and the augmented data and input the combined data together with geometric information into the generative model, thereby obtaining a facial animation output from the generative model.

[0237] Alternatively, in generating a facial video, the generator 234 may obtain a facial video output from the generative model by inputting each of the base data and the augmented data together with geometric information into the generative model without combining the base data and the augmented data. Alternatively, the augmented data may be prediction residual data for the base data, and the generator 234 may obtain a facial video output from the generative model by inputting decoded data decoded using inter prediction together with geometric information into the generative model.

[0238] An example use case of the present disclosure is face reconstruction for the video conferencing and entertainment industries. In the use case of the present disclosure, the bitstream is transmitted under narrow bandwidth and low latency conditions. The present disclosure helps to reduce the fluctuation in the transmission amount of the bitstream and the fluctuation in the data amount in the buffer under such conditions. In other words, the present disclosure enables a small buffer size and enables face reconstruction with low latency.

[0239] 22 is a flowchart showing an example of the operation of the encoding device 100 according to this embodiment. For example, the encoding device 100 encodes base data into a bitstream (S101). The base data includes information about a person's face, and is encoded into the first frame of the bitstream.

[0240] The encoding device 100 also encodes one or more pieces of extended data into a bitstream (S102), where the one or more pieces of extended data contain facial information of the same person and are encoded into one or more frames in the bitstream that are different from the initial frame.

[0241] Furthermore, the encoding device 100 may generate frames of facial video images from the base data and one or more extension data using a generative model (S103), which allows the encoding device 100 to verify the frames generated by the decoding device 200.

[0242] Furthermore, the encoding device 100 may further encode geometric information for each frame. Then, in generating frames of facial moving images, the encoding device 100 may generate frames of facial moving images from the base data, one or more pieces of extended data, and the geometric information using a generative model.

[0243] 23 is a flowchart showing an example of the operation of the decoding device 200 according to this embodiment. For example, the decoding device 200 decodes base data from a bitstream (S201). The base data includes information about a person's face, and is decoded from the first frame in the bitstream.

[0244] The decoding device 200 also decodes one or more pieces of extended data from the bitstream (S202). The one or more pieces of extended data contain face information of the same person and are decoded from one or more frames in the bitstream that are different from the initial frame.

[0245] Furthermore, the decoding device 200 generates a frame of a facial image from the base data and one or more pieces of extended data using a generative model (S203). The decoding device 200 may further decode geometric information and generate a frame of a facial moving image from the base data, one or more pieces of extended data, and the geometric information using a generative model.

[0246] For example, the decoding device 200 may reconstruct facial image data including facial information of the same person using the base data and one or more extension data, and then input the reconstructed facial image data to a generative model to obtain frames of facial video from the generative model.

[0247] Alternatively, the decoding device 200 may input the base data and one or more extension data to a generative model and obtain frames of facial video from the generative model.

[0248] In one example, the base data, the one or more augmented data, and the reconstructed facial image data may be a plurality of vectors representing facial features. In another example, the base data, the one or more augmented data, and the reconstructed facial image data may be a plurality of images. The generative model may be used to generate frames of facial video from the base data, the one or more augmented data, the reconstructed facial image data, or a combination thereof.

[0249] A neural network may be used as the generative model. An example of such a neural network is a generative network. Examples of generative networks include a generative adversarial network (GAN), a variational autoencoder (VAE), an autoregressive model, and a diffusion model.

[0250] A generative network is an example of a machine learning framework that generates new data based on a provided dataset. To ensure that multiple samples of the new data are similar to the original dataset, the generative model analyzes and learns from the underlying distribution of the dataset before generating new data.

[0251] 24 is a block diagram showing yet another example configuration of the encoding device 100 according to this embodiment. In this example, the encoding device 100 generates a bitstream including compressed facial images, compressed driving videos, and compressed background images from facial images, driving videos, and background images. The encoding device 100 includes a compressor 131, a deriver 132, a compressor 133, and a compressor 134. Each component is, for example, an electrical circuit that performs information processing. Two or more of the compressors 131, 133, and 134 may be integrated together.

[0252] Compressor 131, deriver 132, compressor 133 and compressor 134 in FIG. 24 correspond to compressor 131, deriver 132, compressor 133 and compressor 134 in FIG.

[0253] 25 is a flowchart showing another example of the operation of the encoding device 100 according to this embodiment. For example, the components of the encoding device 100 shown in FIG. 24 operate in accordance with the flowchart of FIG.

[0254] In this example, the compressor 131 first compresses a facial image by encoding it into a bitstream (S301). The facial image may be encoded according to a video codec such as VVC. The facial image may be a frame from a driving video, a pre-prepared image including a person's face, or an avatar.

[0255] The deriver 132 also derives geometric information indicating geometric attributes corresponding to each frame of the driving video from the driving video (S302). The geometric information indicating geometric attributes is also simply referred to as geometric attributes. Specifically, the deriver 132 inputs each frame of the driving video into a recognition model such as a neural network and obtains geometric information corresponding to each frame from the recognition model. The geometric information corresponds to a time instance of each frame of the driving video.

[0256] Here, the geometric attributes correspond to, for example, dynamic attributes and may be represented by a point cloud such as a facial landmark, or may be represented by a polygon model for representing the shape of an object by combining multiple polygons. Alternatively, the geometric attributes may be represented by other geometric models. Alternatively, the geometric attributes may be represented by the positions of facial features. Alternatively, the geometric attributes may be treated as a geometric attribute set.

[0257] For example, facial landmarks used as geometric attributes indicate the locations of points on key areas of the face, including the facial contours, eyes, eyebrows, nose, mouth, lips, and chin. Because such geometric attributes can be interpreted by other people or devices, the attributes can be modified and the processing of the attributes can be improved.

[0258] The compressor 133 compresses the geometric information by encoding it into a bitstream using a method such as entropy coding (S303).

[0259] The compressor 134 compresses at least one background image by encoding it into a bitstream (S304). The background image may be encoded according to a video codec such as VVC. The background image is used for a background region in the synthetic face video sequence. That is, the background image represents a background to be overlaid on the face-containing face video sequence.

[0260] The encoding device 100 may generate a composite facial moving image based on a facial image, geometric information, and a background image (S305), similar to the operation performed by the decoding device 200. The encoding device 100 may include multiple components similar to those of the decoding device 200 in order to generate a composite facial moving image. This makes it possible to check the composite facial moving image generated by the decoding device 200 on the encoding device 100. Note that this process may be omitted.

[0261] The encoding device 100 encodes the face image, the geometric information, and the background image into a bitstream, and then transmits the bitstream to the decoding device 200 via a transmission channel. For example, the compressed geometric information is transmitted as a bitstream from the encoding device 100 to the decoding device 200 for each frame of a driving video, i.e., for each time instance. The compressed geometric information may be transmitted as SEI (Supplemental Enhancement Information).

[0262] Note that the base data and one or more pieces of extended data that constitute a facial image may be treated as a facial image. For example, the compressor 131 may separate the facial image into the base data and one or more pieces of extended data, and encode the base data and the one or more pieces of extended data into a bitstream as a plurality of pieces of data that respectively correspond to a plurality of different frames.

[0263] Furthermore, the background image is not an essential element. Therefore, the encoding device 100 does not need to include the compressor 134. Furthermore, the compressed background image does not need to be included in the bitstream. Furthermore, input and output related to the background image may be omitted.

[0264] Alternatively, similar to the face image, the background image may be separated into background base data and one or more pieces of background extension data. The background base data and one or more pieces of background extension data constituting the background image may be treated as the background image. The compressor 134 may then separate the background image into the background base data and the one or more pieces of background extension data, and encode the background base data and the one or more pieces of background extension data into a bitstream as multiple pieces of data corresponding to different frames.

[0265] 26 is a block diagram showing yet another example configuration of the decoding device 200 according to this embodiment. In this example, the decoding device 200 generates a synthetic facial moving image from a bitstream. The decoding device 200 includes a decompressor 231, a deriving device 232, a decompressor 233, a generator 234, a decompressor 235, and a synthesizer 236. Each component is, for example, an electric circuit that performs information processing. Two or more of the decompressor 231, the decompressor 233, and the decompressor 235 may be integrated together.

[0266] 4. Decompressor 231, deriver 232, decompressor 233, generator 234 and decompressor 235 in FIG. 26 correspond to decompressor 231, deriver 232, decompressor 233, generator 234 and decompressor 235 in FIG.

[0267] Fig. 27 is a flowchart showing another example of the operation of the decoding device 200 according to this embodiment. For example, the components of the decoding device 200 shown in Fig. 26 operate according to the flowchart in Fig. 27. Note that, in the following, explanations that are the same as those for encoding may be omitted.

[0268] The decompressor 231 decompresses the facial image by decoding it from the bitstream (S401). The facial image may be decoded according to a video codec such as VVC. The decompressor 231 then provides the facial image to the deriver 232.

[0269] The deriver 232 derives face information indicating face attributes from the face image (S402). Here, the face information indicating face attributes may also be simply referred to as face attributes. The face attributes are static visual attributes and may also be referred to as identity. The face attributes may include information regarding at least one of hair, glasses, facial hair, eyebrows, eyes, mouth, nose, skin, facial contours, clothing, and accessories.

[0270] The decompressor 233 decompresses the geometric information by decoding it for each frame from the bitstream using a method such as entropy decoding (S403).

[0271] The generator 234 generates an intermediate facial video image from the facial information and geometric information using a generation model such as a neural network (S404).

[0272] The generative model may be a Generative Adversarial Network (GAN), a Variational Autoencoder (VAE), an autoregressive model, a diffusion model, etc. For example, the generative model may be a machine learning framework that generates new data based on a provided dataset, and may analyze and learn the underlying distribution of the dataset.

[0273] For example, the generator 234 inputs facial information and geometric information into a generative model for each frame, and obtains an intermediate facial video and a segmentation mask from the generative model. More specifically, the generator 234 inputs facial information and geometric information into the generative model for each frame, and obtains a frame of the intermediate facial video and a segmentation mask for that frame from the generative model. This segmentation mask indicates the foreground and background regions in the intermediate facial video (specifically, the frame of the intermediate facial video).

[0274] The segmentation mask may be represented by a two-dimensional map in which all pixel values ​​in the foreground region are 1 and all pixel values ​​in the background region are 0, or may be represented by a two-dimensional map in which all pixel values ​​in the foreground region are 0 and all pixel values ​​in the background region are 1. For example, the foreground region is a region that includes a face or the like and includes movement, and the background region is a region that does not include a face or the like and does not include movement. The segmentation mask is also called segmentation information.

[0275] The generator 234 may use the facial image itself instead of or in addition to the facial information to render the intermediate facial animation. The deriving unit 232 may or may not be included in the generator 234. The recognition model used to derive the facial information in the deriving unit 232 may be included in a generative model used to generate the intermediate facial animation, etc. in the generator 234. Other variations of the deriving unit 232 and the facial information are similar.

[0276] That is, the generator 234 may generate a segmentation mask and intermediate facial motion images from a facial image and geometric information using a generative model. In this case, the generator 234 may input the facial image and geometric information to the generative model and obtain the segmentation mask and intermediate facial motion images from the generative model.

[0277] The decompressor 235 decompresses by decoding at least one background image from the bitstream (S405). The background image may be decoded according to a video codec such as VVC. Instead of the background image, a selection parameter for selecting a background image from a plurality of background image candidates may be decoded. The selection parameter may be an identifier of a background image corresponding to one of the plurality of background image candidates.

[0278] The synthesizer 236 generates a composite facial video image by embedding corresponding regions in the background image into background regions in the intermediate facial video image using the intermediate facial video image, the segmentation mask, and the background image (S406).

[0279] Note that the base data and one or more pieces of extended data constituting a facial image may be treated as a facial image. For example, the decompressor 231 may decode the base data and one or more pieces of extended data coded as a plurality of pieces of data corresponding to different frames from the bitstream.

[0280] Furthermore, the background image is not an essential element. Therefore, the decoding device 200 does not need to include the decompressor 235 and the combiner 236. Furthermore, the compressed background image does not need to be included in the bitstream. Furthermore, input and output related to the background image may be omitted.

[0281] Alternatively, similar to a face image, the background base data and one or more pieces of background extension data constituting the background image may be treated as the background image, and the decompressor 235 may decode the background base data and one or more pieces of background extension data coded as a plurality of pieces of data corresponding to different frames from the bitstream.

[0282] Fig. 28 is a block diagram showing yet another example configuration of the decoding device 200 according to this embodiment. In the example described above, i.e., the example of Fig. 26, the generator 234 inputs face information and geometric information to a generative model, and acquires an intermediate face video and a segmentation mask from the generative model.

[0283] 28, the generator 234 inputs facial information and geometric information into a generative model, acquires intermediate facial video images from the generative model, and then performs segmentation processing on the intermediate facial video images to acquire a segmentation mask.

[0284] Specifically, the generator 234 inputs face information and geometric information into a generative model for each frame, acquires a frame of intermediate facial video from the generative model, and then performs segmentation processing on each frame of the intermediate facial video to acquire a segmentation mask for that frame.

[0285] This allows the processing to be subdivided, which may facilitate the processing. Instead of the generator 234, a segmentation processor (not shown) may perform the segmentation processing.

[0286] The segmentation process may be performed using a machine learning model such as a neural network, as well as other segmentation processes of the present disclosure.

[0287] The foreground and background regions in the intermediate facial video and the composite facial video generated by the decoding device 200 correspond to the foreground and background regions in the driving video. Therefore, the encoding device 100 may perform a segmentation process on the driving video and encode a segmentation mask of the driving video. The decoding device 200 may then decode the segmentation mask and generate a composite facial video using the segmentation mask.

[0288] Specifically, in the encoding device 100, the deriver 132 may perform a segmentation process on the driving video for each frame to generate a segmentation mask indicating foreground and background regions in the driving video, and the compressor 133 may compress the segmentation mask by encoding it into a bitstream.

[0289] Then, in the decoding device 200, the decompressor 233 may decompress the segmentation mask by decoding it from the bitstream. Furthermore, the synthesizer 236 may generate a composite facial moving image using the segmentation mask, etc. This may reduce the amount of processing in the decoding device 200.

[0290] In addition, in the encoding device 100, a segmentation processor (not shown) different from the deriver 132 may perform the segmentation process. In addition, a compressor (not shown) different from the compressor 133 may encode the segmentation mask into a bitstream. In the decoding device 200, a decompressor (not shown) different from the decompressor 233 may decode the segmentation mask from the bitstream.

[0291] Alternatively, the segmentation mask may be transmitted from the encoding device 100 to the decoding device 200 for each frame in the SEI.

[0292] Each pixel sample in the background region of the face image may be assigned a predetermined background color code for the background. This generates an intermediate face moving image in which each pixel sample in the background region is assigned a predetermined background color code. Therefore, the background region in the intermediate face moving image can be efficiently characterized without performing a segmentation process.

[0293] FIG. 29 is a block diagram showing yet another example configuration of the decoding device 200 according to this embodiment. In this example, the generator 234 generates a composite facial motion image using a generative model from facial information, geometric information, and a background image. Specifically, the generator 234 inputs the facial information, geometric information, and background image into the generative model, and obtains the composite facial motion image from the generative model, thereby generating the composite facial motion image. This can simplify processing. In this case, the decoding device 200 does not need to include a separate synthesizer 236.

[0294] [Examples of Generative Models] Fig. 30 is a diagram showing examples of various models that can be used as generative models. For example, neural networks can be used as generative models. Specifically, Fig. 30 shows a generative adversarial network, a variational autoencoder, a flow-based generative model, and a diffusion model.

[0295] In a generative adversarial network, new data instances similar to input data are generated through learning the features of the input data. Specifically, the unsupervised task in the generative model is converted into a supervised task by two types of submodels.

[0296] For example, the generator sub-model generates fake samples, the classifier sub-model discriminates between real inputs and fake samples generated by the generator sub-model, and an output image is generated through a minimax game that maximizes the classification probability of the classifier sub-model in assigning correct labels to the real inputs and fake samples while minimizing the distribution difference between the real inputs and fake samples.

[0297] In a variational autoencoder, the input data is first compressed into a multivariate latent distribution that reconstructs the data from the latent space as accurately as possible, resulting in efficient data compression and dimensionality reduction. In a flow-based generative model, the source distribution is transformed into the training data distribution through a sequence of one or more reversible transformations, which allows learning of the data distribution and accurate computation of the end-goal likelihood.

[0298] Diffusion models also generate new data instances similar to the training data. First, they degrade the structure of the training data through repeated injection of perturbations and noise before initiating a denoising process in an attempt to recover the original data. As a result, the data is iteratively mapped to a latent distribution via a Markov chain, where the latent state at each step depends only on the latent state at the previous step. The data is then recovered through hierarchical denoising.

[0299] For example, the neural network may be a face picture generation neural network that can be used to generate an output picture using geometric information represented in a fixed format for face parameters and a picture, i.e., the neural network corresponds to a process of generating a plurality of sample values ​​that constitute an output picture, which is a picture included in an output video sequence.

[0300] Alternatives to the neural networks described above may be any combination of the neural networks described above, other types of generative models, etc. may also be used.

[0301] Furthermore, a machine learning model such as a neural network may be used for the segmentation process, and may also be used to derive geometric information or facial information.

[0302] [Configuration Example for Encoding and Decoding Video] Figure 31 is a block diagram showing a configuration example for encoding a video by the encoding device 100 in this embodiment. For example, the encoding device 100 may include the multiple components shown in Figure 31 as multiple components for encoding images included in a video block by block in accordance with VVC. The encoding device 100 may include the multiple components shown in Figure 31 in addition to the multiple components described above, or at least some of the multiple components described above may be integrated into the multiple components shown in Figure 31.

[0303] 31 , the encoding device 100 includes a dividing unit 102, a subtraction unit 104, a transform unit 106, a quantization unit 108, an entropy encoding unit 110, an inverse quantization unit 112, an inverse transform unit 114, an addition unit 116, a block memory 118, a loop filter unit 120, a frame memory 122, an intra prediction unit 124, an inter prediction unit 126, a prediction control unit 128, and a prediction parameter generation unit 130. Note that the intra prediction unit 124 and the inter prediction unit 126 are each configured as part of a prediction processing unit.

[0304] The partitioning unit 102 partitions an image into multiple blocks and provides parameters related to the partitioning to the entropy coding unit 110. The subtraction unit 104 subtracts a predicted image block from a current block to obtain a predicted residual block. The transformation unit 106 performs transformation on the predicted residual block to obtain a transform coefficient block. The quantization unit 108 performs quantization on the transform coefficient block to obtain a quantized coefficient block. The entropy coding unit 110 performs entropy coding on the quantized coefficient block and the parameters to generate a bitstream.

[0305] The inverse quantization unit 112 performs inverse quantization on the quantized coefficient block to obtain a transform coefficient block. The inverse transform unit 114 performs inverse transform on the transform coefficient block to obtain a prediction residual block. The adder 116 adds the prediction image block to the prediction residual block to obtain a reconstructed image block. The block memory 118 stores the reconstructed image block. The loop filter unit 120 applies a loop filter to the reconstructed image block. The frame memory 122 stores the reconstructed image block to which the loop filter has been applied.

[0306] The intra prediction unit 124 performs intra prediction with reference to the block memory 118 to generate a predicted image block. The inter prediction unit 126 performs inter prediction with reference to the frame memory 122 to generate a predicted image block. The prediction control unit 128 provides the predicted image block generated by the intra prediction unit 124 or the predicted image block generated by the inter prediction unit 126 to the subtraction unit 104 and the addition unit 116. The prediction parameter generation unit 130 provides parameters related to intra prediction or inter prediction to the entropy coding unit 110.

[0307] Fig. 32 is a block diagram showing an example configuration for the decoding device 200 in this embodiment to decode a moving image. For example, the decoding device 200 may include the multiple components shown in Fig. 32 as multiple components for decoding images included in a moving image in units of blocks according to VVC. The decoding device 200 may include the multiple components shown in Fig. 32 in addition to the multiple components described above, or at least some of the multiple components described above may be integrated into the multiple components shown in Fig. 32.

[0308] 32 , the decoding device 200 includes an entropy decoding unit 202, an inverse quantization unit 204, an inverse transform unit 206, an adder 208, a block memory 210, a loop filter unit 212, a frame memory 214, an intra prediction unit 216, an inter prediction unit 218, a prediction control unit 220, a prediction parameter generation unit 222, and a partition determination unit 224. Note that the intra prediction unit 216 and the inter prediction unit 218 are each configured as part of a prediction processing unit.

[0309] The entropy decoding unit 202 performs entropy decoding on the bitstream to obtain a quantized coefficient block and parameters. The inverse quantization unit 204 performs inverse quantization on the quantized coefficient block to obtain a transform coefficient block. The inverse transform unit 206 performs inverse transform on the transform coefficient block to obtain a prediction residual block. The adder 208 adds the prediction image block to the prediction residual block to obtain a reconstructed image block. The loop filter unit 212 applies a loop filter to the reconstructed image block.

[0310] The block memory 210 stores reconstructed image blocks, and the frame memory 214 stores reconstructed image blocks to which the loop filter has been applied.

[0311] The intra prediction unit 216 performs intra prediction by referring to the block memory 210 to generate a predicted image block. The inter prediction unit 218 performs inter prediction by referring to the frame memory 214 to generate a predicted image block. The prediction control unit 220 provides the predicted image block generated by the intra prediction unit 216 or the predicted image block generated by the inter prediction unit 218 to the adder 208. The prediction parameter generation unit 222 provides parameters related to intra prediction or inter prediction to the prediction control unit 220, etc.

[0312] The division determination unit 224 determines blocks for decoding the image block by block in accordance with the division parameters.

[0313] [Combination] Multiple configuration examples in the present disclosure may be combined in any manner. Furthermore, multiple operation examples in the present disclosure may be combined in any manner. Furthermore, overlapping descriptions may be omitted in multiple examples in the present disclosure. Furthermore, configurations and processes corresponding to encoding configurations and processes may be applied to decoding, and configurations and processes corresponding to decoding configurations and processes may be applied to encoding. Furthermore, only a portion of the examples included in multiple examples in the present disclosure may be implemented.

[0314] 33 is a block diagram showing an implementation example of the encoding device 100. The encoding device 100 includes a circuit 151 and a memory 152. For example, several components of the encoding device 100 described above are implemented by the circuit 151 and the memory 152.

[0315] The circuit 151 is an electric circuit that performs information processing and can access the memory 152. For example, the circuit 151 may be a dedicated circuit that executes the encoding method of the present disclosure, or may be a general-purpose circuit that executes a program corresponding to the encoding method of the present disclosure. The circuit 151 may also be a processor such as a CPU. Furthermore, the circuit 151 may be a collection of multiple circuits.

[0316] The memory 152 is a dedicated or general-purpose memory that stores information used by the circuit 151 to encode an image. The memory 152 may be an electric circuit and may be connected to the circuit 151. The memory 152 may also be included in the circuit 151. The memory 152 may also be a collection of multiple circuits. The memory 152 may also be a magnetic disk, an optical disk, or the like, and may also be expressed as a storage, a recording medium, or the like. The memory 152 may also be a non-volatile memory or a volatile memory.

[0317] For example, the memory 152 may store data to be coded, such as an image, or may store coded data, such as a bit stream. The memory 152 may also store a program for causing the circuit 151 to perform image processing. The memory 152 may also store a generative model for the circuit 151.

[0318] 34 is a flowchart showing a first basic operation example of the encoding device 100. In the operation of this example, the circuit 151 of the encoding device 100 uses the memory 152 to perform the following.

[0319] Specifically, the circuit 151 encodes base data of a face image related to a face moving image and one or more extension data of the face image into a bitstream (S501).

[0320] Furthermore, the base data is added to data corresponding to a first frame in the bitstream. The one or more extension data are added to one or more data corresponding to one or more second frames in the bitstream. The first frame is a frame of the facial motion image. The one or more second frames are one or more frames of the facial motion image that are one or more frames after the first frame.

[0321] The circuit 151 also encodes the geometric information into a bit stream (S502). Here, the geometric information corresponds to each of a plurality of frames of the facial moving image and indicates geometric attributes within an area including a person's face.

[0322] This may make it possible to encode the base data and one or more extension data related to a face image separately into multiple frames, thereby reducing the amount of code corresponding to one frame and suppressing delays.

[0323] The geometric information may correspond to each of the plurality of third frames of the facial moving image. The first frame may be included in the plurality of third frames, or may not be included in the plurality of third frames. Similarly, one or more second frames may be included in the plurality of third frames, or may not be included in the plurality of third frames.

[0324] In addition, in the bitstream, data corresponding to a frame of a facial video image may include geometric information corresponding to the frame, and may include encoding parameters such as a decoding time and a display time corresponding to the frame. In the bitstream, data corresponding to a frame of a facial video image may be an access unit corresponding to the frame of the facial video image.

[0325] For example, the circuit 151 may encode control information relating to the control of at least one of the plurality of pieces of facial image data, which are the base data and one or more pieces of extended data, into the header area, which may enable appropriate processing to be applied to the facial image data in accordance with the control information when reproducing a facial moving image.

[0326] Furthermore, for example, the control information may include presence information indicating whether or not the access unit controlled by the header field includes facial image data, which is one of multiple facial image data. This may make it possible to identify whether or not the access unit includes facial image data in accordance with the control information when reproducing a facial moving image. Therefore, it may be possible to apply appropriate processing to the facial image data.

[0327] Furthermore, for example, when an access unit controlled by the header field includes facial image data that is one of a plurality of pieces of facial image data, the control information may include type information regarding whether the facial image data is base data or extended data. The extended data is one of one or more types of extended data.

[0328] Specifically, the type information may indicate that, when an access unit includes base data, the facial image data included in the access unit is base data and is used continuously until the next base data, or, when an access unit includes extended data, the type information may indicate that the facial image data included in the access unit is extended data and is used together with the base data.

[0329] This may enable, in reproducing a facial moving image, identification of whether the facial image data is base data or extended data according to the control information, and may enable application of appropriate processing to the facial image data depending on whether the facial image data is base data or extended data.

[0330] Furthermore, for example, the control information may include application information when an access unit controlled by the header field includes facial image data that is one of multiple sets of facial image data. Here, the application information indicates whether the facial image data is applicable to generating and displaying a frame corresponding to the access unit among multiple frames of a facial video. This may make it possible to appropriately control whether or not to apply the facial image data added to the data corresponding to a frame of the facial video in accordance with the control information when reproducing the facial video.

[0331] Furthermore, for example, each of the base data and one or more pieces of extended data may be expressed as a vector indicating facial features included in the facial image. This may reduce the amount of code related to the facial image, thereby suppressing delays.

[0332] Furthermore, for example, each of the base data and one or more pieces of extended data may be expressed as an image related to a facial image, which may make it possible to appropriately reflect each of the base data and extended data related to a facial image as image data in a frame of the facial moving image when reproducing the facial moving image.

[0333] Furthermore, for example, the circuit 151 may derive and encode data of a portion of the face included in the facial image as base data. Alternatively, the circuit 151 may derive and encode data of other portions of the face included in the facial image as extended data, which is one of one or more sets of extended data. This may make it possible to divide the facial image into multiple portions and encode them without complex processing.

[0334] The part of the face may be a part of a plurality of facial features or a part of a plurality of facial regions. The other part of the face may be a part of a plurality of facial features or a part of a plurality of facial regions. The base data may be data of a first part. The one or more pieces of extended data may be one or more pieces of data of one or more second parts different from the first part. When multiple pieces of extended data are used, multiple second parts different from each other may be used.

[0335] For example, the circuit 151 may derive and encode data in a first frequency range in the facial image as base data, and may derive and encode data in a second frequency range in the facial image that is higher than the first frequency range as extended data that is one of one or more extended data.

[0336] This may make it possible to encode the low-frequency component data in the facial image as base data and encode the high-frequency component data in the facial image as extended data. Furthermore, in reproducing the facial motion image, it may be possible to apply the low-frequency component data to generating a first frame of the facial motion image and apply both the low-frequency component data and the high-frequency component data to generating a second frame of the facial motion image. Therefore, it may be possible to suppress delays and to suppress unnaturalness in the facial motion image.

[0337] The data in the first frequency range may include data of the lowest frequency component, i.e., direct current (DC) component. The one or more extension data may be one or more data in one or more second frequency ranges higher than the first frequency range. When multiple extension data are used, multiple different second frequency ranges may be used.

[0338] Furthermore, for example, the circuit 151 may encode a first image as base data. Furthermore, for example, the circuit 151 may encode a second image as base data. Here, the first image is an image related to a facial image and has a first resolution. The second image is an image related to a facial image, is encoded with reference to the first image, and has a second resolution higher than the first resolution.

[0339] This may make it possible to encode a low-resolution facial image as base data and encode a high-resolution facial image as extended data. Furthermore, in reproducing a facial motion image, it may be possible to apply the low-resolution facial image to generating a first frame of the facial motion image and the high-resolution facial image to generating a second frame of the facial motion image. Therefore, it may be possible to suppress delays and to suppress unnaturalness in the facial motion image.

[0340] The one or more pieces of extended data may correspond to one or more second images having one or more second resolutions higher than the first resolution. When multiple pieces of extended data are used, multiple second resolutions different from each other may be used. The multiple pieces of extended data may correspond to multiple second images. In addition, when encoding a second image, another second image may be referenced instead of the first image.

[0341] Furthermore, the one or more second frames may be after the first frame in display order, or may be after the first frame in encoding order or decoding order.

[0342] The base data may include first information that is information about a face image, and each of the one or more pieces of extended data may include second information that is information about a face image and different from the first information.

[0343] Furthermore, the encoding device 100 may include an input terminal, an entropy encoder, and an output terminal. The operations performed by the circuit 151 may be performed by the entropy encoder. Data used in the operation of the entropy encoder may be input to the input terminal. Data obtained by the operation of the entropy encoder may be output from the output terminal.

[0344] 35 is a block diagram showing an implementation example of the decoding device 200. The decoding device 200 includes a circuit 251 and a memory 252. For example, several components of the decoding device 200 described above are implemented by the circuit 251 and the memory 252.

[0345] The circuit 251 is an electric circuit that performs information processing and can access the memory 252. For example, the circuit 251 may be a dedicated circuit that executes the decoding method of the present disclosure, or may be a general-purpose circuit that executes a program corresponding to the decoding method of the present disclosure. The circuit 251 may also be a processor such as a CPU. Furthermore, the circuit 251 may be a collection of multiple circuits.

[0346] The memory 252 is a dedicated or general-purpose memory that stores information for the circuit 251 to decode images. The memory 252 may be an electric circuit and may be connected to the circuit 251. The memory 252 may also be included in the circuit 251. The memory 252 may also be a collection of multiple circuits. The memory 252 may also be a magnetic disk, an optical disk, or the like, and may also be expressed as a storage, a recording medium, or the like. The memory 252 may also be a non-volatile memory or a volatile memory.

[0347] For example, the memory 252 may store data to be decoded, such as a bitstream, or may store decoded data, such as an image. The memory 252 may also store a program for causing the circuit 251 to perform image processing. The memory 252 may also store a generative model for the circuit 251.

[0348] 36 is a flowchart showing a first basic operation example of the decoding device 200. In the operation of this example, the circuit 251 of the decoding device 200 uses the memory 252 to perform the following.

[0349] Specifically, the circuit 251 decodes base data of a face image related to a face moving image and one or more pieces of extended data of the face image from the bitstream (S601).

[0350] Furthermore, the base data is added to data corresponding to a first frame in the bitstream. The one or more extension data are added to one or more data corresponding to one or more second frames in the bitstream. The first frame is a frame of a facial video sequence. The one or more second frames are one or more frames of the facial video sequence, and are one or more frames after the first frame.

[0351] The circuit 251 also decodes geometric information from the bitstream (S602). Here, the geometric information corresponds to each of multiple frames of the facial video and indicates geometric attributes within an area including the person's face. The circuit 251 then generates a facial video using a generative model from the base data, one or more extension data, and the geometric information (S603).

[0352] This may make it possible to decode the base data and one or more extension data related to the face image by dividing them into multiple frames, thereby reducing the amount of code corresponding to one frame, and thus reducing delays.

[0353] The geometric information may correspond to each of the plurality of third frames of the facial moving image. The first frame may be included in the plurality of third frames, or may not be included in the plurality of third frames. Similarly, one or more second frames may be included in the plurality of third frames, or may not be included in the plurality of third frames.

[0354] In addition, in the bitstream, data corresponding to a frame of a facial video image may include geometric information corresponding to the frame, and may include encoding parameters such as a decoding time and a display time corresponding to the frame. In the bitstream, data corresponding to a frame of a facial video image may be an access unit corresponding to the frame of the facial video image.

[0355] For example, the circuit 251 may decode control information relating to the control of at least one of the plurality of pieces of facial image data, which are the base data and one or more pieces of extended data, from the header area, which may enable appropriate processing to be applied to the facial image data in accordance with the control information when reproducing the facial moving image.

[0356] Furthermore, for example, the control information may include presence information indicating whether or not an access unit controlled by the header field includes facial image data, which is one of multiple facial image data. This may make it possible to identify whether or not an access unit includes facial image data in accordance with the control information when reproducing a facial moving image. Therefore, it may be possible to apply appropriate processing to the facial image data.

[0357] Furthermore, for example, when an access unit controlled by the header field includes facial image data that is one of a plurality of pieces of facial image data, the control information may include type information regarding whether the facial image data is base data or extended data. The extended data is one of one or more types of extended data.

[0358] Specifically, the type information may indicate that, when an access unit includes base data, the facial image data included in the access unit is base data and is used continuously until the next base data, or, when an access unit includes extended data, the type information may indicate that the facial image data included in the access unit is extended data and is used together with the base data.

[0359] This may enable, in reproducing a facial moving image, identification of whether the facial image data is base data or extended data according to the control information, and may enable application of appropriate processing to the facial image data depending on whether the facial image data is base data or extended data.

[0360] Furthermore, for example, the control information may include application information when an access unit controlled by the header field includes facial image data that is one of multiple sets of facial image data. Here, the application information indicates whether the facial image data is applicable to generating and displaying a frame corresponding to the access unit among multiple frames of a facial video. This may make it possible to appropriately control whether or not to apply the facial image data added to the data corresponding to a frame of the facial video in accordance with the control information when reproducing the facial video.

[0361] Furthermore, for example, each of the base data and one or more pieces of extended data may be expressed as a vector indicating facial features included in the facial image. This may reduce the amount of code related to the facial image, thereby suppressing delays.

[0362] Furthermore, for example, each of the base data and one or more pieces of extended data may be expressed as an image related to a facial image, which may make it possible to appropriately reflect each of the base data and extended data related to a facial image as image data in a frame of the facial moving image when reproducing the facial moving image.

[0363] Furthermore, for example, the circuit 251 may input the base data, at least one of one or more extended data, and geometric information into a generative model to generate a frame of a facial motion image. This may make it possible to omit the process of generating an intermediate image from the base data and extended data related to the facial image. Therefore, it may be possible to simplify the process of generating a facial motion image. Note that a frame corresponding to the geometric information may be generated as a frame of the facial motion image.

[0364] Also, for example, circuitry 251 may generate intermediate images from the base data and at least one of the one or more augmented data, and circuitry 251 may input the intermediate images and geometric information into a generative model to generate frames of facial motion images.

[0365] This may allow an intermediate image relating to the facial image to be appropriately generated from the base data and the extended data relating to the facial image. Furthermore, it may allow the intermediate image relating to the facial image to be appropriately reflected in a frame of the facial moving image. Note that a frame corresponding to the geometric information may be generated as a frame of the facial moving image.

[0366] Also, for example, the circuit 251 may decode the augmented data, which is one of the one or more augmented data, by referring to the base data, and may input the augmented data and the geometric information into a generative model to generate a frame of facial motion imagery.

[0367] This may enable efficient decoding of extended data having higher accuracy than the accuracy of the base data. Furthermore, it may be possible to generate frames of facial motion images with high accuracy using the extended data with high accuracy. Note that frames corresponding to the geometric information may be generated as frames of facial motion images.

[0368] Furthermore, for example, the base data may be data of a part of a face included in the facial image. Furthermore, one of the one or more pieces of extended data may be data of another part of the face included in the facial image. This may make it possible to decode the facial image by dividing it into multiple parts without complex processing.

[0369] The part of the face may be a part of a plurality of facial features or a part of a plurality of facial regions. The other part of the face may be a part of a plurality of facial features or a part of a plurality of facial regions. The base data may be data of a first part. The one or more pieces of extended data may be one or more pieces of data of one or more second parts different from the first part. When multiple pieces of extended data are used, multiple second parts different from each other may be used.

[0370] For example, the base data may be data in a first frequency range in the facial image, and one of the one or more extended data may be data in a second frequency range in the facial image that is higher than the first frequency range.

[0371] This may make it possible to decode low-frequency component data in the facial image as base data and high-frequency component data in the facial image as extended data. Furthermore, in reproducing a facial motion image, it may be possible to apply the low-frequency component data to generating a first frame of the facial motion image and apply both the low-frequency component data and the high-frequency component data to generating a second frame of the facial motion image. Therefore, it may be possible to suppress delays and to suppress unnaturalness in the facial motion image.

[0372] The data in the first frequency range may include data of the lowest frequency component, i.e., direct current (DC) component. The one or more extension data may be one or more data in one or more second frequency ranges higher than the first frequency range. When multiple extension data are used, multiple different second frequency ranges may be used.

[0373] Furthermore, for example, the base data may correspond to a first image. One of the one or more pieces of extended data may correspond to a second image. Here, the first image is an image related to a facial image and has a first resolution. The second image is an image related to a facial image, is an image decoded with reference to the first image, and has a second resolution higher than the first resolution.

[0374] This may enable decoding a low-resolution facial image as base data and decoding a high-resolution facial image as extended data. Furthermore, in reproducing a facial motion image, the low-resolution facial image may be applied to generating a first frame of the facial motion image, and the high-resolution facial image may be applied to generating a second frame of the facial motion image. Therefore, it may be possible to suppress delays and to suppress unnaturalness in the facial motion image.

[0375] The one or more pieces of extended data may correspond to one or more second images having one or more second resolutions higher than the first resolution. When multiple pieces of extended data are used, multiple second resolutions different from each other may be used. Furthermore, the multiple pieces of extended data may correspond to multiple second images. Furthermore, when decoding a second image, another second image may be referenced instead of the first image.

[0376] Furthermore, for example, the base data may correspond to a first image. One of the one or more pieces of extended data may correspond to a second image. Here, the first image is an image related to a facial image and is decoded with a first quantization width. The second image is an image related to a facial image and is decoded with a second quantization width that is finer than the first quantization width by referring to the first image.

[0377] This may make it possible to decode the coarse facial image as base data and decode the fine facial image as extended data. Furthermore, in reproducing a facial motion image, it may be possible to apply the coarse facial image to generating a first frame of the facial motion image and apply the fine facial image to generating a second frame of the facial motion image. Therefore, it may be possible to suppress delays and to suppress unnaturalness in the facial motion image.

[0378] The fine quantization width may be a small quantization width. Furthermore, the one or more pieces of extended data may correspond to one or more second images decoded with one or more second quantization widths that are finer than the first quantization width. When multiple pieces of extended data are used, multiple second quantization widths different from each other may be used. Furthermore, the multiple pieces of extended data may correspond to multiple second images. Furthermore, in decoding a second image, another second image may be referenced instead of the first image.

[0379] Furthermore, for example, the control information may include identification information for identifying each of one or more pieces of extended data. This may make it possible to identify each piece of extended data according to the control information when reproducing a facial moving image. Therefore, it may be possible to individually specify each piece of extended data and control the application of each piece of extended data.

[0380] Furthermore, for example, the control information may include (i) information included in a header region of an access unit including base data, and (ii) total number information indicating the total number of one or more pieces of extended data. This may make it possible to identify the total number of one or more pieces of extended data in accordance with the control information when reproducing a facial video. Therefore, it may be possible to efficiently determine one or more pieces of extended data that can be used for reproducing a facial video according to the total number of one or more pieces of extended data.

[0381] Furthermore, for example, the control information may include (i) specific information that is information included in a header region of an access unit that includes the base data, and (ii) information for identifying extension data of one or more extension data that is applicable to the generation and display of one or more second frames that correspond to the access unit that includes the extension data.

[0382] This may enable appropriate identification of extension data applicable to generating and displaying frames of facial motion images in accordance with the control information when reproducing facial motion images.

[0383] Furthermore, for example, the circuit 251 may decode at least one control parameter for controlling a stream buffer in which the bitstream is stored in the memory 252. The at least one control parameter may be at least one parameter for controlling the buffer size of the stream buffer to a reference size or less and for controlling the initial delay time for starting the decoding process to a reference delay time or less. This makes it possible to reduce resources for decoding and shorten the delay time.

[0384] Furthermore, for example, the one or more second frames may be after the first frame in display order, or may be after the first frame in encoding order or decoding order.

[0385] The base data may include first information that is information about a face image, and each of the one or more pieces of extended data may include second information that is information about a face image and different from the first information.

[0386] Furthermore, for example, the decoding device 200 may include an input terminal, an entropy decoder, and an output terminal. The operations performed by the circuit 251 may be performed by the entropy decoder. Data used in the operation of the entropy decoder may be input to the input terminal. Data obtained by the operation of the entropy decoder may be output from the output terminal.

[0387] Also, for example, a computer-readable non-transitory recording medium for storing a bitstream may be used. The bitstream may include base data of a facial image related to a facial motion image, one or more extension data of the facial image, and geometric information.

[0388] Furthermore, the base data is attached to data corresponding to a first frame in the bitstream. The one or more extension data are attached to one or more data corresponding to one or more second frames in the bitstream. The first frame is a frame of the facial video. The one or more second frames are one or more frames of the facial video, and are one or more frames after the first frame. Furthermore, the geometric information is information corresponding to each of the multiple frames of the facial video, and is information indicating geometric attributes within an area including a person's face.

[0389] The bitstream may then cause the decoding device 200 to (i) decode the base data, one or more extension data, and geometric information, and (ii) perform a process of generating a facial moving image from the base data, one or more extension data, and geometric information using a generative model.

[0390] This may enable the realization of a recording medium that stores one or more bitstreams corresponding to the above-described decoding device and decoding method, and therefore may enable the recording medium to achieve the same effects as the above-described decoding device 200, etc.

[0391] [Other Examples] The encoding device 100 and the decoding device 200 in each of the above-described examples may be used as an image encoding device and an image decoding device, or as a video encoding device and a video decoding device, respectively. Furthermore, multiple components included in the encoding device 100 and multiple components included in the decoding device 200 may perform corresponding operations.

[0392] Additionally, the term "encode" may be substituted with terms such as "store," "include," "write," "write," "signal," "send," "notify," or "preserve," and these terms may be interchangeable. For example, encoding information may mean including the information in a bitstream. Also, encoding information into a bitstream may mean encoding the information to generate a bitstream that includes the encoded information.

[0393] Furthermore, the term "decode" may be replaced with terms such as "read," "decode," "read," "load," "derive," "obtain," "receive," "extract," or "restore," and these terms may be interchangeable. For example, decoding information may mean obtaining information from a bitstream. Decoding information from a bitstream may mean decoding the bitstream to obtain information contained in the bitstream.

[0394] Furthermore, for example, the coding information and compression information contained in the bitstream may be simply referred to as information.

[0395] Furthermore, at least some of the examples described above may be used as an encoding method, a decoding method, an entropy encoding method, an entropy decoding method, or some other method.

[0396] Each component may be configured with dedicated hardware, or may be realized by executing a software program suitable for that component. Each component may be realized by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.

[0397] Specifically, each of the encoding device 100 and the decoding device 200 may include a processing circuit and a storage device electrically connected to and accessible from the processing circuit. For example, the processing circuit corresponds to the circuit 151 or 251, and the storage device corresponds to the memory 152 or 252.

[0398] The processing circuit includes at least one of dedicated hardware and a program execution unit, and executes processing using a storage device. If the processing circuit includes a program execution unit, the storage device stores a software program to be executed by the program execution unit.

[0399] An example of the above-mentioned software program is a bitstream. The bitstream includes an encoded image and a syntax for performing a decoding process to decode the image. The bitstream causes the decoding device 200 to decode the image by executing a process based on the syntax. Furthermore, for example, software for realizing the above-mentioned encoding device 100 or decoding device 200 is a program such as the following.

[0400] For example, this program may cause a computer to execute an encoding method in which base data of a facial image related to a facial video and one or more extension data of the facial image are encoded into a bit stream, geometric information corresponding to each of multiple frames of the facial video and indicating geometric attributes within an area including a person's face is encoded into the bit stream, the base data is added to data corresponding to a first frame, which is a frame of the facial video, and the one or more extension data are added to one or more data corresponding to one or more second frames, which are one or more frames of the facial video and are one or more frames after the first frame, in the bit stream.

[0401] Furthermore, for example, this program may cause a computer to execute a decoding method in which base data of a facial image related to a facial video and one or more extension data of the facial image are decoded from a bit stream, geometric information corresponding to each of multiple frames of the facial video and indicating geometric attributes within an area including a person's face is decoded from the bit stream, and the facial video is generated from the base data, the one or more extension data and the geometric information using a generative model, wherein the base data is attached to data corresponding to a first frame, which is a frame of the facial video, in the bit stream, and the one or more extension data are attached to one or more data corresponding to one or more second frames, which are one or more frames of the facial video and are one or more frames after the first frame, in the bit stream.

[0402] Furthermore, each of the above-described components may be a circuit. These circuits may form a single circuit as a whole, or each may be a separate circuit. Furthermore, each component may be realized by a general-purpose processor or a dedicated processor.

[0403] Furthermore, a process performed by a specific component may be performed by another component. The order in which the processes are performed may be changed, or multiple processes may be performed in parallel. Any two or more of the multiple examples of the present disclosure may be appropriately combined and implemented. The encoding / decoding device may include the encoding device 100 and the decoding device 200.

[0404] Furthermore, not all of the components in the present disclosure may be implemented, and only some of the components in the present disclosure may be implemented. Similarly, not all of the processes in the present disclosure may be executed, and only some of the processes in the present disclosure may be executed.

[0405] Furthermore, ordinal numbers such as "first" and "second" used in the description may be changed as appropriate. Furthermore, new ordinal numbers may be assigned to components or removed. Furthermore, these ordinal numbers may be assigned to elements in order to identify them, and may not correspond to a meaningful order.

[0406] Also, for example, a phrase "at least one of" a first element, a second element, and a third element corresponds to the first element, the second element, the third element, or any combination thereof.

[0407] Although the aspects of the encoding device 100 and the decoding device 200 have been described above based on a number of examples, the aspects of the encoding device 100 and the decoding device 200 are not limited to these examples. As long as they do not deviate from the spirit of the present disclosure, various modifications conceivable by those skilled in the art to each example, or configurations constructed by combining components of different examples, may also be included within the scope of the aspects of the encoding device 100 and the decoding device 200.

[0408] One or more aspects disclosed herein may be implemented in combination with at least a part of other aspects of the present disclosure. Also, some processes shown in the flowcharts of one or more aspects disclosed herein, some configurations of devices, some syntax, etc. may be implemented in combination with other aspects.

[0409] [Implementation and Application] In each of the above embodiments, each of the functional or operational blocks can typically be realized by an MPU (micro processing unit), memory, etc. Furthermore, the processing by each of the functional blocks may be realized as a program execution unit such as a processor that reads and executes software (programs) recorded on a recording medium such as a ROM. The software may be distributed. The software may be recorded on various recording media such as semiconductor memory. It is also possible to realize each functional block by hardware (dedicated circuitry).

[0410] The processing described in each embodiment may be realized by centralized processing using a single device (system), or may be realized by distributed processing using multiple devices. Furthermore, the processor that executes the program may be a single processor or multiple processors. That is, centralized processing or distributed processing may be performed.

[0411] The aspects of the present disclosure are not limited to the above examples, and various modifications are possible, and these modifications are also included within the scope of the aspects of the present disclosure.

[0412] Furthermore, application examples of the video coding method (image coding method) or video decoding method (image decoding method) shown in each of the above embodiments and various systems for implementing the application examples will be described below. Such systems may be characterized by having an image coding device using the image coding method, an image decoding device using the image decoding method, or an image coding / decoding device that includes both. Other configurations of such systems can be appropriately changed depending on the situation.

[0413] [Example of Use] Fig. 37 shows the overall configuration of an appropriate content supply system ex100 that realizes a content distribution service. The area where communication services are provided is divided into cells of a desired size, and base stations ex106, ex107, ex108, ex109, and ex110, which are fixed wireless stations in the illustrated example, are installed in each cell.

[0414] In this content supply system ex100, devices such as a computer ex111, a game console ex112, a camera ex113, a home appliance ex114, and a smartphone ex115 are connected to the Internet ex101 via an Internet service provider ex102 or a communication network ex104 and base stations ex106 to ex110. The content supply system ex100 may connect a combination of any of the above devices. In various implementations, the devices may be connected to each other directly or indirectly via a telephone network, short-range wireless communication, or the like, without going through the base stations ex106 to ex110. Furthermore, the streaming server ex103 may be connected to devices such as the computer ex111, the game console ex112, the camera ex113, the home appliance ex114, and the smartphone ex115 via the Internet ex101, etc. The streaming server ex103 may also be connected to a terminal in a hotspot on an airplane ex117 via a satellite ex116.

[0415] Note that wireless access points, hot spots, etc. may be used instead of the base stations ex106 to ex110. Furthermore, the streaming server ex103 may be directly connected to the communication network ex104 without going through the Internet ex101 or the Internet service provider ex102, or may be directly connected to the airplane ex117 without going through the satellite ex116.

[0416] The camera ex113 is a device such as a digital camera that can take still images and videos. The smartphone ex115 is a smartphone, mobile phone, or PHS (Personal Handyphone System) that supports mobile communication systems such as 2G, 3G, 3.9G, 4G, and the upcoming 5G.

[0417] The home appliance ex114 is a refrigerator, or an appliance included in a home fuel cell cogeneration system, or the like.

[0418] In the content supply system ex100, a terminal having a photographing function is connected to a streaming server ex103 via a base station ex106 or the like, thereby enabling live streaming and the like. In live streaming, a terminal (such as a computer ex111, a game console ex112, a camera ex113, a home appliance ex114, a smartphone ex115, or a terminal on an airplane ex117) may perform the encoding process described in each of the above embodiments on still image or video content captured by a user using the terminal, may multiplex the video data obtained by encoding with audio data obtained by encoding audio corresponding to the video, and may transmit the obtained data to the streaming server ex103. In other words, each terminal functions as an image encoding device according to one aspect of the present disclosure.

[0419] Meanwhile, the streaming server ex103 streams the transmitted content data to the requesting client. The client is a computer ex111, a game console ex112, a camera ex113, a home appliance ex114, a smartphone ex115, a terminal on an airplane ex117, or the like, which is capable of decoding the encoded data. Each device that receives the distributed data decodes and plays back the received data. That is, each device may function as an image decoding device according to one aspect of the present disclosure.

[0420] [Distributed Processing] The streaming server ex103 may also be multiple servers or multiple computers that process, record, and distribute data in a distributed manner. For example, the streaming server ex103 may be implemented using a CDN (Content Delivery Network), where content distribution is achieved through a network connecting numerous edge servers distributed around the world. In a CDN, a physically nearby edge server is dynamically assigned depending on the client. Content is then cached and distributed to that edge server, thereby reducing delays. Furthermore, when certain types of errors occur or communication conditions change due to increased traffic, processing can be distributed among multiple edge servers, the distribution entity can be switched to another edge server, or distribution can be continued by bypassing the failed portion of the network, thereby achieving high-speed and stable distribution.

[0421] In addition to the distributed processing of the distribution itself, the encoding of captured data may be performed by each terminal, by the server, or by multiple terminals. For example, encoding generally involves two processing loops. The first loop detects the complexity of the image or the amount of code for each frame or scene. The second loop maintains image quality while improving encoding efficiency. For example, a terminal may perform the first encoding process, and the server that receives the content may perform the second encoding process, thereby improving content quality and efficiency while reducing the processing load on each terminal. In this case, if there is a request to receive and decode the data in near real time, the data encoded by a terminal can be received and played back by another terminal, enabling more flexible real-time distribution.

[0422] As another example, the camera ex113 or the like extracts features from an image, compresses the data related to the features as metadata, and transmits the compressed data to the server. The server performs compression according to the meaning (or importance of the content) of the image, for example, by determining the importance of an object from the features and switching the quantization precision accordingly. The feature data is particularly effective in improving the accuracy and efficiency of motion vector prediction when the server re-compresses the image. Alternatively, the terminal may perform simple encoding such as VLC (variable length coding), and the server may perform encoding with a high processing load such as CABAC (context-adaptive binary arithmetic coding).

[0423] As another example, in a stadium, shopping mall, factory, or the like, there may be multiple pieces of video data that have been shot by multiple terminals of almost the same scene. In this case, using the multiple terminals that shot the video and, as necessary, other terminals and servers that did not shoot the video, encoding processes are assigned to each of them, for example, in units of GOPs (Group of Pictures), pictures, or tiles obtained by dividing a picture, for distributed processing. This reduces delays and achieves better real-time performance.

[0424] Since multiple video data are of almost the same scene, the server may manage and / or instruct the video data shot by each terminal to be mutually referential. The server may also receive encoded data from each terminal and change the reference relationships between multiple data, or correct or replace the pictures themselves and re-encode them. This allows for the generation of streams with improved quality and efficiency for each piece of data.

[0425] Furthermore, the server may perform transcoding to change the encoding method of the video data before distributing it. For example, the server may convert an MPEG-based encoding method into a VP-based encoding method (e.g., VP9), or convert H.264 to H.265.

[0426] In this way, the encoding process can be performed by a terminal or one or more servers. Therefore, although the following uses terms such as "server" or "terminal" to refer to the entity performing the process, some or all of the processing performed by the server may be performed by the terminal, and some or all of the processing performed by the terminal may be performed by the server. The same applies to the decoding process.

[0427] [3D, Multi-Angle] Images or videos of different scenes or the same scene taken from different angles using multiple devices such as cameras ex113 and / or smartphones ex115 that are approximately synchronized with each other are increasingly being integrated and used. The videos taken by each device are integrated based on the relative positional relationship between the devices obtained separately, or on areas where feature points in the videos match.

[0428] The server may not only encode two-dimensional video, but also encode still images automatically or at a time specified by the user based on scene analysis of the video and transmit them to the receiving terminal. Furthermore, if the server can acquire the relative positional relationship between the capturing terminals, it can generate a three-dimensional shape of the scene based on not only two-dimensional video but also video of the same scene captured from different angles. The server may separately encode three-dimensional data generated by a point cloud or the like, or may select or reconstruct the video to be transmitted to the receiving terminal from video captured by multiple terminals based on the results of recognizing or tracking people or objects using the three-dimensional data.

[0429] In this way, the user can enjoy a scene by arbitrarily selecting each video corresponding to each shooting terminal, or can enjoy content in which a video from a selected viewpoint is cut out from 3D data reconstructed using multiple images or videos. Furthermore, together with the video, sound may also be collected from multiple different angles, and the server may multiplex the sound from a specific angle or space with the corresponding video and transmit the multiplexed video and sound.

[0430] In recent years, content that associates the real world with a virtual world, such as Virtual Reality (VR) and Augmented Reality (AR), has also become popular. In the case of VR images, the server creates viewpoint images for the right eye and left eye, respectively, and may perform encoding that allows reference between each viewpoint video using Multi-View Coding (MVC) or the like, or may encode them as separate streams without referencing each other. When decoding the separate streams, it is preferable to play them in synchronization with each other so that a virtual three-dimensional space is reproduced according to the user's viewpoint.

[0431] In the case of AR images, the server superimposes virtual object information in the virtual space onto camera information in the real space based on the three-dimensional position or the movement of the user's viewpoint. The decoding device may acquire or store virtual object information and three-dimensional data, generate a two-dimensional image according to the movement of the user's viewpoint, and smoothly connect the two-dimensional image to create superimposed data. Alternatively, the decoding device may send the movement of the user's viewpoint to the server in addition to a request for virtual object information. The server may create superimposed data according to the movement of the viewpoint received from the three-dimensional data stored on the server, encode the superimposed data, and distribute it to the decoding device. Note that the superimposed data may have an α value indicating transparency in addition to RGB, and the server may set the α value of parts other than the object created from the three-dimensional data to 0, etc., to encode the parts in a transparent state. Alternatively, the server may generate data by setting a predetermined RGB value as the background, like a chromakey, and using the background color for parts other than the object.

[0432] Similarly, the decoding of distributed data may be performed by each client terminal, by the server, or by multiple terminals. For example, one terminal may first send a reception request to the server, and then other terminals may receive and decode content according to the request, after which the decoded signal is transmitted to a device having a display. By distributing the processing and selecting appropriate content regardless of the capabilities of the communication terminals themselves, high-quality data can be reproduced. As another example, large-sized image data may be received on a TV or other device, and only a portion of the picture, such as a tile into which the picture is divided, may be decoded and displayed on the viewer's personal device. This allows the viewer to share the overall picture while checking their own area of ​​responsibility or an area of ​​interest in more detail.

[0433] In situations where multiple short-, medium-, or long-range wireless communications are available indoors and outdoors, seamless content reception may be possible using distribution system standards such as MPEG-DASH (Dynamic Adaptive Streaming over HTTP). Users may freely select and switch in real time between decoding devices or display devices, such as their own terminals and indoor / outdoor displays. Furthermore, decoding can be performed while switching between decoding and display devices using their own location information, etc. This allows information to be mapped and displayed on a part of the wall or ground of a neighboring building with an embedded display device while the user is traveling to their destination. It is also possible to switch the bit rate of received data based on the accessibility of the encoded data on the network, such as when the encoded data is cached on a server that can be quickly accessed from the receiving terminal or copied to an edge server in a content delivery service.

[0434] [Web Page Optimization] FIG. 38 is a diagram showing an example of a display screen of a web page on a computer ex111 or the like. FIG. 39 is a diagram showing an example of a display screen of a web page on a smartphone ex115 or the like. As shown in FIGS. 38 and 39 , a web page may include multiple link images that are links to image content, and the appearance of the link images may differ depending on the device used to view the page. When multiple link images are visible on the screen, the display device (decoding device) may display a still image or I-picture contained in each content as a link image until the user explicitly selects the link image, or until the link image approaches the center of the screen or the entire link image is within the screen. Alternatively, the display device (decoding device) may display a video such as a GIF animation using multiple still images or I-pictures, or may receive only the base layer and decode and display the video.

[0435] When a link image is selected by a user, the display device performs decoding while giving top priority to the base layer. Note that if the HTML (HyperText Markup Language) constituting the web page contains information indicating that the content is scalable, the display device may decode up to the enhancement layer. Furthermore, to ensure real-time performance, before selection or when the communication bandwidth is very limited, the display device decodes and displays only forward-reference pictures (I pictures, P pictures, and forward-reference-only B pictures), thereby reducing the delay between the decoding time of the first picture and the display time (the delay from the start of content decoding to the start of display). Furthermore, the display device may intentionally ignore the picture reference relationships and roughly decode all B pictures and P pictures using forward reference, and then perform normal decoding as the number of received pictures increases over time.

[0436] [Autonomous Driving] When transmitting and receiving still image or video data such as two-dimensional or three-dimensional map information for automatic driving or driving assistance of a vehicle, the receiving terminal may receive weather or construction information as meta information in addition to image data belonging to one or more layers, and may associate and decode these. Note that the meta information may belong to a layer, or may simply be multiplexed with the image data.

[0437] In this case, since a vehicle, drone, airplane, or the like including the receiving terminal is moving, the receiving terminal can transmit location information of the receiving terminal, thereby realizing seamless reception and decoding while switching between base stations ex106 to ex110. Furthermore, the receiving terminal can dynamically switch how much meta information to receive or how much to update map information depending on the user's selection, the user's situation, and / or the state of the communication bandwidth.

[0438] In the content supply system ex100, the client can receive, decode, and play back encoded information sent by a user in real time.

[0439] [Distribution of Personal Content] The content supply system ex100 also allows for unicast or multicast distribution of not only high-quality, long-duration content from video distribution companies, but also low-quality, short-duration content from individuals. It is expected that such personal content will continue to increase in the future. To improve the quality of personal content, the server may perform editing before encoding. This can be achieved, for example, using the following configuration.

[0440] During shooting, either in real time or after accumulating and shooting, the server performs recognition processing such as detecting shooting errors, scene search, semantic analysis, and object detection from the original image data or encoded data. Based on the recognition results, the server manually or automatically corrects out-of-focus or camera shake, deletes less important scenes such as scenes with lower brightness or out-of-focus compared to other pictures, emphasizes object edges, changes color, and performs other editing. The server then encodes the edited data based on the editing results. It is also known that viewing rates decrease if the shooting time is too long. Therefore, the server may automatically clip not only less important scenes as described above but also scenes with little movement, based on the image processing results, so that the content falls within a specific time range depending on the shooting time. Alternatively, the server may generate and encode a digest based on the results of the semantic analysis of the scenes.

[0441] Personal content may contain content that, if left as is, violates copyright, moral rights, or portrait rights, and may cause the scope of sharing to exceed the intended scope, resulting in inconvenience to individuals. Therefore, for example, the server may intentionally defocus images of people's faces on the periphery of the screen or the interior of a house before encoding. Furthermore, the server may recognize whether the image to be encoded contains the face of a person other than a pre-registered person, and if so, perform processing such as blurring the face. Alternatively, as pre- or post-processing before encoding, the user may specify a person or background area they wish to modify in the image for copyright or other reasons. The server may replace the specified area with another image or blur the focus. If the image contains a person, the server may track the person in the video and replace the image of the person's face.

[0442] Because viewing personal content with small data volumes requires high real-time performance, the decoding device first receives the base layer as a top priority, and then decodes and plays it back, depending on the bandwidth. The decoding device may also receive an enhancement layer during this time, and if the content is played back more than once, such as when playback is looped, it may play back high-quality video including the enhancement layer. A stream that has undergone scalable encoding in this way can provide an experience in which the video appears rough when not selected or when viewing begins, but gradually becomes smarter and the image quality improves. In addition to scalable encoding, a similar experience can also be provided when a rough stream played the first time and a second stream that is encoded with reference to the first video are configured as a single stream.

[0443] [Other Application Examples] Furthermore, these encoding or decoding processes are generally performed by the LSIex500 possessed by each terminal. The LSI (large scale integration circuitry) ex500 (see FIG. 37) may be a single-chip or multi-chip configuration. Furthermore, video encoding or decoding software may be embedded in some kind of recording medium (such as a CD-ROM, flexible disk, or hard disk) readable by the computer ex111, and the encoding or decoding process may be performed using that software. Furthermore, if the smartphone ex115 is equipped with a camera, video data captured by the camera may be transmitted. This video data is data encoded and processed by the LSIex500 possessed by the smartphone ex115.

[0444] The LSIex500 may be configured to download and activate application software. In this case, the terminal first determines whether it supports the content encoding method or has the capability to execute a specific service. If the terminal does not support the content encoding method or does not have the capability to execute a specific service, the terminal downloads a codec or application software, and then acquires and plays the content.

[0445] Furthermore, at least one of the moving image encoding device (image encoding device) or moving image decoding device (image decoding device) of each of the above embodiments can be incorporated into a digital broadcasting system, not limited to the content supply system ex100 via the Internet ex101. Since multiplexed data in which video and audio are multiplexed is transmitted and received over broadcast radio waves using a satellite or the like, there is a difference in that it is more suited to multicast than the content supply system ex100, which has a configuration that is easy to use for unicast, but similar applications are possible with regard to encoding and decoding processes.

[0446] [Hardware Configuration] Fig. 40 is a diagram showing further details of the smartphone ex115 shown in Fig. 37. Fig. 41 is a diagram showing an example configuration of the smartphone ex115. The smartphone ex115 includes an antenna ex450 for transmitting and receiving radio waves to and from the base station ex110, a camera unit ex465 capable of capturing video and still images, and a display unit ex458 for displaying video captured by the camera unit ex465 and decoded data of the video and the like received by the antenna ex450. The smartphone ex115 further includes an operation unit ex466 such as a touch panel, an audio output unit ex457 such as a speaker for outputting voice or sound, an audio input unit ex456 such as a microphone for inputting voice, a memory unit ex467 capable of storing captured video or still images, recorded voice, received video or still images, encoded data such as email, or decoded data, and a slot unit ex464 that is an interface with a SIM (Subscriber Identity Module) ex468 for identifying a user and authenticating access to various data including the network. Note that an external memory may be used instead of the memory unit ex467.

[0447] A main control unit ex460 that comprehensively controls the display unit ex458 and operation unit ex466, etc., is connected to a power supply circuit unit ex461, an operation input control unit ex462, a video signal processing unit ex455, a camera interface unit ex463, a display control unit ex459, a modulation / demodulation unit ex452, a multiplexing / separation unit ex453, an audio signal processing unit ex454, a slot unit ex464, and a memory unit ex467 via a synchronization bus ex470.

[0448] When the power key is turned on by a user's operation, the power supply circuit unit ex461 starts up the smartphone ex115 to an operable state and supplies power to each unit from the battery pack.

[0449] The smartphone ex115 performs processes such as telephone calls and data communications under the control of a main control unit ex460 having a CPU, ROM, RAM, etc. During a call, an audio signal collected by an audio input unit ex456 is converted into a digital audio signal by an audio signal processing unit ex454, subjected to spectrum spread processing by a modulation / demodulation unit ex452, subjected to digital-to-analog conversion and frequency conversion processing by a transmission / reception unit ex451, and the resulting signal is transmitted via an antenna ex450. The received data is also amplified and subjected to frequency conversion and analog-to-digital conversion processing, subjected to spectrum despreading processing by a modulation / demodulation unit ex452, and converted into an analog audio signal by an audio signal processing unit ex454, which is then output from an audio output unit ex457. During data communication mode, text, still images, or video data is sent to the main control unit ex460 via an operation input control unit ex462 based on operations on the main unit's operation unit ex466, etc. Similar transmission and reception processing is performed. When transmitting video, still images, or video and audio in the data communication mode, the video signal processing unit ex455 compresses and encodes the video signal stored in the memory unit ex467 or the video signal input from the camera unit ex465 using the moving image encoding method described in each of the above embodiments, and sends the encoded video data to the multiplexing / demultiplexing unit ex453. The audio signal processing unit ex454 encodes the audio signal collected by the audio input unit ex456 while the camera unit ex465 is capturing video or still images, and sends the encoded audio data to the multiplexing / demultiplexing unit ex453. The multiplexing / demultiplexing unit ex453 multiplexes the encoded video data and encoded audio data using a predetermined method, and modulates and converts the multiplexed video data and audio data in the modulation / demodulation unit (modulation / demodulation circuit unit) ex452 and the transmission / reception unit ex451, before transmitting the multiplexed video data and audio data via the antenna ex450.

[0450] In order to decode the multiplexed data received via the antenna ex450, such as when receiving video attached to an email or chat, or video linked to a web page, the multiplexing / separation unit ex453 separates the multiplexed data into a video data bit stream and an audio data bit stream, and supplies the encoded video data to the video signal processing unit ex455 and the encoded audio data to the audio signal processing unit ex454 via the synchronization bus ex470. The video signal processing unit ex455 decodes the video signal using a video decoding method corresponding to the video encoding method described in each of the above embodiments, and the video or still image contained in the linked video file is displayed on the display unit ex458 via the display control unit ex459. The audio signal processing unit ex454 decodes the audio signal, and audio is output from the audio output unit ex457. As real-time streaming becomes increasingly common, audio playback may be socially inappropriate depending on the user's situation. Therefore, it is preferable that the initial setting be a configuration in which only the video data is played without playing the audio signal, and audio may be played in sync only when the user performs an operation such as clicking on the video data.

[0451] Although the smartphone ex115 has been used as an example, three other implementation formats are possible: a transmitting / receiving terminal having both an encoder and a decoder, a transmitting terminal having only an encoder, and a receiving terminal having only a decoder. In the digital broadcasting system, multiplexed data in which audio data is multiplexed with video data is received or transmitted. However, in addition to audio data, text data related to the video may also be multiplexed into the multiplexed data. Furthermore, the video data itself may be received or transmitted instead of the multiplexed data.

[0452] Although the main control unit ex460 including a CPU has been described as controlling the encoding or decoding process, various terminals often include a GPU (Graphics Processing Unit). Therefore, a configuration may be adopted in which a memory shared by the CPU and GPU, or a memory whose addresses are managed for common use, is used to take advantage of the GPU's performance to process a large area in a batch. This shortens the encoding time, ensures real-time performance, and achieves low latency. It is particularly efficient to perform motion estimation, deblocking filtering, SAO (Sample Adaptive Offset), and transformation / quantization processes in a batch, such as by picture, by the GPU rather than by the CPU.

[0453] The present disclosure is applicable to encoding devices that encode moving images, and is applicable to video conferencing systems, etc.

[0454] 100 Encoding device 102 Division unit 104 Subtraction unit 106 Transform unit 108 Quantization unit 110 Entropy coding unit 112, 204 Inverse quantization unit 114, 206 Inverse transformation unit 116, 208 Addition unit 118, 210 Block memory 120, 212 Loop filter unit 122, 214 Frame memory 124, 216 Intra prediction unit 126, 218 Inter prediction unit 128, 220 Prediction control unit 130, 222 Prediction parameter generation unit 131, 133, 134, 142, 143 Compressor 132, 232, 237 Derivation unit 141 Preprocessor 151, 251 Circuit 152, 252 Memory 200 Decoding device 202 Entropy decoding unit 224 Division determination unit 231, 233, 235, 241, 242 Decompressor 234 Generator 236 Combiner

Claims

1. Memory and a circuit coupled to the memory; The circuit uses the memory to: decoding base data of a facial image relating to a facial motion image and one or more extension data of the facial image from the bitstream; Decoding geometric information from the bitstream, the geometric information corresponding to each of a plurality of frames of the face moving image and indicating geometric attributes within an area including a person's face; generating the facial motion image from the base data, the one or more pieces of extended data, and the geometric information using a generative model; the base data is added to data corresponding to a first frame, which is a frame of the facial moving image, in the bit stream; the one or more pieces of extended data are added to one or more pieces of data corresponding to one or more second frames, which are one or more frames of the facial moving image and are one or more frames subsequent to the first frame, in the bitstream; Decryption device.

2. The circuit decodes control information relating to control of at least one of the plurality of facial image data, which are the base data and the one or more extension data, from a header area. The decoding device according to claim 1 .

3. the control information includes presence information indicating whether or not face image data that is one of the plurality of face image data is included in the access unit controlled by the header area; The decoding device according to claim 2 .

4. the control information includes, when an access unit controlled by the header area includes facial image data that is one of the plurality of facial image data, type information regarding whether the facial image data is the base data or extended data that is one of the one or more extended data; the type information indicates that, when the access unit includes the base data, the facial image data included in the access unit is the base data and is continuously used until the next base data; the type information indicates that, when the access unit includes the extended data, the facial image data included in the access unit is the extended data and is used together with the base data; 4. The decoding device according to claim 2 or 3.

5. the control information includes application information indicating whether or not, when an access unit controlled by the header area includes facial image data that is one of the plurality of facial image data, the facial image data is applicable to generation and display of a frame corresponding to the access unit among the plurality of frames of the facial moving image; 4. The decoding device according to claim 2 or 3.

6. each of the base data and the one or more pieces of extended data is expressed by a vector indicating facial features included in the face image; The decoding device according to any one of claims 1 to 3.

7. each of the base data and the one or more pieces of extended data is expressed as an image related to the facial image; The decoding device according to any one of claims 1 to 3.

8. the circuit inputs the base data, at least one of the one or more augmented data, and the geometric information into the generative model to generate frames of the facial motion image; The decoding device according to any one of claims 1 to 3.

9. the circuit generates an intermediate image from the base data and at least one of the one or more augmented data, and inputs the intermediate image and the geometric information into the generative model to generate a frame of the facial motion image; The decoding device according to any one of claims 1 to 3.

10. the circuitry refers to the base data, decodes augmented data that is one of the one or more augmented data, and inputs the augmented data and the geometric information into the generative model to generate a frame of the facial motion image; The decoding device according to any one of claims 1 to 3.

11. the base data is data of a part of a face included in the face image, The one or more pieces of extended data are data of other parts of the face included in the face image. The decoding device according to any one of claims 1 to 3.

12. the base data is data of the face image in a first frequency range, the one or more pieces of extended data are data in a second frequency range in the face image that is higher than the first frequency range; The decoding device according to any one of claims 1 to 3.

13. the base data is an image related to the face image and corresponds to a first image which is an image having a first resolution; The one or more pieces of extended data are an image related to the face image, are decoded by referring to the first image, and correspond to a second image having a second resolution higher than the first resolution. The decoding device according to any one of claims 1 to 3.

14. the base data corresponds to a first image, which is an image related to the face image and is decoded using a first quantization width; the extended data being one of the one or more extended data is an image related to the face image, and corresponds to a second image being an image decoded with a second quantization width finer than the first quantization width by referring to the first image; The decoding device according to any one of claims 1 to 3.

15. the control information includes identification information for identifying each of the one or more extension data.

4. The decoding device according to claim 2 or 3.

16. The control information (i) is information included in the header area of ​​the access unit including the base data, and (ii) includes total number information that is information indicating a total number of the one or more extension data.

4. The decoding device according to claim 2 or 3.

17. The control information (i) is information included in the header area of ​​the access unit including the base data, and (ii) includes specific information that is extension data among the one or more extension data and is information for identifying extension data applicable to generation and display of a second frame among the one or more second frames corresponding to the access unit including the extension data.

4. The decoding device according to claim 2 or 3.

18. the circuit decodes at least one control parameter for controlling a stream buffer in the memory in which the bitstream is stored, the control parameter being for controlling a buffer size of the stream buffer to be equal to or less than a reference size and for controlling an initial delay time for starting a decoding process to be equal to or less than a reference delay time; The decoding device according to any one of claims 1 to 3.

19. Memory and a circuit coupled to the memory; The circuit uses the memory to: encoding base data of a facial image relating to a facial motion image and one or more extension data of the facial image into a bitstream; encoding geometric information, which corresponds to each of a plurality of frames of the face moving image and indicates geometric attributes within an area including a person's face, into the bitstream; the base data is added to data corresponding to a first frame, which is a frame of the facial moving image, in the bit stream; the one or more pieces of extended data are added to one or more pieces of data corresponding to one or more second frames, which are one or more frames of the facial moving image and are one or more frames subsequent to the first frame, in the bitstream; Encoding device.

20. The circuit encodes control information relating to control of at least one of the plurality of facial image data, which are the base data and the one or more extension data, into a header area.

20. The encoding device of claim 19.

21. the control information includes presence information indicating whether or not face image data that is one of the plurality of face image data is included in the access unit controlled by the header area; 21. The encoding device according to claim 20.

22. the control information includes, when an access unit controlled by the header area includes facial image data that is one of the plurality of facial image data, type information regarding whether the facial image data is the base data or extended data that is one of the one or more extended data; the type information indicates that, when the access unit includes the base data, the facial image data included in the access unit is the base data and is continuously used until the next base data; the type information indicates that, when the access unit includes the extended data, the facial image data included in the access unit is the extended data and is used together with the base data; 22. Encoding device according to claim 20 or 21.

23. the control information includes application information indicating whether or not, when an access unit controlled by the header area includes facial image data that is one of the plurality of facial image data, the facial image data is applicable to generation and display of a frame corresponding to the access unit among the plurality of frames of the facial moving image; 22. Encoding device according to claim 20 or 21.

24. each of the base data and the one or more pieces of extended data is expressed by a vector indicating facial features included in the face image; The encoding device according to any one of claims 19 to 21.

25. each of the base data and the one or more pieces of extended data is expressed as an image related to the facial image; The encoding device according to any one of claims 19 to 21.

26. the circuit derives and encodes data of a portion of the face included in the face image as the base data, and derives and encodes data of another portion of the face included in the face image as extended data that is one of the one or more extended data; The encoding device according to any one of claims 19 to 21.

27. the circuit derives and encodes data in a first frequency range in the facial image as the base data, and derives and encodes data in a second frequency range in the facial image that is higher than the first frequency range as extended data that is one of the one or more extended data; The encoding device according to any one of claims 19 to 21.

28. The circuit encodes a first image, which is an image related to the face image and has a first resolution, as the base data, and encodes a second image, which is an image related to the face image and is encoded with reference to the first image, and has a second resolution higher than the first resolution, as extended data, which is one of the one or more extended data. The encoding device according to any one of claims 19 to 21.

29. decoding base data of a facial image relating to a facial motion image and one or more extension data of the facial image from the bitstream; Decoding geometric information from the bitstream, the geometric information corresponding to each of a plurality of frames of the face moving image and indicating geometric attributes within an area including a person's face; generating the facial motion image from the base data, the one or more pieces of extended data, and the geometric information using a generative model; the base data is added to data corresponding to a first frame, which is a frame of the facial moving image, in the bit stream; the one or more pieces of extended data are added to one or more pieces of data corresponding to one or more second frames, which are one or more frames of the facial moving image and are one or more frames subsequent to the first frame, in the bitstream; Decryption method.

30. encoding base data of a facial image relating to a facial motion image and one or more extension data of the facial image into a bitstream; encoding geometric information, which corresponds to each of a plurality of frames of the face moving image and indicates geometric attributes within an area including a person's face, into the bitstream; the base data is added to data corresponding to a first frame, which is a frame of the facial moving image, in the bit stream; the one or more pieces of extended data are added to one or more pieces of data corresponding to one or more second frames, which are one or more frames of the facial moving image and are one or more frames subsequent to the first frame, in the bitstream; Encoding method.