Decoding device, encoding device, decoding method, and encoding method

By using neural networks in video encoding devices to generate human-corresponding images, the problems of low encoding efficiency, poor image quality, and resource waste in existing technologies are solved, achieving more efficient information transmission and resource utilization.

CN118947124APending Publication Date: 2024-11-12PANASONIC INTELLECTUAL PROPERTY CORP OF AMERICA
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202380029850.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-05-17
Filing Date
2023-03-31
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

Existing video coding technologies struggle to effectively improve coding efficiency, image quality, reduce processing load, and minimize circuit size when processing video data. Furthermore, improper selection of elements or actions such as filters, blocks, sizes, motion vectors, and reference images can lead to resource waste and communication difficulties.

Method used

By employing decoding and encoding devices and utilizing neural networks to generate human-corresponding images, and by decoding and encoding performance data, images corresponding to humans are generated, reflecting information about human performance, thereby improving information transmission efficiency and reducing resource waste.

Benefits of technology

Human-to-human images generated through neural networks can effectively improve coding efficiency, image quality, reduce processing load and circuit size, enhance information transmission efficiency, facilitate smooth communication, and reduce time and resource waste.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118947124A_ABST
    Figure CN118947124A_ABST
Patent Text Reader

Abstract

A decoding device (200) is provided with a circuit (251) and a memory (252) connected to the circuit (251), the circuit (251), during operation, decodes performance data indicating information represented by a person, generates a person-corresponding image, which is an image corresponding to the person, via a neural network in accordance with the performance data and at least one profile image of the person, and outputs the person-corresponding image.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to a decoding device, an encoding device, a decoding method, and an encoding method. Background Art

[0002] Video coding technology has progressed from H.261 and MPEG-1 to H.264 / AVC (Advanced Video Coding), MPEG-LA, H.265 / HEVC (High Efficiency Video Coding), and H.266 / VVC (Versatile Video Codec). With this progress, there is always a need to provide improvements and optimizations in video coding technology in order to handle the ever-increasing amount of digital video data in various applications. The present disclosure relates to further progress, improvements, and optimizations in video coding.

[0003] Furthermore, Non-Patent Document 1 relates to an example of existing standards related to the above-mentioned video encoding technology.

[0004] Prior Art Literature

[0005] Non-patent literature

[0006] Non-patent document 1: H.265 (ISO / IEC 23008-2 HEVC) / HEVC (High Efficiency Video Coding) Summary of the invention

[0007] Problems to be solved by the invention

[0008] Regarding the encoding method described above, it is expected that new methods will be proposed in order to improve encoding efficiency, improve image quality, reduce processing volume, reduce circuit scale, or appropriately select elements or actions such as filters, blocks, sizes, motion vectors, reference images or reference blocks.

[0009] The present disclosure provides, for example, a configuration or method that can contribute to one or more of improving coding efficiency, improving image quality, reducing processing volume, reducing circuit size, improving processing speed, and appropriately selecting elements or actions, etc. In addition, the present disclosure may include a configuration or method that can contribute to benefits other than the above.

[0010] Means for solving problems

[0011] For example, a decoding device involved in one embodiment of the present invention includes a circuit and a memory connected to the circuit, and the circuit, when in operation, decodes performance data representing information performed by a person, generates a person-corresponding image as an image corresponding to the person through a neural network based on the performance data and at least one profile image of the person, and outputs the person-corresponding image.

[0012] Each embodiment or a part of the structure or method in the present disclosure can improve coding efficiency, improve image quality, reduce the amount of processing for coding / decoding, reduce circuit scale, or improve the processing speed of coding / decoding, for example, at least one of the following aspects. Alternatively, each embodiment or a part of the structure or method in the present disclosure can appropriately select components / actions such as filters, blocks, sizes, motion vectors, reference pictures, reference blocks, etc. in coding and decoding. In addition, the present disclosure also includes the disclosure of structures or methods that can provide benefits other than those mentioned above. For example, structures or methods that suppress the increase in processing volume and improve coding efficiency, etc.

[0013] Further advantages and effects in one embodiment of the present disclosure are clear from the description and drawings. The advantages and / or effects can be obtained by several embodiments and the features described in the description and drawings, but in order to obtain one or more advantages and / or effects, it is not necessary to provide all of them.

[0014] In addition, these overall or specific methods can also be implemented by systems, integrated circuits, computer programs or computer-readable recording media such as CD-ROMs, and can also be implemented by any combination of systems, methods, integrated circuits, computer programs and recording media.

[0015] Effects of the Invention

[0016] The structure or method involved in one embodiment of the present disclosure can, for example, contribute to improving coding efficiency, improving image quality, reducing processing volume, reducing circuit scale, improving processing speed, and appropriately selecting elements or actions. In addition, the structure or method involved in one embodiment of the present disclosure can also contribute to benefits other than the above. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is a conceptual diagram showing a video conferencing system in the first embodiment.

[0018] Figure 2 This is a conceptual diagram showing an example in which a human image is replaced with a human profile image.

[0019] Figure 3 This is a block diagram showing a configuration example of a video conferencing system in the second embodiment.

[0020] Figure 4 This is a block diagram showing an example configuration of an encoding device in the second embodiment.

[0021] Figure 5 This is a conceptual diagram showing an example of determining an index for an input image via an expression recognition neural network.

[0022] Figure 6 This is a block diagram showing a configuration example of a decoding device in the second embodiment.

[0023] Figure 7 This is a block diagram showing another configuration example of the decoding device in the second embodiment.

[0024] Figure 8 This is a flowchart showing an example of the operation of the encoding device in the second embodiment.

[0025] Fig. 9 This is a flowchart showing an example of the operation of the decoding device in the second embodiment.

[0026] Fig.10 This is a conceptual diagram showing an example of the position of an index in a bit stream.

[0027] Fig.11 This is a conceptual diagram showing another example of the position of an index in a bit stream.

[0028] Fig.12 This is a conceptual diagram showing an example of a plurality of profile images.

[0029] Fig.13 This is a conceptual diagram showing another example of a plurality of profile images.

[0030] Fig.14 This is a conceptual diagram showing still another example of a plurality of profile images.

[0031] Fig.15 This is a conceptual diagram showing an example of data expressed by coordinates.

[0032] Fig.16 This is a conceptual diagram showing another example of data expressed by coordinates.

[0033] Fig.17 This is a conceptual diagram showing an example of changes in the orientation of a face.

[0034] Fig.18 This is a conceptual diagram showing an example of changes in lips.

[0035] Fig.19 This is a conceptual diagram showing an example of the movement of the head corresponding to the position of the head.

[0036] Fig. 20 This is a conceptual diagram showing an example of standardization.

[0037] Fig.21 This is a conceptual diagram showing another example of standardization.

[0038] Fig. 22 This is a conceptual diagram showing yet another example of standardization.

[0039] Fig.23 This is a conceptual diagram showing an example of generating an output image through a neural network based on coded data and a profile image.

[0040] Fig.24 This is a conceptual diagram showing an example in which an index is associated with each of a plurality of profile images.

[0041] Fig.25 This is a conceptual diagram showing the selection of a profile image and the generation of an output image.

[0042] Fig.26 is a conceptual diagram showing an example of a neural network for generating an output image.

[0043] Fig. 27 is a conceptual diagram showing another example of a neural network for generating an output image.

[0044] Fig.28 This is a conceptual diagram showing yet another example of a neural network for generating an output image.

[0045] Fig.29 This is a conceptual diagram showing a video conferencing system in the third embodiment.

[0046] Fig.30 This is a block diagram showing a configuration example of a video conferencing system in the third embodiment.

[0047] Fig.31 This is a block diagram showing an example configuration of an encoding device in the third embodiment.

[0048] Fig.32 This is a block diagram showing another configuration example of the encoding device in the third embodiment.

[0049] Fig.33 This is a block diagram showing still another configuration example of the encoding device in the third embodiment.

[0050] Fig.34 This is a block diagram showing still another configuration example of the encoding device in the third embodiment.

[0051] Fig.35 This is a block diagram showing still another configuration example of the encoding device in the third embodiment.

[0052] Fig.36 This is a block diagram showing still another configuration example of the encoding device in the third embodiment.

[0053] Fig.37 This is a block diagram showing an example configuration of a decoding device in the third embodiment.

[0054] Fig.38 This is a block diagram showing another configuration example of the decoding device in the third embodiment.

[0055] Fig.39 This is a block diagram showing still another configuration example of the decoding device in the third embodiment.

[0056] Fig.40 This is a block diagram showing still another configuration example of the decoding device in the third embodiment.

[0057] Fig.41 This is a block diagram showing still another configuration example of the decoding device in the third embodiment.

[0058] Fig.42 This is a block diagram showing still another configuration example of the decoding device in the third embodiment.

[0059] Fig.43 This is a flowchart showing an example of the operation of the encoding device in the third embodiment.

[0060] Fig.44 This is a conceptual diagram showing examples of mouth movements corresponding to different voice data feature quantities.

[0061] Fig.45 This is a conceptual diagram showing an example of the movement of the head according to the orientation of the face.

[0062] Fig.46 This is a conceptual diagram showing an example of eye movement.

[0063] Fig.47 This is a conceptual diagram showing another example of eye movement.

[0064] Fig.48 This is a flowchart showing another operation example of the encoding device in the third embodiment.

[0065] Fig.49 This is a block diagram showing an installation example of an encoding device.

[0066] Fig.50 This is a flowchart showing an example of basic operation of the encoding device.

[0067] Fig.51 This is a block diagram showing an implementation example of a decoding device.

[0068] Fig.52This is a flowchart showing an example of basic operation of the decoding device.

[0069] Fig.53 This is the overall structure of the content supply system that implements content distribution services.

[0070] Fig.54 This is a diagram showing an example of a display screen of a web page.

[0071] Fig.55 This is a diagram showing an example of a display screen of a web page.

[0072] Fig.56 The diagram shows an example of a smart phone.

[0073] Fig.57 This is a block diagram showing an example of the configuration of a smart phone. DETAILED DESCRIPTION

[0074] [Introduction]

[0075] Video coding technology can also be used in video conferencing systems. For example, at the first base, a coding device encodes an image into a bit stream. Then, the bit stream is transmitted from the first base to the second base. At the second base, a decoding device decodes the image from the bit stream. In addition, at the second base, a coding device encodes an image into a bit stream. Then, the bit stream is transmitted from the second base to the first base. At the first base, a decoding device decodes the image from the bit stream.

[0076] The video conference system exchanges images between the first site and the second site through the above-mentioned operation. As a result, the image of the person at the first site is displayed at the second site, and the image of the person at the second site is displayed at the first site. On the other hand, due to the large amount of code of the image, it is sometimes difficult to transmit the bit stream. In addition, in order to protect privacy, it is sometimes not desirable to transmit the image as a bit stream. Therefore, a profile image of the person is sometimes used instead of the image.

[0077] However, in a fixed profile image, it is difficult to visually convey information expressed by a person, which may hinder smooth communication. As a result, time and resources may be wasted.

[0078] Therefore, the decoding device of Example 1 includes a circuit and a memory connected to the circuit, and the circuit, when in operation, decodes performance data representing information expressed by a person, generates a person-corresponding image as an image corresponding to the person through a neural network according to the performance data and at least one profile image of the person, and outputs the person-corresponding image.

[0079] Thus, sometimes the information expressed by a person can be reflected in the person-corresponding image of the output object. Therefore, the information expressed by a person is visually easy to convey, and sometimes smooth communication is promoted. As a result, it is possible to suppress the waste of time and resources.

[0080] In addition, the decoding device of Example 2 may be the decoding device of Example 1, wherein the performance data includes data derived from the image of the person.

[0081] Thus, data derived from the image of the person can be reflected on the person-corresponding image to be output. Therefore, a person-corresponding image matching the image of the person can be output.

[0082] In addition, the decoding device of Example 3 may also be in the decoding device of Example 1 or 2, wherein the performance data includes voice data of the person.

[0083] Thus, the person's voice data may be reflected on the person-corresponding image to be output. Therefore, the person-corresponding image matching the person's voice data may be output.

[0084] In addition, the decoding device of Example 4 may be a decoding device of any one of Examples 1 to 3, wherein the at least one profile image is a plurality of profile images, the circuit selects one profile image from the plurality of profile images according to the performance data, and generates the person corresponding image through the neural network according to the one profile image.

[0085] Thus, one profile image corresponding to the information expressed by the person can be selected from a plurality of profile images. In addition, a person-corresponding image of the output object can be generated based on one profile image corresponding to the information expressed by the person. Therefore, a person-corresponding image matching the information expressed by the person can be output.

[0086] Furthermore, the decoding device of Example 5 may be the decoding device of Example 4, wherein the expression data includes an index representing an expression of the person, and the plurality of profile images correspond to a plurality of expressions of the person.

[0087] Thus, sometimes the expression of a person can be reflected in the person-corresponding image of the output object. Therefore, sometimes the person-corresponding image matching the expression of the person can be output. In addition, since the code amount of the index is assumed to be small, the code amount can sometimes be reduced.

[0088] Furthermore, the decoding device of Example 6 may be the decoding device of any one of Examples 1 to 5, wherein the circuit decodes the representation data from each of a plurality of data areas of the bit stream.

[0089] Thereby, a situation that is different for each data region in the bit stream may be reflected in the person-corresponding image to be output. Therefore, a person's timely situation may be reflected in the person-corresponding image.

[0090] Furthermore, the decoding device of Example 7 may be the decoding device of any one of Examples 1 to 5, wherein the circuit decodes the representation data from a header region of the bit stream.

[0091] This may allow a fixed situation in the bit stream to be reflected in the person-corresponding image to be output. Also, the amount of code may be reduced.

[0092] Furthermore, the decoding device of Example 8 may be the decoding device of any one of Examples 1 to 7, wherein the expression data includes data representing at least one of an expression, a facial orientation, a movement of a facial part, and a movement of the head.

[0093] Thereby, the expression of a person or the like can be reflected in the person-corresponding image to be outputted. Therefore, the person-corresponding image matching the expression of a person or the like can be outputted.

[0094] Furthermore, the decoding device of Example 9 may be the decoding device of any one of Examples 1 to 8, wherein the representation data includes data represented by coordinates.

[0095] As a result, the spatial position can be expressed more finely in the expression data. Therefore, the information expressed by the person can be reflected more finely in the person-corresponding image to be output.

[0096] Furthermore, the decoding device of Example 10 may be the decoding device of any one of Examples 1 to 9, wherein the circuit decodes the at least one profile image.

[0097] Thus, even if the profile image is not stored in the memory, the profile image can be used in some cases. Therefore, the memory capacity can be reduced in some cases. In addition, the profile image determined in the encoding can be used in the decoding in some cases. Therefore, the encoding action and the decoding action can be matched in some cases.

[0098] Furthermore, the decoding device of Example 11 may be the decoding device of any one of Examples 1 to 10, wherein the circuit decodes the representation data from a first bit stream and decodes the at least one profile image from a second bit stream different from the first bit stream.

[0099] This may allow the performance data and the profile image to be managed separately. Also, the performance data and the profile image may be updated separately and efficiently.

[0100] Furthermore, the decoding device of Example 12 may be the decoding device of any one of Examples 1 to 9, wherein the circuit reads the at least one profile image from the memory.

[0101] This may omit the decoding of the profile image, thereby reducing the amount of processing and the amount of code.

[0102] In addition, the decoding device of Example 13 may also be the decoding device of Example 3, wherein the at least one profile image is one profile image, the circuit derives a first feature value set representing the movement of the mouth from the speech data, and generates the person corresponding image via the neural network according to the first feature value set and the one profile image.

[0103] Thus, sometimes, based on the voice data associated with the movement of the mouth, it is possible to derive the first feature quantity set that accurately represents the movement of the mouth. In addition, sometimes, the movement of the mouth can be reflected in the person-corresponding image of the output object. Therefore, sometimes, it is possible to output a person-corresponding image that accurately reflects the information expressed by the person.

[0104] In addition, the decoding device of Example 14 may also be a decoding device described in Example 3 or 13, wherein the at least one profile image is a profile image, the circuit simulates the movement of the head or the eyes to derive a second feature value set representing the movement of the head or the eyes, and generates the corresponding image of the person via the neural network according to the voice data, the second feature value set and the one profile image.

[0105] As a result, the person-corresponding image to be outputted can be matched with the simulated head or eye movement, and thus, a more natural person-corresponding image without any sense of incongruity can be outputted.

[0106] In addition, the decoding device of Example 15 may be a decoding device of any one of Examples 3, 13 and 14, wherein the circuit matches the expression of the person in the corresponding image with the expression estimated based on the voice data.

[0107] Thus, the expression corresponding to the voice data can be reflected in the person-corresponding image of the output object. Therefore, the person-corresponding image matching the person's expression can be output in some cases.

[0108] In addition, the encoding device of Example 16 includes a circuit and a memory connected to the circuit, and the circuit, when in operation, encodes performance data representing information expressed by a person, generates a person-corresponding image as an image corresponding to the person through a neural network based on the performance data and at least one profile image of the person, and outputs the person-corresponding image.

[0109] Thus, in the decoding environment corresponding to the coding environment, the information expressed by the person can sometimes be reflected in the person corresponding image of the output object. Therefore, the information expressed by the person is easy to convey visually, which sometimes promotes smooth communication. As a result, it is possible to suppress the waste of time and resources. In addition, in the coding environment, the person corresponding image that is assumed to be output in the decoding environment corresponding to the coding environment can sometimes be output for confirmation.

[0110] In addition, the encoding device of Example 17 may also be in the encoding device of Example 16, wherein the performance data includes data derived from the image of the person.

[0111] Thus, data derived from the image of the person can be reflected on the person-corresponding image to be output. Therefore, a person-corresponding image matching the image of the person can be output.

[0112] In addition, the encoding device of Example 18 may also be in the encoding device of Example 16 or 17, wherein the performance data includes voice data of the person.

[0113] Thus, the person's voice data may be reflected on the person-corresponding image to be output. Therefore, the person-corresponding image matching the person's voice data may be output.

[0114] In addition, the encoding device of Example 19 may be any one of the encoding devices of Examples 16 to 18, wherein the at least one profile image is a plurality of profile images, the circuit selects one profile image from the plurality of profile images according to the performance data, and generates the person corresponding image through the neural network according to the one profile image.

[0115] Thus, one profile image corresponding to the information expressed by the person can be selected from a plurality of profile images. In addition, a person-corresponding image of the output object can be generated based on one profile image corresponding to the information expressed by the person. Therefore, a person-corresponding image matching the information expressed by the person can be output.

[0116] In addition, the encoding device of Example 20 may be the encoding device of Example 19, wherein the expression data includes an index representing an expression of the person, and the plurality of profile images correspond to the plurality of expressions of the person.

[0117] Thus, sometimes the expression of a person can be reflected in the person-corresponding image of the output object. Therefore, sometimes the person-corresponding image matching the expression of the person can be output. In addition, since the code amount of the index is assumed to be small, the code amount can sometimes be reduced.

[0118] In addition, the encoding device of Example 21 can also be in the encoding device of Example 20, wherein the circuit derives the index according to the image of the person through an expression recognition neural network.

[0119] Thus, an index corresponding to an expression accurately recognized from a person's image may be derived. Therefore, the person's expression may be accurately reflected in the person-corresponding image to be output.

[0120] Furthermore, the encoding device of Example 22 may be the encoding device of any one of Examples 16 to 21, wherein the circuit encodes the representation data into each of a plurality of data areas of a bit stream.

[0121] Thereby, a situation that is different for each data region in the bit stream may be reflected in the person-corresponding image to be output. Therefore, a person's timely situation may be reflected in the person-corresponding image.

[0122] In addition, the encoding device of Example 23 may be the encoding device of any one of Examples 16 to 21, wherein the circuit encodes the representation data into a header area of ​​a bit stream.

[0123] This may allow a fixed situation in the bit stream to be reflected in the person-corresponding image to be output. Also, the amount of code may be reduced.

[0124] In addition, the encoding device of Example 24 may be the encoding device of any one of Examples 16 to 23, wherein the expression data includes data representing at least one of an expression, a facial orientation, a movement of a facial part, and a movement of the head.

[0125] Thereby, the expression of a person or the like can be reflected in the person-corresponding image to be outputted. Therefore, the person-corresponding image matching the expression of a person or the like can be outputted.

[0126] In addition, the encoding device of Example 25 may be in the encoding device of any one of Examples 16 to 24, wherein the expression data includes data expressed by coordinates.

[0127] As a result, the spatial position can be expressed more finely in the expression data. Therefore, the information expressed by the person can be reflected more finely in the person-corresponding image to be output.

[0128] Furthermore, the encoding device of Example 26 may be the encoding device of any one of Examples 16 to 25, wherein the circuit encodes the at least one profile image.

[0129] Thus, in a decoding environment corresponding to the encoding environment, even if the profile image is not stored in the memory, it is sometimes possible to use the profile image. Therefore, in the decoding environment, the memory capacity can sometimes be reduced. In addition, in decoding, the profile image determined in the encoding can sometimes be used. Therefore, the encoding action and the decoding action can sometimes be matched.

[0130] Furthermore, the encoding device of Example 27 may be the encoding device of any one of Examples 16 to 26, wherein the circuit encodes the representation data into a first bit stream and encodes the at least one profile image into a second bit stream different from the first bit stream.

[0131] This may allow the performance data and the profile image to be managed separately. Also, the performance data and the profile image may be updated separately and efficiently.

[0132] Furthermore, the encoding device of Example 28 may be the encoding device of any one of Examples 16 to 27, wherein the circuit reads the at least one profile image from the memory.

[0133] This may reduce the amount of processing compared to newly generating a profile image.

[0134] In addition, the encoding device of Example 29 may be the encoding device of Example 18, wherein the at least one profile image is one profile image, the circuit derives a first feature value set representing the movement of the mouth from the speech data, and generates the person corresponding image via the neural network according to the first feature value set and the one profile image.

[0135] Thus, sometimes, based on the voice data associated with the movement of the mouth, it is possible to derive the first feature quantity set that accurately represents the movement of the mouth. In addition, sometimes, the movement of the mouth can be reflected in the person-corresponding image of the output object. Therefore, sometimes, it is possible to output a person-corresponding image that accurately reflects the information expressed by the person.

[0136] In addition, the encoding device of Example 30 may be the encoding device of Example 18 or 29, wherein the at least one profile image is a profile image, the circuit simulates the movement of the head or the eyes to derive a second feature value set representing the movement of the head or the eyes, and generates the corresponding image of the person via the neural network according to the voice data, the second feature value set and the one profile image.

[0137] As a result, the person-corresponding image to be outputted can be matched with the simulated head or eye movement, and thus, a more natural person-corresponding image without any sense of incongruity can be outputted.

[0138] In addition, the encoding device of Example 31 may be in the encoding device of any one of Examples 18, 29 and 30, wherein the circuit matches the expression of the person in the corresponding image with the expression inferred based on the voice data.

[0139] Thus, the expression corresponding to the voice data can be reflected in the person-corresponding image of the output object. Therefore, the person-corresponding image matching the person's expression can be output in some cases.

[0140] In addition, the decoding method of Example 32 includes: a step of decoding performance data representing information expressed by a person; a step of generating a person-corresponding image as an image corresponding to the person via a neural network based on the performance data and at least one profile image of the person; and a step of outputting the person-corresponding image.

[0141] Thus, sometimes the information expressed by a person can be reflected in the person-corresponding image of the output object. Therefore, the information expressed by a person is visually easy to convey, and sometimes smooth communication is promoted. As a result, it is possible to suppress the waste of time and resources.

[0142] In addition, the encoding method of Example 33 includes: a step of encoding performance data representing information expressed by a person; a step of generating a person-corresponding image as an image corresponding to the person via a neural network based on the performance data and at least one profile image of the person; and a step of outputting the person-corresponding image.

[0143] Thus, in the decoding environment corresponding to the coding environment, the information expressed by the person can sometimes be reflected in the person corresponding image of the output object. Therefore, the information expressed by the person is easy to convey visually, which sometimes promotes smooth communication. As a result, it is possible to suppress the waste of time and resources. In addition, in the coding environment, the person corresponding image that is assumed to be output in the decoding environment corresponding to the coding environment can sometimes be output for confirmation.

[0144] Furthermore, these general or specific methods may also be implemented by systems, devices, methods, integrated circuits, computer programs, or non-volatile recording media such as computer-readable CD-ROMs, and may also be implemented by any combination of systems, devices, methods, integrated circuits, computer programs, and recording media.

[0145] [Definition of terms]

[0146] Each term can also be defined as follows as an example.

[0147] (1) Image

[0148] An image is a unit of data composed of a collection of pixels, and is composed of a picture or a block smaller than a picture, and includes both moving images and still images.

[0149] (2) Pictures

[0150] A picture is a processing unit of an image consisting of a collection of pixels, sometimes also called a frame or a field.

[0151] (3) Block

[0152] A block is a processing unit that includes a set of a specific number of pixels, and as described in the following examples, the name is not limited. In addition, the shape is not limited, for example, it obviously includes a rectangle composed of M×N pixels, a square composed of M×M pixels, and also includes a triangle, a circle, or other shapes.

[0153] (Block example)

[0154] Slice / tile / brick

[0155] CTU / super block / basic partition unit

[0156] VPDU / Hardware Processing Division Unit

[0157] CU / processing block unit / prediction block unit (PU) / orthogonal transform block unit (TU) / unit

[0158] Sub block

[0159] (4) Pixels / sample

[0160] A pixel / sample is a point that is the smallest unit constituting an image, and includes not only pixels at integer positions but also pixels at decimal positions generated based on pixels at integer positions.

[0161] (5) Pixel value / sample value

[0162] The pixel value / sample value is the inherent value of the pixel, which obviously includes the brightness value, color difference value, RGB color scale, and also includes the depth value, or binary values ​​such as 0 and 1.

[0163] (6) Flag

[0164] The flag may be a single-bit flag or a plurality of bits flag, for example, a parameter or index of two or more bits. In addition, the flag may be a binary flag or a plurality of values ​​in other bases.

[0165] (7) Signal

[0166] Signals are symbolized and encoded in order to transmit information. In addition to discrete digital signals, they also include analog signals that take continuous values.

[0167] (8) Stream / Bitstream

[0168] A stream / bitstream refers to a data sequence of digital data or a stream of digital data. In addition to a single stream, a stream / bitstream may be divided into multiple layers and composed of multiple streams. In addition, in addition to the case of transmission via serial communication via a single transmission path, it also includes the case of transmission via packet communication via multiple transmission paths.

[0169] (9) Difference / difference (differential)

[0170] In the case of scalars, the difference / difference includes not only the simple difference (xy), as long as the difference operation is included, but also the absolute value of the difference (|xy|), the square difference (x^2-y^2), the square root of the difference (√(xy)), the weighted difference (ax-by: a, b is a constant), and the bias difference (x-y+a: a is the bias).

[0171] (10) and

[0172] In the case of a scalar, the sum includes not only the simple sum (x+y), as long as it includes the sum operation, but also the absolute value of the sum (|x+y|), the sum of squares (x^2+y^2), the square root of the sum (√(x+y)), the weighted sum (ax+by: a, b is a constant), and the biased sum (x+y+a: a is a bias).

[0173] (11) based on

[0174] Based on also includes the case where an element other than the element to be based on is added. In addition, in addition to the case where the result is directly obtained, the case where the result is obtained via an intermediate result is also included.

[0175] (12) Use (used, using)

[0176] The use includes the case where an element other than the element to be used is added. In addition, in addition to the case where the result is directly obtained, the case where the result is obtained via an intermediate result is also included.

[0177] (13) prohibit (prohibit, forbid)

[0178] Prohibition can be replaced by non-permission. In addition, non-prohibition or permission does not necessarily mean an obligation.

[0179] (14) Restriction (limit, restriction / restrict / restricted)

[0180] Restriction can be replaced by non-permission. In addition, non-prohibition or permission does not necessarily mean an obligation. Furthermore, it is sufficient to prohibit part of it in terms of quantity or quality, including a complete prohibition.

[0181] (15) Chromatic aberration (chroma)

[0182] Chroma is an adjective expressed by the symbols Cb and Cr, which specifies that a sample array or a single sample represents one of two color difference signals associated with a primary color. The term chrominance can also be used instead of the term chroma.

[0183] (16) Brightness (luma)

[0184] Brightness is an adjective that specifies that a sample array or a single sample represents a black and white signal associated with a primary color, and is represented by the symbol or subscript Y or L. Brightness can also be represented by the term luminance instead of the term luma.

[0185] [Explanation related to the record]

[0186] In the drawings, the same reference numerals represent the same or similar components. In addition, the sizes and relative positions of the components in the drawings are not necessarily drawn according to a fixed scale.

[0187] The following embodiments are described in detail with reference to the accompanying drawings. In addition, the embodiments described below are all general or specific examples. The numerical values, shapes, materials, components, configuration positions and connection methods of components, steps, relationships and sequences of steps, etc. shown in the following embodiments are examples, and their purpose is not to limit the claims.

[0188] The following describes implementations of the encoding device and the decoding device. The implementations are examples of encoding devices and decoding devices that can use the processing and / or configuration described in each embodiment of the present disclosure. The processing and / or configuration can also be implemented in encoding devices and decoding devices that are different from the implementations. For example, with respect to the processing and / or configuration applicable to the implementations, for example, any of the following can also be implemented.

[0189] (1) Any of the plurality of components of the encoding device or decoding device of the embodiment described in each aspect of the present disclosure may be replaced with or combined with other components described in any of the aspects of the present disclosure.

[0190] (2) In the coding device or decoding device of the embodiment, any changes such as addition, replacement, deletion, etc. of functions or processes performed by some of the multiple components of the coding device or decoding device may be made. For example, any function or process may be replaced or combined with other functions or processes described in any of the various embodiments of the present disclosure.

[0191] (3) In the method implemented by the encoding device or decoding device of the embodiment, any changes such as addition, replacement, and deletion may be made to a part of the multiple processes included in the method. For example, any process in the method may be replaced or combined with other processes described in any of the various embodiments of the present disclosure.

[0192] (4) Some of the multiple components constituting the encoding device or decoding device of the implementation method may be combined with the components described in any of the various embodiments of the present disclosure, or may be combined with components having some of the functions described in any of the various embodiments of the present disclosure, or may be combined with components that implement a part of the processing implemented by the components described in the various embodiments of the present disclosure.

[0193] (5) A constituent element having a portion of the functions of the encoding device or decoding device of an embodiment, or a constituent element that implements a portion of the processing of the encoding device or decoding device of an embodiment, may also be combined with or replaced with a constituent element described in any of the various embodiments of the present disclosure, a constituent element having a portion of the functions described in any of the various embodiments of the present disclosure, or a constituent element that implements a portion of the processing described in any of the various embodiments of the present disclosure.

[0194] (6) In the method implemented by the encoding device or decoding device of the embodiment, any of the multiple processes included in the method may be replaced or combined with the process described in any of the embodiments of the present disclosure or any similar process.

[0195] (7) Some of the multiple processes included in the method implemented by the encoding device or decoding device of the embodiment may be combined with the process described in any of the various aspects of the present disclosure.

[0196] (8) The means of implementing the processing and / or configuration described in each embodiment of the present disclosure is not limited to the encoding device or decoding device of the embodiment. For example, the processing and / or configuration may also be implemented in a device used for a purpose different from the dynamic image encoding or dynamic image decoding disclosed in the embodiment.

[0197] [First method]

[0198] Figure 1 1 is a conceptual diagram showing a video conferencing system 300 in this embodiment. In this embodiment, the video conferencing system 300 includes an encoding device 100 and a decoding device 200. In the video conferencing system 300, the encoding device 100 and the decoding device 200 communicate in real time. For example, the encoding device 100 obtains an image of a person and sends it to the decoding device 200. Then, the decoding device 200 reconstructs the image of the person.

[0199] In the reconstruction of the image, a person may select a profile image representing the person to replace the image of the person. The profile image may be a virtual image of the person, an illustration (graphic) of the person, or a still image of the person, i.e., an actual image. In addition, the profile image may not necessarily have a human shape, but may have a shape similar to an animal, etc.

[0200] The decoding device 200 may include a neural network (NN). In addition, the decoding device 200 may use the neural network (NN) to reproduce the facial expressions included in the video of a person.

[0201] Here, expression is, for example, an appearance on a person's face based on the emotion in the person's heart, and corresponds to the emotion in the person's heart. Therefore, the expression in the present disclosure can also be replaced by emotion.

[0202] Figure 2 This is a conceptual diagram showing an example in which a human image is replaced with a human profile image. For example, by replacing an input image with a profile image, the profile image is displayed in an output image.

[0203] Here, the decoding device 200 may also input a single profile image with a normal expression to the neural network. Here, the input image has a happy expression. However, it is difficult to reproduce an image with a happy expression based on a single profile image with a normal expression. As a result, Figure 2 As shown, sometimes images with normal expressions are reconstructed.

[0204] [Second method]

[0205] Figure 3 1 is a block diagram showing a configuration example of a video conference system 300 in this embodiment. In this embodiment, the video conference system 300 includes a coding device 100 and a decoding device 200. The coding device 100 and the decoding device 200 are connected to each other via a network 400. The video conference system 300 may further include a network 400.

[0206] The encoding device 100 generates a bit stream by encoding data. Here, generating a bit stream by encoding data may be expressed as encoding data into a bit stream. In addition, encoding of data corresponds to compression of data.

[0207] For example, a profile image and an image are input as input images to the encoding device 100. A plurality of profile images or a plurality of images in an image may be input to the encoding device 100. The encoding device 100 encodes the associated data obtained from the profile image and the image into a bit stream. The encoding device 100 may also generate a plurality of bit streams. The bit stream generated in the encoding device 100 is transmitted from the encoding device 100 to the decoding device 200 via the network 400.

[0208] The decoding device 200 generates data by decoding a bit stream. Here, generating data by decoding a bit stream may be expressed as decoding data from a bit stream. In addition, decoding of data corresponds to decompression of data. For example, the decoding device 200 decodes associated data from a bit stream. Then, the decoding device 200 generates an image based on the associated data and outputs it.

[0209] Figure 4 1 is a block diagram showing a configuration example of the encoding device 100. In this example, the encoding device 100 includes a data encoder 101, an image selector 102, and an image generator 103. The image generator 103 includes a neural network.

[0210] The data encoder 101 reads an input image (input object image) included in the input video, and determines an index representing an expression shown in the input image. The index is an index for selecting a profile image corresponding to the expression shown in the input image from a plurality of profile images. Then, the data encoder 101 encodes the index into encoded data. The encoded data may be a single bit stream or a plurality of bit streams.

[0211] The data encoder 101 is not limited to extracting an index representing an expression from an input image, but may also extract additional data for representing a person from an input image and encode it into encoded data. In addition, the data encoder 101 may also encode a plurality of profile images. For example, the data encoder 101 performs entropy encoding during encoding.

[0212] The image selector 102 selects one profile image from among the plurality of profile images based on the index determined by the data encoder 101 .

[0213] The plurality of profile images may be read from a memory inside or outside the encoding device 100. In other words, the plurality of profile images may be determined in advance regardless of the input video, and the plurality of profile images common to the encoding device 100 and the decoding device 200 may be stored in the memory.

[0214] Alternatively, multiple profile images may be derived from the input image. Specifically, in the initial stage, multiple profile images may be extracted from the input image. Alternatively, multiple profile images may be extracted from the input image at regular intervals. Alternatively, other references may be used to perform inter-frame prediction to derive multiple profile images.

[0215] Furthermore, the profile image may be an image including a face shape, such as an avatar of a person or an animal, an illustration (graphic), or a still image, etc. in any form.

[0216] The image generator 103 generates an output image (output target image) via a neural network according to the selected profile image, and outputs the generated output image. The output image may also be output by being displayed. The image generator 103 may also use additional data extracted from the input image when generating the output image.

[0217] Figure 5 This is a conceptual diagram showing an example of determining an index for an input image via an expression recognition neural network. For example, an input image is input to the expression recognition neural network. As a result, an index representing an expression included in the input image is output from the expression recognition neural network.

[0218] Figure 6 2 is a block diagram showing a configuration example of the decoding device 200. In this example, the decoding device 200 includes a data decoder 201, an image selector 202, and an image generator 203. The image generator 203 includes a neural network.

[0219] The data decoder 201 performs entropy decoding on the coded data, for example, to decode the index and the plurality of profile images from the coded data. The image selector 202 selects one profile image from among the plurality of profile images based on the index.

[0220] The coded data may be a single bit stream or multiple bit streams. In the case of multiple bit streams, multiple profile images and indexes may also be decoded from different bit streams. The image generator 203 generates an output image using the selected one profile image and outputs it. The image generator 203 may also output the output image by displaying the output image.

[0221] The encoded data may include additional data related to facial posture, features, expression, head movement, etc. separately from the index and profile image. In addition, the image generator 203 may generate and output an output image using these additional data in addition to the profile image.

[0222] Figure 7 1 is a block diagram showing another configuration example of the decoding device 200. In this example, the multiple profile images are not the coded data but are acquired separately. For example, the multiple profile images are acquired from a memory. The memory used to acquire the multiple profile images may be a component of the decoding device 200 or may not be a component of the decoding device 200. The multiple profile images may also be acquired from another network different from the network used to acquire the coded data.

[0223] Alternatively, other references may be used for inter-frame prediction to derive multiple profile images.

[0224] For example, the data decoder 201 decodes the index from the coded data. The image selector 202 reads a plurality of profile images from the memory. Then, the image selector 202 selects one profile image from the plurality of profile images based on the index. Alternatively, the image selector 202 may only read the profile image corresponding to the index. The image generator 203 generates an output image using the selected one profile image and outputs it. The output image may also be displayed.

[0225] Figure 8 1 is a flowchart showing an operation example of the encoding device 100. First, the data encoder 101 encodes the index into encoded data and outputs the encoded data including the index (S101). Then, the image selector 102 selects a profile image from a plurality of profile images based on the index (S102). Then, the image generator 103 generates an output image through a neural network according to the profile image (S103).

[0226] The encoded data may also correspond to a bit stream. The index indicates one of the multiple profile images. For example, the multiple profile images correspond to multiple different expressions, and the index indicates one of the multiple profile images by indicating one of the multiple expressions.

[0227] Specifically, the data encoder 101 determines the index so that the expression of the profile image selected by the index matches the expression of the input image. In other words, the data encoder 101 determines the index so that the index represents the profile image having the highest degree of consistency with the expression of the input image among the multiple profile images.

[0228] The set of multiple profile images may also be a set of multiple images of the same image having different expressions. Alternatively, the index may be selected via an expression recognition neural network.

[0229] Alternatively, the set of multiple profile images may be a set of multiple images of the person himself, a set of multiple images of an avatar, a set of multiple images of others, or a set including any combination of the above.

[0230] For example, any one of the above-mentioned sets may be selected by human input. In addition, a profile image may be selected from a plurality of profile images included in the selected set via a neural network. In other words, an index corresponding to a profile image may be selected by human input and a neural network. Alternatively, for example, in the case where one set is used instead of multiple sets, an index may be selected by a neural network instead of human input.

[0231] In addition, the order of the steps may be reversed. For example, the encoding of the index (S101) may be performed after the selection of the profile image (S102), or after the generation of the output image (S103). Specifically, after the profile image is selected, the index corresponding to the selected profile image may be encoded.

[0232] In addition, encoding processing other than indexing may be performed. For example, multiple profile images may be encoded. In addition, additional data related to facial posture, features, expressions, and head movements may be encoded separately from the index and the profile image.

[0233] The selection ( S102 ) and generation ( S103 ) performed by the encoding device 100 may also be performed in the decoding device 200 in the same manner.

[0234] Fig. 9 201. The flowchart shows an operation example of the decoding device 200. First, the data decoder 201 obtains the coded data including the index and decodes the index from the coded data (S201). Then, the image selector 202 selects a profile image from a plurality of profile images based on the index (S202). Then, the image generator 203 generates an output image through a neural network according to the profile image (S203).

[0235] The profile image includes a face, and the output image also includes a face. The multiple profile images are different images and represent different expressions. The multiple profile images may also include the face of the same person. The multiple profile images may also be decoded from the coded data including the index. Alternatively, the multiple profile images may also be decoded from other coded data different from the coded data including the index. Alternatively, the multiple profile images may also be read from a memory inside or outside the decoding device 200.

[0236] In addition, additional data for expressing a person may be decoded from the coded data including the index or other coded data, and the decoded additional data may be used in generating the output image (S203).

[0237] Fig.10 This is a conceptual diagram showing an example of the position of an index in a bitstream. The index may also be included in each data area of ​​the bitstream. The value of the index may be represented by 0, 1, 2, 3, A, B, C, or I, II, III.

[0238] Fig.11This is a conceptual diagram showing other examples of the position of the index in the bitstream. The index may also be included in the header area of ​​the bitstream. That is, a common index may also be applied in the bitstream. In addition, the index may also be switched by switching the bitstream at a certain time.

[0239] Specifically, for example, the index may be switched in units that can be randomly accessed. In this case, the index may be stored in an SPS (Sequence Parameter Set) or SEI in an image coding method such as H.265 / HEVC or H.266 / VVC. Alternatively, when the index is switched in units of one image (picture), it may be stored in a PPS (Picture Parameter Set) or the like.

[0240] In addition, the plurality of profile images may be included in the same region as the index or in a region different from the index. In addition, the additional data extracted from the input image may be included in the same region as the index or in a region different from the index.

[0241] Fig.12 is a conceptual diagram showing an example of a plurality of profile images. Fig.12 As shown in FIG. 1 , the orientation of the face corresponding to the facial posture may be different in a plurality of profile images. The orientation of the face may also be expressed by a yaw angle, a pitch angle, and a roll angle.

[0242] Fig.13 is a conceptual diagram showing another example of multiple profile images. Fig.13 As shown, facial features may also be different in multiple profile images. Specifically, features associated with the shape of the corners of the eyes, the outline of the nose, the shape and width of the lips, the length of the eyebrows, the outline of the face, or any combination thereof may also be different in multiple profile images.

[0243] Fig.14 is a conceptual diagram showing still another example of a plurality of profile images. Fig.14 As shown, in multiple profile images, the expressions may be different, such as happy, sad, angry, or normal.

[0244] The multiple profile images may be obtained by photographing the face of the same person at different times. Alternatively, the multiple profile images may be obtained by photographing the face of the same person at different positions. Alternatively, the multiple profile images may be obtained by photographing the face of the same person using different devices.

[0245] In addition, for example, the encoded data includes data related to facial posture, features, expressions, and head movements, etc. The encoded data may also include coordinates indicating the movement of the head.

[0246] Fig.15 is a conceptual diagram showing an example of data represented by coordinates. Fig.15 As in the example of , the three-dimensional coordinates of the yaw angle, the pitch angle, and the roll angle may be included in the encoded data as coordinates representing the orientation of the face corresponding to the movement of the head.

[0247] Fig.16 is a conceptual diagram showing another example of data represented by coordinates. Fig.16 As in the example of FIG. 1 , orthogonal coordinates may also be included in the encoded data as coordinates representing a position corresponding to the movement of the head.

[0248] Fig.17 This is a conceptual diagram showing an example of a change in the orientation of a face. For example, the change in the orientation of a face is represented by (yaw angle, pitch angle, roll angle) = (0, 0.71, 0). Such data representing the change in the orientation of a face using three-dimensional coordinates may also be included in the encoded data.

[0249] Fig.18 This is a conceptual diagram showing an example of the change of lips. For example, the change of lips is represented by (yaw angle, pitch angle, roll angle) = (0.34, -0.21, -0.91). Such data representing the change of lips by 3D coordinates may also be included in the encoded data. In addition, not limited to lips, data representing the movement of parts of the face by 3D coordinates may also be included in the encoded data. For example, the parts of the face are eyes, eyelids, lips, eyebrows, or facial muscles.

[0250] Fig.19 : is a conceptual diagram showing an example of the movement of the head. The movement of the head corresponds to the translational movement of the face. For example, as the movement of the head, the changed position of the head is represented by the orthogonal coordinates of (-40, -40, 0).

[0251] In addition, a combination of the above-mentioned examples may be used. In addition, the output image and the profile image may have the same background or different backgrounds.

[0252] In addition, for example, the plurality of profile images are a plurality of images including the face of the same person and being different from each other. The plurality of profile images may also be standardized.

[0253] Fig. 20is a conceptual diagram showing an example of standardization. In this example, the backgrounds of a plurality of profile images are unified. Specifically, the plurality of profile images before standardization have different backgrounds, while the plurality of profile images after standardization have the same background.

[0254] For example, the backgrounds of the multiple profile images may be unified by removing the backgrounds. The backgrounds of one profile image among the multiple profile images may be unified by arranging the background of the other profile images. In addition, the backgrounds may be unified by loading the background from the memory and inserting the loaded background into the multiple profile images.

[0255] Fig.21 is a conceptual diagram showing another example of standardization. In this example, the sizes of faces in multiple profile images are unified. Specifically, multiple profile images before standardization have faces of different sizes, while multiple profile images after standardization have faces of the same size. For example, in multiple profile images, faces in each profile image may be scaled so that the sizes of the faces become the same.

[0256] Fig. 22 : is a conceptual diagram showing another example of standardization. In this example, the orientations of faces in a plurality of profile images are unified. Specifically, a plurality of profile images before standardization have faces with different orientations, while a plurality of profile images after standardization have faces with the same orientation. For example, in a plurality of profile images, faces in each profile image may be corrected so that the orientations of the faces become the same.

[0257] Furthermore, any two or more of the above-mentioned multiple standardizations may be combined.

[0258] Fig.23 This is a conceptual diagram showing an example of generating an output image through a neural network according to coded data and a profile image. Here, the coded data includes data related to facial posture, features, expressions, and head movements. In addition, the coded data may also include data representing the movement of the head by coordinates.

[0259] Specifically, for example, a profile image and data included in the coded data are input to the neural network. The data included in the coded data may also be input to the neural network before encoding or after decoding. Then, an output image is output from the neural network. By causing the data included in the coded data to act on the profile image through the neural network, an output image including an expression obtained by correcting the expression in the profile image can be output.

[0260] For example, the coded data may include data related to facial posture, features, expressions, head movements, etc. for each input image. In other words, these data may be coded for each input image.

[0261] Alternatively, these data may not be encoded for all input images. In this case, the data not encoded for the input image may be generated by interpolation from other data encoded for the input image. Alternatively, the reconstructed image of the input image not encoded with these data may be interpolated based on the reconstruction result of the input image encoded with these data.

[0262] Fig.24 This is a conceptual diagram showing an example of associating an index with each of a plurality of profile images. In this example, index 0, index 1, index 2, and index 3 are associated with a sad face, a happy face, a normal face, and a panicked face, respectively.

[0263] Fig.25 is a conceptual diagram showing the selection of a profile image and the generation of an output image. Fig.24 Similarly to the example, multiple indexes are associated with multiple profile images. Then, from the multiple profile images associated with indexes 0 to 3, the profile image associated with index 1 is selected. Furthermore, the selected profile image and the encoded data are input to the neural network. Thus, an output image is generated.

[0264] Fig.26 This is a conceptual diagram showing an example of a neural network for generating an output image. The neural network for generating an output image can also be a Generative Adversarial Network (GAN). GAN provides an intelligent solution to modeling data generation as an unsupervised learning problem as a supervised learning problem. The discriminator model learns to distinguish between pseudo data generated by the generator model and real data. The two models are learned by playing a minimax game.

[0265] Fig. 27 is a conceptual diagram showing another example of a neural network for generating an output image. The neural network for generating an output image can also be a variational autoencoder. A variational autoencoder compresses input information into a constrained multivariate latent distribution and reconstructs the input information as accurately as possible.

[0266] Fig.28 : is a conceptual diagram showing another example of a neural network for generating an output image. The neural network for generating an output image can also be a flow-based generative model. The flow-based generative model is constructed by a series of reversible transformations. Since this model explicitly learns the data distribution F(X), the loss function simply becomes the negative log-likelihood.

[0267] The neural network used to generate the output image may be a hybrid model corresponding to a combination of any two or more of the three models described above, or may be a neural network different from the three models described above.

[0268] In this method, a profile image corresponding to a person's image is selected from a plurality of profile images. In addition, an index indicating the selected profile image is made a signal, and the same profile image is also selected in a decoding environment. Then, an output image is generated by the selected profile image and a neural network. Thus, the selected profile image is reflected in the output image. Therefore, it is possible to help make the output image close to the image of a person.

[0269] In addition, the operation of the decoding environment can also be performed in the encoding environment, and the operation of the encoding environment can also be performed in the decoding environment. In addition, the multiple components included in the encoding device 100 and the multiple components included in the decoding device 200 can also perform corresponding operations. In addition, all the components described in this embodiment are not necessarily required. It is also possible to install only a part of the components of this embodiment.

[0270] [Third Method]

[0271] Fig.29 1 is a conceptual diagram showing a video conferencing system 300 in this embodiment. In this embodiment, the video conferencing system 300 includes an encoding device 100 and a decoding device 200. In the video conferencing system 300, the encoding device 100 and the decoding device 200 communicate in real time. For example, the encoding device 100 obtains an image of a person and sends it to the decoding device 200. Then, the decoding device 200 reconstructs the image of the person.

[0272] In the reconstruction of the image, a profile image representing the person may be selected by the person to replace the image of the person. The profile image may be a virtual image of the person, an illustration of the person, or a still image of the person. In this case, instead of using the image of the person, the voice data of the person may be used. For example, the decoding device 200 reconstructs the image accompanied by the movement of the head of the person, etc., using the voice data of the person and a neural network (NN).

[0273] Fig.30 4 is a block diagram showing a configuration example of a video conference system 300. In this embodiment, the video conference system 300 includes a coding device 100 and a decoding device 200. The coding device 100 and the decoding device 200 are connected to each other via a network 400. The video conference system 300 may further include a network 400.

[0274] The encoding device 100 generates a bit stream by encoding data. Here, generating a bit stream by encoding data may be expressed as encoding data into a bit stream. In addition, encoding of data corresponds to compression of data.

[0275] For example, a profile image and voice data are input to the encoding device 100. The encoding device 100 encodes the voice data into a bit stream. The bit stream generated in the encoding device 100 is transmitted from the encoding device 100 to the decoding device 200 via the network 400.

[0276] The decoding device 200 generates data by decoding a bit stream. Generating data by decoding a bit stream is sometimes expressed as decoding data from a bit stream. In addition, decoding of data corresponds to decompression of data. For example, the decoding device 200 decodes voice data from a bit stream. Then, the decoding device 200 generates an image based on the voice data and outputs it.

[0277] The encoding device 100 may also encode the profile image in addition to the voice data. The decoding device 200 may also decode the profile image in addition to the voice data. In addition, the decoding device 200 may also generate an image based on the voice data and the profile image and output it. In addition, a neural network may also be used when generating the image.

[0278] Fig.31 1 is a block diagram showing a configuration example of the coding device 100. In this example, the coding device 100 includes a data encoder 101 and an image generator 103. The image generator 103 includes a neural network.

[0279] The data encoder 101 reads input voice data, encodes the input voice data into coded data, and outputs the coded data. For example, the data encoder 101 performs entropy coding during encoding. The coded data may be a single bit stream or multiple bit streams. In addition, the coded data may also be included in a header of a bit stream.

[0280] The encoded data may also include additional data for representing a person. The data encoder 101 may also extract features of the voice data from the voice data, and encode the extracted features as additional data into the encoded data. For example, the data encoder 101 may also derive character data (text data) from the voice data, and encode the derived character data as additional data into the encoded data.

[0281] Alternatively, the data encoder 101 may read an image, extract features of the image, and encode the extracted features as additional data into encoded data. Alternatively, the data encoder 101 may encode a profile image into encoded data.

[0282] The image generator 103 may also generate an output image through a neural network according to the voice data and the profile image, and output the output image. The image generator 103 may also output by displaying the output image. In addition, the image generator 103 may also use additional data encoded as encoded data when generating the output image.

[0283] The profile image may be read from a memory inside or outside the encoding device 100. Alternatively, the profile image may be derived from an input image obtained by reading the input image. Specifically, in the initial stage, the input image may be read and the profile image may be extracted from the input image.

[0284] In addition, the image generator 103 may also generate the output image in real time. Alternatively, the image generator 103 may also generate the output image in advance. In this case, the image generator 103 generates a plurality of output image candidates in advance based on a plurality of predefined speech data samples.

[0285] Then, the image generator 103 finds a speech data sample corresponding to the input speech data among a plurality of predefined speech data samples, and then outputs an output image candidate corresponding to the found speech data sample among a plurality of pre-generated output image candidates as an output image.

[0286] Fig.32 1 is a block diagram showing another configuration example of the coding device 100. In this example, the coding device 100 includes a speech deriver 104, a data encoder 101, and an image generator 103. In this example, input character data is read into the coding device 100 instead of input speech data.

[0287] The speech exporter 104 reads the input character data and exports the speech data from the input character data. Fig.31 Compared with the example, in the data encoder 101 and the image generator 103, the voice data derived by the voice deriver 104 is used instead of the input voice data.

[0288] In addition, the data encoder 101 may encode the input character data instead of the voice data or in addition to the voice data. In addition, the image generator 103 may generate an output image using the input character data in addition to the voice data, the profile image and the neural network.

[0289] Regarding other aspects, the data encoder 101 and the image generator 103 are Fig.31 The same action is performed as in the example.

[0290] Fig.331 is a block diagram showing still another configuration example of the encoding device 100. In this example, the encoding device 100 includes a data encoder 101, a feature deriver 105, and an image generator 103.

[0291] The feature deriver 105 reads the input speech data and derives a first feature quantity set representing the movement of the mouth from the input speech data. The image generator 103 generates an output image through a neural network according to the first feature quantity set and the profile image, and outputs the output image.

[0292] In addition, the image generator 103 may generate an output image in real time. Alternatively, the image generator 103 may generate an output image in advance. In this case, the image generator 103 generates a plurality of output image candidates in advance based on a plurality of first feature quantity sets defined in advance.

[0293] In addition, the image generator 103 finds a first feature value set corresponding to the derived first feature value set from among the plurality of predefined first feature value sets. Then, the image generator 103 outputs the output image candidate corresponding to the found first feature value set from among the plurality of pre-generated output image candidates as an output image.

[0294] The data encoder 101 may encode the first feature set derived from the voice data instead of the voice data, or may encode the first feature set derived from the voice data in addition to the voice data. In addition, the image generator 103 may generate an output image using the input voice data in addition to the first feature set, the profile image, and the neural network.

[0295] Regarding other aspects, the data encoder 101 and the image generator 103 are Fig.31 The same action is performed as in the example.

[0296] Fig.34 1 is a block diagram showing another configuration example of the encoding device 100. In this example, the encoding device 100 includes a speech deriver 104, a data encoder 101, a feature deriver 105, and an image generator 103. In addition, in this example, input character data is read into the encoding device 100 instead of input speech data.

[0297] The speech deriver 104 reads the input character data and derives speech data from the input character data. The feature deriver 105 derives a first feature quantity set representing the movement of the mouth from the derived speech data. That is, the feature deriver 105 uses the derived speech data instead of the input speech data.

[0298] Regarding other aspects, the data encoder 101, the feature deriver 105 and the image generator 103 are Fig.33In addition, the speech exporter 104 and Fig.32 The same action is performed as in the example.

[0299] Fig.35 1 is a block diagram showing still another configuration example of the encoding device 100 . In this example, the encoding device 100 includes a data encoder 101 , a simulator 106 , and an image generator 103 .

[0300] The simulator 106 simulates the movement of the head or eyes and derives a second feature quantity set representing the movement of the head or eyes. Here, the movement of the head or eyes is not limited to the movement of only the head or only the movement of the eyes, but may correspond to the movement of both the head and the eyes. For example, the simulator 106 simulates the natural movement of the head or eyes based on one or more general models. Thus, the sense of incongruity of the head or eyes being continuously stopped in the output image is reduced.

[0301] In addition, the simulator 106 may also read input images and simulate the movement of the head or eyes based on the input images.

[0302] The image generator 103 generates an output image via a neural network according to the input speech data, the derived second feature quantity set, and the profile image, and outputs the output image.

[0303] In addition, the image generator 103 may generate the output image in real time. Alternatively, the image generator 103 may generate the output image in advance. For example, the image generator 103 generates a plurality of output image candidates in advance based on a plurality of predefined speech data samples and a plurality of predefined second feature quantity sets.

[0304] In addition, the image generator 103 finds the speech data sample and the second feature set corresponding to the input speech data and the derived second feature set from among the predefined plurality of speech data samples and the predefined plurality of second feature sets. Then, the image generator 103 outputs the output image candidate corresponding to the found speech data sample and the second feature set from among the pre-generated plurality of output image candidates as the output image.

[0305] The data encoder 101 may encode the second feature quantity set derived by simulation in addition to the speech data.

[0306] Regarding other aspects, the data encoder 101 and the image generator 103 are Fig.31 The same action is performed as in the example.

[0307] Fig.361 is a block diagram showing another configuration example of the coding device 100. In this example, the coding device 100 includes a speech deriver 104, a data encoder 101, a simulator 106, and an image generator 103. In this example, input character data is read into the coding device 100 instead of input speech data.

[0308] The speech exporter 104 reads the input character data and exports the speech data from the input character data. The image generator 103 generates an output image through a neural network according to the exported speech data, the exported second feature quantity set, and the profile image, and outputs the output image. That is, the image generator 103 uses the exported speech data to replace the input speech data.

[0309] Regarding other aspects, the data encoder 101, the simulator 106 and the image generator 103 are Fig.35 In addition, the speech exporter 104 and Fig.32 The same action is performed as in the example.

[0310] Fig.37 2 is a block diagram showing a configuration example of the decoding device 200. In this example, the decoding device 200 includes a data decoder 201 and an image generator 203. The image generator 203 includes a neural network.

[0311] The data decoder 201 performs entropy decoding on the coded data, for example, to decode the profile image and the voice data from the coded data. Then, the data decoder 201 outputs the voice data. The data decoder 201 may also output the voice data as voice.

[0312] The coded data may be a single bit stream or multiple bit streams. In the case of multiple bit streams, the profile image and voice data may be decoded from different bit streams. In addition, the coded data may be included in the header of the bit stream.

[0313] Alternatively, the profile image may be read from a memory or the like inside or outside the decoding device 200. In this case, the profile image may not be decoded from the encoded data.

[0314] The encoded data may include additional data for expressing a person. In addition, the data decoder 201 may decode the additional data from the encoded data.

[0315] The image generator 203 may also generate an output image through a neural network according to the decoded voice data and the profile image, and output the output image. The image generator 203 may also output by displaying the output image. In addition, the image generator 203 may also use additional data decoded from the encoded data when generating the output image.

[0316] In addition, the image generator 203 may also generate the output image in real time. Alternatively, the image generator 203 may also generate the output image in advance. In this case, the image generator 203 generates a plurality of output image candidates in advance based on a plurality of predefined speech data samples.

[0317] In addition, the image generator 203 finds a speech data sample corresponding to the decoded speech data among a plurality of predefined speech data samples, and then outputs an output image candidate corresponding to the found speech data sample among a plurality of pre-generated output image candidates as an output image.

[0318] Fig.38 2 is a block diagram showing another configuration example of the decoding device 200. In this example, the decoding device 200 includes a data decoder 201, a speech deriver 204, and an image generator 203.

[0319] The data decoder 201 performs entropy decoding on the coded data, for example, thereby decoding the profile image and character data from the coded data. Fig.37 Compared with the example of , the data decoder 201 decodes character data from the coded data to replace the voice data. Then, the data decoder 201 outputs the character data. The data decoder 201 can also output by displaying the character data.

[0320] The encoded data may be a single bit stream or a plurality of bit streams. In the case of a plurality of bit streams, the profile image and the character data may be decoded from different bit streams.

[0321] The speech exporter 204 exports the speech data from the character data. Then, the speech exporter 204 outputs the speech data. The speech exporter 204 may also output the speech data as speech.

[0322] The image generator 203 generates an output image through a neural network according to the derived voice data and the profile image, and outputs the output image. Fig.37 Compared with the example of , the image generator 203 uses the voice data derived from the character data instead of the voice data decoded from the encoded data, and generates an output image.

[0323] Regarding other aspects, the data decoder 201 and the image generator 203 are Fig.37 The same action is performed as in the example.

[0324] Fig.39 2 is a block diagram showing still another configuration example of the decoding device 200. In this example, the decoding device 200 includes a data decoder 201, a feature deriver 205, and an image generator 203.

[0325] The feature deriver 205 derives a first feature quantity set expressing the movement of the mouth from the decoded speech data. The image generator 203 generates an output image through a neural network according to the first feature quantity set and the profile image, and outputs the output image.

[0326] In addition, the image generator 203 may generate an output image in real time. Alternatively, the image generator 203 may generate an output image in advance. In this case, the image generator 203 generates a plurality of output image candidates in advance based on a plurality of predefined first feature quantity sets.

[0327] Then, the image generator 203 finds the first feature value set corresponding to the derived first feature value set from among the plurality of predefined first feature value sets. Then, the image generator 203 outputs the output image candidate corresponding to the found first feature value set from among the plurality of pre-generated output image candidates as the output image.

[0328] Furthermore, the image generator 203 may generate an output image using decoded speech data in addition to the first feature quantity set, the profile image, and the neural network.

[0329] Regarding other aspects, the data decoder 201 and the image generator 203 are Fig.37 The same action is performed as in the example.

[0330] Fig.40 2 is a block diagram showing still another configuration example of the decoding device 200. In this example, the decoding device 200 includes a data decoder 201, a speech deriver 204, a feature deriver 205, and an image generator 203.

[0331] The data decoder 201 performs entropy decoding on the coded data, for example, thereby decoding the profile image and character data from the coded data. Fig.39 Compared with the example of FIG. 1 , the data decoder 201 decodes character data from the encoded data to replace the voice data.

[0332] The speech deriver 204 derives speech data from the decoded character data. The feature deriver 205 derives a first feature quantity set representing the movement of the mouth from the derived speech data. Fig.39 Compared with the example, the feature exporter 205 uses the voice data derived from the character data instead of the voice data decoded from the encoded data to export the first feature quantity set.

[0333] Regarding other aspects, the data decoder 201, the feature deriver 205 and the image generator 203 are Fig.39 In addition, the speech exporter 204 and Fig.38The same action is performed as in the example.

[0334] Fig.41 2 is a block diagram showing still another configuration example of the decoding device 200 . In this example, the decoding device 200 includes a data decoder 201 , a simulator 206 , and an image generator 203 .

[0335] The simulator 206 simulates the movement of the head or eyes and derives a second feature quantity set representing the movement of the head or eyes. Here, the movement of the head or eyes is not limited to the movement of only the head or only the movement of the eyes, but may correspond to the movement of both the head and the eyes. For example, the simulator 206 simulates the natural movement of the head or eyes based on one or more general models. As a result, the sense of incongruity of the head or eyes being continuously stopped in the output image is reduced.

[0336] Furthermore, the simulator 206 may simulate the movement of the head or eyes based on the additional data decoded from the encoded data.

[0337] The image generator 203 generates an output image via a neural network according to the decoded speech data, the derived second feature quantity set, and the profile image, and outputs the output image. In addition, the image generator 203 may also generate the output image in real time. Alternatively, the image generator 203 may also generate the output image in advance. For example, the image generator 203 generates a plurality of output image candidates in advance based on a plurality of predefined speech data samples and a plurality of predefined second feature quantity sets.

[0338] Then, the image generator 203 finds the speech data sample and the second feature set corresponding to the decoded speech data and the derived second feature set from among the predefined plurality of speech data samples and the predefined plurality of second feature sets. Then, the image generator 203 outputs the output image candidate corresponding to the found speech data sample and the second feature set from among the pre-generated plurality of output image candidates as the output image.

[0339] Regarding other aspects, the data decoder 201 and the image generator 203 are Fig.37 The same action is performed as in the example.

[0340] Fig.42 2 is a block diagram showing still another configuration example of the decoding device 200. In this example, the decoding device 200 includes a data decoder 201, a speech deriver 204, a simulator 206, and an image generator 203.

[0341] The data decoder 201 performs entropy decoding on the coded data, for example, thereby decoding the profile image and character data from the coded data. Fig.41Compared with the example of FIG. 1 , the data decoder 201 decodes character data from the encoded data to replace the voice data.

[0342] The speech exporter 204 exports speech data from the decoded character data. The image generator 203 generates an output image through a neural network according to the exported speech data, the exported second feature quantity set, and the profile image, and outputs the output image. Fig.41 Compared with the example of , the image generator 203 uses the voice data derived from the character data instead of the voice data decoded from the encoded data.

[0343] Regarding other aspects, the data decoder 201, the simulator 206 and the image generator 203 are Fig.41 In addition, the speech exporter 204 and Fig.38 The same action is performed as in the example.

[0344] Fig.43 1 is a flowchart showing an operation example of the encoding device 100. First, the data encoder 101 encodes the speech data into the encoded data (S301). The encoded data may also correspond to a bit stream. The data encoder 101 may also compress the speech data using the AAC codec when encoding the speech data. In addition, the data encoder 101 may also encode the feature quantity used to generate the output image into the encoded data.

[0345] The coded data may be a single bit stream or multiple bit streams. In addition, the coded data may be included in the header of the bit stream.

[0346] In addition, the image generator 103 generates an output image (S302) via a neural network according to the voice data and the profile image. The output image is an image corresponding to a person. For example, the output image at least represents a head including a face of the person. In addition, a first feature quantity set representing the movement of the mouth can also be derived from the voice data. Then, the image generator 103 can also generate an output image via a neural network according to the first feature quantity set and the profile image.

[0347] For example, a speech recognition model, an acoustic model, a natural language model, a lattice model, etc. may be applied to speech data to calculate spectral envelope, pitch, sound, transient detection, phoneme classification, and speech, thereby deriving a first feature quantity set.

[0348] The first feature quantity set may also be represented by coordinates for changing the direction, position, shape or size of the mouth. Fig.15 The first feature set can also be represented by Fig.16 Such orthogonal coordinates.

[0349] Fig.44 This is a conceptual diagram showing examples of mouth movements corresponding to different speech data feature quantities. For example, from speech data with a higher pitch, the first feature quantity set representing the movement of the mouth is derived as (yaw angle, pitch angle, roll angle) = (0.34, -0.51, 0.91). In addition, from speech data with a lower pitch, the first feature quantity set representing the movement of the mouth is derived as (yaw angle, pitch angle, roll angle) = (0.34, -0.21, 0.91).

[0350] Alternatively, an image may be generated through a neural network based on a first feature set representing the movement of the mouth and a second feature set representing the movement of the head or eyes. The first feature set representing the movement of the mouth is derived from speech data, and the second feature set representing the movement of the head or eyes is derived by simulating the movement of the head or eyes.

[0351] Fig.45 : is a conceptual diagram showing an example of the movement of the head. This example corresponds to an example of the second feature quantity set. In addition, in this example, the movement of the head corresponds to the orientation of the face.

[0352] At time T=1, the orientation of the face corresponds to the front, and the second feature set is represented by (yaw angle, pitch angle, roll angle)=(0, 0, 0). At time T=2, the orientation of the face corresponds to the right side, and the second feature set is represented by (yaw angle, pitch angle, roll angle)=(0, 0.23, 0). At time T=3, the orientation of the face corresponds to the left side, and the second feature set is represented by (yaw angle, pitch angle, roll angle)=(0, -0.23, 0).

[0353] Fig.46 : is a conceptual diagram showing an example of eye movement. This example corresponds to another example of the second feature quantity set. In this example, the eye movement corresponds to the movement of the eyelid.

[0354] At time T=1, the eyelids are closed, and the second feature quantity set is represented by (yaw angle, pitch angle, roll angle)=(0, 0, 0). At time T=2, the eyelids are slightly open, and the second feature quantity set is represented by (yaw angle, pitch angle, roll angle)=(0, 0.15, 0). At time T=3, the eyelids are widely open, and the second feature quantity set is represented by (yaw angle, pitch angle, roll angle)=(0, 0.35, 0).

[0355] Fig.47 : is a conceptual diagram showing another example of eye movement. This example corresponds to another example of the second feature quantity set. In this example, the eye movement corresponds to the direction of the eye.

[0356] At time T=1, the eye orientation corresponds to the front, and the second feature set is represented by (yaw angle, pitch angle, roll angle)=(0, 0, 0). At time T=2, the eye orientation corresponds to the right side, and the second feature set is represented by (yaw angle, pitch angle, roll angle)=(0, 0.56, 0). At time T=3, the eye orientation corresponds to the left side, and the second feature set is represented by (yaw angle, pitch angle, roll angle)=(0, -0.56, 0).

[0357] The second feature quantity set may be a combination of the above-mentioned examples. An output image may be generated through a neural network according to the second feature quantity set and the speech data.

[0358] The output image may also be generated in real time from the neural network. Alternatively, multiple output image candidates may be generated in advance via the neural network according to a predetermined plurality of speech data samples. In addition, the output image candidate corresponding to the speech data to be processed may also be selected as the output image.

[0359] Alternatively, a plurality of output image candidates may be generated in advance through a neural network according to a plurality of predetermined first feature quantity sets. Alternatively, an output image candidate corresponding to the first feature quantity set of the processing target speech data may be selected as the output image.

[0360] Alternatively, multiple output image candidates may be generated in advance via a neural network according to multiple predetermined speech data samples and multiple predetermined second feature sets. Alternatively, an output image candidate corresponding to the speech data to be processed and the second feature set as a simulation result may be selected as the output image.

[0361] For example, the image generator 103 may also estimate the expression based on the voice data. Specifically, the image generator 103 may also analyze the frequency of the voice data and the amplitude of the voice data. Here, the frequency represents the height (pitch) of the sound, and the amplitude represents the size (loudness, volume) of the sound. Furthermore, the image generator 103 may also estimate the meaning of the tone and vocabulary based on the analysis results of the voice data, and determine the expression. The image generator 103 may also estimate the expression through a neural network according to the voice data.

[0362] In addition, the image generator 103 may also generate an output image so that the expression estimated from the voice data is consistent with the expression of the person in the output image. In other words, the image generator 103 may also generate an output image via a neural network to match the actual expression.

[0363] In addition, the words spoken by a person sometimes do not match the emotions in the person's mind. The image generator 103 may also generate an output image that better matches the actual emotion by using non-language information included in the speech data, compared to a case where the non-language information is not used.

[0364] In addition, the neural network may be prepared for each language. Thus, sometimes the neural network corresponding to the language can be smoothly learned. In addition, the neural network may be prepared for each combination of language and country. Thus, sometimes the neural network corresponding to the language and culture can be smoothly learned.

[0365] Fig.48 4 is a flowchart showing another example of the operation of the encoding device 100. First, the data encoder 101 encodes character data into encoded data (S401). The encoded data may also correspond to a bit stream. The data encoder 101 may also compress the character data when encoding the voice data. In addition, the data encoder 101 may also encode a feature quantity for generating an output image into encoded data.

[0366] The coded data may be a single bit stream or multiple bit streams. In addition, the coded data may be included in the header of the bit stream.

[0367] In addition, the speech exporter 104 exports the speech data from the character data (S402). In addition, the data encoder 101 may also encode the exported speech data. In the case where the character data is not used when generating the output image, the data encoder 101 may not encode the character data.

[0368] In addition, the image generator 103 generates an output image through a neural network according to the voice data and the profile image (S403). The output image is an image corresponding to a person. For example, the output image at least represents a head including a face of the person.

[0369] For example, character data may also be translation data. In this method, voice data converted from character data is input to the neural network. Regardless of whether the metadata is character data or voice data, learning can be easily performed by inputting voice data to the neural network.

[0370] However, the speech data converted from the character data sometimes does not include non-language information (such as intonation). Therefore, in an environment using character data, the character data may be used in a neural network without being converted into speech data. The neural network in an environment using character data may also be different from the neural network in an environment not using character data.

[0371] In this case, the encoding device 100 may encode the character data into encoded data, and generate an output image representing at least a head including a human face through a neural network according to the character data.

[0372] The neural network used to generate the output image can also be Fig.26The Generative Adversarial Networks (GAN) shown in Figure 1. Alternatively, the neural network can also be Fig. 27 Alternatively, the neural network can also be Fig.28 The flow-based generative model shown.

[0373] Alternatively, the neural network may be any combination of two or more of the above three models. Alternatively, a neural network different from the above three models may be used.

[0374] In this method, an output image is generated via a neural network according to the voice data and the profile image. Thus, the characteristics included in the voice data are reflected in the output image. Therefore, sometimes the information expressed by a person can be reflected in the output image. Therefore, the information expressed by a person is visually easy to convey, which sometimes promotes smooth communication.

[0375] In addition, instead of inputting voice data from a person, character data may be input from a person. In this case, the same operation can be performed by converting the character data into voice data.

[0376] In addition, the operation of the decoding environment can also be performed in the encoding environment, and the operation of the encoding environment can also be performed in the decoding environment. In addition, the multiple components included in the encoding device 100 and the multiple components included in the decoding device 200 can also perform corresponding operations. In addition, all the components described in this embodiment are not necessarily required. Only a part of the components of this embodiment can also be installed.

[0377] [Installation example]

[0378] Fig.49 15 is a block diagram showing an implementation example of the coding device 100. The coding device 100 includes a circuit 151 and a memory 152. For example, the plurality of components of the coding device 100 described above are implemented via the circuit 151 and the memory 152.

[0379] The circuit 151 is a circuit for performing information processing and can access the memory 152. For example, the circuit 151 can be a dedicated circuit for executing the encoding method of the present disclosure, or a general-purpose circuit for executing a program corresponding to the encoding method of the present disclosure. In addition, the circuit 151 can also be a processor such as a CPU. Furthermore, the circuit 151 can also be a collection of multiple circuits.

[0380] The memory 152 is a dedicated or general memory for storing information used by the circuit 151 to encode an image. The memory 152 may be a circuit, or may be connected to the circuit 151. In addition, the memory 152 may be included in the circuit 151. In addition, the memory 152 may be a collection of multiple circuits. In addition, the memory 152 may be a magnetic disk or an optical disk, or may be a storage device or a recording medium. In addition, the memory 152 may be either a non-volatile memory or a volatile memory.

[0381] For example, the memory 152 may store encoding target data such as an image or encoding completed data such as a bit stream. In addition, the memory 152 may store a program for causing the circuit 151 to perform image processing. In addition, the memory 152 may store a neural network for causing the circuit 151 to perform inference processing. In addition, the memory 152 may store a profile image.

[0382] Fig.50 : is a flowchart showing a basic operation example of the coding device 100. Specifically, the circuit 151 of the coding device 100 performs the following processing during operation.

[0383] The circuit 151 encodes the performance data representing the information expressed by the person (S501). Next, the circuit 151 generates a person-corresponding image as an image corresponding to the person through a neural network according to the performance data and at least one profile image of the person (S502). Then, the circuit 151 outputs the person-corresponding image (S503).

[0384] Thus, in the decoding environment corresponding to the coding environment, the information expressed by the person can sometimes be reflected in the person corresponding image of the output object. Therefore, the information expressed by the person is easy to convey visually, which sometimes promotes smooth communication. As a result, it is possible to suppress the waste of time and resources. In addition, in the coding environment, the person corresponding image that is assumed to be output in the decoding environment corresponding to the coding environment can sometimes be output for confirmation.

[0385] For example, the expression data may include data derived from a person's image. Thus, the data derived from the person's image may be reflected in the person-corresponding image of the output object. Therefore, the person-corresponding image matching the person's image may be output.

[0386] In addition, for example, the performance data may also include human voice data. Thus, the human voice data may be reflected in the human corresponding image of the output object. Therefore, the human corresponding image matching the human voice data may be output.

[0387] In addition, for example, the at least one profile image may be a plurality of profile images. The circuit 151 may select one profile image from the plurality of profile images according to the performance data. Then, the circuit 151 may generate a person-corresponding image according to the one profile image through a neural network.

[0388] Thus, one profile image corresponding to the information expressed by the person can be selected from a plurality of profile images. In addition, a person-corresponding image of the output object can be generated based on one profile image corresponding to the information expressed by the person. Therefore, a person-corresponding image matching the information expressed by the person can be output.

[0389] In addition, for example, the performance data may also include an index representing a person's expression. Then, a plurality of profile images may also correspond to a plurality of expressions of a person. Thus, sometimes the person's expression can be reflected in the person-corresponding image of the output object. Therefore, sometimes a person-corresponding image matching the person's expression can be output. In addition, since the code amount of the index is assumed to be small, the code amount can sometimes be reduced.

[0390] In addition, for example, the circuit 151 may also derive the index via an expression recognition neural network according to an image of a person. Thus, an index corresponding to an expression accurately recognized from an image of a person may be derived. Therefore, an expression of a person may be accurately reflected in an image corresponding to the person of the output object.

[0391] In addition, for example, the circuit 151 may encode the performance data into each of the plurality of data regions of the bitstream. Thus, sometimes different conditions in each data region in the bitstream can be reflected in the image corresponding to the person of the output object. Therefore, sometimes the timely condition of the person can be reflected in the image corresponding to the person.

[0392] In addition, for example, the circuit 151 may encode the expression data in the header area of ​​the bit stream. In this way, a fixed condition in the bit stream may be reflected in the person-corresponding image to be output. In addition, the amount of code may be reduced.

[0393] In addition, for example, the performance data may also include data representing at least one of expression, facial orientation, movement of facial parts, and movement of the head. Thus, the expression of a person, etc. may be reflected in the person-corresponding image of the output object. Therefore, a person-corresponding image matching the expression of a person, etc. may be output.

[0394] In addition, for example, the expression data may include data expressed by coordinates. Thus, the expression data may be able to express the spatial position more precisely. Therefore, the information expressed by the person may be reflected more precisely in the person-corresponding image of the output object.

[0395] In addition, for example, circuit 151 may encode at least one profile image.

[0396] Thus, in a decoding environment corresponding to the encoding environment, even if the profile image is not stored in the memory 152, it is sometimes possible to use the profile image. Therefore, in the decoding environment, the memory capacity can sometimes be reduced. In addition, in decoding, the profile image determined in the encoding can sometimes be used. Therefore, the encoding operation and the decoding operation can sometimes be matched.

[0397] In addition, for example, the circuit 151 may encode the performance data into the first bit stream. In addition, the circuit 151 may encode at least one profile image into a second bit stream different from the first bit stream. Thus, the performance data and the profile image may be managed separately. In addition, the performance data and the profile image may be updated separately and efficiently.

[0398] In addition, for example, the circuit 151 may read at least one profile image from the memory 152. This may reduce the amount of processing compared to newly generating a profile image.

[0399] In addition, for example, at least one profile image may be one profile image. Circuit 151 may derive a first feature quantity set representing the movement of the mouth from the speech data. Then, circuit 151 may generate a person-corresponding image through a neural network according to the first feature quantity set and one profile image.

[0400] Thus, sometimes, based on the voice data associated with the movement of the mouth, it is possible to derive the first feature quantity set that accurately represents the movement of the mouth. In addition, sometimes, the movement of the mouth can be reflected in the person-corresponding image of the output object. Therefore, sometimes, it is possible to output a person-corresponding image that accurately reflects the information expressed by the person.

[0401] In addition, for example, at least one profile image may be one profile image. Circuit 151 may also derive a second feature quantity set representing the movement of the head or eyes by simulating the movement of the head or eyes. Then, circuit 151 generates a person-corresponding image through a neural network according to the voice data, the second feature quantity set, and one profile image.

[0402] As a result, the person-corresponding image to be outputted can be matched with the simulated head or eye movement, and thus, a more natural person-corresponding image without any sense of incongruity can be outputted.

[0403] In addition, for example, the circuit 151 may also match the expression in the person corresponding image with the expression estimated from the voice data. Thus, sometimes the expression corresponding to the voice data can be reflected in the person corresponding image of the output object. Therefore, sometimes the person corresponding image matching the person's expression can be output.

[0404] Alternatively, the coding device 100 may also include an input terminal, an entropy encoder, and an output terminal. In addition, the operation performed by the circuit 151 may also be performed by the entropy encoder. In addition, data used in the operation of the entropy encoder may be input to the input terminal. In addition, data obtained by the operation of the entropy encoder may also be output from the output terminal.

[0405] Fig.51 2 is a block diagram showing an implementation example of the decoding device 200. The decoding device 200 includes a circuit 251 and a memory 252. For example, the plurality of components of the decoding device 200 described above are implemented via the circuit 251 and the memory 252.

[0406] The circuit 251 is a circuit for performing information processing and can access the memory 252. For example, the circuit 251 can be a dedicated circuit for executing the decoding method of the present disclosure, or a general-purpose circuit for executing a program corresponding to the decoding method of the present disclosure. In addition, the circuit 251 can also be a processor such as a CPU. Furthermore, the circuit 251 can also be a collection of multiple circuits.

[0407] The memory 252 is a dedicated or general memory for storing information used by the circuit 251 to decode the image. The memory 252 may be a circuit, or may be connected to the circuit 251. In addition, the memory 252 may be included in the circuit 251. In addition, the memory 252 may be a collection of multiple circuits. In addition, the memory 252 may be a magnetic disk or an optical disk, or may be a storage device or a recording medium. In addition, the memory 252 may be either a non-volatile memory or a volatile memory.

[0408] For example, the memory 252 may store decoding target data such as a bit stream or decoded data such as an image. The memory 252 may store a program for causing the circuit 251 to perform image processing. The memory 252 may store a neural network for causing the circuit 251 to perform inference processing. The memory 252 may store a profile image.

[0409] Fig.52 2 is a flowchart showing a basic operation example of the decoding device 200. Specifically, the circuit 251 of the decoding device 200 performs the following processing during operation.

[0410] The circuit 251 decodes the performance data representing the information performed by the person (S601). Next, the circuit 251 generates a person-corresponding image as an image corresponding to the person through a neural network according to the performance data and at least one profile image of the person (S602). Then, the circuit 251 outputs the person-corresponding image (S603).

[0411] Thus, sometimes the information expressed by a person can be reflected in the person-corresponding image of the output object. Therefore, the information expressed by a person is visually easy to convey, and sometimes smooth communication is promoted. As a result, it is possible to suppress the waste of time and resources.

[0412] For example, the expression data may include data derived from a person's image. Thus, the data derived from the person's image may be reflected in the person-corresponding image of the output object. Therefore, the person-corresponding image matching the person's image may be output.

[0413] In addition, for example, the performance data may also include human voice data. Thus, the human voice data may be reflected in the human corresponding image of the output object. Therefore, the human corresponding image matching the human voice data may be output.

[0414] In addition, for example, the at least one profile image may be a plurality of profile images. The circuit 251 may select one profile image from the plurality of profile images according to the performance data. Then, the circuit 251 may generate a person-corresponding image according to the one profile image through a neural network.

[0415] Thus, one profile image corresponding to the information expressed by the person can be selected from a plurality of profile images. In addition, a person-corresponding image of the output object can be generated based on one profile image corresponding to the information expressed by the person. Therefore, a person-corresponding image matching the information expressed by the person can be output.

[0416] In addition, for example, the performance data may also include an index representing a person's expression. Then, a plurality of profile images may also correspond to a plurality of expressions of a person. Thus, sometimes the person's expression can be reflected in the person-corresponding image of the output object. Therefore, sometimes a person-corresponding image matching the person's expression can be output. In addition, since the code amount of the index is assumed to be small, the code amount can sometimes be reduced.

[0417] In addition, for example, the circuit 251 may decode the performance data from each of the plurality of data regions in the bit stream. Thus, sometimes different conditions in each data region in the bit stream can be reflected in the image corresponding to the person of the output object. Therefore, sometimes the timely condition of the person can be reflected in the image corresponding to the person.

[0418] In addition, for example, the circuit 251 may decode the representation data from the header region of the bit stream. In this way, a fixed condition in the bit stream may be reflected in the person-corresponding image to be output. In addition, the amount of code may be reduced.

[0419] In addition, for example, the performance data may also include data representing at least one of expression, facial orientation, movement of facial parts, and movement of the head. Thus, the expression of a person, etc. may be reflected in the person-corresponding image of the output object. Therefore, a person-corresponding image matching the expression of a person, etc. may be output.

[0420] In addition, for example, the expression data may include data expressed by coordinates. Thus, the expression data may be able to express the spatial position more precisely. Therefore, the information expressed by the person may be reflected more precisely in the person-corresponding image of the output object.

[0421] In addition, for example, circuit 251 may decode at least one profile image.

[0422] Thus, even if the profile image is not stored in the memory 252, the profile image can be used in some cases. Therefore, the memory capacity can be reduced in some cases. In addition, the profile image determined in the encoding can be used in the decoding in some cases. Therefore, the encoding action and the decoding action can be matched in some cases.

[0423] In addition, for example, the circuit 251 may decode the performance data from the first bit stream. In addition, the circuit 251 may decode at least one profile image from a second bit stream different from the first bit stream. Thus, the performance data and the profile image may be managed separately. In addition, the performance data and the profile image may be updated separately and efficiently.

[0424] In addition, for example, the circuit 251 may read the at least one profile image from the memory 252 .

[0425] This may omit the decoding of the profile image, thereby reducing the amount of processing and the amount of code.

[0426] In addition, for example, the at least one profile image may be one profile image. The circuit 251 may derive a first feature value set representing the movement of the mouth from the speech data. Then, the circuit 251 may generate a person-corresponding image through a neural network according to the first feature value set and one profile image.

[0427] Thus, sometimes, based on the voice data associated with the movement of the mouth, it is possible to derive the first feature quantity set that accurately represents the movement of the mouth. In addition, sometimes, the movement of the mouth can be reflected in the person-corresponding image of the output object. Therefore, sometimes, it is possible to output a person-corresponding image that accurately reflects the information expressed by the person.

[0428] In addition, for example, the at least one profile image may be one profile image. The circuit 251 may also derive a second feature quantity set representing the movement of the head or the eyes by simulating the movement of the head or the eyes. Then, the circuit 251 may also generate a person-corresponding image through a neural network according to the voice data, the second feature quantity set, and one profile image.

[0429] As a result, the person-corresponding image to be outputted can be matched with the simulated head or eye movement, and thus, a more natural person-corresponding image without any sense of incongruity can be outputted.

[0430] In addition, for example, the circuit 251 may also match the expression in the person corresponding image with the expression estimated from the voice data. Thus, the expression corresponding to the voice data may be reflected in the person corresponding image of the output object. Therefore, the person corresponding image matching the person's expression may be output.

[0431] Alternatively, the decoding device 200 may also include an input terminal, an entropy decoder, and an output terminal. In addition, the operation performed by the circuit 251 may also be performed by the entropy decoder. In addition, data used in the operation of the entropy decoder may also be input to the input terminal. In addition, data obtained by the operation of the entropy decoder may also be output from the output terminal.

[0432] [Other examples]

[0433] The encoding device 100 and the decoding device 200 in the above-mentioned examples can also be used as an image encoding device and an image decoding device, respectively, and can also be used as a dynamic image encoding device and a dynamic image decoding device. In addition, the multiple components included in the encoding device 100 and the multiple components included in the decoding device 200 can also perform corresponding operations.

[0434] In addition, the expression of encoding can be replaced by expressions such as storing, including, writing, describing, signaling, sending, notifying or saving, and these expressions can also be replaced with each other. For example, encoding information can also mean including the information in a bit stream. In addition, encoding information into a bit stream can also mean encoding the information and generating a bit stream including the encoded information.

[0435] In addition, the expression of decoding can be replaced by expressions such as reading, interpreting, reading, reading in, deriving, obtaining, accepting, extracting or restoring, and these expressions can also be replaced with each other. For example, decoding information can also be obtained from a bit stream. In addition, decoding information from a bit stream can also mean decoding a bit stream and obtaining information included in the bit stream.

[0436] In addition, at least a part of the above-mentioned examples may be used as an encoding method, may be used as a decoding method, may be used as an entropy encoding method, may be used as an entropy decoding method, or may be used as other methods.

[0437] In addition, each component may be formed by dedicated hardware or implemented by executing a software program suitable for each component. Each component may also be implemented by a program execution unit such as a CPU or a processor reading and executing a software program recorded in a recording medium such as a hard disk or a semiconductor memory.

[0438] Specifically, the encoding device 100 and the decoding device 200 may each include a processing circuit and a storage device electrically connected to the processing circuit and accessible from the processing circuit. For example, the processing circuit corresponds to the circuit 151 or 251, and the storage device corresponds to the memory 152 or 252.

[0439] The processing circuit includes at least one of dedicated hardware and a program execution unit, and uses the storage device to execute the processing. In addition, when the processing circuit includes the program execution unit, the storage device stores a software program for the program execution unit to execute.

[0440] An example of the above-mentioned software program is a bit stream. The bit stream includes a coded image and a syntax for performing a decoding process for decoding the image. The bit stream causes the decoding device 200 to perform a process based on the syntax, thereby causing the decoding device 200 to decode the image. In addition, for example, the software for realizing the above-mentioned coding device 100 or decoding device 200 is the following program.

[0441] For example, the program may also cause a computer to execute a coding method, which includes: a step of encoding performance data representing information performed by a person; a step of generating a person-corresponding image as an image corresponding to the person via a neural network based on the performance data and at least one profile image of the person; and a step of outputting the person-corresponding image.

[0442] In addition, for example, the program can also cause a computer to execute a decoding method, which includes: a step of decoding performance data representing information performed by a person; a step of generating a person-corresponding image as an image corresponding to the person via a neural network based on the performance data and at least one profile image of the person; and a step of outputting the person-corresponding image.

[0443] In addition, each of the above-mentioned components may also be a circuit. These circuits may constitute one circuit as a whole, or may be separate circuits. In addition, each component may be implemented by a general-purpose processor or a dedicated processor.

[0444] In addition, the processing performed by a specific component may also be performed by other components. In addition, the order of executing the processing may be changed, and multiple processing may be performed in parallel. In addition, any two or more of the multiple examples disclosed in the present invention may also be appropriately combined and implemented. In addition, the encoding and decoding device may also include an encoding device 100 and a decoding device 200.

[0445] In addition, the ordinal numbers 1 and 2 used in the description may be replaced and added as appropriate. In addition, for the constituent elements, ordinal numbers may be newly assigned or removed. In addition, these ordinal numbers are sometimes added to the elements in order to identify the elements, and sometimes do not correspond to a meaningful order.

[0446] In the above, the modes of the encoding device 100 and the decoding device 200 are described based on multiple examples, but the modes of the encoding device 100 and the decoding device 200 are not limited to these examples. As long as it does not depart from the gist of the present disclosure, modes obtained by applying various modifications that can be conceived by those skilled in the art to each example, and modes constructed by combining constituent elements in different examples can all be included in the scope of the modes of the encoding device 100 and the decoding device 200.

[0447] It is also possible to combine and implement one or more methods disclosed herein with at least a portion of other methods in the present disclosure. In addition, it is also possible to combine and implement a portion of the processing, a portion of the device structure, a portion of the syntax, etc. recorded in the flowchart of one or more methods disclosed herein with other methods.

[0448] [Implementation and Application]

[0449] In each of the above embodiments, each functional or action module can usually be implemented by an MPU (micro processing unit) and a memory. In addition, the processing performed by each functional module can also be implemented as a program execution unit of a processor that reads and executes the software (program) recorded in a recording medium such as a ROM. The software can also be distributed. The software can also be recorded in various recording media such as semiconductor memories. In addition, each functional module can also be implemented by hardware (dedicated circuit).

[0450] The processing described in each embodiment can be implemented by centralized processing using a single device (system), or can be implemented by distributed processing using multiple devices. In addition, the processor that executes the above program can be single or multiple. That is, centralized processing or distributed processing can be performed.

[0451] The aspects of the present disclosure are not limited to the above-described embodiments, and various modifications can be made, and these modifications are also included in the scope of the aspects of the present disclosure.

[0452] Furthermore, application examples of the dynamic image encoding method (image encoding method) or dynamic image decoding method (image decoding method) shown in each of the above embodiments and various systems implementing the application examples are described here. Such a system may also be characterized by having an image encoding device using the image encoding method, an image decoding device using the image decoding method, or an image encoding and decoding device having both. Other configurations of such a system can be appropriately changed according to circumstances.

[0453] [Example of use]

[0454] Fig.53 The figure shows the overall structure of a content providing system ex100 suitable for realizing content distribution services. The area for providing communication services is divided into cells of desired sizes, and base stations ex106, ex107, ex108, ex109, and ex110, which are fixed wireless stations in the example shown in the figure, are installed in each cell.

[0455] In the content supply system ex100, devices such as a computer ex111, a game machine ex112, a camera ex113, a home appliance ex114, and a smartphone ex115 are connected to the Internet ex101 via an Internet service provider ex102 or a communication network ex104, and base stations ex106 to ex110. The content supply system ex100 may also connect any of the above devices in combination. In various embodiments, the devices may be directly or indirectly connected to each other via a telephone network or short-range wireless without going through the base stations ex106 to ex110. Furthermore, the streaming server ex103 may be connected to the devices such as the computer ex111, the game machine ex112, the camera ex113, the home appliance ex114, and the smartphone ex115 via the Internet ex101. In addition, the streaming server ex103 may be connected to a terminal in a hot spot in an airplane ex117 via a satellite ex116.

[0456] Alternatively, wireless access points or hotspots may be used instead of the base stations ex106 to ex110. Furthermore, the streaming server ex103 may be directly connected to the communication network ex104 without going through the Internet ex101 or the Internet service provider ex102, or directly connected to the aircraft ex117 without going through the satellite ex116.

[0457] The camera ex113 is a device such as a digital camera capable of taking still and moving images. The smartphone ex115 is a smartphone, a mobile phone, or a PHS (Personal Handyphone System) that supports 2G, 3G, 3.9G, 4G, and a mobile communication system to be called 5G in the future.

[0458] The household appliance ex114 is a refrigerator or a device included in a household fuel cell cogeneration system.

[0459] In the content supply system ex100, a terminal having a camera function is connected to the streaming server ex103 via the base station ex106, etc., so that real-time distribution (live broadcast) etc. can be performed. In real-time distribution, a terminal (computer ex111, game machine ex112, camera ex113, home appliance ex114, smartphone ex115, terminal in airplane ex117, etc.) can perform the encoding process described in the above-mentioned embodiments on the still image or moving image content photographed by the user using the terminal, multiplex the image data obtained by encoding and the sound data obtained by encoding the sound corresponding to the image, and transmit the obtained data to the streaming server ex103. That is, each terminal functions as an image encoding device according to one embodiment of the present disclosure.

[0460] On the other hand, the streaming server ex103 streams the content data sent to the client that has issued the request. The client is a computer ex111, a game console ex112, a camera ex113, a home appliance ex114, a smartphone ex115, or a terminal in an airplane ex117, etc., which can decode the coded data. Each device that has received the distributed data decodes and reproduces the received data. That is, each device can also function as an image decoding device according to one embodiment of the present disclosure.

[0461] [Distributed Processing]

[0462] In addition, the streaming server ex103 can also be a plurality of servers or a plurality of computers, which process, record or distribute the data in a distributed manner. For example, the streaming server ex103 can also be implemented by a CDN (Content Delivery Network), which realizes content distribution by connecting a plurality of edge servers distributed all over the world to the network between the edge servers. In the CDN, physically close edge servers are dynamically allocated according to the client. In addition, by caching and distributing content to the edge server, delay can be reduced. In addition, in the case of several types of errors or when the communication state changes due to an increase in traffic, etc., the processing can be dispersed by a plurality of edge servers, or the distribution subject can be switched to another edge server, or the portion of the network where the failure occurred can be bypassed and distribution can be continued, so that high-speed and stable distribution can be achieved.

[0463] In addition, the distributed processing is not limited to the distribution itself. The encoding processing for the photographic data can be performed in each terminal, can be performed on the server side, or can be shared. As an example, two processing cycles are generally performed in the encoding process. In the first cycle, the complexity or code volume of the image in units of frames or scenes is detected. In addition, in the second cycle, processing is performed to maintain the image quality and improve the encoding efficiency. For example, by performing the first encoding process by the terminal and the second encoding process by the server side that has received the content, the processing load in each terminal can be reduced and the quality and efficiency of the content can be improved. In this case, if there is a requirement for receiving and decoding in approximately real time, the first encoding completed data performed by the terminal can also be received and reproduced by other terminals, so more flexible real-time distribution can also be performed.

[0464] As another example, the camera ex113 or the like extracts feature quantities from an image, compresses data related to the feature quantities as metadata, and transmits the data to the server. The server, for example, determines the importance of an object based on the feature quantities and switches the quantization accuracy, thereby performing compression in accordance with the meaning of the image (or the importance of the content). The feature quantity data is particularly effective in improving the accuracy and efficiency of motion vector prediction when the image is recompressed in the server. Alternatively, a simple encoding such as VLC (variable length coding) may be performed in the terminal, while a encoding with a heavy processing load such as CABAC (context adaptive binary arithmetic coding) may be performed in the server.

[0465] As another example, in a stadium, shopping mall, or factory, there are sometimes multiple image data obtained by multiple terminals taking photos of roughly the same scene. In this case, multiple terminals that took photos are used, and other terminals and servers that did not take photos are used as needed, for example, in units of GOP (Group of Picture), in units of pictures, or in units of tiles obtained by dividing pictures, etc., and encoding processing is allocated and distributed processing is performed. This can reduce latency and achieve better real-time performance.

[0466] Since the plurality of image data are substantially the same scene, the server can also manage and / or instruct the image data photographed in each terminal to be mutually referenced. In addition, the server can also receive the encoded data from each terminal, change the reference relationship between the plurality of data, or correct or replace the image itself and re-encode it. In this way, a stream with improved quality and efficiency of each data can be generated.

[0467] Furthermore, the server may distribute the video data after performing transcoding to change the encoding method of the video data. For example, the server may convert the encoding method of the MPEG type to the VP type (such as VP9) or convert H.264 to H.265.

[0468] As such, the encoding process can be performed by a terminal or one or more servers. Therefore, although "server" or "terminal" is used as the subject of the processing below, part or all of the processing performed by the server can also be performed by the terminal, and part or all of the processing performed by the terminal can also be performed by the server. In addition, in these aspects, the decoding process is also the same.

[0469] [3D, multi-angle]

[0470] There are increasing cases where images or videos taken by terminals such as multiple cameras ex113 and / or smartphone ex115 that are substantially synchronized with each other, or the same scene taken from different angles, are integrated and used. The images taken by the terminals are integrated based on the relative positional relationship between the terminals obtained separately, or the area where the feature points included in the images are consistent.

[0471] The server may encode not only the 2D dynamic image, but also the static image automatically or at the time specified by the user based on the scene analysis of the dynamic image and send it to the receiving terminal. The server can further generate the 3D shape of the scene based not only on the 2D dynamic image but also on the images obtained by photographing the same scene from different angles, if the relative position relationship between the photographing terminals can be obtained. The server may separately encode the 3D data generated by the point cloud, etc., or select or reconstruct the images photographed by multiple terminals to generate the images to be sent to the receiving terminal based on the results of identifying or tracking people or objects using the 3D data.

[0472] In this way, the user can arbitrarily select each image corresponding to each camera terminal to experience the scene, and can also experience the content obtained by cutting out the image of the selected viewpoint from the 3D data reconstructed using multiple images or images. Furthermore, it is also possible to collect sound from multiple different angles together with the image, and the server multiplexes the sound from a specific angle or space with the corresponding image and sends the multiplexed image and sound.

[0473] In addition, in recent years, content that establishes a correspondence between the real world and the virtual world, such as Virtual Reality (VR) and Augmented Reality (AR), has become increasingly popular. In the case of VR images, the server can either produce viewpoint images for the right eye and the left eye respectively, and perform encoding that allows reference between viewpoint images through multi-view coding (Multi-View Coding (MVC)), or encode them as different streams without mutual reference. When decoding different streams, they can be reproduced synchronously with each other to reproduce the virtual 3D space corresponding to the user's viewpoint.

[0474] In the case of AR images, the server superimposes virtual object information in the virtual space on the camera information in the real space based on the 3D position or the movement of the user's viewpoint. The decoding device may also obtain or maintain virtual object information and 3D data, generate a 2D image corresponding to the movement of the user's viewpoint, and create overlapping data by smoothly connecting. Alternatively, the decoding device may also send the movement of the user's viewpoint to the server in addition to the entrustment of the virtual object information. The server may also create overlapping data corresponding to the movement of the received viewpoint based on the 3D data maintained by the server, encode the overlapping data and distribute it to the decoding device. In addition, the overlapping data may have an α value indicating the transmittance in addition to RGB, and the server encodes the portion other than the object created based on the 3D data by setting the α value of the portion other than the object to 0, etc. to make the portion transparent. Alternatively, the server may also set the RGB value of a specified value as the background, as in the case of a chroma key, to generate data that sets the portion other than the object as the background color.

[0475] Similarly, the decoding process of the distributed data can be performed on each terminal as a client, on the server side, or shared. As an example, a terminal can also send a receiving request to the server, and other terminals receive the content corresponding to the request and perform decoding processing, and the decoded signal is sent to the device with a display. By not relying on the performance of the terminal itself that can communicate, distributed processing and selecting appropriate content can reproduce data with good picture quality. In addition, as another example, larger image data can also be received by TV, etc., and a portion of the area such as tiles divided from the picture can be decoded and displayed on the viewer's personal terminal. In this way, the overall situation can be shared, and the area of ​​one's own responsibility or the area that one wants to confirm in more detail can be confirmed at hand.

[0476] In a situation where multiple short-range, medium-range or long-range wireless communications can be used indoors and outdoors, it is possible to receive content seamlessly using distribution system specifications such as MPEG-DASH (Dynamic Adaptive Streaming over HTTP). Users can also freely select their terminals, decoding devices or display devices such as displays configured indoors and outdoors, and switch them in real time. In addition, the terminal for decoding and the terminal for display can be switched and decoded using their own location information, etc. As a result, while the user is moving to the destination, information can be mapped and displayed on a part of the wall or ground of a neighboring building in which a device capable of display is embedded. In addition, the bit rate of the received data can also be switched based on the ease of accessing the encoded data on the network, and the encoded data on the network includes: encoded data cached in a server that can access the encoded data from the receiving terminal in a short time, or encoded data copied in an edge server in a content distribution service, etc.

[0477] [Web page optimization]

[0478] Fig.54 This is a diagram showing an example of a display screen of a web page on the computer ex111 or the like. Fig.55 FIG. 4 is a diagram showing an example of a display screen of a web page on a smartphone ex115 or the like. Fig.54 and Fig.55 As shown, a web page sometimes includes a plurality of link images that are links to image content, and the display method thereof differs depending on the device being browsed. When a plurality of link images can be viewed on the screen, until the user explicitly selects a link image, or until the link image approaches the center of the screen or the entire link image enters the screen, the display device (decoding device) may display a still image or I picture of each content as a link image, or may display an image such as a GIF animation through a plurality of still images or I pictures, or may receive only the base layer and decode and display the image.

[0479] When a linked image is selected by the user, the display device gives top priority to decoding the base layer. In addition, if there is information indicating that it is adaptive content in the HTML (HyperText Markup Language) that constitutes the web page, the display device can also decode up to the enhancement layer. Furthermore, in order to ensure real-time performance, before selection or when the communication band is very strict, the display device can reduce the delay between the decoding time and the display time of the front-end picture (the delay from the start of decoding to the start of display of the content) by decoding and displaying only the pictures that refer to the front (I pictures, P pictures, and B pictures that refer only to the front). Furthermore, the display device can also intentionally ignore the reference relationship of the pictures, so that all B pictures and P pictures refer to the front and are roughly decoded, and normal decoding is performed as time passes and the number of received pictures increases.

[0480] [Automatic driving]

[0481] In addition, when still images or video data such as 2D or 3D map information are transmitted and received for the purpose of automatic driving or driving assistance of a vehicle, the receiving terminal may receive weather or construction information as meta-information in addition to image data belonging to one or more layers, and establish correspondence between them and decode them. Furthermore, the meta-information may belong to a layer or may simply be multiplexed with the image data.

[0482] In this case, since the vehicle, drone or airplane including the receiving terminal moves, the receiving terminal can switch base stations ex106 to ex110 and perform reception and decoding seamlessly by transmitting the location information of the receiving terminal. In addition, the receiving terminal can dynamically switch the degree of receiving meta information or the degree of updating map information according to the user's selection, the user's condition and / or the state of the communication band.

[0483] In the content providing system ex100, the client can receive, decode, and reproduce the encoded information sent by the user in real time.

[0484] [Distribution of Personal Content]

[0485] In addition, in the content supply system ex100, not only can long-term content be distributed by image distribution practitioners with high image quality, but also short-term content can be distributed by individuals with low image quality by unicast or multicast. It is expected that such personal content will continue to increase in the future. In order to make personal content of higher quality, the server may perform encoding processing after editing. This can be achieved, for example, using the following configuration.

[0486] In real time during photography, or after accumulating and photographing, the server performs recognition processing such as photography errors, scene exploration, meaning analysis and object detection based on the original picture data or the encoded data. Then, based on the recognition results, the server manually or automatically corrects focus deviation or hand shaking, deletes scenes of low importance such as scenes with lower brightness than other pictures or out of focus, emphasizes the edges of objects or changes the color tone, etc. The server encodes the edited data based on the editing results. In addition, it is also known that if the shooting time is too long, the viewing rate will decrease. The server can also automatically cut off not only the scenes of low importance as mentioned above but also the scenes with less dynamics based on the shooting time and the image processing results, so that the content is within a specific time range. Alternatively, the server can also generate a summary based on the results of the scene's meaning analysis and encode it.

[0487] In personal content, there are also situations where, if no changes are made, the content that infringes on copyright, the author's personality rights or portrait rights may appear in it, and there are also situations that are not desirable for individuals, such as the scope of sharing exceeds the intended scope. Therefore, for example, the server can also be changed to intentionally not focus on the faces of people in the peripheral part of the screen or images of the home, etc. and encode them. Furthermore, the server can also identify whether the face of a person different from the pre-registered person is reflected in the encoded object image, and if it is reflected, mosaic or other processing is applied to the facial part. Alternatively, as a pre-processing or post-processing of the encoding, the user can also specify the person or background area that he wants to be processed in the image from the perspective of copyright, etc. The server can also replace the designated area with other images or blur the focus. If it is a person, the person can be tracked in the dynamic image and the image of part of the person's face can be replaced.

[0488] Since the viewing of personal content with a small amount of data has a strong demand for real-time performance, the decoding device first gives top priority to receiving the base layer, decoding it, and reproducing it, although it also depends on the bandwidth. The decoding device can also receive the enhancement layer during this period, and in the case of loop reproduction or reproduction more than twice, it can reproduce high-quality images including the enhancement layer. In this way, if the stream is adaptively encoded, the following experience can be provided: although it is a rough dynamic image when it is not selected or at the beginning of viewing, the stream gradually becomes refined and the image becomes better. In addition to adaptive encoding, if the rough stream reproduced for the first time and the second stream encoded with reference to the dynamic image of the first time are composed of one stream, the same experience can be provided.

[0489] [Other application examples]

[0490] The above encoding or decoding processing is generally performed in an LSI ex500 included in each terminal. Fig.53 ) may be composed of a single chip or a plurality of chips. In addition, software for encoding or decoding moving images may be embedded in any recording medium (CD-ROM, floppy disk, hard disk, etc.) that can be read by the computer ex111 or the like, and the encoding or decoding process may be performed using the software. Furthermore, if the smartphone ex115 is equipped with a camera, moving image data acquired by the camera may be transmitted. In this case, the moving image data is data encoded by the LSI ex500 of the smartphone ex115.

[0491] In addition, LSIex500 can also be configured to download application software and activate it. In this case, the terminal first determines whether the terminal supports the encoding method of the content or has the ability to execute a specific service. If the terminal does not support the encoding method of the content or does not have the ability to execute a specific service, the terminal downloads the codec or application software, and then obtains and reproduces the content.

[0492] Furthermore, not limited to the content supply system ex100 via the Internet ex101, at least one of the video encoding device (video encoding device) or video decoding device (video decoding device) in the above-mentioned embodiments can be incorporated into a digital broadcasting system. Since multiplexed data obtained by multiplexing video and audio is transmitted and received over broadcasting radio waves using a satellite or the like, the content supply system ex100 is suitable for multicasting compared to the unicast-friendly configuration of the content supply system ex100, but the same application can be made to the encoding and decoding processes.

[0493] [Hardware Configuration]

[0494] Fig.56 is a more detailed representation Fig.53 FIG. 115 is a diagram of a smartphone ex115 shown in FIG. Fig.571 is a diagram showing an example of the configuration of a smartphone ex115. The smartphone ex115 includes an antenna ex450 for transmitting and receiving radio waves with the base station ex110, a camera unit ex465 capable of shooting video and still images, and a display unit ex458 for displaying decoded data such as the video shot by the camera unit ex465 and the video received by the antenna ex450. The smartphone ex115 further includes an operation unit ex466 such as a touch panel, a voice output unit ex457 such as a speaker for outputting voice or sound, a voice input unit ex456 such as a microphone for inputting voice, a memory unit ex467 capable of storing coded data or decoded data such as shot video or still images, recorded voice, received video or still images, and mails, and a slot unit ex464 as an interface unit with a SIM (Subscriber Identity Module) ex468 for identifying a user and authenticating access to various data including the network. In addition, an external memory can also be used to replace the memory unit ex467.

[0495] The main control unit ex460, which controls the display unit ex458 and the operation unit ex466, is connected to the power circuit unit ex461, the operation input control unit ex462, the image signal processing unit ex455, the camera interface unit ex463, the display control unit ex459, the modulation / demodulation unit ex452, the multiplexing / separation unit ex453, the audio signal processing unit ex454, the slot unit ex464 and the memory unit ex467 via the synchronous bus ex470.

[0496] When the power key is turned on by a user's operation, the power circuit unit ex461 activates the smartphone ex115 to be operable and supplies power from the battery pack to each unit.

[0497] The smartphone ex115 performs processes such as calls and data communications under the control of a main control unit ex460 including a CPU, ROM, and RAM. During a call, the voice signal received by the voice input unit ex456 is converted into a digital voice signal by the voice signal processing unit ex454, the modulation / demodulation unit ex452 performs spectrum spreading, the transmission / reception unit ex451 performs digital-to-analog conversion and frequency conversion, and the resulting signal is transmitted via the antenna ex450. In addition, the received data is amplified, frequency converted and analog-to-digital converted, the modulation / demodulation unit ex452 performs inverse spectrum spreading, the voice signal processing unit ex454 converts the received data into an analog voice signal, and the analog-to-digital voice signal is output from the voice output unit ex457. In the data communication mode, text, still image, or video data is transmitted to the main control unit ex460 via the operation input control unit ex462 based on the operation of the operation unit ex466 of the main unit. The same transmission and reception processes are performed. When transmitting video, still images, or video and audio in the data communication mode, the video signal processing unit ex455 compresses and encodes the video signal stored in the memory unit ex467 or the video signal input from the camera unit ex465 by the video encoding method described in the above embodiments, and sends the encoded video data to the multiplexing / demultiplexing unit ex453. The audio signal processing unit ex454 encodes the audio signal received by the audio input unit ex456 when the camera unit ex465 takes a video or still image, and sends the encoded audio data to the multiplexing / demultiplexing unit ex453. The multiplexing / demultiplexing unit ex453 multiplexes the encoded video data and the encoded audio data in a predetermined manner, and the modulation / demodulation unit (modulation / demodulation circuit unit) ex452 and the transmission / reception unit ex451 perform modulation processing and conversion processing, and transmit the data via the antenna ex450.

[0498] When receiving an image attached to an e-mail or chat, or an image linked to a web page, in order to decode the multiplexed data received via the antenna ex450, the multiplexing / demultiplexing unit ex453 separates the multiplexed data into a bit stream of image data and a bit stream of voice data, and supplies the encoded image data to the image signal processing unit ex455 via the synchronous bus ex470, and supplies the encoded voice data to the voice signal processing unit ex454. The image signal processing unit ex455 decodes the image signal using a moving image decoding method corresponding to the moving image encoding method shown in the above-mentioned embodiments, and displays the image or still image included in the linked moving image file on the display unit ex458 via the display control unit ex459. The voice signal processing unit ex454 decodes the voice signal and outputs the voice from the voice output unit ex457. As real-time streaming becomes more and more popular, there are cases where the reproduction of voice is socially inappropriate depending on the user's situation. Therefore, it is also possible to configure that, as an initial value, the audio signal is not reproduced but only the video data is reproduced, and the audio is reproduced synchronously only when the user performs an operation such as clicking on the video data.

[0499] In this description, the smartphone ex115 is used as an example. However, as a terminal, in addition to a transmitting and receiving terminal having both an encoder and a decoder, a transmitting terminal having only an encoder and a receiving terminal having only a decoder are conceivable. In the digital broadcasting system, it is described that multiplexed data is received or transmitted by multiplexing voice data with video data. However, in addition to voice data, character data related to the video may also be multiplexed in the multiplexed data. In addition, it is also possible to receive or transmit the video data itself instead of the multiplexed data.

[0500] In addition, it is described that the encoding or decoding process is controlled by the main control unit ex460 including the CPU, but various terminals are mostly equipped with a GPU (Graphics Processing Unit). Therefore, through a memory common to the CPU and the GPU, or a memory that manages addresses in a manner that can be used in common, it is also possible to flexibly utilize the performance of the GPU to process a wide range at once. As a result, it is possible to shorten the encoding time, ensure real-time performance, and achieve low latency. In particular, it is more efficient to perform motion exploration, deblocking filters, SAO (Sample Adaptive Offset), and conversion / quantization processing on a picture basis instead of on the CPU, and on the GPU.

[0501] Industrial Applicability

[0502] The present disclosure can be used in a coding device and a decoding device for visually transmitting information, and can be applied to a video conferencing system, etc.

[0503] Description of reference numerals:

[0504] 100 Encoding device

[0505] 101 Data Encoder

[0506] 102, 202 Image Selector

[0507] 103, 203 Image Generator

[0508] 104, 204 Voice Exporter

[0509] 105, 205 Feature Exporter

[0510] 106, 206 simulator

[0511] 151, 251 circuits

[0512] 152, 252 memory

[0513] 200 Decoding device

[0514] 201 Data Decoder

[0515] 300 Video Conference System

[0516] 400 Network

Claims

1. A decoding device, comprising: Circuits; and a memory connected to the circuit, The circuit is in action. decoding the performance data representing the information performed by the person, generating a person-corresponding image as an image corresponding to the person via a neural network based on the performance data and at least one profile image of the person, Output the image corresponding to the person.

2. The decoding device according to claim 1, The performance data includes data derived from an image of the person.

3. The decoding device according to claim 1 or 2, The performance data includes voice data of the person.

4. The decoding device according to claim 1 or 2, The at least one profile image is a plurality of profile images, The circuit, selecting one profile image from among the plurality of profile images according to the performance data, According to the one profile image, the person corresponding image is generated via the neural network.

5. The decoding device according to claim 4, The performance data includes an index representing the expression of the person, The plurality of profile images correspond to a plurality of expressions of the person.

6. The decoding device according to claim 1 or 2, The circuit decodes the presentation data from respective ones of a plurality of data regions of a bitstream.

7. The decoding device according to claim 1 or 2, The circuit decodes the presentation data from a header region of a bitstream.

8. The decoding device according to claim 1 or 2, The expression data includes data representing at least one of facial expression, facial orientation, movement of a facial part, and movement of the head.

9. The decoding device according to claim 1 or 2, The representation data includes data represented by coordinates.

10. The decoding device according to claim 1 or 2, The circuit decodes the at least one profile image.

11. The decoding device according to claim 1 or 2, The circuit, decoding the representation data from a first bit stream, The at least one profile picture is decoded from a second bit stream different from the first bit stream.

12. The decoding device according to claim 1 or 2, The circuit reads the at least one profile image from the memory.

13. The decoding device according to claim 3, The at least one profile image is one profile image, The circuit, deriving a first feature quantity set representing the movement of the mouth from the speech data, The person-corresponding image is generated via the neural network according to the first feature value set and the one profile image.

14. The decoding device according to claim 3, The at least one profile image is one profile image, The circuit, Simulating the movement of the head or the eyes to derive a second feature quantity set representing the movement of the head or the eyes, The person-corresponding image is generated via the neural network according to the voice data, the second feature value set and the one profile image.

15. The decoding device according to claim 3, The circuit matches the expression of the person in the corresponding image with the expression inferred from the voice data.

16. An encoding device comprising: Circuits; and a memory connected to the circuit, The circuit is in action. encoding performance data representing information performed by a person, generating a person-corresponding image as an image corresponding to the person via a neural network based on the performance data and at least one profile image of the person, Output the image corresponding to the person.

17. The encoding device according to claim 16, The performance data includes data derived from an image of the person.

18. The encoding device according to claim 16 or 17, The performance data includes voice data of the person.

19. The encoding device according to claim 16 or 17, The at least one profile image is a plurality of profile images, The circuit, selecting one profile image from among the plurality of profile images according to the performance data, According to the one profile image, the person corresponding image is generated via the neural network.

20. The encoding device according to claim 19, The performance data includes an index representing the expression of the person, The plurality of profile images correspond to a plurality of expressions of the person.

21. A decoding method, comprising: The step of decoding the performance data representing the information performed by the person; A step of generating a person-corresponding image as an image corresponding to the person via a neural network based on the performance data and at least one profile image of the person; as well as The step of outputting an image corresponding to the person.

22. A coding method, comprising: The step of encoding performance data representing information performed by a person; A step of generating a person-corresponding image as an image corresponding to the person via a neural network based on the performance data and at least one profile image of the person; as well as The step of outputting an image corresponding to the person.