Encoding apparatus, decoding apparatus, encoding method, and decoding method

Abstract

The encoding device includes a circuit and a memory connected to the circuit. During operation, the circuit obtains first information (S301) containing multiple feature quantities representing a time dimension contained in the model, configures the first information on one or more images (S302), and encodes the one or more images to generate a bit stream (S303). For example, the first information may contain multiple data, and the bit stream may contain second information indicating the configuration order of the multiple data.

Description

Technical Field

[0001] This disclosure relates to encoding devices, decoding devices, encoding methods, and decoding methods. Background Technology

[0002] Various methods are known for encoding three-dimensional data. For example, there are methods for encoding models representing three-dimensional data. In addition, methods for representing dynamic models are known (see, for example, Non-Patent Literature 1).

[0003] Existing technical documents Non-patent literature Non-patent literature 1: "K-Planes: Explicit Radiance Fields in Space, Time, and Appearance", https: / / arxiv.org / pdf / 2301.10241 Summary of the Invention

[0004] The problem that the invention aims to solve Regarding the encoding methods described above, in order to improve encoding efficiency, improve image quality, reduce processing load, reduce circuit size, or appropriately select elements or actions such as filters, blocks, sizes, motion vectors, reference images or reference blocks, we hope to propose new methods.

[0005] This disclosure provides, for example, a structure or method that can help improve one or more of the following: improve encoding efficiency, improve image quality, reduce processing load, reduce circuit size, improve processing speed, and appropriately select elements or actions. Additionally, this disclosure may include structures or methods that can contribute to benefits other than those described above.

[0006] Methods for solving problems For example, one aspect of the present disclosure involves an encoding device comprising a circuit and a memory connected to the circuit, wherein the circuit, in operation, acquires first information contained in a model, the first information representing multiple feature quantities having a time dimension, configures the first information on one or more images, and generates a bitstream by encoding the one or more images.

[0007] One aspect of the present disclosure relates to a decoding apparatus comprising a circuit and a memory connected to the circuit, wherein the circuit, in operation, decodes the one or more images from a bitstream generated by encoding the one or more images, and obtains the first information from the one or more images, wherein the one or more images are configured with the first information contained in a model, the first information representing a plurality of feature quantities having a time dimension.

[0008] Each embodiment or part of the structure or method in this disclosure can, for example, improve encoding efficiency, improve image quality, reduce the amount of processing required for encoding / decoding, reduce circuit size, or improve the processing speed of encoding / decoding. Alternatively, each embodiment or part of the structure or method in this disclosure can appropriately select constituent elements / actions such as filters, blocks, sizes, motion vectors, reference images, and reference blocks during encoding and decoding. Furthermore, this disclosure also includes structures or methods that can provide beneficial effects other than those described above. For example, structures or methods that improve encoding efficiency while suppressing an increase in processing load.

[0009] Further advantages and effects of one aspect of this disclosure become apparent from the specification and accompanying drawings. The aforementioned advantages and / or effects are obtained through various embodiments and features described in the specification and accompanying drawings, but it is not necessary to provide all of them in order to obtain one or more advantages and / or effects.

[0010] Furthermore, all or all of these methods can be implemented by systems, integrated circuits, computer programs, or computer-readable recording media such as CD-ROMs, or by any combination of systems, methods, integrated circuits, computer programs, and recording media.

[0011] The effects of the invention The structure or method disclosed herein may contribute to one or more of the following: improved encoding efficiency, improved image quality, reduced processing load, reduced circuit size, improved processing speed, and appropriate selection of elements or actions. Furthermore, the structure or method disclosed herein may also contribute to other beneficial effects beyond those described above. Attached Figure Description

[0012] Figure 1 This is a schematic diagram illustrating an example of the structure of the transmission system involved in the implementation method.

[0013] Figure 2 This is a diagram representing an example of the hierarchical structure of data in a stream.

[0014] Figure 3 This is a diagram illustrating an example of the structure of a slice.

[0015] Figure 4 This is a diagram illustrating an example of the structure of a tile.

[0016] Figure 5 This is a diagram illustrating an example of a scalable coding structure.

[0017] Figure 6 This is a diagram illustrating an example of a scalable coding structure.

[0018] Figure 7 This is a block diagram illustrating an example of the structure of the encoding device involved in the embodiment.

[0019] Figure 8 This is a block diagram showing an example of the installation of an encoding device.

[0020] Figure 9 This is a flowchart illustrating an example of the overall encoding process of an encoding device.

[0021] Figure 10 This is a block diagram illustrating the structure of the decoding device involved in the implementation.

[0022] Figure 11 This is a block diagram illustrating an example of the installation of a decoding device.

[0023] Figure 12 This is a flowchart illustrating an example of the overall decoding process of a decoding device.

[0024] Figure 13 This is a diagram illustrating an example of the resolution of a dynamic 3D scene along the X, Y, and Z axes involved in the implementation.

[0025] Figure 14 This is a diagram illustrating examples of multiple scales involved in the implementation method.

[0026] Figure 15 This is a diagram illustrating an example of a dynamic 3D scene with multiple frames involved in the implementation method.

[0027] Figure 16 This is a diagram illustrating an example of a 3D video sequence with multiple dynamic 3D scenes involved in the implementation method.

[0028] Figure 17 This is a diagram showing an example of a feature plane involved in an implementation method.

[0029] Figure 18 This is a diagram illustrating examples of multiple feature transformations in a dynamic 3D scene involved in the implementation method.

[0030] Figure 19 This is a diagram illustrating an example of a group of time-independent feature planes involved in an implementation.

[0031] Figure 20 This is a diagram illustrating an example of a group of time-dependent feature planes involved in an implementation.

[0032] Figure 21 This is a diagram illustrating examples of multiple scales of the time-independent feature plane involved in the implementation.

[0033] Figure 22This is a diagram illustrating examples of multiple scales of the time-dependent feature plane involved in the implementation method.

[0034] Figure 23 This is a diagram illustrating an example of a model of a 3D scene involved in an implementation method.

[0035] Figure 24 This is a diagram illustrating an example of another model of the 3D scene involved in the implementation method.

[0036] Figure 25 This diagram illustrates the rendering process performed by the rendering unit in the embodiment.

[0037] Figure 26 This is a diagram illustrating an example of an image used in the training process involved in the implementation method.

[0038] Figure 27 This is a diagram illustrating an example of the training process of the model involved in the implementation method.

[0039] Figure 28 This is a block diagram illustrating the structure of the encoding device involved in the implementation.

[0040] Figure 29 This is a block diagram illustrating the structure of the decoding device involved in the implementation.

[0041] Figure 30 This is a block diagram illustrating another structural example of the encoding device involved in the embodiment.

[0042] Figure 31 This is a block diagram illustrating another structural example of the decoding device involved in the implementation.

[0043] Figure 32 This is a flowchart of the decoding process performed by the decoding device involved in the first method.

[0044] Figure 33 This is a diagram illustrating an example of packing the feature planes involved in the implementation method into multiple images.

[0045] Figure 34 This is a diagram illustrating another example of how the feature planes involved in the implementation are packaged in multiple images.

[0046] Figure 35 This is a diagram illustrating another example of how the feature planes involved in the implementation are packaged in multiple images.

[0047] Figure 36 This is a diagram illustrating another example of how the feature planes involved in the implementation are packaged in multiple images.

[0048] Figure 37 This is a diagram illustrating another example of how the feature planes involved in the implementation are packaged in multiple images.

[0049] Figure 38 This is a diagram illustrating another example of how the feature planes involved in the implementation are packaged in multiple images.

[0050] Figure 39 This is a diagram illustrating another example of how the feature planes involved in the implementation are packaged in multiple images.

[0051] Figure 40 This is a diagram illustrating another example of how the feature planes involved in the implementation are packaged in multiple images.

[0052] Figure 41 This is a diagram illustrating another example of how the feature planes involved in the implementation are packaged in multiple images.

[0053] Figure 42 This is a diagram illustrating another example of how the feature planes involved in the implementation are packaged in multiple images.

[0054] Figure 43 This is a diagram illustrating another example of how the feature planes involved in the implementation are packaged in multiple images.

[0055] Figure 44 This is a diagram illustrating an example of time-independent feature plane data for unpacking a feature map image from a feature plane according to the implementation method.

[0056] Figure 45 This diagram illustrates an example of unpacking a segment of time-dependent feature plane data involved in an implementation method onto a feature plane.

[0057] Figure 46 This is a diagram illustrating an example of updating a model of dynamic 3D representation involved in an implementation method.

[0058] Figure 47 This is a diagram illustrating an example of a decoding apparatus according to an implementation method using time-dependent feature plane data to update a model.

[0059] Figure 48 This is a diagram illustrating another example of how the feature planes involved in the implementation are packaged in multiple images.

[0060] Figure 49 This is a flowchart of the encoding process of the encoding device involved in the first method.

[0061] Figure 50 This is a diagram illustrating an example of packaging the feature planes involved in the implementation method within a feature map image.

[0062] Figure 51 This is a diagram illustrating an example of dividing the time-dependent feature plane involved in the implementation into segments.

[0063] Figure 52 This is a flowchart of the encoding process performed by the encoding device involved in the implementation method.

[0064] Figure 53 This is a flowchart of the decoding process performed by the decoding device involved in the implementation method.

[0065] Figure 54 This is a flowchart of the decoding process performed by the decoding device involved in the second method.

[0066] Figure 55 This is a block diagram illustrating the structure of the encoding device involved in the implementation.

[0067] Figure 56 This is a block diagram illustrating the structure of the decoding device involved in the implementation.

[0068] Figure 57 This is a diagram illustrating an example of packing feature planes and neural network weights in an image involved in the implementation.

[0069] Figure 58 This is a diagram illustrating another example of packing feature planes and neural network weights in an image involved in the implementation.

[0070] Figure 59 This is a diagram illustrating another example of packing feature planes and neural network weights in an image involved in the implementation.

[0071] Figure 60 This is a diagram illustrating another example of packing feature planes and neural network weights in an image involved in the implementation.

[0072] Figure 61 This is a diagram illustrating another example of packing feature planes and neural network weights in an image involved in the implementation.

[0073] Figure 62 This is a diagram illustrating an example of updating a model of dynamic 3D representation involved in an implementation method.

[0074] Figure 63 This is a diagram illustrating an example of a model that uses neural network weights to update the dynamic 3D representation involved in the implementation.

[0075] Figure 64 This is a diagram illustrating an example of the processing of updating a model of dynamic 3D representation involved in an implementation using voxel grid information.

[0076] Figure 65This is a flowchart of the encoding process performed by the encoding device involved in the second method.

[0077] Figure 66 This is a flowchart of the encoding process performed by the encoding device involved in the implementation method.

[0078] Figure 67 This is a flowchart of the decoding process performed by the decoding device involved in the implementation method.

[0079] Figure 68 This is a diagram showing the overall structure of the content supply system that enables content publishing services.

[0080] Figure 69 This is an example of a web page display.

[0081] Figure 70 This is an example of a web page display.

[0082] Figure 71 This is a diagram illustrating an example of a smartphone.

[0083] Figure 72 This is a block diagram illustrating the structure of a smartphone. Detailed Implementation

[0084] [Introduction] One aspect of this disclosure involves an encoding apparatus comprising a circuit and a memory connected to the circuit. During operation, the circuit acquires first information contained in a model, the first information representing multiple features having a time dimension, arranges the first information on one or more images, and generates a bitstream by encoding the one or more images. Thus, the encoding apparatus can use image encoding to encode the first information representing multiple features having a time dimension contained in the model. Therefore, the encoding processing of information contained in the model can be easily implemented.

[0085] For example, the first information could contain multiple data items, and the bitstream could contain second information indicating the configuration order of the multiple data items. Thus, the decoding device can use the second information to appropriately extract multiple data items from the image.

[0086] For example, the first information may contain multiple data items, and the bitstream may contain third information indicating the start position of at least one of the multiple data items in the more than one image. Thus, the decoding device can use the third information to appropriately extract multiple data items from the image.

[0087] For example, the first information could include multiple feature planes, each representing a multiple feature quantity configured in two dimensions. Thus, the encoding device can use image encoding to encode the information of the multiple feature planes contained in the model. This allows for easy encoding of the information contained in the model.

[0088] For example, the first information could include: first data, comprising multiple features that do not have the time dimension; and multiple second data, comprising multiple features that have the time dimension, each of the multiple second data corresponding to a different time interval. The one or more images could comprise multiple first images and multiple second images, with the first data segmented across the multiple first images and the multiple second data segmented across the multiple second images. This reduces the padding data contained in one or more images, thus improving encoding efficiency.

[0089] For example, the first information may include: first data, comprising multiple feature quantities that do not have the time dimension; and multiple second data, comprising multiple feature quantities that have the time dimension, the multiple second data corresponding to different time intervals, the one or more images comprising a first image and a second image, the first data being configured in the first image, and the multiple second data being centrally configured in the second image. This reduces the number of images being encoded, thus improving encoding efficiency.

[0090] For example, the first information may include first data and second data with a higher resolution than the first data, and the more than one image may include a first image and a second image encoded after the first image, with the first data configured in the first image and the second data configured in the second image. Thus, the decoding device can first decode the first data and use the first data to generate a model.

[0091] For example, the first information may include first data and second data with a higher resolution than the first data. Each of the more than one image includes a first component, a second component smaller than the first component, and a third component. The first data is configured in at least one of the second component and the third component, and the second data is configured in the first component. Thus, multiple data can be efficiently configured using multiple components.

[0092] One aspect of this disclosure involves a decoding apparatus comprising a circuit and a memory connected to the circuit. During operation, the circuit decodes one or more images from a bitstream generated by encoding one or more images containing first information, and obtains the first information from the images. This first information is contained in a model and represents multiple feature quantities having a time dimension. Thus, the decoding apparatus can use image-encoding-based decoding processing to decode the first information contained in the model representing multiple feature quantities having a time dimension. Therefore, decoding processing of information contained in the model can be easily implemented.

[0093] For example, the first information could contain multiple data items, and the bitstream could contain second information indicating the configuration order of the multiple data items. Thus, the decoding device can use the second information to appropriately extract multiple data items from the image.

[0094] For example, the first information may contain multiple data items, and the bitstream may contain third information indicating the start position of at least one of the multiple data items in the more than one image. Thus, the decoding device can use the third information to appropriately extract multiple data items from the image.

[0095] For example, the first information could include multiple feature planes, each representing a multiple feature quantity configured in two dimensions. Therefore, the decoding device can use image-encoding-based decoding processing to decode the information of the multiple feature planes contained in the model. This allows for easy decoding of the information contained in the model.

[0096] For example, the first information could include: first data, comprising multiple features that do not have the time dimension; and multiple second data, comprising multiple features that have the time dimension, each of the multiple second data corresponding to a different time interval. The one or more images could comprise multiple first images and multiple second images, with the first data segmented across the multiple first images and the multiple second data segmented across the multiple second images. This reduces the padding data contained in one or more images, thus improving encoding efficiency.

[0097] For example, the first information may include: first data, comprising multiple features that do not have the time dimension; and multiple second data, comprising multiple features that have the time dimension, each of the multiple second data corresponding to a different time interval, wherein the one or more images comprises a first image and a second image, the first data is configured in the first image, and the multiple second data are centrally configured in the second image. This reduces the number of images being encoded, thus improving encoding efficiency.

[0098] For example, the first information could include first data and second data with a higher resolution than the first data, and the more than one image could include a first image and a second image encoded after the first image, with the first data configured in the first image and the second data configured in the second image. Thus, the decoding device can first decode the first data and use it to generate a model.

[0099] For example, the first information could include first data and second data with a higher resolution than the first data. Each of the more than one image could include a first component, a second component smaller than the first component, and a third component. The first data could be configured in at least one of the second component and the third component, and the second data could be configured in the first component. Thus, multiple data can be efficiently configured using multiple components.

[0100] In one aspect of the encoding method disclosed herein, first information representing multiple features having a time dimension, contained in a model, is obtained; the first information is configured on one or more images; and a bitstream is generated by encoding the one or more images. Thus, the encoding method can use image encoding to encode the first information representing multiple features having a time dimension contained in the model. Therefore, the encoding processing of information contained in the model can be easily implemented.

[0101] In one aspect of the decoding method disclosed herein, the one or more images are decoded from a bitstream generated by encoding one or more images containing first information in a model, and the first information, representing multiple feature quantities having a time dimension, is obtained from the one or more images. Thus, the decoding method can use image-encoding-based decoding processing to decode the first information in the model representing multiple feature quantities having a time dimension. Therefore, decoding processing of information contained in the model can be easily implemented.

[0102] One aspect of this disclosure involves an encoding apparatus comprising a circuit and a memory connected to the circuit. During operation, the circuit acquires multiple neural network weights included in a model, assigns these weights to one or more images, and generates a bitstream by encoding the one or more images. Thus, the encoding apparatus can encode multiple neural network weights included in a model using image encoding. This allows for easy encoding of information contained in the model.

[0103] For example, the bitstream may contain first information indicating the configuration order of the plurality of neural network weights. Thus, the decoding device can use the first information to appropriately extract the plurality of neural network weights from the image.

[0104] For example, the bitstream could also contain metadata, which includes the first information and multiple parameters associated with the model. Thus, the decoding device can use multiple parameters to generate the model.

[0105] For example, the circuit may also acquire second information contained in the model, which represents multiple feature quantities having a time dimension, wherein the multiple neural network weights and the second information are configured on one or more images. Thus, the encoding device can use image encoding to encode the second information representing multiple feature quantities having a time dimension contained in the model. Therefore, the encoding processing of information contained in the model can be easily implemented.

[0106] For example, the multiple neural network weights and the second information can be configured in the same image contained in the one or more images. This allows for the efficient configuration of multiple neural network weights and the second information.

[0107] For example, the multiple neural network weights and the second information can be configured in different images contained within the one or more images. Thus, the decoding device can independently decode the multiple neural network weights and the second information.

[0108] For example, each of the more than one image may contain multiple components, and the multiple neural network weights and the second information may be configured on different components among the multiple components. This allows for the efficient configuration of multiple neural network weights and the second information.

[0109] For example, each of the more than one image may contain multiple layers, with the multiple neural network weights and the second information configured in different layers. This allows for the efficient configuration of multiple neural network weights and the second information.

[0110] One aspect of this disclosure relates to a decoding apparatus comprising a circuit and a memory connected to the circuit. During operation, the circuit decodes one or more images from a bitstream generated by encoding one or more images containing weights of multiple neural networks included in a model, and obtains the multiple neural network weights from the images. Thus, the decoding apparatus can decode the multiple neural network weights included in a model using image encoding-based decoding processing. This allows for easy implementation of decoding processing of information contained in the model.

[0111] For example, the bitstream may contain first information indicating the configuration order of the plurality of neural network weights. Thus, the decoding device can use the first information to appropriately extract the plurality of neural network weights from the image.

[0112] For example, the bitstream could also contain metadata, which includes the first information and multiple parameters associated with the model. Thus, the decoding device can use multiple parameters to generate the model.

[0113] For example, the plurality of neural network weights and second information contained in the model can be configured in one or more images. This second information represents multiple feature quantities with a time dimension. In the acquisition process, the plurality of neural network weights and the second information are acquired from the one or more images. Therefore, the decoding device can use image-encoding-based decoding processing to decode the second information contained in the model that represents multiple feature quantities with a time dimension. This allows for easy implementation of the decoding processing of information contained in the model.

[0114] For example, the multiple neural network weights and the second information can be configured in the same image contained in the one or more images. This allows for the efficient configuration of multiple neural network weights and the second information.

[0115] For example, the multiple neural network weights and the second information can be configured in different images contained within the one or more images. Thus, the decoding device can independently decode the multiple neural network weights and the second information.

[0116] For example, each of the more than one image may contain multiple components, and the multiple neural network weights and the second information may be configured on different components among the multiple components. This allows for the efficient configuration of multiple neural network weights and the second information.

[0117] For example, each of the more than one image may contain multiple layers, with the multiple neural network weights and the second information configured in different layers. This allows for the efficient configuration of multiple neural network weights and the second information.

[0118] One aspect of this disclosure involves an encoding method that obtains multiple neural network weights contained in a model, configures these weights on one or more images, and generates a bitstream by encoding the one or more images. Thus, the encoding method can encode multiple neural network weights contained in a model using image encoding. This allows for easy encoding of information contained in the model.

[0119] One aspect of this disclosure relates to a decoding method that decodes one or more images from a bitstream generated by encoding one or more images containing weights of multiple neural networks included in a model, and obtains the multiple neural network weights from the one or more images. Thus, the decoding method can decode the multiple neural network weights included in a model using image encoding-based decoding processing. Therefore, decoding processing of information contained in a model can be easily implemented.

[0120] Furthermore, these general or specific methods can be implemented through systems, methods, integrated circuits, computer programs, or computer-readable recording media such as CD-ROMs, or through any combination of systems, methods, integrated circuits, computer programs, and recording media.

[0121] [Definition of the term] As an example, the terms can also be defined as follows.

[0122] (1) Image It is a unit of data consisting of a collection of pixels, which consists of images or blocks smaller than images, and includes still images in addition to moving images.

[0123] (2) Pictures A frame is a unit of image processing consisting of a collection of pixels, sometimes referred to as a frame or field.

[0124] (3) blocks A processing unit is a collection of a specific number of pixels, as illustrated in the examples below, and the name is not limited. Furthermore, the shape is not limited; for example, it includes rectangles composed of M×N pixels, squares composed of M×M pixels, as well as triangles, circles, and other shapes.

[0125] (Example of a block) • Slices / tiles / bricks • CTU / Superblock / Basic Unit of Division •VPDU / Hardware processing unit • CU / Processing Block Unit / Prediction Block Unit (PU) / Orthogonal Transformation Block Unit (TU) / Unit ·Sub-block (4) Pixels / Samples A point is the smallest unit that makes up an image. It includes not only pixels at integer positions but also pixels at fractional positions generated based on pixels at integer positions.

[0126] (5) Pixel value / sample value These are the inherent values ​​of a pixel, including brightness, color difference, RGB grayscale, and depth values ​​or binary values ​​of 0 and 1.

[0127] (6) Sign Besides 1 bit, it also includes cases with multiple bits, such as parameters or indices of 2 bits or more. Furthermore, it can be not only a binary value but also a multi-valued value using other bases.

[0128] (7) Signal In order to transmit information, signals are symbolized and encoded, including not only discretized digital signals but also analog signals that take continuous values.

[0129] (8) Stream / Bitstream This refers to a data string or stream of digital data. A stream / bit stream can be a single, hierarchical structure composed of multiple streams. Furthermore, besides transmission via serial communication over a single path, it also includes transmission via data packet communication over multiple paths.

[0130] (9) Difference / Difference In the case of scalars, any operation involving difference is acceptable except for simple difference (xy), including the absolute value of the difference (|xy|), the difference of squares (x^2-y^2), the square root of the difference (√(xy)), the weighted difference (ax-by: a and b are constants), and the offset difference (x-y+a: a is the offset).

[0131] (10) and In the case of scalars, any operation involving summation is acceptable, except for the simple sum (x+y), including the absolute value of the sum (|x+y|), the sum of squares (x^2+y^2), the square root of the sum (√(x+y)), the weighted sum (ax+by: a and b are constants), and the offset sum (x+y+a: a is the offset).

[0132] (11) based on This also includes cases where elements other than those of the object on which the result is based are included. Furthermore, in addition to cases where the result is obtained directly, it also includes cases where the result is obtained through intermediate results.

[0133] (12) Use (used, using) This also includes cases where elements other than those used as the object are incorporated. Furthermore, in addition to cases where the result is obtained directly, it also includes cases where the result is obtained through intermediate results.

[0134] (13) Prohibit (prohibit, forbid) It can also be described as not being allowed. Furthermore, not prohibiting or allowing does not necessarily imply an obligation.

[0135] (14) Limit (limit, restriction / restrict / restricted) This can also be termed "not allowed." Furthermore, not prohibiting or allowing does not necessarily imply an obligation. Moreover, prohibition can be partial in quantity or quality, or it can include cases of complete prohibition.

[0136] (15) Color difference (chroma) Chroma is an adjective, denoted by the symbols Cb and Cr, that describes one of two color difference signals associated with a primary color, either in a specified sample arrangement or as a single sample representation. The term chrominance can also be used instead of chroma.

[0137] (16) Luminance Luma is an adjective, indicated by the symbol or subscript Y or L, that specifies a monochrome signal associated with a primary color, whether representing an arrangement or a single sample. The term luminance can also be used instead of luma.

[0138] [Explanations related to the records] In the accompanying drawings, the same reference numerals denote the same or similar constituent elements. Furthermore, the size and relative positions of the constituent elements in the drawings are not necessarily depicted to a fixed scale.

[0139] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Furthermore, the embodiments described below are all general or specific examples. The numerical values, shapes, materials, constituent elements, the arrangement and connection methods of the constituent elements, steps, relationships and sequences of steps, etc., represented by the following embodiments are all examples and are not intended to limit the scope of the claims.

[0140] Hereinafter, embodiments of the encoding and decoding apparatus will be described. The embodiments are examples of encoding and decoding apparatuses to which the processing and / or structures described in the various embodiments of this disclosure can be applied. The processing and / or structures may also be implemented in encoding and decoding apparatuses different from the embodiments. For example, regarding the processing and / or structures applied to the embodiments, any of the following may also be implemented.

[0141] (1) Any of the multiple components of the encoding or decoding apparatus of the embodiments described in the various aspects of this disclosure may be replaced or combined with other components described in the various aspects of this disclosure.

[0142] (2) In the encoding or decoding apparatus of the embodiments, the functions or processes performed by a portion of the multiple components of the encoding or decoding apparatus may be arbitrarily changed, such as by adding, replacing, or deleting functions or processes. For example, any function or process may be replaced or combined with other functions or processes described in any of the embodiments of this disclosure.

[0143] (3) In the method implemented by the encoding or decoding apparatus of the embodiment, any changes may be made to some of the multiple processes included in the method, such as adding, replacing, or deleting. For example, any process in the method may be replaced or combined with other processes described in any of the embodiments of this disclosure.

[0144] (4) Some of the constituent elements of the encoding or decoding apparatus constituting the embodiment may also be combined with the constituent elements described in any of the embodiments of the present disclosure, or may be combined with a portion of the constituent elements having the functions described in any of the embodiments of the present disclosure, or may be combined with a portion of the constituent elements that implement the processing performed by the constituent elements described in the embodiments of the present disclosure.

[0145] (5) The constituent elements that have a part of the function of the encoding or decoding device of the embodiment, or the constituent elements that have a part of the processing of the encoding or decoding device of the embodiment, may be combined or replaced with the constituent elements described in any of the embodiments of the present disclosure, the constituent elements that have a part of the function of the embodiment of the present disclosure, or the constituent elements that have a part of the processing of the embodiment of the present disclosure.

[0146] (6) In the method implemented by the encoding or decoding apparatus of the embodiment, any one of the multiple processes included in the method may be replaced or combined with any of the processes described in any of the various embodiments of the present disclosure or any of the same processes.

[0147] (7) Some of the multiple processes included in the method implemented by the encoding or decoding device of the embodiment may also be combined with the processes described in any of the embodiments of this disclosure.

[0148] (8) The implementation of the processing and / or structure described in the various embodiments of this disclosure is not limited to the encoding or decoding apparatus of the embodiments. For example, the processing and / or structure may also be implemented in an apparatus used for a different purpose than the motion picture encoding or motion picture decoding disclosed in the embodiments.

[0149] [System Architecture] Figure 1This is a schematic diagram illustrating an example of the structure of the transmission system involved in this embodiment.

[0150] A transmission system (Trs) is a system that transmits streams generated by encoding images and decodes the transmitted streams. For example, ... Figure 1 As shown, such a transmission system TRS includes an encoding device 100, a network Nw, and a decoding device 200.

[0151] An image is input to the encoding device 100. The encoding device 100 generates a stream by encoding the input image and outputs the stream to the network Nw. The stream contains, for example, the encoded image and control information for decoding the encoded image. Through this encoding, the image is compressed.

[0152] Furthermore, the original image input to the encoding device 100 before encoding is also referred to as the original image, original signal, or original sample. Additionally, the image can be a moving image or a still image. Furthermore, an image is a higher-level concept such as a sequence, picture, or block, and is not limited by spatial or temporal regions unless otherwise specified. Furthermore, an image is composed of an arrangement of pixels or pixel values, and the signal or pixel values ​​representing that image are also referred to as samples. Furthermore, a stream can also be referred to as a bitstream, coded bitstream, compressed bitstream, or coded signal. Moreover, the encoding device can also be referred to as an image encoding device or a moving image encoding device, and the encoding method of the encoding device 100 can also be referred to as an encoding method, an image encoding method, or a moving image encoding method.

[0153] Network Nw transmits the stream generated by encoding device 100 to decoding device 200. Network Nw can also be the Internet, a wide area network (WAN), a local area network (LAN), or a combination thereof. Network Nw is not necessarily limited to a two-way communication network; it can also be a one-way communication network transmitting broadcast waves such as terrestrial digital broadcasts or satellite broadcasts. In addition, network Nw can also be replaced by recording media that record streams, such as DVD (Digital Versatile Disc) or BD (Blu-Ray Disc).

[0154] The decoding device 200 decodes the stream transmitted over the network Nw, for example, to generate a decoded image as an uncompressed image. For instance, the decoding device decodes the stream according to a decoding method corresponding to the encoding method performed by the encoding device 100.

[0155] In addition, the decoding device can also be called an image decoding device or a moving image decoding device, and the decoding method of the decoding device 200 can also be called a decoding method, an image decoding method, or a moving image decoding method.

[0156] [Data Structures] Figure 2 This is a diagram representing an example of the hierarchical structure of data in a stream. A stream, for example, contains video sequences. For instance, as shown... Figure 2 As shown in (a), the video sequence contains VPS (Video Parameter Set), SPS (Sequence Parameter Set), PPS (Picture Parameter Set), SEI (Supplemental Enhancement Information), and multiple images.

[0157] In a moving image composed of multiple layers, VPS contains common coding parameters for multiple layers, as well as coding parameters associated with the multiple layers or individual layers contained in the moving image.

[0158] The SPS contains parameters used for the sequence, that is, encoding parameters referenced by the decoding device 200 for decoding the sequence. For example, these encoding parameters can also represent the width or height of the image. Furthermore, multiple SPSs can exist.

[0159] The PPS contains parameters used for the images, i.e., encoding parameters referenced by the decoding device 200 for decoding each image in the sequence. For example, these encoding parameters may also include a reference value for the quantization width used in image decoding and a flag indicating the application of weighted prediction. Furthermore, multiple PPSs may exist. Additionally, SPS and PPS are sometimes simply referred to as a parameter set.

[0160] like Figure 2 As shown in (b), an image may also contain an image header and one or more slices. The image header contains encoding parameters referenced by the decoding device 200 for decoding the one or more slices.

[0161] like Figure 2 As shown in (c), the slice includes a slice header and one or more bricks. The slice header includes encoding parameters referenced by the decoding device 200 for decoding the one or more bricks.

[0162] like Figure 2 As shown in (d), a brick contains more than one CTU (Coding Tree Unit).

[0163] Alternatively, the image may not contain a slice, but instead contain a set of tiles. In this case, the set of tiles contains more than one tile. Furthermore, slices can also be included within bricks.

[0164] CTU is also known as a superblock or basic partitioning unit. For example... Figure 2 As shown in (e), such a CTU includes a CTU header and one or more CUs (Coding Units). The CTU header includes encoding parameters referenced by the decoding device 200 for decoding one or more CUs.

[0165] A CU can also be divided into multiple smaller CUs. Furthermore, as... Figure 2 As shown in (f), the CU includes a CU header, prediction information, and residual coefficient information. The prediction information is used to predict the CU, and the residual coefficient information represents the prediction residuals, which will be described later. Furthermore, the CU is essentially the same as the PU (Prediction Unit) and TU (Transform Unit), but, for example, in the SBT described later, it may also contain multiple TUs smaller than the CU. Additionally, the CU can be processed according to each VPDU (Virtual Pipeline Decoding Unit) that constitutes the CU. A VPDU is, for example, a fixed unit that can be processed in one stage during pipelined processing in hardware.

[0166] Alternatively, a stream may not have... Figure 2 Any part of the layers shown. Furthermore, the order of these layers can be interchanged, and any layer can be replaced with another layer. Additionally, the image of the object being processed by the encoding device 100 or decoding device 200 at the current point in time is called the current image. If the processing is encoding, the current image is synonymous with the encoded object image; if the processing is decoding, the current image is synonymous with the decoded object image. Furthermore, a block, such as a CU or CU block, of the object being processed by the encoding device 100 or decoding device 200 at the current point in time is called the current block. If the processing is encoding, the current block is synonymous with the encoded object block; if the processing is decoding, the current block is synonymous with the decoded object block.

[0167] [Structural slices / tiles of the image] To decode images in parallel, images are sometimes composed of slice units or tile units.

[0168] A slice is the basic coded unit that makes up an image. An image is composed of one or more slices. Furthermore, a slice is composed of one or more consecutive CTUs.

[0169] Figure 3This is an example diagram illustrating the structure of a slice. For instance, an image contains 11×8 CTUs and is divided into four slices (slices 1-4). Slice 1, for example, consists of 16 CTUs, slice 2 consists of 21 CTUs, slice 3 consists of 29 CTUs, and slice 4 consists of 22 CTUs. Here, each CTU within the image belongs to a specific slice. The shape of a slice is the shape of the image after horizontal division. The boundary of a slice does not need to be at the edge of the image; it can be a position within the boundaries of the CTUs in the image. The processing order (encoding or decoding order) of the CTUs in a slice is, for example, a raster scan order. Furthermore, a slice contains a slice header and encoded data. The slice header may also describe the characteristics of the slice, such as the CTU addresses at the beginning of the slice and the slice type.

[0170] A tile is a unit that makes up a rectangular area of ​​an image. Tiles can also be assigned a number called TileId according to the raster scan sequence.

[0171] Figure 4 This is an example diagram illustrating the structure of a tile. For example, the image contains 11×8 CTUs and is divided into four rectangular regions (tiles 1-4). When using tiles, the processing order of the CTUs is changed compared to when not using tiles. Without tiles, multiple CTUs within the image are processed, for example, in raster scan order. With tiles, in each of the multiple tiles, at least one CTU is processed, for example, in raster scan order. For example, as... Figure 4 As shown, the processing order of the multiple CTUs contained in tile 1 is from the left end of the first column of tile 1 toward the right end of the first column of tile 1, and then from the left end of the second column of tile 1 toward the right end of the second column of tile 1.

[0172] In addition, a tile sometimes contains more than one slice, and a slice sometimes contains more than one tile.

[0173] Alternatively, an image can also be composed of tile sets. A tile set can contain more than one tile group, or more than one tile. An image can also consist only of tile sets, tile groups, and individual tiles. For example, the order in which multiple tiles are scanned in raster order for each tile set can be used as the basic encoding order of the tiles. A set of more than one tile with consecutive basic encoding orders within each tile set can be used as a tile group. Such an image can also be composed of the segmentation section 102 described later (see...). Figure 7 )constitute.

[0174] [Hyper-level coding] Figure 5 as well as Figure 6 This is a diagram illustrating an example of a hierarchical flow structure.

[0175] like Figure 5 As shown, the encoding device 100 can also generate a temporally / spatially scalable stream by encoding each of multiple images into one of multiple layers. For example, by encoding the images layer by layer, the encoding device 100 achieves scalability where the enhancement layer exists above the base layer. Such encoding of each image is called scalable encoding. Therefore, the decoding device 200 can switch the image quality displayed by decoding the stream. That is, the decoding device 200 determines which layer to decode based on intrinsic factors such as its own performance and extrinsic factors such as the state of communication bandwidth. As a result, the decoding device 200 can freely switch between decoding the same content as low-resolution content and high-resolution content. For example, a user of the stream might use a smartphone while on the move to watch the moving images halfway through the stream, and then watch the rest of the moving images at home using a device such as an internet TV. Furthermore, each of the aforementioned smartphones and devices is equipped with a decoding device 200 with similar or different performance. In this case, if the device decodes to the higher layer in the stream, the user can watch high-quality moving images at home. Therefore, the encoding device 100 does not need to generate multiple streams with the same content but different image quality, which can reduce the processing load.

[0176] Furthermore, the enhancement layer may also contain metadata such as image-based statistical information. The decoding device 200 can also generate high-quality motion images by performing super-resolution on the base layer image based on the metadata. Super-resolution can be either improving the signal-to-noise ratio (SNR) at the same resolution or increasing the resolution. The metadata may also contain information for determining the linear or nonlinear filtering coefficients used in the super-resolution processing, or information for determining the parameter values ​​in filtering, machine learning, or least squares operations used in the super-resolution processing.

[0177] Alternatively, the image can be segmented into tiles based on the meaning of each object within it. In this case, the decoding device 200 can also decode only a portion of the image by selecting tiles of the object to be decoded. Furthermore, the attributes of the object (people, cars, balls, etc.) and its position within the image (coordinates within the same image, etc.) can be stored as metadata. In this case, the decoding device 200 can determine the desired target's location based on the metadata and decide which tile contains that target. For example, as... Figure 6 As shown, metadata is stored using data storage structures different from image data, such as SEI in HEVC. This metadata represents, for example, the location, size, or color of the main target.

[0178] Furthermore, metadata can be stored in units consisting of multiple images, such as streams, sequences, or random access units. Thus, the decoding device 200 can obtain information such as the moment a specific person appears in a moving image, and can determine the image containing the target and the target's position within that image by using this moment and image unit information.

[0179] [Encoding device] Next, the encoding device 100 according to the embodiment will be described. Figure 7 This is a block diagram illustrating an example of the structure of the encoding apparatus 100 according to the embodiment. The encoding apparatus 100 encodes images in blocks.

[0180] like Figure 7 As shown, the encoding apparatus 100 is an apparatus for encoding images in blocks, including a segmentation unit 102, a subtraction unit 104, a transform unit 106, a quantization unit 108, an entropy encoding unit 110, an inverse quantization unit 112, an inverse transform unit 114, an addition unit 116, a block memory 118, a loop filtering unit 120, a frame memory 122, an intra-frame prediction unit 124, an inter-frame prediction unit 126, a prediction control unit 128, and a prediction parameter generation unit 130. Furthermore, the intra-frame prediction unit 124 and the inter-frame prediction unit 126 are each configured as part of the prediction processing unit.

[0181] [Installation example of the encoding device] Figure 8 This is a block diagram showing an example of the installation of the encoding device 100. The encoding device 100 includes a processor a1 and a memory a2. For example, Figure 7 The multiple components of the encoding device 100 shown are composed of Figure 8 The processor a1 and memory a2 shown are installed and implemented.

[0182] Processor a1 is a circuit that processes information and can access memory a2. For example, processor a1 is a dedicated or general-purpose electronic circuit that encodes images. Processor a1 can also be a processor like a CPU. Furthermore, processor a1 can also be a collection of multiple electronic circuits. Additionally, for example, processor a1 can also function as... Figure 7 The functions of multiple components of the encoding device 100 shown, excluding the component for storing information.

[0183] Memory a2 is a dedicated or general-purpose memory that stores information used by processor a1 to encode images. Memory a2 can be an electronic circuit or connected to processor a1. Furthermore, memory a2 can also be included within processor a1. Additionally, memory a2 can be a collection of multiple electronic circuits. Furthermore, memory a2 can be a disk or optical disc, or it can take the form of a storage device or recording medium. Furthermore, memory a2 can be non-volatile memory or volatile memory.

[0184] For example, the encoded image or the stream corresponding to the encoded image can be stored in memory a2. Furthermore, the program used by processor a1 to encode the image can also be stored in memory a2.

[0185] In addition, for example, memory a2 can also serve as Figure 7 The encoding device 100 shown here has multiple components, including the component for storing information. Specifically, the memory a2 can also serve as... Figure 7 The block memory 118 and frame memory 122 shown have the following functions. More specifically, reconstructed images (specifically reconstructed blocks or reconstructed pictures, etc.) can also be stored in memory a2.

[0186] Alternatively, the encoding device 100 may not require installation of [something]. Figure 7 All of the multiple constituent elements shown can also be processed without the above-mentioned multiple processing steps. Figure 7 Some of the constituent elements shown may also be included in other devices, and some of the aforementioned processes may also be performed by other devices.

[0187] The following section will describe the overall processing flow of the encoding device 100, followed by a description of the constituent elements included in the encoding device 100.

[0188] [Overall Encoding Process] Figure 9 This is a flowchart illustrating an example of the overall encoding process performed by the encoding device 100.

[0189] First, the segmentation unit 102 of the encoding device 100 segments the image contained in the original image into multiple fixed-size blocks (128×128 pixels) (step Sa_1). Then, the segmentation unit 102 selects a segmentation mode for each fixed-size block (step Sa_2). That is, the segmentation unit 102 further segments the fixed-size blocks into multiple blocks constituting the selected segmentation mode. Then, the encoding device 100 performs the processing steps Sa_3 to Sa_9 on each of the multiple blocks.

[0190] The prediction processing unit and prediction control unit 128, which consist of the intra-frame prediction unit 124 and the inter-frame prediction unit 126, generate the prediction image of the current block (step Sa_3). In addition, the prediction image is also referred to as the prediction signal, prediction block, or prediction sample.

[0191] Next, the subtraction unit 104 generates the difference between the current block and the predicted image as the prediction residual (step Sa_4). The prediction residual is also referred to as the prediction error.

[0192] Next, the transformation unit 106 and the quantization unit 108 transform and quantize the predicted image to generate multiple quantization coefficients (step Sa_5).

[0193] Next, the entropy coding unit 110 generates a stream (step Sa_6) by encoding the plurality of quantization coefficients and prediction parameters related to the generation of the predicted image (specifically, entropy coding).

[0194] Next, the inverse quantization unit 112 and the inverse transformation unit 114 recover the prediction residual by performing inverse quantization and inverse transformation on multiple quantization coefficients (step Sa_7).

[0195] Next, the addition unit 116 reconstructs the current block by adding the predicted image to the recovered prediction residual (step Sa_8). This generates a reconstructed image. Furthermore, the reconstructed image is also referred to as a reconstructed block, and in particular, the reconstructed image generated by the encoding device 100 is also referred to as a local decoding block or a local decoding image.

[0196] When the reconstructed image is generated, the loop filter unit 120 filters the reconstructed image as needed (step Sa_9).

[0197] Then, the encoding device 100 determines whether the encoding of the entire image is complete (step Sa_10). If it is determined that the encoding is not complete (no in step Sa_10), the processing starting from step Sa_2 is repeatedly executed.

[0198] Furthermore, in the example above, the encoding device 100 selects a partitioning pattern for blocks of a fixed size and encodes each block according to that partitioning pattern. However, it can also encode each block according to each of multiple partitioning patterns. In this case, the encoding device 100 can also evaluate the cost for each of the multiple partitioning patterns, for example, selecting the stream obtained by encoding according to the partitioning pattern with the lowest cost as the final output stream.

[0199] Furthermore, these steps Sa_1 to Sa_10 can be performed sequentially by the encoding device 100, and multiple processes of a portion of these processes can be performed in parallel or in reverse order.

[0200] The encoding process performed by this encoding apparatus 100 uses a hybrid encoding method combining predictive coding and transform coding. Furthermore, predictive coding is performed through an encoding loop, which consists of a subtraction unit 104, a transform unit 106, a quantization unit 108, an inverse quantization unit 112, an inverse transform unit 114, an addition unit 116, a loop filter unit 120, a block memory 118, a frame memory 122, an intra-frame prediction unit 124, an inter-frame prediction unit 126, and a prediction control unit 128. That is, the prediction processing unit, composed of the intra-frame prediction unit 124 and the inter-frame prediction unit 126, constitutes a part of the encoding loop.

[0201] [Decoding device] Next, the decoding device 200, which is capable of decoding the stream output from the encoding device 100, will be described. Figure 10 This is a block diagram illustrating an example of the structure of the decoding apparatus 200 according to the embodiment. The decoding apparatus 200 is a device that decodes a stream of encoded images on a block-by-block basis.

[0202] like Figure 10 As shown, the decoding device 200 includes an entropy decoding unit 202, an inverse quantization unit 204, an inverse transform unit 206, an adder unit 208, a block memory 210, a loop filter unit 212, a frame memory 214, an intra-frame prediction unit 216, an inter-frame prediction unit 218, a prediction control unit 220, a prediction parameter generation unit 222, and a segmentation determination unit 224. Furthermore, the intra-frame prediction unit 216 and the inter-frame prediction unit 218 are each configured as part of the prediction processing unit.

[0203] [Installation example of the decoding device] Figure 11 This is a block diagram illustrating an installation example of the decoding device 200. The decoding device 200 includes a processor b1 and a memory b2. For example, Figure 10 The multiple components of the decoding device 200 shown are composed of Figure 11 The processor b1 and memory b2 shown are installed and implemented.

[0204] Processor b1 is a circuit that processes information and can access memory b2. For example, processor b1 may be a dedicated or general-purpose electronic circuit for decoding streams. Processor b1 can also be a processor like a CPU. Furthermore, processor b1 can be an assembly of multiple electronic circuits. Additionally, for example, processor b1 can also function as... Figure 10 The functions of multiple components of the decoding device 200 shown, excluding the component for storing information.

[0205] Memory b2 is a dedicated or general-purpose memory that stores information used by processor b1 to decode the stream. Memory b2 can be an electronic circuit or connected to processor b1. Furthermore, memory b2 can also be included within processor b1. Additionally, memory b2 can be an assembly of multiple electronic circuits. Furthermore, memory b2 can be a disk or optical disk, or it can take the form of a storage device or recording medium. Furthermore, memory b2 can be non-volatile memory or volatile memory.

[0206] For example, memory b2 can store images or streams. Furthermore, memory b2 can also store programs for the processor b1 to decode the streams.

[0207] In addition, for example, memory b2 can also serve as Figure 10 The decoding device 200 shown above functions as an information storage component among its various constituent elements. Specifically, the memory b2 can also serve as... Figure 10 The block memory 210 and frame memory 214 shown below illustrate their functions. More specifically, reconstructed images (specifically, reconstructed blocks or reconstructed pictures, etc.) can also be stored in memory b2.

[0208] Alternatively, it is not necessary to install in the decoding device 200. Figure 10 All of the multiple constituent elements shown above can also be processed without the above-mentioned multiple processing steps. Figure 10 A portion of the multiple constituent elements shown may also be included in other devices, and a portion of the aforementioned multiple processes may also be performed by other devices.

[0209] The following describes the overall processing flow of the decoding apparatus 200, followed by an explanation of each component included in the decoding apparatus 200. However, detailed descriptions of components in the decoding apparatus 200 that undergo the same processing as those in the encoding apparatus 100 are omitted. For example, the inverse quantization unit 204, inverse transform unit 206, adder 208, block memory 210, frame memory 214, intra-frame prediction unit 216, inter-frame prediction unit 218, prediction control unit 220, and loop filter unit 212 in the decoding apparatus 200 undergo the same processing as the inverse quantization unit 112, inverse transform unit 114, adder 116, block memory 118, frame memory 122, intra-frame prediction unit 124, inter-frame prediction unit 126, prediction control unit 128, and loop filter unit 120 in the encoding apparatus 100.

[0210] [Overall Decoding Process] Figure 12 This is a flowchart illustrating an example of the overall decoding process performed by the decoding device 200.

[0211] First, the segmentation determination unit 224 of the decoding device 200 determines the segmentation mode (step Sp_1) for each of the multiple fixed-size blocks (128×128 pixels) contained in the image based on the parameters input from the entropy decoding unit 202. This segmentation mode is selected by the encoding device 100. Then, the decoding device 200 performs steps Sp_2 to Sp_6 on each of the multiple blocks constituting the segmentation mode.

[0212] The entropy decoding unit 202 decodes the encoded quantization coefficients and prediction parameters of the current block (specifically, entropy decoding) (step Sp_2).

[0213] Next, the inverse quantization unit 204 and the inverse transformation unit 206 recover the prediction residual of the current block by performing inverse quantization and inverse transformation on multiple quantization coefficients (step Sp_3).

[0214] Next, the prediction processing unit, which consists of the intra-frame prediction unit 216, the inter-frame prediction unit 218, and the prediction control unit 220, generates the prediction image of the current block (step Sp_4).

[0215] Next, the addition unit 208 reconstructs the current block into a reconstructed image (also known as a decoded image block) by adding the predicted image to the predicted residual (step Sp_5).

[0216] Then, when the reconstructed image is generated, the loop filter unit 212 filters the reconstructed image (step Sp_6).

[0217] Then, the decoding device 200 determines whether the decoding of the entire image is complete (step Sp_7). If it is determined that the decoding is not complete (no in step Sp_7), the processing starting from step Sp_1 is repeatedly executed.

[0218] Furthermore, these steps Sp_1 to Sp_7 can be performed sequentially by the decoding device 200, and multiple processes of a portion of these processes can be performed in parallel or in reverse order.

[0219] [3D Video] This section describes the field of immersive video streaming, also known as 3D (three-dimensional) video or volumetric video. Through immersive video streaming, viewers can experience six degrees of freedom (6DoF) within a video scene, enabling them to move and rotate in any direction. This technology is used in interactive web viewers or virtual reality (VR) glasses.

[0220] To achieve a fully immersive experience, the scene needs to be rendered at high speed, ideally, and in real-time, based on the viewer's position and gaze direction. However, immersive video streaming presents several technical challenges, including significant bandwidth and computational resource requirements.

[0221] For example, in virtual reality experiences, higher resolution and frame rates are needed to create realistic and immersive environments. In immersive videos, information from multiple viewpoints needs to be encoded, resulting in considerably larger file sizes.

[0222] Furthermore, dynamic streaming is essential for adjusting video quality based on the viewer's orientation and focus, and for adapting to various network conditions. Therefore, effective compression and streaming technologies are needed to improve the viewer's experience while reducing throughput.

[0223] Thanks to advancements in recent years, various methods have been introduced to efficiently represent 3D scenes using feature planes and neural networks trained specifically for each scene. These methods are able to efficiently encode 3D video data with small file sizes while maintaining high-quality rendered images.

[0224] Examples of immersive audiovisual experiences include sporting events, concerts, and reality shows. In sporting events, viewers can experience movements from various viewpoints within a stadium or arena, becoming immersed in the live sporting event. In concerts, the audience can virtually exist in the center of the performance, enjoying an immersive experience. In reality shows, viewers can explore the environment from various viewpoints, thus enhancing their experience.

[0225] Furthermore, immersive audiovisual experiences can be applied to games or interactive entertainment. In immersive gaming experiences, for example, players can move within a three-dimensional environment or interact with three-dimensional characters in a virtual world. Additionally, in interactive narrative experiences, viewers can influence the plot's progression by making choices that impact the story.

[0226] Furthermore, immersive audiovisual experiences can be applied in education, design, and training. Immersive learning environments provide students with interactive experiences that enhance comprehension and retention of learning materials. In the design field, architects, interior designers, and engineers can use immersive technologies to visualize and manipulate 3D models of projects. Additionally, in the training field, immersive simulations can provide realistic training scenarios for various professions, including healthcare, military, and aviation.

[0227] Furthermore, immersive audiovisual experiences can be applied to architecture or heritage. Through immersive architectural visualization, clients can experience proposed designs before construction begins, explore virtual buildings, evaluate spatial relationships, and provide feedback to architects or developers. Additionally, immersive experiences allow users to explore historical sites or relics in detail, protecting cultural heritage and promoting virtual tourism.

[0228] Furthermore, as a method for streaming immersive videos, one approach is to encode and decode only the spatial information within the feature planes of each frame of the 3D video and treat it as a static scene. However, this method cannot flexibly utilize the inherent temporal redundancy in 3D videos, hindering effective data compression.

[0229] In this embodiment, feature planes containing temporal information are encoded as images using existing video codecs (image coding standards) and transmitted. Furthermore, an optimal packing method for feature planes that minimizes latency while maintaining high quality is described. Additionally, supplementary information sent to the decoding device for reconstructing a 3D scene is explained. Thus, a highly versatile decoding device capable of rendering various 3D scenes can be provided.

[0230] [Resolution of 3D video] Figure 13 This is a diagram illustrating an example of the resolution of a dynamic 3D scene along the X, Y, and Z axes. A bounding box defines the boundary of a 3D region in space that includes the target scene captured throughout the entire duration of the dynamic 3D scene. Specifically, the bounding box is composed of... Figure 13 The smallest coordinate point (in meters) of the bounding box of the 3D scene shown is (X... min Y min Z min The maximum coordinates (in meters) of the bounding box of the 3D scene and the bounding box of the 3D scene, i.e., (X). max Y max Z max Defined by ).

[0231] In addition, set the resolution (in pixels) of the 3D scene along the X-axis, i.e., W. X The resolution (in pixels) of the 3D scene along the Y-axis, i.e., W. Y The resolution (in pixels) of the 3D scene along the Z-axis, i.e., W. Z .

[0232] Here, the aforementioned information from the bounding box is used for effective reconstruction in the decoding device. This information is necessary for both the scene at the center of the object and the scene facing forward. In the scene at the center of the object, the camera typically focuses on a specific object or point of interest within the scene, and in most cases, rotates around it to display various viewpoints of the object.

[0233] In a forward-facing scene, the camera pose is characterized by a viewing direction that is roughly the same as the environment, and mainly involves only parallel movement of the camera pose.

[0234] Figure 14 This is a diagram illustrating examples of multiple scales used to capture different levels of detail in a dynamic 3D scene. The scale is a parameter representing the resolution, defined by a one-dimensional arrangement of scale resolution multipliers (e.g., [1, 2, ..., N]). Additionally, the scale when the scale resolution multiplier = N will also be denoted as "scale N" below.

[0235] For example, the scale resolution multiplier is a power of 2, such as [1, 2, 4, 8]. However, the scale resolution multiplier is not limited to powers of 2. For example, the scale resolution multiplier can also be a floating-point number. In another example, only one scale exists; that is, only a scale resolution multiplier of 1 is possible.

[0236] Figure 15 This is a diagram representing an example of a dynamic 3D scene with multiple frames. The temporal aspect of the video depicting the dynamic 3D scene (represented by S) is expressed by the parameter W, which represents its temporal resolution. T And thus, it assigns features. In this representation, the initial frame of a dynamic 3D scene is aligned with the value 0, but the frame L of the video sequence... S Corresponding to W T L S The total number of frames in scene S, according to L S Decision W T The value. Additionally, a 3D scene refers to a series of continuous actions within a specific environment, comprising multiple consecutive frames.

[0237] Figure 16 This diagram illustrates an example of a 3D video sequence with multiple dynamic 3D scenes. Here, each frame can be identified by a combination of its scene number and the frame number within that scene. The resolution of a scene (e.g., W) is also considered. X W Y W Z W T The resolution is common throughout the entire period (all frames) of the scene. In other words, the resolution is set for each scene. Alternatively, the resolution can be common across multiple scenes. Furthermore, it can be combined... Figures 13-16 The example shown.

[0238] [Characteristic plane] The encoding device transforms the 3D scene into a feature plane and encodes the transformed feature plane. Figure 17 This is a diagram illustrating an example of a feature plane. A feature plane, for example, consists of multiple two-dimensional layers. Each layer, called a channel, is composed of multiple pixels arranged in a two-dimensional configuration. Each pixel in the feature plane represents multi-channel feature data, which is a one-dimensional arrangement of floating-point data. For example, the arrangement of pixel values ​​taken from all channels represents a feature quantity.

[0239] For example, a feature represents the presence of an object at a corresponding spatial location, the object's attribute value, or a combination thereof. Here, attribute values ​​include, for example, at least one of color, transmittance, and reflectance.

[0240] Furthermore, the height resolution W2 and width resolution W1 of the feature plane are obtained from the resolution information of the dynamic 3D scene. The number of channels C is determined, for example, based on the specific 3D scene representation method used. That is, the number of channels C is independent of the dynamic 3D scene and is common across multiple scenes.

[0241] For example, the values ​​stored in the cells of a feature plane comprising multiple channels are 32-bit or 64-bit floating-point values. As an example, the dimension of the feature plane is 32(C) × 64(W1) × 64(W2). Furthermore, the values ​​stored in the cells of the channels of the feature plane can be 32-bit or 64-bit fixed-point values, or floating-point or fixed-point values ​​exceeding 64 bits, or values ​​smaller than 32-bit floating-point or fixed-point values.

[0242] Figure 18 is a diagram illustrating examples of multiple feature transformations in a dynamic 3D scene. In Figure 18 In this context, we have multiple feature planes corresponding to multiple basic planes (XY, YZ, XZ, XT, YT, ZT) in a single frame at a single scale (scale=1). This representation is generated for each scale of a dynamic 3D scene, and six feature planes are generated at each scale.

[0243] As an example of mapping information from 4D (four-dimensional) points (x, y, z, t) in a 3D scene to mesh nodes on a feature plane, it involves simple projection and interpolation. Temporally independent feature planes (XY, YZ, XZ), which are independent of time, primarily contain information about the static parts of the 3D scene. For example, the feature values ​​of each pixel on a temporally independent feature plane are obtained by projecting objects (moving objects) onto the corresponding planes (XY, YZ, XZ), indicating whether an object exists at the corresponding location, the object's attribute values, or combinations thereof.

[0244] Furthermore, the time-dependent feature planes (XT, YT, ZT), which are time-dependent feature planes, contain dynamic information about the scene. For example, each pixel of the time-dependent feature plane represents a feature quantity projected onto the corresponding plane (XT, YT, ZT). This feature quantity indicates whether an object has moved, the amount of movement, the direction of movement, the difference, or a combination thereof.

[0245] Figure 19 This is a diagram representing an example of a group of time-independent characteristic planes. For example... Figure 19 As shown, a time-independent feature plane has dimensions of C × W1 × W2. Here, W1 and W2 are the products of the dimensions of the bounding box of the 3D scene at scale 1 along each axis and the scale resolution multiplier. For example, in the scale 1 feature plane of XY, W1 = W X W2=W Y In scale 2 of the characteristic plane of XY, W1 = 2 × Wx, W2 = 2 × W Y In the characteristic plane of XY at scale N, W1 = N × Wx, W2 = N × W Y .

[0246] In addition, such as Figure 19 As shown, the time-independent feature plane group includes the feature plane group of XY, the feature plane group of YZ, and the feature plane group of XZ. The feature plane groups of XY, YZ, and XZ each include feature planes of multiple scales (1 to N).

[0247] Figure 20 This is a diagram representing an example of a group of time-dependent feature planes. For example... Figure 20 As shown, the size of a time-dependent feature plane is C×W T ×W1. Here, W T Corresponding to the time dimension, W1 represents the total time length T of the 3D scene. W1 is the product of the dimensions of the bounding box at scale 1 of the 3D scene along each axis and the scale resolution multiplier. For example, at scale 1 of the feature plane in XT, W1 = Wx. At scale 2 of the feature plane in XT, W1 = 2 × Wx. At scale N of the feature plane in XT, W1 = N × Wx.

[0248] In addition, such as Figure 20 As shown, the groups of time-dependent feature planes include groups of feature planes for XT, YT, and ZT. Each group of feature planes for XT, YT, and ZT includes feature planes of multiple scales (1 to N).

[0249] In addition, Figure 20In this context, the temporal resolution scale is only one, but multiple scales can be applied to it. By adding a time dimension scale, the accuracy of the temporal resolution is improved. Furthermore, it enhances the accessibility of the video.

[0250] Figure 21 This is a graph representing examples of time-independent feature planes at multiple scales. Figure 22 This is a graph representing examples of time-dependent feature planes at multiple scales. For example... Figure 21 as well as Figure 22 As shown, changes in the 3D region of space (3D scene) are recorded as features across multiple feature planes with different resolutions at multiple scales. Figure 21 as well as Figure 22 This example uses M scales, resulting in M×3 time-independent feature planes and M×3 time-dependent feature planes. Furthermore, the scale multiplier for scale M is N.

[0251] Figure 23 This is a diagram illustrating an example of a 3D scene model. For example... Figure 23 As shown, by training (learning) the features of the feature plane and the weights of the neural network on a specific dynamic 3D scene, a model representing the 3D scene is generated.

[0252] A dynamic 3D scene with multiple frames is represented by a single model. The model contains temporally independent feature planes and neural network weights as static information. Additionally, the model contains temporally dependent feature planes as dynamic information. Temporally independent feature planes are spatially independent of time, while temporally dependent feature planes are temporally and spatially dependent. The neural network weights are the weights of the trained neural network. 3D scene parameters used to provide additional information to the model are associated with each model element.

[0253] Additionally, examples of using feature planes (time-independent feature planes and time-dependent feature planes) and neural network weights are shown here, but it is also possible to generate a model using only feature planes without using neural network weights.

[0254] Figure 24 This is a diagram representing another example of a 3D scene model. For example... Figure 24As shown, the model can also include voxel gratings as static information. A voxel grating is a 3D voxel grid with a smaller resolution compared to feature planes. For example, a voxel grating can contain information related to the density of the 3D scene. For instance, a voxel grating can represent the object space using an octree structure, with each leaf containing information indicating whether a 3D point is contained within that leaf, or the number of 3D points contained within that leaf. Using voxel gratings can significantly speed up the reconstruction process. Additionally, voxel gratings can also contain information related to movement within the 3D scene.

[0255] Figure 25 This diagram represents the rendering process performed by the rendering unit. The rendering unit generates a rendered image by using a model representing a dynamic 3D scene. Specifically, the rendering unit obtains the model (feature planes and neural network weights), 3D scene parameters, input viewpoint, and input time t. The viewpoint represents the camera's position and pose. The rendering unit uses the features of the feature planes and the neural network weights contained in the model to render (reconstruct) the rendered image observed from the input viewpoint at input time t. In this way, the model is used to generate a rendered image observed from any input viewpoint (free viewpoint).

[0256] More specifically, based on the input viewpoint, the four-dimensional coordinates (x, y, z, t) of the 3D model corresponding to the coordinates (x', y', t) in the rendered image plane are calculated. Feature quantities corresponding to these coordinates (x, y, z, t) are derived from the feature plane. Furthermore, color information, etc., is generated based on these feature quantities and a neural network using neural network weights, and reflected in the rendered image plane. By performing these processes on each pixel (coordinate) of the rendered image plane, the rendered image is generated.

[0257] Furthermore, the 3D scene parameters include parameters that provide the basic information required for rendering the 3D scene. For example, these parameters may include at least one of the following: parameters for recognizing the 3D video sequence, parameters for recognizing the scene, start frame number, end frame number, size of the 3D scene bounding box, feature plane mesh resolution, scale resolution multiplier, feature plane quantization parameters, start time, end time, and duration. Additionally, the model and 3D scene parameters are set during the training process described later.

[0258] Figure 26 This is a diagram representing an example of images used in the training process. Images captured from multiple viewpoints (camera positions and poses) of a dynamic 3D scene are used in the training process.

[0259] Figure 27This is a diagram illustrating an example of the model's training process (learning process). The features of the feature plane and the weights of the neural network are optimized based on the metering loss. Here, the metering loss refers to the difference in pixel values ​​between the correct image (the ground truth image) and the rendered image.

[0260] The rendering unit uses viewpoint information (camera position and pose), 3D scene parameters, and information from the model (color, etc.) from the training data (teacher data) to generate rendered images. Here, the 3D scene parameters are generated during training on a specific dynamic 3D scene. Furthermore, the model is initialized based on these 3D scene parameters. The difference between the rendered image and the correct image from the training data is calculated, and this difference is fed back into the model. As a result, the feature planes and neural network weights within the model are optimized.

[0261] Figure 28 This is a block diagram illustrating the structure of the encoding apparatus 300 according to the embodiment. A model of a 3D scene representation method trained on a 3D scene within a 3D video sequence is input to the encoding apparatus 300. The encoding apparatus 300 includes a transformation unit 301, an NNC encoding unit 302, a metadata transformation unit 303, and an image encoding unit 304.

[0262] The transform unit 301 extracts feature planes, neural network weights, and associated 3D scene parameters from the model. The transform unit 301 outputs the neural network weights to the NNC encoding unit 302.

[0263] The NNC (Neural Network Codec) encoding unit 302 compresses (encodes) the neural network weights to generate compressed neural network weights, and outputs the generated compressed neural network weights to the image encoding unit 304. For example, the NNC encoding unit 302 uses the NNC (Neural Network Coding) of the MPEG standard specification to compress the neural network weights. This enables efficient encoding of data related to the neural network. However, the method for compressing neural network weights is not limited to this.

[0264] Furthermore, the transformation unit 301 reconfigures (packs) the extracted feature planes onto the image and outputs the generated image to the image encoding unit 304. Additionally, the transformation unit 301 outputs packing information related to the reconfiguration to the metadata transformation unit 303. The transformation unit 301 updates the 3D scene parameters required for 3D scene reconstruction in the decoding device and outputs the updated 3D scene parameters to the metadata transformation unit 303.

[0265] The metadata transformation unit 303 transforms the 3D scene parameters and packing information into metadata (parameter set) that is sent along with the bitstream. That is, the metadata transformation unit 303 generates metadata that includes the 3D scene parameters and packing information.

[0266] The image coding unit 304 generates a bitstream by encoding the image, the weights of the compressed neural network, and metadata. That is, the bitstream contains the image (encoded image), the weights of the compressed neural network, and metadata. For example, the neural network weights are signaled as SEI (Supplemental Enhancement Information) or VUI (Video Usability Information) data.

[0267] For example, the image encoding unit 304 encodes the image using a known video codec. The structure of the image encoding unit 304 may also be the same as that of the encoding device 100 described above.

[0268] Figure 29 This is a block diagram illustrating the structure of the decoding apparatus 400 according to the embodiment. A bitstream is input to the decoding apparatus 400. For example, the bitstream is generated by the encoding apparatus 300 described above.

[0269] The decoding device 400 includes an image decoding unit 401, a metadata inverse transformation unit 402, an NNC decoding unit 403, an inverse transformation unit 404, and a rendering unit 405.

[0270] The image decoding unit 401 obtains the image, compressed neural network weights, and metadata by decoding the bitstream. For example, the image decoding unit 401 uses a known video codec to decode the image. The structure of the image decoding unit 401 may also be the same as that of the decoding device 200 described above.

[0271] The metadata inverse transformation unit 402 transforms the metadata into 3D scene parameters and packaging information. That is, the metadata inverse transformation unit 402 obtains the 3D scene parameters and packaging information from the metadata.

[0272] The NNC decoding unit 403 generates neural network weights by decompressing (decoding) the compressed neural network weights. The inverse transform unit 404 uses the packing information to transform the image into a feature plane.

[0273] The rendering unit 405 uses neural network weights, feature planes, and 3D scene parameters to generate or update the model. Furthermore, the rendering unit 405 uses the model to generate a rendered image as an image observed from an input viewpoint requested by the software or the user.

[0274] Figure 30 This is a block diagram representing another structural example of an encoding device. Figure 30 The encoding device 310 shown and Figure 28 The difference in the encoding device 300 shown is that the weights of the compressed neural network are stored in metadata.

[0275] The encoding device 310 is input with a model of a 3D scene representation method trained on a 3D scene within a 3D video sequence. The encoding device 310 includes a transformation unit 311, an NNC encoding unit 312, a metadata transformation unit 313, and an image encoding unit 314.

[0276] The transformation unit 311 extracts feature planes, neural network weights, and 3D scene parameters from the model. The transformation unit 311 outputs the neural network weights to the NNC encoding unit 312. Furthermore, the transformation unit 311 reconfigures (packs) the extracted feature planes onto the image and outputs the generated image to the image encoding unit 314. Additionally, the transformation unit 311 outputs packing information related to the reconfiguration to the metadata transformation unit 313. The transformation unit 311 updates the 3D scene parameters required for 3D scene reconstruction in the decoding device and outputs the updated 3D scene parameters to the metadata transformation unit 313.

[0277] The NNC encoding unit 312 generates compressed neural network weights by compressing the neural network weights, and outputs the generated compressed neural network weights to the metadata transformation unit 313.

[0278] The metadata transformation unit 313 transforms the 3D scene parameters, compressed neural network weights, and packaging information into metadata. For example, the compressed neural network weights are packaged and stored in the URL, and the URL information representing the URL is included in the metadata.

[0279] The image encoding unit 314 generates a bitstream by encoding the image and metadata. That is, the bitstream contains the image (encoded image) and metadata. For example, the image encoding unit 314 encodes the image using a known video codec. The structure of the image encoding unit 314 may also be the same as that of the encoding device 100 described above.

[0280] Figure 31 This is a block diagram illustrating another structural example of a decoding device. A bitstream is input to the decoding device 410. For example, the bitstream is generated by the encoding device 310 described above.

[0281] The decoding device 410 includes an image decoding unit 411, a metadata inverse transformation unit 412, an NNC decoding unit 413, an inverse transformation unit 414, and a rendering unit 415.

[0282] The image decoding unit 411 obtains the image and metadata by decoding the bitstream. The metadata inverse transformation unit 412 transforms the metadata into 3D scene parameters, packing information, and compressed neural network weights. For example, the image decoding unit 411 uses a known video codec to decode the image. The structure of the image decoding unit 411 may also be the same as that of the decoding device 200 described above.

[0283] The NNC decoding unit 413 generates neural network weights by decompressing (decoding) the compressed neural network weights. The inverse transform unit 414 uses the packing information to transform the image into a feature plane.

[0284] The rendering unit 415 uses neural network weights, feature planes, and 3D scene parameters to generate or update the model. Furthermore, the rendering unit 415 uses the model to generate a rendered image as an image observed from an input viewpoint requested by the software or the user.

[0285] Furthermore, when the URL information indicating the storage device containing encoded data such as neural network weights is included in the metadata, the decoding device 410 obtains the encoded data such as compressed neural network weights from the URL indicated by the URL information. The NNC decoding unit 413 generates neural network weights by decoding the obtained compressed neural network weights. In this way, the encoded data of the neural network weights (compressed neural network weights) is not appended to the bitstream, but transmitted to the decoding device via a URL or other means, thereby reducing the size of the bitstream.

[0286] [First Method] The decoding process performed by the decoding device involved in the first method will be described below. Figure 32 This is a flowchart of the decoding process performed by the decoding device involved in the first method.

[0287] First, the decoding device decodes the image and metadata from the bitstream (S101). Furthermore, the decoding device obtains the packing information and 3D scene parameters from the metadata.

[0288] Next, the decoding device transforms the image into a feature map image (S102). The feature map image has a higher bit depth than the decoded image. For example, the decoding device generates high-bit image data by inverse quantizing low-bit image data. For example, by inverse quantizing a 10-bit integer value (which is a low bit), a 32-bit floating-point value (which is a high bit) is generated. In other examples, a 32-bit floating-point value is generated by inverse quantizing a 12-bit or 8-bit integer value. Inverse quantization is, for example, a left shift operation used to improve precision. Furthermore, the 3D scene parameters include feature plane quantization parameters used to determine the inverse quantization step (quantization step). The decoding device performs inverse quantization using the step represented by the feature plane quantization parameters.

[0289] Next, the decoding device uses the packing information to unpack multiple segments of the feature plane from the feature map image (S103).

[0290] Next, the decoding device updates the model using multiple segments of the feature planes (S104). The model contains multiple previously acquired feature planes, and new feature planes are added through the update.

[0291] Next, the decoding device uses the updated model to generate a rendered image at the specified viewpoint and time (S105).

[0292] Furthermore, this example illustrates an instance where the image contained in the bitstream is quantized, but the image contained in the bitstream may also not be quantized. In this case, the decoding device does not perform step S102, and in step S103, it unpacks multiple segments of the feature plane from the image obtained from the bitstream.

[0293] Figure 33 This is a diagram illustrating an example of packing (reconfiguring) feature planes across multiple images. Figure 33 In the example shown, the image resolution is determined based on the size of the time-dependent feature plane data. Specifically, the large amount of time-independent feature plane data is segmented and packed into multiple frames.

[0294] exist Figure 33 In the example, there exists one time-independent feature plane data and x time-dependent feature plane data (time-dependent feature plane data 1 to x). The multiple time-dependent feature plane data correspond to different times or time intervals.

[0295] Here, time-independent feature plane data is typically larger than time-dependent feature plane data. Therefore, in Figure 33 In the previous section, the image resolution (size) is set to w1×h1 so that time-dependent feature plane data is configured in one frame (image), while time-independent feature plane data is sent as a large frame. Furthermore, x time-dependent feature plane data are sent as separate frames. In this case, each frame of time-dependent feature plane data contains large padding data.

[0296] On the other hand, Figure 33 In the next section, the image resolution is set to w2×h2 to configure a time-dependent feature plane data. In this case, the time-independent feature plane data is segmented into y frames. Furthermore, x time-dependent feature plane data are transmitted as different frames. This allows for the reduction of padding data contained in the frames of the time-dependent feature plane data. In this case, y+x frames from frame m to frame m+y+x-1 are contained in the bitstream.

[0297] Furthermore, information indicating whether the information contained in each frame is time-independent or time-dependent feature plane data can also be included in the bitstream. For example, this information can be appended to metadata or the SEI, VUI, or header information of the image coding standard. Thus, the decoding device can identify whether the decoded frame is time-independent or time-dependent feature plane data. Therefore, the decoding device can use the decoded time-independent or time-dependent feature plane data to appropriately update the model.

[0298] Figure 34 This is another example of packing feature planes into multiple images. Figure 34 In the example shown, with Figure 33 Similarly, the example shown has one time-independent feature plane data and x time-dependent feature plane data (time-dependent feature plane data 1 to x).

[0299] exist Figure 34 In the example shown, the image resolution is set to w1×h1 so that time-dependent feature plane data is configured in one frame, while time-independent feature plane data is sent as a large frame.

[0300] Furthermore, the x time-dependent feature planes are combined and packaged into a single frame. In this case, with Figure 33 Compared to the previous method, this method has the advantage of reducing the number of frames. Furthermore, frames m and m+1 are included in the bitstream.

[0301] Additionally, information indicating whether the data contained in each frame is time-independent or time-dependent feature plane data can also be included in the bitstream. For example, this information can also be appended to metadata or the SEI, VUI, or header information of the image coding standard.

[0302] In addition, information indicating the number of temporally dependent feature plane data points contained in a frame and the intra-frame mapping start position of each temporally dependent feature plane data point (e.g., Figure 34 Information such as the start positions of data 1 to x shown can also be included in the bitstream. For example, this information can also be appended to metadata or the SEI, VUI, or header information of an image coding standard. Thus, when the decoded frame contains multiple time-dependent feature plane data, the decoding device can appropriately obtain multiple time-dependent feature plane data by using information indicating the start position of the mapping to extract each time-dependent feature plane data.

[0303] Additionally, in the initial data (e.g.) Figure 34If the starting position of data 1) is a predetermined position such as (0, 0) of the image, the information indicating the starting position of the initial data may not be included in the bitstream.

[0304] Figure 35 This is another example of packing feature planes into multiple images. Figure 35 In the example shown, with Figure 33 Similarly, the example shown has one time-independent feature plane data and x time-dependent feature plane data (time-dependent feature plane data 1 to x).

[0305] exist Figure 35 In the example shown, the time-dependent feature plane data contains time-dependent feature plane data at multiple scales (1 to N). Furthermore, with... Figure 33 Similarly, in the example shown, the image resolution is set to w2×h2 to allow configuration of a time-dependent feature plane data. Furthermore, the time-independent feature plane data is segmented into y+1 frames. x time-dependent feature plane data are transmitted as separate frames.

[0306] Furthermore, in Figure 35 In the example shown, low-scale temporally independent feature plane data is initially encoded. That is, low-scale temporally independent feature plane data is configured in the first y+1 frames of the frames containing temporally independent feature plane data. Furthermore, high-scale temporally independent feature plane data is segmented and configured in the last y frames of the y+1 frames containing temporally independent feature plane data.

[0307] Here, low-scale temporally independent feature plane data refers to one or more scales on the lower scale side of scales 1 to N. High-scale temporally independent feature plane data refers to one or more scales on the higher scale side of scales 1 to N, such as scales other than those included in the low-scale temporally independent feature plane data of scales 1 to N. For example, low-scale temporally independent feature plane data includes scales 1 to 2, while high-scale temporally dependent feature plane data includes the remaining scales.

[0308] Therefore, it has the advantage of being able to quickly reconstruct the low-scale data contained in the initial frame of a 3D scene. Furthermore, frames m through m+y+x are contained in the bitstream.

[0309] Additionally, information indicating whether the data contained in each frame is time-independent or time-dependent feature plane data can also be included in the bitstream. For example, this information can also be appended to metadata or the SEI, VUI, or header information of the image coding standard.

[0310] Furthermore, information indicating the scale of the temporally independent feature plane data contained in each frame can also be included in the bitstream. For example, this information can be appended to metadata or the SEI, VUI, or header information of the image coding standard. Thus, when the decoded frame contains temporally independent feature plane data, the decoding device can identify which scale of temporally independent feature plane data the decoded frame represents. Therefore, the decoding device can appropriately acquire multi-scale temporally independent feature plane data.

[0311] Figure 36 This is another example of packing feature planes into multiple images. Figure 36 In the example shown, with Figure 35 Similarly, the example shown has one time-independent feature plane data and x time-dependent feature plane data (time-dependent feature plane data 1 to x). The time-dependent feature plane data contains time-dependent feature plane data at multiple scales (1 to N). Furthermore, the image resolution w2×h2 is set to be smaller than the resolution w1×h1.

[0312] exist Figure 36 In the example shown, both time-independent and time-dependent feature plane data are segmented into multiple frames. Furthermore, low-scale time-independent and time-dependent feature plane data are initially encoded. Specifically, the low-scale time-independent and time-dependent feature plane data are segmented into u frames. For example, the low-scale time-independent feature plane data is segmented into u, and the time-dependent feature plane data 1 to x are also segmented into u. Each frame stores one of the u segmented low-scale time-independent feature plane data and the u segmented time-dependent feature plane data. Additionally, the high-scale time-independent feature plane data is segmented into v and stored in v frames.

[0313] Therefore, by configuring low-scale time-independent feature plane data and time-dependent feature plane data in the preceding frames, dynamic scenes can be reproduced at high speed.

[0314] Additionally, information indicating whether a frame contains time-independent feature plane data, time-dependent feature plane data, or both can also be included in the bitstream. For example, this information can also be appended to metadata or the SEI, VUI, or header information of an image coding standard.

[0315] Furthermore, in the case where a frame contains two sides, information indicating the number of time-independent or time-dependent feature plane data points contained in the frame, and information indicating the start position of the mapping within the frame for each data point (e.g., Figure 36The information at start position A and start position B (as shown) can also be included in the bitstream. For example, this information can also be appended to metadata or the SEI, VUI, or header information of an image encoding standard. Thus, when the decoded frame contains multiple data, the decoding device can appropriately obtain multiple data by using the information indicating the start position of the mapping.

[0316] Additionally, at the beginning position of the initial data (e.g.) Figure 36 If the start position A) is a predetermined position such as (0, 0) of the image, the information indicating the start position of the initial data may not be included in the bitstream.

[0317] Figure 37 This is another example of packing feature planes into multiple images. Figure 37 In the example shown, two scales, 1 and 2, of the time-independent feature plane are reconfigured in the YUV420 format image. Furthermore, the feature plane has C channels, which are configured in two frames, m and m+1. Here, the time-independent feature plane can be data corresponding to one feature plane (e.g., any one of XY, YZ, and XZ) or data corresponding to multiple feature planes (e.g., more than two of XY, YZ, and XZ).

[0318] Specifically, channels 1 to k of scale 2 are configured in the Y component of frame m, and channels k+1 to C of scale 2 are configured in the Y component of frame m+1. Furthermore, channels 1 to k of scale 1 are configured in the U component of frame m, and channels k+1 to C of scale 1 are configured in the U component of frame m+1.

[0319] Here, in the YUV420 format, a 2×2 pixel in the Y component corresponds to a 1 pixel in the U or V component. This is the same relationship between the sizes of the two scales when the scale resolution multiplier is a power of 2. Therefore, by setting it to the above configuration, the data of the two scales can be effectively reconfigured. Furthermore, multiple channels of the feature plane are arranged adjacently in a rectangular manner. As a result, parallel decoding using tiles, slices, or sub-images can be implemented.

[0320] Furthermore, by dividing the multiple channels contained in the feature plane into multiple frames, the frame resolution can be reduced. Thus, for example, the feature plane can be encoded at resolutions below those supported by existing image coding standards, thereby enabling feature plane encoding processing to be implemented using existing image coding standards.

[0321] Additionally, information indicating the dimension of the channels contained in the frame (e.g., in...) Figure 37In the example shown, information representing k), information representing the scale of the feature plane data contained in the frame, or mapping information representing the scale of the data contained in each component of the YUV component can also be included in the bitstream. For example, this information can also be appended to metadata or the SEI, VUI, or header information of the image coding standard. Thus, the decoding device can use the mapping information to appropriately obtain the feature planes from the decoded multiple frames.

[0322] In addition, Figure 37 The example shown illustrates a configuration with two scales, but the number of scales can also be three or more.

[0323] Figure 38 This is another example of packing feature planes into multiple images. Figure 38 In the example shown, the four scales 1 to 4 of the time-independent feature plane are reconfigured in the YUV420 format image. Furthermore, the feature plane has C channels, which are configured in two frames, m and m+1. In this case, the same effect as described above can be achieved.

[0324] In addition, when configuring more than five scales, for example, the largest scale is configured in the Y component, the second largest scale is configured in one of the U or V components, and the remaining scales are configured in the other of the U or V components.

[0325] Furthermore, this example illustrates C channels configured separately across two frames, but C channels can be configured in one frame or split across three or more frames.

[0326] Figure 39 This is another example of packing feature planes into multiple images. Figure 39 In the example shown, the two scales 1 and 2 of the temporally independent feature plane are reconfigured in the YUV420 format image. Furthermore, the temporally independent feature planes include the XY feature plane, the YZ feature plane, and the XZ feature plane. Moreover, the XY, YZ, and XZ feature planes are configured in different frames.

[0327] In this example, all channels of the feature plane are packed into one frame. Specifically, all channels of scale 2 (channels 1 to C) of the XY feature plane are configured in the Y component of frame m, and all channels of scale 1 of the XY feature plane are configured in the U component of frame m. All channels of scale 2 of the YZ feature plane are configured in the Y component of frame m+1, and all channels of scale 1 of the YZ feature plane are configured in the U component of frame m+1. All channels of scale 2 of the XZ feature plane are configured in the Y component of frame m+2, and all channels of scale 1 of the XZ feature plane are configured in the U component of frame m+2.

[0328] Furthermore, multiple channels of the feature plane are arranged adjacently in a rectangular configuration. This enables parallel decoding using tiles, slices, or sub-images.

[0329] In this way, by storing multiple channels contained in a feature plane (XY, YZ, or XZ) in a single frame, the number of frames to be encoded can be reduced. Consequently, feature planes can be encoded, for example, at frame rates below those supported by existing image coding standards, thus enabling the encoding of feature planes using existing image coding standards.

[0330] Furthermore, information indicating which of the XY, YZ, or XZ feature planes a frame contains can also be included in the bitstream. For example, this information can be appended to metadata or the SEI, VUI, or header information of an image coding standard. Thus, the decoding device can identify which of the XY, YZ, or XZ feature planes the decoded frame belongs to. Therefore, the decoding device can, for example, use the decoded XY, YZ, or XZ feature plane to appropriately update the model.

[0331] In addition, Figure 39 The example shown illustrates a configuration with two scales, but the number of scales can also be three or more.

[0332] Figure 40 This is another example of packing feature planes into multiple images. Figure 40 In the example shown, the four scales 1–4 of the temporally independent feature planes are reconfigured in a YUV420 format image. Furthermore, the temporally independent feature planes include the XY feature plane, the YZ feature plane, and the XZ feature plane. Moreover, the XY, YZ, and XZ feature planes are configured in different frames.

[0333] Specifically, all channels (channels 1 to C) of scale 4 in the XY feature plane are configured in the Y component of frame m, all channels of scale 3 in the XY feature plane are configured in the U component of frame m, and all channels of scales 2 and 1 in the XY feature plane are configured in the V component of frame m. All channels of scale 4 in the YZ feature plane are configured in the Y component of frame m+1, all channels of scale 3 in the YZ feature plane are configured in the U component of frame m+1, and all channels of scales 2 and 1 in the YZ feature plane are configured in the V component of frame m+1. All channels of scale 4 in the XZ feature plane are configured in the Y component of frame m+2, all channels of scale 3 in the XZ feature plane are configured in the U component of frame m+2, and all channels of scales 2 and 1 in the XZ feature plane are configured in the V component of frame m+2. In this case, the same effect as described above can be achieved.

[0334] In addition, when configuring more than five scales, for example, the largest scale is configured in the Y component, the second largest scale is configured in one of the U or V components, and the remaining scales are configured in the other of the U or V components.

[0335] Figure 41 This is another example of packing feature planes into multiple images. Figure 41 In the example shown, the two scales 1 and 2 of the time-independent feature plane are reconfigured in the YUV420 format image. Furthermore, the time-independent feature planes include the XY feature plane, the YZ feature plane, and the XZ feature plane. Each feature plane also has C channels.

[0336] In this example, the XY feature plane, YZ feature plane, and XZ feature plane are stored in one frame. Specifically, the scale 2 of the XY feature plane, YZ feature plane, and XZ feature plane is stored in the Y component, and the scale 1 of the XY feature plane, YZ feature plane, and XZ feature plane is stored in the U component.

[0337] Furthermore, while an example of storing all feature planes of time-independent feature planes in a single frame has been provided, similarly, all feature planes of time-dependent feature planes can also be stored in a single frame.

[0338] Furthermore, multiple channels of the feature plane are arranged adjacently in a rectangular configuration. This enables parallel decoding using tiles, slices, or sub-images. Additionally, in this example, the feature plane data is positioned on the left side of the image, thus speeding up the decoding process.

[0339] Additionally, information indicating which of XY, YZ, XZ, or combinations thereof is included in the frame can also be contained in the bitstream. For example, this information can be appended to metadata or the SEI, VUI, or header information of the image coding standard. Furthermore, in the case where the frame contains XY, YZ, and XZ, the starting position of the mapping within the frame indicating each feature plane data (e.g., Figure 41 The information indicating the start positions (A-F) can also be included in the bitstream. For example, this information can also be appended to metadata or the SEI, VUI, or header information of the image coding standard. Thus, when the decoded frame contains XY, YZ, and XZ, the decoding device can use the information indicating the start position of the mapping to appropriately obtain the data of the XY feature plane, YZ feature plane, and XZ feature plane.

[0340] Additionally, at the beginning position of the initial data (e.g.) Figure 41If the start position A and start position D are predetermined positions such as (0, 0) of the image, the information indicating the start position of the initial data may not be included in the bitstream.

[0341] Furthermore, the mapping methods for XY, YZ, and XZ are not limited to those described above. For example, mapping can be performed on the image in different orders such as YZ, XZ, and XY. Moreover, the starting position does not necessarily have to be a pixel at the end of the frame; the starting position can be set at any location. Therefore, the encoding device can flexibly map multiple feature planes. For example, by performing mapping, the encoding device can improve the encoding efficiency of image encoding, thereby reducing the amount of data.

[0342] Figure 42 This is another example of packing feature planes into multiple images. Figure 42 In the example shown, time-dependent feature planes are reconfigured within a frame in YUV420 format. These time-dependent feature planes include the XT, YT, and ZT feature planes. The XT, YT, and ZT feature planes are stored within the Y component. Furthermore, each feature plane contains N scales from scale 1 to N.

[0343] Furthermore, the number of channels C is represented by p × q. In this example, p = 2, q = 8, and the number of channels C is p × q = 16. In this case, the height of the configured scale N is k × q, and the width is N × W. X ×p. k corresponds to the temporal resolution of one channel and the number of frames contained in the 3D scene. N is the scale resolution multiplier corresponding to the largest scale N. Additionally, if lower-level scales are also included, they can be configured in the U or V components.

[0344] Additionally, information indicating which of XT, YT, XT, or combinations thereof is included in the frame can also be contained in the bitstream. For example, this information can be appended to metadata or the SEI, VUI, or header information of the image coding standard. Furthermore, in the case where the frame contains XT, YT, and XT, the starting position of the mapping within the frame for each feature plane (e.g., Figure 42 The information indicating the start positions (A-C) can also be included in the bitstream. For example, this information can also be appended to metadata or the SEI, VUI, or header information of the image coding standard. Thus, when the decoded frame contains XT, YT, and XT, the decoding device can use the information indicating the start position of the mapping to appropriately obtain the data of the XT feature plane, the YT feature plane, and the XT feature plane.

[0345] Furthermore, the mapping methods for XT, YT, and XT are not limited to those described above. For example, mapping can be performed from the image in different orders such as YT, ZT, and XT. Moreover, the starting position does not necessarily have to be a pixel at the end of the frame; the starting position can be set at any location. Therefore, the encoding device can flexibly map multiple feature planes. For example, by performing mapping, the encoding device can improve the encoding efficiency of image encoding, thereby reducing the amount of data.

[0346] Figure 43 This is another example of packing feature planes into multiple images. Figure 43 In the example shown, time-independent and time-dependent feature planes are reconfigured within a frame in YUV420 format. The time-independent feature planes comprise two scales, 1 and 2. These include the XY, YZ, and XZ feature planes. The time-dependent feature planes include the XT, YT, and ZT feature planes. The time-independent feature planes at scale 2 are stored in the Y component. The time-independent feature planes at scale 1 are stored in the Y component.

[0347] In addition, Figure 43 The example shown is a configuration containing two time-independent feature planes of different scales, but the number of scales can also be more than three.

[0348] Furthermore, information indicating which of XY, YZ, XZ, XT, YT, XT, or combinations thereof is contained in the frame can also be included in the bitstream. For example, this information can be appended to the metadata or the SEI, VUI, or header information of the image coding standard. Additionally, when the frame contains XY, YZ, XZ, XT, YT, and XT, information indicating the mapping start position information within the frame for each feature plane can also be included in the bitstream. For example, this information can also be appended to the metadata or the SEI, VUI, or header information of the image coding standard. Therefore, when the decoded frame contains XY, YZ, XZ, XT, YT, and XT, the decoding device can use the information indicating the mapping start position to appropriately obtain the feature plane data for XY, YZ, XZ, XT, YT, and ZT.

[0349] Furthermore, the mapping methods for XY, YZ, XZ, XT, YT, and XT are not limited to those described above. For example, mapping can be performed on the image in different orders such as YZ, YT, XZ, ZT, XY, and XT. Moreover, the starting position does not necessarily have to be a pixel at the end of the frame; the starting position can be set at any location. Therefore, the encoding device can flexibly map multiple feature planes. For example, by performing mapping, the encoding device can improve the encoding efficiency of image encoding, thereby reducing the amount of data.

[0350] Figure 44 It means in Figure 32 The diagram illustrates an example of time-independent feature plane data unpacking from the feature map image to the feature plane in step S103. Multiple rectangular segments of the feature plane are stacked to form a feature plane with C channels. Furthermore, this processing is performed at each scale. Figure 44 Examples representing scale 1 and scale N.

[0351] Furthermore, these multiple rectangular segments are received via more than one frame. In this example, the received feature plane data from C channels are received as p columns and q rows of rectangular image segments in a p×q=C manner.

[0352] Figure 45 It means in Figure 32 The diagram illustrates an example of unpacking segments of time-dependent feature plane data into a feature plane in step S103. In this example, the time-dependent feature plane comprises scales 1 to N. Each scale comprises channels 1 to C. Multiple rectangular segments of the feature plane are stacked to form a feature plane with C channels.

[0353] Here, Figure 45 The k shown corresponds to the number of frames contained in the 3D scene. That is, when k is small, there is the advantage of being able to quickly decode the 3D scene and begin playback. Furthermore, when k is large, it is possible to encode 3D scenes of longer durations.

[0354] Figure 46 It means Figure 32 A diagram illustrating an example of updating the dynamic 3D representation model in step S104. (See diagram for example.) Figure 46 As shown, the received temporally independent feature plane data is used to update the temporally independent feature planes in the model. The received temporally dependent feature plane data is used to update the temporally dependent feature planes in the model. The received neural network weights are used to update the neural network weights in the model.

[0355] Furthermore, model updates can be performed at multiple scales simultaneously or on a per-scale basis. For example, the lower-level scales can be updated initially. This allows for faster reconstruction of the initial frames of the 3D scene. For instance, the model update is performed by rendering units 405 or 415.

[0356] Figure 47 It means Figure 32 The diagram shows an example of a time-dependent feature plane data update model used in the decoding device in step S104. Figure 47As shown, the model is updated by appending newly received time-dependent feature plane data to the previously received time-dependent feature plane data. In this way, model updates are performed in stages within the rendering unit. This provides the advantage of being able to update the initial frame initially and begin reproducing dynamic 3D scenes more quickly.

[0357] In addition, the model obtained by the decoding device based on previously received model data may also include a model initialized by default values.

[0358] Figure 48 This diagram illustrates another example of packaging a feature plane across multiple images. In this example, multiple segments of the feature plane are categorized based on their level of interest. Then, in the initial frame, segments of high-interest regions (or foreground regions) are sent preferentially compared to segments of low-interest regions (or background regions). In this example, information about the high-interest region is retained in the lower left corner of the feature plane.

[0359] Specifically, the segments with the highest attention are configured in the initial frame M, the segments with the second highest attention are configured in the subsequent frame M+1, the segments with the third highest attention are configured in the subsequent frames M+2 and M+3, and the segments with the lowest attention are configured in the subsequent frames M+4 and M+5. In this way, segments of each level of attention are concentrated in one or more frames. Furthermore, the frames containing the segments with high attention are encoded (transmitted) first. This provides the advantage that the decoding device can quickly reconstruct specific areas of the 3D scene that the user is interested in displaying.

[0360] The following describes an encoding apparatus for generating a bitstream that is decoded by the decoding apparatus described in the first method above. Figure 49 This is a flowchart of the encoding process performed by the encoding device involved in the first method.

[0361] First, the encoding device extracts multiple segments of the feature planes from the model, which includes multiple feature planes (S201).

[0362] Next, the encoding device generates a feature map image by packing (reconfiguring) multiple segments of the feature plane. Furthermore, the encoding device stores the reconfiguration order and start address of the multiple segments of the feature plane in metadata (S202). At this time, the encoding device, for example, configures padding data in areas outside the feature plane data within the image.

[0363] Next, the encoding device quantizes the high-bit feature map image to transform it into a low-bit-depth image (S203). The low-bit-depth image has a lower bit depth than the feature map image.

[0364] For example, the encoding device quantizes the 32-bit floating-point data, which is the high bit, to obtain a 10-bit integer value. Alternatively, the encoding device quantizes the 32-bit floating-point value to obtain a 12-bit or 8-bit integer value. Quantization processing includes, for example, a process of reducing precision through a right shift operation. Furthermore, the 3D scene parameters include feature plane quantization parameters used to determine the quantization steps.

[0365] Next, the encoding device encodes the low bit depth image and metadata into a bit stream (S204). That is, the bit stream contains the low bit depth image and metadata.

[0366] Furthermore, while an example of quantizing the feature map image is shown here, the feature map image may also not be quantized. In this case, the encoding device does not perform step S203, and the feature map image is encoded in step S204.

[0367] Figure 50 This is a diagram illustrating an example of packing feature planes into a feature map image. Figure 50 Multiple channels 1 to C of the feature plane shown are configured in a low bit depth image. Furthermore, the encoding device generates packing information related to packing and stores the generated packing information in the bitstream.

[0368] like Figure 50 As shown, the packing information includes (XY_N) as an ID for identifying the feature plane. This ID indicates the category (XY, YZ, XZ, XT, YT, ZT) and scale of the feature plane. For example, (XY_N) indicates that the data in this frame is data of scale N of the XY feature plane.

[0369] Furthermore, (u, v) represents the start position of the data within the frame; in this example, it represents the pixel position (coordinates) as the start position. In this example, it represents... Figure 50 The starting position is shown as P11.

[0370] (1, C) indicates the number of channels contained in the data. In this example, it means that the data contains channels 1 to C.

[0371] (p, q) represents the configuration of data for multiple channels. Specifically, it means that multiple channels are configured with p rows and q columns.

[0372] (0) indicates the configuration order of multiple channels. In this example, the configuration order is the raster scan order.

[0373] Figure 51 It means in Figure 49 A diagram illustrating an example of segmenting the time-dependent feature plane into segments in step S202. (See diagram for example.) Figure 51As shown, the time-dependent feature plane is segmented into multiple segments. Furthermore, this example illustrates that the segments have the same shape, but the segments can also have different shapes.

[0374] As described above, the encoding device involved in the first method transmits time-dependent feature plane data in a single frame. Furthermore, the encoding device divides large feature plane data into multiple segments for transmission. The decoding device uses the received segments to update the model, thereby immediately reconstructing the 3D scene. Additionally, the encoding device prioritizes transmitting the initial settings of the time-dependent feature planes and feature plane segments belonging to lower-level time-independent feature planes. Moreover, the encoding device combines multiple scales of time-dependent or time-independent feature planes into a single frame.

[0375] This improves the robustness of 3D scene reconstruction and reduces the processing load on the decoding device. Furthermore, low latency is achieved, allowing the decoding device to quickly reconstruct the initial frames of the 3D scene. This also enables the decoding device to begin reproducing the 3D scene earlier, thus providing a higher quality immersive experience.

[0376] Furthermore, this method can also be implemented by combining at least a portion of other methods disclosed herein. Moreover, this method can also be implemented by combining a portion of the process shown in any of these methods, a portion of the structure of any apparatus, a portion of a statement, etc., with other methods.

[0377] Furthermore, some or all of the processing in the aforementioned decoding apparatus can also be performed in the encoding apparatus in the same way. Moreover, not all of the constituent elements described in this approach are always necessary; sometimes only a portion of the constituent elements of the first approach are included in the apparatus.

[0378] Furthermore, while the above description illustrates an example of arranging feature planes in an image, the data that can be arranged is not limited to this. Any information contained in the model outside of feature planes can also be similarly arranged in the image. For example, this information could be information representing feature quantities with a time dimension.

[0379] Furthermore, in the above description, YUV components were used as examples of multiple components contained in an image, but multiple components are not limited to this. For example, multiple components could also be YCbCr, YPbPr, or RGB, etc.

[0380] As described above, the encoding device according to the embodiment performs... Figure 52The process is as follows: The encoding device obtains first information (e.g., multiple feature planes) representing multiple features with a time dimension contained in the model (S301), configures the first information on one or more images (S302), and generates a bitstream by encoding one or more images (S303). Thus, the encoding device can encode the first information representing multiple features with a time dimension contained in the model using image encoding. Therefore, the encoding processing of information contained in the model can be easily implemented.

[0381] For example, the first information contains multiple data (e.g., time-independent feature plane data and multiple time-dependent feature plane data), and the bitstream contains second information indicating the configuration order of the multiple data. Thus, the decoding device can use the second information to appropriately extract the multiple data from the image.

[0382] For example, the first information contains multiple data (e.g., time-independent feature plane data and multiple time-dependent feature plane data), and the bitstream contains third information indicating the start position of at least one of the multiple data in more than one image. Thus, the decoding device can use the third information to appropriately extract multiple data from the image.

[0383] For example, the first information includes multiple feature planes, each representing a multiple feature quantity configured in two dimensions. Therefore, the encoding device can use image encoding to encode the information of the multiple feature planes contained in the model. This allows for easy encoding of the information contained in the model.

[0384] For example, such as Figure 33 As shown, the first information includes: first data (e.g., time-independent feature plane data), containing multiple feature quantities without a time dimension; and multiple second data (e.g., time-dependent feature plane data), containing multiple feature quantities with a time dimension, the multiple second data corresponding to different time intervals. One or more images comprise multiple first images and multiple second images, with the first data segmented across the multiple first images and the multiple second data segmented across the multiple second images. Thus, the padding data contained in one or more images is reduced, thereby improving encoding efficiency.

[0385] For example, such as Figure 34As shown, the first information includes: first data (e.g., time-independent feature plane data), containing multiple feature quantities without a time dimension; and multiple second data (e.g., time-dependent feature plane data), containing multiple feature quantities with a time dimension, each corresponding to a different time interval. One or more images comprise a first image and a second image, with the first data configured in the first image and the multiple second data items centrally configured in the second image. This reduces the number of images to be encoded, thus improving encoding efficiency.

[0386] For example, such as Figure 35 as well as Figure 36 As shown, the first information includes first data (low-scale time-independent feature plane data) and second data (high-scale time-independent feature plane data) with a higher resolution than the first data. One or more images include a first image and a second image encoded after the first image. The first data is configured in the first image, and the second data is configured in the second image. Therefore, the decoding device can first decode the first data and use it to generate a model.

[0387] For example, such as Figures 37-41 As shown, the first information includes first data (low-scale time-independent feature plane data) and second data (high-scale time-independent feature plane data) with a higher resolution than the first data. Each of the more than one image includes a first component (e.g., a Y component) and a second component and a third component (e.g., a U component and a V component) smaller than the first component. The first data is configured in at least one of the second and third components, and the second data is configured in the first component. Thus, multiple data can be efficiently configured using multiple components.

[0388] In addition, the structure of the encoding device is, for example, similar to Figure 8 The encoding device shown is the same. The encoding device includes a processor (or circuit) and a memory, and the processor (or circuit) uses the memory to perform the above-mentioned processing during operation.

[0389] Furthermore, the decoding device involved in the implementation method performs... Figure 53 The processing is shown. The decoding device decodes one or more images from a bitstream generated by encoding one or more images containing first information in the model (S401), and obtains the first information (S402) from the one or more images. The first information (e.g., multiple feature planes) represents multiple feature quantities having a time dimension. Thus, the decoding device can use image encoding-based decoding processing to decode the first information in the model that represents multiple feature quantities having a time dimension. Therefore, the decoding processing of information contained in the model can be easily implemented.

[0390] For example, the first information contains multiple data (e.g., time-independent feature plane data and multiple time-dependent feature plane data), and the bitstream contains second information indicating the configuration order of the multiple data. Thus, the decoding device can use the second information to appropriately extract the multiple data from the image.

[0391] For example, the first information contains multiple data (e.g., time-independent feature plane data and multiple time-dependent feature plane data), and the bitstream contains third information indicating the start position of at least one of the multiple data in more than one image. Thus, the decoding device can use the third information to appropriately extract multiple data from the image.

[0392] For example, the first information includes multiple feature planes, each representing a multiple feature quantity configured in two dimensions. Therefore, the decoding device can use image-encoding-based decoding processing to decode the information of the multiple feature planes contained in the model. This allows for easy decoding of the information contained in the model.

[0393] For example, such as Figure 33 As shown, the first information includes first data (e.g., time-independent feature plane data), containing multiple feature quantities without a time dimension; and multiple second data (e.g., time-dependent feature plane data), containing multiple feature quantities with a time dimension, each of the multiple second data corresponding to a different time interval. One or more images comprise multiple first images and multiple second images, with the first data segmented across the multiple first images and the multiple second data segmented across the multiple second images. Thus, the padding data contained in one or more images is reduced, thereby improving encoding efficiency.

[0394] For example, such as Figure 34 As shown, the first information includes: first data (e.g., time-independent feature plane data), containing multiple feature quantities without a time dimension; and multiple second data (e.g., time-dependent feature plane data), containing multiple feature quantities with a time dimension, each of the multiple second data corresponding to a different time interval. One or more images include a first image and a second image, with the first data configured in the first image and the multiple second data centrally configured in the second image. This reduces the number of images to be encoded, thus improving encoding efficiency.

[0395] For example, such as Figure 35 as well as Figure 36As shown, the first information includes first data (low-scale time-independent feature plane data) and second data (high-scale time-independent feature plane data) with a higher resolution than the first data. One or more images include a first image and a second image encoded after the first image. The first data is configured in the first image, and the second data is configured in the second image. Therefore, the decoding device can first decode the first data and use it to generate a model.

[0396] For example, such as Figures 37-41 As shown, the first information includes first data (low-scale time-independent feature plane data) and second data (high-scale time-independent feature plane data) with a higher resolution than the first data. Each of the more than one image includes a first component (e.g., a Y component) and a second component and a third component (e.g., a U component and a V component) smaller than the first component. The first data is configured in at least one of the second and third components, and the second data is configured in the first component. Thus, multiple data can be efficiently configured using multiple components.

[0397] In addition, the structure of the decoding device is, for example, similar to... Figure 11 The decoding device 200 shown has the same structure. The decoding device includes a processor (or circuit) and a memory, and the processor (or circuit) uses the memory to perform the above-mentioned processing during operation.

[0398] [Second Method] The decoding process of the decoding device involved in the second method will be explained below. Figure 54 This is a flowchart of the decoding process of the decoding device involved in the second method.

[0399] First, the decoding device decodes the image and metadata from the bitstream (S501). Furthermore, the decoding device obtains packing information and 3D scene parameters from the metadata.

[0400] Next, the decoding device transforms the image into a feature map image (S502). The feature map image has a higher bit depth than the decoded image. For example, the decoding device inverse-quantizes the low-bit image data to generate high-bit image data. For example, by inverse-quantizing a 10-bit integer value as a low bit, a 32-bit floating-point value as a high bit is generated. In other examples, a 32-bit floating-point value is generated by inverse-quantizing a 12-bit or 8-bit integer value. Inverse quantization is, for example, a left shift operation used to improve precision. Furthermore, the 3D scene parameters include feature plane quantization parameters used to determine the inverse quantization step (quantization step). The decoding device performs inverse quantization using the step represented by the feature plane quantization parameters.

[0401] Next, the decoding device uses the packing information to unpack multiple segments of the feature plane and multiple neural network weights from the feature map image (S503).

[0402] Next, the decoding device updates the model using multiple segments of the feature planes and neural network weights (S504). The model contains multiple previously obtained feature planes, and new feature planes are added through the update.

[0403] Next, the decoding device uses the updated model to generate a rendered image at the specified viewpoint and time (S505).

[0404] Furthermore, this example illustrates an instance where the image contained in the bitstream is quantized, but the image in the bitstream may also not be quantized. In this case, the decoding device does not perform step S502, and in step S503, multiple segments of the feature plane and multiple neural network weights are unpacked from the image obtained from the bitstream.

[0405] Figure 55 This is a block diagram illustrating the structure of the encoding apparatus according to the embodiment. A model of a 3D scene representation method trained on a 3D scene within a 3D video sequence is input into the encoding apparatus 500. This model includes feature planes and neural network weights.

[0406] The encoding device 500 includes a transformation unit 501, a metadata transformation unit 502, and an image encoding unit 503. The transformation unit 501 extracts feature planes, neural network weights, and associated 3D scene parameters from the model.

[0407] Furthermore, the transformation unit 501 reconfigures (packs) the extracted feature planes and neural network weights into the image and outputs the generated image to the image encoding unit 503. Additionally, the transformation unit 501 outputs packing information related to the reconfiguration to the metadata transformation unit 502.

[0408] The packing information contains information used to unpack the neural network weights from the frame. For example, this information includes at least one of the following: the starting address of the neural network weights, the number of neural network weights, the number of neural network layers, the type of neural network layers, the number of neurons in each layer, and the neural network identifier.

[0409] In addition, the transformation unit 501 updates the 3D scene parameters required for the reconstruction of the 3D scene in the decoding device and outputs the updated 3D scene parameters to the metadata transformation unit 502.

[0410] The metadata transformation unit 502 transforms the 3D scene parameters and packing information into metadata (parameter set) that is sent along with the bit stream.

[0411] The image encoding unit 503 generates a bitstream by encoding the image and metadata. That is, the bitstream contains the image (encoded image) and metadata. For example, the image encoding unit 503 encodes the image using a known video codec. The structure of the image encoding unit 503 may also be the same as that of the encoding apparatus 100 described above.

[0412] Figure 56 This is a block diagram illustrating the structure of the decoding apparatus according to the embodiment. A bitstream is input to the decoding apparatus 600. For example, the bitstream is generated by the encoding apparatus 500 described above.

[0413] The decoding device 600 includes an image decoding unit 601, a metadata inverse transformation unit 602, an inverse transformation unit 603, and a rendering unit 604.

[0414] The image decoding unit 601 decodes the bitstream to obtain the image and metadata. The metadata inverse transformation unit 602 transforms the metadata into 3D scene parameters and packing information. For example, the image decoding unit 601 uses a known video codec to decode the image. The structure of the image decoding unit 601 may also be the same as that of the decoding device 200 described above.

[0415] The inverse transform unit 603 uses the packing information to transform the image into feature planes and neural network weights.

[0416] The rendering unit 604 uses neural network weights, feature planes, and 3D scene parameters to generate or update the model. Furthermore, the rendering unit 604 uses the model to generate a rendered image as an image observed from an input viewpoint requested by the software or the user.

[0417] Figure 57 This is a diagram illustrating an example of packing (reconfiguring) feature planes and neural network weights within an image. Frame m is the image decoded from the bitstream.

[0418] exist Figure 54 In step S503, the decoding device unpacks the neural network weights from the frame. The decoding device uses the packing information and neural network weights contained in the metadata to generate the neural network.

[0419] Furthermore, the mapping method is not limited to Figure 57 The example shown illustrates this. For instance, neural network weights can also be mapped to the top-left pixel within an image. Furthermore, the starting position does not necessarily have to be a pixel at the end of the frame; the starting position can be set at any location. Thus, the encoding device can flexibly map multiple feature planes. For example, by performing mapping, the encoding device can improve the encoding efficiency of image encoding, thereby reducing the amount of data.

[0420] Therefore, the encoding device can use unused regions within a frame to send additional data. Since high precision is required for the neural network weights, the encoding device can also apply reversible or low-quantization (low QP) in frames, slices, tiles, or coding units containing the neural network weights. Alternatively, when using high QP, the encoding device can ensure high precision by making the neural network weights redundant. For example, forward error correction can be used as a method of redundancy. For example, the neural network weights can also be stored three times. In this case, the precision is determined based on the similarity of at least two of the three repetitions in the decoding device.

[0421] Additionally, information indicating whether a frame contains neural network weights can also be included in the bitstream. For example, this information can be appended to metadata or the SEI, VUI, or header information of an image encoding standard. Furthermore, if a frame contains neural network weights, information indicating the number of neural network weights in one frame, and the start position of the mapping within the frame for the neural network weights (e.g., ...) can also be included. Figure 57 The information at the start position A) shown can also be included in the bitstream. For example, this information can be appended to metadata or the SEI, VUI, or header information of an image encoding standard. Thus, when the decoded frame contains more than one neural network weight, the decoding device can use the information indicating the start position of the mapping to appropriately obtain more than one neural network weight.

[0422] Figure 58 This is another example of packing feature planes and neural network weights into an image. Figure 58 In the example shown, frame m contains slice 1 and slice 2. Slice 1 contains feature plane data, and slice 2 contains neural network weights. Thus, feature planes and neural network weights can be stored in different slices within a single frame. Alternatively, tiles can be used instead of slices. This allows for parallel encoding and decoding of feature planes and neural network weights.

[0423] Figure 59 This is another example of packing feature planes and neural network weights into an image. Figure 59 In the example shown, the feature plane and neural network weights are stored in different frames.

[0424] Specifically, frame m is an image decoded from the bitstream, containing only data on the neural network weights. Frame m+1 is an image decoded from the bitstream, containing only feature plane data (e.g., temporally independent feature plane data and temporally dependent feature plane data). Figure 54 In step S503 shown, the decoding device obtains neural network weights from the frame (feature map image) and uses the obtained neural network weights to generate a neural network.

[0425] Additionally, information indicating the regions containing padding data can also be included in the bitstream. For example, this information can be appended to metadata or the SEI, VUI, or header information of an image coding standard. Thus, the decoding device can identify which regions within the decoded frame contain padding data, and therefore can appropriately obtain neural network weights or feature plane data.

[0426] In this way, the feature planes and neural network weights are stored in different frames. Therefore, the encoding device can apply processing appropriate to each feature plane and neural network weight on a frame-by-frame basis. For example, the encoding device can apply different quantization processing to each frame. For instance, the encoding device can apply reversible quantization or low quantization (low QP) to the neural network weight frames and high quantization (high QP) to the frames containing feature plane data.

[0427] Furthermore, information indicating whether the information contained in the frame is neural network weights or feature plane data can also be included in the bitstream. This information can also indicate whether the information in the frame is time-independent feature plane data, time-dependent feature plane data, or both. For example, this information can also be appended to metadata or the SEI, VUI, or header information of the image coding standard. Thus, the decoding device can identify whether the information contained in the decoded frame is neural network weights or feature plane data, and can appropriately obtain either the neural network weights or feature plane data.

[0428] Figure 60 This is another example of packing feature planes and neural network weights into an image. Figure 60 In the example shown, feature planes and neural network weights are stored in the YUV420 format image.

[0429] Specifically, the Y component of a frame stores feature plane data, while the U component stores neural network weights. Furthermore, the neural network weights can be configured in the V component, or in both the U and V components. Figure 54 In step S503 shown, the decoding device unpacks the neural network weights from the frame and uses the neural network weights to generate the neural network contained in the model.

[0430] Furthermore, the mapping method is not limited to Figure 60 The example shown illustrates this. For instance, the neural network weights could be configured on the Y component, and the feature plane data on the U or V component. This allows the encoding device to flexibly map different information. For example, by performing mapping, the encoding device can improve the encoding efficiency of image coding, thereby reducing the amount of data.

[0431] Furthermore, this method can effectively utilize the U or V components contained in the frame, thus enabling the transmission of more data simultaneously.

[0432] Additionally, information representing the Y, U, and V components of a frame (feature planes or neural network weights) can also be included in the bitstream. This information can also indicate whether the information contained in the component is time-independent feature plane data, time-dependent feature plane data, or both. For example, this information can also be appended to metadata or the SEI, VUI, or header information of an image coding standard. Figure 60 In the example shown, feature plane data (both time-independent and time-dependent feature plane data) is included in the Y component, and information about neural network weights can also be included in the bitstream in the U component. Therefore, the decoding device can identify which component within the decoded frame contains neural network weights or feature plane data, and thus can appropriately obtain the neural network weights or feature plane data.

[0433] Figure 61 This is another example of packing feature planes and neural network weights into an image. Figure 61 In the example shown, the multiple frames contained in the bitstream are classified into enhancement layers and base layers. For example, the base layer can be decoded without referring to other layers, while the decoding of the enhancement layer requires reference to the base layer.

[0434] exist Figure 61 In the example shown, the frames of the base layer contain only feature plane data (both time-independent and time-dependent feature plane data). The frames of the extended layer (spatial layer) contain only neural network weight data. Furthermore, the frames of the base layer containing feature plane data and the frames of the extended layer containing neural network weights are frames at the same time t.

[0435] exist Figure 54 In step S503 shown, the decoding device unpacks the neural network weights from the frame and uses the neural network weights to generate a neural network.

[0436] Furthermore, the mapping method is not limited to Figure 61 The example shown illustrates this. For instance, neural network weights could be stored in the base layer, while feature plane data could be stored in the extended layer. This allows the encoding device to flexibly map the information. For example, by performing mapping, the encoding device can improve the coding efficiency of image encoding, thereby reducing the amount of data.

[0437] Furthermore, this method allows the use of multiple layers of the video to send both the feature plane and the neural network weights.

[0438] Additionally, information indicating the contents of the base or extended layers (feature plane data or neural network weights) can also be included in the bitstream. This information can also indicate whether the layer contains time-independent feature plane data, time-dependent feature plane data, or both. For example, this information can also be appended to metadata or the SEI, VUI, or header information of an image coding standard. Figure 61 In the example shown, it indicates that feature plane data (both time-independent and time-dependent feature plane data) is included in the base layer, and information about neural network weights in the extended layers can also be included in the bitstream. Therefore, the decoding device can identify whether neural network weights or feature plane data are included in each layer of the decoded frame, and thus can appropriately obtain the neural network weights or feature plane data.

[0439] Furthermore, while the example of configuring neural network weights in an image has been described above, at least one of other types of information, such as voxel raster data, 3D Gaussian distribution, or point group data, can be configured in the image using the same method, instead of neural network weights or in addition to neural network weights. Moreover, multiple types of these different types of information can be combined and configured within the same frame.

[0440] Figure 62 It means Figure 54 A diagram illustrating an example of updating the dynamic 3D representation model in step S504. (See diagram for example.) Figure 62 As shown, the received temporally independent feature plane data is used to update the temporally independent feature planes in the model. The received temporally dependent feature plane data is used to update the temporally dependent feature planes in the model. The received neural network weights are used to update the neural network weights in the model. The received voxel raster information is used to update the voxel raster in the model. For example, this model update is performed by the rendering unit 604.

[0441] Figure 63 This is a diagram illustrating an example of a model in step S504 that uses neural network weights to update the dynamic 3D representation.

[0442] The received neural network weights are used to update the neural network. For example, the neural network is updated via the rendering unit.

[0443] The received neural network weights include: (1) a list of neural network weights [w 11 w 12 …w 1N b 11 ... b L2(2) The number of layers in the neural network L; (3) A list of the number of neurons in each layer [1, N, ..., 2]; (4) The type of neural network layer (e.g., convolutional layer or fully combined layer); (5) The neural network identifier (e.g., NN_1).

[0444] In addition, Figure 63 In the example shown, the neural network contains L layers. Layer 1 contains one neuron, layer 2 contains N neurons, and layer L contains two neurons.

[0445] The received information about the neural network weights is copied to construct the neural network. Furthermore, by being signaled about the type of layer (e.g., convolutional or fully combined layer), the decoding device can, for example, learn the characteristics of the layer being decoded among multiple layers, each with different functions.

[0446] Figure 64 It means Figure 54 The figure shows an example of the processing of the model in step S504, which uses voxel raster information to update the dynamic 3D representation of the model.

[0447] Voxel raster information includes: (1) a list of voxel raster features (e.g., data values ​​of each node); (2) the dimensions of the voxel raster along the X, Y, and Z directions (e.g., [w]). x w y w z (3) Voxel raster identifier (e.g., VG_1).

[0448] The received voxel raster information is used to update the nodes of the voxel raster. For example, the feature values ​​of each node in the voxel raster are a trained feature arrangement consisting of floating-point values. The node features can be used to calculate detailed information related to the 3D scene in the voxel raster region, such as density or color. The voxel raster identifier is used to identify the voxel raster's representation type (density or color, etc.). That is, the voxel raster identifier indicates the voxel raster's representation type.

[0449] The following describes an encoding device for generating a bitstream decoded by the decoding device involved in the second method described above. Figure 65 This is a flowchart of the encoding process performed by the encoding device involved in the second method.

[0450] First, the encoding device extracts multiple segments of the feature planes from the model containing multiple feature planes (S601).

[0451] Next, the encoding device generates a feature map image by packaging (reconfiguring) multiple segments of the feature plane and neural network weights. Furthermore, the encoding device stores (i) the reconfiguration order and start address of the multiple segments of the feature plane, and (ii) the reconfiguration order and start address of the neural network weights in metadata (S602). At this time, the encoding device, for example, configures padding data in the feature plane data within the image and in the regions outside the neural network weights.

[0452] Next, the encoding device quantizes the high-bit feature map image to transform it into a low-bit-depth image (S603). The low-bit-depth image has a lower bit depth than the feature map image.

[0453] For example, the encoding device quantizes the 32-bit floating-point data, which is the high-order bit, to obtain a 10-bit integer value. Alternatively, the encoding device quantizes the 32-bit floating-point value to obtain a 12-bit or 8-bit integer value. Quantization processing includes, for example, a process of reducing precision through a right shift operation. Furthermore, the 3D scene parameters include feature plane quantization parameters used to determine the quantization steps.

[0454] Furthermore, since high precision is required for neural network weights, the encoding device can also apply reversible or low-quantization (low QP) in frames containing neural network weights. Alternatively, when using high QP, the encoding device can also ensure high precision by making the neural network weights redundant.

[0455] Next, the encoding device encodes the low bit depth image and metadata into a bit stream (S604). That is, the bit stream contains the low bit depth image and metadata.

[0456] Furthermore, while an example of quantizing the feature map image is shown here, the feature map image may also not be quantized. In this case, the encoding device does not perform step S603, and the feature map image is encoded in step S604.

[0457] As described above, the encoding apparatus involved in the second approach uses existing video codecs without requiring other codec-related methods or devices, and can effectively encode neural network weight data. Furthermore, the encoding apparatus replaces padding data with useful data such as intra-frame neural network weight data. Moreover, the encoding apparatus can encode both feature planes and neural network weights using a single video codec. Thus, efficient encoding and decoding processes can be achieved.

[0458] Furthermore, this method can also be implemented by combining at least a portion of other methods disclosed herein. Moreover, this method can also be implemented by combining a portion of the process shown in any of these methods, a portion of the structure of any apparatus, a portion of a statement, etc., with other methods.

[0459] Furthermore, some or all of the processing in the aforementioned decoding apparatus can also be performed in the encoding apparatus in the same way. Moreover, not all of the constituent elements described in this method are always necessary; sometimes only a portion of the constituent elements of the first method are included in the apparatus.

[0460] Furthermore, while the above description illustrates an example of configuring feature planes on an image, the configured data is not limited to this. Any information contained in the model, other than feature planes, can also be configured on the image in the same way. For example, this information could be information representing feature quantities with a time dimension.

[0461] Furthermore, the above description illustrates an example of configuring both the feature plane and the neural network weights onto an image, but the feature plane may also be encoded using other methods instead of being configured onto the image.

[0462] Furthermore, the above description illustrates YUV components as examples of multiple components contained in an image, but multiple components are not limited to this. For example, multiple components could also be YCbCr, YPbPr, or RGB, etc.

[0463] As described above, the encoding device according to the embodiment performs... Figure 66 The process is as follows: The encoding device obtains the weights of multiple neural networks contained in the model (S701), configures the multiple neural network weights on one or more images (S702), and generates a bitstream by encoding one or more images (S703). Thus, the encoding device can encode the weights of multiple neural networks contained in the model using image encoding. Therefore, the encoding processing of information contained in the model can be easily realized.

[0464] For example, the bitstream contains first information indicating the configuration order of multiple neural network weights. Thus, the decoding device can use the first information to appropriately extract the multiple neural network weights from the image.

[0465] For example, the bitstream contains metadata, which includes initial information and multiple parameters related to the model (such as 3D scene parameters). Thus, the decoding device can use these multiple parameters to generate the model.

[0466] For example, the encoding device also acquires second information contained in the model, which (e.g., feature planes) represents multiple feature quantities with a time dimension. In the configuration, multiple neural network weights and the second information are configured on more than one image. Thus, the encoding device can use image encoding to encode the second information contained in the model representing multiple feature quantities with a time dimension. Therefore, the encoding processing of information contained in the model can be easily implemented.

[0467] For example, such as Figure 57 as well as Figure 58 As shown, multiple neural network weights and second information are configured in the same image contained in more than one image. This allows for the efficient configuration of multiple neural network weights and second information.

[0468] For example, such as Figure 59 As shown, multiple neural network weights and second information are configured in different images contained in one or more images. Therefore, the decoding device can independently decode multiple neural network weights and second information.

[0469] For example, such as Figure 60 As shown, each of the more than one images contains multiple components (e.g., Y component, U component, and V component), and multiple neural network weights and second information are configured on different components. This allows for the efficient configuration of multiple neural network weights and second information.

[0470] For example, such as Figure 61 As shown, each of the multiple images contains multiple layers (e.g., extended layers and base layers), and multiple neural network weights and second information are configured in different layers. This allows for the efficient configuration of multiple neural network weights and second information.

[0471] In addition, the structure of the encoding device is, for example, similar to Figure 8 The encoding device shown is the same. The encoding device includes a processor (or circuit) and a memory, and the processor (or circuit) uses the memory to perform the above-mentioned processing during operation.

[0472] Furthermore, the decoding device involved in the implementation method performs... Figure 67 The processing is shown. The decoding device decodes one or more images from a bitstream generated by encoding one or more images containing multiple neural network weights included in the model (S801), and obtains multiple neural network weights from one or more images (S802). Thus, the decoding device can decode multiple neural network weights included in the model using image encoding-based decoding processing. Therefore, decoding processing of information included in the model can be easily implemented.

[0473] For example, the bitstream contains first information indicating the configuration order of multiple neural network weights. Thus, the decoding device can use the first information to appropriately extract the multiple neural network weights from the image.

[0474] For example, the bitstream contains metadata, which includes initial information and multiple parameters related to the model (such as 3D scene parameters). Thus, the decoding device can use these multiple parameters to generate the model.

[0475] For example, multiple neural network weights and second information (e.g., feature planes) representing multiple features with a time dimension are configured in one or more images. In the acquisition process, the multiple neural network weights and the second information are obtained from one or more images. Therefore, the decoding device can use image-encoding-based decoding processing to decode the second information representing multiple features with a time dimension contained in the model. Thus, decoding processing of the information contained in the model can be easily implemented.

[0476] For example, such as Figure 57 as well as Figure 58 As shown, multiple neural network weights and second information are configured in the same image contained in more than one image. This allows for the efficient configuration of multiple neural network weights and second information.

[0477] For example, such as Figure 59 As shown, multiple neural network weights and second information are configured in different images contained in one or more images. Therefore, the decoding device can independently decode multiple neural network weights and second information.

[0478] For example, such as Figure 60 As shown, each of the more than one images contains multiple components (e.g., Y component, U component, and V component), and multiple neural network weights and second information are configured on different components. This allows for the efficient configuration of multiple neural network weights and second information.

[0479] For example, such as Figure 61 As shown, each of the multiple images contains multiple layers (e.g., extended layers and base layers), and multiple neural network weights and second information are configured in different layers. This allows for the efficient configuration of multiple neural network weights and second information.

[0480] In addition, the structure of the decoding device is, for example, similar to Figure 11 The decoding device 200 shown has the same structure. The decoding device includes a processor (or circuit) and a memory, and the processor (or circuit) uses the memory to perform the above-mentioned processing during operation.

[0481] The embodiments disclosed herein may also be implemented in combination with at least a portion of the other embodiments disclosed herein. Furthermore, portions of the processing described in the flowcharts of the embodiments disclosed herein, portions of the apparatus structure, portions of statements, etc., may be combined with other embodiments.

[0482] [Implementation and Application] In the above embodiments, each functional or operational block can typically be implemented using an MPU (microprocessor) and memory. Furthermore, the processing of each functional block can also be implemented as a program execution unit such as a processor that reads and executes software (programs) recorded in a recording medium such as ROM. This software can also be distributed. The software can also be recorded in various recording media such as semiconductor memory. Alternatively, each functional block can be implemented using hardware (dedicated circuitry).

[0483] The processes described in each embodiment can be implemented centrally using a single device (system), or they can be implemented distributedly using multiple devices. Furthermore, the processor executing the above-described program can be single or multiple. That is, centralized processing or distributed processing is possible.

[0484] The present disclosure is not limited to the above embodiments, and various modifications can be made, which are also included within the scope of the present disclosure.

[0485] Furthermore, examples of application of the motion picture encoding method (image encoding method) or motion picture decoding method (image decoding method) described in the above embodiments, and various systems implementing such examples will be described here. Such a system may also be characterized by having an image encoding device using the image encoding method, an image decoding device using the image decoding method, or having both image encoding and decoding devices. Other structures of such a system can be appropriately modified depending on the circumstances.

[0486] [Usage Example] Figure 68 This is a diagram showing the overall structure of an appropriate content delivery system ex100 for implementing content publishing services. The area for providing communication services is divided into desired sizes, and each cell is equipped with base stations ex106, ex107, ex108, ex109, and ex110, which serve as fixed wireless stations in the example shown in the diagram.

[0487] In this content delivery system ex100, various devices such as a computer ex111, a game console ex112, a camera ex113, a home appliance ex114, and a smartphone ex115 are connected to the Internet ex101 via an Internet service provider ex102 or a communication network ex104 and base stations ex106 to ex110. The content delivery system ex100 can also connect any of these devices in combination. In various embodiments, the devices can also be directly or indirectly interconnected via telephone networks or short-range wireless connections without using base stations ex106 to ex110. Furthermore, the streaming server ex103 can also connect to the computer ex111, game console ex112, camera ex113, home appliance ex114, and smartphone ex115 via the Internet ex101, etc. Additionally, the streaming server ex103 can also connect to terminals within a hotspot on an airplane ex117 via satellite ex116.

[0488] Alternatively, wireless access points or hotspots can be used to replace base stations ex106 to ex110. Furthermore, the streaming server ex103 can connect directly to the communication network ex104 without going through the Internet ex101 or the Internet service provider ex102, and can also connect directly to the aircraft ex117 without going through the satellite ex116.

[0489] The camera ex113 is a digital camera or similar device capable of capturing still and moving images. Furthermore, the smartphone ex115 refers to a smartphone, portable telephone, or PHS (Personal Handyphone System) that corresponds to mobile communication systems known as 2G, 3G, 3.9G, 4G, and in the future, 5G.

[0490] Home appliance EX114 refers to equipment included in refrigerators or household fuel cell combined heat and power systems.

[0491] In the content supply system ex100, terminals with photography capabilities are connected to the streaming server ex103 via base station ex106, enabling real-time publishing. During real-time publishing, terminals (such as computer ex111, game console ex112, camera ex113, home appliance ex114, smartphone ex115, and terminals within an airplane ex117) can perform the encoding processing described in the above embodiments on still or moving images captured by the user using the terminal. They can also multiplex the encoded image data and the encoded audio data corresponding to the image, and send the obtained data to the streaming server ex103. In other words, each terminal functions as an image encoding device according to one aspect of this disclosure.

[0492] On the other hand, the streaming server ex103 streams content data sent by requesting clients. Clients can be terminals such as computers ex111, game consoles ex112, cameras ex113, home appliances ex114, smartphones ex115, or airplanes ex117, capable of decoding the encoded data described above. Each device receiving the streamed data decodes and reproduces the received data. That is, each device can also function as an image decoding apparatus according to one aspect of this disclosure.

[0493] [Distributed processing] Furthermore, the streaming server ex103 can also consist of multiple servers or computers, distributing, recording, or publishing data. For example, the streaming server ex103 can also be implemented using a CDN (Contents Delivery Network), which delivers content through multiple edge servers distributed globally and a network connecting these edge servers. In a CDN, physically proximate edge servers are dynamically allocated based on the client. Moreover, by caching and publishing content to these edge servers, latency can be reduced. Furthermore, in the event of errors or changes in communication due to increased traffic, processing can be distributed across multiple edge servers, the publishing entity can be switched to other edge servers, or publishing can continue by bypassing faulty parts of the network, thus enabling high-speed and stable publishing.

[0494] Furthermore, the processing of the data can be distributed not only across the terminals themselves, but also on the server side, or the workload can be shared among them. For example, the encoding process typically involves two processing loops. In the first loop, the complexity or amount of code for the image, measured in frames or scenes, is assessed. In the second loop, processing is performed to maintain image quality and improve encoding efficiency. For instance, the terminal performs the first encoding process, while the server receiving the content performs the second, thus reducing the processing load on each terminal while improving content quality and efficiency. In this scenario, if there is a request for near real-time reception and decoding, the data encoded in the first iteration by the terminal can be received and reproduced by other terminals, enabling more flexible real-time publishing.

[0495] As another example, cameras like the ex113 extract features from images, compress the data associated with those features into metadata, and send it to a server. The server, for instance, determines the importance of a target based on the features and adjusts the quantization precision accordingly, performing compression that corresponds to the meaning (or importance) of the image's content. The feature data is particularly effective in improving the accuracy and efficiency of motion vector prediction during further compression on the server. Alternatively, simple encoding such as VLC (Variable Length Coding) can be performed at the terminal, while more demanding encoding such as CABAC (Context Adaptive Binary Arithmetic Coding) can be performed on the server.

[0496] Furthermore, as another example, in stadiums, shopping malls, or factories, there are sometimes multiple image data sets obtained by capturing roughly the same scene from multiple terminals. In this case, the data is distributed and encoded separately using the multiple terminals that captured the images, as well as other terminals and servers that did not capture images as needed, for example, by group of pictures (GOPs), by individual images, or by tiles obtained from dividing images. This reduces latency and further enhances real-time performance.

[0497] Multiple image datasets depict roughly the same scene, and therefore can be managed and / or directed by the server to cross-reference image data captured by various terminals. Furthermore, the server can receive encoded data from each terminal, change the reference relationships between multiple datasets, or correct or replace the images themselves for re-encoding. This allows for the generation of streams with improved quality and efficiency for each dataset.

[0498] Furthermore, the server can also transcode the image data by changing its encoding method before distributing the image data. For example, the server can convert MPEG encoding to VP encoding (such as VP9), or H.264 to H.265.

[0499] Thus, encoding processing can be performed by a terminal or one or more servers. Therefore, in the following descriptions, "server" or "terminal" will be used as the subject of processing, but the processing performed on the server can be performed by the terminal in part or all of the same way. Furthermore, the same applies to decoding processing.

[0500] [3D, Multi-angle] There is an increasing trend of merging images or videos of different scenes captured by multiple cameras (ex113) and / or smartphones (ex115) that are roughly synchronized with each other, or of the same scene captured from different angles. Images captured by each device are merged based on the relative positional relationship between the devices or regions containing consistent feature points within the images.

[0501] The server not only encodes 2D moving images, but also automatically or at a user-specified time based on scene analysis of the moving images, and sends them to the receiving terminal. Furthermore, when the relative positional relationships between the photographic terminals are available, the server can generate the 3D shape of the scene not only based on 2D moving images, but also based on images of the same scene captured from different angles. The server can also encode 3D data generated from point clouds, or select or reconstruct images from multiple terminals based on the results of identifying or tracking people or targets using 3D data to generate images to be sent to the receiving terminal.

[0502] In this way, users can freely select images corresponding to each camera terminal to appreciate the scene, and can also appreciate the content of images extracted from the selected viewpoint from the 3D data reconstructed using multiple images or videos. Furthermore, sound and images are collected together from multiple different angles, and the server can also multiplex sound from a specific angle or space with the corresponding image and send the multiplexed image and sound.

[0503] Furthermore, in recent years, content that establishes a correspondence between the real world and the virtual world, such as Virtual Reality (VR) and Augmented Reality (AR), has become increasingly popular. In the case of VR images, the server can create separate viewpoint images for the right and left eyes. These images can be encoded using multi-view coding (MVC) to allow reference between the viewpoint images, or they can be encoded as separate streams without reference. During the decoding of another stream, the images can be synchronously reproduced according to the user's viewpoint to recreate a virtual three-dimensional space.

[0504] In the case of AR images, the server can overlay virtual object information in virtual space onto camera information in real space based on 3D position or user viewpoint movement. The decoding device can also acquire or retain virtual object information and 3D data, generate a 2D image based on user viewpoint movement, and create overlay data by smoothly connecting them. Alternatively, the decoding device can send the user's viewpoint movement to the server in addition to the virtual object information. The server can also cooperate with the viewpoint movement received from the 3D data held on the server to create overlay data, encode the overlay data, and publish it to the decoding device. Furthermore, in addition to RGB, the overlay data also has an α value representing transmittance. The server can also set the α value of parts other than the target created from the 3D data to 0, etc., and encode the transmittance state of these parts. Alternatively, the server can set a specified RGB value as the background, similar to a chroma key, to generate data where parts other than the target are set as the background color.

[0505] Similarly, the decoding process of the published data can be performed on the client terminals, on the server side, or distributed among them. For example, one terminal could temporarily send a send / receive request to the server, while other terminals receive the content corresponding to that request, decode it, and send a signal indicating completion to a device with a display. By distributing the processing and selecting appropriate content regardless of the performance of the communicating terminals themselves, data with good image quality can be reproduced. Furthermore, as another example, while a large image data file is received by a TV or similar device, a viewer's personal terminal can decode and display a segmented area of ​​the image, such as tiles. This allows for the sharing of the overall image while simultaneously allowing the viewer to identify their own area of ​​responsibility or areas they wish to examine in more detail.

[0506] In situations where multiple short-range, medium-range, or long-range wireless communications are available both indoors and outdoors, content can be received seamlessly using publishing system standards such as MPEG-DASH (Dynamic Adaptive Streaming over HTTP). Users can also freely switch between their terminals and decoding or display devices, such as indoor or outdoor displays, in real time. Furthermore, decoding can be performed using location information and other methods, switching between the decoding and display terminals. This allows information to be mapped and displayed on parts of the walls or floors of adjacent buildings containing display devices while the user is moving to their destination. Additionally, the bit rate of the received data can be switched based on the ease of access to encoded data on the network, such as data cached on a server accessible only briefly from the receiving terminal or copied to an edge server of the content distribution service.

[0507] [Web page optimization] Figure 69 This is an example of a web page display screen in a computer such as ex111. Figure 70 This is an example image showing the display of a web page on a smartphone such as the EX115. Figure 69 as well as Figure 70 As shown, web pages sometimes contain multiple linked images that serve as links to image content, and the viewing method varies depending on the browsing device. When multiple linked images are displayed on the screen, before the user explicitly selects a linked image, or before the linked image is near the center of the screen or before the entire linked image enters the screen, the display device (decoding device) can display still images or I-images of each content as linked images. It can also display images such as GIF animations through multiple still images or I-images, or it can decode and display images by receiving only the base layer.

[0508] When a user selects a linked image, the display device prioritizes the base layer and decodes it. Additionally, if the HTML (HyperText Markup Language) constituting the web page contains information indicating tiered content, the display device can also decode up to the enhancement layer. Furthermore, to ensure real-time performance, before selection or when communication bandwidth is extremely limited, the display device can reduce the delay between the decoding and display of the initial image (the delay from the start of content decoding to the start of display) by decoding and displaying only the preceding reference images (I images, P images, and only B images for preceding reference). Moreover, the display device can also forcibly ignore the reference relationships between images and coarsely decode all B and P images as preceding references, performing normal decoding as more images are received over time.

[0509] [Autonomous Driving] Furthermore, when receiving still images or video data such as two-dimensional or three-dimensional map information for the purpose of autonomous driving or driving assistance, the receiving terminal can also receive information such as weather or construction data as metadata, in addition to image data belonging to more than one layer, and decode them by establishing correspondences. Moreover, metadata can belong to a layer or be simply multiplexed with image data.

[0510] In this scenario, since the receiving terminal, including vehicles, drones, or aircraft, is moving, the receiving terminal can seamlessly receive and decode data while switching between base stations ex106 to ex110 by transmitting its location information. Furthermore, the receiving terminal can dynamically adjust the level of metadata reception and map information updates based on user selection, user status, and / or communication bandwidth conditions.

[0511] In the content delivery system ex100, the client can receive, decode, and reproduce the encoded information sent by the user in real time.

[0512] [Publishing personal content] Furthermore, the content delivery system ex100 can not only unicast or multicast high-quality, long-duration content provided by video publishers, but also unicast or multicast low-quality, short-duration content provided by individuals. It can be assumed that such personal content will increase in the future. To improve the quality of personal content, the server can also encode it after editing. This can be achieved, for example, using the following structure.

[0513] During or after capturing images, the server performs image processing on the raw or encoded data, including error detection, scene search, meaning analysis, and target detection. Based on the recognition results, the server then manually or automatically corrects for focus deviations or camera shake, removes less important scenes (such as those with lower brightness or out-of-focus areas), emphasizes target edges, or adjusts tones. The server then encodes the edited data. Furthermore, recognizing that longer shooting times can decrease audiovisual quality, the server can automatically limit not only less important scenes like those mentioned above but also scenes with minimal movement to a specific time frame based on image processing results. Alternatively, the server can generate and encode summaries based on the meaning analysis results of the scenes.

[0514] Personal content may contain elements that infringe on copyrights, the author's moral rights, or portrait rights, or it may be shared beyond the intended scope, causing inconvenience to the individual. Therefore, servers can encode images such as faces in the periphery of a scene or a home scene by forcibly changing them to out-of-focus images. Furthermore, servers can identify whether a face different from a pre-registered person is captured within the image to be encoded, and if so, apply processing such as mosaic to the face. Alternatively, as pre- or post-processing for encoding, from a copyright perspective, users can specify the people or background areas they want to process in the image. The server can then replace the specified area with another image or blur the focus. If it involves a person, the server can track the person in a moving image and replace the image of their face.

[0515] Because personal content with small data volumes has high real-time requirements for audiovisual quality, the decoding device prioritizes receiving, decoding, and reproducing the base layer, although bandwidth also plays a role. During this process, the decoding device can also receive enhancement layers, including them in reproducing high-quality images, especially in cases where the playback is looped or repeated more than once. This allows for the provision of a stream with scalable encoding, providing an experience where the motion images are initially coarse but gradually become smoother and the image quality improves. In addition to scalable encoding, the same experience can be provided when a coarser first playback stream and a second stream encoded based on the first motion image are combined into a single stream.

[0516] [Other Implementation Examples] Furthermore, these encoding or decoding processes are generally performed within the LSI ex500 of each terminal. LSI (large scale integration circuitry) ex500 (see reference) Figure 68 The chip can be a single chip or a structure composed of multiple chips. Alternatively, software for motion image encoding or decoding can be installed on a recording medium (CD-ROM, floppy disk, or hard disk, etc.) that can be read by a computer such as the ex111, and the software can be used for encoding or decoding. Furthermore, if the ex115 smartphone has a camera, motion image data acquired by that camera can also be transmitted. This motion image data is encoded in the LSIex500 chip of the ex115 smartphone.

[0517] Alternatively, the LSIex500 can also be a structure that downloads and activates application software. In this case, the terminal first determines whether it corresponds to the content's encoding method or whether it has the capability to execute a specific service. If the terminal does not correspond to the content's encoding method or does not have the capability to execute a specific service, the terminal downloads the codec or application software, and then retrieves and reproduces the content.

[0518] Furthermore, the content delivery system ex100, which is not limited to the Internet ex101, can also be equipped in a digital broadcasting system with at least one of the motion picture encoding device (image encoding device) or motion picture decoding device (image decoding device) of the above embodiments. Since multiplexed data that multiplexes images and sounds is carried and transmitted and received using radio waves for broadcasting via satellites or the like, there is a difference in the multicast direction compared to the easy unicast structure of the content delivery system ex100, but the same application can be performed regarding encoding and decoding processing.

[0519] [Hardware Structure] Figure 71 It means Figure 68 More detailed images of the EX115 smartphone are shown. Additionally, Figure 72This diagram illustrates a structural example of the smartphone ex115. The smartphone ex115 includes: an antenna ex450 for transmitting and receiving radio waves with a base station ex110; a camera unit ex465 capable of capturing images and still images; and a display unit ex458 for displaying decoded data such as images captured by the camera unit ex465 and images received by the antenna ex450. The smartphone ex115 also includes: an operation unit ex466 as a touch panel; a sound output unit ex457 as a speaker for outputting sound or audio; a sound input unit ex456 as a microphone for inputting sound; a memory unit ex467 capable of storing encoded or decoded data such as captured images or still images, recorded audio, received images or still images, and emails; and a slot ex464 serving as an interface with a SIM (Subscriber Identity Module) ex468, which is used to identify the user and authenticate access to various data sources, including the network. Alternatively, an external storage unit can be used to replace the storage unit ex467.

[0520] The main control unit ex460, which performs integrated control of the display unit ex458 and the operation unit ex466, is connected to the power supply circuit unit ex461, the operation input control unit ex462, the image signal processing unit ex455, the camera interface unit ex463, the display control unit ex459, the modulation / demodulation unit ex452, the multiplexing / demultiplexing unit ex453, the audio signal processing unit ex454, the slot unit ex464, and the memory unit ex467 via the synchronization bus ex470.

[0521] If the power button is turned on by the user, the power circuit section ex461 will activate the smartphone ex115 to operate, supplying power from the battery pack to various parts.

[0522] The smartphone ex115 performs call and data communication processing based on the control of the main control unit ex460, which includes a CPU, ROM, and RAM. During a call, the audio signal processing unit ex454 converts the audio signal collected by the audio input unit ex456 into a digital audio signal. The modulation / demodulation unit ex452 performs spectral diffusion processing, and the transmit / receive unit ex451 performs digital-to-analog conversion and frequency conversion processing. The resulting signal is then transmitted via the antenna ex450. Furthermore, the received data is amplified, undergoes frequency conversion and analog-to-digital conversion processing, and performs inverse spectral diffusion processing via the modulation / demodulation unit ex452. After being converted into an analog audio signal by the audio signal processing unit ex454, it is output from the audio output unit ex457. In data communication mode, based on the operation of the main unit's operation unit ex466, text, still images, or video data are sent to the main control unit ex460 via the operation input control unit ex462. The same transmit and receive processing is performed. In data communication mode, when transmitting images, still images, or images and sound, the image signal processing unit ex455 compresses and encodes the image signal stored in the memory unit ex467 or the image signal input from the camera unit ex465 using the motion picture encoding method described in the above embodiments, and sends the encoded image data to the multiplexing / demultiplexing unit ex453. The sound signal processing unit ex454 encodes the sound signal collected by the sound input unit ex456 during the capture of images or still images by the camera unit ex465, and sends the encoded sound data to the multiplexing / demultiplexing unit ex453. The multiplexing / demultiplexing unit ex453 multiplexes the encoded image data and the encoded sound data in a predetermined manner, performs modulation and conversion processing through the modulation / demodulation unit (modulation / demodulation circuit unit) ex452 and the transmitting / receiving unit ex451, and transmits it via the antenna ex450.

[0523] In cases where images are attached to emails or chat messages, or images linked to web pages are received, the multiplexing / demultiplexing unit ex453 separates the multiplexed data into a bitstream of image data and a bitstream of audio data by demultiplexing the multiplexed data received via antenna ex450 in order to decode the multiplexed data. The encoded image data is then supplied to the image signal processing unit ex455 via the synchronization bus ex470, and the encoded audio data is supplied to the audio signal processing unit ex454. The image signal processing unit ex455 decodes the image signal using a motion picture decoding method corresponding to the motion picture encoding method described in the above embodiments, and displays the image or still image contained in the linked motion picture file from the display unit ex458 via the display control unit ex459. The audio signal processing unit ex454 decodes the audio signal and outputs audio from the audio output unit ex457. Since real-time streaming is increasingly popular, audio reproduction may be socially inappropriate depending on the user's situation. Therefore, as an initial consideration, a structure that reproduces only the image data without reproducing the audio signal is preferred; however, audio can also be reproduced synchronously only when the user performs an operation such as clicking on the image data.

[0524] Furthermore, while the EX115 smartphone was used as an example here, as a terminal, besides a transmit / receive terminal with both an encoder and a decoder, three other installation forms can be considered: a transmit terminal with only an encoder and a receive terminal with only a decoder. In digital broadcasting systems, the reception or transmission of multiplexed data, which multiplexes audio data within video data, was explained. However, in addition to audio data, multiplexed data can also multiplex text data associated with the video. Furthermore, it is also possible to receive or transmit the video data itself without receiving or transmitting multiplexed data.

[0525] Furthermore, while the main control unit of the CPU, ex460, was described for controlling encoding or decoding processing, most terminals possess a GPU (Graphics Processing Unit). Therefore, a structure can also be implemented where the GPU's performance is effectively utilized to process a wider area simultaneously by using shared memory between the CPU and GPU, or memory that manages addresses in a way that allows for common use. This shortens encoding time, ensures real-time performance, and achieves low latency. In particular, it is even more efficient if motion search, deblocking filtering, SAO (Sample Adaptive Offset), and transform / quantization processing are performed simultaneously on an image-by-image basis using the GPU, rather than the CPU.

[0526] Industrial availability This disclosure can be applied, for example, to encoding devices and decoding devices.

[0527] Explanation of reference numerals in the attached figures 100 encoding device 102 Division 104 Subtraction Section 106 Transformer 108 Quantitative Department 110 Entropy Coding Department 112, 204 Inverse Quantization Section Inverse Transformation Units 114 and 206 116, 208 Addition Department 118 and 210 memory blocks 120, 212 Loop Filter Section 122, 214 frame memory Intra-frame prediction units 124 and 216 Inter-frame prediction units 126 and 218 Predictive Control Department 128, 220 Prediction parameter generation unit 130, 222 200 Decoding Device 202 Entropy Decoding Department 224 Division Decision Department 300, 310, 500 encoding devices Transformer Units 301, 311, and 501 302, 312 NNC Encoding Section 303, 313, 502 Metadata Transformation Department Image coding units 304, 314, and 503 400, 410, 600 decoding devices Image Decoding Units 401, 411, and 601 402, 412, 602 Metadata Inverse Transformation Unit 403, 413 NNC Decoding Section Inverse Transformation Units 404, 414, and 603 Rendering Departments 405, 415, and 604 processors a1 and b1 a2 and b2 memory

Claims

1. An encoding device comprising: Circuits; and The memory connected to the circuit, The circuit is in operation. Obtain the first information contained in the model, which represents multiple feature quantities with a time dimension. The first information is configured on more than one image. A bitstream is generated by encoding one or more images.

2. The encoding device as described in claim 1, The first information contains multiple data points. The bitstream contains second information indicating the configuration order of the plurality of data.

3. The encoding device as described in claim 1, The first information contains multiple data points. The bitstream contains third information, which indicates the start position of at least one of the plurality of data in the one or more images.

4. The encoding device as described in claim 1, The first information includes multiple feature planes that represent multiple feature quantities configured in two dimensions.

5. The encoding device as described in claim 1, The first information includes: first data, comprising multiple feature quantities that do not have the time dimension; and multiple second data, comprising multiple feature quantities that have the time dimension, wherein the multiple second data correspond to different time intervals. The more than one image includes multiple first images and multiple second images. The first data is segmented and configured across the plurality of first images. The plurality of second data are respectively configured in the plurality of second images.

6. The encoding device as claimed in claim 1, The first information includes: first data, comprising multiple feature quantities that do not have the time dimension; and multiple second data, comprising multiple feature quantities that have the time dimension, wherein the multiple second data correspond to different time intervals. The more than one image includes a first image and a second image. The first data is configured in the first image. The plurality of second data are centrally configured in the second image.

7. The encoding device as claimed in claim 1, The first information includes first data and second data with a higher resolution than the first data. The one or more images include a first image and a second image encoded after the first image. The first data is configured in the first image. The second data is configured in the second image.

8. The encoding device as claimed in claim 1, The first information includes first data and second data with a higher resolution than the first data. Each of the more than one images includes a first component, and a second component and a third component that are smaller than the first component. The first data is configured in at least one of the second component and the third component. The second data is configured in the first component.

9. A decoding device, have: Circuits; and The memory connected to the circuit, The circuit is in operation. Decoding the one or more images from a bitstream generated by encoding one or more images containing first information contained in a model, wherein the first information represents multiple feature quantities having a time dimension. The first information is obtained from one or more images.

10. The decoding apparatus as described in claim 9, The first information contains multiple data points. The bitstream contains second information indicating the configuration order of the plurality of data.

11. The decoding apparatus as described in claim 9, The first information contains multiple data points. The bitstream contains third information, which indicates the start position of at least one of the plurality of data in the one or more images.

12. The decoding apparatus as described in claim 9, The first information includes multiple feature planes that represent multiple feature quantities configured in two dimensions.

13. The decoding apparatus as described in claim 9, The first information includes: first data, comprising multiple feature quantities that do not have the time dimension; and multiple second data, comprising multiple feature quantities that have the time dimension, wherein the multiple second data correspond to different time intervals. The more than one image includes multiple first images and multiple second images. The first data is segmented and configured across the plurality of first images. The plurality of second data are respectively configured in the plurality of second images.

14. The decoding apparatus as described in claim 9, The first information includes: first data, comprising multiple feature quantities that do not have the time dimension; and multiple second data, comprising multiple feature quantities that have the time dimension, wherein the multiple second data correspond to different time intervals. The more than one image includes a first image and a second image. The first data is configured in the first image. The plurality of second data are centrally configured in the second image.

15. The decoding apparatus as described in claim 9, The first information includes first data and second data with a higher resolution than the first data. The one or more images include a first image and a second image encoded after the first image. The first data is configured in the first image. The second data is configured in the second image.

16. The decoding apparatus as described in claim 9, The first information includes first data and second data with a higher resolution than the first data. Each of the more than one images includes a first component, and a second component and a third component that are smaller than the first component. The first data is configured in at least one of the second component and the third component. The second data is configured in the first component.

17. An encoding method, Obtain the first information contained in the model, which represents multiple feature quantities with a time dimension. The first information is configured on more than one image. A bitstream is generated by encoding one or more images.

18. A decoding method, Decoding the one or more images from a bitstream generated by encoding one or more images containing first information contained in a model, wherein the first information represents multiple feature quantities having a time dimension. The first information is obtained from one or more images.

Images