Signaling for Implicit Neural Representation Reconstruction

Abstract

A method and apparatus for encoding or decoding one or more syntactic elements for use with an INR decoder are provided. The one or more syntactic elements provide for reconstructing at least one portion of an image using at least one implicit neural representation network that represents at least one portion of an image. The one or more syntactic elements comprise at least one of instructions related to an implicit neural representation network, an element providing information for configuring the inputs to an implicit neural representation network, or information related to the outputs of an implicit neural representation network.

Description

Technical Field

[0001] This embodiment generally relates to the compression of images, videos and / or 3D scenes using implicit neural representations (INRs). This embodiment relates to a method and apparatus for encoding, decoding, and transmitting metadata used by a decoder to reconstruct a signal encoded using an INR network.

Background Art

[0002] This application claims the priority of European application No. EP23305937.7 filed on Jun. 13, 2023, the entire content of which is incorporated herein by reference.

[0003] To achieve high compression efficiency, image and video coding schemes typically employ prediction and transform to exploit spatial and temporal redundancies within video content. Emerging technologies use neural networks. Among them, implicit neural representation (INR) aims to parameterize a function that receives coordinates as inputs and outputs the values of the signals at those coordinates. INR can be used, for example, for the compression of images, videos, 3D objects and scenes. It can also be applied to any type of signal. Techniques for constructing INR networks for the encoding of 2D or 3D images are known. Techniques for compressing neural networks are also known. However, no image / video decoder can reconstruct display images and videos based solely on INR. Additional information is required for any image / video decoder to reconstruct the output signal.

Brief Description of the Drawings

[0004] [Figure 1] It is a diagram showing an example of a neural network for implicit neural representation. [Figure 2]This figure shows an example of a method for encoding information used by a decoder to reconstruct at least a portion of the signal representing a scene from an encoded INR, according to one embodiment. [Figure 3] This figure shows an example of a method for decoding information used by a decoder to reconstruct at least a portion of the signal representing a scene from an encoded INR, according to one embodiment. [Figure 4] This figure shows an example of a method for reconstructing at least a portion of the scene-representing signal from encoded INR according to one embodiment. [Figure 5] This figure shows an example of a method, according to one embodiment, for generating coordinates used by a decoder to reconstruct at least a portion of the signal representing a scene from an encoded INR. [Figure 6] This figure shows an example of a method for performing inference on an INR network according to one embodiment. [Figure 7] This figure shows an example of a method for reconstructing at least a portion of the scene-representing signals from the output of an INR network, according to one embodiment. [Figure 8] This is a block diagram showing a system in which an embodiment of this model may be implemented. [Figure 9] This is a block diagram showing a system in which aspects of this embodiment can be implemented according to other embodiments. [Figure 10] This figure shows two remote devices communicating via a communication network, based on an example of this principle. [Figure 11] This diagram shows the syntax of a signal using an example of this principle. [Modes for carrying out the invention]

[0005] According to one embodiment, a method is provided for signaling one or more syntactic elements for using an INR decoder. The one or more syntactic elements provide the use of at least one implicit neural representation (INR) network representing at least one portion of the signal representing a scene in order to reconstruct at least one portion of the signal representing a scene. In some embodiments, the method also includes encoding the INR network.

[0006] In another embodiment, a device is provided for signaling one or more syntactic elements for use with an INR decoder. This device includes one or more processors, which are operable to signal one or more syntactic elements that provide one or more implicit neural representation networks representing at least one portion of a signal representing a scene, in order to reconstruct at least one portion of a signal representing a scene. In some embodiments, the device can also encode the INR network.

[0007] In another embodiment, a method is provided for decoding one or more syntactic elements for use with an INR decoder. The one or more syntactic elements provide at least one implicit neural representation (INR) network representing at least one portion of the signal representing a scene in order to reconstruct at least one portion of the signal representing a scene. In some embodiments, the method also includes decoding the INR network.

[0008] In some embodiments, the method also includes reconstructing at least a portion of the signal representing the scene using one or more syntactic elements and an implicit neural representation network.

[0009] In another embodiment, a device is provided for decoding one or more syntactic elements for use with an INR decoder. The device comprises one or more processors and is operable to decode one or more syntactic elements using one or more implicit neural representation networks that represent at least one portion of the signal representing at least one scene, in order to reconstruct at least one portion of the signal representing at least one scene. In some embodiments, the device can also decode the INR network. In some embodiments, the device can also reconstruct at least one portion of the signal representing a scene using one or more syntactic elements and an implicit neural representation network.

[0010] Further embodiments that can be used individually or in combination are described herein.

[0011] One or more embodiments also provide a computer program that, when executed by one or more processors, causes one or more processors to perform a method for signaling / decoding one or more syntactic elements for using an INR decoder according to any embodiment described herein. One or more embodiments also provide non-temporary computer-readable media and / or computer-readable storage media which store instructions for signaling / decoding one or more syntactic elements for using an INR decoder in the manner described herein.

[0012] One or more embodiments also provide a computer-readable storage medium on which a bitstream generated by the method described herein is recorded. One or more embodiments also provide a method and apparatus for transmitting or receiving a bitstream generated by the above method.

[0013] This application describes various aspects, including tools, functions, embodiments, models, and approaches. Many of these aspects are described in a way that may sound restrictive in order to illustrate their individual characteristics. However, this is for the purpose of clarifying the description and is not intended to limit the scope or application of these aspects. In fact, each different aspect can be combined and interchangeable with each other to provide further aspects. Furthermore, these aspects can also be combined and interchangeable with aspects described in prior applications.

[0014] The embodiments described and considered in this application can be implemented in various forms. Figures 1-11 show several embodiments, but other embodiments are also conceivable, and the discussion of Figures 1-11 does not limit the scope of embodiments.

[0015] In this application, the terms “reconstructed” and “decoded” may be used interchangeably, the terms “pixel” and “sample” may be used interchangeably, and the terms “image,” “video,” and “frame” may be used interchangeably.

[0016] This specification describes various methods, each comprising one or more steps or actions to achieve the described method. Unless a particular order of steps or actions is required for the proper operation of the method, the order and / or use of any particular steps and / or actions may be modified or combined. Furthermore, terms such as “first,” “second,” etc., may be used in various embodiments to modify elements, components, steps, actions, etc., for example, “first decode” and “second decode.” Unless specifically required, the use of these terms does not imply an order to the modified actions. Therefore, in this example, the first decode does not need to be performed before the second decode, and may occur, for example, before, simultaneously with, or within overlapping timeframes of the second decode.

[0017] Unless otherwise specified or technically impossible, the technical aspects described in this application can be used alone or in combination.

[0018] At least one aspect is generally related to the encoding and decoding of images, videos or 3D data using implicit neural representations. More generally, at least one aspect described herein is related to using implicit neural representations for the encoding / decoding of any signal representing a scene.

[0019] At least another aspect is generally related to the transmission of generated or encoded bitstreams. These aspects and other aspects can be implemented as a method, an apparatus, or a computer-readable storage medium storing instructions for encoding or decoding a data signal representing a scene according to any of the described methods, and / or a computer-readable storage medium storing a bitstream generated by any of the described methods.

[0020] Although not yet standardized, new neural compression technologies are being considered in the MPEG group standardization. In the AhG within WG4, compression technologies based on INR (implicit neural representation) have been investigated by the academic community and several research groups, and the number of papers has been increasing exponentially.

[0021] Research on INR mainly focuses on 2D and video compression, but INR has also been investigated for many other signals, especially 3D scenes and objects. Furthermore, these methods have much lower computational complexity compared to end-to-end neural compression methods.

[0022] Figure 1 shows an example of a neural network used for implicit neural representation (INR). Such a neural network used for INR can be called an INR network. The INR parameterizes a signal as a function 100, taking coordinates 110 as input and outputting the signal value 120 at these coordinates. In recent years, INRs have been used in various application fields such as images, videos, and 3D objects. In the case of an image, the input 110 is the pixel coordinate (x,y), and the INR can output 120 the color value (r,g,b) or (y,u,v) of the input pixel. The input coordinates can be modified by a transformation before being used as input to the neural network. This transformation can be a Fourier map, coordinate transformation, normalization, etc. The INR can be used to reconstruct a signal by calculating the signal value for each required coordinate input. For example, in 2x upsampling, it can be used to upsample a signal by generating an output for the input coordinates corresponding to the upsampled pixel, such as outputting the average value of the coordinates between two consecutive pixels.

[0023] The INR network 100 is a neural network typically composed of multiple neural layers, such as fully connected layers. In Figure 1, the network has four layers. Intermediate outputs are represented by circles. Each neural layer can be described as a function that first multiplies the input by a tensor, adds a vector called a bias, and then applies a nonlinear function to the resulting value. The shape (and other properties) of the tensor and the type of nonlinear function are called the network architecture. The values ​​of the tensor and bias are referred to as "weights" in this specification. The weights and, if applicable, the parameters of the nonlinear function are called the network parameters θ. The architecture and parameters define the "model". The notation f is used to represent the INR function parameterized by θ. θ This is used.

[0024] A typical process for encoding a signal using INR is as follows: First, the weights θ (or a portion thereof) of the INR network are optimized to reconstruct the signal. Next, these weights are further encoded to generate the output bitstream. For example, for an image I of size (M×N), the weights θ can be optimized by minimizing the following loss function.

[0025]

number

[0026] Here, D is the distortion that quantifies the difference between the image reconstructed by fθ and the original image I, R is the bitrate of the encoded parameter, and λ is the trade-off parameter between D and R. D can be any differentiable distortion measure, such as the root mean square error in the second equation. M and N are the width and height of the image. In this case, other metrics such as LPIPS (Learned Perceptual Image Patch Similarity) can also be used. Optimization of the weights θ is typically performed by machine learning approaches such as batch gradient descent.

[0027] To decode the signal, fθ is evaluated in all relevant coordinates. These coordinates can be selected during decoding. A typical selection is all pixel coordinates of an image or video. For example, in the case of a 256x256 pixel image, these coordinates can be all pairs of (x,y) for all x∈{0,1,···,255} and y∈{0,1,···,255}. Other selections are also possible, such as upsampling, downsampling, or augmenting the original image.

[0028] Scientific literature describes numerous methods for constructing INR networks for encoding 2D or 3D images. Furthermore, many existing methods exist for compressing and encoding neural networks, such as the MPEG Neural Network Compression (NNC) standard.

[0029] However, an INR network, optimized by the encoder and potentially encoded by the NNC, is insufficient for the decoder to reconstruct the image. During operation, the decoder requires additional information to properly reconstruct the encoded input signal.

[0030] Several embodiments provide information on encoding the information necessary for a decoder to reconstruct an input signal (e.g., image, video, or 3D data). Some variations describe a bitstream containing both an INR network and additional information necessary for signal reconstruction (including SEI syntax). The method by which the decoder uses this information to reconstruct the signal is also described.

[0031] Some embodiments describe the essential information that the bitstream must transmit, regardless of the INR scheme used.

[0032] Figure 2 shows an example of a method 200 for encoding information used by a decoder to reconstruct at least a portion of the signal representing a scene from encoded INRs, according to one embodiment. A bitstream containing an INR network is configured, optimized and encoded using a commercially available method, and the information necessary for the decoder to reconstruct the signal using the INR network is encoded. In 201, the INRs are encoded, for example, using an NNC encoder. The INRs represent at least a portion of the signal representing the scene. This can be a region of an image, a component of an image, a part of a video, or data from a static or dynamic 3D scene. The weights of the INRs are determined using a commercially available method. The INR weights are encoded in the bitstream.

[0033] In 202, one or more syntactic elements (SEs) are also signaled within the bitstream. These one or more syntactic elements provide the decoder with the encoded INR to reconstruct at least a portion of the signal representing the scene.

[0034] Figure 3 shows an example of a method 300 in which a decoder decodes information used to reconstruct at least a portion of the signal representing a scene from an encoded INR, according to one embodiment. For example, a bitstream encoded in the manner described in relation to Figure 2 is decoded. In 301, one or more syntactic elements (SEs) are decoded from the bitstream. In 302, the syntactic elements are provided to the decoder, which uses the syntactic elements and an implicit neural representation network to reconstruct at least a portion of the signal representing a scene.

[0035] Several embodiments illustrating examples of the above syntactic elements are provided below. Also provided below are several embodiments for reconstructing at least a portion of the signal representing the scene using the decoded syntactic elements.

[0036] As described above, the bitstream should contain signals for some or all of the following elements:

[0037] In the modified example, the type of INR used to represent the scene as at least part of the signal is signalized. This signalization relates to the INR used, for example, a single network for the entire image or scene, a per-channel or per-channel group network (e.g., for Y and UV), a per-patch network within the image, or a per-sub-volume network of the scene.

[0038] In another variation, information related to the input to the INR network is signaled. These elements describe the information necessary to constitute the INR input.

[0039] The dimensions of a domain can be represented by a signal; for example, an image is a two-dimensional signal. This describes the dimensions of the output image or scene.

[0040] Input coordinate normalization can be signaled. Coordinate normalization refers to the range of coordinates assumed by the INR. For example, a certain INR has an input value range of [0,1] 2 , (1,···,width)x(1,···,height), or [-1,1] 2 This can be assumed.

[0041] Input coordinate transformations can be signalized. Input coordinates are often transformed by a function before being fed into a neural network. To reconstruct an image, the transformations that must be used must be signalized. If multiple transformations exist, their order may also be signalized.

[0042] The maximum upsampling size can be signaled. This information can be additional. It is interesting to signal the maximum recommended upsampling possible with this INR. Maximum upsampling can be defined in various ways, for example, the maximum upsampling rate that does not result in a distortion increase beyond a set value, or the upsampling rate at which INR upsampling is superior to the upsampling algorithms available in the decoder.

[0043] In another variation, information related to the model is signaled. In practice, the model is usually integerized; that is, weights are quantized and represented as fixed-point values, and floating-point operations are converted to corresponding integer operations. For example, an activation layer (such as sin(x)) may be converted as a lookup table (LUT) when the network is integerized. For this reason, the input / output range and bit depth must be described precisely.

[0044] In another variation, information related to signal reconstruction is quantified as a signal.

[0045] For example, the output format can be signalized. This signalization refers to the format of the image encoding generated by the INR network, such as RGB or YUV. Depending on the encoding, additional signalization may be required. For example, when the YUV format is used, signalization related to chroma subsampling and / or phase may be required.

[0046] The output range of an INR network can be signaled as, for example, [0,255] or [0,1].

[0047] Assuming a quantized network, Table 1 below shows examples of syntax for signaling one or more of the above pieces of information, and Table 1 lists some of the syntactic elements that may be necessary to describe the inputs and outputs of the network.

[0048] [Table 1]

[0049] The semantics of the syntactic elements shown in Table 1 are as follows:

[0050] -inr_model_type: A tag that describes the type of INR approach used to generate the INR output. For example, the following types may be defined: ○inr_model_type=0: The model takes a d-dimensional vector (specified by -inr_input_dimension_count_minus1) as input and outputs an output. The input is sampled uniformly within the range specified for each dimension. ○inr_model_type=1: Samples the input using a 32x downsampling function at the center of a 32x32 patch within the specified range for each dimension. Alternatively, an additional parameter can specify the downsampling range. ○inr_model_type=2: INR includes two neural networks; the first network is applied before the coordinate transformation, and the second network is applied after the coordinate transformation. ○Xxxx: Other approaches may also be specified. -inr_inference_type:Type of operation:A tag that describes whether it is an integer or floating-point operation. This can affect the syntax of several elements. For example, a value of 0 means an integer operation, and 1 means a floating-point operation. -inr_input_dimension_count_minus1: An integer representing the input dimension minus 1. For example, in the case of an image INR model, the input is 2-dimensional (width and height), so inr_input_dimension_count_minus1 is 1. -inr_input_range_minus1[i]: For each dimension of the input, the input range minus 1. The number of valid values ​​will be inr_input_range[i] = inr_input_range_minus1[i] + 1. -inr_input_quantizer[i]: An integer representing the fixed-point integer quantizer for each dimension of the input. For example, if inr_input_quantizer[i]=0, the input x in dimension d represents the value x. If inr_input_quantizer[i]=3, the input x in dimension d represents the value (x / 2 3 ) represents. -inr_input_zero_centered[i]: If the flag is 0 (false), the input in dimension d is in the range [0, inr_input_range[i]], and if the flag is 1 (true), the value is in the range [inr_input_range[i] / 2 - inr_input_range[i], inr_input_range[i] / 2]. Alternatively, the flag inr_input_zero_centered can be replaced by an explicit encoding of the zero value offset.

[0051] For example, a model aimed at encoding an image of size W×H with a subpixel precision of 1 / 8 pixel (i.e., the network can upsample the original image eightfold) can be described with the following parameters:

[0052] -inr_inference_type = 0 -inr_input_dimension_count_minus1=1 -inr_input_range_minus1[0]=W-2 -inr_input_quantizer [0]=3 -inr_input_zero_centered[0]=0 -inr_input_range_minus1 [1]=H-2 -inr_input_quantizer [1]=3 -inr_input_zero_centered[1]=0

[0053] The input coordinate generator, in order to restore the image to its original size, will look like this:

[0054]

number

[0055] Here, infer(x,y) applies model inference using the input coordinates (x,y).

[0056] As another example, consider a model that generates a 3D point cloud with a voxel coordinate precision of 1 / 256 for input coordinates in the range [-1,1].

[0057] -inr_inference_type = 0 -inr_input_dimension_count_minus1=2 -inr_input_range_minus1[0]=1 -inr_input_quantizer [0]=8 -inr_input_zero_centered[0]=1 -inr_input_range_minus1[1]=1 -inr_input_quantizer [1]=8 -inr_input_zero_centered[1]=1 -inr_input_range_minus1[2]=1 -inr_input_quantizer [2]=8 -inr_input_zero_centered[2]=1

[0058] The input coordinate generator is as follows to reconstruct the volume.

[0059]

number

[0060] -inr_input_transformation_count: An integer representing the number of consecutive transformations applied to the input before it is fed into the INR network. For example, if one transformation is applied, the value is 0. If three transformations are applied, the value is 2. -inr_input_transformation_type[i]: A tag that describes the type of transformation applied to the input. Type 0 is left to define custom types. For example, the following types may be defined: ○inr_input_transformation_type[i]=1: Hyperspherical coordinate transformation. ○inr_input_transformation_type[i]=2: Fourier transform mapping using a custom Fourier transform matrix. For this type of value, inr_inference_type must be a float. In this case, the custom matrix is ​​written as follows: ■inr_Fourier_mapping_coefficient_count_minus1[i]: The value obtained by subtracting 1 from the integer representing the number of Fourier transform coefficients. ■inr_Fourier_mapping_coefficient[i][j]: Values ​​of the Fourier transform coefficients. ○inr_input_transformation_type[i]=3: Normalization. ○inr_input_transformation_type[i]=4: Fourier transform using the first predefined Fourier transform mapping matrix. ○inr_input_transformation_type[i]=5: Fourier transform using a second predefined Fourier transform mapping matrix. ○inr_input_transformation_type[i]=6: Fourier transform using a predefined Fourier transform LUT. ○ etc. -inr_output_type: Specifies the output type. Type 0 is used to define a custom type. Other types can be predefined, for example, as follows: ○inr_output_type=1: RGB output encoded in 8 bits, with each component in the range of [0,255]. ○inr_output_type=2: RGB output encoded in 10 bits, with each component in the range of [0, 1023]. ○inr_output_type=3: YUV output encoded in 8 bits, with each component in the range of [0,255]. ○inr_output_type=4: YUV output encoded in 10 bits, with each component in the range of [0, 1023]. ○inr_output_type=5: RGBA output (encodes each component in the range of [0,255] using 8 bits), where A represents the alpha value. ○ etc.

[0061] If the type is 0, the output characteristics are described using the same logic as the input (number of components, range, quantizer, zero-value offset).

[0062] For example, a model that does not perform any conversion and whose output RGB is in the range of [0,255] is described by the following parameters.

[0063] -inr_input_transformation_count =0 -inr_output_type=1

[0064] In floating-point INR, the coordinates are mapped using a custom Fourier mapping with three coefficients a, b, and c, then normalized, and the output is in YUV format. This process is described as follows:

[0065] -inr_input_transformation_count =2 -Inr_input_transformation_type[0]=2 -inr_Fourier_mapping_coefficient_count_minus1[0]=2 -inr_Fourier_mapping_coefficient[0][0]=a -inr_Fourier_mapping_coefficient[0][1]=b -inr_Fourier_mapping_coefficient[0][2]=c -Inr_input_transformation_type[1]=3 -inr_output_type=3

[0066] The transformation can typically be represented as a lookup table (LUT) for integer inference. Please note the following. For example, based on the previous example involving Fourier mapping (3D point cloud reconstruction), a certain signal decoding can be calculated as follows:

[0067]

number

[0068] The LUT_transform function returns the mapped coordinates to the "mapped_coord" variable. One way to construct a LUT_transform for Fourier mapping is as follows: Assume a set of Fourier mapping coefficients Fourier_mapping_coef and an input range [start,end].

[0069]

number

[0070] Here, len(Fourier_mapping_coef) returns the length of the input set Fourier_mapping_coef, and get_range(x) returns the range in which a real number x is converted to an integer representation. For example, if the uniform quantization step is δ and x'=inv_quantization(x) is a real number converted to x, then typically get_range(x)=(x'-δ / 2', x'+δ / 2). sin_int_rep(range,coef) is a function that returns the integer representation of the values ​​of the sine function in this interval. For example, the following may be used:

[0071]

number

[0072] cos_int_rep(range, coef) can be defined similarly.

[0073]

number

[0074] Furthermore, it is possible to add other exponents, such as π or its multiples, to the sin and cos expressions. Moreover, by utilizing the fact that cos(x) = sin(π-x), it is possible to avoid storing a table for cosine values ​​and instead store a LUT_complementary_angle table that stores the integer representation of π-x. In that case, for example, it is possible to use LUT_table_c[x][i] := LUT_table_s[LUT_complementary_angle table [x]][i].

[0075] Next, LUT_transform(x,y,z) can be implemented as follows:

[0076]

number

[0077] In some embodiments, the syntactic elements described in Figures 2 and 3 may also include network configuration information. For example, the neural network encoding may be signaled. This element signals how the INR neural network is encoded within the bitstream. For example, the value may be associated with a specific format, such as NNC or Open Neural Network Exchange format, and / or a specific version of the format.

[0078] An inference engine configuration can be signalized. Adding signalization related to the inference engine configuration may be interesting and could include, for example, the precision used for operation, the memory required to store the INR network, or the inference engine to use.

[0079] Table 2 below lists some of the syntactic elements that may be necessary to describe a network configuration.

[0080] [Table 2]

[0081] The semantics for Table 2 are as follows:

[0082] inr_mode_idc indicates the format used to encode the neural network. For example, a value of 0 indicates that this SEI message contains an ISO / IEC 15938-17 bitstream, and a value of 1 indicates that this SEI message contains a neural network encoded in the format identified by the inr_tag_uri tag URI.

[0083] inr_tag_uri contains a tag URI having the syntax and semantics specified in IETF RFC 4151 that identifies the format and related information of the INR network or the updates encoded herein.

[0084] inr_uri contains a URI with syntax and semantics defined in IETF Internet Standard 66 that identifies a neural network used as an INR network, or an update to that network.

[0085] The `inr_complexity_info_present_flag` flag specifies whether there are syntactic elements indicating the complexity of the INR network. For example, a value of 1 indicates that there are one or more syntactic elements indicating the complexity of the INR network, while a value of 0 indicates that there are no syntactic elements indicating the complexity of the INR network.

[0086] If inr_parameter_type_idc is 0, it indicates that the neural network will only use integer parameters. If inr_parameter_type_flag is 1, it indicates that the neural network may use floating-point or integer parameters. If inr_parameter_type_idc is 2, it indicates that the neural network will only use binary parameters. If inr_parameter_type_idc is 3, it is reserved for future use.

[0087] If inr_log2_parameter_bit_length_minus3 is 0, 1, 2, or 3, it indicates that the neural network will not use parameters with bit lengths exceeding 8, 16, 32, or 64, respectively. If inr_parameter_type_idc exists and inr_log2_parameter_bit_length_minus3 does not exist, the neural network will not use parameters with a bit length greater than 1.

[0088] `inr_num_parameters_idc` indicates the maximum number of neural network parameters in the INR network in powers of 2 (or another power of 2). A value of 0 for `inr_num_parameters_idc` indicates that the maximum number of neural network parameters is unknown. The value of `inr_num_parameters_idc` ranges from 0 to 63 (inclusive).

[0089] If the value of inr_num_parameters_idc is greater than 0, the variable maxNumParameters is derived as follows:

[0090]

number

[0091] The number of neural network parameters in the post-processing filter is less than or equal to maxNumParameters.

[0092] If inr_num_mac_operations_idc is greater than 0, it indicates that the maximum number of multiplication operations per inference for the INR network is less than or equal to inr_num_mac_operations_idc. If inr_num_mac_operations_idc is 0, it indicates that the maximum number of multiplication operations for the network is unknown. The value of inr_num_mac_operations_idc is between 0 and 2. 32 The range is up to -1 (including both endpoints).

[0093] If inr_total_kilobyte_size is greater than 0, it indicates the total size (in kilobytes) required to store the uncompressed parameters of the neural network. The total size (in bits) is equal to or greater than the sum of the bits used to store each parameter. inr_total_kilobyte_size is the total size in bits divided by 8000 and rounded to the nearest whole number. If inr_total_kilobyte_size is 0, it indicates that the total size required to store the parameters of the neural network is unknown. The value of inr_total_kilobyte_size is between 0 and 2 32 The range is up to -1 (including both endpoints).

[0094] inr_reserved_zero_bit_b is 0 in a bitstream conforming to the syntax defined herein. The decoder ignores INR SEI messages where inr_reserved_zero_bit_b is not 0.

[0095] inr_payload_byte[i] contains the i-th byte of the bitstream in accordance with ISO / IEC 15938-17. The byte sequence of inr_payload_byte[i] for all current values ​​of i is the complete bitstream in accordance with ISO / IEC 15938-17.

[0096] The syntactic elements listed above are merely examples; other syntactic elements may be added, or some may be removed. Furthermore, the order in which the syntactic elements are presented is for illustrative purposes only; one or more syntactic elements can be expressed in any order.

[0097] Figure 4 provides a high-level overview of exemplary methods 400 on which a decoder may rely to decode a bitstream 410 containing encoded INRs representing at least a portion of the signal representing the scene. Here, embodiments are described for signals representing an image. However, the embodiments described are applicable to any other kind of signal representing a scene. The bitstream also contains one or more syntactic elements, for example, as described above in relation to Table 1 or Table 2.

[0098] The input bitstream 410 or a portion thereof is supplied to the process modules. Module 420 extracts information from the bitstream and generates and outputs an input coordinate set for the INR network. Module 430 performs INR inference. This module is responsible for decoding the INR network and generating the output of the INR network, i.e., associating the output of the INR network with each input coordinate set. Module 440 combines these outputs and reconstructs the image. This may include transformations to conform to the requested output format. However, this module is also responsible for appropriately positioning these values ​​in the image format.

[0099] Figure 5 shows one possible embodiment of the input generation module 520. In step 510, the size of the original image, represented by a bitstream, is taken in as input 515 and used to generate input coordinates corresponding to the original image. For example, these input coordinates are typically pairs of positions (x,y) where x and y each range from 1 to the width or height of the image. Alternatives are possible depending on the type of INR used; for example, coordinates associated with a pixel block may be used. For example, for an INR that outputs the color value of a pixel block, the input would typically be the coordinates of that block. Such an INR is defined, for example, with inr_model_type=1 and uses the syntax provided below, in which case the input coordinates are subsampled accordingly.

[0100] Step 530 takes signals in the bitstream related to the coordinate transformation 535 as input and applies these transformations to the coordinates provided in step 510.

[0101] Step 550 arranges the coordinates based on the type of INR555 signaled in the bitstream. This step may also be additional. This step ensures that the appropriate value is used for each input of the INR network. This module outputs the coordinates 560 used in the INR network.

[0102] Figure 6 shows one embodiment of the INR inference module 430. In step 610, the INR network encoding signaled in bitstream 615 is used as input to prepare for decoding. This information is used to configure the INR decoder module 620, which takes the bitstream of the INR network 625 (i.e., encoded INR) as input and outputs the decoded INR network.

[0103] Module 630 initializes the inference engine 640 using the inference engine configuration 635 and the decoded INR network. Module 630 may be involved in steps such as allocating computational resources, loading the decoded model into the memory of the processor that will perform the inference, transforming the network into the format expected by the inference engine, and adapting network weights for the inference engine. It may also be involved in the procedure of defining which coordinates are used in which network if multiple networks are used in the INR. In that case, the input 635 of module 630 may also include a signal indicating the type of INR to be used.

[0104] The initialized inference engine 640 receives coordinates 560 as input. These coordinates are fed into the INR network, where inference is performed to calculate the values ​​associated with these pixels. Several variations of inference are possible. For example, inference can be performed on one set of coordinates at a time, or on multiple sets of coordinates simultaneously, or in parallel using multiple processors. This module outputs the calculated coordinates and associated pixel values ​​660.

[0105] Figure 7 shows one embodiment of the inference output merging module 440. Module 710 receives coordinates and associated pixel values ​​660. It may also take the type of INR used and the network output format as input (716). This module is responsible for rearranging the pixel values ​​into a conventional image format (e.g., a row-major or column-major matrix). If necessary, this module may also accept the type of INR used as input. For example, input coordinates may be associated with pixel block values ​​(to determine the output order), or a step of merging the outputs of multiple INR networks may be involved. In another example, different networks may each generate Y and UV values.

[0106] Module 720 receives the range of output values ​​from network 726 as input. It then performs a coordinate transformation on the pixel values ​​to obtain values ​​that fall within a standard range for the image (e.g., [0,255] or [0,1]). The final output 730 is the decoded image.

[0107] This output can be further modified to obtain different image formats (e.g., YUV to RGB, or vice versa).

[0108] The process described above is just one example. Depending on the INR type considered or indicated in the bitstream, additional steps may be required, or some steps may be omitted.

[0109] The order of some steps may also differ. For example, upsampling may be performed independently of input range generation 420. Although a sequential process is described, some operations may be performed in parallel. For example, network decoding 620 may be performed simultaneously with coordinate preparation 420. Also, some steps may be moved to different modules.

[0110] Figure 8 is a block diagram showing an example of a system in which various configurations and embodiments can be implemented. System 800 may be embodied as a device comprising various components described below and configured to perform one or more embodiments described in this application. Examples of such devices include, but are not limited to, various electronic devices such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances, and servers. The components of System 800 may be embodied individually or in combination as a single integrated circuit, a plurality of ICs, and / or discrete components. For example, in at least one embodiment, the processing elements and encoder / decoder elements of System 800 are distributed across a plurality of ICs and / or discrete components. In various embodiments, System 800 is communicably connected to other systems or other electronic devices, for example, via a communication bus or through dedicated input and / or output ports. In various embodiments, System 800 is configured to implement one or more embodiments described in this application.

[0111] System 800 includes, for example, at least one processor 810 configured to execute instructions loaded therein to implement various embodiments described in this application. Processor 810 may include embedded memory, input / output interfaces, and various other circuits known in the art. System 800 includes at least one memory 820 (e.g., a volatile memory device and / or a non-volatile memory device). System 800 includes a storage device 840 which may include, but is not limited to, an EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash memory, a magnetic disk drive, and / or an optical disk drive, including non-volatile memory and / or volatile memory. Storage device 840 may, in non-limiting examples, include an internal storage device, a connected storage device, and / or a network-accessible storage device.

[0112] System 800 includes, for example, an encoder / decoder module 830 that processes and provides signals representing at least a portion of the signal representing the scene, and / or encoded INR representing at least a portion of the signal representing the scene, and / or decoded INR representing at least a portion of the signal representing the scene, and / or at least a portion of the signal reconstructed from the decoded INR, and the encoder / decoder module 830 may include its own processor and memory. The encoder / decoder module 830 represents a module included in the device to perform encoding and / or decoding functions. As is well known, the device may include either an encoding module or a decoding module, or both. Furthermore, the encoder / decoder module 830 may be implemented as an independent component of System 800, or it may be incorporated into the processor 810 as a combination of hardware and software well known to those skilled in the art.

[0113] To perform the various embodiments described in this application, program code loaded into the processor 810 or encoder / decoder 830 may be stored in the storage device 840, then loaded into the memory 820 and executed by the processor 810. According to various embodiments, one or more of the processor 810, memory 820, storage device 840, and encoder / decoder module 830 may store one or more of various items during the execution of the processes described in this application. Such stored items may include, but are not limited to, input images, videos, 3D data, INR weights, decoded images, decoded videos, decoded 3D data or portions of decoded data, bitstreams, matrices, variables, and intermediate or final results obtained from the processing of equations, formulas, operations, and arithmetic logic.

[0114] In some embodiments, memory within the processor 810 and / or the encoder / decoder module 830 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, memory located outside the processing device (for example, the processing device is either the processor 810 or the encoder / decoder module 830) is used for one or more of these functions. The external memory may be, for example, memory 820 and / or storage device 840, i.e., dynamic volatile memory and / or non-volatile flash memory. In some embodiments, external non-volatile flash memory is used to store the television's operating system.

[0115] Inputs to the elements of System 800 may be provided through various input devices, as shown in Block 805. Such input devices include, but are not limited to, (i) a radio frequency (RF) section that receives, for example, radio frequency (RF) signals transmitted in the air by a broadcaster, (ii) a component (COMP) input terminal (or set of COMP input terminals), (iii) a universal serial bus (USB) input terminal, and / or (iv) a high-definition multimedia interface (HDMI) input terminal. Other examples not shown in Figure 8 include composite video.

[0116] In various embodiments, the input devices of block 805 are associated with corresponding input processing elements, as known in the prior art. For example, the RF section may be associated with elements suitable for processing: (i) selecting a predetermined frequency (also called signal selection, or band-limiting a signal to a specific frequency band); (ii) down-converting the selected signal; (iii) re-band-limiting to a narrower frequency band (e.g., to select a signal frequency band that can be called a channel in a particular embodiment); (iv) demodulating the down-converted and band-limited signal; (v) performing error correction; and (vi) demultiplexing to select a data packet stream of interest. The RF section in various embodiments may include one or more elements for performing these functions, such as frequency selectors, signal selectors, band limiters, channel selectors, filters, downconverters, demodulators, error correctors, and demultiplexers. The RF section may include a tuner that performs several of these functions, such as down-converting a received signal to a lower frequency (e.g., an intermediate frequency or quasi-baseband frequency) or baseband. In one embodiment of a set-top box, the RF section and associated input processing elements receive an RF signal transmitted via a wired (e.g., cable) medium and perform frequency selection to a desired frequency band by filtering, down-converting, and re-filtering. Various embodiments rearrange the order of the above (and other) elements, remove some of these elements, or add other elements that perform similar or different functions. Adding elements includes inserting elements between existing elements, such as inserting an amplifier or an analog-to-digital converter. In various embodiments, the RF section includes an antenna.

[0117] Furthermore, the USB and / or HDMI terminals may include their respective interface processors for connecting the system 800 to other electronic devices via USB and / or HDMI connections. It should be understood that various aspects of input processing, such as Reed-Solomon error correction, may be implemented as needed, for example, within a separate input processing IC or within the processor 810. Similarly, aspects of USB or HDMI interface processing may be implemented as needed, within a separate interface IC or within the processor 810. The demodulated, error-corrected, and multiplexed streams are provided to various processing elements, such as the processor 810 and an encoder / decoder 830 operating in conjunction with memory and storage elements, for data stream processing necessary for display on an output device.

[0118] Various components of system 800 may be housed within an integrated housing, within which the various components may be interconnected, and data may be transmitted between them using appropriate connection means 815 (e.g., internal buses, wiring, printed circuit boards, etc., including an I2c bus, which are well known in the art).

[0119] System 800 includes a communication interface 850 that enables communication with other devices via a communication channel 890. The communication interface 850 may include, but is not limited to, a transceiver configured to send and receive data via the communication channel 890. The communication interface 850 may include, but is not limited to, a modem or a network card, and the communication channel 890 may be implemented, for example, in a wired and / or wireless medium.

[0120] In various embodiments, data is streamed to system 800 using a Wi-Fi network such as IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers). In these embodiments, the Wi-Fi signal is received via a communication channel 890 and a communication interface 850 adapted for Wi-Fi communication. In these embodiments, communication channel 890 is typically connected to an access point or router that provides access to an external network, including the Internet, enabling streaming applications and other OTT communications. Other embodiments provide streaming data to system 800 using a set-top box that distributes data via an HDMI connection on input block 805. Yet another embodiment provides streaming data to system 800 using an RF connection on input block 805. As described above, various embodiments provide data in non-streaming manners. In addition, various embodiments use wireless networks other than Wi-Fi (e.g., cellular networks or Bluetooth networks).

[0121] System 800 can provide output signals to various output devices, including a display 865, a speaker 875, and other peripheral devices 885. In various embodiments, the display 865 includes, for example, one or more of a touchscreen display, an organic light-emitting diode (OLED) display, a curved display, and / or a foldable display. The display 865 may be for a television, tablet, laptop, mobile phone, or other device. The display 865 may be integrated with other components (e.g., in a smartphone) or separate (e.g., an external monitor for a laptop). In various embodiments, the other peripheral devices 885 include, in various embodiments, one or more of a standalone digital video disc (or digital multipurpose disc) (DVR, both terms), a disc player, a stereo system, and / or a lighting system. Various embodiments use one or more peripheral devices 885 that provide functionality based on the output of System 800. For example, a disc player performs the function of playing back the output of System 800.

[0122] In various embodiments, control signals are transmitted between the system 800 and the display 865, speaker 875, or other peripheral devices 885 using signaling such as AV.Link, CEC, or other communication protocols that enable device-to-device control with or without user intervention. Output devices may be coupled to the system 800 via dedicated connections through their respective interfaces 860, 870, and 880. Alternatively, output devices may be connected to the system 800 using a communication channel 890 via a communication interface 850. The display 865 and speaker 875 may be integrated into a single unit with other components of the system 800 in an electronic device (e.g., a television). In various embodiments, the display interface 860 includes a display driver, such as a timing controller (TCon) chip.

[0123] The display 865 and speaker 875 may, alternatively, be isolated from one or more other components, such as when the RF portion of input 805 is part of a separate set-top box. In various embodiments where the display 865 and speaker 875 are external components, the output signal may be provided via a dedicated output connection, such as an HDMI port, a USB port, or a COMP output.

[0124] The embodiments can be implemented by computer software, hardware, or a combination of hardware and software implemented by the processor 810. In non-limiting examples, the embodiments can be implemented by one or more integrated circuits. The memory 820 can be any type appropriate for the technical environment and, in non-limiting examples, can be implemented using any appropriate data storage technology such as optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory. The processor 810 can be any type appropriate for the technical environment and, in non-limiting examples, can include one or more microprocessors, general-purpose computers, dedicated computers, and processors based on multi-core architectures.

[0125] Figure 9 is a block diagram of a system in which an aspect of this embodiment may be implemented according to another embodiment. Figure 9 shows one embodiment of a device 900 for encoding or decoding metadata used by an INR decoder according to any embodiment described herein. The device includes a processor 910 which is interconnectable to a memory 920 via at least one port. Both the processor 910 and the memory 920 may also have one or more additional interconnections to external connections.

[0126] The processor 910 is configured to encode at least one implicit neural representation network representing at least a portion of the signal representing the scene, and to signal one or more syntactic elements using any of the embodiments described herein to reconstruct at least a portion of the signal representing the scene using the encoded implicit neural representation network.

[0127] In another modification, the processor 910 is configured to decode one or more syntactic elements that specify the use of an implicit neural representation network representing at least a portion of the signal representing the scene, and to reconstruct at least a portion of the signal representing the scene using one or more syntactic elements and the implicit neural representation network, in which case any embodiment described herein is used. For example, the processor 910 is configured using a computer program product that includes code instructions implementing any embodiment described herein.

[0128] In the embodiment shown in Figure 10, in a transmission context between two remote devices A and B via a communication network NET, device A includes a processor associated with memory RAM and ROM configured to implement a method for encoding one or more syntactic elements for use with an INR decoder, as described in relation to Figure 1-7, and device B includes a processor associated with memory RAM and ROM configured to implement a method for decoding one or more syntactic elements for use with an INR decoder, as described in relation to Figure 1-7. For example, the network is a broadcast network adapted to broadcast / transmit the encoded INR and one or more syntactic elements from device A to a decoding device including device B. In some embodiments, the encoded INR and one or more syntactic elements are transmitted on the same signal. In other embodiments, the encoded INR and one or more syntactic elements are transmitted separately on separate signals.

[0129] Figure 11 shows an example of the syntax of a signal transmitted via a packet-based transmission protocol. Each transmitted packet P includes a header H and a payload PAYLOAD. In some embodiments, the payload PAYLOAD may include one or more syntactic elements for using an INR decoder, as in any of the embodiments described above.

[0130] In some embodiments, one or more syntactic elements include at least one of the following: instructions relating to an implicit neural representation network, information provided to constitute the input to the implicit neural representation network, or information relating to the output of the implicit neural representation network.

[0131] In one modified example, the representation related to the implicit neural representation network includes at least one of the following: the model type of the implicit neural representation network, the type of computation of the implicit neural representation network, the encoding format of the implicit neural representation network, a tag URI indication that identifies the format of the implicit neural representation network, a URI indication that identifies the implicit neural representation network, an indicator indicating whether complexity information related to the implicit neural representation network exists, a specification of the type of parameters used by the implicit neural representation network, a specification related to the bit length of the parameters used by the implicit neural representation network, an indicator indicating the maximum number of parameters of the implicit neural representation network, a specification of the maximum number of multiplication and accumulation operations per inference of the implicit neural representation network, or a size indication for storing the uncompressed parameters of the implicit neural representation network.

[0132] In another variation, the information for configuring the input to an implicit neural representation network includes at least one of the following: an indication of the number of dimensions of the input, an indication of the range of dimensions of the input, an indication of the quantizer of the dimensions of the input, an indication of the zero-value offset in the range, an indication of the number of transformations applied to the input, an indication of the types of transformations applied to the input, or an indication of one or more parameters of the transformations.

[0133] Various embodiments involve decoding. As used in this application, “decode” encompasses all or part of the processes performed on the received encoded INR to produce, for example, a final output suitable for display. In various embodiments, such processing includes one or more of the processes typically performed by a decoder (e.g., entropy decoding, inverse quantization, inverse transform, differential decoding). In various embodiments, such processing additionally or alternatively includes processing performed by the decoder of the various embodiments described in this application (e.g., entropy decoding of a binary symbol sequence for reconstructing image, video, or 3D data).

[0134] Please note that the syntactic elements used herein are descriptive terms; therefore, they do not preclude the use of other syntactic element names.

[0135] This disclosure describes various types of information that can be transmitted or stored, such as syntax. This information can be packaged or placed in various ways, including methods common in image, video, and neural network standards, such as storing it in SPS, PPS, NAL units, headers (e.g., NAL unit headers and slice headers), and SEI messages. Other methods are also available, including methods common in system-level or application-level standards, which involve storing the information in one or more of the following:

[0136] a. SDP (Session Description Protocol). This is a format for describing multimedia communication sessions for purposes such as session announcements and session invitations. For example, it is defined in RFCs and used in combination with RTP (Real-Time Transport Protocol) transmission.

[0137] b. DASH MPD (Media Representation Description) descriptors. For example, descriptors used in DASH and transmitted over HTTP are associated with a representation or set of representations and provide additional characteristics to the content representation.

[0138] c. RTP header extension. For example, used during RTP streaming.

[0139] d. ISO-based media file formats. For example, the format used by OMAF, which uses boxes, which are object-oriented building blocks, defined by a unique type identifier and length, and are also called "atoms" in some definitions.

[0140] e. An HLS (HTTP Live Streaming) manifest sent via HTTP. The manifest can be associated with, for example, a version or set of versions of content and can provide characteristics of that version or set of versions.

[0141] When a diagram is presented as a flowchart, it should be understood that a corresponding block diagram of the device is also provided. Similarly, when a diagram is presented as a block diagram, it should be understood that a corresponding method / process flowchart is also provided.

[0142] The embodiments and aspects described herein can be implemented, for example, as methods or processes, apparatus, software programs, data streams, or signals. Even when discussed in the context of a single embodiment (for example, when discussed only as a method), embodiments of the discussed features can be implemented in other forms (for example, apparatus or programs). Apparatus can be implemented, for example, with appropriate hardware, software, and firmware. Methods can be implemented, for example, with a processor, which generally refers to a processing device, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. Processors include, for example, communication devices such as computers, mobile phones, portable / personal digital assistants (PDAs), and other devices that facilitate information communication between end users.

[0143] References to “one embodiment,” “an embodiment,” “one example,” “an example,” and other variations of the expression mean that the specific functions, structures, characteristics, etc. described in relation to that embodiment are included in at least one embodiment. Therefore, expressions such as “in one embodiment,” “in one example,” “in one embodiment,” and “one example,” and all other variations of the expression appearing in various parts of this application do not necessarily all refer to the same embodiment.

[0144] In addition, this application may refer to the "determination" of various types of information. Determining information may include one or more processes such as estimating information, calculating information, predicting information, or retrieving information from memory.

[0145] Furthermore, this application may refer to "access" to various types of information. Access to information may include one or more actions such as receiving information, retrieving information (e.g., retrieving from memory), storing information, moving information, duplicating information, calculating information, determining information, predicting information, or estimating information.

[0146] Furthermore, this application may refer to the "reception" of various types of information. Reception, like "access," is intended as a broad term. Reception of information includes, for example, access to information and retrieval of information (e.g., retrieval from memory). Moreover, "reception" is generally involved in some form in operations such as storing, processing, transmitting, moving, copying, erasing, calculating, deciding, predicting, and estimating information.

[0147] The use of expressions such as " / ", "and / or", and "at least one" is intended to encompass the cases of selecting only the first listed option (A), or only the second listed option (B), or both options (A and B), for example, in the cases of "A / B", "A and / or B", and "at least one of A and B". As further examples, expressions such as "A, B, and / or C" and "at least one of A, B, and C" are intended to encompass the cases of selecting only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first listed option and the second listed option (A and B), or only the first listed option and the third listed option (A and C), or only the second listed option and the third listed option (B and C), or all three options (A, B, and C). This can be extended to a number of items listed, as will be obvious to those skilled in the art in the technical field and related technical fields.

[0148] Furthermore, as used herein, the term “signaling” refers, in particular, to instructing a corresponding decoder to do something. In this way, in one embodiment, the same parameters are used on both the encoder and decoder sides. For example, the encoder can transmit a specific parameter to the decoder (explicit signaling), allowing the decoder to use the same specific parameter. Conversely, if the decoder already holds that specific parameter or other parameters, it can simply recognize and select that specific parameter using signaling (implicit signaling) without transmitting it. Bit saving is achieved in various embodiments by not transmitting any actual function. Note that signaling can be achieved in various ways. For example, in various embodiments, one or more syntactic elements, flags, etc., are used to transmit information to the corresponding decoder. The foregoing refers to the verb form “signaling,” but “signal” may also be used as a noun in this specification.

[0149] As those skilled in the art will understand, implementations can generate various signals formatted to transmit, for example, information that can be stored or transmitted. This information may include, for example, instructions for performing a method or data generated by any of the described embodiments. For example, a signal can be formatted to transmit a bitstream of the described embodiment. Such a signal can be formatted, for example, as an electromagnetic wave (e.g., using the radio frequency portion of the spectrum) or as a baseband signal. The formatting includes, for example, encoding a data stream or modulating a carrier wave with the encoded data stream. The information transmitted by the signal can be, for example, analog or digital information. The signal can be transmitted over various wired or wireless links, as is well known. The signal can be stored in a processor-readable medium.

[0150] Several embodiments have been described above. The features of these embodiments can be provided individually or in any combination across various categories and types of claims.

Claims

1. A method comprising signalizing one or more syntactic elements, which provides for reconstructing at least a portion of a signal representing a scene, and using at least one implicit neural representation network representing at least a portion of the signal representing the scene.

2. A device comprising one or more processors, the one or more processors capable of signalizing one or more syntactic elements, which provides for reconstructing at least a portion of a signal representing a scene, using at least one of an implicit neural representation network representing at least a portion of the signal representing the scene.

3. A method comprising decoding one or more syntactic elements, which provides for reconstructing at least a portion of a signal representing a scene, and using at least one implicit neural representation network representing at least a portion of the signal representing the scene.

4. The method according to claim 3, further comprising reconstructing at least a portion of the signal representing the scene using the one or more syntactic elements and the implicit neural representation network.

5. A device comprising one or more processors, the one or more processors capable of decoding one or more syntactic elements, which provides for reconstructing at least a portion of a signal representing a scene, using at least one of an implicit neural representation network representing at least a portion of the signal representing the scene.

6. The method according to claim 5, wherein the one or more processors can further operate to reconstruct the one or more syntactic elements and the implicit neural representation network of the signal representing the scene.

7. The aforementioned one or more syntactic elements are: Instructions related to the implicit neural representation network, Information that provides for configuring the input to the implicit neural representation network, or Information related to the output of the implicit neural representation network, A method according to any one of claims 1, 3, or 4, or an apparatus according to any one of claims 2, 5, or 6, comprising at least one of the above.

8. The instructions relating to the implicit neural representation network are, The model type of the implicit neural representation network mentioned above, The type of computation of the implicit neural representation network, The encoding format of the implicit neural representation network, An instruction for a tag URI that identifies the format of the implicit neural representation network, Instructions for a URI that identifies the implicit neural representation network, An indicator that indicates whether complexity information related to the implicit neural representation network exists, The type of parameter used by the implicit neural representation network, Instructions relating to the bit length of the parameters used by the implicit neural representation network, An indicator that indicates the maximum number of parameters of the implicit neural representation network, The instruction for the maximum number of multiplication and integration operations per inference of the implicit neural representation network, or A size instruction for storing uncompressed parameters of an implicit neural representation network. The method or apparatus according to claim 7, comprising at least one of the following.

9. The information that provides for configuring the input to the implicit neural representation network is, The number of dimensions of the input is indicated. Instructions for the range of dimensions of the input, Instructions for the quantizer of the aforementioned input dimension, An indication of the offset of the zero value within the aforementioned range, Instructions regarding the number of transformations applied to the aforementioned input, An instruction for the type of conversion applied to the aforementioned input, or, Instructions relating to one or more parameters of the aforementioned conversion, The method or apparatus according to claim 7 or 8, comprising at least one of the following.

10. Reconstructing at least a portion of the signal representing the scene using the aforementioned one or more syntactic elements and the implicit neural representation network is: Generating input from one or more syntactic elements, The generated input is provided to the implicit neural representation network, A method according to any one of claims 4, 7 to 9, or an apparatus according to any one of claims 6 to 9, comprising the above.

11. Reconstructing at least a portion of the signal representing the scene using the aforementioned one or more syntactic elements and the implicit neural representation network is: Performing implicit neural representation network inference using one or more of the aforementioned syntactic elements, A method according to any one of claims 4, 7 to 10, or the apparatus according to any one of claims 6 to 10, further comprising the above.

12. The method or apparatus according to claim 11, wherein performing the implicit neural representation network inference using the one or more syntactic elements comprises decoding the implicit neural representation network using the one or more syntactic elements.

13. The method or apparatus according to claim 11 or 12, wherein performing the implicit neural representation network inference comprises configuring an inference engine using the one or more syntactic elements.

14. Reconstructing at least a portion of the signal representing the scene using the aforementioned one or more syntactic elements and the implicit neural representation network is: Using one or more syntactic elements, combine the output of the implicit neural representation network with the input of the implicit neural representation network. A method according to any one of claims 4, 7 to 13, or the apparatus according to any one of claims 6 to 13, further comprising the above.

15. The method or apparatus according to claim 14, wherein combining the output of the implicit neural representation network with the input of the implicit neural representation network using one or more syntactic elements comprises arranging the output of the implicit neural representation network into a predetermined format or within a predetermined range of values.

16. The method according to claim 1, further comprising encoding, or the apparatus according to claim 2, wherein one or more processors can further perform encoding the at least one implicit neural representation network.

17. A computer program product comprising instructions for causing one or more processors to perform the method described in any one of claims 1, 3, 4, 7 to 15.

18. An executable program instruction, stored in a non-temporary computer-readable medium for a computer, which contains the program instruction for performing the method according to any one of claims 1, 3, 4, 7 to 15.

19. A bitstream comprising data representing one or more syntactic elements, which provides for reconstructing at least a portion of a signal representing a scene, and using at least one implicit neural representation network representing at least a portion of the signal representing the scene.

20. The bitstream according to claim 19, further comprising data encoding at least a portion of the implicit neural representation network, or data for updating at least one of the implicit neural representation networks.

21. The bitstream according to claim 19 or 20, wherein the one or more syntactic elements comprise at least one of instructions relating to the implicit neural representation network, information providing to constitute an input to the implicit neural representation network, or information relating to an output to the implicit neural representation network.

22. The instructions relating to the implicit neural representation network are, The model type of the implicit neural representation network mentioned above, The type of computation of the implicit neural representation network, The encoding format of the implicit neural representation network, An instruction for a tag URI that identifies the format of the implicit neural representation network, Instructions for a URI that identifies the implicit neural representation network, An indicator that indicates whether complexity information related to the implicit neural representation network exists, The type of parameter used by the implicit neural representation network, Instructions relating to the bit length of the parameters used by the implicit neural representation network, An indicator that indicates the maximum number of parameters of the implicit neural representation network, The instruction for the maximum number of multiplication and integration operations per inference of the implicit neural representation network, or A size instruction for storing uncompressed parameters of an implicit neural representation network. The bitstream according to claim 21, comprising at least one of the following.

23. The information that provides for configuring the input to the implicit neural representation network is, The number of dimensions of the input is indicated. Instructions for the range of dimensions of the input, Instructions for the quantizer of the aforementioned input dimension, An indication of the offset of the zero value within the aforementioned range, Instructions regarding the number of transformations applied to the aforementioned input, An instruction for the type of conversion applied to the aforementioned input, or, Instructions relating to one or more parameters of the aforementioned conversion, The bitstream according to claim 21 or 22, comprising at least one of the following.

24. A non-temporary computer-readable medium for storing the bitstream described in any one of claims 19 to 24.

25. It is a device, The apparatus according to claim 3 or 4, (i) an antenna configured to receive or transmit a signal, wherein the signal includes a bitstream as described in any one of claims 19 to 24, or (ii) a band limiter configured to restrict the signal to a frequency band including the bitstream, or (iii) a display configured to display at least a portion of the signal representing the scene, A device equipped with the following features.

26. The device according to claim 25, wherein the device comprises at least one of a television, a mobile phone, a tablet, or a set-top box.

Images